Skip to content
Barman (PostgreSQL backup server) S3 archive scripts
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Type Name Latest commit message Commit time
Failed to load latest commit information.


S3 archive scripts for Barman PostgreSQL Backup Server

About BarmanS3

BarmanS3 consists of 2 separate processes - basebackup sync and WAL archive sync. It uses AWS CLI utility to perform S3 folder sync actions while using script wrappers for orchestration and automation. Currently it consists of 3 scripts, single shared config file and a cron file:

  • /opt/barmans3/bin/barmans3-wal-sync: for syncing server WAL archive from Barman to S3 bucket
  • /opt/barmans3/bin/barmans3-base-sync: for syncing server basebackup (ie full backup) from Barman to S3 bucket
  • /opt/barmans3/bin/barmans3-full-sync: for doing full base & wals sync together with delete (ie cleanup)
  • /opt/barmans3/bin/barmans3-queue-add: barman post-backup hook-script for creating basebackup sync task in barmans3-base-sync queue
  • /opt/barmans3/bin/barmans3-base-ls: listing basebackups from S3 bucket
  • /opt/barmans3/barmans3.conf: shared config file holding S3 URIs amongst other things
  • /etc/cron.d/barmans3: cron entries for automated barmans3-wal-sync and barmans3-base-sync execution

Theory of Operation

  1. barmans3-queue-add script is added to main-server barman configuration as post-backup hook script - where barman passes (full) backup run details - once barman backup is completed
  2. barmans3-queue-add creates basebackup S3 sync task as: /var/run/barmans3/queue.d/<backup_id>.run
  3. barmans3-base-sync is run from cron (once-per-day) looking for <backup_id>.run tasks to execute aws cli s3 sync for basebackup taken by barman
  4. sync task is transferred into /var/run/barmans3/queue.d/<backup_id>.running state while executing
  5. if sync task errors or task input params were incorrect (ie barman backup errored for example) - then task is trasferred into /var/run/barmans3/queue.d/<backup_id>.error state (and not processed any further)
  6. if sync task completes with no errors on its awscli return status - then task is put into /var/run/barmans3/queue.d/<backup_id>.done state (and kept for history, no automated removal yet)
  7. barmans3-wal-sync is run from cron every 15 minutes and requiring server name as an argument - syncing server WALs from Barman to S3 bucket
  8. sync scripts outputs are directed to relevant log files under /var/log/barmans3 (redirection configured in /etc/cron.d/barmans3)


We assume CentOS 7 Barman host here - althou BarmanS3 scripts should work also with other Linux OSes.


  • AWS CLI utility
  • GIT utility
  • rsync utility

Satisfying barmans3 dependencies

# EPEL7 repo is required to install awscli
yum -y install epel-release

# install awscli and other utilities
yum -y install awscli git screen rsync

Deploying BarmanS3

# clone barmans3 git repo
git clone
cd barmans3


# review default /opt/barmans3/barmans3.conf contents for:
vi /opt/barmans3/barmans3.conf

# switch to barman user
su - barman
# create aws-cli config and S3 credential files
mkdir ~/.aws
chmod 700 ~/.aws

# review: all params 
cat << EOF > ~/.aws/config
output = text
s3 =
    max_concurrent_requests = 24
    multipart_threshold = 256MB
    multipart_chunksize = 128MB
chmod 600 ~/.aws/config
# change: aws_access_key_id
# change: aws_secret_access_key
cat << EOF > ~/.aws/credentials
aws_access_key_id = <aws_key_id>
aws_secret_access_key = <aws_access_key>
chmod 600 ~/.aws/credentials


Manual operation

# verifying aws-cli operation
# load configuration file variables
source /opt/barmans3/barmans3.conf

# list bucket contents (might be empty)
aws --endpoint-url $S3_ENDPOINT_URI s3 ls $S3_BUCKET_URI
# list bucket contents with debug enabled
aws --debug --endpoint-url $S3_ENDPOINT_URI s3 ls $S3_BUCKET_URI
# initial (manual) sync of base backups
# launch task inside screen
time aws --endpoint-url $S3_ENDPOINT_URI s3 sync /var/data/barman/main-server/base/ s3://backup/barman/main-server/base/
# initial (manual) sync of WALs
time aws --endpoint-url $S3_ENDPOINT_URI s3 sync /var/data/barman/main-server/wals/ s3://backup/barman/main-server/wals/


# add post-backup hook script configuration for barman main-server config
echo "post_backup_script = /opt/barmans3/bin/barmans3-queue-add" >> /etc/barman.d/main-server.conf
# add cron entry for sync automation
# redirect output to logfiles
# and remove upload stats lines from log output
cat << 'EOF' > /etc/cron.d/barmans3
# m    h  dom mon dow   user     command
  */15 *    *   *   *   barman   /opt/barmans3/bin/barmans3-wal-sync main-server | sed 's/\o015/\n/g' | grep -v 'file(s) remaining' >> /var/log/barmans3/walsync.log 2>&1
  5    23   *   *   *   barman   /opt/barmans3/bin/barmans3-base-sync | sed 's/\o015/\n/g' | grep -v 'file(s) remaining' >> /var/log/barmans3/basesync.log 2>&1
  0    3    1   *   *   barman   /opt/barmans3/bin/barmans3-full-sync main-server | sed 's/\o015/\n/g' | grep -v 'file(s) remaining' > /var/log/barmans3/fullsync.log 2>&1
# reload crond 
# NOT required if cron runs with inotify (like CentOS 7 crond.service does)
systemctl reload crond.service
systemctl status crond.service
You can’t perform that action at this time.