Skip to content
Switch branches/tags
Go to file

Latest commit


Git stats


Failed to load latest commit information.
Latest commit message
Commit time

Zenko Backbeat

backbeat logo


Backbeat is an engine with a messaging system at its heart. It's part of Zenko, Scality’s Open Source Multi-Cloud Data Controller. Learn more about Zenko at

Backbeat is optimized for queuing metadata updates and dispatching work to long-running tasks in the background. The core engine can be extended for many use cases, which are called extensions, as listed below.


Asynchronous Replication

This feature replicates objects from one S3 bucket to another S3 bucket in a different geographical region. The extension uses the local Metadata journal as the source of truth and replicates object updates in a FIFO order.



This guide assumes the following:

  • Using MacOS
  • brew is installed (get it here)
  • node is installed (version 6.9.5)
  • yarn is installed (version 3.10.10)
  • aws is installed (version 1.11.1)

Run kafka and zookeeper

Install kafka and zookeeper

brew install kafka && brew install zookeeper

Make sure you have /usr/local/bin in your PATH env variable (or wherever your homebrew programs are installed):

echo 'export PATH="$PATH:/usr/local/bin"' >> ~/.bash_profile

Start kafka and zookeeper servers

mkdir ~/kafka && \
cd ~/kafka && \
curl | tar xvz && \
sed 's/zookeeper.connect=.*/zookeeper.connect=localhost:2181\/backbeat/' \
kafka_2.11- > \

Start the zookeeper server:

zookeeper-server-start ~/kafka/kafka_2.11-

In a new shell, start the kafka server:

kafka-server-start ~/kafka/kafka_2.11-

Create a zookeeper node and kafka topic

In a new shell, connect to the zookeeper server with the ZooKeeper chroot /backbeat path:

zkCli -server localhost:2181/backbeat

Create the replication-populator node:

create /replication-populator my_data

We may leave the zookeeper server now:


Create the backbeat-replication kafka topic:

kafka-topics --create \
--zookeeper localhost:2181/backbeat \
--replication-factor 1 \
--partitions 1 \
--topic backbeat-replication

Run Scality Components

Start Vault and Scality S3 servers

Start the Vault server (this requires access to the private Vault repository):

git clone ~/replication/vault && \
cd ~/replication/vault && \
yarn i && \
chmod 400 ./tests/utils/keyfile && \

In a new shell, start the Scality S3 server:

git clone ~/replication/s3 && \
cd ~/replication/s3 && \
yarn i && \
S3BACKEND=file S3VAULT=scality yarn start

Setup replication with backbeat

In a new shell, clone backbeat:

git clone ~/replication/backbeat && \
cd ~/replication/backbeat && \
yarn i

Now, create an account and keys:

VAULTCLIENT=~/replication/backbeat/node_modules/vaultclient/bin/vaultclient && \
$VAULTCLIENT create-account \
--name backbeatuser \
--email dev@null \
--port 8600 >> backbeat_user_credentials && \
$VAULTCLIENT generate-account-access-key \
--name backbeatuser \
--port 8600 >> backbeat_user_credentials && \
cat backbeat_user_credentials

Output will look something like (this output is stored for reference in the file backbeat_user_credentials):

    "id": "8CFJQ2Z3R6LR0WTP5VDS",
    "value": "gB53GM7/LpKrm6DktUUarcAOcqHS2tvKI/=CxFxR",
    "createDate": "2017-08-03T00:17:57Z",
    "lastUsedDate": "2017-08-03T00:17:57Z",
    "status": "Active",
    "userId": "038628340774"

Store the account's credentials using the "id" and "value" fields:

aws configure --profile backbeatuser

The completed prompt should look like:

AWS Access Key ID [None]: 8CFJQ2Z3R6LR0WTP5VDS
AWS Secret Access Key [None]: gB53GM7/LpKrm6DktUUarcAOcqHS2tvKI/=CxFxR
Default region name [None]:
Default output format [None]:

Set up replication on your buckets:

node ~/replication/backbeat/bin/setupReplication.js setup \
--source-bucket source-bucket \
--source-profile backbeatuser \
--target-bucket target-bucket \
--target-profile backbeatuser

Run the backbeat queue populator:

yarn --prefix ~/replication/backbeat run queue_populator

In a new shell, run the backbeat queue processor:

yarn --prefix ~/replication/backbeat run queue_processor

You are now ready to put data on source-bucket and watch it replicate to target-bucket!

Put an object on the source-bucket:

echo 'content to be replicated' > replication_contents && \
aws s3api put-object \
--bucket source-bucket \
--key object-to-replicate \
--body replication_contents \
--endpoint http://localhost:8000 \
--profile backbeatuser

Check the replication status of the object we have just put:

aws s3api head-object \
--bucket source-bucket \
--key object-to-replicate \
--endpoint http://localhost:8000 \
--profile backbeatuser

The object's "ReplicationStatus" should either be "PENDING", or if some time has passed, then it should be "COMPLETED".

Check if the object has been replicated to the target bucket:

aws s3api head-object \
--bucket target-bucket \
--key object-to-replicate \
--endpoint http://localhost:8000 \
--profile backbeatuser

After some time, the object's "ReplicationStatus" should be "REPLICA". 😺


In our $HOME directory, we now have the following directories:

├── kafka
│   └── kafka_2.11-
├── replication
    ├── backbeat
    ├── s3
    └── vault


Zenko Backbeat is the core engine for asynchronous replication, optimized for queuing metadata updates and dispatching work to long-running tasks in the background.





No packages published