Switch branches/tags
Nothing to show
Find file Copy path
Fetching contributors…
Cannot retrieve contributors at this time
122 lines (88 sloc) 6.62 KB
title description keywords kbid weight date updated_at author
Universal Object Storage seeding
Initial backup into object storage when operating with massive data is very slow and consumes a lot of bandwidth. This KB describes simple seeding S3 compatible storage on-prem.
backup seeding, S3 compatible


Initial full backup is always painful and may take significant amount of time when offloading data to object storage over Internet uplink. Amazon Web Services (AWS) has Snowball service (basically it is removable storage they ship to customer for initial seeding, and it should be returned back once data is on it and they seed it for further incremental runs). Other providers have their options to seed data. Cloudberry Backup supports out-of-the-box AWS Snowball, but not for other S3 compatible providers. Data seeding is not that complicated, but from the other hand Cloudberry requires certain folders / files structure of is backups, what is not possible to accomplish over simple data offload into directly attached storage that can be shipped further to storage provider (e.g. Windows / Windows file systems does not support colon symbol in files / folder names and this is required by Cloudberry backup structure).


Seeding to object storage can be done over S3 Compatible appliance (Virtual Machine) that customers can download and use as middleware between data and colon supported storage (e.g. NAS with linux file system).


  • Download Cloudberry S3 compatible appliance;
    • It is tiny linux machine (Debian 8 jessie);
    • It is with based system, no UI;
    • It is with SSH server;
    • It is with Minio S3 compatible as our seeding middleware;
    • It is with v2/v4 S3 API signature support.
  • Get NAS (or other storage) with file system that supports colon (consult with your storage provider / NAS supplier);
  • Prepare your backup strategy (e.g. backup plan groups, consider encryption / compression policies).

Configuration of Appliance

S3 compatible appliance is available in VMware Virtual Machine (OVF 2.0) form-factor. Deploy new virtual machine (can be on one of your VMware ESXi or VMware Workstation). Request another form-factor from or if you experience issues with downloading this archive once this one does not work or you don't have certain facilities to run it.

Cloudberry Backup initial seeding

Start the Virtual Appliance and login with cloudberry:Pa$$w0rd. Elevate your permissions to root if you experience issues with mounting shares (Yes, this is not best practice) with root:Pa$$w0rd. Mount your remote share (NAS) with the following example:

    mount -t nfs your_IP:/folder /mnt/nas/
    ls /mnt/nas // check your mount, create / delete files / folders, just to make sure you have permissions

Go to your home (cloudberry user) directory (/home/cloudberry/) and execute the following:

    ./minio server /mnt/nas/

    Endpoint:  http://:9000  http://:9000
    Region:    us-east-1

    Browser Access:
       http://:9000  http://:9000

    Command-line Access:
       $ mc config host add myminio http://:9000  

    Object API (Amazon S3 compatible):

This means, that you have your S3 compatible server within Appliance running with v4 Signature support. Now we need proxy, do the following:

    cd $GOPATH/bin
    ./s3v2tov4-proxy -l :8000 -f http://localhost:9000 -access  -secret 

If you don't see any errors, you are set now with S3 compatible middleware and ready to do rest of configuration in Cloudberry lab products.

Configuration of Cloudberry Backup

Download and install (if you have not done yet) Cloudberry Backup for Windows (or Mac / Linux). And start configuartion of your storage account.

Cloudberry Backup configuration seeding

Next step is to configure your backup plan (Image Based, File / folder etc), use your recently created storage account for this. Run backup plan and make sure you have data written to your NAS storage and the structure is not changed (you should be able to see colon element in folders / files). For example in path with files where disk letter (e.g. C:). For example:

    cloudberry@cloudberry:/mnt/qnap/backups/backup/CBB_DC$ tree .
    |-- C:
    |   |-- :acl:
    |   |   `-- 20160902151528
    |   |       `-- 8bae0af739186d8461df58591a9720f1
    |   `-- Distribs
    |       |-- bootable.iso:
    |       |   `-- 20160726133432
    |       |       |-- bootable.iso

Seeding is ready and can be shipped to storage provider.

Reuse seeding and keep incremental runs

Your storage provider should offload data to datacenter and provider credentials and endpoint to connect. Configure new object storage endpoint accordingly (in the similar way we did for the seeding storage). Reconfigure your backup plan by changing storage account. You may need to "Syncronize Repository" in «Tools» → «Options» → «Repository» in order to list existing backup data and fit it with Cloudberry SQLite backend.

Nota bene!

Please consult with team prior you go seeding!