Universal Object Storage seeding
Initial backup into object storage when operating with massive data is very slow and consumes a lot of bandwidth. This KB describes simple seeding S3 compatible storage on-prem.
backup seeding, S3 compatible
Initial full backup is always painful and may take significant amount of time when offloading data to object storage over Internet uplink. Amazon Web Services (AWS) has Snowball service (basically it is removable storage they ship to customer for initial seeding, and it should be returned back once data is on it and they seed it for further incremental runs). Other providers have their options to seed data. Cloudberry Backup supports out-of-the-box AWS Snowball, but not for other S3 compatible providers. Data seeding is not that complicated, but from the other hand Cloudberry requires certain folders / files structure of is backups, what is not possible to accomplish over simple data offload into directly attached storage that can be shipped further to storage provider (e.g. Windows / Windows file systems does not support colon symbol in files / folder names and this is required by Cloudberry backup structure).
Seeding to object storage can be done over S3 Compatible appliance (Virtual Machine) that customers can download and use as middleware between data and colon supported storage (e.g. NAS with linux file system).
- Download Cloudberry S3 compatible appliance;
- It is tiny linux machine (Debian 8 jessie);
- It is with based system, no UI;
- It is with SSH server;
- It is with Minio S3 compatible as our seeding middleware;
- It is with v2/v4 S3 API signature support.
- Get NAS (or other storage) with file system that supports colon (consult with your storage provider / NAS supplier);
- Prepare your backup strategy (e.g. backup plan groups, consider encryption / compression policies).
Configuration of Appliance
S3 compatible appliance is available in VMware Virtual Machine (OVF 2.0) form-factor. Deploy new virtual machine (can be on one of your VMware ESXi or VMware Workstation). Request another form-factor from email@example.com or if you experience issues with downloading this archive once this one does not work or you don't have certain facilities to run it.
Start the Virtual Appliance and login with
cloudberry:Pa$$w0rd. Elevate your permissions to root if you experience issues with mounting shares (Yes, this is not best practice) with
Mount your remote share (NAS) with the following example:
mount -t nfs your_IP:/folder /mnt/nas/ ls /mnt/nas // check your mount, create / delete files / folders, just to make sure you have permissions
Go to your home (cloudberry user) directory (/home/cloudberry/) and execute the following:
This means, that you have your S3 compatible server within Appliance running with v4 Signature support. Now we need proxy, do the following:
cd $GOPATH/bin ./s3v2tov4-proxy -l :8000 -f http://localhost:9000 -access -secret
If you don't see any errors, you are set now with S3 compatible middleware and ready to do rest of configuration in Cloudberry lab products.
Configuration of Cloudberry Backup
Download and install (if you have not done yet) Cloudberry Backup for Windows (or Mac / Linux). And start configuartion of your storage account.
Next step is to configure your backup plan (Image Based, File / folder etc), use your recently created storage account for this. Run backup plan and make sure you have data written to your NAS storage and the structure is not changed (you should be able to see colon element in folders / files). For example in path with files where disk letter (e.g. C:). For example:
cloudberry@cloudberry:/mnt/qnap/backups/backup/CBB_DC$ tree . . |-- C: | |-- :acl: | | `-- 20160902151528 | | `-- 8bae0af739186d8461df58591a9720f1 | `-- Distribs | |-- bootable.iso: | | `-- 20160726133432 | | |-- bootable.iso
Seeding is ready and can be shipped to storage provider.
Reuse seeding and keep incremental runs
Your storage provider should offload data to datacenter and provider credentials and endpoint to connect. Configure new object storage endpoint accordingly (in the similar way we did for the seeding storage). Reconfigure your backup plan by changing storage account. You may need to "Syncronize Repository" in «Tools» → «Options» → «Repository» in order to list existing backup data and fit it with Cloudberry SQLite backend.
Please consult with firstname.lastname@example.org team prior you go seeding!