Turbolift, the Cloud Files Uploader

Date:	2013-09-05 09:51
tags:	rackspace, upload, mass, Cloud Files, files, api
category:	*nix

Turbolift Files to Rackspace Cloud Files -Swift-

General Overview

If you have found yourself in a situation where you needed or wanted to upload a whole bunch of files to Cloud Files quickly, this is what you are looking for. Turbolift is an assistant for uploading files to the the Rackspace Cloud Files Repository with a bunch of options.

You'll also probably want to read The Rackspace Cloud Files API guide (PDF) : The guide will provide you an overview of all of the available functions for Cloud Files.

The Process by which this application works is simple. All you have to do is Literally fill in the command line arguments and press enter. The application is a multiprocessing Cloud Files Uploader which will upload any directory or file provided to it.

Functions of the Application :

• Upload a directory, (recursively)
• Upload a single file
• Upload a local directory (recursively) and sync it with a Cloud Files Container
• Download a Container to a local directory
• Download changed objects from a Container to a local directory
• Compresses a Local Directory, then uploads it
• List all Containers
• List all Objects in a Container
• Delete an Object in a Container
• Delete an entire Container
• Clone one Container in one region to another Container in the same or Different Region.
• Set Custom headers on Objects/Containers

Turbolift can be managed with a config file. The option --system-config references a config file. Additionally, the Environment Variable TURBO_CONFIG can be used to reference a config file as well. All of Turbolift's options can be set in the config file. This makes managing Turbolift very simple.

Please read command_line_args for more information on Command Line Arguments and functions.

If you would like to setup Turbolift to use an environment variable file I would recommend you have a look at the turboliftrc_example file and the environment_vars document.

Please also review the Turbolift Wiki for more information.

Prerequisites :

• For all the things to work right please make sure you have python-dev.
• All systems require the python-setuptools package.
• Python => 2.6 but < 3.0
• A File or some Files you want uploaded

Installation :

Installation is simple:

git clone git://github.com/cloudnull/turbolift.git
cd turbolift
python setup.py install

Installation NOTICE :

If you are installing on a system that is not Running Python => 2.6 but < 3.0, please review the document INSTALL_EMBED. This is a tested installation of Turbolift on Ubuntu 8.04, which ships with Python 2.5. The guide is a simple walk through for compiling stand alone python and installing Turbolift in that stand alone installation.

Python Modules Needed, for full operation :

NOTE : All of the modules needed should be part of the "standard" Library as of Python 2.6. The setup file will install the two needed dependencies which may not have made it onto your system.

Application Usage

Here is the Basic Usage

turbolift -u [CLOUD-USERNAME] -a [CLOUD-API-KEY] --os-rax-auth [REGION] upload -s [PATH-TO-DIRECTORY] -c [CONTAINER-NAME]

Run turbolift -h for a full list of available flags and operations

Turbolift can easily be set to run on via script or as a CRON job, please see, turbolift_example_script for more ideas / information on using Turbolift as a script.

Systems Tested on

The application has been tested on :

• Debian 6
• Ubuntu 10.04 - 12.04
• Mac OS X 10.8
• CentOS[RHEL] 6

Known Limitations:

1. Turbolift does not use SLO / DLO on files larger than 5GB in size. If you have a file that you would like uploaded that is larger than 5GB in size you will need to chunk the file prior to uploading at this time. Please refer to the SLO / DLO docs for more information on Large Object storage.

For information on Benchmarks from my own testing, please have a look here at the benchmarks file.

Turbolift is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation. The License in service for this program is GPLv3. please see http://www.gnu.org/licenses/gpl-3.0.txt for more information.