Transferatu

Transferatu moves your data from one place to another. As a service.

Currently supported transfer methods:

• postgres -> postgres
• postgres -> S3
• S3 -> postgres

Transferatu is a pliny app.

Setup

Transferatu is designed as a Heroku app:

$ heroku create <your-app-name>

It needs a Heroku API token (to manage its workers) and AWS credentials to access S3. The best way to create the API token is with the oauth plugin.

$ heroku authorizations:create --description transferatu ----scope read-protected,write-protected
Created OAuth authorization.
  ID:          105a7bfa-34c3-476e-873a-b1ac3fdc12fb
  Description: transferatu
  Token:       <your-token>
  Scope:       read-protected,write-protected

Create an S3 bucket and then create an AWS user with an access policy restricted to read and write that bucket and its contents:

{
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "s3:*",
      "Resource": [
        "arn:aws:s3:::<your-transferatu-bucket>",
        "arn:aws:s3:::<your-transferatu-bucket>/*"
      ]
    }
  ]
}

Set the config vars related to these, as well as the app name (Transferatu needs to know this for run process management), and bucket name:

$ heroku config:set HEROKU_API_TOKEN=<your-token> \
  HEROKU_APP_NAME=<your-app-name> \
  AWS_ACCESS_KEY_ID=<your-role-id> \
  AWS_SECRET_ACCESS_KEY=<your-role-key> \
  S3_BUCKET_NAME=<your-transferatu-bucket>
Setting config vars and restarting <your-app-name>... done, v56

Transferatu also needs a Postgres database:

$ heroku addons:add heroku-postgresql:premium-yanari

Once everything is set up, you can deploy, run a schema migration to set up the database, and scale up the clock process:

$ git push heroku master
...
$ heroku run bundle exec rake db:migrate
$ heroku ps:scale clock=1 scheduler=1

Quiescence

Occasionally, you may want to have your backup workers take a breather and pause transfer processing (e.g., when performing maintenance on the app). To this end, Transferatu includes a mechanism to stop accepting new jobs:

$ heroku run bundle exec rake quiescence:enable

You can optionally cancel any in-progress transfers as well:

$ heroku run bundle exec rake quiescence:enable[hard]

To restore normal operation, disable quiescence:

$ heroku run bundle exec rake quiescence:disable

Note also that Transferatu workers are designed to run as standalone heroku run processes. This makes it possible to deploy changes without interrupting in-progress transfers. However, it does mean that workers can end up running out-of-date code. To work around this, workers are automatically soft-quiesced (and, naturally, restarted) after every code push.