Home

anglocide edited this page Dec 16, 2016 · 44 revisions
Clone this wiki locally

Introduction

Got Your Back (GYB) is a command line tool that backs up and restores your Gmail account. This page provides simple instructions for downloading, installing and starting to use GYB.

GYB works with Gmail.com and Google Apps accounts.

Step 1: Download GYB

Windows Users

Head to the releases page and download the latest Windows version of GYB, do not download the Python Source version (unless you really know what you're doing).

Mac and Linux Users

Head to the releases page and download the latest Python source version of GYB, do not download the Windows version.

Step 2: Extract GYB

Windows Users

Use the archive extraction tool of your choice to extract the files from the GYB .zip you downloaded. Windows XP and higher have a built in tool that works just fine. When specifying where to extract GYB, I suggest extracting directly to C:\ so that the files will reside in C:\gyb. Once completed, GYB should exist at C:\gyb\gyb.exe

Mac and Linux Users

Use the archive extraction tool of your choice to extract the files from the GYB .zip you downloaded. I suggest extracting the files to a sub-folder of your home directory.

Step 3: Running GYB for the First Time

Windows Users

Open a command prompt on your computer. You can do this by going to Start -> Programs -> Accessories -> Command Prompt or by opening the Run... dialog on the start menu and typing CMD then enter. Now change to the directory where you extracted GAM. The command to change directories looks like:

cd \gyb

this works if you extracted GYB to c:\gyb. If you extracted it elsewhere, specify that location instead. Now type:

gyb --email youremail@gmail.com --action estimate

except use your real email address in place of youremail@gmail.com. GYB will open up a web page in order for you to grant access to your Gmail account. This authorization makes it possible for GYB to connect to your Google Account for Gmail data only, GYB will have no rights to any of your other Google Data. Make sure you are logged in to the Google account you specified before granting access. Once you've granted access, switch back to the command prompt window and hit enter. If no errors are printed, GYB should start estimating the size of your Gmail mailbox. Note that GYB only estimates the size of messages in the All Mail folder, it does not check Spam or Trash although these do count against your Gmail quota displayed at the bottom of your Gmail inbox. To accurately compare GYB's estimate and the Gmail inbox web page quota display, first empty your Trash and Spam folders.

Congratulations, you're up and running with GYB! You probably want to move on to performing a backup now.

Mac and Linux Users

Open up a terminal window on your computer. On Linux, this is generally under Accessories -> Terminal. On Mac, it's under Applications -> Utilities -> Terminal. Now change to the directory where you extracted GYB. Try:

cd ~/gyb

this will work if you extracted the GYB files to a subfolder named gyb in your home directory. If you extracted them elsewhere, replace ~/gyb with the full path to them. Now run:

python3 gyb.py --email youremail@gmail.com --action estimate

If you get an error about python not being a valid program, make sure you have the Python 3 interpreter installed on your machine. All Macs and most Linux installs should include Python 3 but if not, you may need to research how to install it on your OS/Distribution.

GYB will open up a web page in order for you to grant GYB access to your Gmail account. This authorization makes it possible for GYB to connect to your Google Account Gmail data only, GYB will have no rights to any of your other Google Data. Make sure you are logged in to the Google account you specified before granting access. Once you've granted access, switch back to the command prompt window and hit enter. If no errors are printed, GYB should start estimating the size of your Gmail mailbox. Note that GYB only estimates the size of messages in the All Mail folder, it does not check Spam or Trash although these do count against your Gmail quota displayed at the bottom of your Gmail inbox. To accurately compare GYB's estimate and the Gmail inbox web page quota display, first empty your Trash and Spam folders.

Instead of needing to type "python3 gyb.py" for every command, you can mark the gyb.py file as executable or we can use the alias command to shorten it to just "gyb":

alias gyb="python3 gyb.py"

Now when we can just type commands like:

gyb --email myemail@gmail.com --action estimate

you'll need to type the alias command each time you open a Terminal to run GYB.

Congratulations, you're up and running with GYB! You probably want to move on to performing a backup now.

Step 4: Performing A Backup

A basic GYB backup is very easy to start. Just run:

gyb --email youremail@gmail.com --action backup

the "--action backup" is not strictly necessary since GYB defaults to backing up if an action is not specified. Assuming you've already granted GYB access to your Gmail messages, GYB will load the access token from youremail@gmail.com.cfg and use it to get access to your messages. By default, GYB will download and save all messages to a folder named "GYB-GMail-Backup-youremail@gmail.com". You can specify another folder for GYB to use with the --local-folder argument:

gyb --email youremail@gmail.com --local-folder "C:\Users\John\Documents\Johns_Gmail_Backup"

GYB will keep you update you as the backup progresses.

Step 5: Performing a Restore

Restores on GYB are also very simple:

gyb --email youremail@gmail.com --action restore --local-folder "c:\my_gmail_backup"

the specified folder should exist and should have been used in a previous GYB backup. If not specified, restores default to using the GYB-GMail-Backup-youremail@gmail.com folder just like the backup does. GYB will connect to your Gmail account and perform the restore of all messages in the backup folder.

Note that if you perform a restore to the same Gmail account, GYB will not create duplicate messages, instead you'll only see messages restored which were backed up by GYB then later deleted from the Gmail account.

If you want to restore messages to an account other than the one you backed up, it's necessary to specify the backup folder. As an example,

gyb --email newaddress@gmail.com --action restore --local-folder GYB-GMail-Backup-oldaddress@gmail.com

will look for messages in the backup of the oldaddress@gmail.com account but restore them to the newaddress@gmail.com account.

You can also use the "--label-restored NEWLABELNAME" argument to set a label on all restored messages. For example:

gyb --email newaddress@gmail.com --action restore --local-folder GYB-GMail-Backup-oldaddress@gmail.com --label-restored "Old Address"

will restore the message, always including an extra label of "Old Address" on the restored messages.

Selective Backups With Gmail Searching

GYB supports selective backups using Gmail style mailbox searches. For example, suppose you wanted to only backup important or starred messages:

gyb --email youremail@gmail.com --search "is:important OR is:starred"

would cause GYB to only backup messages matching that search query. Virtually any Gmail search will work with GYB. The only exception being that specifying "in:anywhere" will not backup Trash and Spam, there's currently no way to backup Trash and Spam with GYB. See here for a detailed article on all of the possible Gmail Search parameters.

Note that Gmail searches also work with the "--action estimate" command. Suppose you wanted to know how much space emails with .PDF attachments are using in your Gmail mailbox:

gyb --email youremail@gmail.com --action estimate --search "filename:PDF"

will estimate the size of messages with PDF attachments only. Try substituting DOC, JPG, ZIP and other common file attachments for PDF.

Advanced options

--search

Optional: On backup, estimate, count and purge, Gmail search to scope operation against.

--local-folder

On backup, restore, estimate, local folder to use. Default is GYB-GMail-Backup-

--label-restored

On restore, all messages will additionally receive this label. For example, "--label_restored gyb-restored" will label all uploaded messages with a gyb-restored label.

--strip-labels

On restore and restore-mbox, strip existing labels from messages except for those explicitly declared with the --label-restored parameter.

--noresume

GYB keeps a record of messages restored to each account and will pick up where it left off should the restore not finish. The --noresume switch will make GYB ignore messages already restored and restart the restore at the beginning.

--fast-incremental

By default, GYB will refresh the stored labels and flags for messages that have already been backed up, just in case they changed after the backup. This step can be skipped by supplying the --fast-incremental switch on the command line.

--action restore-mbox

Restore mbox files that you've exported from Gmail Takeout, Google Vault, GAM Email Audit Exports or any other MBOX format file you have.

Use the --local-folder option to specify the path where you've extracted all of your mbox files. GYB will restore messages from all .mbox and .mbx files in the directory and any sub-directories.

--action restore-group

Google Apps for Work and Education only. This feature allows you to restore messages to a Google Group rather than a user mailbox. It's important to note that:

  • Message labels, read/unread status, stars and other metadata are not preserved with restore-group.
  • There is no API or method to extract or backup messages stored in Google Groups. GYB can restore messages to a group but cannot backup message in a group, it's a one-way process.
  • The Groups Migration API supports a maximum message size of 16mb so not all Gmail-stored messages can be imported into a group.
  • Groups have no quota! If you're okay with the above issues, you can offload an unlimited amount of data to a group. This may be a good solution for users approaching their Gmail quota.

This option requires both the --service-account and --use-admin option to be specified. The --email option should be the Google Group to restore messages into. Archiving for the group should be enabled.

A good use case for restore-group would be a user who is nearing Gmail quota. You could do a selective backup of the user's mailbox with a GYB backup using --search before:2011/04/13 smaller:16M to get only messages older than 2 years and smaller than 16mb. Then restore the messages to a Google Group and give the user exclusive access to the new group. Finally, free up the user's mailbox by performing a purge using the same search parameters. I'd also recommend holding on to the local backup of the user's mail should you ever wish to restore to the mailbox.

--use-admin

Specify the Google Apps admin to utilize when restoring messages to a group with --action restore-group. This user should be a super administrator, delegated admins do not have sufficient privileges to perform group restores.

--action count

Just count the number of messages in a user mailbox. Note, to compare this number to what you see in Gmail, you should turn conversation mode off in general settings and search for "-is:chat". This ensures you are counting individual messages (not conversations) and that archived chats which are not backed up by GYB by default are not counted.

--action purge

DANGEROUS!!! This option will completely delete messages. Running this command without a --search parameter will EMPTY YOUR ENTIRE MAILBOX. This removes messages from Trash and Spam folders also so there is no ability to restore from the mailbox itself. Use this option with extreme caution. You have been warned. It is highly recommended to do a backup before a purge.

--action purge-labels

DANGEROUS!!! This option will delete all user labels for the mailbox. No messages will be deleted but everything will be left unlabeled!

--batch-size

By default, GYB grabs the full content of 100 messages at a time for backup. If the mailbox has many very large messages, it may take a very long time for GYB to backup anything as it could be pulling down up to 2500mb (100 messages x 25mb each) of data for each batch. Try specifying something smaller like --batch-size 4. The batch size range is 1-100 as of version 0.44.

--fast-restore

Perform a faster restore of messages. It's important to note that when performing a fast restore, restored messages will not be threaded into Gmail conversations nor will they be de-dupped. This makes viewing and managing the messages in the mailbox at a later time much more difficult.

--vault

On restore and --fast-restore, skips adding restored messages to the user's visible Gmail mailbox and only lets the messages be visible to Google Vault. This option is meant mostly for system administrator who wish to have the restored messages be a part of the user's Vault discovery but not their visible mailbox.

--spam-trash

Include messages in the Spam and Trash folders for backup, estimate and count actions. This allows these messages to be acted upon where normally they would be skipped.

--service-account

Use a Google Service Account to authenticate rather than standard 3-legged OAuth authentication. This option is only for Google Apps for Business and Education users. See below for details.

Google Apps for Work and Education admins: Backup, Restore and Estimate Users and Restore to Groups

If you're using Google Apps for Work or Education, it's possible to use GYB with your users without needing to know their password. This works because GYB makes use of a special Google Apps feature called Service Accounts.

There are a few steps involved with creating and authorizing a service account for GYB.

  1. Go to the Google Developers Console
  2. Select Yes and click "Agree and continue". It will take a moment for the project to be created.
  3. Click "Go to credentials"
  4. Click "New credentials" and choose "Service account key".
  5. Click "Select..." and choose "New service account".
  6. Give your service account a name like "GYB Service account".
  7. Keep JSON as key type. Click "Create".
  8. Open the file in a text editor and look for the line showing something like:

    "client_id": "107634805914295539364",

    in this example, 107634805914295539364 is your Client ID. Remember this value for later steps.

  9. Your browser will download a .json file. Save the file with a name of oauth2service.json and put it in the same folder as gyb.py or gyb.exe.
  10. Click "Manage service accounts" to the right.
  11. Click the 3 dots to the right of your service account. Choose Edit.
  12. Place a checkmark next to "Enable Google Apps Domain-wide Delegation" and Save.
  13. Go to your Google Apps Admin console (https://admin.google.com)
  14. Click Security, Show more, Advanced settings.
  15. Click Manage API Client Access
  16. For Client Name, enter the Client ID from above.
  17. For API Scopes, enter exactly:

    https://mail.google.com/,https://www.googleapis.com/auth/apps.groups.migration,https://www.googleapis.com/auth/drive.appdata

    Your service account setup is complete.

Now you can run GYB with the --service-account option. Try running:

gyb --email yourusersemail@yourcompany.com --service-account

WARNING: Service Accounts offer very powerful control over your Google Apps domain. Do not use this option on a computer you do not trust! Do not leave the oauth2service.json file in places where others can find it! If you suspect your Service Account has been stolen, delete the API project in the API console and unauthorize its access to your domain.

Troubleshooting

Below you'll find some possible errors and what they could mean.

access_denied

This can occur if you've failed to add the proper API scopes (mail, groups migration) for a service account.

invalid_request

This can occur if you're attempting to download mail from an account that is suspended; only active accounts can be backed up via Got Your Back.