Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP
tree: 7a8d8e642e
Fetching contributors…

Cannot retrieve contributors at this time

498 lines (341 sloc) 18.407 kB
============
TKLBAM-FAQ
============
--------------------------
Frequently Asked Questions
--------------------------
:Author: Liraz Siri <liraz@turnkeylinux.org>
:Date: 2013-11-07
:Manual section: 7
:Manual group: backup
GENERAL QUESTIONS
=================
Is TKLBAM open source?
----------------------
Yes, TKLBAM is licensed under the GPL3. You don't have to care about
free software ideology to appreciate the advantages. Any code running on
your server doing something as critical as encrypted backups should be
available for peer review and modification.
Where can I use TKLBAM?
-----------------------
Using TKLBAM on TurnKey Linux provides the best experience but it will also
work with any Debian or Ubuntu derived system and even with other Linux
distributions if you use the --skip-packages option to disable integration with
APT, the Debian package manager.
TKLBAM was originally designed to work with TurnKey Linux, where it takes
advantage of an appliance's fixed installation state to make the smallest
possible backup and make migration easier. The design has since been extended
to work well with any Debian derived system, including Ubuntu.
In principle it should backup/restore files and databases on any Linux system
(e.g., you can always install it and its dependencies from source), it just
doesn't know how to talk with non-APT package management systems (yet).
What can I use for backup storage?
----------------------------------
Pretty much anything, though storing backups to Amazon S3 is easiest because
authentication and key management are automatic. You just need to run::
tklbam-backup
But you can also backup to any storage target supported by TKLBAM's
back-end Duplicity including the local filesystem, NFS, Rsync, SSH, FTP,
WebDAV, Rackspace CloudFiles and even IMAP.
The local filesystem is one of the easier storage targets to use because
you don't need to mess around with authentication credentials.
So assuming you want to store your backup at /mnt/otherdisk::
tklbam-backup --address file:///mnt/otherdisk/tklbam/backup
tklbam-escrow /mnt/otherdisk/tklbam/key
And restore like this::
tklbam-restore --address file:///mnt/otherdisk/tklbam/backup \
--keyfile=/mnt/otherdisk/tklbam/key
Not as easy as the Hub-enabled "automatic" mode, but still vastly easier than
your conventional backup process. The disadvantage is that you won't be able to
restore/test your backup in the cloud, or from a VM running in another office
branch (for example). Also keep in mind that a physical hard disk, even a RAID
array, provides much lower data reliability compared with Amazon S3.
For this reason we recommend users use local backups to supplement cloud
backups (e.g., providing fast local access).
Which databases are supported?
------------------------------
Currently, only MySQL and PostgreSQL have built-in support but TKLBAM can work
with other databases so long as you configure custom
serialization/unserialization procedures in a hook script.
USAGE QUESTIONS
===============
How do I tune and optimize a TKLBAM backup?
-------------------------------------------
One of my favorite ways to do this::
# step 1: generate a backup dump
tklbam-backup --dump=/tmp/mybackup
# step 2: interactively review the dump's file contents & disk usage
cd /tmp/mybackup
apt-get install ncdu
ncdu
# step 3: add includes or excludes, go back to step 1, rinse, repeat
vim /etc/tklbam/overrides
# Everything perfect?
tklbam-backup --upload-raw=/tmp/mybackup
By default, TKLBAM will automatically determine what paths and databases need
to be backed up on a given TurnKey system according to the backup profile it
gets from the Hub. The default profile tracks changes to the user-servicable,
customizable parts of the filesystem (e.g., /etc /root /home /var /usr/local
/var /opt /srv) while ignoring changes in areas maintained by the package
management system.
You can "override" the default backup profile configuration by specifying
overrides, either on the command line, or preferably by editing
the /etc/tklbam/overrides configuration file.
How does TKLBAM know what to backup on my system?
-------------------------------------------------
Every TurnKey appliance that TKLBAM supports has a corresponding backup
profile, which is downloaded from the Hub when you initialize TKLBAM. The
profile is used to calculate the list of system changes we need to backup. It
usually describes the installation state of a TurnKey appliance and contains a
list of packages, filesystem paths to scan for changes and an index of the
contents of those paths which records timestamps, ownership and permissions.
You can also generate your own custom profiles with the following command::
tklbam-internal create-profile
The backup profile is stored in /var/lib/tklbam/profile and contains the
following text files:
1) dirindex.conf: a list of directories to check for changes by default.
This list does not include any files or directories maintained by the
package management system.
2) dirindex: appliance installation state - filesystem index
3) packages: appliance installation state - list of packages
Users can override which files and directories are checked for changes by
configuring overrides (See below).
Why is an override called an override?
--------------------------------------
Because it "overrides" the default backup configuration in the appliance
profile.
How do I remove a file or directory from being included in my backup?
---------------------------------------------------------------------
By adding a negative override to /etc/tklbam/overrides::
echo -/var/www/*/logs >> /etc/tklbam/overrides
You can also specify overrides on the command line at backup time::
tklbam-backup -- -/var/www/*/logs
How do I add a directory to my backup?
--------------------------------------
By adding an override to /etc/tklbam/overrides::
echo /mnt/images >> /etc/tklbam/overrides
Or on the command line::
tklbam-backup /var/www/*/logs
Make sure you understand the implications of doing this. For example, if
you add a directory handled by package management this may break package
management on the system you restore to.
How do I exclude a database or table from my backup?
----------------------------------------------------
By adding a negative database override to /etc/tklbam/overrides::
# exclude drupal5 database
echo -mysql:drupal5 >> /etc/tklbam/overrides
# exclude sessions table in drupal6 database
echo -mysql:drupal6/sessions >> /etc/tklbam/overrides
Or on the command line::
tklbam-backup -- -mysql:drupal6/page_cache
By default ALL databases are backed up so adding a negative database override
excludes only that database or table from the backup.
Excluding a table only excludes its data. The schema is still backed up as long
as the database is included.
Specifying a positive database override changes the default behavior so that
only the database or table specified in the override is included in the backup.
You can mix positive overrides with negative overrides.
Can I have multiple TKLBAM backups on a single system?
------------------------------------------------------
Yes. For example, let's say your default TKLBAM backup is several gigabytes in
size and you'd like to create a lighter 100 MB backup that will be updated more
frequently and take less time to update/restore::
cp -a /etc/tklbam /etc/tklbam.light
echo -/var/www/\*/logs >> /etc/tklbam.light/overrides
echo -/home/liraz/bigfiles >> /etc/tklbam.light/overrides
echo -mysql:mydatabase/bigtable >> /etc/tklbam.light/overrides
export TKLBAM_CONF=/etc/tklbam.light
mkdir /var/lib/tklbam.light
export TKLBAM_REGISTRY=/var/lib/tklbam.light
tklbam-init
tklbam-backup
For convenience you may want to create a script that sets the TKLBAM_REGISTRY
and TKLBAM_CONF environment variables::
cat > /usr/local/bin/tklbam-backup-light << EOF
#!/bin/bash
export TKLBAM_CONF=/etc/tklbam.light
export TKLBAM_REGISTRY=/var/lib/tklbam.light
tklbam-backup
EOF
chmod +x /usr/local/bin/tklbam-backup-light
Can I use TKLBAM to only backup a single directory?
---------------------------------------------------
Yes. Here are a couple of recommended ways to do this:
1) Create a separate backup with an empty backup profile::
export TKLBAM_REGISTRY=/var/lib/tklbam.srv
export TKLBAM_CONF=/etc/tklbam.srv
tklbam-init --force-profile=empty
tklbam-backup --skip-packages --skip-database -- /srv
2) Use the --raw-upload option
This lobotomizes TKLBAM so instead of creating a system level backup it just
backs up the directory you specify. In other words, --raw-upload turns
TKLBAM into a directory-level backup tool rather than a system-level backup
tool.
For example, let's say you have a collection of big files at /srv that you don't
want to include in your system backup (e.g., because you don't want to bloat your backup).
So you configure an overrides to exclude the /srv directory from your default
backup and create another TKLBAM backup just for the big files::
echo -/srv >> /etc/tklbam/overrides
export TKLBAM_REGISTRY=/var/lib/tklbam.srv-raw
tklbam-backup --raw-upload=/srv
Later, you'll need to use the --raw-download option to restore::
tklbam-restore --raw-download=/srv <your-backup-id>
If you don't use the raw-download option, TKLBAM will assume you are trying
to restore a system-level backup and you'll get an error.
What's the difference between a full backup and an incremental backup?
----------------------------------------------------------------------
A full backup is a backup that can be restored independently of any
other backup. An incremental backup links with the last backup before it
and only includes changes made since.
Backup chains are links of backup sessions which start with a full
backup, and then a series of incremental backups each recording only the
changes made since the backup before it. Incremental backups are useful
because they are fast and efficient.
Restoring an incremental backup requires retrieving the volumes of all
backup sessions made before it, up to and including the full backup that
started the chain. The longer the backup chain, the more time it will
take to restore.
How often does a does a full backup happen, how can I configure this?
---------------------------------------------------------------------
By default, a full backup will happen if the last full backup is
older than 30 days. Between full backups, all backup sessions are
incremental.
We recommend enabling the daily backup cron job so that daily
incremental backups happen automatically::
chmod +x /etc/cron.daily/tklbam-backup
You can override the default by setting the full-backup parameter in
the tklbam configuration::
# create a full backup every 14 days
echo full-backup 14D >> /etc/tklbam/conf
I forgot my passphrase, and I "lost" my escrow key. Can you help me?
--------------------------------------------------------------------
Sorry, if your server is gone (e.g., terminated EC2 instance) nobody can
help you. Next time either save an escrow key somewhere[s] safe or don't
set a passphrase.
The encryption key for your backup was generated locally on your server not
ours. Passphrase protection uses special cryptographic countermeasures to make
typical cracking techniques (e.g., dictionary attacks) very difficult even with
access to massive computer resources.
Note, if the system you backed up is still available, just log into it
as root and change the passphrase (you don't need to know the old
passphrase)::
tklbam-passphrase
AMAZON S3 QUESTIONS
===================
How do I monitor how much traffic is being uploaded or downloaded?
------------------------------------------------------------------
We recommend the following package::
apt-get install iftop
iftop
How can I throttle how much bandwidth TKLBAM uses?
--------------------------------------------------
Here's one way to do it::
apt-get install trickle
trickle -u 100 -d 100 tklbam-backup
Do I have to store my backups in the Amazon S3 storage cloud?
-------------------------------------------------------------
No! TKLBAM stores backups in the cloud for convenience, but it also supports
local / custom backup storage targets.
There are two main alternatives to letting TKLBAM store a backup in the cloud:
1) Low-level tklbam-backup --dump option: lets you dump the raw TKLBAM backup
extract to a directory, which you can then store anyway you like.
For example here's how we'd a system backup into a simple unencrypted tarball::
cd /tmp
mkdir mybackup
tklbam-backup --dump=mybackup/
tar jcvf mybackup.tar.bz2 mybackup/
And later restore it like this::
cd /tmp
tar jxvf mybackup.tar.bz2
tklbam-restore mybackup/
The --dump option bypasses Duplicity, which usually create a series of
encrypted archive files that can be incrementally updated. These archive files
are stored by default in the Amazon S3 storage cloud but you can override this
with the --address option and specify any storage back-end supported by
Duplicity (e.g., local directory, rsync over ssh, ftp, sftp, etc).
2) High-level tklbam-backup --address option: lets you specify a custom backup
target URL that is passed on to Duplicity.
It is highly recommended to rehearse a trial restore. Testing your backups
is always a good idea, and even more so with a custom --address as this may
complicate usage.
The Hub normally helps you manage your backup's metadata when it
auto-configures the storage address. If you specify a manual address you
need to manage storage locations, encryption keys and authentication
credentials by hand.
Why can't I access TKLBAM storage buckets with other Amazon S3 tools?
---------------------------------------------------------------------
TKLBAM doesn't store its data in generic S3 buckets, but in an isolated
TKLBAM-specific area on S3. This means generic S3 tools such as the AWS
management console, or S3Fox will not be able to access the storage buckets in
which TKLBAM backup volumes reside.
What are the advantages of isolating TKLBAM Amazon S3 storage?
--------------------------------------------------------------
1) Easier sign up process. Users don't need to know anything about S3
API keys or understand the implications of giving them to us.
2) Security: you don't need to give us access to your generic S3
account. If someone compromises your regular AWS API Key they still
can't get to your encrypted backup volumes and say... delete them.
3) Cost transparency: TKLBAM related storage charges show up separately
from your generic S3 storage.
What happens if my payment method to Amazon is invalidated?
-----------------------------------------------------------
Amazon supports payment by credit card and bank account. We recommend
heavy users add a bank account as their payment method, as it's usually
more permanent than a credit card.
In any case, if your payment method is invalidated (e.g., cancelled or
expired credit card), billing will fail and Amazon will attempt to
contact you (e.g., by e-mail) to provide a new, valid payment method.
How much does cloud backup storage cost?
----------------------------------------
Amazon S3 cloud storage fees are around $0.15/GB per month.
You can use simulation mode to calculate how much uncompressed data
TKLBAM is going to store in a full backup::
$ tklbam-backup --simulate
CREATING /TKLBAM
FULL UNCOMPRESSED FOOTPRINT: 148.30 MB in 6186 files
In practice, the actual footprint of a full backup will usually be
smaller due to compression, but this depends on the type of data being
compressed (e.g., text compresses very well, video very poorly).
By default, a full backup is performed if one month has passed since the
last full backup. In between, incremental backups will be performed
which only record changes since the last backup. The full backup
frequency can be customized. See this manual page for details.
The Hub says my backup costs $0.00, what am I really paying?
------------------------------------------------------------
If you notice $0.00 in the backups console, there's no need to open a
support request. It's not a bug. At 15 cents per gigabyte, if you have
just a few megabytes of data Amazon doesn't charge you anything.
Backups start from around 100KB for a freshly installed TurnKey appliance.
Remember, TKLBAM only saves changes you've made since the appliance was
installed.
FAULT TOLERANCE FOR THE PARANOID IT GUY
=======================================
Is the Hub a central point of failure for TKLBAM?
-------------------------------------------------
No, for a couple of reasons:
1) After the initial setup, TKLBAM communicates directly with Amazon S3. Even
if the Hub does down, backups will not be interrupted.
2) You can use TKLBAM without linking it to the Hub at all. See the tklbam-init
--solo option.
If the Hub goes down, will my backup cron jobs still work?
----------------------------------------------------------
Yes. Backups which have already been configured will continue to work
normally. If TKLBAM can't reach the Hub it just uses the locally cached
profile and S3 address.
If my connection to the Hub goes down, can I still restore?
-----------------------------------------------------------
Yes - manually. It just won't be as easy. You'll need to do a couple of
steps by hand:
1) transfer the escrow key to the restore target.
This means you'll need to have stored the escrow key somewhere safe
or be able to create it on the backed up machine.
2) specify the S3 address and the key manually when you restore.
For more details see the tklbam-restore documentation.
If the Hub fails can I still create a new backup?
-------------------------------------------------
Yes - but you'll have to manage it manually. The Hub won't know anything about
these backups so you'll have manage keys and authentication credentials by
hand.
SEE ALSO
========
``tklbam`` (8)
Jump to Line
Something went wrong with that request. Please try again.