Backup Strategy for Container Host

[srkunze]

Building on top of #1302 what's would a sensible backup strategy look like?

I know that we need the LXD database in order to get the LXD daemon up and running again.
What about the containers themselves? Snapshots? Configs?

In other words: what's all necessary in order to recreate the complete container host in case of a hardware failure?

[stgraber]

A complete backup of /var/lib/lxd and separate SQL dump of the database would be the easiest way to backup everything LXD cares about.

[stgraber]

You may also consider backing up /var/log/lxd/ if those logs matter to you, LXD won't mind if they're gone though.

[srkunze]

Sounds good. Thanks. :)

[srkunze]

Can /var/lib/lxd/images be excluded from backup? Or do containers reference images which they are created from?

[stgraber]

If you're using zfs, the containers do reference their image, for other backends, they don't, though if you decide not to backup the images, you should also clear the image related tables from the database on restore.

[srkunze]

I understand. I pondered over this in order to reduce the backup size.

As it seems, though, using ZFS would be an even better way of optimizing backup size without the need to introduce some exceptions and other weird hacky things.

[stgraber]

Correct. If using zfs you'll want to backup the zpool separately, that will contain all containers, images and snapshots, then /var/lib/lxd itself will be mostly empty except for the database and certificates.

[srkunze]

Haven't made experience with ZFS, yet. Seems like a good alternative.

[yoniy0]

@stgraber do you mind correcting me if I'm wrong on any of the following, and clarifying a few points?

If using ZFS as storage backend, then LXD creates the following data-sets: tank/lxd (mounted at /var/lib/lxd), tank/lxd/containers, tank/lxd/images, then tank/lxd/images/<giberrish-image-name> for each image, and tank/lxd/containers/<container-name> for each container (where any data-set under containers is mounted relative to the original mount).

Using lxc snapshot utilizes ZFS snapshots to create a snapshot of the specific container data-set.

One could use zfs send to copy snapshot (incrementally) to a remote machine, thus creating a backup. To create a complete backup, is it sufficient to run sudo zfs snapshot -r tank/lxd (then sudo zfs send -R it), or should one create snapshots for all data-sets mentioned above (using zfs snapshot--no -r!) except the container (which was created with lxc snapshot), then send all of them? Is the DB backed up by sending a snapshot of tank/lxd?

Would a feasible restore strategy then be simply sending all snapshots in the reverse direction? Would LXD accept the new data, or should it be manually made aware of it?

Finally, are any parts of srv3_tank/lxd/deleted (such as deleted images) referenced by LXD and should be included?

[CalebEverett]

Here's one approach using aws volume snapshots:

• Mount a separate volume dedicated to zfs for lxd
• Snapshot the entire volume (snapshots on aws are compressed and incremental so can be taken regularly without much increase in storage consumption if your data isn't changing that much)
• Take snapshots of the entire root volume as well (which includes the lxd database)
• The snapshots can be easily automated in a script on your server or an aws lambda function
• If you have to restore, create volumes from relevant snapshots and mount them in a new instance

It also seemed like you had more options for backing up and restoring the lxd file system if you configured lxd to use a file system below the top level zpool. I had lxd configured to use a pool directly, which made it difficult to send to a new zfs pool with the same directory structure since a destination pool had to exist to receive the stream. With the local pool set up as:

lxd
lxd/containers
lxd/images
lxd/deleted

and the remote pool set up initially as:

lxd

I couldn't send to the remote to end up with the same directory structure as the local pool. I could only send to

lxd/lxd

But with lxd below the top pool like this, you could send the entire lxd file system to a pool on another volume and then mount the volume and import the pool to restore (provided you had the associated lxd database and other info).

top/lxd
top/lxd
top/lxd/containers
top/lxd/images
top/lxd/deleted

https://github.com/lxc/lxd/blob/master/doc/storage-backends.md#zfs
https://insights.ubuntu.com/2016/02/16/zfs-is-the-fs-for-containers-in-ubuntu-16-04/
https://wiki.ubuntu.com/Kernel/Reference/ZFS

[AceSlash]

I had to rescue a lxd host recently and to my surprise, I saw that /var/lib/lxd was (by default) not on the zfs dataset used by everything else.

This is very dangerous: if you use zfs snapshot and send to backup tank/lxd (or whatever your pool name is), you won't have the lxd database and you'll have a lot of trouble to restore the host.

Furthermore, this way of backup (taking everything, including the database), works in some case (when you want to restore the whole host), but doesn't in others common cases: like you want to import all the containers on an other host that already runs several containers. Currently, you can't do that easily without hacking the database.

You also can't "lxc copy" to an other host and then send the incremental snapshots using lxd (you can with zfs).

I think a simple way to export a container configuration to a sql file to be able to import it to an other host is necessary. With this, you can backup the container data with ZFS to a backup server and have a way to import the configuration. You could then restore a target container to an other lxd host.

[yoniy0]

@stgraber I'm still looking for a good LXD backup strategy. The documentation isn't quite complete, but can you comment on whether lxc import is mature enough to rely on, or if using images (through publish>export>image import) is preferable?