How to restore a single container from backup zfs snapshot

[AceSlash]

I'll describe my issue so you can better understand the problem.

I have a backup machine, that handle the backup of several lxd hosts. I pull zfs snapshots for critical containers (not necessarily all the containers) from the lxd hosts to the backup server every hour (ssh root@lxdhost1 zfs send -I tank/lxd/container2@2016081615 tank/lxd/container2@2016081615 | zfs receive tank/backup/lxdhost1/lxdhost1/container2/system).

On my backup server, I have something like this as zfs tree:

tank
└── backup
    ├── lxdhost1
    │   ├── container1
    │   │   ├── system
    │   │   ├── data1
    │   │   └── data2
    │   └── container2
    │       └── system
    ├── lxdhost2
    │   └── container3
    │       ├── system
    │       └── data1
    └── lxdhost3
        └── container4
            └── system

I also backup the lxd database of all hosts (lxd.db) and the content of the /var/lib/lxd tree (without the mounted zfs devices containing the containers data of course).

If lxdhost1 dies, I need to import container1 to lxdhost2 and container2 to lxdhost3.

To my knowledge, there is no easy way to do that.

For example, to restore the backup of container1 to lxdhost2, I suppose I have to:

• lxc init a new container with the same name/config/os on lxdhost2
• lxc config edit to adjust the mac addresses to their old values
• zfs destroy the new container dataset
• zfs send/receive container1 dataset backup to lxdhost2
• adjust the sqlite database (adjust the containers_config "volatile.base_image" to the original, or remove the line if the image doesn't exist on lxdhost2 I think)
• lxc start container1

I would like to be able to do something like this instead:

• export container1 configuration (as a sql file?) during the backup procedure (don't save the lxd.db in this case)
• zfs send/receive the container1 dataset to lxdhost2
• import the container1 configuration to lxdhost2 lxd.db
• lxc start container1

I'm not sure how to handle the zfs dataset origin (eg: container link to their image). But it should simply be dropped since we just send the zfs dataset without any reference to the original image.

An other way would be to have the backup host as an lxd host and lxc copy every containers to it, then send incremental snapshots to keep the backup up-to-date. This way a simple lxc copy from the backup host to lxdhost2 would work, but you need a backup system with lxd, which is not always possible.

Any more idea how to be able to export/import containers from backup is welcome.

[AceSlash]

This issue is somewhat related to #1987 and #1304 .

[tych0]

On Tue, Aug 16, 2016 at 05:37:49AM -0700, AceSlash wrote:

I'll describe my issue so you can better understand the problem.

I have a backup machine, that handle the backup of several lxd hosts. I pull zfs snapshots for critical containers (not necessarily all the containers) from the lxd hosts to the backup server every hour (ssh root@lxdhost1 zfs send -I tank/lxd/container2@2016081615 tank/lxd/container2@2016081615 | zfs receive tank/backup/lxdhost1/lxdhost1/container2/system).

On my backup server, I have something like this as zfs tree:

tank
└── backup
    ├── lxdhost1
    │   ├── container1
    │   │   ├── system
    │   │   ├── data1
    │   │   └── data2
    │   └── container2
    │       └── system
    ├── lxdhost2
    │   └── container3
    │       ├── system
    │       └── data1
    └── lxdhost3
        └── container4
            └── system

I also backup the lxd database of all hosts (lxd.db) and the content of the /var/lib/lxd tree (without the mounted zfs devices containing the containers data of course).

If lxdhost1 dies, I need to import container1 to lxdhost2 and container2 to lxdhost3.

To my knowledge, there is no easy way to do that.

For example, to restore the backup of container1 to lxdhost2, I suppose I have to:

• lxc init a new container with the same name/config/os on lxdhost2
• lxc config edit to adjust the mac addresses to their old values
• zfs destroy the new container dataset
• zfs send/receive container1 dataset backup to lxdhost2
• adjust the sqlite database (adjust the containers_config "volatile.base_image" to the original, or remove the line if the image doesn't exist on lxdhost2 I think)
• lxc start container1

I would like to be able to do something like this instead:

• export container1 configuration (as a sql file?) during the backup procedure (don't save the lxd.db in this case)
• zfs send/receive the container1 dataset to lxdhost2
• import the container1 configuration to lxdhost2 lxd.db
• lxc start container1

I'm not sure how to handle the zfs dataset origin (eg: container link to their image). But it should simply be dropped since we just send the zfs dataset without any reference to the original image.

An other way would be to have the backup host as an lxd host and lxc copy every containers to it, then send incremental snapshots to keep the backup up-to-date. This way a simple lxc copy from the backup host to lxdhost2 would work, but you need a backup system with lxd, which is not always possible.

Any more idea how to be able to export/import containers from backup is welcome.

Why not use lxc copy to copy the containers to the second system
running in LXD? Then you can just use lxc start right away on
failover.

You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
#2286

[AceSlash]

@tych0 : Your approach has several issues from my point of view:

• you must know on which host you will start the containers if the first host dies, this is not necessary the case, as host and load change over time
• you are using production lxd host as backup hosts, with the whole history (all zfs snapshots) on them, this can take a lot of space where you could just send the latest snapshot from the central backup server
• from one backup host where you have all your data clearly identified, you now have several backup/prod hosts, this complicate the backup checking and process significantly

Furthermore, I backup the data on the backup host on different site (a mirror backup server and an external encrypted storage) and the backup host is the only one that have access to all the different lxd hosts, some lxd hosts don't see each others (different network, different location). It would be difficult to have the same level of security and check that everything works on 10+ lxd hosts.

But your way works if you need HA and quick fail over, and for some containers, I actually replicate the data one more time this way (lxc copy than zfs send/receive to keep it up-to-date), but I purge the history on the lxd hosts very frequently (I only keep 7 days of data on prod servers).

[tych0]

On Tue, Aug 16, 2016 at 08:09:12AM -0700, AceSlash wrote:

@tych0 : Your approach has several issues from my point of view:

• you must know on which host you will start the containers if the first host dies, this is not necessary the case, as host and load change over time

True, although you could just lxc copy them to the host you want
them to live on finally.

• you are using production lxd host as backup hosts, with the whole history (all zfs snapshots) on them, this can take a lot of space where you could just send the latest snapshot from the central backup server

Not necessarily, you can copy just snapshots:

lxc copy foo/snap0 otherhost:foobackup

• from one backup host where you have all your data clearly identified, you now have several backup/prod hosts, this complicate the backup checking and process significantly

Furthermore, I backup the data on the backup host on different site (a mirror backup server and an external encrypted storage) and the backup host is the only one that have access to all the different lxd hosts, some lxd hosts don't see each others (different network, different location). It would be difficult to have the same level of security and check that everything works on 10+ lxd hosts.

I don't understand these points. Why not just run LXD on the backup
server?

But your way works if you need HA and quick fail over, and for some containers, I actually replicate the data one more time this way (lxc copy than zfs send/receive to keep it up-to-date), but I purge the history on the lxd hosts very frequently (I only keep 7 days of data on prod servers).

You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub:
#2286 (comment)

[AceSlash]

From IRC discussion, I see your point: with a backup server running lxd, you can keep up-to-date all the containers from different lxd hosts and if an host dies, you can transfer the container from the backup host to the new lxd host.

That's indeed a solution. This is exactly what I meant by:

An other way would be to have the backup host as an lxd host and lxc copy every containers to it, then send incremental snapshots to keep the backup up-to-date. This way a simple lxc copy from the backup host to lxdhost2 would work, but you need a backup system with lxd, which is not always possible.

The main issue is that your backup server must run lxd, which may not be the case (Freebsd, older Linux, etc).

While I understand that this may not seem like a big issue, you are basically telling me that lxd containers cannot be saved and restored individually outside a lxc copy operation.

Don't you think that the ability to create/restore a container backup should exist beside this solution?

[tych0]

We talked this over in IRC, but for the record: I think we'd be happy to accept patches that made this possible (e.g. by adding another image type called "zfs" or something, and allowing uploading a zfs blob), but it is not a priority for us right now.

[nlienard]

i've also run a freebsd as backup machine and would love to zfs send | ssh backup zfs recv datavols of my containers on this machine. But before; it requires that lxd accept zfs image for restore.

[givre]

Thanks for that guys.