backup ganeti cluster using lvm snapshots and rsync to zfs snapshots

This tools depend on scripts in https://github.com/ffzg/gnt-info repository
to create lvm snapshots and push them to rsync server (machine with zfs).



Add new ganeti cluster (r1u28) to backup server (lib15):

0. configure rsync_server to accept new backups

root@lib20:/srv/zfs-tools# zfs list lib20/oscar
NAME          USED  AVAIL  REFER  MOUNTPOINT
lib20/oscar   103K  4.50T  28.2K  /lib20/oscar

First configure rsync targets in /etc/rsyncd.conf:

[oscar]
        comment = oscar.gnt.ffzg.hr
        path = /lib20/oscar/
        use chroot = yes
        lock file = /var/lock/rsyncd.oscar
        read only = no
        list = yes
        uid = root
        gid = root
        strict modes = yes
        ignore errors = no
        ignore nonreadable = yes
        transfer logging = yes
        timeout = 600
        refuse options = checksum dry-run
        dont compress = *.gz *.tgz *.zip *.z *.rpm *.deb *.iso *.bz2 *.tbz

1. create ssh key to access r1u28

root@lib20:~/.ssh# ssh-keygen -f ./oscar-lib20

2. add it to authorized_keys on r1u28 and configure backup
   destination on lib20 and path to lvm+rsync script

command="rsync_server=lib20 backup=oscar /srv/gnt-info/gnt-lv-snap-rsync.sh -" ssh-dss ...

3. generate ssh key on r1u28 to create snapshot on lib15 and distribute it

root@r1u28:# ssh-keygen -t dsa -f /etc/ganeti/id_dsa-zfs-snap

root@r1u28:# gnt-cluster copyfile /etc/ganeti/id_dsa-zfs-snap

4. add it to authorized_keys on lib15

command="/srv/zfs-tools/ssh/zfs-snap.sh" ssh-dss ...

4.1 ensure that rsync_server lib15 is known from all nodes in cluster

root@r1u28:# gnt-cluster command "sh -cx 'yes | ssh -i /etc/ganeti/id_dsa-zfs-snap lib15'"

5. add it to cron backup script

export backup=oscar
/srv/zfs-tools/backup-instances-today.sh | xargs -i echo \
 'echo {} | ssh -i /root/.ssh/r1u28-id_dsa r1u28 | tee "/dev/shm/cron.{}.log"' \
 | tee /dev/shm/cron-$backup.sh
sh -xe /dev/shm/cron-$backup.sh



Create backup from ganeti intance SOP:

1. export instance using gnt-backup

gnt-backup export --noshutdown -n r1u28 delta

2. import backup into zfs filesystem
   (backup enviroment variable is top-level directory for backups)

backup=oscar /srv/zfs-tools/restore-to-zfs-fs.sh \
	/var/lib/ganeti/export/delta/a39fa7b8-8a5d-4bf2-a9a9-438761357446

3. now rename backup directory to contain disk number starting from 0
   instead of uuid used by ganeti

zfs rename lib15/oscar/delta/3978-85-42-99-438761357446 lib15/oscar/delta/0



Create backup from ganeti without existing gnt-backup export

1. create new directory for instance disk 0

zfs create -p lib15/oscar/a1.ffzg.unizg.hr/0

2. manually invoke first rsync

echo a1.ffzg.unizg.hr 0 | ssh -i /root/.ssh/r1u28-id_dsa r1u28



rsync exclude and other parametars

If you create lib15/oscar/instance/rsync.args file and put
"--exclude path/to/exclude" in one or more lines, this will be
applied when transfering files to zfs rsync server from
https://github.com/ffzg/gnt-info/blob/master/gnt-lv-snap-rsync.sh



Tweaks you can do to installed instances:

optimize logrotate to include date in filename (to prevent multiple copies
on copy-on-write filesystem like zfs)

perl -p -i -n -e 's/(\s+)daily/$1daily\n$1dateext/' /etc/logrotate.d/*



List backups showing written which is real size of uncompressed data:

zfs list -r -t all -o space,logicalreferenced,written,compressratio lib20/backup