Method to backup user-data -- Any possibilities?

[LawGuy2000]

I have been reading the documents all over and it appears there is no coherent system to backup user-data. qvm-backup will backup the entire VM, which of course is a bit excessive to do that every couple hours or days.

qvm-copy might be the beginning of a tool that could be built on for user-data backups, but the issue is that it would cause a LOT of writing to the disk every backup as there is no way to not copy a file if it has not changed or to sync only the changed parts. It would be very inefficient I imagine.

The Problem

User-data lives on many virtual machines. Given the potential for malware to corrupt the data or hard disk errors, etc, it is important to have a good backup strategy. The backup strategy should avoid excessive disk writes and work across all VMs even those without internet I think.

Potential Soution, Possible?

I imagine a system where VMs can send specific data to another VM and have to avoid duplicate copying. At a minimum that would be nice. Even better would be the ability to synchronize file structure with another VM. Preferably, encrypted.

For example, imagine sys-backup stores the backed up user data. AppVm will generate a list of files that are marked to be backed up. It will then send a message to sys-backup asking for the files it currently has and the hash that was given to it for each file. The AppVM will then utilize its list and that list to which files have been deleted and changed. It will send over the changed files and the hash to sys-backup. The actual file it sends could be encrypted or not.

Then, the user can do whatever they want with the data on sys-backup. For example, they can run rsync to a remote server if they want keeping versioned files.

Is there any plans to build some tools that will enable user-data level backups in a secure way in Qubes?

[mirrorway]

The backup strategy should avoid excessive disk writes and work across all VMs even those without internet I think.

I suggest this borgbackup setup for efficient (delta-only), compressed, encrypted, versioned, remote backup: https://groups.google.com/d/msg/qubes-users/cNC-dthpB1o/pFVTXHfXAgAJ

It runs from dom0 by way of a proxy VM, instead of using the VM's own networking, so works from offline or tor VMs. Neither the proxy VM nor the backup server is trusted.

Extra implementation notes:

• Use --read-special with borg create on the thin volume associated with the vm's private data.
• If your VM is called my-potato, then the private volume is at /dev/mapper/qubes_dom0-vm--my--potato--private or if it's running, /dev/mapper/qubes_dom0-vm--my--potato--private--snap.
• Use xl pause, unpause to freeze the VM during backup (don't need to shut it down).

[mirrorway]

The backup strategy should avoid excessive disk writes and work across all VMs even those without internet I think.

I suggest this borgbackup setup for efficient (delta-only), compressed, encrypted, versioned, remote backup: https://groups.google.com/d/msg/qubes-users/cNC-dthpB1o/pFVTXHfXAgAJ

It runs from dom0 by way of a proxy VM, instead of using the VM's own networking, so works from offline or tor VMs. Neither the proxy VM nor the backup server is trusted.

Extra implementation notes:

• Use --read-special with borg create on the thin volume associated with the vm's private data.
• If your VM is called my-potato, then the private volume is at /dev/mapper/qubes_dom0-vm--my--potato--private or if it's running, /dev/mapper/qubes_dom0-vm--my--potato--private--snap.
• Use xl pause, unpause to freeze the VM during backup (don't need to shut it down).

[andrewdavidwong]

Based on our issue reporting guidelines, this does not appear to be suitable for qubes-issues. We ask that you please send this to the qubes-users mailing list instead. If, after reading our issue reporting guidelines, you believe we are mistaken, please leave a brief comment explaining why. We'll be happy to take another look, and, if appropriate, reopen this issue. Thank you for your understanding.

[andrewdavidwong]

Based on our issue reporting guidelines, this does not appear to be suitable for qubes-issues. We ask that you please send this to the qubes-users mailing list instead. If, after reading our issue reporting guidelines, you believe we are mistaken, please leave a brief comment explaining why. We'll be happy to take another look, and, if appropriate, reopen this issue. Thank you for your understanding.

[andrewdavidwong]

To clarify: Intra-VM file backup is not something that Qubes is intended to handle, and there are existing mailing list threads on this already. qvm-backup with compression is usable for frequent backups (I use it daily on a large subset of my backups), so this issue rests on a mistaken (or at least controversial) assumption. Asking and discussing whether other tools will be built for this is fine, but it should be done on the mailing lists. This is not the place for such discussions.

[andrewdavidwong]

To clarify: Intra-VM file backup is not something that Qubes is intended to handle, and there are existing mailing list threads on this already. qvm-backup with compression is usable for frequent backups (I use it daily on a large subset of my backups), so this issue rests on a mistaken (or at least controversial) assumption. Asking and discussing whether other tools will be built for this is fine, but it should be done on the mailing lists. This is not the place for such discussions.