Consider official Btrfs support in R4.0

[andrewdavidwong]

See discussion here:

https://groups.google.com/d/topic/qubes-users/3x7m6nNJmJU/discussion

[nrgaway]

Here are my experiences with BTRFS and issues I have run into that may want to be taken into consideration when considering adding full support. I use BTRFS for most of my systems except QubesOS.

For the first year or two I used BTRFS with QubesOS. During that period I was building and testing many templates I kept running into issues with the disk becoming full due to COW which resulted in needing to run a balance every week or two. It becomes a PITA to fix a system when all blocks had been allocated resulting in a disk full message. but I have written scripts to do so.

I found the best option when working with BTRFS was to disable COW for the virtual machine images as recommend for virtual machines which then one loses the ability to quickly snapshot any images. I figured then there was no benefit to using BTRFS if I did not want to actively balance the system and disabled COW so I have been running with the defaults for the last year or so.

[nrgaway]

Here are my experiences with BTRFS and issues I have run into that may want to be taken into consideration when considering adding full support. I use BTRFS for most of my systems except QubesOS.

For the first year or two I used BTRFS with QubesOS. During that period I was building and testing many templates I kept running into issues with the disk becoming full due to COW which resulted in needing to run a balance every week or two. It becomes a PITA to fix a system when all blocks had been allocated resulting in a disk full message. but I have written scripts to do so.

I found the best option when working with BTRFS was to disable COW for the virtual machine images as recommend for virtual machines which then one loses the ability to quickly snapshot any images. I figured then there was no benefit to using BTRFS if I did not want to actively balance the system and disabled COW so I have been running with the defaults for the last year or so.

[csirac2]

In my experience with server workloads and btrfs, shipping a system with file-backed VM storage on a snapshotted btrfs volume is a recipe for disaster.

I would suggest snapshots within the guests, and a mechanism to send/receive them to other storage. However, btrfs receive is completely unsuited to receiving data across security domains. A full solution would require btrfs receive running in a dvm attached to dedicated block device per domain. A btrfs dom0 shouldn't be required.

Curious to hear others thoughts on this.

[csirac2]

In my experience with server workloads and btrfs, shipping a system with file-backed VM storage on a snapshotted btrfs volume is a recipe for disaster.

I would suggest snapshots within the guests, and a mechanism to send/receive them to other storage. However, btrfs receive is completely unsuited to receiving data across security domains. A full solution would require btrfs receive running in a dvm attached to dedicated block device per domain. A btrfs dom0 shouldn't be required.

Curious to hear others thoughts on this.

[ideologysec]

Benefits of btrfs: https://fedoraproject.org/wiki/Features/SystemRollbackWithBtrfs

separate from being able to make backups of VMs on a system instead of individual level, and the deduplication features potentially helping template VMs take up even less space, booting into snapshots gives the ability to test potentially even more destructive/radical configuration changes, without worry of permanent harm.

[ideologysec]

Benefits of btrfs: https://fedoraproject.org/wiki/Features/SystemRollbackWithBtrfs

separate from being able to make backups of VMs on a system instead of individual level, and the deduplication features potentially helping template VMs take up even less space, booting into snapshots gives the ability to test potentially even more destructive/radical configuration changes, without worry of permanent harm.

[tasket]

I have been using Btrfs as my root and VM storage volume since Qubes R3.0 RC1. I've had one problem with "running out of space" which caused the fs to go into read-only mode. This was fairly easy to recover from. Since then I have kept at least 40GB "free" on the volume.

BTW, balancing doesn't seem to help a non-raid configuration. If anything, it reduced the amount of shared exents between files, leaving less free space.

I run with everything under COW and the performance is fine. I haven't bothered with setting NOCOW on images because I keep snapshots, which makes NOCOW files behave like COW anyway.

Anecdotally, Btrfs gets a bad rap because it has attracted early adopters who like to experiment and are very vocal (every alternative Linux fs suffers from this). But I understand that systematic testing of storage failure modes shows Btrfs to be much more reliable than Ext4 overall.

The plusses for Btrfs so far outweigh the negatives, especially after Kernel 3.17. The tools have also improved since then, and so has the performance. The only negative that bothers me is the free-space reporting; there has been improvement here and I expect it to get better.

[tasket]

I have been using Btrfs as my root and VM storage volume since Qubes R3.0 RC1. I've had one problem with "running out of space" which caused the fs to go into read-only mode. This was fairly easy to recover from. Since then I have kept at least 40GB "free" on the volume.

BTW, balancing doesn't seem to help a non-raid configuration. If anything, it reduced the amount of shared exents between files, leaving less free space.

I run with everything under COW and the performance is fine. I haven't bothered with setting NOCOW on images because I keep snapshots, which makes NOCOW files behave like COW anyway.

Anecdotally, Btrfs gets a bad rap because it has attracted early adopters who like to experiment and are very vocal (every alternative Linux fs suffers from this). But I understand that systematic testing of storage failure modes shows Btrfs to be much more reliable than Ext4 overall.

The plusses for Btrfs so far outweigh the negatives, especially after Kernel 3.17. The tools have also improved since then, and so has the performance. The only negative that bothers me is the free-space reporting; there has been improvement here and I expect it to get better.

[rustybird]

@tasket:

I haven't bothered with setting NOCOW on images because I keep snapshots, which makes NOCOW files behave like COW anyway.

IIRC btrfs has an internal mode called COW1 or similar, which is causes "the least possible amount of COW" and is used automatically in this situation (snapshotting NOCOW files).

[rustybird]

@tasket:

I haven't bothered with setting NOCOW on images because I keep snapshots, which makes NOCOW files behave like COW anyway.

IIRC btrfs has an internal mode called COW1 or similar, which is causes "the least possible amount of COW" and is used automatically in this situation (snapshotting NOCOW files).

[andrewdavidwong]

Related user feedback:
https://groups.google.com/d/topic/qubes-users/w2Vo5Tjaric/discussion

[andrewdavidwong]

Related user feedback:
https://groups.google.com/d/topic/qubes-users/w2Vo5Tjaric/discussion

[rustybird]

I've opened pull request QubesOS/qubes-core-admin#188 - "file-reflink, a storage driver optimized for CoW filesystems".

[rustybird]

I've opened pull request QubesOS/qubes-core-admin#188 - "file-reflink, a storage driver optimized for CoW filesystems".