btrfs send: extended attribute security.capability lost during incremental send after chown

[ste-fan]

tl;dr When changing ownership of a file with an extended security.capability attribute and manually restoring said attribute to the same value it was before the chown, an incremental btrfs send -p will effectively remove the attribute on the receiving side. (send only transfers the chown but not a set_xattr.)

---

Consider two minimal btrfs filesystems fs1 and fs2.

Create 2 files, format as btrfs and mount as loop devices.

for x in fs1 fs2; do
    head -c 114294784 /dev/zero > $x.loop
    mkfs.btrfs $x.loop
    mkdir -p $x
    mount -t btrfs -o loop $x.loop $x
done
#> btrfs-progs v5.2.1 
#> See http://btrfs.wiki.kernel.org for more information.
#> 
#> Label:              (null)
#> UUID:               f61cf2f4-5446-4fed-9003-ce70da5ed176
#> Node size:          16384
#> Sector size:        4096
#> Filesystem size:    109.00MiB
#> Block group profiles:
#>   Data:             single            8.00MiB
#>   Metadata:         DUP              32.00MiB
#>   System:           DUP               8.00MiB
#> SSD detected:       no
#> Incompat features:  extref, skinny-metadata
#> Number of devices:  1
#> Devices:
#>    ID        SIZE  PATH
#>     1   109.00MiB  fs1.loop
#> 
#> btrfs-progs v5.2.1 
#> See http://btrfs.wiki.kernel.org for more information.
#> 
#> Label:              (null)
#> UUID:               375ce0cd-f830-430e-8b18-0da08c6a8afe
#> Node size:          16384
#> Sector size:        4096
#> Filesystem size:    109.00MiB
#> Block group profiles:
#>   Data:             single            8.00MiB
#>   Metadata:         DUP              32.00MiB
#>   System:           DUP               8.00MiB
#> SSD detected:       no
#> Incompat features:  extref, skinny-metadata
#> Number of devices:  1
#> Devices:
#>    ID        SIZE  PATH
#>     1   109.00MiB  fs2.loop

Create foo.bar on fs1 with an xattr security.capability set.

touch fs1/foo.bar
setcap cap_sys_nice+ep fs1/foo.bar 
getcap -v fs1/foo.bar 
#> fs1/foo.bar = cap_sys_nice+ep

Create read-only snapshot snap_init and transfer to fs2.

btrfs subvol snap -r fs1 fs1/snap_init
#> Create a readonly snapshot of 'fs1' in 'fs1/snap_init'
btrfs send fs1/snap_init | btrfs receive fs2/
#> At subvol fs1/snap_init
#> At subvol snap_init

Capabilities are present as expected.

getcap -v fs[12]/snap_init/foo.bar
#> fs1/snap_init/foo.bar = cap_sys_nice+ep
#> fs2/snap_init/foo.bar = cap_sys_nice+ep

Change group ownership of foo.bar and manually restore caps.

chgrp adm fs1/foo.bar
getcap -v fs1/foo.bar
#> fs1/foo.bar
setcap cap_sys_nice+ep fs1/foo.bar
getcap -v fs1/foo.bar
#> fs1/foo.bar = cap_sys_nice+ep

Create read-only snapshots snap_changed_complete which is transferred non-incrementally to fs2 and snap_changed_incremental which is incrementally transferred to fs2.

btrfs subvol snap -r fs1 fs1/snap_changed_complete
#> Create a readonly snapshot of 'fs1' in 'fs1/snap_changed_complete'
btrfs subvol snap -r fs1 fs1/snap_changed_incremental
#> Create a readonly snapshot of 'fs1' in 'fs1/snap_changed_incremental'
btrfs send fs1/snap_changed_complete | btrfs receive fs2/
#> At subvol fs1/snap_changed_complete
#> At subvol snap_changed_complete
btrfs send -p fs1/snap_init fs1/snap_changed_incremental | btrfs receive fs2/
#> At subvol fs1/snap_changed_incremental
#> At snapshot snap_changed_incremental

The incremental transfer effectively deleted the caps.

getcap -v fs[12]/snap_changed_{complete,incremental}/foo.bar
#> fs1/snap_changed_complete/foo.bar = cap_sys_nice+ep
#> fs2/snap_changed_complete/foo.bar = cap_sys_nice+ep
#> fs1/snap_changed_incremental/foo.bar = cap_sys_nice+ep
#> fs2/snap_changed_incremental/foo.bar

Looking at the dumped send stream, one sees the problem: Since caps have not changed only the change of ownership is transferred, which (b/c of security implications in kernel, I guess) removes all capabilities.

btrfs send -p fs1/snap_init fs1/snap_changed_incremental | btrfs receive --dump
#> At subvol fs1/snap_changed_incremental
#> snapshot        ./snap_changed_incremental      uuid=46558541-1e89-314a-8d3e-b559e17ba1a5 transid=11 parent_uuid=e8e9a984-8222-4843-af6b-742fec3bf87a parent_transid=7
#> utimes          ./snap_changed_incremental/     atime=2019-09-04T14:03:40+0200 mtime=2019-09-04T14:06:41+0200 ctime=2019-09-04T14:06:41+0200
#> truncate        ./snap_changed_incremental/foo.bar size=0
#> chown           ./snap_changed_incremental/foo.bar gid=4 uid=0
#> utimes          ./snap_changed_incremental/foo.bar atime=2019-09-04T14:01:36+0200 mtime=2019-09-04T14:01:36+0200 ctime=2019-09-04T14:05:10+0200

Whereas for the complete send (starting from scratch), set_xattr and chown are present.

btrfs send fs1/snap_changed_complete | btrfs receive --dump
#> At subvol fs1/snap_changed_complete
#> subvol          ./snap_changed_complete         uuid=184f8c5e-c798-224e-b604-632305a3835a transid=10
#> chown           ./snap_changed_complete/        gid=0 uid=0
#> chmod           ./snap_changed_complete/        mode=755
#> utimes          ./snap_changed_complete/        atime=2019-09-04T14:03:40+0200 mtime=2019-09-04T14:03:30+0200 ctime=2019-09-04T14:03:30+0200
#> mkfile          ./snap_changed_complete/o257-6-0
#> rename          ./snap_changed_complete/o257-6-0 dest=./snap_changed_complete/foo.bar
#> utimes          ./snap_changed_complete/        atime=2019-09-04T14:03:40+0200 mtime=2019-09-04T14:03:30+0200 ctime=2019-09-04T14:03:30+0200
#> set_xattr       ./snap_changed_complete/foo.bar name=security.capability data=� len=20
#> truncate        ./snap_changed_complete/foo.bar size=0
#> chown           ./snap_changed_complete/foo.bar gid=4 uid=0
#> chmod           ./snap_changed_complete/foo.bar mode=644
#> utimes          ./snap_changed_complete/foo.bar atime=2019-09-04T14:01:36+0200 mtime=2019-09-04T14:01:36+0200 ctime=2019-09-04T14:05:10+0200

A possible solution would be to always send set_xattr for files that also have a chown operation, even if the extended attributes have not changed.

---

Verbose output of receive during complete send.

btrfs send fs1/snap_changed_complete | btrfs receive -vv fs2/
#> At subvol fs1/snap_changed_complete
#> At subvol snap_changed_complete
#> receiving subvol snap_changed_complete uuid=184f8c5e-c798-224e-b604-632305a3835a, stransid=10
#> chown  - uid=0, gid=0
#> chmod  - mode=0755
#> utimes 
#> mkfile o257-6-0
#> rename o257-6-0 -> foo.bar
#> utimes 
#> set_xattr foo.bar - name=security.capability data_len=20 data=�
#> truncate foo.bar size=0
#> chown foo.bar - uid=0, gid=4
#> chown: restore capabilities
#> chmod foo.bar - mode=0644
#> utimes foo.bar
#> BTRFS_IOC_SET_RECEIVED_SUBVOL uuid=184f8c5e-c798-224e-b604-632305a3835a, stransid=10

Verbose output of receive during incremental send.

btrfs send -p fs1/snap_init fs1/snap_changed_incremental | btrfs receive -vv fs2/
#> At subvol fs1/snap_changed_incremental
#> At snapshot snap_changed_incremental
#> receiving snapshot snap_changed_incremental uuid=46558541-1e89-314a-8d3e-b559e17ba1a5, ctransid=11 parent_uuid=e8e9a984-8222-4843-af6b-742fec3bf87a, parent_ctransid=7
#> utimes 
#> truncate foo.bar size=0
#> chown foo.bar - uid=0, gid=4
#> utimes foo.bar
#> BTRFS_IOC_SET_RECEIVED_SUBVOL uuid=46558541-1e89-314a-8d3e-b559e17ba1a5, stransid=11

---

Complete code block

## Consider two minimal btrfs filesystems `fs1` and `fs2`.
## Create 2 files, format as btrfs and mount as loop devices.
for x in fs1 fs2; do
    head -c 114294784 /dev/zero > $x.loop
    mkfs.btrfs $x.loop
    mkdir -p $x
    mount -t btrfs -o loop $x.loop $x
done
#> btrfs-progs v5.2.1 
#> See http://btrfs.wiki.kernel.org for more information.
#> 
#> Label:              (null)
#> UUID:               f61cf2f4-5446-4fed-9003-ce70da5ed176
#> Node size:          16384
#> Sector size:        4096
#> Filesystem size:    109.00MiB
#> Block group profiles:
#>   Data:             single            8.00MiB
#>   Metadata:         DUP              32.00MiB
#>   System:           DUP               8.00MiB
#> SSD detected:       no
#> Incompat features:  extref, skinny-metadata
#> Number of devices:  1
#> Devices:
#>    ID        SIZE  PATH
#>     1   109.00MiB  fs1.loop
#> 
#> btrfs-progs v5.2.1 
#> See http://btrfs.wiki.kernel.org for more information.
#> 
#> Label:              (null)
#> UUID:               375ce0cd-f830-430e-8b18-0da08c6a8afe
#> Node size:          16384
#> Sector size:        4096
#> Filesystem size:    109.00MiB
#> Block group profiles:
#>   Data:             single            8.00MiB
#>   Metadata:         DUP              32.00MiB
#>   System:           DUP               8.00MiB
#> SSD detected:       no
#> Incompat features:  extref, skinny-metadata
#> Number of devices:  1
#> Devices:
#>    ID        SIZE  PATH
#>     1   109.00MiB  fs2.loop

## Create `foo.bar` on `fs1` with an `xattr` `security.capability` set.
touch fs1/foo.bar
setcap cap_sys_nice+ep fs1/foo.bar 
getcap -v fs1/foo.bar 
#> fs1/foo.bar = cap_sys_nice+ep

## Create read-only snapshot `snap_init` and transfer to `fs2`.
btrfs subvol snap -r fs1 fs1/snap_init
#> Create a readonly snapshot of 'fs1' in 'fs1/snap_init'
btrfs send fs1/snap_init | btrfs receive fs2/
#> At subvol fs1/snap_init
#> At subvol snap_init

## Capabilities are present as expected.
getcap -v fs[12]/snap_init/foo.bar
#> fs1/snap_init/foo.bar = cap_sys_nice+ep
#> fs2/snap_init/foo.bar = cap_sys_nice+ep

## Change group ownership of `foo.bar` and manually restore caps.
chgrp adm fs1/foo.bar
getcap -v fs1/foo.bar
#> fs1/foo.bar
setcap cap_sys_nice+ep fs1/foo.bar
getcap -v fs1/foo.bar
#> fs1/foo.bar = cap_sys_nice+ep

## Create read-only snapshots `snap_changed_complete` which is transferred
## non-incrementally to `fs2` and `snap_changed_incremental` which is
## incrementally transferred to `fs2`.
btrfs subvol snap -r fs1 fs1/snap_changed_complete
#> Create a readonly snapshot of 'fs1' in 'fs1/snap_changed_complete'
btrfs subvol snap -r fs1 fs1/snap_changed_incremental
#> Create a readonly snapshot of 'fs1' in 'fs1/snap_changed_incremental'
btrfs send fs1/snap_changed_complete | btrfs receive -vv fs2/
#> At subvol fs1/snap_changed_complete
#> At subvol snap_changed_complete
#> receiving subvol snap_changed_complete uuid=184f8c5e-c798-224e-b604-632305a3835a, stransid=10
#> chown  - uid=0, gid=0
#> chmod  - mode=0755
#> utimes 
#> mkfile o257-6-0
#> rename o257-6-0 -> foo.bar
#> utimes 
#> set_xattr foo.bar - name=security.capability data_len=20 data=�
#> truncate foo.bar size=0
#> chown foo.bar - uid=0, gid=4
#> chown: restore capabilities
#> chmod foo.bar - mode=0644
#> utimes foo.bar
#> BTRFS_IOC_SET_RECEIVED_SUBVOL uuid=184f8c5e-c798-224e-b604-632305a3835a, stransid=10
btrfs send -p fs1/snap_init fs1/snap_changed_incremental | btrfs receive -vv fs2/
#> At subvol fs1/snap_changed_incremental
#> At snapshot snap_changed_incremental
#> receiving snapshot snap_changed_incremental uuid=46558541-1e89-314a-8d3e-b559e17ba1a5, ctransid=11 parent_uuid=e8e9a984-8222-4843-af6b-742fec3bf87a, parent_ctransid=7
#> utimes 
#> truncate foo.bar size=0
#> chown foo.bar - uid=0, gid=4
#> utimes foo.bar
#> BTRFS_IOC_SET_RECEIVED_SUBVOL uuid=46558541-1e89-314a-8d3e-b559e17ba1a5, stransid=11

## The incremental transfer effectively deleted the caps.
getcap -v fs[12]/snap_changed_{complete,incremental}/foo.bar
#> fs1/snap_changed_complete/foo.bar = cap_sys_nice+ep
#> fs2/snap_changed_complete/foo.bar = cap_sys_nice+ep
#> fs1/snap_changed_incremental/foo.bar = cap_sys_nice+ep
#> fs2/snap_changed_incremental/foo.bar

## Looking at the dumped `send` stream, one sees the problem: Since caps have
## not changed *only* the change of ownership is transferred, which (b/c of
## security implications in kernel, I guess) removes all capabilities.
btrfs send -p fs1/snap_init fs1/snap_changed_incremental | btrfs receive --dump
#> At subvol fs1/snap_changed_incremental
#> snapshot        ./snap_changed_incremental      uuid=46558541-1e89-314a-8d3e-b559e17ba1a5 transid=11 parent_uuid=e8e9a984-8222-4843-af6b-742fec3bf87a parent_transid=7
#> utimes          ./snap_changed_incremental/     atime=2019-09-04T14:03:40+0200 mtime=2019-09-04T14:06:41+0200 ctime=2019-09-04T14:06:41+0200
#> truncate        ./snap_changed_incremental/foo.bar size=0
#> chown           ./snap_changed_incremental/foo.bar gid=4 uid=0
#> utimes          ./snap_changed_incremental/foo.bar atime=2019-09-04T14:01:36+0200 mtime=2019-09-04T14:01:36+0200 ctime=2019-09-04T14:05:10+0200

## Whereas for the complete `send` (starting from scratch), `set_xattr` and
## `chown` are present.
btrfs send fs1/snap_changed_complete | btrfs receive --dump
#> At subvol fs1/snap_changed_complete
#> subvol          ./snap_changed_complete         uuid=184f8c5e-c798-224e-b604-632305a3835a transid=10
#> chown           ./snap_changed_complete/        gid=0 uid=0
#> chmod           ./snap_changed_complete/        mode=755
#> utimes          ./snap_changed_complete/        atime=2019-09-04T14:03:40+0200 mtime=2019-09-04T14:03:30+0200 ctime=2019-09-04T14:03:30+0200
#> mkfile          ./snap_changed_complete/o257-6-0
#> rename          ./snap_changed_complete/o257-6-0 dest=./snap_changed_complete/foo.bar
#> utimes          ./snap_changed_complete/        atime=2019-09-04T14:03:40+0200 mtime=2019-09-04T14:03:30+0200 ctime=2019-09-04T14:03:30+0200
#> set_xattr       ./snap_changed_complete/foo.bar name=security.capability data=� len=20
#> truncate        ./snap_changed_complete/foo.bar size=0
#> chown           ./snap_changed_complete/foo.bar gid=4 uid=0
#> chmod           ./snap_changed_complete/foo.bar mode=644
#> utimes          ./snap_changed_complete/foo.bar atime=2019-09-04T14:01:36+0200 mtime=2019-09-04T14:01:36+0200 ctime=2019-09-04T14:05:10+0200

## A possible solution would be to always `send` `set_xattr` for files that
## also have a `chown` operation, even if the extended attributes have not
## changed.

[kdave]

Thanks for the very detailed report and analysis! Sending the xattrs in case there's a chown operation as well seems tobe the safest way. The capabilities are specially handled at receive time but the incremental change does not pass enough information indeed.

The send/receive should follow what would a user do, so a plain 'chown' would normally drop the capabilities and that would be transferred as 'delete xattr' to the receiving side. So to preserve owner change and capabilities, this would have to be followed by a manual step to restore them.
And this manual step needs to be mimicked by send to recognize new owner and send capabilities.

So this technically makes it a kernel bug, but should be fixed nevertheless. I wonder if there are other implied changes for various security related properties/attributes. IIRC we've had problems only with capabilities.

[adam900710]

@kdave I tend to consider send stream as a diff between two subvolumes, other than simple instructions to rebuild the subvolume.

From that respect of view, the XATTR restore should be then handled by receive, not send part.

Btrfs receive can easily determine if the original file has any XATTR, and back it up then restore after uid/gid change.
We can even allow user to toggle such behavior, as we are more flex in user space.

How does this plan sound to you?

[kdave]

The chown and capabilities is a special case and a well known one so fixing it on the send side seems as a reasonable option and has been implemented. Patch is scheduled for 5.8 and tagged for stable so once it propagates to the stable kernels we'll remove the workaround.

[kdave]

Patch now in linux kernel as commit https://git.kernel.org/linuis/89efda52e6b6930 .