Unable to receive encrypted raw send

[putnam]

System information

Type	Version/Name
Distribution Name	Debian
Distribution Version	bookworm/testing
Kernel Version	5.14.0-2
Architecture	amd64
OpenZFS Version	2.0.6-1

Describe the problem you're observing

I am trying to transfer an encrypted dataset to another pool using a raw send. Partially through this send, the receive side fails with an unclear error message.

The receiving system is also on 2.0.6, running Debian Bullseye on kernel 5.10.0-9.

Here are the commands that reproduce the issue reliably:

source# zfs send -Lcw -v tank/enc/private@zfs-auto-snap_monthly-2020-07-01-1152 | mbuffer -O 10.200.200.2:13337

dest# mbuffer -I 13337 | zfs receive -uv archive/dwight/tank/private

The dataset is about 1.3TB; at the 360GB mark, the receive reliably fails with the following error:

cannot receive new filesystem stream: invalid backup stream

Oddly, if you provide the -s flag to the recv (enable resume support) the error changes to this:

cannot receive new filesystem stream: kernel modules must be upgraded to receive this stream.

I tried sending the stream through zstreamdump, which did not complain. I also maximized debug logging, generated a resume token and tried resuming (right at whatever transaction caused the recv to fail), which generated these messages (gist). It's noisy; see line 1786 for when receive_object failed.

I don't know the exact steps to reproduce this problem with another dataset. Originally this dataset was on another pool in an unencrypted state. It was sent into the encrypted pool just fine, then I wanted to raw-send the newly-encrypted dataset offsite to another pool. It is at this point that I ran into this show-stopper.

See also the gist below which is the output of zstreamdump -vvv on the resumed send.

https://gist.github.com/putnam/3a467fdab6fe96ca048264c1bcd8a4b0

UPDATE: In the discussion below, I was able to shrink the affected dataset down to a ~70MB repro. See this comment for an attachment.

[putnam]

I'm actually not sure what to do here. I can't send a raw backup, full stop. And if I send it non-raw, I can't switch the incrementals to be raw because the IVs won't be the same. What am I meant to do in this situation? I will try running 2.1.0 in a VM or something on the target host to see if it makes any difference.

[rincebrain]

In theory, that message isn't supposed to happen unless you're sending from a newer version than being received on, and somehow it didn't notice a missing feature required until later, at which point the "fix" is pretty obvious.

In practice, that message comes out for all sorts of edge cases, and could probably stand to be refined, though I'd personally just suggest changing it to say something like "Either you need to upgrade your receiver to a newer version or, if they're not different (or the receiver is newer), you've hit an edge case we aren't handling, please report a bug."

Would you possibly be willing to share a full (e.g. not solely the resume portion, though that might be helpful in addition to the whole) raw (e.g. encrypted data blocks, though not some metadata) send stream with someone to debug? I'm probably not an ideal candidate, not being familiar with the over the wire format of send/recv or all the places encrypted recv touches, but it'd probably be very helpful if someone could repro it locally and iterate on whatever's going awry rather than having to round trip to you for testing each time. (It's perfectly okay to say no if you're not comfortable;; it would just be useful to know )

I suspect it'll just say "lol it's encrypted go pound sand", from recollection, but what do zdb -dbdbdbdbdbdb tank/enc/private@zfs-auto-snap_monthly-2020-07-01-1152 269839 and [...] 269838 have to say?

You said the other system is on bullseye with 2.0.6, is that from backports? Could you give us the full output of zfs version on both hosts, just for completeness?

Also just for completeness, could you try receiving the stream locally somewhere on the sender to see if it errors the same way?

One final remark - while you're correct that you couldn't go from non-raw to raw send, you could still do non-raw send and enable encryption on the receiver like with any non-raw send being received. This of course doesn't work if you don't trust the receiver to have access to the encryption key, but that may or may not be an issue in your environment.

[putnam]

Thanks for your reply, @rincebrain

I'll try to address everything in order.

Sharing a full send is going to be hard. It's 1.3TB, and it's a dataset I'd prefer not to directly share anyway. But I realize that it may only repro on this exact snapshot on this exact dataset. I don't mind the back-and-forth, though.

Attached is the output of zdb for each of those objects as requested:
zdb_269838.txt
zdb_269839.txt

The target system is on 2.0.6 via backports.

Source zfs version:

zfs-2.0.6-1
zfs-kmod-2.0.6-1

Target zfs version:

zfs-2.0.6-1~bpo11+1
zfs-kmod-2.0.6-1~bpo11+1

I just tried sending the stream locally from source to a new dataset with the same error at the same position:

# zfs send -Lcw tank/enc/private@zfs-auto-snap_monthly-2020-07-01-1152 | mbuffer | zfs recv -u tank/privatetest
in @  128 MiB/s, out @  0.0 kiB/s,  360 GiB total, buffer  10% full
cannot receive new filesystem stream: invalid backup stream

As a side note, the performance of zfs send here has been relatively poor. Before I switched to encryption I could zfs send at a rate of about 1-1.5 GiB/second. Unlike sequential scrubs, zfs send seems to be IOP bound. I'm not sure what makes the encrypted version so much slower. The pool topology on the source machine is 5x11 raidz2. Target is 3x12 raidz2. All are spinning disks between 12T-18T in size.

Regarding the ability to still send a backup unencrypted, it is indeed the case that I want raw sends to work since they'll be going to an untrusted offsite system.

[rincebrain]

Sharing a full send is going to be hard. It's 1.3TB, and it's a dataset I'd prefer not to directly share anyway. But I realize that it may only repro on this exact snapshot on this exact dataset. I don't mind the back-and-forth, though.

Okay, I just figured it'd be worth asking (though it'd also be interesting to see if you could create a clone with only the minimal data necessary to reproduce - possibly by doing a zfs clone of the snapshot, finding just the specific files those IDs I asked you for the zdb data for map to, and nuking everything but them and their containing dirs...you could find the specific files with find /path/to/mounted/snapshot/or/clone -type f -inum [the number in question], though this may take a bit. It's also likely to be error prone, so mayhaps clone, snapshot the clone, verify the clone snapshot still errors on send in the same place, then nuke everything but the entire unmodified directory and parents of those objects, then snapshot that and make sure the snapshot still reproduces, and so on...)

I don't know of any performance bottlenecks specific to only raw send - I'd be curious to check out a flamegraph of send -Lec versus send -w on the same dataset over, say, sleep 120 - it might be IO bound, but I'd like to see if there's any visibly different callstacks that turn up...

RAIDZs aren't particularly made of IOPs in general - what makes you think it's IOP bound and not CPU or other bound? (If it's "it's in iowait a bunch", that could mean many things.)

Yeah, I'm not surprised that unencrypted doesn't work for you, just thought I'd ask. :/

[putnam]

Good idea on the clone strategy. Worked a charm, repros now with a 73M dataset.

Here are my notes as I did this, just for reference:

# Created clone
zfs clone tank/enc/private@zfs-auto-snap_monthly-2020-07-01-1152 tank/privatetest

# Initial clone snapshot (no modifications)
zfs snapshot tank/privatetest@clonesnap

# Sending full clone triggers same error at same position
zfs send -Lcw tank/privatetest@clonesnap |mbuffer|zfs recv -u tank/privatetest_receive
in @  128 MiB/s, out @  0.0 kiB/s,  360 GiB total, buffer  21% full
cannot receive new filesystem stream: invalid backup stream

# Whittling down filesystem to only affected inodes
find /tank/privatetest -type f -inum 269838 -or -inum 269839

# deleted all other folders, size is now 285M
zfs snapshot tank/privatetest@afterdeletion

# tried send again, failed again (faster this time)
zfs send -Lcw tank/privatetest@afterdeletion | zfs recv -u tank/privatetest_receive
cannot receive new filesystem stream: invalid backup stream

# deleted sibling files to minimize size, snapped, tested again (73M)
zfs snapshot tank/privatetest@afterdeletion2
zfs send -Lcw tank/privatetest@afterdeletion2 | zfs recv -u tank/privatetest_receive
cannot receive new filesystem stream: invalid backup stream

Here is a dropbox link of the zfs send:
https://www.dropbox.com/s/p7yykx47syl7jm5/zfs_send_culled_private.bin.zip?dl=0

You're right that I sort of assumed it was IOP-bound simply because of frequent iowait on zfs send. CPU usage is only 15-20% of one core. I will definitely dig into this separately once I get past the send issue.

[putnam]

To add a little more context if it matters, here is this dataset's zfs send/recv "provenance". Originally it was unencrypted on a single host running normally. When I prepared the offsite, I wanted to also take this opportunity to transfer the data back-and-forth to get encryption on both hosts and also rebalance data across vdevs.

The unencrypted dataset was sent to the backup machine into an encrypted dataset, then sent back into the primary pool so it would be encrypted -- I did not use a raw send here, though, and in order for raw sends to work on the offsite I needed to once again wipe the pool on the offsite and now send the dataset with the raw flag.

It would have been much easier to have used a raw send, but there is a long standing bug that directly affected my workflow here. Once I sent the dataset to the offsite, I needed to briefly wipe the source pool and send everything back, but I kept the source system online by mounting the offsite over nfs. This resulted in the dataset changing and needing to be sent back, which is affected by that bug. I confirmed it by dry-running before I started this journey and didn't want to take a risk. So I decided to do an extra transfer once everything was situated on the primary host so the raw sends can start clean and never risk being modified by the offsite and being sent back.

tl;dr:

1. Dataset began unencrypted on SOURCE (2.0.6)
2. Dataset was sent into a new, encrypted dataset on TARGET (2.0.3)
3. Source pool destroyed and recreated. Dataset was then sent back from TARGET to SOURCE, with a new encryption key.
4. Finally, attempted to replicate SOURCE to TARGET again using a raw send, which led to this ticket. In this process I updated TARGET to 2.0.6 just to see if it affected anything, which it did not.

The most recent zfs send to work fine was #3. The affected files are not new at all. Old stuff.

[rincebrain]

Just to be clear, did you send just this dataset, or was it part of a hierarchy of datasets being sent around?

You could always try cherrypicking/un-reverting d1d4769 and seeing if it helps your life.

[putnam]

Only this dataset in this case. Not a hierarchy.

…

On Thu, Nov 11, 2021 at 4:07 PM Rich Ercolani ***@***.***> wrote: Just to be clear, did you send *just* this dataset, or was it part of a hierarchy of datasets being sent around? You could always try cherrypicking/un-reverting d1d4769 <d1d4769> and seeing if it helps your life. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#12720 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAASBT6QMF2ZHMRNQCWIEC3ULQ5AHANCNFSM5HH4C6DQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

[putnam]

I'm a little surprised -- is nobody working on bugs on raw sends? I can reliably reproduce this and provided an easy repro case. I'd definitely like to dig in myself but it's a lot to spin up. Maybe I hit some edge case as I'm surprised nobody else seems to have run into this. As it stands I'm just waiting until 2.1.1 lands in Debian (the packager finally uploaded to experimental a week ago) to try to test further.

[rincebrain]

Not quite nobody, no. Just almost. I've failed to convince anybody else so far to work on these, but I'm trying to knock them out one at a time, unfortunately slowly. I believe the root cause(s) for some of the bugs that work similarly to this one are understood, just no fix written and remaining committed yet. As far as how this situation arose, it seems it works well for the few big companies that use it, and there haven't been many in the intersection of {willing to try fixing, uses native encryption} - even I don't meet that criteria, I don't use it normally. I'm displeased by this, but I'm already doing all I can about it.

…

On Wed, Dec 8, 2021, 4:38 PM Chris Putnam ***@***.***> wrote: I'm a little surprised -- is nobody working on bugs on raw sends? I can reliably reproduce this and provided an easy repro case. I'd definitely like to dig in myself but it's a lot to spin up. Maybe I hit some edge case as I'm surprised nobody else seems to have run into this. As it stands I'm just waiting until 2.1.1 lands in Debian (the packager finally uploaded to experimental a week ago) to try to test further. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#12720 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AABUI7NITOV47BODKQ6VEGLUP7F5VANCNFSM5HH4C6DQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

[bghira]

I'm a little surprised -- is nobody working on bugs on raw sends?

for what it's worth, this feature has been broken since it landed and it's received no love from the dev team. Datto contributed it but doesn't wish to contribute to its maintenance. this is one reason I'm skeptical of all the new features (dRAID, Object storage) - they're running the BTRFS playbook, adding feature after feature without validating the existing functionality. all of that, on top of major reorganisation of code as platforms consolidate on OpenZFS.

i honestly don't blame anyone who doesn't want to upgrade from 0.8.x

[putnam]

I have updated to 2.1.1 after it landed in Debian unstable.

The send stream generated by 2.1.1 is identical to the send stream generated by 2.0.6. The operation fails in the same way:

$ cat /sys/module/zfs/version 
2.1.1-3

$ zfs send -Lcw tank/privatetest@afterdeletion2 > zfs_send_culled_private_2.1.1.bin

$ diff -s zfs_send_culled_private.bin zfs_send_culled_private_2.1.1.bin 
Files zfs_send_culled_private.bin and zfs_send_culled_private_2.1.1.bin are identical

# (this is the equivalent of downloading my repro and catting it to zfs recv)
$ zfs send -Lcw tank/privatetest@afterdeletion2 | zfs recv -u tank/privatetest_receive
cannot receive new filesystem stream: invalid backup stream