Input/output error mounting encrypted filesystem after raw send/receive

[rjhind]

I've just started storing Parallels virtual machines on encrypted filesystems on a zpool and am trying to back them up to another zpool using raw send receive however more often than not, I'm unable to mount the received file system and get an input/output error.

I've managed to reproduce it and have put an example virtual machine at: https://www.dropbox.com/s/djpqjm4z9122sqi/ubuntu-pvm.tar.bz2?dl=0

The bz2 contains a pre-allocated 64GB disk but if you expand it on to a zpool with zfs then it'll only take ~1GB actual space as the disk is fairly blank.

To reproduce the errors, I've been using the following:

$ sudo -s
$ cd /Volumes/tank
$ zfs create -o encryption=on -o keyformat=passphrase -o keylocation=prompt tank/test
Enter passphrase: testtest
Re-enter passphrase: testtest
$ cd test
$ tar xf ~/Downloads/ubuntu-pvm.tar.bz2
$ zfs snap tank/test@snap1
$ zfs send -w tank/test@snap1 | zfs recv -u tank/test-2
$ zfs load-key tank/test-2
Enter passphrase for 'tank/test-2': testtest
$ zfs mount tank/test-2
cannot mount 'tank/test-2': Input/output error
$

I've run the same thing on an Ubuntu 18 box with ZOL-0.8.0-rc4 (build from source) and can successfully mount the received filesystem. But if I send from the OS X box to ZOL, I could mount the file system but got errors trying to ls the contents.

[rjhind]

The actual ZOL version I was testing with is

zfs-0.8.0-rc4_5_g2de17298d
zfs-kmod-0.8.0-rc4_5_g2de17298d

[lundman]

dmu_recv.c is identical to ZOL, but

dmu_send.c has this difference

 dump_freeobjects(dmu_sendarg_t *dsp, uint64_t firstobj, uint64_t numobjs)
 {
        struct drr_freeobjects *drrfo = &(dsp->dsa_drr->drr_u.drr_freeobjects);
+       uint64_t maxobj = DNODES_PER_BLOCK *
+           (DMU_META_DNODE(dsp->dsa_os)->dn_maxblkid + 1);
+
+       /*
+        * ZoL < 0.7 does not handle large FREEOBJECTS records correctly,
+        * leading to zfs recv never completing. to avoid this issue, don't
+        * send FREEOBJECTS records for object IDs which cannot exist on the
+        * receiving side.
+        */
+       if (maxobj > 0) {
+               if (maxobj < firstobj)
+                       return (0);
+
+               if (maxobj < firstobj + numobjs)
+                       numobjs = maxobj - firstobj;
+       }

[lundman]

We should keep an eye on ZOL openzfs/zfs#8852

[rjhind]

Sending from 03X to ZOL provoked the issue. If I create a bz2 from the send stream, perhaps that will give something to investigate on the ZOL ticket. If the send stream is corrupted, then really it should be rejected rather than received OK but the fs fail to mount/ls.

[behlendorf]

@aabc123 an example send stream that reproduces the issue would be tremendously helpful.

[rjhind]

@behlendorf Here is a snap which can be received but fails to mount on both ZOL and O3X. https://www.dropbox.com/s/no0ehylt59vtieh/ubuntu-pvm.snap.bz2?dl=0

It was created with -wR. I've tested just receiving with -u then load-key and mount. On O3X the mount fails with input/output error. On ZOL I get I/o error and invalid argument.

filesystem 'rpool/home/russell/vm-3' can not be mounted: Input/output error
cannot mount 'rpool/home/russell/vm-3': Invalid argument

[rjhind]

Sorry, just realised bzipping was pointless as its encrypted.

[behlendorf]

@aabc123 would it be possible to get the original pool as well, or is it too large? Being able to compare the received blocks against the originals should help us find the discrepancy.

[rjhind]

That pool is too large. I can do a non-raw send (I've tried -ceL) and that seems to work fine and give you that snap shot?

If that's no use then I should be able to create a new file-based pool, try and provoke the issue in that and generate a raw snap from that along with a copy of the file-based pool for you?

[tcaputi]

Having the pool file itself would be very helpful. Doing a non-raw send will cause a few changes that might be hard to track.

[rjhind]

OK, I've got them created.

8gb File-based pool 'test': https://www.dropbox.com/s/ehb0g2jx8cqt427/test.zpool.bz2?dl=0
Snapshot of -wR test/vm@snap2 from the pool: https://www.dropbox.com/s/300dm8xyc6xh3hj/test_vm.zsnap?dl=0

Both are ~1.3GB. Passphrase for test/vm testtest

You shouldn't really need the snapshot file as importing the pool on to my ZOL-0.8.0-rc4 system and do a send there, the resultant file system is still not mountable due to the I/o and invalid argument errors.

$ sudo zfs send -wR test/vm@snap2 | sudo zfs recv -u rpool/home/russell/test_vm2
$ sudo zfs load-key rpool/home/russell/test_vm2
Enter passphrase for 'rpool/home/russell/test_vm2':
$ sudo zfs mount rpool/home/russell/test_vm2
filesystem 'rpool/home/russell/test_vm2' can not be mounted: Input/output error
cannot mount 'rpool/home/russell/test_vm2': Invalid argument

[tcaputi]

@aabc123 Can you try this patch from ZoL? openzfs/zfs#8857

[lundman]

https://github.com/openzfsonosx/zfs/tree/pr8857

[rjhind]

@tcaputi @lundman This seems to have fixed the raw sends for me (I've tried both ZOL and O3X).

[rjhind]

Closed fixed by 3699861