`zfs send` immediately stops with "Assertion failed: (progress)"

[lapo-luchini]

System information

Type	Version/Name
Distribution Name	FreeBSD
Distribution Version	12.2-RELEASE-p2
Architecture	FreeBSD
ZFS Version	base system

Describe the problem you're observing

I migrated this server a few times already on new hardware over the years, by using zfs send.
Even right now I'm using zrepl to periodically have a remote copy to a distant site (for disaster recovery).
But trying to migrate on a new server in the same server farm recently I'm seeing this error:

$ sudo zfs snapshot -r z@recentSnapshot
$ sudo zfs send -Rv z@recentSnapshot >/dev/null
Assertion failed: (progress), file /usr/src/cddl/contrib/opensolaris/lib/libzfs/common/libzfs_sendrecv.c, line 1511.

This is happening in FreeBSD's libzfs_sendrecv.c line 1511 but as far as I can tell it is identical to current libzfs_sendrecv.c line L1343.

Describe how to reproduce the problem

I don't know how to reproduce it (or debug it) and the server is serving production data, so I can only do non-intrusive tests.

I guess it has something to do with clones, given the comments at line 1312 and line 1324 so maybe I can print something with zdb?
Any idea how to debug (and solve)?

[mabod]

sudo zfs send -Rv z@recentSnapshot >/dev/null

Do you get the same error when you redirect the send to a regular file instead of /dev/null ?

I'm not seeing this error when following the steps as described. Could you elaborate on the layout of the datasets with regard to those clones you mentioned?

[lapo-luchini]

Do you get the same error when you redirect the send to a regular file instead of /dev/null ?

Yes, sure, original use-case is really a zfs send | ssh to do a remote backup, I put the /dev/null there just to simplify the example.

I'm not seeing this error when following the steps as described. Could you elaborate on the layout of the datasets with regard to those clones you mentioned?

Actually I'm not sure… I was citing clones just because the assertions fails with (needagain=true, progress=false) and those seem to be relative to clones.

I didn't create any clone personally, but I'm using beadm and it certainly does; I'm also using cbsd which probably does too.
But I've been using those for a very long time, and zfs send never had any problem.
More recently I started using zrepl for periodic backup of some filesystems, but I don't think it creates any clone; it does probably set bookmarks though.
Is there any way to "print debug" and see at which filesystem exactly it stops?
On the other hand that's happening on a production system, so re-compiling kernel and restarting, while possible, isn't very easy and I'd keep that as a last resort.
If there's any zdb command I can issue to check the values of the filesystem…?

zfs send -v will be more verbose, showing each filesystem as it is sent.

[lapo-luchini]

It fails immediately, before anything is sent at all. :(

% sudo zfs snapshot -r z@migrate-5
% sudo zfs send -Rv z@migrate-5 >/dev/null
Assertion failed: (progress), file /usr/src/cddl/contrib/opensolaris/lib/libzfs/common/libzfs_sendrecv.c, line 1511.
zsh: abort      sudo zfs send -Rv z@migrate-5 > /dev/null

[mabod]

sudo zfs send -Rv z@migrate-5 >/dev/null

I am still concerned about the redirection to /dev/null. On my PC zfs send to /dev/null exits with an error:

8# zfs send -Rv zM2/zM2-BENCHMARK@2021-01-19--08:02 >/dev/null
full send of zM2/zM2-BENCHMARK@2021-01-19--08:02 estimated size is 4.53M
total estimated size is 4.53M
internal error: warning: cannot send 'zM2/zM2-BENCHMARK@2021-01-19--08:02': Invalid argument
[1]    33326 abort (core dumped)  zfs send -Rv zM2/zM2-BENCHMARK@2021-01-19--08:02 > /dev/null

If I take a regular file instead it works fine:

9# zfs send -Rv zM2/zM2-BENCHMARK@2021-01-19--08:02 >aaa      
full send of zM2/zM2-BENCHMARK@2021-01-19--08:02 estimated size is 4.53M
total estimated size is 4.53M

[lapo-luchini]

Here goes, on file:

% sudo zfs snapshot -r z@migrate-6
% sudo zfs send -Rv z@migrate-6 >a
Assertion failed: (progress), file /usr/src/cddl/contrib/opensolaris/lib/libzfs/common/libzfs_sendrecv.c, line 1511.
zsh: abort      sudo zfs send -Rv z@migrate-6 > a
% ls -al a
-rw-r--r--  1 lapo  users  694992 21 gen 14:26 a

Is there any way to decode (in a easy to read way) that file?
That might give me information about the last filesystem "properly" analyzed, or an approximation of that.

zstreamdump, though I am not confident it will work very well with a broken stream.

[lapo-luchini]

% sudo zfs snapshot -r z@motoko5-migrate-7
% sudo zfs send -Rv z@motoko5-migrate-7 >a
Assertion failed: (progress), file /usr/src/cddl/contrib/opensolaris/lib/libzfs/common/libzfs_sendrecv.c, line 1511.
% zstreamdump <a | fgrep "      name =" | wc -l
      65
% zfs list -Hr z | wc -l
      65

It seems to list all filesystems (and, by glancing with less, all snapshots as well)… I don't understand where it stops.

Any suggestions how to debug further?
(at this point, I'm even willing to patch kernel sources for increased logs and reboot the server)

[stale]

This issue has been automatically marked as "stale" because it has not had any activity for a while. It will be closed in 90 days if no further activity occurs. Thank you for your contributions.