Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

(Solved) - Syncing Issue (Possible duplicate of 678 681 - but my setup is different so didt think i should write it there) #683

Closed
Darkyere opened this issue Oct 5, 2021 · 4 comments

Comments

@Darkyere
Copy link

Darkyere commented Oct 5, 2021

Every time im trying to receive a Dataset. At the end of the transfer i get.

cannot receive: failed to read from stream
cannot receive new filesystem stream: dataset does not exist

If i use the same command again there is allways data missing that it resyncs.

Followed by another error


This is my setup

Im receiving from Ubuntu 21.04 on RPI4 (ARM64)
Kernel version: Linux rpi4-ubuntudesktop 5.11.0-1019-raspi #20-Ubuntu SMP PREEMPT Tue Sep 21 15:23:42 UTC 2021 aarch64 aarch64 aarch64 GNU/Linux

And receiving on Ubuntu 20.04 on RPI4 (x86_64)
Kernel Version: Linux zboxri531-lubuntults 5.11.0-37-generic #41~20.04.2-Ubuntu SMP Fri Sep 24 09:06:38 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

On both version im using the built in ZFS in kernel.
No dkms module in use here.

I followed the install guide for ubuntu and this is my syncoid version

/usr/sbin/syncoid version 2.1.0
(Getopt::Long::GetOptions version 2.50; Perl version 5.30.0)

I have done as described in Syncoid wiki on the end sending for non root user.

sudo zfs allow -u <MyUser> send,snapshot,hold <MyPool>

I have used sudo su on the receiving end so all commands are Super User.


My command with debug is as following.

syncoid SendingUser@192.168.1.170:"Storage/OS/Linux/Chromium OS" "BackUpAndSync/OS/Linux/Chromium OS" --compress none --sshcipher chacha20-poly1305@openssh.com --sshport 12345 --sshkey "/dest/to/key" --no-privilege-elevation --debug
DEBUG: SSHCMD: ssh -c chacha20-poly1305@openssh.com  -p 12345 -i /dest/to/key
Enter passphrase for key '/dest/to/key': 
DEBUG: compression forced off from command line arguments.
DEBUG: checking availability of mbuffer on source...
DEBUG: checking availability of mbuffer on target...
DEBUG: checking availability of pv on local machine...
DEBUG: checking availability of zfs resume feature on source...
DEBUG: checking availability of zfs resume feature on target...
DEBUG: syncing source Storage/OS/Linux/Chromium OS to target BackUpAndSync/OS/Linux/Chromium OS.
DEBUG: getting current value of syncoid:sync on Storage/OS/Linux/Chromium OS...
ssh -c chacha20-poly1305@openssh.com  -p 12345 -i /dest/to/key -S /tmp/syncoid-SendingUser@192.168.1.170-1633466255 SendingUser@192.168.1.170  zfs get -H syncoid:sync ''"'"'Storage/OS/Linux/Chromium OS'"'"''
DEBUG: checking to see if BackUpAndSync/OS/Linux/Chromium OS on  is already in zfs receive using  ps -Ao args= ...
DEBUG: checking to see if target filesystem exists using "  zfs get -H name 'BackUpAndSync/OS/Linux/Chromium OS' 2>&1 |"...
DEBUG: getting list of snapshots on Storage/OS/Linux/Chromium OS using ssh -c chacha20-poly1305@openssh.com  -p 12345 -i /dest/to/key -S /tmp/syncoid-SendingUser@192.168.1.170-1633466255 SendingUser@192.168.1.170  zfs get -Hpd 1 -t snapshot guid,creation ''"'"'Storage/OS/Linux/Chromium OS'"'"'' |...
DEBUG: creating sync snapshot using "ssh -c chacha20-poly1305@openssh.com  -p 12345 -i /dest/to/key -S /tmp/syncoid-SendingUser@192.168.1.170-1633466255 SendingUser@192.168.1.170  zfs snapshot ''"'"'Storage/OS/Linux/Chromium OS'"'"''@syncoid_zboxri531-lubuntults_2021-10-05:22:37:55-GMT02:00
"...
DEBUG: target BackUpAndSync/OS/Linux/Chromium OS does not exist.  Finding oldest available snapshot on source Storage/OS/Linux/Chromium OS ...
DEBUG: getting estimated transfer size from source -S /tmp/syncoid-SendingUser@192.168.1.170-1633466255 SendingUser@192.168.1.170 using "ssh -c chacha20-poly1305@openssh.com  -p 12345 -i /dest/to/key -S /tmp/syncoid-SendingUser@192.168.1.170-1633466255 SendingUser@192.168.1.170  zfs send  -nvP ''"'"'Storage/OS/Linux/Chromium OS@syncoid_ReceivingUser-Aspire-TC-115_2021-08-20:09:45:35-GMT02:00'"'"'' 2>&1 |"...
DEBUG: sendsize = 6877753400
INFO: Sending oldest full snapshot Storage/OS/Linux/Chromium OS@syncoid_ReceivingUser-Aspire-TC-115_2021-08-20:09:45:35-GMT02:00 (~ 6.4 GB) to new target filesystem:
DEBUG: ssh -c chacha20-poly1305@openssh.com  -p 12345 -i /dest/to/key -S /tmp/syncoid-SendingUser@192.168.1.170-1633466255 SendingUser@192.168.1.170 ' zfs send  '"'"'Storage/OS/Linux/Chromium OS'"'"'@'"'"'syncoid_ReceivingUser-Aspire-TC-115_2021-08-20:09:45:35-GMT02:00'"'"' | mbuffer  -q -s 128k -m 16M 2>/dev/null' | mbuffer  -q -s 128k -m 16M 2>/dev/null | pv -p -t -e -r -b -s 6877753400 |  zfs receive  -s -F 'BackUpAndSync/OS/Linux/Chromium OS'
DEBUG: checking to see if BackUpAndSync/OS/Linux/Chromium OS on  is already in zfs receive using  ps -Ao args= ...
6,41GiB 0:02:38 [41,3MiB/s] [=============================================================================================================================================>] 100%            
cannot receive: failed to read from stream
cannot receive new filesystem stream: dataset does not exist
CRITICAL ERROR: ssh -c chacha20-poly1305@openssh.com  -p 12345 -i /dest/to/key -S /tmp/syncoid-SendingUser@192.168.1.170-1633466255 SendingUser@192.168.1.170 ' zfs send  '"'"'Storage/OS/Linux/Chromium OS'"'"'@'"'"'syncoid_ReceivingUser-Aspire-TC-115_2021-08-20:09:45:35-GMT02:00'"'"' | mbuffer  -q -s 128k -m 16M 2>/dev/null' | mbuffer  -q -s 128k -m 16M 2>/dev/null | pv -p -t -e -r -b -s 6877753400 |  zfs receive  -s -F 'BackUpAndSync/OS/Linux/Chromium OS' failed: 256 at /usr/sbin/syncoid line 492.

Doing the command again sometimes finishes it.
But then end with a new error.

cannot destroy snapshots: permission denied

Command with debug as follow

syncoid SendingUser@192.168.1.170:"Storage/OS/Linux/Chromium OS" "BackUpAndSync/OS/Linux/Chromium OS" --compress none --sshcipher chacha20-poly1305@openssh.com --sshport 12345 --sshkey "/dest/to/key" --no-privilege-elevation --debug
DEBUG: SSHCMD: ssh -c chacha20-poly1305@openssh.com  -p 12345 -i /dest/to/key
Enter passphrase for key '/dest/to/key': 
DEBUG: compression forced off from command line arguments.
DEBUG: checking availability of mbuffer on source...
DEBUG: checking availability of mbuffer on target...
DEBUG: checking availability of pv on local machine...
DEBUG: checking availability of zfs resume feature on source...
DEBUG: checking availability of zfs resume feature on target...
DEBUG: syncing source Storage/OS/Linux/Chromium OS to target BackUpAndSync/OS/Linux/Chromium OS.
DEBUG: getting current value of syncoid:sync on Storage/OS/Linux/Chromium OS...
ssh -c chacha20-poly1305@openssh.com  -p 12345 -i /dest/to/key -S /tmp/syncoid-SendingUser@192.168.1.170-1633466945 SendingUser@192.168.1.170  zfs get -H syncoid:sync ''"'"'Storage/OS/Linux/Chromium OS'"'"''
DEBUG: checking to see if BackUpAndSync/OS/Linux/Chromium OS on  is already in zfs receive using  ps -Ao args= ...
DEBUG: checking to see if target filesystem exists using "  zfs get -H name 'BackUpAndSync/OS/Linux/Chromium OS' 2>&1 |"...
DEBUG: getting current value of receive_resume_token on BackUpAndSync/OS/Linux/Chromium OS...
  zfs get -H receive_resume_token 'BackUpAndSync/OS/Linux/Chromium OS'
DEBUG: no receive token found 
DEBUG: getting list of snapshots on Storage/OS/Linux/Chromium OS using ssh -c chacha20-poly1305@openssh.com  -p 12345 -i /dest/to/key -S /tmp/syncoid-SendingUser@192.168.1.170-1633466945 SendingUser@192.168.1.170  zfs get -Hpd 1 -t snapshot guid,creation ''"'"'Storage/OS/Linux/Chromium OS'"'"'' |...
DEBUG: getting list of snapshots on BackUpAndSync/OS/Linux/Chromium OS using   zfs get -Hpd 1 -t snapshot guid,creation 'BackUpAndSync/OS/Linux/Chromium OS' |...
DEBUG: creating sync snapshot using "ssh -c chacha20-poly1305@openssh.com  -p 12345 -i /dest/to/key -S /tmp/syncoid-SendingUser@192.168.1.170-1633466945 SendingUser@192.168.1.170  zfs snapshot ''"'"'Storage/OS/Linux/Chromium OS'"'"''@syncoid_zboxri531-lubuntults_2021-10-05:22:49:28-GMT02:00
"...
DEBUG: getting current value of -p used on BackUpAndSync/OS/Linux/Chromium OS...
  zfs get -H -p used 'BackUpAndSync/OS/Linux/Chromium OS'
DEBUG: checking to see if BackUpAndSync/OS/Linux/Chromium OS on  is already in zfs receive using  ps -Ao args= ...
DEBUG: rolling back target to BackUpAndSync/OS/Linux/Chromium OS@syncoid_ReceivingUser-Aspire-TC-115_2021-08-20:09:45:35-GMT02:00...
 zfs rollback -R 'BackUpAndSync/OS/Linux/Chromium OS'@'syncoid_ReceivingUser-Aspire-TC-115_2021-08-20:09:45:35-GMT02:00'
DEBUG: getting estimated transfer size from source -S /tmp/syncoid-SendingUser@192.168.1.170-1633466945 SendingUser@192.168.1.170 using "ssh -c chacha20-poly1305@openssh.com  -p 12345 -i /dest/to/key -S /tmp/syncoid-SendingUser@192.168.1.170-1633466945 SendingUser@192.168.1.170  zfs send  -nvP -I ''"'"'Storage/OS/Linux/Chromium OS@syncoid_ReceivingUser-Aspire-TC-115_2021-08-20:09:45:35-GMT02:00'"'"'' ''"'"'Storage/OS/Linux/Chromium OS@syncoid_zboxri531-lubuntults_2021-10-05:22:49:28-GMT02:00'"'"'' 2>&1 |"...
DEBUG: sendsize = 46304
Sending incremental Storage/OS/Linux/Chromium OS@syncoid_ReceivingUser-Aspire-TC-115_2021-08-20:09:45:35-GMT02:00 ... syncoid_zboxri531-lubuntults_2021-10-05:22:49:28-GMT02:00 (~ 45 KB):
DEBUG: ssh -c chacha20-poly1305@openssh.com  -p 12345 -i /dest/to/key -S /tmp/syncoid-SendingUser@192.168.1.170-1633466945 SendingUser@192.168.1.170 ' zfs send  -I '"'"'Storage/OS/Linux/Chromium OS'"'"'@'"'"'syncoid_ReceivingUser-Aspire-TC-115_2021-08-20:09:45:35-GMT02:00'"'"' '"'"'Storage/OS/Linux/Chromium OS'"'"'@'"'"'syncoid_zboxri531-lubuntults_2021-10-05:22:49:28-GMT02:00'"'"' | mbuffer  -q -s 128k -m 16M 2>/dev/null' | mbuffer  -q -s 128k -m 16M 2>/dev/null | pv -p -t -e -r -b -s 46304 |  zfs receive  -s -F 'BackUpAndSync/OS/Linux/Chromium OS' 2>&1
67,5KiB 0:00:00 [ 504KiB/s] [==============================================================================================================================================] 149%            
DEBUG: pruning up to 10 obsolete sync snapshots...
DEBUG: ssh -c chacha20-poly1305@openssh.com  -p 12345 -i /dest/to/key -S /tmp/syncoid-SendingUser@192.168.1.170-1633466945 SendingUser@192.168.1.170  zfs destroy 'Storage/OS/Linux/Chromium OS'@syncoid_zboxri531-lubuntults_2021-10-05:22:37:55-GMT02:00
cannot destroy snapshots: permission denied
WARNING: ssh -c chacha20-poly1305@openssh.com  -p 12345 -i /dest/to/key -S /tmp/syncoid-SendingUser@192.168.1.170-1633466945 SendingUser@192.168.1.170 ' zfs destroy '"'"'Storage/OS/Linux/Chromium OS'"'"'@syncoid_zboxri531-lubuntults_2021-10-05:22:37:55-GMT02:00' failed: 256 at /usr/sbin/syncoid line 1380.

I hope its possible to be resolved.
Ty for reading,
Darkyere

@Darkyere
Copy link
Author

After reading

syncoid fails in Ubuntu 20.04.2 LTS kernel 5.11.0-25 due to zfs receive -s issue #665

Which explains that a wrong version of ZFS userspace (zfsutils-linux) and ZFS kernel module (in my case build into the kernel) is the reason for this.

And

Ubuntu 20.04.2 LTS kernel 5.11.0-25 zfs send | receive broken


I tried to install Ubuntu 21.10 Beta in a VM.
Which has kernel

Linux Ubuntu21 5.13.0-16-generic #16-Ubuntu SMP Fri Sep 3 14:53:27 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

And ZFS userspace and kernel module

zfs-2.0.6-1ubuntu2
zfs-kmod-2.0.3-8ubuntu7

I know the versions dont match exactly. But the Syncoid send/recieve worked without a hitch.

So suppose i have to wait for Ubuntu 21.10 on some machines.
And either downgrade the kernel with the correct kernel module or possibly install the right ZFS kernel version manually on ubuntu 20.04.


Hope this will help others out there figuring this out.

Best regards,
Darkyere

@ghfields
Copy link

I've expected this problem would eventually happen with Ubuntu having HWE kernels with zfs integrated, but the userspace tools of the original distro release. So... today with a 20.04.3 focal LTS HWE kernel 5.11 and the zfs 0.8.3 userspace, I set out to surgically upgrade the userspace tools. I succeeded using apt pinning. https://help.ubuntu.com/community/PinningHowto

First, I added APT::Default-Release "focal"; to "/etc/apt/apt.conf.d/01-vendor-ubuntu"

Then, since 5.11 is the hirsute kernel, I added the following to "/etc/apt/sources.list" (adjust for newer HWE kernels):

deb http://us.archive.ubuntu.com/ubuntu/ hirsute main restricted
deb http://us.archive.ubuntu.com/ubuntu/ hirsute-updates main restricted
deb http://security.ubuntu.com/ubuntu hirsute-security main restricted

Then I created the file "/etc/apt/preferences.d/zfs.pref" and put the following in it:

Package: zfsutils-linux
Pin: release n=focal
Pin-Priority: -10

Package: libnvpair3linux
Pin: release n=focal
Pin-Priority: -10

Package: libuutil3linux
Pin: release n=focal
Pin-Priority: -10

Package: libzfs4linux
Pin: release n=focal
Pin-Priority: -10

Package: libzpool4linux
Pin: release n=focal
Pin-Priority: -10

Package: libc6
Pin: release n=focal
Pin-Priority: -10

Package: libc6:i386
Pin: release n=focal
Pin-Priority: -10

Package: libc-bin
Pin: release n=focal
Pin-Priority: -10

Package: libc-dev-bin
Pin: release n=focal
Pin-Priority: -10

Package: fakeroot
Pin: release n=focal
Pin-Priority: -10

Package: libfakeroot
Pin: release n=focal
Pin-Priority: -10

Package: locales
Pin: release n=focal
Pin-Priority: -10

Package: libc:i386
Pin: release n=focal
Pin-Priority: -10

Package: libc6-dbg
Pin: release n=focal
Pin-Priority: -10

Package: libc6-dev
Pin: release n=focal
Pin-Priority: -10

Finally, I did a apt update && apt upgrade libc6 libc6:i386 zfsutils-linux

In the end I got:

# zfs version
zfs-2.0.2-1ubuntu5.3
zfs-kmod-2.0.2-1ubuntu5.1

@gnordli
Copy link

gnordli commented Oct 25, 2021 via email

@Darkyere
Copy link
Author

Plz. go to this page and vote this is a problem for you too if u read this post.

When using HWE, zfs-kmod and zfs user tools versions must match

@Darkyere Darkyere changed the title Syncing Issue (Possible duplicate of 678 681 - but my setup is different so didt think i should write it there) (Solved) - Syncing Issue (Possible duplicate of 678 681 - but my setup is different so didt think i should write it there) Jan 18, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants