cannot use zfs send recursivly #1388
Comments
|
This looks new to me. The core would be helpful but I don't have the symbols for your build of the zfs tools. Can you fire up gdb and post the backtrace from the core. |
|
here you go #0 0x00000307a4291b85 in raise () from /lib64/libc.so.6 |
|
now with more debug symbols #0 0x00000307a4291b85 in raise () from /lib64/libc.so.6 |
|
This indicates that the ZFS_IOC_SEND ioctl which was passed to the kernel contained an invalid argument. It would be useful to see what those arguments were, perhaps there was an invalid snapshot name. Can you either:
|
|
Here's the patch http://bpaste.net/show/89072/ #0 0x00000301ae670b18 in vfprintf () from /lib64/libc.so.6 |
|
did this help at all? |
|
Static analysis from issue #1392 suggests a possible cause:
@behlendorf As easy as it should be to fix that, I do not have time right now, so it is all yours. |
|
@prometheanfire Let me get you a patch to try to provide the needed debugging (maybe today). Also if you get a chance the @ryao I don't think #1392 is likely the root cause since the reason for the core is due to an explicit |
|
@behlendorf I don't have access to the trace-cmd but do you have that patch for me to test? |
|
I narrowed it down to a specific dataset that I cannot send. zfs list zfs list -t snap works doesn't work zfs send khorne-m4-zp00/ROOT@20130506-backup > asd zdb khorne-m4-zp00/ROOT@20130506-backup zdb khorne-m4-zp00/ROOT |
|
I noticed a couple of odd things as well about the dataset that is giving the errors. these pop out at me version = - |
|
The output of trace-cmd revealed that failing call path in the kernel is the following which returns EINVAL. Unfortunately the error code from zfs_get_zplprop() is lost we don't know exactly why it was unable to fetch the version. However, based on the previous comment I suspect that somehow the property is outright missing. How that could happen isn't at all clear, but this chunk of kernel code can't cope with it and correctly errors out. You can apply the following patch to get the exact error code. diff --git a/module/zfs/dmu_send.c b/module/zfs/dmu_send.c
index b256803..79ce9da 100644
--- a/module/zfs/dmu_send.c
+++ b/module/zfs/dmu_send.c
@@ -464,7 +464,10 @@ dmu_send(objset_t *tosnap, objset_t *fromsnap, boolean_t fr
#ifdef _KERNEL
if (dmu_objset_type(tosnap) == DMU_OST_ZFS) {
uint64_t version;
- if (zfs_get_zplprop(tosnap, ZFS_PROP_VERSION, &version) != 0) {
+ err = zfs_get_zplprop(tosnap, ZFS_PROP_VERSION, &version);
+ if (err) {
+ printk("zfs_get_zplprop(%p, %d, %d) = %d\n",
+ tosnap, ZFS_PROP_VERSION, version, err);
kmem_free(drr, sizeof (dmu_replay_record_t));
return (EINVAL);
}If you're more ambitious simply you can apply this path to comment out the version check and set diff --git a/module/zfs/dmu_send.c b/module/zfs/dmu_send.c
index b256803..3157b37 100644
--- a/module/zfs/dmu_send.c
+++ b/module/zfs/dmu_send.c
@@ -463,10 +463,13 @@ dmu_send(objset_t *tosnap, objset_t *fromsnap, boolean_t f
#ifdef _KERNEL
if (dmu_objset_type(tosnap) == DMU_OST_ZFS) {
- uint64_t version;
+ uint64_t version = ZPL_VERSION_SA;
+
+#if 0
if (zfs_get_zplprop(tosnap, ZFS_PROP_VERSION, &version) != 0) {
kmem_free(drr, sizeof (dmu_replay_record_t));
return (EINVAL);
+#endif
}
if (version == ZPL_VERSION_SA) {
DMU_SET_FEATUREFLAGS( |
|
I was more ambitious ./tmp/trace-cmd/trace-cmd record -p function_graph -F zfs send khorne-m4-zp00/ROOT@20130807 > /mnt/20130807-backup-zfssend.gz trace-cmd: Interrupted system call http://dev.gentoo.org/~prometheanfire/crash-logs/zfs-send-trace-2.out.xz |
|
closing this since it seems like it's just corruption on my end zpool status -vpool: khorne-m4-zp00 errors: Permanent errors have been detected in the following files: |
|
I ran into something that looks quite similar to this today. I've been using the ZFS Stable Ubuntu PPA (Raring) and just today updated my ZFS debs from 0.6.1 to 0.6.2. After the update, my recursive zfs sends no longer worked: I downgraded to 0.6.1 and the send worked properly. Unfortunately I'm in the middle of a couple projects and am not totally comfortable running additional tests/patches right now. However, I would be happy to do so in a few weeks if you would like additional testing. |
|
@ziuchkovski There was an accidentally incompatibility introduced between the 0.6.2 user space utilities and the 0.6.1 kernel modules related to |
|
@behlendorf Thanks, that was the problem. Userspace/module incompatibility didn't occur to me. Reloading the modules after the 0.6.2 upgrade fixed the issue. |
|
@ziuchkovski Glad to hear it. Unfortunately, you should expect similar breakage when updating to 0.6.3 it looks like it's going to be unavoidable. Sorry. |
|
@behlendorf Good to know, thanks. Now I know what to look for. |
zfs send -Rvvvvvvvv khorne-m4-zp00@20130403-backup | pigz -c > /mnt/backups/${date}-backup-zfssend.gz
send from @ to khorne-m4-zp00@20130403-backup estimated size is 15K
send from @ to khorne-m4-zp00/ROOT@20130403-backup estimated size is 15K
send from @ to khorne-m4-zp00/ROOT/rootfs@20130403-backup estimated size is 70.9M
send from @ to khorne-m4-zp00/ROOT/rootfs/USR@20130403-backup estimated size is 5.30G
send from @ to khorne-m4-zp00/ROOT/rootfs/OPT@20130403-backup estimated size is 595M
send from @ to khorne-m4-zp00/ROOT/rootfs/VAR@20130403-backup estimated size is 1.04G
send from @ to khorne-m4-zp00/GENTOO@20130403-backup estimated size is 14.5K
send from @ to khorne-m4-zp00/GENTOO/portage@20130403-backup estimated size is 340M
send from @ to khorne-m4-zp00/GENTOO/distfiles@20130403-backup estimated size is 4.67G
send from @ to khorne-m4-zp00/GENTOO/packages@20130403-backup estimated size is 2.69G
send from @ to khorne-m4-zp00/HOME@20130403-backup estimated size is 15.5K
send from @ to khorne-m4-zp00/HOME/root@20130403-backup estimated size is 30.3M
send from @ to khorne-m4-zp00/HOME/mthode@20130403-backup estimated size is 95.1G
total estimated size is 110G
TIME SENT SNAPSHOT
internal error: Invalid argument
here, have a core dump
http://dev.gentoo.org/~prometheanfire/crash-logs/zfs-core-0.6.1-20130403.dump
The text was updated successfully, but these errors were encountered: