ZFS have silently corrupted the meta data #1848
Comments
|
Increasing vm.min_free_kbytes from 135168 to 4194304 (according to #1666), gives me: |
|
Ok, so spending over a day getting a new kernel (3.12.0) and then latest spl+zfs checkout compiled, I managed to both load the module and also import the pool. However: Now, the question is: Can I safely enable readonly (I can wait until zfs-rogue catches up - I have only one FS which is encrypted)? |
|
Latest {spl,zfs}-crypto in his '0.6.3' branch seems to work. It was merged with ZoL at the 0c28fb4 (Fri Aug 16 15:20:07 2013 if I'm reading the log correctly) commit. After that, there have been a lot of changes to ZoL. I did get ZFS-8000-72 however: I'm currently running the command and hopfully everything will be ok! I'm absolutly terrified though. I have not been able to find a cheap way to do a backup on 15Tb!! |
|
CRAP! That gave me another oops: |
|
Installing the non-crypto versions of spl/zfs again (#1848 (comment)) and used the '-o readonly=on', I get: An 'import -F' seemed to have worked, but now every time I run 'zpool status', I get a glibc memory corruption: And after a reboot, I had to use 'import -F' again (probably because I can't run a scrub on a read-only pool)... |
|
Going back to the spl/zfs crypto I built earlier (which is pre-libzfs_core), and this time using the parameter 'zvol_inhibit_dev=1', everything seems to be just dandy. So far. It's currently resilvering, so hopfully that won't fuck everything up. I didn't need to use the option '-F' when I imported the pool. I had to use '-f' though (probably because it was previously imported with the non-crypto version): I still get the glibc memalloc stack trace on the subsequent 'zpool status', but 'zfs list' seems to work without a problem. Full stack saved if any one is interested... UPDATE: I only seem to get that the first time. I just ran it again (some five-ten minutes after the import), and this time I didn't get the stacktrace... And the pool seems fine: |
|
Soooo.... My question(s) now...
I guest Q2 is more of a 'if someone have the time to investigate or already seen it and knows exactly what the fix is/was'. ZoL HEAD don't seem to exibit this problem (don't know if that's because I have to import using 'readonly=on' and that might mean something) so I can wait with Q2... But Q1 is very serious and a reason really needs to be found (so that I and everyone else can be sure it won't happen again). I'm actually quite weery of mounting the filesystems now - I figure if I don't mount anything, it won't write anything to disk. I really don't even dare to reboot to verify that everything works! |
|
@FransUrbo First off I'm glad you see you managed to save the pool!
The first issue you encountered appear to be a somewhat unexplainable error down in the memory allocator. I've sporadically seen this sort of thing reported before and it almost always is the result of a build gone wrong. Either the spl was not rebuilt with zfs, or perhaps there was only a partial zfs build, or the linking phase went badly for strange reason. It's rare, but it's been reported. However, in all of these case no damage to the pool was ever done. The system would just panic/oops, rebuilding the modules cleanly usually would resolve the issue. I know that's not a very satisfying answer but it's hard to say for certain what happened. Your pool also has the complication of the crypto bits. While I've never looked at that code, I can easily image that the ZoL upstream code might interpret a pool like this to have "corrupt" metadata.
No, that needs an explanation if someone has time to dig in to it. |
|
@behlendorf 'A build gone wrong'... Could you elaborate? I don't use dkms. Don't like it, don't trust it and I think it's a piece of utter crap. And it's a mess because I try to run all my machines on the same kernel so distributing a installable package with the binary modules in it is so much simpler than having a build environment on every single one of them... So I always build the deb packages, and I always test them on a special VM before I install them on the main server (except this time because I can't access my VM's). And I never install a kernel module package unless I have the intention to reboot. So I can say with ... 98% certainty (based on how I work and my personal methodology - I leave 2% off because I HAVE been known to forget :). So the original modules was installed and loaded in the beginning of june. And since I've had about two-three months uptime before I rebooted, that means I have rebooted that install at least once (I also usually reboot about every three-four months to upgrade spl/zfs just because I'd like to keep as close to HEAD as possible to gain features but mostly speed which still isn't great but just fine for my use case). I can't absolutly rule out 'a build gone wrong' (especially since I'm not quite sure what you mean by that), but I'd say it is very unlikely - I know what I'm doing (famos last words, but still! :). I'm fairly certain I would have detected any troubles before hand. Why would vanilla ZoL think that the pool is corrupted just because of the crypto bit? It DOES notice this (every time I tried) and refuses to import it read-write. In addition to refusing to import r/w, claiming it's corrupted, it also complains about I/O errors. I still get that from 'zdb', even though the pool is now imported (without accessible ZVOLs). I just haven't mounted the filesystems (other than test one or two of them and then unmounted them afterwards). |
|
@FransUrbo By a build gone wrong I just meant a build which might not have been carefully done. For example, say ZFS was accidentally built against version A of the SPL, but then version B of the SPL modules was loaded with your new modules. That doesn't sound like the case here but it can lead to all sorts of weird issues.
I'm just speculating since I've never tried this. But it's my understanding that the way ZFS encryption was done not all of the metadata gets encrypted. Enough of it remains in the clear that even vanilla ZoL can determine a little bit about the pool. I could imagine a case where it can see enough to attempt the import but quickly runs in to encrypted blocks which look it like corruption. But, as I said that's just a guess. |
|
|
I'm closing this, because it's now quite obvious that it's the zfs-crypto patch that fucked everything up. |
I had to reboot after about three month of uptime, so I'm not exactly sure what's changed... There where no (!!) indications that there was anything wrong with either the pool or any of the filesystems before the reboot.
But now I can't load the module.
If I first loads the spl module and then manually tries to load the zfs module, I get:
If I remove the cache device, I instead get a lzjb_decompress oops:
Any ideas? This is unfortunately my main storage system, running zfs version '0.6.1-1.20130604' with zfs-crypto and some extra libshare patches that have yet to be merged to 0.6.2+.
It holds all my development VMs so building a new kernel and/or zfs/spl module(s) will be extremely difficult.
I have a new kernel and spl/zfs module built, but that's in my ZFS root which I have not been able to boot because of various problems, mostly grub.
I actually managed to find a '0.6.2-1.20130824' version of both spl and zfs which had the appropriate extra patches. Unfortunately, that exhibits the exact same behavior.
I've found a number of issues regarding 'spl_kmem_cache_alloc', but none match exactly, and those people have managed to at least load the module, but fail on import. My module won't even load. Sometimes, if I move '/sbin/zpool' out of the way, I can load the module but not import it (get the same problem as above), so maybe it's related... ?
The text was updated successfully, but these errors were encountered: