Upgrading / Updating ZoL from 0.6.2 to 0.6.3 fails #2400
Comments
|
I ran into the same issue and tried a few things. ReinstallI first tried but that produced the same results. After rebooting, I noticed a boot message about the zfs module not loading. This might help in debugging this issue. DowngradeI then tried to downgrade to 0.6.2 but that got nowhere as well. So can't downgrade since dkms-2.2.0.3-14.zfs1.el6.noarch.rpm has been pulled. So we are left where we upgraded and it doesn't work and there's no way to revert back to a working version. |
|
This exact same thing happened to me today. Guys be careful when doing upgrade, make backup and make sure its valid!! |
|
The data isn't lost; just inaccessible until this issue is resolved. Not a production box but just testing. I'm sure I can find and download the zfs/spl 0.6.2 and appropriate dkms rpms, manually install them, and get things working again. The yum downgrade approach is much more preferable. |
|
It seems that earlier ZoL versions kernel modules built for the current kernel stay and dkms don't wan't to update them. You can restore a fully working ZoL 0.6.3 (on RHEL 6 / CentOS 6) by running the following: |
|
Works fine, restored my zpool. |
|
Thank you very much deajan!!! You solved my upgrading problem. |
|
Similar experience here on three machines, slightly complicated by a kernel update immediately prior to the zfs update. It resulted in a weird mix of the old / existing "0.6.2+patches" .ko's and new "0.6.3" .ko's. [Spl modules were 0.6.3, zfs modules were 0.6.2+patches. Even though the rpm's were all 0.6.3.] All fine until next reboot, when the .ko version mismatch caused log spew about missing symbols, blah, blah and it of course failed to start zfs. The installation procedure was "yum update -y" (with existing zfs modules and kernel version 2.6.32-431.5.1.el6.x86_64 installed) followed by reboot. Fair bit of dkms spammage resulted. It was rebuilding 0.6.2+patches module versions for the new kernel (2.6.32-431.17.1.el6.x86_64). Does dkms activity get logged somewhere?? After the reboot, ran the "yum localinstall zfs-release-1.4...." and "yum install zfs" per the website. More dkms spammage as it allegedly built new module versions. It left two of the older version "debuginfo" rpm's installed. Removed those, all showed current 0.6.3 rpm's. Everything fine until next reboot when it went up the spout due to the ver mismatch. FIX: Manually removed the zfs & spl rpm's, rm -rf all zfs & spl-related modules / links from /lib/modules/{kernel-ver}/weak-updates plus any remnant module versions in /lib/modules/{kernel-ver}/extra/[spl | zfs] on two of the servers. Then performed a normal installation per the zfsonlinux.org website instructions. That worked as expected. End result was a system with consistent module versions working. =) Still have a third server that has the inconsistent kernel modules on-disk. It's running a multi-day zfs send out to tape. I can grab any directory structure, package, log file info from that if it's useful. Also planning to update our two primary servers this coming weekend. I can record any dkms errors, etc. on one of the two if that would prove useful. Just let me know. |
|
deejan, thank you for posting this fix! I need help translating it into ubuntu folder hierarchy/commands. Can you help me? |
|
Well i'm not really into ubuntu, but here's a more or less accurate translation that should work as long as the built kernel modules are named the same way they are on RHEL. |
|
Just had a kernel update today on one of my machines. |
|
Hi all. Would anyone have a full set of steps to upgrade ZoL on a CentOS 6 box, bearing in mind the problems deajan has found and worked around? Now we know about this, how would one start the upgrade, without having to install and then re-install? Also, can anyone edit the title of this issue to fix the typo 'Uprading', so others may find this more easily? cheers Andy |
|
Upgrading is fairly easy. I did the following on my daily use machine and had no issues. The first two lines/commands come from deajan's post and just delete any existing zfs related kernel modules for the currently running kernel. The last line is the standard yum update. Cheers, |
|
Hi Scott. Many thanks, was just about to post that I tried that same order of things myself, and it worked fine for me (too). It does take an age though, on a VirtualBox VM anyway. Nearly an hour for the 'yum update', and that was after removing two older kernels first, which took a long time in itself to finish (dracut running for ages, why??). Andy |
|
Andy, One trick I found on accident was installing pigz (parallel implementation of gzip) which sped up the compression portion of dracut quite a bit since it will utilize all your cores instead of just one. In my experience, no configuration required as dracut just found pigz and started using it instead of gzip. I just re-ran dracut on my daily use machine and it took less then a minute. I hope that helps some. |
deajan
changed the title
|
@ahmgithubahm As you pointed out ZoL modules get built for any installed kernel. Also, you might set installonly_limit=2 in /etc/yum.conf to make yum only keep the latest two kernels instead of keeping 5, for future rebuilds. Btw, thanks for pointing out the "Uprading" typo. @srlefevre Your idea with pigz is great. I just tested before and after installing pigz and got a speed increase of 39% for dracut. |
|
Removing older kernels is not a solution, just a workaround. DKMS is supposed to handle this properly. This is either a problem with DKMS, or how zfs is set to use it. It needs to be figured out which, and either fixed here, or submitted as an issue to dkms if it's found to be dkms at fault. I ran into this as well on every box I manage. It's easy to work around with an after upgrade script that manally runs the dkms clean and fix sequence, but this should "just work". |
|
Hi skylis. Deajan's suggestion for removing older kernels wasn't really to address the main problem handled under this github issue, but rather, just to speed up the ZFS upgrade process, since having fewer older kernels installed means less compilation has to be done. Right, deajan, or have I got it all wrong? :-) I entirely agree, the 'yum upgrade zfs' should just work. Andy |
|
@ahmgithubahm Andy, you just got it :) |
|
You can see the issue in the first post; 0.6.3 is built, but then it is confused when running the 'module version sanity check'and subsequenty not actually updated in /lib/modules/*/extras. I have had success using dkms directly to remove zfs and then spl modules, then yum upgrading, and then using dkms directly to add spl and then zfs back in. Uninstalling the zfs* and spl* packages first, and then installing 0.6.3 would likely have the same effect. |
|
@eborisch Uninstalling and reinstalling via yum / rpm won't have the same effect as the kernel modules stay. |
|
@deajan thanks for pointing that out. Should have stopped while I was ahead. ;) |
|
I was able to reproduce this as well. Time permitting I'll do a little investigating. I completely agree that |
|
If you ever end up on this page and your running The issue occurred because a simple
Source: http://bernaerts.dyndns.org/linux/75-debian/312-debian-wheezy-zfsonlinux-zpool-lost |
|
I'm running Ubuntu 14.04 LTS and this fixed the issue as well: |
Thank you, i that solved my problem as well! |
Solved my issue as well. Thanks for this. |
|
That did not fix my issue. In fact, using "apt-get update; apt-get dist-upgrade" followed by a reboot is how I ended up in this mess in the first place. I ended up with two kernels on my system (new one from the upgrade, and the previous one) and ZFS using the old modules with the new kernel. |
|
I have experienced the same issue as Jeff on Ubuntu Server 14.04. The reported ZFS Ubuntu "fix" above is actually what caused the failure of ZFS on my system. |
|
What worked for me (@neutreno) is purging all of the ZFS packages and all old kernels (so only one version of kernel package was on the system), then reinstalling ZoL. Some of those steps may be more extreme than necessary, but it did get my system back to a working state. I believe I may have also had to import my pool, but honestly don't remember if that was necessary or just something I tried along the way. |
|
Thanks @jefferai ! I managed to fix it shortly after my last post. For others in the same boat, you may find my notes below useful in getting ZFS back up and running on Ubuntu 14.04 after upgrading to ZFS 0.6.3: |
|
@neutreno I tried your steps and it worked, but after reboot the pool could not be mounted. So same problem again. |
|
Hi @frieslog, glad my notes could be of use. Try |
|
I experienced the same issue on upgrade from 0.6.3 -> 0.6.4 on CentOS 7 while upgrading to kernel 3.10.0-229.1.2.el7.x86_64. |
|
Same -> 0.6.5 on CentOS 6 while upgrading kernel at the same time as ZFS. |
Hello,
I tried to update ZoL 0.6.2 to 0.6.3 on a production server with the following specs:
CentOS 6.5 x64, 8GB RAM, 4xSAS HDD from which 2x1To ZFS mirror on IBM x3100M4.
Something went wrong with building the new modules.
Here's the transcript of what i've done so far.
First, i removed elder kernels so i won't have to wait building unnecessary old kernel modules:
[root@backupmaster ~]# yum remove kernel-2.6.32-431.3.1.el6.x86_64 kernel-devel-2.6.32-431.3.1.el6.x86_64 kernel-2.6.32-431.11.2.el6.x86_64 kernel-devel-2.6.32-431.11.2.el6.x86_64 kernel-2.6.32-431.5.1.el6.x86_64 kernel-devel-2.6.32-431.5.1.el6.x86_64
Then, i just tried to update my system (including zfs):
From what i read, it seems that the update process thinks that the kernel modules are already up to date (so it keeps ZoL 0.6.2 modules ?).
Here's some additionnal info (i didn't find a way to get the modules version yet):
Also, my system says ZoL 0.6.3 is installed:
I think i could indeed resolve my problem by removing all zfs and spl packages, and reinstalling them, but i've left my system in this state right now to be able to understand what happened, and maybe find a better solution which might help further install processes.
Any idea of what could have gone wrong ? Something to do with the new dkms pacakge which ain't from ZoL ? I heard dkms merged patches from ZoL. Has that happened yet ?
The text was updated successfully, but these errors were encountered: