L2ARC failed to rebuild after reboot #11787
Comments
aaronjwood
added
Status: Triage Needed
No, that is not the case. Curious though what caused this. If it happens again could you post a "cat /proc/spl/kstat/zfs/arcstats" after it fails to rebuild the L2ARC contents? That might give us a clue. You can also look in |
Yeah, will do. Haven't been able to reproduce it yet. I don't see anything relevant in zpool history unfortunately. |
|
Still haven't reproduced this yet but saw something that I thought I should share: Note the |
|
This means the header of the L2ARC device (used for rebuilding its
contents) was corrupted for some reason.
…On Mon, Apr 26, 2021, 5:42 AM Aaron Wood ***@***.***> wrote:
Still haven't reproduced this yet but saw something that I thought I
should share:
...
l2_rebuild_success 4 0
l2_rebuild_unsupported 4 0
l2_rebuild_io_errors 4 0
l2_rebuild_dh_errors 4 1
l2_rebuild_cksum_lb_errors 4 0
l2_rebuild_lowmem 4 0
l2_rebuild_size 4 0
l2_rebuild_asize 4 0
l2_rebuild_bufs 4 0
l2_rebuild_bufs_precached 4 0
l2_rebuild_log_blks 4 0
...
Note the l2_rebuild_dh_errors field. I'm not clear on what this field
means or if it's relevant to the issue I hit.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<#11787 (comment)>, or
unsubscribe
<https://github.com/notifications/unsubscribe-auth/AB2Y2IPKSKYGVFGQSPUBJ63TKTOLJANCNFSM4ZWCG4YA>
.
|
|
This would cause the behavior I saw, right? Is there anything else within ZFS that I can look at that might indicate why this header got corrupted? |
|
I just had this happen to me again last night due to a power outage. Once my server came back I saw my L2 ARC was empty again. FWIW l2_rebuild_dh_errors is now sitting at 3. Anything else I can check here to get more information on why I keep failing to rebuild? |
|
Try a "zdb -lll /dev/cache device" . It seems the header of the L2ARC
device is corrupted and that is why ZFS cannot read it. Only way to restore
normal operation would be removing and re-adding the cache device.
…On Tue, May 4, 2021, 5:10 PM Aaron Wood ***@***.***> wrote:
I just had this happen to me again last night due to a power outage. Once
my server came back I saw my L2 ARC was empty again. FWIW
l2_rebuild_dh_errors is now sitting at 3.
Anything else I can check here to get more information on why I keep
failing to rebuild?
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<#11787 (comment)>, or
unsubscribe
<https://github.com/notifications/unsubscribe-auth/AB2Y2IPHVHIXX66U3SVP7ILTMAE5PANCNFSM4ZWCG4YA>
.
|
I can try removing and adding it back. It's interesting you say that because I had done that (for another reason) after upgrading to ZFS 2.0.0. I'll do it again right now and see if I can reproduce the issue again. |
|
Even after readding the device: Also tried with the ID which is how I actually add it to the pool: |
|
This tells us the whole label of the cache device is corrupted and not only
the header that is required for persistent L2ARC. I have never seen
this before, either in production or in local VM testing.
…On Tue, May 4, 2021, 7:50 PM Aaron Wood ***@***.***> wrote:
zdb -lll /dev/nvme0n1
failed to unpack label 0
failed to unpack label 1
failed to unpack label 2
failed to unpack label 3
I can try removing and adding it back. It's interesting you say that
because I had done that (for another reason) after upgrading to ZFS 2.0.0.
I'll do it again right now and see if I can reproduce the issue again.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<#11787 (comment)>, or
unsubscribe
<https://github.com/notifications/unsubscribe-auth/AB2Y2IMYPRFQMKWTJSXCC73TMAXWFANCNFSM4ZWCG4YA>
.
|
|
Are you sure there is no underlying hardware problem?
You can try zeroing out manually the label and header of the removed device
with "zpool labelclear" and then adding it again.
…On Tue, May 4, 2021, 7:56 PM Georgios Amanakis ***@***.***> wrote:
This tells us the whole label of the cache device is corrupted and not
only the header that is required for persistent L2ARC. I have never seen
this before, either in production or in local VM testing.
On Tue, May 4, 2021, 7:50 PM Aaron Wood ***@***.***> wrote:
> zdb -lll /dev/nvme0n1
> failed to unpack label 0
> failed to unpack label 1
> failed to unpack label 2
> failed to unpack label 3
>
> I can try removing and adding it back. It's interesting you say that
> because I had done that (for another reason) after upgrading to ZFS 2.0.0.
> I'll do it again right now and see if I can reproduce the issue again.
>
> —
> You are receiving this because you commented.
> Reply to this email directly, view it on GitHub
> <#11787 (comment)>, or
> unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AB2Y2IMYPRFQMKWTJSXCC73TMAXWFANCNFSM4ZWCG4YA>
> .
>
|
|
Hah, wow! Guess I am lucky :) Some more info for context: The drive seems to work fine as is, regardless of being used as an L2ARC or not. Here's more data about the drive: ZFS is the one managing the label, partition, etc. of the entire disk right? I'll try your suggestion above and post back in a bit. |
Is ZFS corrupting this? |
|
FWIW this also fails: but this works: It dumps WAY too much info for me to post here. Let me know if you want me to attach it if you think it'd be useful/relevant. |
|
Could you do a |
|
For sure: |
|
So the device is still on its first pass (has not been filled to the end), has written about 31GB out of a total capacity of about 2TB. I suspect that the TRIM code may be interfering with the label (trimming it out), although the device is still on its first pass. |
|
Yeah, it was much more full before the power loss event last night :) Since I've been removing/adding it today it's pretty small right now as you said. I do have TRIM enabled. I noticed that a TRIM is kicked off whenever I add this drive to a pool. Guessing that is intended behavior from ZFS. My conf for reference: Yes to your last question about Proxmox and ZFS. Since I've reported the issue there have been a few patches that came in: |
|
Yes, adding a cache device to a pool with Notably your device reports |
|
Interesting. Let me know if there's any other info you want to see, or if there's anything else you want me to try. I'm not quite sure what to do otherwise. |
|
This is a longshot but there shouldn't be any weirdness in working with the reported (fake) sector size of 512/512, right? |
|
I do not think so, the sector size is handled by core ZFS code, the L2ARC code has barely anything to do with it. |
|
How come we do see the header when doing BTW I tried the process of removing the L2ARC device, labelclear'ing it, and adding it back all while l2arc_trim_ahead was set to 0. Still see the same behavior. |
|
One other thing I noticed: when adding the device to the pool with l2arc_trim_ahead set to 100 I see this until the trimming has completed: Once it's finished it sees the header: Is this expected behavior? The L2ARC device shows as |
|
As far as I remember this is expected behavior. Also the label of a vdev is
present in both partitions on the disk, that is normal.
Could you share all the non default ZFS module parameters you use?
…On Fri, May 21, 2021, 7:40 AM Aaron Wood ***@***.***> wrote:
One other thing I noticed: when adding the device to the pool with
l2arc_trim_ahead set to 100 I see this until the trimming has completed:
zdb -l nvme-HP_SSD_EX950_2TB_HBSE59340600764
------------------------------------
LABEL 0
------------------------------------
version: 5000
state: 4
guid: 2538846175622296943
labels = 0 1 2 3
L2ARC device header not found
Once it's finished it's sees the header:
zdb -l nvme-HP_SSD_EX950_2TB_HBSE59340600764
------------------------------------
LABEL 0
------------------------------------
version: 5000
state: 4
guid: 2538846175622296943
labels = 0 1 2 3
------------------------------------
L2ARC device header
------------------------------------
magic: 6504978260106102853
version: 1
pool_guid: 8691708886196254037
flags: 1
start_lbps[0]: 15741607936
start_lbps[1]: 15607631872
log_blk_ent: 1022
start: 4198400
end: 2000063823872
evict: 4198400
lb_asize_refcount: 2437120
lb_count_refcount: 119
trim_action_time: 1621575040
trim_state: 4
------------------------------------
L2ARC device log blocks
------------------------------------
log_blk_count: 119 with valid cksum
0 with invalid cksum
log_blk_asize: 2437120
Is this expected behavior? The L2ARC device shows as ONLINE from a zpool
status as it is trimming (when the L2ARC header isn't present).
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<#11787 (comment)>, or
unsubscribe
<https://github.com/notifications/unsubscribe-auth/AB2Y2ILRSTTERWUW5YSA45DTOXW5BANCNFSM4ZWCG4YA>
.
|
|
Ok, thanks. Here's what I have: FWIW I've also tried upgrading to 5.11 yesterday (https://forum.proxmox.com/threads/kernel-5-11.86225/#post-378338) but the same behavior persists with all of the different scenarios that I've gone through above. You can assume going forward that I'm now on |
|
I do not think kernel updates affect this. In your case the label is destroyed before the device is completely filled with data (ie l2ad_first=1 seen as flags=1 in the zdb output) which excludes a good chunk of code. Upon importing the pool the following happens:
If the device is marked for trimming or for rebuild, no writes take place until those flags are cleared. |
|
Do you get this problem only after power outage or does it also occur after a normal export of the pool or reboot? |
|
Could you post a |
|
Sure |
|
At this point I am sure L2ARC fails to rebuild because it cannot physically read the device header from the device (ie its not like its reading a header that isn't valid but it actually fails to read from the device). It eludes me though why. |
|
Could it be something silly like the device itself isn't available yet on reboot? Are the reboots proper shutdowns or result of something else? (power failure was mentioned). |
|
I can answer the second part :) Other than the one time where I mentioned it was a power failure, everything else has been from me doing |
|
I was mistaken, actually the IO error returned is 52, EBADE. In OpenZFS in Linux, EBADE is used to declare checksum errors, and that zio_wait() was modified to actually perform a checksum check embedded in the header. So, it's not a failure to physically read from the device, but more likely a checksum error (ZIO_CHECKSUM_LABEL). The question is why. |
|
My assumption is that for some reason the header may be written incompletely when the reboot is happening, leading to corruption in corner cases. I also noticed that I had used a ZIO_PRIORITY_ASYNC_WRITE as a priority in the write command. @aaronjwood do you have the capacity to try the following patch: |
|
Sorry, unfortunately I am not able to deploy a custom build of ZFS on this environment :( |
I am having the exact same error IO 52 on TrueNAS Scale RC.2 after a reboot. |
|
on my system with zfs 2.1.6 after reboot, l2arc shows "0" in alloc/capacity column . before reboot, 1,5gb was allocated there. secondarycache=metadata |
|
I was just hit but this issue: after a reboot, L2ARC was not rebuilt. Before the reboot it was at 870/380 GB (size/compressed), while now it is at 23.5/11.5 GB. What does EDIT: from here it seems that no correct L2 headers were found. This is the first time I experience the issue, which happened at the first reboot after I manually trimmed the pool and then enabled autotrim. My zfs module options are: Reading the previous posts seems to confirm we have some sort of "interference" between trim (manual and/or automatic) and persistent L2ARC. Running ZFS debug log show the following info related to my cache disks: I'm on Rocky Linux 9.1, kernel 5.14.0-162.6.1.el9_1.0.1.x86_64, ZFS 2.1.7-1 Regards. |
|
Was having the same issue in a newly configured pool to which I recently added L2ARC SSD (a couple of days later). The L2ARC was listed in the zpool iostat output, but 0 bytes alloc across reboots. Using Arch Linux, so, I issued a zpool.cache file rebuild again now after adding the L2ARC with:
Then regenerated the init image with mkinitcpio (which adds that cache file to the image? I guess?), and restarted and now there are 26GB allocated in the L2ARC. Arch uses mkinitcpio to load the initrd image as in other systems, or so it goes... so, it will load the zfs module through that and probably in turn will use that zpool.cache to load the ZFS pool. I haven't checked the internals of how it works. It seems to me that upon rebooting, at least in Arch they peek at the cache and load from there, and I didn't had added the L2ARC device to that cache until later. But now it is persistent across reboots (or so it seems after a couple of reboots). Just a couple of cents for somebody stumbling upon this. |
|
I had the same problem. Reboot after crash. L2ARC empty. this was the first reboot after adding it. arcstat shows I just rebuilt the cachefile, based on a comment here. However zdb -C doesn't show the cache devices, and there is no evidence of them in the cache file. Is this normal? zdb -l /dev/sdbx1 shows labels. The log file shows Mar 13 15:29:38 eternal.lcsr.rutgers.edu systemd[1]: Starting Import ZFS pools by cache file... I wonder if there's a timing issue. If so, it might be a good idea to add the cache devices to the cache file, so that import will wait until they are ready. |
|
I've just hit this bug after removing my cache device (a partition on an NVMe drive) and re-adding it.
zpool status shows a cache device: But arc_summary does not: I've tried export/re-importing the pool, I've rebooted several times, and I've secure-wiped the partition and re-created it as a bare partition with no filesystem. I've turned off |
|
If I am not mistaken, I sometime saw |
You're a genius. My arcstat alias with custom columns was erroring-out: Then I fired up a VM and it started working a few seconds later:
|
Recently had to change the L2ARC SSD drive, and after 10 reboots or so, it refused to hold data (iostat alloc = 0). I was using the Also, before it started to work again, arc_summary.py showed something like these lines at zero (now it shows a few megabytes) which had me wondering that L2ARC while being filled, wasn't being properly used?: |
|
I faced the same issue as richardm1 I tried everything that was written here. the cache never survived a reboot. I didn’t see any error before, this one appeared after recreating the pool with ashift=12 |
|
@gamanakis Any ideas on what is failing here? |
|
There seems to be a problem with the ashift of L2ARC not being constant when the pool has an ashift other than 9. issue #14313. This will lead to inability of reading the L2ARC header and rebuilding its contents in ARC. |
|
Excellent find! Glad there is some idea of the root of the problem. The seemingly random loss of the l2arc on reboot has been killer :( |
|
Thanks for looking into this, I can only share that indeed I use |
Wanted to share the same, I use an ashift of 12 in the pool that my cache device lives in. FWIW since experiencing the issue long ago my cache device has changed from an HP EX950 to a set of 3 SATA SSDs (different brands and sizes) to now currently an Intel 670p. No matter the device(s) I always hit the issue in a seemingly random manner. |
|
Thanks gamanakis, I applied #14963 to master, but unfortunately it didn't solve the problem. I don't see any mention of nvme0n1p5 in the log. this is normal? |
|
Your cache device is 1GB and for persistent L2ARC to work it needs to be greater than 1GB. Try again with a slightly bigger partition. |
|
thanks a lot! this solved my issue. |
|
I'm still seeing this issue and have to workaround this with #14963 in v2.2.3 when relying on an implicit ashift value.
(Full command for reference I'm happy to add and remove the cache devices whilst testing, but can't recreate the pool if someone has a potential commit they want me to test. Pool Details: Running zdb gives the true ashift value: ZFS options: (Not sure how to get the cache vdev ashift, I'm happy to edit and add this) |
System information
Describe the problem you're observing
Had a system up and running for months. L2ARC was a few hundred GB. Rebooted the system and found that the L2ARC was empty.
Describe how to reproduce the problem
Not sure about this. I've rebooted in the past and had my L2ARC preserved. I haven't been able to find any logs around this either. If anyone has suggestions on what I can check I'd be happy to post it here. FWIW
/sys/module/zfs/parameters/l2arc_rebuild_enabledis set to 1.One difference from the last reboot I did where it worked: I had a kernel update this time. Is there some special handling that forces the L2ARC to not rebuild when a new kernel version is used or something?
The text was updated successfully, but these errors were encountered: