ZFS intermittently locks up during the tar / untar sycle #2752
Comments
|
I guess I posted the wrong question but in hope anyone actually read it here is and additional information And then So it looks like in 60 seconds drive was reset by kernel Still I am not sure is it a zfs bug or there something else in play. |
|
Looks like building from git makes not much of a difference. |
|
@alitvak69 based on the stack you've provided this sounds like an issue with your hardware. They show that zfs is waiting for a physical IO to the disk to either complete / fail / timeout. This is all managed by the lower level block layer in Linux, the device driver, and the actual hardware. Until one of these three things occurs there's not much zfs can do aside from wait. There is a proposed patch in #2387 which would allow us to timeout more quickly but that would just result in the offending drive being marked as failed. This may not be what we you depending on the situation. |
behlendorf
added
the
Type: Performance
|
you have an option of adding an L2ARC device ? even on a mirrored zpool on WD30EFRX (2x) git is really really slow when working on the kernel for me after having added the l2arc it's rather fast & comparable to e.g. ext4, btrfs |
|
Thank you for your response. I ran a long test on all of the disks using On Mon, Oct 6, 2014 at 12:48 PM, kpande notifications@github.com wrote:
|
|
@alitvak69 it's been noted by others that ZFS is capable of driving the hardware more aggressively that other filesystems. This can cause issues with borderline hardware. |
|
I don't have space for L2ARC device. I wanted to add two SSDs but both On Mon, Oct 6, 2014 at 1:20 PM, Sasha Litvak alexander.v.litvak@gmail.com
|
|
I went through series of the sessions with Dell but it it doesn't seem to be a faulty hardware issue. If you happened to go through thread below you will see an explanation of what this patch does. It seem that similar thing happens with Linux |
|
@alitvak69 That's entirely possible. Due to various hardware quirks the SCSI subsystem contains many of these kind of workarounds. Since the fix for this would need to happen below the ZFS layer my suggestion would be to try a newer kernel with an updated driver and see if it works better. |
|
I have the same issue on my hardware. It's looks like The problem reproduced by I'm running In my case I have 8 disks, no SMART errors, no zpool errors or any other problems (except kernel oops in zfs module). @alitvak69, did you solve you issue? How? Thank you. ADDITIONAL DETAILS. Before I tried zfs this server run ext3/4 on MegaRAID virtual drive (with RAID6), but on outdated Debian and without any problems with hw freezes. |
|
|
|
I have strange behaviour now:
Is it bug in Is somebody know how to produce read operations using dd as it do ZoL? |
|
Excellent. in logs, |
|
I had the same issue. Zpool didn't import at all. Even 'zdb -e' did stick on pread64s randomly. |
|
@thepax, did you disable ReadAhead using |
|
Sure: |
|
Looks like NORA (NoReadAhaed) helped me: Everything is OK with reads now! |
|
@thepax, thank you! |
|
I have not made NORA changes. I went through several changes including On Sun, Dec 7, 2014 at 1:05 PM, umask notifications@github.com wrote:
|
|
I have read about ZFS on MegaRAID here https://calomel.org/zfs_raid_speed_capacity.html ('How do you get ZFS to work through the LSI MegaRAID controller ?'). Next I noticed that scrub speed too slow (10-12MB/sec; it's still so low - changes didn't helped) on my zfs partition and decided to do some "tuning" (enable read-ahead and disk cached access). Probably @alitvak69, which scrub speed do you have on your zfs/zpool? |
|
Well, it looks like pigeon superstition. In order to experiment with the issue I've turned ReadAhead back and the issue doesn't return. So now I believe that any MegaRAID reconfiguration fixes it. Probably Anyway, this definately not a ZFS issue. |
|
@thepax, keep us informed about results of your experiments. Thank you 👍 I agree that problem not in ZFS (it's somewhere in megaraid firmware of kernel module/driver). Nonetheless |
|
My original issue was not a scrub speed although it is slow regardless of readahead settings on my 12 TB RAID. I had it during tar | pv | untar sessions. |
|
@umask The issue with ZFS is that it uses many IO threads to work with hard drives. My current LD settings (the first two are SSDs): Scrub speed was about 260MB/s Everything works fine. After disabling ReadAhead and enabling it back I can't reproduce the issue. |
|
@thepax, as I know Adaptive ReadAhead is deprecated feature in MegaRAID. Just FYI. I suspect that 260MB\s scrub speed on 22 disks is very slow. |
|
Are you sure that after switching ReadAhead to 'off' and turn it back to 'on' again and 'power off'/'power on' server its controller will not fall to problem state? |
|
Where does this issue stand? Is it something which has been clearly resolved as not a ZFS issue? |
|
@behlendorf Yes, problem not related with ZFS. |
|
Not a ZFS issue IMO. Perhaps should be closed. On Fri, Dec 19, 2014 at 1:22 AM, umask notifications@github.com wrote:
|
|
OK, thanks. Closed. |
and others
I have recently upgraded to ZOL release 0.6.3 on SL 6.5. We also added a new Dell PV MD1200 with 12 x 2 TB SAS drives connected to Dell Perc H800 LSI Megaraid based controller.
I created 12 RAID0 disks and and then assigned to a raidz2 in a single pool u3
I originally had another pool u2 with a separate Dell H700 controller and similar disks but in the server chassis.
I tried various kernels: 2.6.32-431.29.2, 3.10.55, 3.16.3 and the issue is still the same.
On some runs of tar / untar copy between volumes:
(example)
tar cf - /u2/logs/old -P | pv -s $(du -sb /u2/logs/old | awk '{print $1}') | tar xf - -C 3
I see that throughput drops to 0 bytes and stays there for extended period of time, more then 30 seconds,
At that time dstat shows some of the block devices on u3 busy 100% and others have 0% utilization.
zpool iostat -v shows 0 read or write activity on u3. On u2 which is older volume I see some activity but it is not related to the current operation. There is no load increase and when i ran strace it would show both tar cf and tar xf are waiting.
I cannot kill the operation unless I kill all underlying shells (no reaction to abort otherwise)
When I run dd against any block device of zpool u3 (example below), operation resumes immediately.
I see no disk errors, controller log or system logs show nothing consistently happening during this lockups. I ran a long smart self-test against all of the disks on MD1200 and they showed no errors (disks are healthy)
At this point I am not sure how to diagnose the issue. Can any of you help?
Thanks,
The text was updated successfully, but these errors were encountered: