ZFS + UAS + AMD (ZEN) = I/O Errors #11741

Redsandro · 2021-03-13T14:34:40Z

Redsandro
Mar 13, 2021

I'm using OpenZFS on a UAS DAS and keep running into I/O errors on AMD hardware only. This happens on both ZEN and ZEN2 architecture. I haven't tested ZEN+ or ZEN3. The same DAS works fine thoroughly tested on multiple Intel devices. I discourage people from purchasing AMD hardware if they plan to use OpenZFS on a UAS DAS.

My USB Attached SCSI Device, a 4x 8TB ZFS Direct Attached Storage (Icy Box IB-3740-C31), gets many read/write and checksum errors when attached to an AMD computer. It does not give these errors when attached to Intel computers.

Originally observed on a Gigabyte Aorus X570 I PRO WIFI + AMD 3700x (ZEN2) desktop connected for testing the DAS, reported to r/Amd and AMD email tech support; I got the impression this was related to the 500 series chipset problem and I could safely purchase another AMD mini PC with a non-500-series chipset. The OpenZFS DAS works fine and successfully completes a scrub on Intel hardware. I've tried three machines, among which an Intel Core i7-7600U and an Intel Xeon CPU D-1518.

Below the steps to reproduce this on Ubuntu 20.04 with zfsutils-linux installed. Assuming the USB ZFS pool is attached.

$ zpool status
  pool: ztest
 state: ONLINE
config:

	NAME           STATE     READ WRITE CKSUM
	ztest          ONLINE       0     0     0
	  mirror-0     ONLINE       0     0     0
	    sdc        ONLINE       0     0     0
	    sdb        ONLINE       0     0     0
	  mirror-1     ONLINE       0     0     0
	    sdd        ONLINE       0     0     0
	    sda        ONLINE       0     0     0

errors: No known data errors

$ lsusb -t
/:  Bus 04.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/4p, 10000M
    |__ Port 4: Dev 22, If 0, Class=Hub, Driver=hub/4p, 5000M
        |__ Port 1: Dev 24, If 0, Class=Mass Storage, Driver=uas, 10000M
        |__ Port 2: Dev 25, If 0, Class=Mass Storage, Driver=uas, 10000M
        |__ Port 3: Dev 23, If 0, Class=Mass Storage, Driver=uas, 10000M
        |__ Port 4: Dev 26, If 0, Class=Mass Storage, Driver=uas, 10000M

$ fio --name=random-read  --filename=fio-temp --rw=randwrite --bs=1m --size=50G --numjobs=1 --iodepth=4 --runtime=300 --sync=1

$ zpool status
  pool: ztest
 state: ONLINE
status: One or more devices has experienced an unrecoverable error.  An
	attempt was made to correct the error.  Applications are unaffected.
action: Determine if the device needs to be replaced, and clear the errors
	using 'zpool clear' or replace the device with 'zpool replace'.
   see: http://zfsonlinux.org/msg/ZFS-8000-9P
  scan: resilvered 1,92G in 0 days 00:00:09 with 0 errors on Thu Feb 25 18:18:30 2021
config:

	NAME           STATE     READ WRITE CKSUM
	ztest          ONLINE       0     0     0
	  mirror-0     ONLINE       0     0     0
	    sdc        ONLINE       0     0    36
	    sdb        ONLINE       0     1    36
	  mirror-1     ONLINE       0     0     0
	    sdd        ONLINE       0     1    18
	    sda        ONLINE       0     1    18

errors: No known data errors

Although this is very likely not related to the DAS and cables due to the succesful function on three Intel machines, I have also tested different cables. I have used both the USB type C port and the USB 3.1 port on the back. I have tried flashing the latest bios for this mainboard (33c). I have tested a different DAS (Fantec QB-35U31). The results are the same. Resilvers fine on Intel hardware, breaks on AMD 500 series chipset.

AMD has since announced a fix for 500 series chipset USB problems.

But here is the problem: I'm experiencing the same issue on a MinisForum UM270 with AMD Ryzen 7 2700u (ZEN), purchased for the NAS build. This is a 2017 CPU, so I am assuming a 300/400 series chipset, although I'm unsure where to find exact details about these mini PC's.

Syslog

13:59:57 nas zed: eid=13 class=scrub_start pool_guid=0x2965B2F20651A54F  
14:08:39 nas kernel: sd 4:0:0:0: [sdd] tag#5 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
14:08:39 nas kernel: sd 4:0:0:0: [sdd] tag#5 Sense Key : Illegal Request [current] 
14:08:39 nas kernel: sd 4:0:0:0: [sdd] tag#5 Add. Sense: Invalid command operation code
14:08:39 nas kernel: sd 4:0:0:0: [sdd] tag#5 CDB: Read(16) 88 00 00 00 00 01 72 09 02 c8 00 00 00 48 00 00
14:08:39 nas kernel: blk_update_request: critical target error, dev sdd, sector 6208160456 op 0x0:(READ) flags 0x700 phys_seg 9 prio class 0
14:08:39 nas kernel: zio pool=pool vdev=/dev/sdd1 error=121 type=1 offset=3178577104896 size=36864 flags=40080ca0
14:08:40 nas zed: eid=15 class=io pool_guid=0x2965B2F20651A54F vdev_path=/dev/sdd1 
14:12:44 nas kernel: sd 4:0:0:0: [sdd] tag#4 uas_eh_abort_handler 0 uas-tag 2 inflight: CMD IN 
14:12:44 nas kernel: sd 4:0:0:0: [sdd] tag#4 CDB: Read(16) 88 00 00 00 00 01 72 25 53 a8 00 00 00 48 00 00
14:12:49 nas kernel: xhci_hcd 0000:04:00.3: xHCI host not responding to stop endpoint command.
14:12:49 nas kernel: xhci_hcd 0000:04:00.3: xHCI host controller not responding, assume dead
14:12:49 nas kernel: xhci_hcd 0000:04:00.3: HC died; cleaning up
14:12:49 nas kernel: usb 1-1: USB disconnect, device number 4
14:12:49 nas kernel: usb 1-1.5: USB disconnect, device number 5
14:12:49 nas kernel: sd 4:0:0:0: [sdd] tag#3 uas_eh_abort_handler 0 uas-tag 1 inflight: CMD 
14:12:49 nas kernel: sd 4:0:0:0: [sdd] tag#3 CDB: Read(16) 88 00 00 00 00 01 72 25 52 98 00 00 00 38 00 00
14:12:49 nas kernel: sd 4:0:0:0: [sdd] tag#2 uas_eh_abort_handler 0 uas-tag 5 inflight: CMD 
14:12:49 nas kernel: sd 4:0:0:0: [sdd] tag#2 CDB: Read(16) 88 00 00 00 00 03 a3 80 e6 10 00 00 00 10 00 00
14:12:49 nas kernel: sd 4:0:0:0: [sdd] tag#1 uas_eh_abort_handler 0 uas-tag 4 inflight: CMD 
14:12:49 nas kernel: sd 4:0:0:0: [sdd] tag#1 CDB: Read(16) 88 00 00 00 00 03 a3 80 e4 10 00 00 00 10 00 00
14:12:49 nas kernel: sd 4:0:0:0: [sdd] tag#0 uas_eh_abort_handler 0 uas-tag 3 inflight: CMD 
14:12:49 nas kernel: sd 4:0:0:0: [sdd] tag#0 CDB: Read(16) 88 00 00 00 00 00 00 00 0a 10 00 00 00 10 00 00
14:12:49 nas kernel: usb 2-1: USB disconnect, device number 2
14:12:49 nas kernel: usb 2-1.1: USB disconnect, device number 4
14:12:49 nas kernel: usb 1-3: USB disconnect, device number 2
14:12:49 nas kernel: sd 3:0:0:0: [sdc] Synchronizing SCSI cache
14:12:49 nas kernel: scsi host4: uas_eh_device_reset_handler FAILED to get lock err -19
14:12:49 nas kernel: sd 4:0:0:0: Device offlined - not ready after error recovery
14:12:49 nas kernel: zio pool=pool vdev=/dev/sdd1 error=5 type=1 offset=3032272011264 size=12288 flags=40080ca0
14:12:49 nas kernel: sd 4:0:0:0: rejecting I/O to offline device
14:12:49 nas kernel: zio pool=pool vdev=/dev/sdd1 error=5 type=1 offset=3032271814656 size=36864 flags=40080ca0
14:12:49 nas kernel: zio pool=pool vdev=/dev/sdc1 error=5 type=1 offset=3032271847424 size=8192 flags=1808a0
14:12:49 nas zed: eid=20 class=delay pool_guid=0x2965B2F20651A54F vdev_path=/dev/sdd1 
14:12:49 nas zed[10406]: Missed 1 events
14:12:49 nas zed: eid=124 class=io pool_guid=0x2965B2F20651A54F  
14:12:49 nas kernel: sd 3:0:0:0: [sdc] Synchronize Cache(10) failed: Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK
14:12:49 nas kernel: xhci_hcd 0000:04:00.3: WARN Can't disable streams for endpoint 0x85, streams are being disabled already
14:12:49 nas kernel: usb 2-1.2: USB disconnect, device number 5
14:12:49 nas kernel: sd 4:0:0:0: [sdd] Synchronizing SCSI cache
14:12:49 nas kernel: sd 4:0:0:0: [sdd] Synchronize Cache(10) failed: Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK
14:12:49 nas kernel: xhci_hcd 0000:04:00.3: WARN Can't disable streams for endpoint 0x85, streams are being disabled already
14:12:49 nas kernel: usb 2-1.3: USB disconnect, device number 3
14:12:49 nas kernel: sd 2:0:0:0: [sdb] Synchronizing SCSI cache
14:12:49 nas kernel: zio pool=pool vdev=/dev/sdb1 error=5 type=1 offset=3032272310272 size=8192 flags=1808a0
14:12:50 nas kernel: sd 2:0:0:0: [sdb] Synchronize Cache(10) failed: Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK
14:12:50 nas kernel: xhci_hcd 0000:04:00.3: WARN Can't disable streams for endpoint 0x85, streams are being disabled already
14:12:50 nas kernel: usb 2-1.4: USB disconnect, device number 6
14:12:50 nas kernel: sd 5:0:0:0: [sde] Synchronizing SCSI cache
14:12:50 nas kernel: zio pool=pool vdev=/dev/sde1 error=5 type=1 offset=3032266256384 size=8192 flags=1808a0
14:12:50 nas kernel: WARNING: Pool 'pool' has encountered an uncorrectable I/O failure and has been suspended.
14:12:50 nas kernel: sd 5:0:0:0: [sde] Synchronize Cache(10) failed: Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK
14:12:50 nas kernel: xhci_hcd 0000:04:00.3: WARN Can't disable streams for endpoint 0x85, streams are being disabled already
14:13:11 nas zed: eid=5182 class=io_failure pool_guid=0x2965B2F20651A54F  
14:13:39 nas smartd[3360793]: Device: /dev/sda [SAT], SMART Usage Attribute: 194 Temperature_Celsius changed from 43 to 44
14:13:39 nas smartd[3360793]: Device: /dev/sdb [SAT], removed ATA device: No such device
14:13:39 nas smartd[3360793]: Device: /dev/sdc [SAT], removed ATA device: No such device
14:13:39 nas smartd[3360793]: Device: /dev/sdd [SAT], removed ATA device: No such device
14:13:39 nas smartd[3360793]: Device: /dev/sde [SAT], removed ATA device: No such device

At some point the UAS simply disappears. Again, I can attach the DAS to my Intel laptop or desktop and it resilvers fine. No errors.

I'm uncertain how to proceed. Given that all USB devices, external storage and single disk non-ZFS UAS devices seem to work fine on both AMD computers tested, the only combination that structurally causes these issues for me is AMD + UAS + OpenZFS.

Can anyone else reproduce this specifically on AMD with UAS hardware? Did anyone experience this? Are there some known factors that I can disable on a system (Ubuntu 20.04) or BIOS (American Megatrends 5.13 1/21/2021) level?

Until this is resolved, I discourage anyone from buying AMD hardware released on ZEN platform or later when planning to use OpenZFS on a DAS.

hugovdb1965 · 2021-05-31T07:27:19Z

hugovdb1965
May 31, 2021

I have the same issue with an icy box DAS (same model as described by the author) using a Zen3 CPU (Ryzen 9 5950X). The icy box is connected with USB3 gen 2 (10Gbit/s), connecting it with USB3 gen 1 (5Gbit/s) does not solve the issue. Using an Intel based workstation or Intel based laptop at either USB speed has no issues. The problem only occurs when applying heavy write I/O to the DAS, for example moving several GB of files to a filesystem on the DAS. After the errors occur I can resilver on the AMD platform. Large read workloads have no issue. The DAS contains 4x6TB WD red disks. I run on Ubuntu 20.04.2, kernel 5.4.0-73-generic, but had the problem with earlier kernels as well.

8 replies

hugovdb1965 Jun 3, 2021

I had a free x4 slot. The card arrived today. When I hook up the DAS to the separate controller everything works fine. When I put it back on an MB USB port (both 10Gbit and 5 Gbit) the problem comes back. I'd say this is final proof something is wrong with the chipset or its firmware. Although I do have a good workaround now, it did increase the price of my MB by 30% and it requires my only x4 slot. It works but AMD should really fix the X570 USB controllers/firmware instead. The good news is of course I don't need to buy another DAS.

hugovdb1965 Jun 6, 2021

Tried the same on Windows 10 and it gives the same picture. ASMedia USB 3.1 controller works fine, AMD USB 3.1 controller gives a ton of errors. That excludes OS, drivers, DAS, disks and cables and leaves only AMD hardware and firmware as possible sources of the problem.

Redsandro Jun 7, 2021
Author

Thank you for sharing this valuable information @hugovdb1965.

I've made one more post at /r/Amd here: https://www.reddit.com/r/Amd/comments/nuawhz/amd_uasuasp_confirmed_broken_on_zen3/
Although it will probably be closed as "tech support outside of megathead", the same megathread were my reports were previously left unaddressed.

I will leave it at that, because we (at the office) and I (at home) replaced AMD hardware with Intel, at least were UAS devices were concerned.

TheHooly Jun 7, 2021

UPDATE

We're all using Linux. AMD doesn't like Linux (judging from the inability to claim their free promotional games on Linux). So perhaps using the latest kernel may yield different results. If I recall correctly, the last kernel I tried was 5.8. You should try 5.12.

Offtopic: I was able to claim the Far Cry 5 code that came with my Ryzen 7 5800X (sold as part of an upgrade kit with the CPU, a B550 Tomahawk and a rudimentary CPU cooler). On their website instead of receiving a .exe I received a .sh file that worked perfectly fine.

Redsandro Jun 7, 2021
Author

On their website instead of receiving a .exe I received a .sh file

This must be new! When I contacted customer support they explicitly told me I needed Windows to claim the game.

Pleytos · 2021-06-07T13:16:00Z

Pleytos
Jun 7, 2021

I thought I might be helpful to share my hardware knowledge: All Ryzen processors (desktop and laptop) are System on a Chip and can work without a chipset. Laptops and mini pc's very often don't have a chipset and depend on the USB- and other controllers integrated in the processor. So your 2700u probably doesn't have a chipset.

0 replies

ppawel · 2022-12-01T19:59:13Z

ppawel
Dec 1, 2022

I can confirm exactly the same problem with two AMD-based ThinkPad laptops - T14s Gen 1 (Zen 2) and T16 Gen 1 (Zen 3+).

This is with the USB enclosure Icy Box IB-3780-C31, with 8 drives (2 mirrors with 2 disks each, 1 raidz2 with 4 disks).

ZFS 2.1.6 on Arch Linux (kernel 6.0.10).

As a workaround, I am forcing UAS(P) to be disabled with the following modprobe configuration:

options usb_storage quirks=174c:1356:u

With this, I get the following logs when connecting the enclosure:

Dec 01 20:35:34 t14s kernel: usb 8-1.1.1: new SuperSpeed Plus Gen 2x1 USB device number 26 using xhci_hcd
Dec 01 20:35:34 t14s kernel: usb 8-1.1.1: New USB device found, idVendor=174c, idProduct=1356, bcdDevice= 1.00
Dec 01 20:35:34 t14s kernel: usb 8-1.1.1: New USB device strings: Mfr=2, Product=3, SerialNumber=1
Dec 01 20:35:34 t14s kernel: usb 8-1.1.1: Product: ASM1352R-PM
Dec 01 20:35:34 t14s kernel: usb 8-1.1.1: Manufacturer: ASMT
Dec 01 20:35:34 t14s kernel: usb 8-1.1.1: SerialNumber: 20200319H8200001
Dec 01 20:35:34 t14s kernel: usb 8-1.1.1: UAS is ignored for this device, using usb-storage instead
Dec 01 20:35:34 t14s kernel: usb-storage 8-1.1.1:1.0: USB Mass Storage device detected
Dec 01 20:35:34 t14s kernel: usb-storage 8-1.1.1:1.0: Quirks match for vid 174c pid 1356: 800000
Dec 01 20:35:34 t14s kernel: scsi host3: usb-storage 8-1.1.1:1.0

Note the "UAS is ignored for this device" line.

With those quirks enabled, drives appear to work fine and ZFS is completely functional. The only problem is that I get the impression that read performance is somewhat affected, at least when I'm moving something between the datasets that are on those disks, I am not able to max out the SSDs.

The enclosure itself is supposed to be USB 3.1 Gen 2 (10GBps) so I think it should be able to handle the traffic.

When I copy some stuff to the local laptop drive and then to the SSDs in the enclosure, the speed is "correct" - so it's definitely "something" with the USB throughput.

However, overall this workaround has been satisfactory for my use case. Without it, the symptoms are exactly the same as the original poster mentioned, and while ZFS is fighting the errors bravely, it is simply unusable.

4 replies

Redsandro Dec 1, 2022
Author

I am forcing UAS(P) to be disabled with the following modprobe configuration:

options usb_storage quirks=174c:1356:u

[While] read performance is somewhat affected, (...) this workaround has been satisfactory for my use case. Without it, the symptoms are exactly the same as the original poster mentioned, and while ZFS is fighting the errors bravely, it is simply unusable.

Good find @ppawel! I wish I knew about that quirk option sooner, so I'm quoting it here for clarity.

For me it's too late. I was so intensely frustrated with AMD, their ignorant customer support which I have contacted multiple times, and their toxic community of victim blaming fans telling me I'm an idiot for buying an UAS device in the first place, that I have sold/replaced my AMD devices with Intel devices. I'm all for supporting the underdog, but not when they are so ignorant. All they do is cater to gamers and support some video cards, but they don't really gaf about Linux or big data analysts and supporting their preferred data storage peripherals.

ppawel Dec 8, 2022

Just a small note that I have now reconnected the aforementioned USB enclosure to my new Intel-based home server system and all the issues have disappeared - no quirks needed, everything works smoothly and performance has improved as well.

mkobel Jan 9, 2024

I have the same issue with an Intel-based server system.

HPE ProLiant MicroServer G10+ V2 (Intel Xeon E-2314)
USB controller: Intel Corporation Tiger Lake-H USB 3.2 Gen 2x1 xHCI Host Controller (rev 11)
Icy Box IB-3780-C31
Debian GNU/Linux Kernel 6.1.69-1
ZFS raidz2

Disks always disappeared pairwise during high load situations (copy a large amount of files) with errors like critical target error, dev sdj, sector 4148124960 op 0x1:(WRITE) flags 0x700 phys_seg 119 prio class 2 / CDB: Write(16) 8a 00 00 00 00 00 f7 3f 5d 20 00 00 03 b8 00 00 / Sense: Invalid field in cdb / Sense Key : Illegal Request [current] / Sense: Medium not present / Device: /dev/sdl [SAT], is in SLEEP mode

As workaround I had to shut down the server, turn off and on the Icy Box and restart the server. (Just restarting the server did not help).

Interestingly some controllers where using uas, some usb-storage:

    |__ Port 4: Dev 2, If 0, Class=Hub, Driver=hub/5p, 480M
        |__ Port 5: Dev 13, If 0, Class=, Driver=, 480M
        |__ Port 3: Dev 9, If 0, Class=Mass Storage, Driver=usb-storage, 480M
        |__ Port 1: Dev 5, If 0, Class=Mass Storage, Driver=uas, 480M
        |__ Port 4: Dev 11, If 0, Class=Mass Storage, Driver=usb-storage, 480M
        |__ Port 2: Dev 7, If 0, Class=Mass Storage, Driver=uas, 480M
    |__ Port 5: Dev 3, If 0, Class=Hub, Driver=hub/5p, 480M
        |__ Port 1: Dev 6, If 0, Class=Mass Storage, Driver=uas, 480M
        |__ Port 2: Dev 8, If 0, Class=Mass Storage, Driver=uas, 480M
        |__ Port 3: Dev 10, If 0, Class=Mass Storage, Driver=uas, 480M
        |__ Port 4: Dev 12, If 0, Class=Mass Storage, Driver=usb-storage, 480M
        |__ Port 5: Dev 14, If 0, Class=, Driver=, 480M

Since setting options usb_storage quirks=174c:1356:u the issues did not occur any more.

iSOcH May 20, 2024

I am using the 4-drive variant and am seeing the problem on multiple machines (Intel and AMD). The only one where it seems stable is a intel 13700k on a z690 motherboard.

Currently trying the quirks (for my device it seems to be options usb_storage quirks=2109:0715:u). In a zfs scrub, I see a reduced speed of close to 600mbyte/s compared to almost 800mbyte/s seen with UAS.

Pectojin · 2023-03-08T10:04:57Z

Pectojin
Mar 8, 2023

Sadly, with my 5600G in a B550 mobo with AGESA updated to 1.2.0.7, I still can't use my DAS neither using UAS or regular usb-storage mode. It "works" in in both modes until I start writing data and I get tons of write failures, data corruption, and the whole connection hanging for minutes at a time. The 3.1 support seems abysmal. The only way I got my DAS to work consistently was by using a USB 2 cable with 480M throughput, so I'm currently waiting on a PCIE card with a chipset that hopefully actually works..

So, for others coming across this: your mileage may wary, but nothing has worked for me.

2 replies

francoism90 May 30, 2023

@Pectojin I've just seen this topic, however my problems went away after disabling USB-suspend (both with a service like TLP and also the port itself). See https://gist.github.com/francoism90/7dd103249c5ffe3dc36be44a8b291ef1 when using Thunderbolt over USB.

It has been a while, and I'm not using it anymore, but I got the same issues on my Intel NUC.

Pectojin May 31, 2023

I got my PCIE card (an icybox one) and it's been working without issue for ~2 months. It's a bit disappointing, but spending ~40 euro on an external card has been well worth it for me.

Michaelcode2 · 2023-09-29T07:03:52Z

Michaelcode2
Sep 29, 2023

I had SATA device error, (IO error) or switching to read-only state. Error appears on the disks on the heavy load process, backup for example. iommu configuration solved my issue.
Here is the action sequence:

grep "sdb" /var/log/syslog | tail -n100

Sep 28 10:40:55 kernel: [404556.968434] sd 1:0:0:0: [sdb] tag#26 CDB: Read(16) 88 00 00 00 00 01 40 00 08 00 00 00 01 00 00 00
Sep 28 10:40:55 kernel: [404556.968435] blk_update_request: I/O error, dev sdb, sector 5368711168 op 0x0:(READ) flags 0x0 phys_seg 32 prio class 0
Sep 28 10:40:55 kernel: [404556.973909] sd 1:0:0:0: [sdb] tag#27 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK cmd_age=0s
Sep 28 10:40:55 kernel: [404556.973910] sd 1:0:0:0: [sdb] tag#27 CDB: Read(16) 88 00 00 00 00 00 00 00 00 00 00 00 01 00 00 00
Sep 28 10:40:55 kernel: [404556.973911] blk_update_request: I/O error, dev sdb, sector 0 op 0x0:(READ) flags 0x0 phys_seg 32 prio class 0
Sep 28 10:40:59 smartd[975]: Device: /dev/sdb [SAT], state written to /var/lib/smartmontools/smartd.HGST_HUS726060ALE610-K8GENJGD.ata.state

In ProxMox VM backup process log

INFO:  52% (981.8 GiB of 1.8 TiB) in 2h 47m 56s, read: 93.6 MiB/s, write: 92.9 MiB/s
INFO:  52% (985.1 GiB of 1.8 TiB) in 2h 53m 51s, read: 9.8 MiB/s, write: 9.7 MiB/s
ERROR: job failed with err -5 - Input/output error
INFO: aborting backup job
INFO: resuming VM again

For AMD Epic series processor I solved the issue with adding boot parameter

"iommu=pt" and "amd_iommu=on"

Check if iommu is supported

sudo dmesg | grep -i iommu

sudo dmesg | grep -e DMAR -e IOMMU

Set IOMMU

vim /etc/default/grub

add parameters to the string GRUB_CMDLINE_LINUX_DEFAULT

GRUB_CMDLINE_LINUX_DEFAULT="amd_iommu=on iommu=pt consoleblank=0"

reboot the server

after rebook check for boot errors
sudo dmesg | grep "error"

I have got a bit slower IO to the SATA disks, but more stable work with SATA devices. Now SATA devices do not dissappear or switches to read-only state.

1 reply

Redsandro Sep 29, 2023
Author

Thanks for sharing your findings.

I would have loved to test this with multi-disk UAS hardware on AMD, but I have replaced all AMD hardware with Intel machines to solve my problem. If anyone can confirm this also fixes the high load USB Attached SCSI Device problem, it will be beneficial for the community, and frankly for AMD as well.

With AMD tech support being very similar to /dev/null in helping us solve this issue, I'd need to see peer testimony before choosing AMD again.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ZFS + UAS + AMD (ZEN) = I/O Errors #11741

{{title}}

Replies: 5 comments 15 replies

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

Select a reply

ZFS + UAS + AMD (ZEN) = I/O Errors #11741

Syslog

Replies: 5 comments · 15 replies

Redsandro Jun 7, 2021 Author

Redsandro Jun 7, 2021 Author

Redsandro Dec 1, 2022 Author

Redsandro Sep 29, 2023 Author

Replies: 5 comments 15 replies

Redsandro Jun 7, 2021
Author

Redsandro Jun 7, 2021
Author

Redsandro Dec 1, 2022
Author

Redsandro Sep 29, 2023
Author