[vioscsi] - Reset to device \Device\Raidport1\ was issued - system unresponsive.

[casper]

I'm running vioscsi on a Windows 2012 R2 Hetzner CPX11 Cloud instance (AMD EPYC).

There is a very strange issue at random intervals (hour, to days, and sometimes weeks, but usually with heavy/heavier disk activity), I get a "Reset to device \Device\Raidport1\ was issued", Event ID 129 in the event log.

When this happens whatever process was responsible for triggering this event freezes, and there is no way to kill that process or get rid of it through task manager or any other method that I tried. Even rebooting does not help, as the system refuses to reboot because of this stuck process when this issue hits.

The only way to get the VM working properly again is a hard power cycle.

I've tried all the tips and tricks I could find on the net in order to resolve this issue. Fiddling with the Windows Advanced Power Settings, change AHCI settings, PCI Express Link State Power management, even changing some registry values related to power management. Disabled all sleep from everything. Disabled HD drive caching in device manager. Nothing seems to help.

I've also copied my VM image to another server, to make sure it was not caused by a particular VM instance. Did not help either.

Does anyone know what could be causing this? Is there some way I could debug it further, or any tips on how to get around it?

Thanks.

[vrozenfe]

@casper
Event 129 means IO timeout. Usually happens if SCSI Target doesn't return response after certont amount of time (30 sec by default IIRC). What is your storage backend? Is is a local disk/file of SAN/iSCSI target?

Best,
Vadim.

[casper]

The disks are local NVMe SSD drives with RAID. That is what Hetzner says at the VM checkout page.

"Servers with local disks keep all data on a local RAID mirror on the host system. They are optimized for high I/O performance and low latency and are especially suited for applications which require fast access to disks with low latency, such as databases."

I also tried moving the VM to a different physical server, to make sure it was not a local server problem, but I got the same issue still.

[vrozenfe]

you can try adjust TimeoutValue
https://docs.microsoft.com/en-us/windows-hardware/drivers/storage/registry-entries-for-scsi-miniport-drivers

or IoTimeoutValue
https://docs.microsoft.com/en-us/windows-hardware/drivers/storage/registry-entries-for-storport-miniport-drivers

in the Windows Registry.

I would also suggests to confirm with your provider, that they actually provide a dedicated physical storage for you VM.
If you want, I can take a look at your VM system event log file. Just ping me at vrozenfe_at_redhat_dot_com

Regards,
Vadim.

[casper]

Awesome. Thanks a lot @vrozenfe . I will try to adjust the timeouts and get back to you. I just had another freeze today. Going to try to change those settings now.

[casper]

I had it freeze again even with the new timeout settings. I sent you an email with the event log. Pinging you here in case the email goes to your spam folder.

[casper]

Update for anyone else running into the same problem:
Switching from AMD EPYC to the Intel-based servers on Hetzner seems to fix the issue. Now running for 3 weeks without a freeze. They cost a bit more, but it beats constantly having to reboot.

[YanVugenfirer]

Update for anyone else running into the same problem:
Switching from AMD EPYC to the Intel-based servers on Hetzner seems to fix the issue. Now running for 3 weeks without a freeze. They cost a bit more, but it beats constantly having to reboot.

Closing based on this comment.

[vrozenfe]

Hopefully, the problem has been fixed for both vioscsi and viostor with #684 and #683 commits respectively.

Vadim.

[ylluminate]

Still experiencing this kind of problem. This followed an upgrade from the 0.1.204-1 drivers (which worked without issue) to the 0.1.217-2 drivers.

(Host: Proxmox 7.2)

Windows Logs → System:

Source: vioscsi
Event ID: 129
Level: Warning
Message: Reset to device, \Device\RaidPort2, was issued.

This results in the virtual machine guest being entirely unresponsive even to direct console attachment and having to force stop/start the Windows guest.

Two drivers potentially in question that are provoking the freeze / problem:

[ylluminate]

Any reason this hasn't received any direct attention so far? It's really creating some trouble for a lot of folks who aren't familiar with this situation.

[vrozenfe]

There is a good chance that the problem is not in the vioscsi driver at all.
Event 129 is just an indication that a request was not completed in time and the system is
asking miniport driver to cancel andcomplete all outstanding requests.
If such problem happens occasionally, then #684 should be able to handle this situation. Hoverer, if the problem is persistent and happens at the storage subsystem level (https://bugzilla.redhat.com/show_bug.cgi?id=1973317 as an example) then vioscsi driver has nothing to do with that.
We went through the very intensive testing of #684 and #683 patches, emulating the storage subsystem errors and delays
on different setups. As for now a storage subsystem malfunction seems to be the only reason for getting events 129 and153