Repository navigation
Replies: 2 comments
|
I haven't seen it, or heard about it before. And we can rule out both DSM 7.4.1 Update 1 and the latest SynoOnlinePack_v2 as they were both released after your 2026-09-20 logged instance. DSM 7.4.1 was released in July so if it changed something there'd be more reports of this issue. |
0 replies
|
Thanks for checking. Agreed, I don't suspect the update either. One detail I should have mentioned: the NVMe pool was only created on 2026-09-16, so all three incidents happened within the first seven weeks of this setup, which fits it being specific to this configuration rather than a DSM change. The irqbalance workaround survived a reboot today (all 16 NVMe IRQs on their hinted CPUs). I'll report back in a couple of months whether the timeouts stay away. |
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
Sharing an analysis in case others see the same pattern. The root cause is a hypothesis and the fix is under observation.
Setup: DS1621+ (Ryzen V1500B, 8 CPUs), DSM 7.4.1-90080 Update 1 (kernel 4.4.302), 2× Kingston KC3000 1 TB (FW EIFK51.2) as a RAID1 storage pool in the internal M.2 slots, enabled with Synology_HDD_db v3.6.139 and Synology_enable_M2_volume v1.1.22. Holds Docker configs and databases.
Symptom: three incidents in seven weeks (2026-09-20, 2026-10-22, 2026-11-05). Each time, both NVMe drives logged
I/O <n> QID <q> timeout, abortingin the same second, on the same queue number (QID 2, 6 and 3). Afterwards camecompleting aborted command with status: 0000. On 10-22 the system recovered by itself. On 09-20 and 11-05 atimeout, reset controllerfollowed 60 s later, writes failed on both legs and md failed the array (on 11-05 the pool crashed).Ruled out: SMART clean on both (critical_warning 0, media_errors 0, 0% used), APST disabled, PCIe ASPM disabled (policy performance), temperatures normal, no scheduled task matching the times.
Observation: the NVMe driver's
affinity_hintmaps qN → CPU N−1, but irqbalance kept migrating the NVMe IRQs. At one point 8 of 16 NVMe vectors were stacked on CPU 5, including q3 of both drives. Two independent drives timing out together on the same queue three times, with the commands completing successfully, looks like lost or unprocessed completion interrupts on the host side rather than a drive problem. DSM'sirqbalance.shalready has special handling for 10GbE and AHCI IRQs (--synoethirq,--synoahciirq), but none for NVMe.Workaround (in testing since 2026-11-06): a systemd drop-in that starts irqbalance with
--banmod=nvme, plus pinning each NVMe IRQ to itsaffinity_hint, re-applied by a boot-up task:# /etc/systemd/system/irqbalance.service.d/nvme.conf [Service] ExecStart= ExecStart=/usr/sbin/irqbalance --banmod=nvmeToo early to call it a fix, since the incidents were 4–6 weeks apart. I'll report back.
Questions: Has anyone else seen simultaneous timeouts on both M.2 drives with the same QID? Since Synology's own NVMe drives use the same driver and irqbalance, I don't think this is specific to third-party drives, but I can't prove it.
All reactions