perf: INTERNAL DEVICE ERROR (00/06) nvme_set_num_queues failed!

[hlitz]

$ sudo ./perf -s 4096 -q 64 -w randrw -M 0 -c 1 -t 60
...
Created task_pool
Initializing NVMe Controllers
SET FEATURES (09) sqid:0 cid:15 nsid:0 cdw10:00000007 cdw11:ffffffff
INTERNAL DEVICE ERROR (00/06) sqid:0 cid:15 cdw0:0 sqhd:0002 p:1 m:0 dnr:0
nvme_set_num_queues failed!
nvme_attach failed for controller at pci bdf 4:0:0

$ sudo ./identify
...
EAL: TSC frequency is ~2299998 KHz
EAL: Master lcore 0 is ready (tid=feab4940;cpuset=[0])
SET FEATURES (09) sqid:0 cid:15 nsid:0 cdw10:00000007 cdw11:ffffffff
INTERNAL DEVICE ERROR (00/06) sqid:0 cid:15 cdw0:0 sqhd:0002 p:1 m:0 dnr:0
nvme_set_num_queues failed!
failed to attach to NVMe controller at PCI BDF 4:0:0

$ sudo ./nvme-cli/nvme error-log /dev/nvme0
NVMe Status:INTERNAL(6)

[ghost]

SET FEATURES (09) sqid:0 cid:15 nsid:0 cdw10:00000007 cdw11:ffffffff

This is a Set Features command with feature = 7, Number of Queues, which is normal during startup.

However, the value of CDW11 here means we are requesting 65536 submission and completion queues, which is invalid; the maximum allowed number of queues is 65535, which translates to 65534 since this is a 0-based value, and that would be encoded as 0xfffefffe. Even more confusingly, the SPDK driver requests only 1024 queues by default, which means CDW11 should be 0x3ff03ff, unless the app overrides the default num_io_queues.

Is this an unmodified, up-to-date SPDK library + perf example?

What NVMe device(s) are attached?

There is a possibly bug here which I've just noticed: if num_io_queues is overriden to 0, the nvme_ctrlr_set_num_queues() function will subtract 1 and get the noted 0xffffffff value; however, this shouldn't be happening unless something has been modified (perf does not override the default num_io_queues).

[hlitz]

just did a git pull on the Intel SPDK repo and ran an unmodified perf test. This is my output:

EAL: Master lcore 0 is ready (tid=68e22940;cpuset=[0])
Initializing NVMe Controllers
Attaching to 0000:04:00.00
SET FEATURES (09) sqid:0 cid:15 nsid:0 cdw10:00000007 cdw11:03ff03ff
INTERNAL DEVICE ERROR (00/06) sqid:0 cid:15 cdw0:0 sqhd:0002 p:1 m:0 dnr:0
nvme_set_num_queues failed!

cdw11 is no longer ffffffffff but it’s still much larger than expected and the same error occurs.

From print statements with this Intel perf test, I can tell that ctrlr->ops.num_io_queues is 1024 (which sounds like what it should be according to the thread on github). It’s the spdk_nvme_qpair_process_completions() function, which checks the admin queue, that fails. Perhaps something is wrong with the admin queue?

[ghost]

The new CDW11 value looks correct - it is actually two copies of 1023 in the high and low words (number of I/O completion queues and number of I/O submission queues - these should always be the same since SPDK allocates submission and completion queues in pairs).

However, I am not sure what could be causing the command to fail; if the device supports fewer than the requested number of queues, it is supposed to choose its maximum value and report it back via CDW0.

The failure is noticed in the admin queue checking function, since that is where we poll for completions on admin commands, but if we are actually receiving completions here, I doubt anything is wrong with the admin queue.

Can you verify that this behavior still occurs after e.g. power cycling the host machine and/or NVMe device (especially if the kernel NVMe driver is also not working)?

[hlitz]

So I looked into this issue again. Rebooted the machine but the error stays the same. Is there any way to hard reset the device? Or is there a hardware defect?

[benlwalker]

We believe this was a hardware error. Please let us know if there is still a problem, but closing the issue for now.

[hradl]

Hi - I have a similar problem - was the problem resolved?

[benlwalker]

We believe the SSD in question failed, but we weren't able to confirm. You're welcome to open a new issue with a log of the problem you are seeing and we'll take a look.

[hradl]

I have 4 cards in one server - 3 of them went offline at the sametime and returns the same error
I have not been able to regain control these cards but i think its unlikely that 3 cards go bad, no hw reboot / power off / on has resolved the problem. Can you help me getting the necessary logs/debug info?

[hradl]

firmware version is 8EV101F0
from kernel log
nvme 020d:01:00.0: rtas_msi: allocated virq 37
nvme 020d:01:00.0: Could not set queue count (6)
nvme nvme1: IO queues not created

[benlwalker]

Reboot and then dump the output from dmesg (and filter it down to the messages with 'nvme' in them maybe). But if the Linux kernel driver isn't loading (which is what the above log indicates) then it definitely isn't an SPDK problem.

Can you switch which PCIe slots the cards are in, maybe?

[hradl]

I found other articles having similar problem at intel support. I don't think its a SPDK problem
thanks anyway
https://communities.intel.com/thread/109279