Low iSCSI performance, target threads & cores underutilized

[karlatec]

Sighting report

When running iSCSI Target with multiple cores, all connections seem to be processed
using the same poll_group with the rest being idle. This results in low performance.

Using SPDK v25.05 for both Target and Initiator (665a105).
Fio bdev plugin used as initiator.

Steps to Reproduce

0. HW Configuration:

2 bare metal systems connected back to back with 100GbE Link.
Used NVMe drive is Intel Optane P4800X with about 550k IOPS max performance for used 4KiB Random Read workload.

1. Target configuration:

spdk/build/bin/spdk_tgt --wait-for-rpc -L nvmf_tcp -L nvmf -m [0-19]
spdk/scripts/rpc.py iscsi_set_options -c 8
spdk/scripts/rpc.py sock_impl_set_options -i posix --enable-zerocopy-send-server
spdk/scripts/rpc.py iobuf_set_options --small-pool-count=32767 --large-pool-count=16383
spdk/scripts/rpc.py framework_start_init

spdk/scripts/rpc.py bdev_nvme_attach_controller -b NVMe1 -t PCIe -a 0000:85:00.0
spdk/scripts/rpc.py iscsi_create_portal_group 1 192.168.100.100:3262
spdk/scripts/rpc.py iscsi_create_initiator_group 2 ANY ANY
spdk/scripts/rpc.py iscsi_create_target_node iscsi1 spdk_iscsi1 'NVMe1n1:0' 1:2 256 -d

2. Initiator configuration:

fio_bdev_iscsi.conf 
{
  "subsystems": [
    {
      "subsystem": "bdev",
      "config": [
        {
          "method": "bdev_iscsi_create",
          "params": {
            "name": "iSCSI1",
            "initiator_iqn": "iqn.2016-06.io.spdk:disk1/0",
            "url": "iscsi://192.168.100.100:3262/iqn.2016-06.io.spdk:iscsi1/0"
          }
        }
      ]
    },
    {
      "subsystem": "iscsi",
      "config": [
        {
          "method": "iscsi_set_options",
          "params": {
            "node_base": "iqn.2016-06.io.spdk",
            "max_queue_depth": 256
          }
        }
      ]
    }
  ]
}

fio.conf
[workload]
group_reporting=1
per_job_logs=0
ioengine=/root/spdk/build/fio/spdk_bdev
spdk_json_conf=/root/manual_tests/fio_bdev_iscsi.conf
allrandrepeat=1
direct=1
exitall=1
exitall_on_error=1
thread=1
blocksize=4096
filename=iSCSI1
numjobs=4
iodepth=128
readwrite=randread

Current results

1. IOPS max out at about 100k for above configuration and workload.
2. Inspecting spdk_top on Target side shows only 1 poll_group / core being Busy despite numjobs > 1 on fio initiator side (more than 1 connection). Screenshots attached:
   Cores:

Threads:

Pollers:

3. Running the same workload using Kernel iSCSI initiator (iscsiadm discover & login) and fio results in the same performance and threads/pollers behavior.

4. Running the same workload with equivalent SPDK NVMe-oF TCP Target & Initiator configuration results in maxed out performance (~550k IOPS). spdk_top on Target side shows expected threads & pollers behavior.

Context (Environment including OS version, SPDK version, etc.)

SPDK 665a105

[jimharris]

Hi @karlatec,

This is all working as expected.

On the target side, the SCSI layer is not multi-threaded. It means that all connections for one target node must be processed by one thread only.

On the initiator side, all I/O are funneled to one CPU core for processing. We only create one connection per iSCSI target node from the initiator side. Not all iSCSI targets support MC/S (multiple connections per session). It should work with SPDK iSCSI target but I don't think it's anything being actively tested.

Marking this as an enhancement request.

-Jim

[karlatec]

Oh really! So many years with SPDK and I didn't know :) In my defense, I've never done anything perf-related with iSCSI.
Thanks @jimharris !

[jimharris]

Oh really! So many years with SPDK and I didn't know :) In my defense, I've never done anything perf-related with iSCSI. Thanks @jimharris !

I think it's been a long time since anyone has done anything perf-related with iSCSI. :-)