[NVME-OF TCP] Crash in perf running on more than single core

[sashakot]

perf application crashes in NVME-OF test running more than 1 core.
The crash happens with any block size, read/write operations, any number of cores used in the target.

Details:

Version: "7a39a68"

Command lines:

Target: sudo ./app/nvmf_tgt/nvmf_tgt -c ./nvmf.conf -m 0x3

Client: sudo examples/nvme/perf/perf -q 5 -o 1036288 -w randwrite -t 60 -c 0x1100 -D -r 'trtype:TCP adrfam:IPv4 traddr:1.1.75.1 trsvcid:1023 nqn.2016-06.io.spdk.r-dcs75:rd0'

Backtrace:

#0  0x0000000000420515 in nvme_tcp_qpair_process_send_queue (tqpair=0xe19040) at nvme_tcp.c:447
447                     pdu_length = pdu->hdr.common.plen - pdu->writev_offset;
Missing separate debuginfos, use: debuginfo-install glibc-2.17-222.el7.x86_64 libaio-0.3.109-13.el7.x86_64 libgcc-4.8.5-28.el7_5.1.x86_64 libibverbs-41mlnx1-OFED.4.5.0.1.0.45037.x86_64 libnl3-3.2.28-4.el7.x86_64 librdmacm-41mlnx1-OFED.4.2.0.1.3.45037.x86_64 libuuid-2.23.2-52.el7.x86_64 numactl-libs-2.0.9-7.el7.x86_64 openssl-libs-1.0.2k-12.el7.x86_64 zlib-1.2.7-17.el7.x86_64
(gdb) bt
#0  0x0000000000420515 in nvme_tcp_qpair_process_send_queue (tqpair=0xe19040) at nvme_tcp.c:447
#1  0x000000000042322c in nvme_tcp_qpair_process_completions (qpair=0xe19040, max_completions=0) at nvme_tcp.c:1555
#2  0x000000000041e01a in nvme_transport_qpair_process_completions (qpair=0xe19040, max_completions=0) at nvme_transport.c:224
#3  0x000000000041a78d in spdk_nvme_qpair_process_completions (qpair=0xe19040, max_completions=0) at nvme_qpair.c:413
#4  0x000000000041b586 in spdk_nvme_wait_for_completion_robust_lock (qpair=0xe19040, status=0x7ffe66d56dd0, robust_mutex=0x0) at nvme.c:128
#5  0x000000000041b5fe in spdk_nvme_wait_for_completion (qpair=0xe19040, status=0x7ffe66d56dd0) at nvme.c:142
#6  0x0000000000411542 in nvme_fabric_prop_get_cmd (ctrlr=0xe1ed60, offset=20, size=0 '\000', value=0x7ffe66d56e88) at nvme_fabric.c:97
#7  0x0000000000411664 in nvme_fabric_ctrlr_get_reg_4 (ctrlr=0xe1ed60, offset=20, value=0x7ffe66d56f40) at nvme_fabric.c:130
#8  0x000000000042024c in nvme_tcp_ctrlr_get_reg_4 (ctrlr=0xe1ed60, offset=20, value=0x7ffe66d56f40) at nvme_tcp.c:369
#9  0x000000000041d91f in nvme_transport_ctrlr_get_reg_4 (ctrlr=0xe1ed60, offset=20, value=0x7ffe66d56f40) at nvme_transport.c:139
#10 0x000000000040b846 in nvme_ctrlr_get_cc (ctrlr=0xe1ed60, cc=0x7ffe66d56f40) at nvme_ctrlr.c:49
#11 0x000000000040bd4f in spdk_nvme_ctrlr_alloc_io_qpair (ctrlr=0xe1ed60, user_opts=0x7ffe66d56f80, opts_size=12) at nvme_ctrlr.c:251
#12 0x000000000040666d in init_ns_worker_ctx (ns_ctx=0xe18d20) at perf.c:826
#13 0x0000000000406739 in work_fn (arg=0xe198c0) at perf.c:862
#14 0x00000000004088ff in main (argc=14, argv=0x7ffe66d57148) at perf.c:1709

Environment:

OS: CentOS Linux release 7.5.1804 (Core)

Kernel: 3.10.0-862.el7.x86_64

Target output"

Starting SPDK v19.01-pre / DPDK 18.08.0 initialization...
[ DPDK EAL parameters: nvmf --no-shconf -c 0x3 --base-virtaddr=0x200000000000 --file-prefix=spdk_pid30815 ]
EAL: Detected 24 lcore(s)
EAL: Detected 2 NUMA nodes
EAL: No free hugepages reported in hugepages-1048576kB
EAL: Probing VFIO support...
app.c: 609:spdk_app_start: *NOTICE*: Total cores available: 2
reactor.c: 293:_spdk_reactor_run: *NOTICE*: Reactor started on core 1
reactor.c: 293:_spdk_reactor_run: *NOTICE*: Reactor started on core 0
EAL: PCI device 0000:81:00.0 on NUMA socket 1
EAL:   probe driver: 8086:2700 spdk_nvme
conf.c: 167:spdk_nvmf_parse_nvmf_tgt: *ERROR*: Deprecated options detected for the NVMe-oF target.
The following options are no longer controlled by the target
and should be set in the transport on a per-transport basis:
MaxQueueDepth, MaxQueuesPerSession, InCapsuleDataSize, MaxIOSize, IOUnitSize
This can be accomplished by setting the options through the create_nvmf_transport RPC.
You may also continue to configure these options in the conf file under each transport.tcp.c: 566:spdk_nvmf_tcp_create: *NOTICE*: *** TCP Transport Init ***
tcp.c: 767:spdk_nvmf_tcp_listen: *NOTICE*: *** NVMe/TCP Target Listening on 1.1.75.1 port 1023 *** 

[ak-gh]

Just to give you guys some extra pointer, pdu at this point is NULL and send_queue for the connection is empty.

Regards,
Andrey

[optimistyzy]

https://review.gerrithub.io/435577 2018年11月30日星期五，ak-gh <notifications@github.com> 写道：

…

Just to give you guys some extra pointer, pdu at this point is NULL and send_queue for the connection is empty. Regards, Andrey — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#528 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AD7kgwOD4mgbE-eecvFuuxmwQQ2PeJSUks5u0RXGgaJpZM4Y7dL4> .

[optimistyzy]

@ak-gh Can you tell me why the following not work? Since I tried both, it should work. Thanks

https://review.gerrithub.io/#/c/spdk/spdk/+/436612/

[ak-gh]

Hi guys, sorry for confusion, I just didn't test the first patch, so feel free to choose the one which is best for you. Thanks, Andrey

On Tue, Dec 11, 2018, 07:05 Ziye Yang ***@***.***> wrote: @ak-gh <https://github.com/ak-gh> Can you tell me why the following not work? Since I tried both, it should work. Thanks https://review.gerrithub.io/#/c/spdk/spdk/+/436612/ — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#528 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/Ak3rBxYd8LvU-AJhe_SlFLii5WEdjPXlks5u3y7sgaJpZM4Y7dL4> .

-- Regards, Andrey

[jimharris]

Patches to fix this have been merged and @sashakot confirmed on the mailing list.