nginx -s reload can't shutdown "worker process"

[lulin2008]

QAT.L.4.17.0-00002
QAT_Engine-v0.6.11
openssl-1.1.1k

nginx.conf

ssl_engine {
use_engine qatengine;
default_algorithms ALL;
qat_engine {
qat_offload_mode async;
qat_notify_mode poll;
qat_poll_mode heuristic;
qat_sw_fallback on;
}
}

error:
When nginx work cpu is 100%, a large number of close-waits are generated, and nginx -s reload cannot shutting down the old nginx process.

debug:

pstack 363897
#0 0x00007fd4b2a796f0 in __read_nocancel () from /lib64/libpthread.so.0
#1 0x00007fd4b0aad8b5 in read (__nbytes=8, __buf=0x55c24bc9c7f8, __fd=) at /usr/include/bits/unistd.h:44
#2 qat_pause_job (job=, jobStatus=jobStatus@entry=2) at qat_events.c:278
#3 0x00007fd4b0ab465b in qat_chained_ciphers_do_cipher (ctx=0x55c24b51fc40, out=0x55c24b6b5638 "\233'Q\346\033\b\303F\367\323\323\301\377͎a\254\034\003u\033\066Ƥ:㐘泸7\351w}%\362\064\371J\216J\222\227\241\202\030\263\032\070+>\260\037\313\327\r\222\361\366\224\263\300\251Ș؀s\031D\344\221a\235ʝpt\321&\351{\254\304\373'j\241\346\374\370rIM\257ޖ\353@\242\021lO\217\375\222B\330\371\005I\354UWL\017U\274Ӹ\276Bq)l_nP\b\341\375ti|\b\b\313d\305\004\365?\177\020\324\251B.9\211r\231\322'\031\326h!\037\274\003ij\001^S\321\064ZlQ\022y/\256\213B\261\216\236\"\346\363\376\240\206KA\257\237\313\325\345/\033\231", <incomplete sequence \311>..., in=0x55c24b6b5638 "\233'Q\346\033\b\303F\367\323\323\301\377͎a\254\034\003u\033\066Ƥ:㐘泸7\351w}%\362\064\371J\216J\222\227\241\202\030\263\032\070+>\260\037\313\327\r\222\361\366\224\263\300\251Ș؀s\031D\344\221a\235ʝpt\321&\351{\254\304\373'j\241\346\374\370rIM\257ޖ\353@\242\021lO\217\375\222B\330\371\005I\354UWL\017U\274Ӹ\276Bq)l_nP\b\341\375ti|\b\b\313d\305\004\365?\177\020\324\251B.9\211r\231\322'\031\326h!\037\274\003ij\001^S\321\064ZlQ\022y/\256\213B\261\216\236\"\346\363\376\240\206KA\257\237\313\325\345/\033\231", <incomplete sequence \311>..., len=4208) at qat_hw_ciphers.c:1483
#4 0x00007fd4b1deaefc in tls1_enc () from /etc/nginx/.openssl/lib/libssl.so.1.1
#5 0x00007fd4b1deb9f6 in ssl3_get_record () from /etc/nginx/.openssl/lib/libssl.so.1.1
#6 0x00007fd4b1de8ec3 in ssl3_read_bytes () from /etc/nginx/.openssl/lib/libssl.so.1.1
#7 0x00007fd4b1df0b7a in ssl3_read () from /etc/nginx/.openssl/lib/libssl.so.1.1
#8 0x00007fd4b197e884 in async_start_func () from /etc/nginx/.openssl/lib/libcrypto.so.1.1
#9 0x00007fd4b155dd40 in ?? () from /lib64/libc.so.6
#10 0x0000000000000000 in ?? ()

pstack 363892
#0 0x00007fd4b2a796f0 in __read_nocancel () from /lib64/libpthread.so.0
#1 0x00007fd4b0aad8b5 in read (__nbytes=8, __buf=0x55c24c81dfc8, __fd=) at /usr/include/bits/unistd.h:44
#2 qat_pause_job (job=, jobStatus=jobStatus@entry=2) at qat_events.c:278
#3 0x00007fd4b0ab465b in qat_chained_ciphers_do_cipher (ctx=0x55c24bf744d0, out=0x55c24b5bdfc8 "$\244D\255\336\066\037\002\346Ƹ/ɂS\032V\254\060\251ش#w\372\362\367\061\241\025ʆ\035W\266\225)m@\331\n/:\020=\237雁a\n\340C!\354\270\367|\232\236\031\304<K\254.\267\246\376v|\026\310aa\005\202Kզ!m\265\344\262F\372\204\244\021=\263^\326.a\210\tD"\254\066M$t?BoT]\254n\356*\216\372\231\247\066\237]jj\304T\235\026q%tmC!\301Ɩ\267\340K?1\302\200\361^q", in=0x55c24b5bdfc8 "$\244D\255\336\066\037\002\346Ƹ/ɂS\032V\254\060\251ش#w\372\362\367\061\241\025ʆ\035W\266\225)m@\331\n/:\020=\237雁a\n\340C!\354\270\367|\232\236\031\304<K\254.\267\246\376v|\026\310aa\005\202Kզ!m\265\344\262F\372\204\244\021=\263^\326.a\210\tD"\254\066M$t?BoT]\254n\356*\216\372\231\247\066\237]jj\304T\235\026q%tmC!\301Ɩ\267\340K?1\302\200\361^q", len=3088) at qat_hw_ciphers.c:1483
#4 0x00007fd4b1deaefc in tls1_enc () from /etc/nginx/.openssl/lib/libssl.so.1.1
#5 0x00007fd4b1deb9f6 in ssl3_get_record () from /etc/nginx/.openssl/lib/libssl.so.1.1
#6 0x00007fd4b1de8ec3 in ssl3_read_bytes () from /etc/nginx/.openssl/lib/libssl.so.1.1
#7 0x00007fd4b1df0b7a in ssl3_read () from /etc/nginx/.openssl/lib/libssl.so.1.1
#8 0x00007fd4b197e884 in async_start_func () from /etc/nginx/.openssl/lib/libcrypto.so.1.1
#9 0x00007fd4b155dd40 in ?? () from /lib64/libc.so.6
#10 0x0000000000000000 in ?? ()

help me!!!

[ShuaiYuan21]

@lulin2008 Could you please also share one of the qat configure files which locate at /etc/c6xx_dev0.conf ?
If possible, please update your qat_driver and qat_engine to the latest version, and try again.
Let us know if this issue still exists after that.

Thanks.

[lulin2008]

@ShuaiYuan21
nginx:
worker_processes 16;

cat /etc/c6xx_dev0.conf |grep -v "#"

[GENERAL]
ServicesEnabled = cy

ServicesProfile = DEFAULT

ConfigVersion = 2

CyNumConcurrentSymRequests = 512
CyNumConcurrentAsymRequests = 64

statsGeneral = 1
statsDh = 1
statsDrbg = 1
statsDsa = 1
statsEcc = 1
statsKeyGen = 1
statsDc = 1
statsLn = 1
statsPrime = 1
statsRsa = 1
statsSym = 1

DcIntermediateBufferSizeInKB = 64

AutoResetOnError = 0

[KERNEL]
NumberCyInstances = 0
NumberDcInstances = 0

[SHIM]
NumberCyInstances = 1
NumberDcInstances = 0
NumProcesses = 32
LimitDevAccess = 1

Cy0Name = "UserCY0"
Cy0IsPolled = 1
Cy0CoreAffinity = 0

[ShuaiYuan21]

@lulin2008 I cannot reproduce the problem. Does this issue appear every time after reload?
It seems that this problem is caused by a thread local variable counting, so please upgrade qat_engine (v0.6.11) to the latest version, which has been fixed this issue.

Thanks.

[lulin2008]

@ShuaiYuan21
When I upgrade to qat_engine (v0.6.11), the issue remains unresolved.When nginx work cpu is 100%, a large number of close-waits are generated, and nginx -s reload can not shutting down the old nginx process(1 or 2,not all).

[ShuaiYuan21]

hi, @lulin2008

1. Are you running nginx in another environment, like Docker or something else?
2. Could you please list the steps to fully reproduce the issue? I would like to confirm whether it is caused by other factors, because just running nginx in system does not have this issue.

[ShuaiYuan21]

No response from user for a long time, close this ticket.