when server is trying to respond with multiple threads, server is crashing

[rameshrajagopal]

I want to create a server which routes the request to the worker threads, which will do the work, once the work is done, it responds back to the client. This is not working with the libnghttp2_asio library. it is crashing.

Core trace is as below:

Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `./worker'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 stream_weight_less (lhsx=0x7f6f8401c5f0, rhsx=0x0) at nghttp2_stream.c:39

39 return lhs->cycle < rhs->cycle;
Traceback (most recent call last):
File "/usr/share/gdb/auto-load/usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.19-gdb.py", line 63, in
from libstdcxx.v6.printers import register_libstdcxx_printers
ImportError: No module named 'libstdcxx'
(gdb) bt
#0 stream_weight_less (lhsx=0x7f6f8401c5f0, rhsx=0x0) at nghttp2_stream.c:39
#1 0x00007f6f91952596 in bubble_up (pq=pq@entry=0x7f6f84000dd8, index=1)

at nghttp2_pq.c:58

#2 0x00007f6f919526b5 in nghttp2_pq_push (pq=pq@entry=0x7f6f84000dd8,

item=item@entry=0x7f6f8401c5f0) at nghttp2_pq.c:84

#3 0x00007f6f91954914 in stream_obq_push (dep_stream=0x7f6f84000dc0,

stream=stream@entry=0x7f6f8401c5e0) at nghttp2_stream.c:135

#4 0x00007f6f91954cba in stream_update_dep_on_attach_item (stream=0x7f6f8401c5e0)

at nghttp2_stream.c:379

#5 nghttp2_stream_attach_item (stream=0x7f6f8401c5e0,

item=item@entry=0x7f6f80000a20) at nghttp2_stream.c:410

#6 0x00007f6f919564ac in nghttp2_session_add_item (

session=session@entry=0x7f6f84000da0, item=item@entry=0x7f6f80000a20)
at nghttp2_session.c:764

#7 0x00007f6f9195ce0d in submit_headers_shared (attach_stream=,

stream_user_data=0x0, data_prd=0x7f6f8dc37ca0, nvlen=2, nva_copy=0x7f6f80000990, 
pri_spec=0x7f6f8dc37bc0, stream_id=15, flags=0 '\000', session=0x7f6f84000da0)
at nghttp2_submit.c:98

#8 submit_headers_shared_nva (session=0x7f6f84000da0, flags=,

stream_id=15, pri_spec=pri_spec@entry=0x0, nva=<optimized out>, nvlen=2, 
data_prd=0x7f6f8dc37ca0, stream_user_data=stream_user_data@entry=0x0, 
attach_stream=attach_stream@entry=1 '\001') at nghttp2_submit.c:154

#9 0x00007f6f9195d4bc in nghttp2_submit_response (session=,

stream_id=<optimized out>, nva=<optimized out>, nvlen=<optimized out>, 
data_prd=<optimized out>) at nghttp2_submit.c:424

#10 0x00007f6f92f0f976 in nghttp2::asio_http2::server::http2_handler::start_response

(this=this@entry=0x7f6f840009a8, strm=...) at asio_server_http2_handler.cc:341

#11 0x00007f6f92f1201f in nghttp2::asio_http2::server::response_impl::start_response

(this=0x7f6f8401c770) at asio_server_response_impl.cc:98

#12 0x00007f6f92f1234f in nghttp2::asio_http2::server::response_impl::write_head (

this=<optimized out>, status_code=<optimized out>, h=...)
at asio_server_response_impl.cc:59

#13 0x00007f6f92f11ca6 in nghttp2::asio_http2::server::response::write_head (

this=<optimized out>, status_code=<optimized out>, h=...)
at asio_server_response.cc:42

#14 0x00000000004090f6 in std::__deque_buf_size (__size=18656976)

at /usr/include/c++/4.8/bits/stl_deque.h:90

#15 0x00000000004091f6 in Worker::dowork (this=0x7f6f8dc37da8, res=...)

at worker_queue.cpp:36

#16 0x000000000040f4bf in std::__get_helper<0ul, std::_Mem_fn<void (Worker::)()>, Worker> (__t=...) at /usr/include/c++/4.8/tuple:745
#17 0x000000000040f40f in boost::exception_detail::clone_implboost::exception_detail::error_info_injector<boost::system::system_error >::clone_impl (this=0x11caf40,

x=..., __in_chrg=<optimized out>, __vtt_parm=<optimized out>)
at /usr/include/boost/exception/exception.hpp:439

#18 0x000000000040f263 in std::thread::_Impl<std::_Bind_simple<std::_Mem_fn<void (Worker::)()> (Worker)> >::_M_run() (this=0x0) at /usr/include/c++/4.8/thread:115
#19 0x000000000040f128 in std::_Sp_counted_ptr_inplace<std::thread::_Impl<std::_Bind_s---Type to continue, or q to quit---q

Quit

The code for my example resides @
https://github.com/rameshrajagopal/nghttp2_asio_example

I would like to know, is there any problem with the usage of library in my code or is it a known problem on library side ?

Regards,
Ramesh

[rameshrajagopal]

I tried to enable multithreading via setting num_threads(2) but tried to print how many threads are accessing the handle callback, i don't see two threads are getting created, I have updated my example to include this change, how do i enable multithreading with this library ?

[tatsuhiro-t]

num_threads sets the number of worker threads, and each thread has its own event loop.
The callback function passed to server.handle is executed on one of these threads.
As is the case of most event driven frame work like Node.js, application must not block event loop in callback.
There is 2 problems in your code.
The first problem is the life time of request and response object. It is undefined to access them after the underlying stream is closed. Its closure is notified by setting callback to response.on_close.
Since you are creating your own threads to run application code which is outside of library event loop thread, you have to set this callback, and if it is called, avoid accessing request and response objects.
The second problem was there, but it seems you modified the file, so now it is done.
For the record, request/response objects should be used in the same thread as the event loop thread. To achieve this, you can use strand. The underlying io_service is accessed from response.io_service.
The modified program introduced new problem. That is calling sleep(1). This kind of blocking call must not be called inside event loop callback. We have sample code to achieve delayed response in examples/asio-sv.cc.
If you'd like to do some work which takes very long, then create your own worker thread, and do the task, and commit its result to response object in the same thread the event loop run using strand.

[tatsuhiro-t]

It seems io_service.post do the job without using strand explicitly.
See the example https://gist.github.com/tatsuhiro-t/ba3f7d72d037027ae47b

[rameshrajagopal]

Thanks, I will try it out

[rameshrajagopal]

I have updated my code. This seems to be working. Thanks.