assetion failed

[minaevmike]

i have failed assetion in nghttp2_session.c on line 6576, in gdb i see that datamax = 16384 and nghttp2_buf_avail(buf) = 16384.
And sometimes i have failed assertion here:

(gdb) 
#0  0x00007ffff5c77e37 in __GI_raise (sig=sig@entry=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:56
#1  0x00007ffff5c79528 in __GI_abort () at abort.c:89
#2  0x00007ffff5c70ce6 in __assert_fail_base (fmt=0x7ffff5dc0c08 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=assertion@entry=0x7ffff71090ee "!inside_callback_", 
    file=file@entry=0x7ffff71090d1 "asio_server_http2_handler.cc", line=line@entry=391, function=function@entry=0x7ffff7109280 "void nghttp2::asio_http2::server::http2_handler::enter_callback()") at assert.c:92
#3  0x00007ffff5c70d92 in __GI___assert_fail (assertion=0x7ffff71090ee "!inside_callback_", file=0x7ffff71090d1 "asio_server_http2_handler.cc", line=391, 
    function=0x7ffff7109280 "void nghttp2::asio_http2::server::http2_handler::enter_callback()") at assert.c:101
#4  0x00007ffff70e6aa1 in nghttp2::asio_http2::server::http2_handler::enter_callback() () from /usr/lib/x86_64-linux-gnu/libnghttp2_asio.so.1
#5  0x00007ffff70e0155 in int nghttp2::asio_http2::server::http2_handler::on_write<65536ul>(boost::array<unsigned char, 65536ul>&, unsigned long&) () from /usr/lib/x86_64-linux-gnu/libnghttp2_asio.so.1
#6  0x00007ffff70e1498 in nghttp2::asio_http2::server::connection<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_service<boost::asio::ip::tcp> > >::do_write() ()
   from /usr/lib/x86_64-linux-gnu/libnghttp2_asio.so.1
#7  0x00007ffff70e18ac in boost::asio::detail::write_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_service<boost::asio::ip::tcp> >, boost::asio::mutable_buffers_1, boost::asio::detail::transfer_all_t, nghttp2::asio_http2::server::connection<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_service<boost::asio::ip::tcp> > >::do_write()::{lambda(boost::system::error_code const&, unsigned long)#1}>::operator()(boost::system::error_code const&, unsigned long, int) () from /usr/lib/x86_64-linux-gnu/libnghttp2_asio.so.1
#8  0x00007ffff70e19c8 in boost::asio::detail::reactive_socket_send_op<boost::asio::mutable_buffers_1, boost::asio::detail::write_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_service<boost::asio::ip::tcp> >, boost::asio::mutable_buffers_1, boost::asio::detail::transfer_all_t, nghttp2::asio_http2::server::connection<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_service<boost::asio::ip::tcp> > >::do_write()::{lambda(boost::system::error_code const&, unsigned long)#1}> >::do_complete(boost::asio::detail::task_io_service*, boost::asio::detail::task_io_service_operation*, boost::system::error_code const&, unsigned long) () from /usr/lib/x86_64-linux-gnu/libnghttp2_asio.so.1
#9  0x00007ffff70d192d in boost::asio::detail::task_io_service::do_run_one(boost::asio::detail::scoped_lock<boost::asio::detail::posix_mutex>&, boost::asio::detail::task_io_service_thread_info&, boost::system::error_code const&) () from /usr/lib/x86_64-linux-gnu/libnghttp2_asio.so.1
#10 0x00007ffff70d2a61 in boost::asio::detail::task_io_service::run(boost::system::error_code&) () from /usr/lib/x86_64-linux-gnu/libnghttp2_asio.so.1
#11 0x00007ffff70d2e66 in boost::asio::io_service::run() () from /usr/lib/x86_64-linux-gnu/libnghttp2_asio.so.1
#12 0x00007ffff70d057e in std::_Function_handler<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> (), std::__future_base::_Task_setter<std::unique_ptr<std::__future_base::_Result<unsigned long>, std::__future_base::_Result_base::_Deleter>, unsigned long> >::_M_invoke(std::_Any_data const&) () from /usr/lib/x86_64-linux-gnu/libnghttp2_asio.so.1
#13 0x00007ffff70d0cb2 in std::__future_base::_State_baseV2::_M_do_set(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>&, bool&) ()
   from /usr/lib/x86_64-linux-gnu/libnghttp2_asio.so.1
#14 0x00007ffff673e730 in pthread_once () at ../nptl/sysdeps/unix/sysv/linux/x86_64/pthread_once.S:103
#15 0x00007ffff70d48a3 in std::__future_base::_Async_state_impl<std::_Bind_simple<std::_Mem_fn<unsigned long (boost::asio::io_service::*)()> (std::shared_ptr<boost::asio::io_service>)>, unsigned long>::_Async_state_impl(std::_Bind_simple<std::_Mem_fn<unsigned long (boost::asio::io_service::*)()> (std::shared_ptr<boost::asio::io_service>)>&&)::{lambda()#1}::operator()() const ()
   from /usr/lib/x86_64-linux-gnu/libnghttp2_asio.so.1
#16 0x00007ffff62d6dd0 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#17 0x00007ffff67390a5 in start_thread (arg=0x7fffef7fe700) at pthread_create.c:309
#18 0x00007ffff5d3acfd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

so what i must do to fix this?
I repoduce this if i send a make a lot(over 100 in for loop) submits in one http2 session.

[tatsuhiro-t]

Thank you for reporting this. As for assertion error in nghttp2_session.c, did you use data_length_callback to extend frame size? Also are you using v1.9.2?

[tatsuhiro-t]

So, line 6576 is

assert(nghttp2_buf_avail(buf) >= datamax);

and datamax = 16384 and nghttp2_buf_avail(buf) = 16384, then assert should be fine.
I'm not sure why it failed there.

[tatsuhiro-t]

Is there any chance that your program called nghttp2 API from multiple thread at the same time?

[minaevmike]

No i don't, i only use asio wrapper. I update my nghhtp2 to master and first assertion dissapears.
For second: yes, there is a big change that i called it from multiply threads. I need some mutex queue to call it?

[tatsuhiro-t]

I think first assertion error also comes with the multiple calls from different thread to the same nghttp2 session object. The call to the same nghttp2 session must be guarded by mutex, otherwise there is plenty of chance for undefined behaviour.

[minaevmike]

I add mutex, and it doesn't help, as i can undestand the problem is bigger. I can add mutex only to calls like response.write_head and response.end. But guard appears in on_write function (asio_server_http2_hander.h) and on_write calls in boost::asio::async_write callback( response.end), after end returns. So i need a callback to write_head and end functions.

[tatsuhiro-t]

You are write, they happens inside libnghttp2_asio library.

I have not reproduced this issue. I used examples/asio-sv. Could you share the simplest server side code to reproduce this issue?

[minaevmike]

i can't reproduce this issue with simple server, but it repoduced on my working server.
While testing i found one interesting thing. I my current implementation of my server, i have my own io_service for my usage. And i call write_head and end methods from my threads. But when i make code like this

res.io_service().post([&res, packed = std::forward<std::string>(packed), code](){
            res.write_head(code);
            res.end(packed); 
        });

Assertion's dissapears, do you have any ideas why this happens?

[tatsuhiro-t]

res.io_service() returns the io_service the request/response is handled. And we assume that all API calls must be done in the same thread where underlying io_service is running. So res.io_service.post will invoke callbacks on the same thread, which follows our assumption.
One thing I have to say is that "res.io_service()" also must be called from the same thread res.io_service is running. If you are using res object from another thread, it will cause undefined behaviour.

[minaevmike]

But why call of res.io_service() can cause UB?

[tatsuhiro-t]

All objects associated to one nghttp2 server object must be used in the same thread that the underlying io_service object is running. If res object is used in the another thread and res.io_service() is called, at the same time, the res object might be destroyed. That would be UB.
I'm not sure this applies to your case really, since I don't really know how you use libnghttp2_asio. But if you are using objects from multiple threads at the same time, we may get UB with the reason described above.

[minaevmike]

if i'll try to get ioservice from server, i can't know which one from the set of io_services is exactly handling this request.
I have another question, if i'll make some kind of refactoring, would you accept it? I want to change the way of starting io_services.

[tatsuhiro-t]

We will accept PR if it is generally useful. I saw similar concerns from other users regarding io_service, so probably we have some issues there.

[tatsuhiro-t]

Closed since asio library has been removed.