Skip to content
This repository has been archived by the owner on Jul 12, 2023. It is now read-only.

Kurento 6.15.0 crushes #599

Open
3 tasks done
avssav opened this issue Jun 1, 2021 · 8 comments
Open
3 tasks done

Kurento 6.15.0 crushes #599

avssav opened this issue Jun 1, 2021 · 8 comments
Projects
Triage

Comments

@avssav
Copy link

avssav commented Jun 1, 2021 •
edited

Prerequisites

These are MANDATORY, otherwise the issue will be automatically closed.

Issue description

We use KMS 6.15.0 on AWS instance linux Ubuntu 18.04. It starts like:

sudo -u kurento docker run --name kms-6.15.0 --restart always -p 8888:8888 -p 8433:8433 --ulimit nofile="$(( ($(cat /proc/sys/fs/file-max) * 50) / 100 ))" \
    -e GST_DEBUG='2,Kurento*:4,kms*:4,sdp*:4,webrtc*:4,*rtpendpoint:4,rtp*handler:4,rtpsynchronizer:4,agnosticbin:4,KurentoMediaElementImpl:4' \
    -e KMS_EXTERNAL_IPV4='auto' -e KMS_NETWORK_INTERFACES="eth0" -v /home/kurento/kurento-config-6.15.0:/etc/kurento -e KMS_MTU='1100' \
    -e KMS_STUN_IP='ip.of.our.stun' -e KMS_STUN_PORT='3478' -e KMS_TURN_URL='turn:xxxxxxxxx@ip.of.our.stun:3478?transport=udp' --network host \
    -v /home/kurento/tmp/kurento:/tmp -v /srv/records:/srv/records --log-driver journald -d kurento/kurento-media-server:6.15.0;

Why is not KMS 6.16.0? I don't see any urgent updates in the version.

We get some crush of our server really rarely.
Next I attach backtrace of a crash.
I see some malloc problems in backtrace. But I don't see any memory problems on the AWS instance.

How to reproduce?

I don't know

Expected & current behavior

[New LWP 25997]
[New LWP 18582]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/usr/bin/kurento-media-server '.
Program terminated with signal SIGBUS, Bus error.
#0  malloc_consolidate (av=av@entry=0x7f403c000020) at malloc.c:4188
4188    malloc.c: No such file or directory.
[Current thread is 1 (Thread 0x7f42c59fc700 (LWP 252))]
(gdb) bt
#0  0x00007f433d1c14d6 in malloc_consolidate (av=av@entry=0x7f403c000020) at malloc.c:4188
#1  0x00007f433d1c4d0c in _int_malloc (av=av@entry=0x7f403c000020, bytes=bytes@entry=7904) at malloc.c:3457
#2  0x00007f433d1c71d4 in __GI___libc_malloc (bytes=7904) at malloc.c:2920
#3  0x00007f433d7b0e78 in operator new(unsigned long) () at /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#4  0x000056477eee9b21 in boost::asio::detail::completion_handler<boost::asio::detail::rewrapped_handler<boost::asio::detail::binder2<boost::asio::detail::write_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_service<boost::asio::ip::tcp> >, std::vector<boost::asio::const_buffer, std::allocator<boost::asio::const_buffer> >, boost::asio::detail::transfer_all_t, boost::asio::detail::wrapped_handler<boost::asio::io_service::strand, websocketpp::transport::asio::custom_alloc_handler<std::_Bind<std::_Mem_fn<void (websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*)(std::function<void (std::error_code const&)>, boost::system::error_code const&, unsigned long)> (std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >, std::function<void (std::error_code const&)>, std::_Placeholder<1>, std::_Placeholder<2>)> >, boost::asio::detail::is_continuation_if_running> >, boost::system::error_code, unsigned long>, websocketpp::transport::asio::custom_alloc_handler<std::_Bind<std::_Mem_fn<void (websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*)(std::function<void (std::error_code const&)>, boost::system::error_code const&, unsigned long)> (std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >, std::function<void (std::error_code const&)>, std::_Placeholder<1>, std::_Placeholder<2>)> > > >::do_complete(boost::asio::detail::task_io_service*, boost::asio::detail::task_io_service_operation*, boost::system::error_code const&, unsigned long) (this=<optimized out>, __n=494)
    at /usr/include/c++/5/ext/new_allocator.h:104
#5  0x000056477eee9b21 in boost::asio::detail::completion_handler<boost::asio::detail::rewrapped_handler<boost::asio::detail::binder2<boost::asio::detail::write_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_service<boost::asio::ip::tcp> >, std::vector<boost::asio::const_buffer, std::allocator<boost::asio::const_buffer> >, boost::asio::detail::transfer_all_t, boost::asio::detail::wrapped_handler<boost::asio::io_service::strand, websocketpp::transport::asio::custom_alloc_handler<std::_Bind<std::_Mem_fn<void (websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*)(std::function<void (std::error_code const&)>, boost::system::error_code const&, unsigned long)> (std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >, std::function<void (std::error_code const&)>, std::_Placeholder<1>, std::_Placeholder<2>)> >, boost::asio::detail::is_continuation_if_running> >, boost::system::error_code, unsigned long>, websocketpp::transport::asio::custom_alloc_handler<std::_Bind<std::_Mem_fn<void (websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*)(std::function<void (std::error_code const&)>, boost::system::error_code const&, unsigned long)> (std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >, std::function<void (std::error_code const&)>, std::_Placeholder<1>, std::_Placeholder<2>)> > > >::do_complete(boost::asio::detail::task_io_service*, boost::asio::detail::task_io_service_operation*, boost::system::error_code const&, unsigned long) (__a=..., __n=494)
    at /usr/include/c++/5/bits/alloc_traits.h:491
#6  0x000056477eee9b21 in boost::asio::detail::completion_handler<boost::asio::detail::rewrapped_handler<boost::asio::detail::binder2<boost::asio::detail::write_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_service<boost::asio::ip::tcp> >, std::vector<boost::asio::const_buffer, std::allocator<boost::asio::const_buffer> >, boost::asio::detail::transfer_all_t, boost::asio::detail::wrapped_handler<boost::asio::io_service::strand, websocketpp::transport::asio::custom_alloc_handler<std::_Bind<std::_Mem_fn<void (websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*)(std::function<void (std::error_code const&)>, boost::system::error_code const&, unsigned long)> (std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >, std::function<void (std::error_code const&)>, std::_Placeholder<1>, std::_Placeholder<2>)> >, boost::asio::detail::is_continuation_if_running> >, boost::system::error_code, unsigned long>, websocketpp::transport::asio::custom_alloc_handler<std::_Bind<std::_Mem_fn<void (websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*)(std::function<void (std::error_code const&)>, boost::system::error_code const&, unsigned long)> (std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >, std::function<void (std::error_code const&)>, std::_Placeholder<1>, std::_Placeholder<2>)> > > >::do_complete(boost::asio::detail::task_io_service*, boost::asio::detail::task_io_service_operation*, boost::system::error_code const&, unsigned long) (this=<optimized out>, __n=494)
    at /usr/include/c++/5/bits/stl_vector.h:170
#7  0x000056477eee9b21 in boost::asio::detail::completion_handler<boost::asio::detail::rewrapped_handler<boost::asio::detail::binder2<boost::asio::detail::write_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_service<boost::asio::ip::tcp> >, std::vector<boost::asio::const_buffer, std::allocator<boost::asio::const_buffer> >, boost::asio::detail::transfer_all_t, boost::asio::detail::wrapped_handler<boost::asio::io_service::strand, websocketpp::transport::asio::custom_alloc_handler<std::_Bind<std::_Mem_fn<void (websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*)(std::function<void (std::error_code const&)>, boost::system::error_code const&, unsigned long)> (std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >, std::function<void (std::error_code const&)>, std::_Placeholder<1>, std::_Placeholder<2>)> >, boost::asio::detail::is_continuation_if_running> >, boost::system::error_code, unsigned long>, websocketpp::transport::asio::custom_alloc_handler<std::_Bind<std::_Mem_fn<void (websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*)(std::function<void (std::error_code const&)>, boost::system::error_code const&, unsigned long)> (std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >, std::function<void (std::error_code const&)>, std::_Placeholder<1>, std::_Placeholder<2>)> > > >::do_complete(boost::asio::detail::task_io_service*, boost::asio::detail::task_io_service_operation*, boost::system::error_code const&, unsigned long) (__n=<optimized out>, this=0x7f42c59fb170) at /usr/include/c++/5/bits/stl_vector.h:185
#8  0x000056477eee9b21 in boost::asio::detail::completion_handler<boost::asio::detail::rewrapped_handler<boost::asio::detail::binder2<boost::asio::detail::write_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_service<boost::asio::ip::tcp> >, std::vector<boost::asio::const_buffer, std::allocator<boost::asio::const_buffer> >, boost::asio::detail::transfer_all_t, boost::asio::detail::wrapped_handler<boost::asio::io_service::strand, websocketpp::transport::asio::custom_alloc_handler<std::_Bind<std::_Mem_fn<void (websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*)(std::function<void (std::error_code const&)>, boost::system::error_code const&, unsigned long)> (std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >, std::function<void (std::error_code const&)>, std::_Placeholder<1>, std::_Placeholder<2>)> >, boost::asio::detail::is_continuation_if_running> >, boost::system::error_code, unsigned long>, websocketpp::transport::asio::custom_alloc_handler<std::_Bind<std::_Mem_fn<void (websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*)(std::function<void (std::error_code const&)>, boost::system::error_code const&, unsigned long)> (std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >, std::function<void (std::error_code const&)>, std::_Placeholder<1>, std::_Placeholder<2>)> > > >::do_complete(boost::asio::detail::task_io_service*, boost::asio::detail::task_io_service_operation*, boost::system::error_code const&, unsigned long) (__a=..., __n=<optimized out>, this=0x7f42c59fb170) at /usr/include/c++/5/bits/stl_vector.h:136
#9  0x000056477eee9b21 in boost::asio::detail::completion_handler<boost::asio::detail::rewrapped_handler<boost::asio::detail::binder2<boost::asio::detail::write_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_service<boost::asio::ip::tcp> >, std::vector<boost::asio::const_buffer, std::allocator<boost::asio::const_buffer> >, boost::asio::detail::transfer_all_t, boost::asio::detail::wrapped_handler<boost::asio::io_service::strand, websocketpp::transport::asio::custom_alloc_handler<std::_Bind<std::_Mem_fn<void (websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*)(std::function<void (std::error_code const&)>, boost::system::error_code const&, unsigned long)> (std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >, std::function<void (std::error_code const&)>, std::_Placeholder<1>, std::_Placeholder<2>)> >, boost::asio::detail::is_continuation_if_running> >, boost::system::error_code, unsigned long>, websocketpp::transport::asio::custom_alloc_handler<std::_Bind<std::_Mem_fn<void (websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*)(std::function<void (std::error_code const&)>, boost::system::error_code const&, unsigned long)> (std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >, std::function<void (std::error_code const&)>, std::_Placeholder<1>, std::_Placeholder<2>)> > > >::do_complete(boost::asio::detail::task_io_service*, boost::asio::detail::task_io_service_operation*, boost::system::error_code const&, unsigned long) (__x=std::vector of length 494, capacity 494 = {...}, this=0x7f42c59fb170) at /usr/include/c++/5/bits/stl_vector.h:320
#10 0x000056477eee9b21 in boost::asio::detail::completion_handler<boost::asio::detail::rewrapped_handler<boost::asio::detail::binder2<boost::asio::detail::write_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_service<boost::asio::ip::tcp> >, std::vector<boost::asio::const_buffer, std::allocator<boost::asio::const_buffer> >, boost::asio::detail::transfer_all_t, boost::asio::detail::wrapped_handler<boost::asio::io_service::strand, websocketpp::transport::asio::custom_alloc_handler<std::_Bind<std::_Mem_fn<void (websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*)(std::function<void (std::error_code const&)>, boost::system::error_code const&, unsigned long)> (std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >, std::function<void (std::error_code const&)>, std::_Placeholder<1>, std::_Placeholder<2>)> >, boost::asio::detail::is_continuation_if_running> >, boost::system::error_code, unsigned long>, websocketpp::transport::asio::custom_alloc_handler<std::_Bind<std::_Mem_fn<void (websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*)(std::function<void (std::error_code const&)>, boost::system::error_code const&, unsigned long)> (std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >, std::function<void (std::error_code const&)>, std::_Placeholder<1>, std::_Placeholder<2>)> > > >::do_complete(boost::asio::detail::task_io_service*, boost::asio::detail::task_io_service_operation*, boost::system::error_code const&, unsigned long) (other=..., this=0x7f42c59fb170)
    at /usr/include/boost/asio/detail/consuming_buffers.hpp:186
#11 0x000056477eee9b21 in boost::asio::detail::completion_handler<boost::asio::detail::rewrapped_handler<boost::asio::detail::binder2<boost::asio::detail::write_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_service<boost::asio::ip::tcp> >, std::vector<boost::asio::const_buffer, std::allocator<boost::asio::const_buffer> >, boost::asio::detail::transfer_all_t, boost::asio::detail::wrapped_handler<boost::asio::io_service::strand, websocketpp::transport::asio::custom_alloc_handler<std::_Bind<std::_Mem_fn<void (websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*)(std::function<void (std::error_code const&)>, boost::system::error_code const&, unsigned long)> (std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >, std::function<void (std::error_code const&)>, std::_Placeholder<1>, std::_Placeholder<2>)> >, boost::asio::detail::is_continuation_if_running> >, boost::system::error_code, unsigned long>, websocketpp::transport::asio::custom_alloc_handler<std::_Bind<std::_Mem_fn<void (websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*)(std::function<void (std::error_code const&)>, boost::system::error_code const&, unsigned long)> (std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >, std::function<void (std::error_code const&)>, std::_Placeholder<1>, std::_Placeholder<2>)> > > >::do_complete(boost::asio::detail::task_io_service*, boost::asio::detail::task_io_service_operation*, boost::system::error_code const&, unsigned long) (other=<unknown type in /usr/lib/debug/.build-id/47/50de7fbda2d6efaf9658acc2b864373a4c161e.debug, CU 0x43c7af, DIE 0x6b88fc>, this=0x7f42c59fb168) at /usr/include/boost/asio/impl/write.hpp:167
#12 0x000056477eee9b21 in boost::asio::detail::completion_handler<boost::asio::detail::rewrapped_handler<boost::asio::detail::binder2<boost::asio::detail::write_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_service<boost::asio::ip::tcp> >, std::vector<boost::asio::const_buffer, std::allocator<boost::asio::const_buffer> >, boost::asio::detail::transfer_all_t, boost::asio::detail::wrapped_handler<boost::asio::io_service::strand, websocketpp::transport::asio::custom_alloc_handler<std::_Bind<std::_Mem_fn<void (websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*)(std::function<void (std::error_code const&)>, boost::system::error_code const&, unsigned long)> (std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >, std::function<void (std::error_code const&)>, std::_Placeholder<1>, std::_Placeholder<2>)> >, boost::asio::detail::is_continuation_if_running> >, boost::system::error_code, unsigned long>, websocketpp::transport::asio::custom_alloc_handler<std::_Bind<std::_Mem_fn<void (websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*)(std::function<void (std::error_code const&)>, boost::system::error_code const&, unsigned long)> (std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >, std::function<void (std::error_code const&)>, std::_Placeholder<1>, std::_Placeholder<2>)> > > >::do_complete(boost::asio::detail::task_io_service*, boost::asio::detail::task_io_service_operation*, boost::system::error_code const&, unsigned long) (this=0x7f42c59fb168)
    at /usr/include/boost/asio/detail/bind_handler.hpp:108
#13 0x000056477eee9b21 in boost::asio::detail::completion_handler<boost::asio::detail::rewrapped_handler<boost::asio::detail::binder2<boost::asio::detail::write_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_service<boost::asio::ip::tcp> >, std::vector<boost::asio::const_buffer, std::allocator<boost::asio::const_buffer> >, boost::asio::detail::transfer_all_t, boost::asio::detail::wrapped_handler<boost::asio::io_service::strand, websocketpp::transport::asio::custom_alloc_handler<std::_Bind<std::_Mem_fn<void (websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*)(std::function<void (std::error_code const&)>, boost::system::error_code const&, unsigned long)> (std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >, std::function<void (std::error_code const&)>, std::_Placeholder<1>, std::_Placeholder<2>)> >, boost::asio::detail::is_continuation_if_running> >, boost::system::error_code, unsigned long>, websocketpp::transport::asio::custom_alloc_handler<std::_Bind<std::_Mem_fn<void (websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*)(std::function<void (std::error_code const&)>, boost::system::error_code const&, unsigned long)> (std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >, std::function<void (std::error_code const&)>, std::_Placeholder<1>, std::_Placeholder<2>)> > > >::do_complete(boost::asio::detail::task_io_service*, boost::asio::detail::task_io_service_operation*, boost::system::error_code const&, unsigned long) (other=<unknown type in /usr/lib/debug/.build-id/47/50de7fbda2d6efaf9658acc2b864373a4c161e.debug, CU 0x43c7af, DIE 0x739142>, this=0x7f42c59fb120) at /usr/include/boost/asio/detail/wrapped_handler.hpp:184
#14 0x000056477eee9b21 in boost::asio::detail::completion_handler<boost::asio::detail::rewrapped_handler<boost::asio::detail::binder2<boost::asio::detail::write_op<boost::asio::basic_stream_socket<boost::asio::ip::tcp, boost::asio::stream_socket_service<boost::asio::ip::tcp> >, std::vector<boost::asio::const_buffer, std::allocator<boost::asio::const_buffer> >, boost::asio::detail::transfer_all_t, boost::asio::detail::wrapped_handler<boost::asio::io_service::strand, websocketpp::transport::asio::custom_alloc_handler<std::_Bind<std::_Mem_fn<void (websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config>::*)(std::function<void (std::error_code const&)>, boost::system::error_code const&, unsigned long)> (std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >, std::function<void (std::error_code const&)>, std::_Placeholder<1>, std::_Placeholder<2>)> >, boost::asio::detail::is_continuation_if_running> >, boost::system::error_code, unsigned long>, websocketpp::transport::asio::custom_alloc_handler<std::_Bind<std::_Mem_fn<void (websocketpp---Type <return> to continue, or q <return> to quit---
::transport::asio::connection<websocketpp::config::asio::transport_config>::*)(std::function<void (std::error_code const&)>, boost::system::error_code const&, unsigned long)> (std::shared_ptr<websocketpp::transport::asio::connection<websocketpp::config::asio::transport_config> >, std::function<void (std::error_code const&)>, std::_Placeholder<1>, std::_Placeholder<2>)> > > >::do_complete(boost::asio::detail::task_io_service*, boost::asio::detail::task_io_service_operation*, boost::system::error_code const&, unsigned long) (owner=0x564780ca3560, base=0x564780ca3f50) at /usr/include/boost/asio/detail/completion_handler.hpp:59
#15 0x000056477ee995fd in boost::asio::detail::strand_service::do_complete(boost::asio::detail::task_io_service*, boost::asio::detail::task_io_service_operation*, boost::system::error_code const&, unsigned long) (bytes_transferred=0, ec=..., owner=..., this=<optimized out>)
    at /usr/include/boost/asio/detail/task_io_service_operation.hpp:38
#16 0x000056477ee995fd in boost::asio::detail::strand_service::do_complete(boost::asio::detail::task_io_service*, boost::asio::detail::task_io_service_operation*, boost::system::error_code const&, unsigned long) (owner=0x564780ca3560, base=0x564780ca2710, ec=...)
    at /usr/include/boost/asio/detail/impl/strand_service.ipp:167
#17 0x000056477ee89fcb in kurento::WebSocketTransport::run() (bytes_transferred=0, ec=..., owner=..., this=<optimized out>) at /usr/include/boost/asio/detail/task_io_service_operation.hpp:38
#18 0x000056477ee89fcb in kurento::WebSocketTransport::run() (ec=..., this_thread=..., lock=..., this=0x564780ca3560) at /usr/include/boost/asio/detail/impl/task_io_service.ipp:372
#19 0x000056477ee89fcb in kurento::WebSocketTransport::run() (ec=..., this=0x564780ca3560) at /usr/include/boost/asio/detail/impl/task_io_service.ipp:149
#20 0x000056477ee89fcb in kurento::WebSocketTransport::run() (this=0x564780c6d248) at /usr/include/boost/asio/impl/io_service.ipp:59
#21 0x000056477ee89fcb in kurento::WebSocketTransport::run() (this=0x564780c6d140) at /build/server/transport/websocket/WebSocketTransport.cpp:103
#22 0x00007f433d7dbc80 in  () at /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#23 0x00007f433f88f6ba in start_thread () at /lib/x86_64-linux-gnu/libpthread.so.0
#24 0x00007f433d24a4dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109
(gdb) quit

(Optional) Possible solution

Info about your environment

About Kurento Media Server

About your Application Server

  • Programming Language: node.js
  • Kurento Client version: 6.15.0

Run these commands

I run the command into docker container

# cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=16.04
DISTRIB_CODENAME=xenial
DISTRIB_DESCRIPTION="Ubuntu 16.04.6 LTS"

kurento-media-server --version
0:00:00.014285031 19688 0x55e36ba44270 INFO    KurentoModuleManager ModuleManager.cpp:90:loadModule: Load file: /usr/lib/x86_64-linux-gnu/kurento/modules/libkmsfiltersmodule.so, module name: /usr/lib/x86_64-linux-gnu/kurento/modules/libkmsfiltersmodule.so
0:00:00.014341591 19688 0x55e36ba44270 INFO    KurentoModuleManager ModuleManager.cpp:137:loadModule: Loaded module: filters, version: 6.15.0, date: Nov  3 2020 18:33:58
0:00:00.014658784 19688 0x55e36ba44270 INFO    KurentoModuleManager ModuleManager.cpp:90:loadModule: Load file: /usr/lib/x86_64-linux-gnu/kurento/modules/libkmscoremodule.so, module name: /usr/lib/x86_64-linux-gnu/kurento/modules/libkmscoremodule.so
0:00:00.014687655 19688 0x55e36ba44270 INFO    KurentoModuleManager ModuleManager.cpp:137:loadModule: Loaded module: core, version: 6.15.0, date: Nov  3 2020 18:12:59
0:00:00.022753644 19688 0x55e36ba44270 INFO    KurentoModuleManager ModuleManager.cpp:90:loadModule: Load file: /usr/lib/x86_64-linux-gnu/kurento/modules/libkmselementsmodule.so, module name: /usr/lib/x86_64-linux-gnu/kurento/modules/libkmselementsmodule.so
0:00:00.022791185 19688 0x55e36ba44270 INFO    KurentoModuleManager ModuleManager.cpp:137:loadModule: Loaded module: elements, version: 6.15.0, date: Nov  3 2020 18:22:58
0:00:00.023270089 19688 0x55e36ba44270 INFO    KurentoModuleManager ModuleManager.cpp:90:loadModule: Load file: /usr/lib/x86_64-linux-gnu/kurento/modules/libkmspointerdetectormodule.so, module name: /usr/lib/x86_64-linux-gnu/kurento/modules/libkmspointerdetectormodule.so
0:00:00.023295380 19688 0x55e36ba44270 INFO    KurentoModuleManager ModuleManager.cpp:137:loadModule: Loaded module: pointerdetector, version: 6.15.0, date: Nov  4 2020 12:26:22
0:00:00.023724744 19688 0x55e36ba44270 INFO    KurentoModuleManager ModuleManager.cpp:90:loadModule: Load file: /usr/lib/x86_64-linux-gnu/kurento/modules/libkmsplatedetectormodule.so, module name: /usr/lib/x86_64-linux-gnu/kurento/modules/libkmsplatedetectormodule.so
0:00:00.023748634 19688 0x55e36ba44270 INFO    KurentoModuleManager ModuleManager.cpp:137:loadModule: Loaded module: platedetector, version: 6.15.0, date: Nov  4 2020 12:26:25
0:00:00.024299370 19688 0x55e36ba44270 INFO    KurentoModuleManager ModuleManager.cpp:90:loadModule: Load file: /usr/lib/x86_64-linux-gnu/kurento/modules/libkmscrowddetectormodule.so, module name: /usr/lib/x86_64-linux-gnu/kurento/modules/libkmscrowddetectormodule.so
0:00:00.024326290 19688 0x55e36ba44270 INFO    KurentoModuleManager ModuleManager.cpp:137:loadModule: Loaded module: crowddetector, version: 6.15.0, date: Nov  4 2020 12:26:52
0:00:00.024734014 19688 0x55e36ba44270 INFO    KurentoModuleManager ModuleManager.cpp:90:loadModule: Load file: /usr/lib/x86_64-linux-gnu/kurento/modules/libkmschromamodule.so, module name: /usr/lib/x86_64-linux-gnu/kurento/modules/libkmschromamodule.so
0:00:00.024760604 19688 0x55e36ba44270 INFO    KurentoModuleManager ModuleManager.cpp:137:loadModule: Loaded module: chroma, version: 6.15.0, date: Nov  4 2020 12:20:06
Kurento Media Server version: 6.15.0
Found modules:
        'chroma' version 6.15.0
        'core' version 6.15.0
        'crowddetector' version 6.15.0
        'elements' version 6.15.0
        'filters' version 6.15.0
        'platedetector' version 6.15.0
        'pointerdetector' version 6.15.0

 dpkg -l | grep -Pi 'kurento|kms-|gst.*1.5|nice'
ii  gstreamer1.5-alsa:amd64              1.8.1-1kurento2.16.04                    amd64        GStreamer plugin for ALSA
ii  gstreamer1.5-libav:amd64             1.8.1-1kurento1.16.04                    amd64        libav plugin for GStreamer
ii  gstreamer1.5-libav-dbg:amd64         1.8.1-1kurento1.16.04                    amd64        libav plugin for GStreamer (debug symbols)
ii  gstreamer1.5-nice:amd64              0.1.18-0kurento1.16.04                   amd64        ICE library (GStreamer 1.5 plugin)
ii  gstreamer1.5-nice-dbgsym:amd64       0.1.18-0kurento1.16.04                   amd64        debug symbols for package gstreamer1.5-nice
ii  gstreamer1.5-plugins-bad:amd64       1.8.1-1kurento5.16.04                    amd64        GStreamer plugins from the "bad" set
ii  gstreamer1.5-plugins-bad-dbg:amd64   1.8.1-1kurento5.16.04                    amd64        GStreamer plugins from the "bad" set (debug symbols)
ii  gstreamer1.5-plugins-base:amd64      1.8.1-1kurento2.16.04                    amd64        GStreamer plugins from the "base" set
ii  gstreamer1.5-plugins-base-dbg:amd64  1.8.1-1kurento2.16.04                    amd64        GStreamer plugins from the "base" set
ii  gstreamer1.5-plugins-good:amd64      1.8.1-1kurento5.16.04                    amd64        GStreamer plugins from the "good" set
ii  gstreamer1.5-plugins-good-dbg:amd64  1.8.1-1kurento5.16.04                    amd64        GStreamer plugins from the "good" set
ii  gstreamer1.5-plugins-ugly:amd64      1.8.1-1kurento1.16.04                    amd64        GStreamer plugins from the "ugly" set
ii  gstreamer1.5-plugins-ugly-dbg:amd64  1.8.1-1kurento1.16.04                    amd64        GStreamer plugins from the "ugly" set (debug symbols)
ii  gstreamer1.5-pulseaudio:amd64        1.8.1-1kurento5.16.04                    amd64        GStreamer plugin for PulseAudio
ii  gstreamer1.5-x:amd64                 1.8.1-1kurento2.16.04                    amd64        GStreamer plugins for X11 and Pango
ii  kms-chroma                           6.15.0-0kurento1.16.04                   amd64        Kurento Chroma filter
ii  kms-core                             6.15.0-0kurento1.16.04                   amd64        Kurento Core module
ii  kms-core-dbg                         6.15.0-0kurento1.16.04                   amd64        Kurento Core module
ii  kms-crowddetector                    6.15.0-0kurento1.16.04                   amd64        Kurento Crowd Detector filter
ii  kms-elements                         6.15.0-0kurento1.16.04                   amd64        Kurento Elements module
ii  kms-elements-dbg                     6.15.0-0kurento1.16.04                   amd64        Kurento Elements module
ii  kms-filters                          6.15.0-0kurento1.16.04                   amd64        Kurento Filters module
ii  kms-filters-dbg                      6.15.0-0kurento1.16.04                   amd64        Kurento Filters module
ii  kms-jsonrpc                          6.15.0-0kurento1.16.04                   amd64        Kurento JSON-RPC library
ii  kms-jsonrpc-dbg                      6.15.0-0kurento1.16.04                   amd64        Kurento JSON-RPC library
ii  kms-platedetector                    6.15.0-0kurento1.16.04                   amd64        Kurento Plate Detector filter
ii  kms-pointerdetector                  6.15.0-0kurento1.16.04                   amd64        Kurento Pointer Detector filter
ii  kmsjsoncpp                           1.6.3-1kurento1.16.04                    amd64        Kurento jsoncpp library
ii  kmsjsoncpp-dbg                       1.6.3-1kurento1.16.04                    amd64        Kurento jsoncpp library
ii  kurento-dbg                          6.15.0-0kurento1.16.04                   amd64        Meta-package that installs debug symbols
ii  kurento-media-server                 6.15.0-0kurento1.16.04                   amd64        Kurento Media Server
ii  kurento-media-server-dbg             6.15.0-0kurento1.16.04                   amd64        Kurento Media Server
ii  libgstreamer-plugins-bad1.5-0:amd64  1.8.1-1kurento5.16.04                    amd64        GStreamer development files for libraries from the "bad" set
ii  libgstreamer-plugins-base1.5-0:amd64 1.8.1-1kurento2.16.04                    amd64        GStreamer libraries from the "base" set
ii  libgstreamer1.5-0:amd64              1.8.1-1kurento2.16.04                    amd64        Core GStreamer libraries and elements
ii  libgstreamer1.5-0-dbg:amd64          1.8.1-1kurento2.16.04                    amd64        Core GStreamer libraries and elements
ii  libnice10:amd64                      0.1.18-0kurento1.16.04                   amd64        ICE library (shared library)
ii  libnice10-dbgsym:amd64               0.1.18-0kurento1.16.04                   amd64        debug symbols for package libnice10
ii  libsrtp0:amd64                       1.6.0-0kurento1.16.04                    amd64        Secure RTP (SRTP) and UST Reference Implementations - shared library
ii  libusrsctp                           0.9.2-1kurento1.16.04                    amd64        sctp-refimpl library
ii  openh264                             1.5.0-0kurento1.16.04                    amd64        H.264 Video Codec provided by Cisco Systems, Inc.
ii  openh264-gst-plugins-bad-1.5:amd64   1.8.1-1kurento5.16.04                    amd64        GStreamer plugins from openh264
ii  openwebrtc-gst-plugins               0.10.0-1kurento1.16.04                   amd64        OpenWebRTC specific GStreamer plugins
ii  openwebrtc-gst-plugins-dbg           0.10.0-1kurento1.16.04                   amd64        OpenWebRTC specific GStreamer plugins
@github-actions
Copy link

github-actions bot commented Jun 1, 2021

Hello @avssav! 👋 we're sorry you found a bug... so first of all, thank you very much for reporting it.

To know about progress, check in Triage. All issues are considered Backlog Candidates until work priorities align and the issue is selected for development. It will then become part of our official Backlog.

@j1elo j1elo added this to Backlog Candidates in Triage Jun 1, 2021
@avssav
Copy link
Author

avssav commented Jul 3, 2021 •
edited

Our another kurento was crashed yesterday. Dump file didn't create but I have some logs. Maybe it will helps

Jul  2 17:09:39 kms11 b5bbd8291fc3[1382]: 14:44:08.558033648     1 0x7f06202514a0 WARN               rtpsource rtpsource.c:1147:update_receiver_stats: duplicate or reordered packet (seqnr 11750, expected 11763)
Jul  2 17:09:39 kms11 b5bbd8291fc3[1382]: 14:44:08.788472544     1 0x7f0560055940 WARN               rtpsource rtpsource.c:1147:update_receiver_stats: duplicate or reordered packet (seqnr 33826, expected 33835)
Jul  2 17:09:39 kms11 b5bbd8291fc3[1382]: 14:44:08.867830372     1 0x7f0604329230 WARN         rtpsynchronizer kmsrtpsynchronizer.c:553:kms_rtp_synchronizer_process_rtp_buffer_mapped:<KmsRtpSynchronizer@0x7f05d51a8b70> [Sorted mode] Fix PTS not increasing monotonically, SSRC: 3952
380878, seq: 32906, rtp_ts: 2703411984, ext_ts: 2703411984, last: 0:08:11.805705539, current: 0:08:11.800219539, fixed = last: 0:08:11.805705539
Jul  2 17:09:39 kms11 b5bbd8291fc3[1382]: 14:44:08.867932717     1 0x7f0604329230 WARN                kmsutils kmsutils.c:1449:kms_utils_depayloader_adjust_pts_out:<rtpopusdepay2900> Fix PTS not strictly increasing, last: 0:08:11.805705539, current: 0:08:11.805705539, fixed = last + 1: 0:08:11.806705539
Jul  2 17:09:39 kms11 b5bbd8291fc3[1382]: *** Error in `/usr/bin/kurento-media-server': corrupted double-linked list: 0x00007f05640c6060 ***
Jul  2 17:09:39 kms11 b5bbd8291fc3[1382]: ======= Backtrace: =========
Jul  2 17:09:39 kms11 b5bbd8291fc3[1382]: /lib/x86_64-linux-gnu/libc.so.6(+0x777f5)[0x7f08bce537f5]
Jul  2 17:09:39 kms11 b5bbd8291fc3[1382]: /lib/x86_64-linux-gnu/libc.so.6(+0x8298b)[0x7f08bce5e98b]
Jul  2 17:09:39 kms11 b5bbd8291fc3[1382]: /lib/x86_64-linux-gnu/libc.so.6(__libc_malloc+0x54)[0x7f08bce601d4]
Jul  2 17:09:39 kms11 b5bbd8291fc3[1382]: /usr/lib/libkmsjsoncpp.so.1(_ZN4Json5Value8CZStringC1ERKS1_+0x46)[0x7f08bdbd5a96]
Jul  2 17:09:39 kms11 b5bbd8291fc3[1382]: /usr/lib/libkmsjsoncpp.so.1(_ZNSt8_Rb_treeIN4Json5Value8CZStringESt4pairIKS2_S1_ESt10_Select1stIS5_ESt4lessIS2_ESaIS5_EE7_M_copyINSB_11_Alloc_nodeEEEPSt13_Rb_tree_nodeIS5_EPKSF_SG_RT_+0x3c)[0x7f08bdbdcbcc]
Jul  2 17:09:39 kms11 b5bbd8291fc3[1382]: /usr/lib/libkmsjsoncpp.so.1(_ZNSt8_Rb_treeIN4Json5Value8CZStringESt4pairIKS2_S1_ESt10_Select1stIS5_ESt4lessIS2_ESaIS5_EE7_M_copyINSB_11_Alloc_nodeEEEPSt13_Rb_tree_nodeIS5_EPKSF_SG_RT_+0xef)[0x7f08bdbdcc7f]
Jul  2 17:09:39 kms11 b5bbd8291fc3[1382]: /usr/lib/libkmsjsoncpp.so.1(_ZNSt8_Rb_treeIN4Json5Value8CZStringESt4pairIKS2_S1_ESt10_Select1stIS5_ESt4lessIS2_ESaIS5_EE7_M_copyINSB_11_Alloc_nodeEEEPSt13_Rb_tree_nodeIS5_EPKSF_SG_RT_+0x7d)[0x7f08bdbdcc0d]
Jul  2 17:09:39 kms11 b5bbd8291fc3[1382]: /usr/lib/libkmsjsoncpp.so.1(_ZNSt8_Rb_treeIN4Json5Value8CZStringESt4pairIKS2_S1_ESt10_Select1stIS5_ESt4lessIS2_ESaIS5_EE7_M_copyINSB_11_Alloc_nodeEEEPSt13_Rb_tree_nodeIS5_EPKSF_SG_RT_+0x7d)[0x7f08bdbdcc0d]
Jul  2 17:09:39 kms11 b5bbd8291fc3[1382]: /usr/lib/libkmsjsoncpp.so.1(_ZN4Json5ValueC2ERKS0_+0x19c)[0x7f08bdbdaebc]
Jul  2 17:09:39 kms11 b5bbd8291fc3[1382]: /usr/lib/libkmsjsoncpp.so.1(_ZNSt8_Rb_treeIN4Json5Value8CZStringESt4pairIKS2_S1_ESt10_Select1stIS5_ESt4lessIS2_ESaIS5_EE7_M_copyINSB_11_Alloc_nodeEEEPSt13_Rb_tree_nodeIS5_EPKSF_SG_RT_+0x49)[0x7f08bdbdcbd9]
Jul  2 17:09:39 kms11 b5bbd8291fc3[1382]: /usr/lib/libkmsjsoncpp.so.1(_ZNSt8_Rb_treeIN4Json5Value8CZStringESt4pairIKS2_S1_ESt10_Select1stIS5_ESt4lessIS2_ESaIS5_EE7_M_copyINSB_11_Alloc_nodeEEEPSt13_Rb_tree_nodeIS5_EPKSF_SG_RT_+0x7d)[0x7f08bdbdcc0d]
Jul  2 17:09:39 kms11 b5bbd8291fc3[1382]: /usr/lib/libkmsjsoncpp.so.1(_ZN4Json5ValueC2ERKS0_+0x19c)[0x7f08bdbdaebc]
Jul  2 17:09:39 kms11 b5bbd8291fc3[1382]: /usr/lib/x86_64-linux-gnu/libkmscoreimpl.so.6(_ZN7kurento16MediaElementImpl6invokeESt10shared_ptrINS_15MediaObjectImplEERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERKN4Json5ValueERSD_+0xd4b)[0x7f08be13729b]
Jul  2 17:09:39 kms11 b5bbd8291fc3[1382]: /usr/lib/x86_64-linux-gnu/libkmscoreimpl.so.6(_ZN7kurento12EndpointImpl6invokeESt10shared_ptrINS_15MediaObjectImplEERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERKN4Json5ValueERSD_+0x42)[0x7f08be121782]
Jul  2 17:09:39 kms11 b5bbd8291fc3[1382]: /usr/lib/x86_64-linux-gnu/libkmscoreimpl.so.6(_ZN7kurento19SessionEndpointImpl6invokeESt10shared_ptrINS_15MediaObjectImplEERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERKN4Json5ValueERSD_+0x41)[0x7f08be11c4e1]
Jul  2 17:09:39 kms11 b5bbd8291fc3[1382]: /usr/lib/x86_64-linux-gnu/libkmscoreimpl.so.6(_ZN7kurento15SdpEndpointImpl6invokeESt10shared_ptrINS_15MediaObjectImplEERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERKN4Json5ValueERSD_+0x16f0)[0x7f08be12a4a0]
Jul  2 17:09:39 kms11 b5bbd8291fc3[1382]: /usr/lib/x86_64-linux-gnu/libkmscoreimpl.so.6(_ZN7kurento19BaseRtpEndpointImpl6invokeESt10shared_ptrINS_15MediaObjectImplEERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERKN4Json5ValueERSD_+0x178a)[0x7f08be12f77a]
Jul  2 17:09:39 kms11 b5bbd8291fc3[1382]: /usr/lib/x86_64-linux-gnu/libkmselementsimpl.so.6(_ZN7kurento18WebRtcEndpointImpl6invokeESt10shared_ptrINS_15MediaObjectImplEERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERKN4Json5ValueERSD_+0x1801)[0x7f08b947aec1]
Jul  2 17:09:39 kms11 b5bbd8291fc3[1382]: /usr/bin/kurento-media-server(_ZN7kurento13ServerMethods6invokeERKN4Json5ValueERS2_+0x258)[0x556cef473b28]
Jul  2 17:09:39 kms11 b5bbd8291fc3[1382]: /usr/lib/x86_64-linux-gnu/libjsonrpc.so.6(_ZN7kurento7JsonRpc7Handler7processERKN4Json5ValueERS3_+0x5da)[0x7f08bddf37fa]
Jul  2 17:09:39 kms11 b5bbd8291fc3[1382]: /usr/bin/kurento-media-server(_ZN7kurento13ServerMethods7processERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERS6_S9_+0x1ca)[0x556cef477e6a]
Jul  2 17:09:39 kms11 b5bbd8291fc3[1382]: /usr/bin/kurento-media-server(_ZN7kurento18WebSocketTransport14processMessageIN11websocketpp6serverINS2_6config4asioEEEEEvPT_St8weak_ptrIvENS7_11message_ptrE+0x16e)[0x556cef4e2e8e]
Jul  2 17:09:39 kms11 b5bbd8291fc3[1382]: /usr/bin/kurento-media-server(_ZNSt17_Function_handlerIFvSt8weak_ptrIvESt10shared_ptrIN11websocketpp14message_buffer7messageINS4_5alloc15con_msg_managerEEEEESt5_BindIFSt7_Mem_fnIMN7kurento18WebSocketTransportEFvPNS3_6serverINS3_6config4asi
oEEES1_S9_EEPSE_SJ_St12_PlaceholderILi1EESO_ILi2EEEEE9_M_invokeERKSt9_Any_dataOS1_OS9_+0x84)[0x556cef4c8fa4]
Jul  2 17:09:39 kms11 b5bbd8291fc3[1382]: /usr/bin/kurento-media-server(_ZN11websocketpp10connectionINS_6config4asioEE17handle_read_frameERKSt10error_codem+0x836)[0x556cef4fc0f6]
Jul  2 17:09:39 kms11 b5bbd8291fc3[1382]: /usr/bin/kurento-media-server(_ZN11websocketpp9transport4asio10connectionINS_6config4asio16transport_configEE17handle_async_readESt8functionIFvRKSt10error_codemEERKN5boost6system10error_codeEm+0x9a)[0x556cef4da0aa]
Jul  2 17:09:39 kms11 b5bbd8291fc3[1382]: /usr/bin/kurento-media-server(_ZN5boost4asio6detail14strand_service8dispatchINS1_7binder2IN11websocketpp9transport4asio20custom_alloc_handlerISt5_BindIFSt7_Mem_fnIMNS7_10connectionINS5_6config4asio16transport_configEEEFvSt8functionIFvRKSt1
0error_codemEERKNS_6system10error_codeEmEESt10shared_ptrISF_ESL_St12_PlaceholderILi1EESV_ILi2EEEEEESN_mEEEEvRPNS2_11strand_implERT_+0xc7)[0x556cef4ebf07]
Jul  2 17:09:39 kms11 b5bbd8291fc3[1382]: /usr/bin/kurento-media-server(_ZN5boost4asio6detail15wrapped_handlerINS0_10io_service6strandEN11websocketpp9transport4asio20custom_alloc_handlerISt5_BindIFSt7_Mem_fnIMNS7_10connectionINS5_6config4asio16transport_configEEEFvSt8functionIFvRKSt10error_codemEERKNS_6system10error_codeEmEESt10shared_ptrISF_ESL_St12_PlaceholderILi1EESV_ILi2EEEEEENS1_26is_continuation_if_runningEEclISN_mEEvRKT_RKT0_+0x152)[0x556cef4ec2b2]
Jul  2 17:09:39 kms11 b5bbd8291fc3[1382]: /usr/bin/kurento-media-server(_ZN5boost4asio6detail7read_opINS0_19basic_stream_socketINS0_2ip3tcpENS0_21stream_socket_serviceIS5_EEEENS0_17mutable_buffers_1ENS1_19transfer_at_least_tENS1_15wrapped_handlerINS0_10io_service6strandEN11websocketpp9transport4asio20custom_alloc_handlerISt5_BindIFSt7_Mem_fnIMNSG_10connectionINSE_6config4asio16transport_configEEEFvSt8functionIFvRKSt10error_codemEERKNS_6system10error_codeEmEESt10shared_ptrISO_ESU_St12_PlaceholderILi1EES14_ILi2EEEEEENS1_26is_continuation_if_runningEEEEclESY_mi+0x30e)[0x556cef4ec63e]
Jul  2 17:09:39 kms11 b5bbd8291fc3[1382]: /usr/bin/kurento-media-server(_ZN5boost4asio6detail18completion_handlerINS1_17rewrapped_handlerINS1_7binder2INS1_7read_opINS0_19basic_stream_socketINS0_2ip3tcpENS0_21stream_socket_serviceIS8_EEEENS0_17mutable_buffers_1ENS1_19transfer_at_least_tENS1_15wrapped_handlerINS0_10io_service6strandEN11websocketpp9transport4asio20custom_alloc_handlerISt5_BindIFSt7_Mem_fnIMNSJ_10connectionINSH_6config4asio16transport_configEEEFvSt8functionIFvRKSt10error_codemEERKNS_6system10error_codeEmEESt10shared_ptrISR_ESX_St12_PlaceholderILi1EES17_ILi2EEEEEENS1_26is_continuation_if_runningEEEEESZ_mEES1C_EEE11do_completeEPNS1_15task_io_serviceEPNS1_25task_io_service_operationES11_m+0x1a1)[0x556cef512471]
Jul  2 17:09:39 kms11 b5bbd8291fc3[1382]: /usr/bin/kurento-media-server(_ZN5boost4asio6detail14strand_service8dispatchINS1_17rewrapped_handlerINS1_7binder2INS1_7read_opINS0_19basic_stream_socketINS0_2ip3tcpENS0_21stream_socket_serviceIS9_EEEENS0_17mutable_buffers_1ENS1_19transfer_at_least_tENS1_15wrapped_handlerINS0_10io_service6strandEN11websocketpp9transport4asio20custom_alloc_handlerISt5_BindIFSt7_Mem_fnIMNSK_10connectionINSI_6config4asio16transport_configEEEFvSt8functionIFvRKSt10error_codemEERKNS_6system10error_codeEmEESt10shared_ptrISS_ESY_St12_PlaceholderILi1EES18_ILi2EEEEEENS1_26is_continuation_if_runningEEEEES10_mEES1D_EEEEvRPNS2_11strand_implERT_+0x253)[0x556cef512793]
Jul  2 17:09:39 kms11 b5bbd8291fc3[1382]: /usr/bin/kurento-media-server(_ZN5boost4asio6detail23reactive_socket_recv_opINS0_17mutable_buffers_1ENS1_7read_opINS0_19basic_stream_socketINS0_2ip3tcpENS0_21stream_socket_serviceIS7_EEEES3_NS1_19transfer_at_least_tENS1_15wrapped_handlerINS0_10io_service6strandEN11websocketpp9transport4asio20custom_alloc_handlerISt5_BindIFSt7_Mem_fnIMNSH_10connectionINSF_6config4asio16transport_configEEEFvSt8functionIFvRKSt10error_codemEERKNS_6system10error_codeEmEESt10shared_ptrISP_ESV_St12_PlaceholderILi1EES15_ILi2EEEEEENS1_26is_continuation_if_runningEEEEEE11do_completeEPNS1_15task_io_serviceEPNS1_25task_io_service_operationESZ_m+0x3b0)[0x556cef512c20]
Jul  2 17:09:39 kms11 b5bbd8291fc3[1382]: /usr/bin/kurento-media-server(_ZN5boost4asio6detail13epoll_reactor16descriptor_state11do_completeEPNS1_15task_io_serviceEPNS1_25task_io_service_operationERKNS_6system10error_codeEm+0x204)[0x556cef4c0bd4]
Jul  2 17:09:39 kms11 b5bbd8291fc3[1382]: /usr/bin/kurento-media-server(_ZN7kurento18WebSocketTransport3runEv+0x4bb)[0x556cef4b0fcb]
Jul  2 17:09:39 kms11 b5bbd8291fc3[1382]: /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xb8c80)[0x7f08bd474c80]
Jul  2 17:09:39 kms11 b5bbd8291fc3[1382]: /lib/x86_64-linux-gnu/libpthread.so.0(+0x76ba)[0x7f08bf5286ba]
Jul  2 17:09:39 kms11 b5bbd8291fc3[1382]: /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7f08bcee34dd]
Jul  2 17:09:39 kms11 b5bbd8291fc3[1382]: ======= Memory map: ========
Jul  2 17:09:39 kms11 b5bbd8291fc3[1382]: 556cef380000-556cef589000 r-xp 00000000 103:01 2824916                   /usr/bin/kurento-media-server
Jul  2 17:09:39 kms11 b5bbd8291fc3[1382]: 556cef788000-556cef790000 r--p 00208000 103:01 2824916                   /usr/bin/kurento-media-server
Jul  2 17:09:39 kms11 b5bbd8291fc3[1382]: 556cef790000-556cef792000 rw-p 00210000 103:01 2824916                   /usr/bin/kurento-media-server
Jul  2 17:09:39 kms11 b5bbd8291fc3[1382]: 556cef792000-556cef793000 rw-p 00000000 00:00 0
Jul  2 17:09:39 kms11 b5bbd8291fc3[1382]: 556cf0f98000-556cf3dc1000 rw-p 00000000 00:00 0                          [heap]

@avssav
Copy link
Author

avssav commented Aug 3, 2021

One more crush on low load service (

Aug  3 10:30:22 kms16 8108644a900d[1329]: 8:04:47.159301192     1 0x7f3464280de0 WARN               rtpsource rtpsource.c:1147:update_receiver_stats: duplicate or reordered packet (seqnr 13690, expected 13702)
Aug  3 10:30:22 kms16 8108644a900d[1329]: 8:04:47.159620470     1 0x7f3464280de0 WARN               rtpsource rtpsource.c:1147:update_receiver_stats: duplicate or reordered packet (seqnr 13691, expected 13702)
Aug  3 10:30:22 kms16 8108644a900d[1329]: *** Error in `/usr/bin/kurento-media-server': malloc(): smallbin double linked list corrupted: 0x00007f3530365e70 ***
Aug  3 10:30:22 kms16 8108644a900d[1329]: ======= Backtrace: =========
Aug  3 10:30:22 kms16 8108644a900d[1329]: /lib/x86_64-linux-gnu/libc.so.6(+0x777f5)[0x7f38454d47f5]
Aug  3 10:30:22 kms16 8108644a900d[1329]: /lib/x86_64-linux-gnu/libc.so.6(+0x82679)[0x7f38454df679]
Aug  3 10:30:22 kms16 8108644a900d[1329]: /lib/x86_64-linux-gnu/libc.so.6(__libc_calloc+0xba)[0x7f38454e1e3a]
Aug  3 10:30:22 kms16 8108644a900d[1329]: /lib/x86_64-linux-gnu/libglib-2.0.so.0(g_malloc0+0x21)[0x7f3846ab2811]
Aug  3 10:30:22 kms16 8108644a900d[1329]: /lib/x86_64-linux-gnu/libglib-2.0.so.0(g_hash_table_new_full+0x85)[0x7f3846a9b7b5]
Aug  3 10:30:22 kms16 8108644a900d[1329]: /usr/lib/x86_64-linux-gnu/gstreamer-1.5/libgstrtpmanager.so(+0x2f0ae)[0x7f36705d60ae]
Aug  3 10:30:22 kms16 8108644a900d[1329]: /usr/lib/x86_64-linux-gnu/gstreamer-1.5/libgstrtpmanager.so(+0x38799)[0x7f36705df799]
Aug  3 10:30:22 kms16 8108644a900d[1329]: /lib/x86_64-linux-gnu/libglib-2.0.so.0(+0x70c55)[0x7f3846ad3c55]
Aug  3 10:30:22 kms16 8108644a900d[1329]: /lib/x86_64-linux-gnu/libpthread.so.0(+0x76ba)[0x7f3847ba96ba]
Aug  3 10:30:22 kms16 8108644a900d[1329]: /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7f38455644dd]
Aug  3 10:30:22 kms16 8108644a900d[1329]: ======= Memory map: ========
Aug  3 10:30:22 kms16 8108644a900d[1329]: 5580b8a25000-5580b8c2e000 r-xp 00000000 103:01 2822457                   /usr/bin/kurento-media-server
Aug  3 10:30:22 kms16 8108644a900d[1329]: 5580b8e2d000-5580b8e35000 r--p 00208000 103:01 2822457                   /usr/bin/kurento-media-server
Aug  3 10:30:22 kms16 8108644a900d[1329]: 5580b8e35000-5580b8e37000 rw-p 00210000 103:01 2822457                   /usr/bin/kurento-media-server
Aug  3 10:30:22 kms16 8108644a900d[1329]: 5580b8e37000-5580b8e38000 rw-p 00000000 00:00 0
Aug  3 10:30:22 kms16 8108644a900d[1329]: 5580ba94b000-5580bc337000 rw-p 00000000 00:00 0                          [heap]

@avssav
Copy link
Author

avssav commented Nov 24, 2021

An another one:

(gdb) bt
#0  0x00007f41f616c1a6 in __GI_abort () at abort.c:125
#1  0x00005599c56c6769 in Debug::DeathHandler::SignalHandler(int, void*, void*) () at /build/server/death_handler.cpp:299
#2  0x00005599c56c6769 in Debug::DeathHandler::SignalHandler(int, void*, void*) (sig=<optimized out>, secret=<optimized out>) at /build/server/death_handler.cpp:600
#3  0x00007f41f888b390 in <signal handler called> () at /lib/x86_64-linux-gnu/libpthread.so.0
#4  0x00007f41f27b2839 in sigc::internal::signal_emit1<void, kurento::DataChannelClose, sigc::nil>::emit(sigc::internal::signal_impl*, kurento::DataChannelClose const&) (this=<optimized out>) at /usr/include/sigc++-2.0/sigc++/functors/slot_base.h:309
#5  0x00007f41f27b2839 in sigc::internal::signal_emit1<void, kurento::DataChannelClose, sigc::nil>::emit(sigc::internal::signal_impl*, kurento::DataChannelClose const&) (impl=0x7f3f240f4130, _A_a1=...) at /usr/include/sigc++-2.0/sigc++/signal.h:1006
#6  0x00007f41f27abc68 in kurento::WebRtcEndpointImpl::onIceCandidate(char*, _KmsIceCandidate*) (this=0x7f3e6406f648, _A_a1=...) at /usr/include/sigc++-2.0/sigc++/signal.h:2814
#7  0x00007f41f27abc68 in kurento::WebRtcEndpointImpl::onIceCandidate(char*, _KmsIceCandidate*) (event=..., sigcSignal=..., this=0x7f3e6406f290) at /usr/include/kurento/modules/core/MediaObjectImpl.hpp:174
#8  0x00007f41f27abc68 in kurento::WebRtcEndpointImpl::onIceCandidate(char*, _KmsIceCandidate*) (this=0x7f3e6406f290, sessId=<optimized out>, candidate=<optimized out>) at /build/src/server/implementation/objects/WebRtcEndpointImpl.cpp:213
#9  0x00007f41f27a6d7e in kurento::signal_handler_adaptor<void, _GstElement*, char*, _KmsIceCandidate*>(gpointer) (__args#2=0x7f3f140d8d00 [KmsIceCandidate], __args#1=0x7f3df0e77c50 "kmswebrtcendpoint5683-sess0", __args#0=0x7f3de85b57e0 [KmsWebrtcEndpoint], this=<optimized out>)
    at /usr/include/c++/5/functional:2267
#10 0x00007f41f27a6d7e in kurento::signal_handler_adaptor<void, _GstElement*, char*, _KmsIceCandidate*>(gpointer) (data=<optimized out>) at /usr/include/kurento/modules/core/SignalHandler.hpp:54
#14 0x00007f41f7a7708f in <emit signal ??? on instance 0x7f3de85b57e0 [KmsWebrtcEndpoint]> (instance=<optimized out>, signal_id=<optimized out>, detail=<optimized out>) at /build/glib2.0-xkQkqE/glib2.0-2.48.2/./gobject/gsignal.c:3441
    #11 0x00007f41f7a5bfa5 in g_closure_invoke (closure=0x7f3e6406c410, return_value=return_value@entry=0x0, n_param_values=3, param_values=param_values@entry=0x7f3bdeb291b0, invocation_hint=invocation_hint@entry=0x7f3bdeb29130)
    at /build/glib2.0-xkQkqE/glib2.0-2.48.2/./gobject/gclosure.c:804
    #12 0x00007f41f7a6dfc1 in signal_emit_unlocked_R (node=node@entry=0x7f3fbc02e350, detail=detail@entry=0, instance=instance@entry=0x7f3de85b57e0, emission_return=emission_return@entry=0x0, instance_and_params=instance_and_params@entry=0x7f3bdeb291b0)
    at /build/glib2.0-xkQkqE/glib2.0-2.48.2/./gobject/gsignal.c:3629
    #13 0x00007f41f7a76d5c in g_signal_emit_valist (instance=<optimized out>, signal_id=<optimized out>, detail=<optimized out>, var_args=var_args@entry=0x7f3bdeb29390) at /build/glib2.0-xkQkqE/glib2.0-2.48.2/./gobject/gsignal.c:3385
#15 0x00007f41f7a5f0a7 in g_cclosure_marshal_VOID__OBJECTv (closure=0x7f3e64062840, return_value=<optimized out>, instance=<optimized out>, args=<optimized out>, marshal_data=0x0, n_params=<optimized out>, param_types=0x7f3fbc041db0)
    at /build/glib2.0-xkQkqE/glib2.0-2.48.2/./gobject/gmarshal.c:2102
#16 0x00007f41f7a5c1d4 in _g_closure_invoke_va (closure=closure@entry=0x7f3e64062840, return_value=return_value@entry=0x0, instance=instance@entry=0x7f3de86a6bc0, args=args@entry=0x7f3bdeb296d0, n_params=<optimized out>, param_types=0x7f3fbc041db0)
    at /build/glib2.0-xkQkqE/glib2.0-2.48.2/./gobject/gclosure.c:867
#17 0x00007f41f7a769a6 in g_signal_emit_valist (instance=0x7f3de86a6bc0, signal_id=<optimized out>, detail=0, var_args=var_args@entry=0x7f3bdeb296d0) at /build/glib2.0-xkQkqE/glib2.0-2.48.2/./gobject/gsignal.c:3294
#18 0x00007f41f7a7708f in g_signal_emit (instance=<optimized out>, signal_id=signal_id@entry=89, detail=detail@entry=0) at /build/glib2.0-xkQkqE/glib2.0-2.48.2/./gobject/gsignal.c:3441
#19 0x00007f41f24f1347 in kms_webrtc_session_new_candidate (cand=0x7f3ebc036160 [KmsIceCandidate], self=0x7f3de86a6bc0 [KmsWebrtcSession]) at /build/src/gst-plugins/webrtcendpoint/kmswebrtcsession.c:564
#20 0x00007f41f24f1347 in kms_webrtc_session_new_candidate (agent=<optimized out>, candidate=0x7f3ebc036160 [KmsIceCandidate], self=0x7f3de86a6bc0 [KmsWebrtcSession]) at /build/src/gst-plugins/webrtcendpoint/kmswebrtcsession.c:603
#21 0x00007f41f7a5f0a7 in g_cclosure_marshal_VOID__OBJECTv (closure=0x7f3e65aad820, return_value=<optimized out>, instance=<optimized out>, args=<optimized out>, marshal_data=0x0, n_params=<optimized out>, param_types=0x7f3fbc04bbf0)
    at /build/glib2.0-xkQkqE/glib2.0-2.48.2/./gobject/gmarshal.c:2102
#22 0x00007f41f7a5c122 in _g_closure_invoke_va (closure=closure@entry=0x7f3e65aad820, return_value=return_value@entry=0x0, instance=instance@entry=0x7f3e3c070120, args=args@entry=0x7f3bdeb29ac8, n_params=1, param_types=0x7f3fbc04bbf0)
    at /build/glib2.0-xkQkqE/glib2.0-2.48.2/./gobject/gclosure.c:867
#23 0x00007f41f7a769a6 in g_signal_emit_valist (instance=instance@entry=0x7f3e3c070120, signal_id=signal_id@entry=101, detail=detail@entry=0, var_args=var_args@entry=0x7f3bdeb29ac8) at /build/glib2.0-xkQkqE/glib2.0-2.48.2/./gobject/gsignal.c:3294
#24 0x00007f41f7a77555 in g_signal_emit_by_name (instance=instance@entry=0x7f3e3c070120, detailed_signal=detailed_signal@entry=0x7f41f24f92ef "on-ice-candidate") at /build/glib2.0-xkQkqE/glib2.0-2.48.2/./gobject/gsignal.c:3481
#25 0x00007f41f24f6da8 in kms_ice_nice_agent_new_candidate_full (agent=0x7f3f74089190 [NiceAgent], candidate=<optimized out>, self=0x7f3e3c070120 [KmsIceNiceAgent]) at /build/src/gst-plugins/webrtcendpoint/kmsiceniceagent.c:94
Python Exception <class 'gdb.error'> There is no member named v_pointer.:

@j1elo
Copy link
Member

j1elo commented Nov 24, 2021

@avssav are you using DataChannel?
I noticed that lines 3 and 4 mention the DataChannelClose method, so that could be a hint

@avssav
Copy link
Author

avssav commented Nov 25, 2021 •
edited

@j1elo Where can I see to use of the method? We connect to the kurento server through our node.js kurento-client. It was forked and rewritten from https://github.com/Kurento/kurento-tutorial-node/tree/master/kurento-one2one-call with using socket.io and ws npm modules and kurento-client-js and kurento-utils-js a few years ago.
Copy of our application is here https://github.com/avssav/novakid-kurento-client (app.js) There is 6.13.1 version, but it doesn't matter, because this client hasn't changed. Only the versions of the kureno server and modules npm in the package.json changed to newer

After that I update npm modules in our package.json, kurento-client-js and kurento-utils-js, Because they were changing from one version of kurento to another.
Our forks of kurento-client-js and kurento-utils-js are here https://github.com/d42f/kurento-client-bower/tree/branch-6.15.0 and https://github.com/d42f/kurento-utils-bower/tree/branch-6.15.0

We are now using the standard docker kms image 6.15.0 (there are plans to upgrade to 6.16.0 now because kurento-client-js and kurento-utils-js have some upgrades)

@j1elo
Copy link
Member

j1elo commented Nov 25, 2021

It doesn't look that you are using DataChannels in https://github.com/avssav/novakid-kurento-client

So the fact that Kurento stack trace points to some DataChannel methods, could indicate a memory corruption bug, that ends up with a crash.

Are you able to identify if the crash happens with some scenario? Like:

  • When some specific client connects to the session.
  • When someone from a geographic area connects.
  • When there are only a couple users, or when there are lots of users.
  • When the server is very loaded, or if load doesn't matter.

So I can try to replicate your conditions with the kurento-one2one-call, to see if I can also get the same crash which would help me investigate the issue.

@avssav
Copy link
Author

avssav commented Nov 29, 2021

There a lot of one2one pair of clients connected (10-70 pair). Different geographic areas. Load doesn't matter. This does not happen often

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
None yet
Projects
Triage
Backlog Candidates
Milestone
No milestone
Development

No branches or pull requests
2 participants
@j1elo @avssav