Skip to content
This repository has been archived by the owner on Jul 12, 2023. It is now read-only.

crash on libnice( priv_mark_pair_nominated ) #505

Open
4 tasks done
darrenhp opened this issue Jul 25, 2020 · 12 comments
Open
4 tasks done

crash on libnice( priv_mark_pair_nominated ) #505

darrenhp opened this issue Jul 25, 2020 · 12 comments
Assignees
@j1elo
Labels
kms Related especifically to the media server libnice stability Crashes or critical bug that affects stability

Comments

@darrenhp
Copy link

darrenhp commented Jul 25, 2020 •
edited
Loading

Prerequisites

  • I have read the SUPPORT document

  • I have checked the Troubleshooting Guide

  • I have tested with the latest version of Kurento

Issue description

crash

the errors.log is

Aborted (thread 139709364483840, pid 1144)
Stack trace:
[__GI_raise]
sysdeps/unix/sysv/linux/raise.c:51 (discriminator 3)
[__GI_abort]
/build/glibc-2ORdQG/glibc-2.27/stdlib/abort.c:81
[g_assertion_message]
glib/gtestutils.c:2532
[g_assertion_message_cmpnum]
glib/gtestutils.c:2590
[priv_mark_pair_nominated]
./agent/conncheck.c:2272 (discriminator 1)
[conn_check_handle_inbound_stun]
./agent/conncheck.c:4793
[agent_recv_message_unlocked]
./agent/agent.c:4287
[component_io_cb]
./agent/agent.c:5611
[socket_source_dispatch]
gio/gsocket.c:3843
[g_main_dispatch]
glib/gmain.c:3180
[g_main_context_iterate]
glib/gmain.c:3902
[g_main_loop_run]
glib/gmain.c:4097
[gst_nice_src_create]
./gst/gstnicesrc.c:292
[gst_base_src_get_range]
./libs/gst/base/gstbasesrc.c:2460
[gst_base_src_loop]
./libs/gst/base/gstbasesrc.c:2737
[gst_task_func]
./gst/gsttask.c:346

the gdb

(gdb) bt
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
#1  0x00007f10fd7508b1 in __GI_abort () at abort.c:79
#2  0x0000557c43f4a9ed in Debug::safe_abort () at ./server/death_handler.cpp:299
#3  Debug::DeathHandler::SignalHandler (sig=<optimized out>, secret=<optimized out>) at ./server/death_handler.cpp:600
#4  <signal handler called>
#5  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
#6  0x00007f10fd7508b1 in __GI_abort () at abort.c:79
#7  0x00007f10fedb8315 in g_assertion_message (domain=domain@entry=0x7f10f8ef336a "libnice", file=file@entry=0x7f10f8ef59f5 "conncheck.c", line=line@entry=2272,
    func=func@entry=0x7f10f8ef5b80 <__func__.28924> "priv_mark_pair_nominated", message=message@entry=0x7f10b000e830 "assertion failed (pair->state == NICE_CHECK_DISCOVERED): (4 == 6)") at ../../../../glib/gtestutils.c:2532
#8  0x00007f10fedb86c2 in g_assertion_message_cmpnum (domain=domain@entry=0x7f10f8ef336a "libnice", file=file@entry=0x7f10f8ef59f5 "conncheck.c", line=line@entry=2272,
    func=func@entry=0x7f10f8ef5b80 <__func__.28924> "priv_mark_pair_nominated", expr=expr@entry=0x7f10f8ef5af8 "pair->state == NICE_CHECK_DISCOVERED", arg1=4, cmp=cmp@entry=0x7f10f8ef3eab "==", arg2=6, numtype=105 'i')
    at ../../../../glib/gtestutils.c:2588
#9  0x00007f10f8ed9ae8 in priv_mark_pair_nominated (agent=agent@entry=0x7f10b8034700, stream=stream@entry=0x7f10c0077a10, component=component@entry=0x7f10c0132fd0, localcand=localcand@entry=0x7f10d81e3320,
    remotecand=remotecand@entry=0x7f10d02f80b0) at conncheck.c:2272
#10 0x00007f10f8edc7b5 in priv_reply_to_conn_check (use_candidate=1, msg=0x7f109f099550, rbuf_len=<optimized out>, sockptr=0x7f10d4252b50, toaddr=0x7f109f099e20, rcand=0x7f10d02f80b0, lcand=0x7f10d81e3320,
    component=0x7f10c0132fd0, stream=0x7f10c0077a10, agent=0x7f10b8034700) at conncheck.c:3268
#11 conn_check_handle_inbound_stun (agent=agent@entry=0x7f10b8034700, stream=stream@entry=0x7f10c0077a10, component=component@entry=0x7f10c0132fd0, nicesock=0x7f10d4252b50, from=0x7f109f099e20, buf=buf@entry=0x7f10b00d8730 "",
    len=100) at conncheck.c:4790
#12 0x00007f10f8ed55ba in agent_recv_message_unlocked (agent=agent@entry=0x7f10b8034700, stream=stream@entry=0x7f10c0077a10, component=component@entry=0x7f10c0132fd0, nicesock=<optimized out>, message=message@entry=0x7f109f099eb0)
    at agent.c:4283
#13 0x00007f10f8ed58d5 in component_io_cb (gsocket=<optimized out>, condition=<optimized out>, user_data=0x557c4500d0e0) at agent.c:5608
#14 0x00007f10fbcc80f9 in socket_source_dispatch (source=source@entry=0x7f10b00bcb30, callback=0x7f10f8ed5650 <component_io_cb>, user_data=0x557c4500d0e0) at ../../../../gio/gsocket.c:3843
#15 0x00007f10fed91285 in g_main_dispatch (context=0x7f10c0286c50) at ../../../../glib/gmain.c:3176
#16 g_main_context_dispatch (context=context@entry=0x7f10c0286c50) at ../../../../glib/gmain.c:3829
#17 0x00007f10fed91650 in g_main_context_iterate (context=0x7f10c0286c50, block=block@entry=1, dispatch=dispatch@entry=1, self=<optimized out>) at ../../../../glib/gmain.c:3902
#18 0x00007f10fed91962 in g_main_loop_run (loop=0x7f10c0286d90) at ../../../../glib/gmain.c:4098
#19 0x00007f10bde55ea4 in gst_nice_src_create (basesrc=<optimized out>, buffer=0x7f109f0aa168) at gstnicesrc.c:291
#20 0x00007f10fb5b2ea9 in gst_base_src_get_range (src=src@entry=0x7f10e40dbbb0, offset=offset@entry=18446744073709551615, length=<optimized out>, buf=buf@entry=0x7f109f0aa248) at gstbasesrc.c:2460
#21 0x00007f10fb5b4b93 in gst_base_src_loop (pad=0x7f10cc103470) at gstbasesrc.c:2736
#22 0x00007f10ff350a10 in gst_task_func (task=0x7f10dc2483b0) at gsttask.c:344
#23 0x00007f10fedb9b40 in g_thread_pool_thread_proxy (data=<optimized out>) at ../../../../glib/gthreadpool.c:307
#24 0x00007f10fedb9175 in g_thread_proxy (data=0x7f10e42a80a0) at ../../../../glib/gthread.c:784
#25 0x00007f10ffe8d6db in start_thread (arg=0x7f109f0ab700) at pthread_create.c:463
#26 0x00007f10fd831a3f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Context

How to reproduce?

Expected & current behavior

(Optional) Possible solution

INFO about Kurento Media Server

  • Kurento version: 6.14.0
  • Server OS: 18.04 (Bionic)
  • Installation method:
    • apt-get
    • [] Docker
    • [] AWS
    • [] Built from sources

INFO about your Application Server

  • Language: Java
  • Kurento Client version:

INFO about end-user clients

  • Device(s):
  • OS(es):
  • Browser(s):

INFO about your environment

Commands output

cat /etc/lsb-release

kurento-media-server --version

dpkg -l | grep -Pi 'kurento|kms-|gst.*1.5|nice'

$ cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=18.04
DISTRIB_CODENAME=bionic
DISTRIB_DESCRIPTION="Ubuntu 18.04.4 LTS"

$ kurento-media-server --version
Kurento Media Server version: 6.14.0
Found modules:
        'core' version 6.14.0
        'elements' version 6.14.0
        'filters' version 6.14.0

$ dpkg -l | grep -Pi 'kurento|kms-|gst.*1.5|nice'
ii  gstreamer1.5-alsa:amd64                1.8.1-1kurento2.18.04                           amd64        GStreamer plugin for ALSA
ii  gstreamer1.5-libav:amd64               1.8.1-1kurento1.18.04                           amd64        libav plugin for GStreamer
ii  gstreamer1.5-libav-dbg:amd64           1.8.1-1kurento1.18.04                           amd64        libav plugin for GStreamer (debug symbols)
ii  gstreamer1.5-nice:amd64                0.1.17-0kurento1.18.04                          amd64        ICE library (GStreamer 1.5 plugin)
ii  gstreamer1.5-nice-dbgsym:amd64         0.1.17-0kurento1.18.04                          amd64        debug symbols for gstreamer1.5-nice
ii  gstreamer1.5-plugins-bad:amd64         1.8.1-1kurento4.18.04                           amd64        GStreamer plugins from the "bad" set
ii  gstreamer1.5-plugins-bad-dbg:amd64     1.8.1-1kurento4.18.04                           amd64        GStreamer plugins from the "bad" set (debug symbols)
ii  gstreamer1.5-plugins-base:amd64        1.8.1-1kurento2.18.04                           amd64        GStreamer plugins from the "base" set
ii  gstreamer1.5-plugins-base-dbg:amd64    1.8.1-1kurento2.18.04                           amd64        GStreamer plugins from the "base" set
ii  gstreamer1.5-plugins-good:amd64        1.8.1-1kurento4.18.04                           amd64        GStreamer plugins from the "good" set
ii  gstreamer1.5-plugins-good-dbg:amd64    1.8.1-1kurento4.18.04                           amd64        GStreamer plugins from the "good" set
ii  gstreamer1.5-plugins-ugly:amd64        1.8.1-1kurento1.18.04                           amd64        GStreamer plugins from the "ugly" set
ii  gstreamer1.5-plugins-ugly-dbg:amd64    1.8.1-1kurento1.18.04                           amd64        GStreamer plugins from the "ugly" set (debug symbols)
ii  gstreamer1.5-pulseaudio:amd64          1.8.1-1kurento4.18.04                           amd64        GStreamer plugin for PulseAudio
ii  gstreamer1.5-x:amd64                   1.8.1-1kurento2.18.04                           amd64        GStreamer plugins for X11 and Pango
ii  kms-core                               6.14.0-0kurento1.18.04                          amd64        Kurento Core module
ii  kms-core-dbg                           6.14.0-0kurento1.18.04                          amd64        Kurento Core module
ii  kms-elements                           6.14.0-0kurento1.18.04                          amd64        Kurento Elements module
ii  kms-elements-dbg                       6.14.0-0kurento1.18.04                          amd64        Kurento Elements module
ii  kms-filters                            6.14.0-0kurento1.18.04                          amd64        Kurento Filters module
ii  kms-filters-dbg                        6.14.0-0kurento1.18.04                          amd64        Kurento Filters module
ii  kms-jsonrpc                            6.14.0-0kurento1.18.04                          amd64        Kurento JSON-RPC library
ii  kms-jsonrpc-dbg                        6.14.0-0kurento1.18.04                          amd64        Kurento JSON-RPC library
ii  kmsjsoncpp                             1.6.3-1kurento1.18.04                           amd64        Kurento jsoncpp library
ii  kmsjsoncpp-dbg                         1.6.3-1kurento1.18.04                           amd64        Kurento jsoncpp library
ii  kurento-dbg                            6.14.0-0kurento1.18.04                          amd64        Meta-package that installs debug symbols
ii  kurento-media-server                   6.14.0-0kurento1.18.04                          amd64        Kurento Media Server
ii  kurento-media-server-dbg               6.14.0-0kurento1.18.04                          amd64        Kurento Media Server
ii  libgstreamer-plugins-bad1.5-0:amd64    1.8.1-1kurento4.18.04                           amd64        GStreamer development files for libraries from the "bad" set
ii  libgstreamer-plugins-base1.5-0:amd64   1.8.1-1kurento2.18.04                           amd64        GStreamer libraries from the "base" set
ii  libgstreamer1.5-0:amd64                1.8.1-1kurento2.18.04                           amd64        Core GStreamer libraries and elements
ii  libgstreamer1.5-0-dbg:amd64            1.8.1-1kurento2.18.04                           amd64        Core GStreamer libraries and elements
ii  libnice10:amd64                        0.1.17-0kurento1.18.04                          amd64        ICE library (shared library)
ii  libnice10-dbgsym:amd64                 0.1.17-0kurento1.18.04                          amd64        debug symbols for libnice10
ii  libsrtp0:amd64                         1.6.0-0kurento1.18.04                           amd64        Secure RTP (SRTP) and UST Reference Implementations - shared library
ii  libusrsctp                             0.9.2-1kurento1.18.04                           amd64        sctp-refimpl library
ii  openh264                               1.4.0-1kurento1.18.04                           amd64        OpenH264 library
ii  openh264-gst-plugins-bad-1.5:amd64     1.8.1-1kurento4.18.04                           amd64        GStreamer plugins from openh264
ii  openwebrtc-gst-plugins                 0.10.0-1kurento1.18.04                          amd64        OpenWebRTC specific GStreamer plugins
ii  openwebrtc-gst-plugins-dbg             0.10.0-1kurento1.18.04                          amd64        OpenWebRTC specific GStreamer plugins
@j1elo j1elo self-assigned this Aug 3, 2020
@j1elo j1elo added the stability Crashes or critical bug that affects stability label Aug 3, 2020
@j1elo j1elo added the kms Related especifically to the media server label Oct 4, 2020
@j1elo j1elo added the libnice label Oct 4, 2020
@darrenhp
Copy link
Author

It looks the same as https://gitlab.freedesktop.org/libnice/libnice/-/issues/95 (fixed);
But this issue show it's not fixed;

@sshsun1990
Copy link

I have the same problem. Now I modify the source code of libnice

// g_assert (pair->state == NICE_CHECK_DISCOVERED);
        if (pair->state != NICE_CHECK_DISCOVERED) {
          nice_debug ("pair->state:%d replace g_assert", pair->state);  
          return ;
        }

@j1elo
Copy link
Member

j1elo commented Oct 27, 2020

You can build with -DG_DISABLE_ASSERT so the GLib asserts are disabled on runtime. This is what we're doing in current builds of libnice: https://github.com/Kurento/libnice/blob/ubuntu/xenial/debian/rules#L18

The packages that are right now in the nightly repository are a Release Candidate of Kurento 6.15, so if you still have this issue you can try with this newer version and help us confirm if the issue is resolved:

https://doc-kurento.readthedocs.io/en/latest/user/installation_dev.html#kurento-media-server

@darrenhp
Copy link
Author

You can build with -DG_DISABLE_ASSERT so the GLib asserts are disabled on runtime. This is what we're doing in current builds of libnice: https://github.com/Kurento/libnice/blob/ubuntu/xenial/debian/rules#L18

The packages that are right now in the nightly repository are a Release Candidate of Kurento 6.15, so if you still have this issue you can try with this newer version and help us confirm if the issue is resolved:

https://doc-kurento.readthedocs.io/en/latest/user/installation_dev.html#kurento-media-server

It happend again on Kurento 6.15;

libnice:ERROR:../../agent/conncheck.c:2248:priv_mark_pair_nominated: assertion failed (pair->state == NICE_CHECK_DISCOVERED): (4 == 6)
Aborted (thread 140254237128448, pid 1250)
Stack trace:
[__GI_raise]
sysdeps/unix/sysv/linux/raise.c:51 (discriminator 3)
[__GI_abort]
/build/glibc-S7xCS9/glibc-2.27/stdlib/abort.c:81
[g_assertion_message]
glib/gtestutils.c:2532
[g_assertion_message_cmpnum]
glib/gtestutils.c:2590
[priv_mark_pair_nominated]
agent/conncheck.c:2248 (discriminator 1)
[priv_reply_to_conn_check]
agent/conncheck.c:3248
[agent_recv_message_unlocked]
agent/agent.c:4339
[component_io_cb]
agent/agent.c:5662
[socket_source_dispatch]
gio/gsocket.c:3843
[g_main_dispatch]
glib/gmain.c:3180
[g_main_context_iterate]
glib/gmain.c:3902
[g_main_loop_run]
glib/gmain.c:4097
[gst_nice_src_create]
gst/gstnicesrc.c:292
[gst_base_src_get_range]
./libs/gst/base/gstbasesrc.c:2460
[gst_base_src_loop]
./libs/gst/base/gstbasesrc.c:2737
[gst_task_func]
./gst/gsttask.c:346

Below is my install infomation;

$ kurento-media-server -v
Kurento Media Server version: 6.15.0
Found modules:
	'core' version 6.15.0
	'elements' version 6.15.0
	'filters' version 6.15.0


$ dpkg -l | grep -Pi 'kurento|kms-|gst.*1.5|nice'
ii  gstreamer1.5-alsa:amd64                1.8.1-1kurento2.18.04                           amd64        GStreamer plugin for ALSA
ii  gstreamer1.5-libav:amd64               1.8.1-1kurento1.18.04                           amd64        libav plugin for GStreamer
ii  gstreamer1.5-libav-dbg:amd64           1.8.1-1kurento1.18.04                           amd64        libav plugin for GStreamer (debug symbols)
ii  gstreamer1.5-nice:amd64                0.1.18-0kurento1.18.04                          amd64        ICE library (GStreamer 1.5 plugin)
ii  gstreamer1.5-nice-dbgsym:amd64         0.1.18-0kurento1.18.04                          amd64        debug symbols for gstreamer1.5-nice
ii  gstreamer1.5-plugins-bad:amd64         1.8.1-1kurento5.18.04                           amd64        GStreamer plugins from the "bad" set
ii  gstreamer1.5-plugins-bad-dbg:amd64     1.8.1-1kurento5.18.04                           amd64        GStreamer plugins from the "bad" set (debug symbols)
ii  gstreamer1.5-plugins-base:amd64        1.8.1-1kurento2.18.04                           amd64        GStreamer plugins from the "base" set
ii  gstreamer1.5-plugins-base-dbg:amd64    1.8.1-1kurento2.18.04                           amd64        GStreamer plugins from the "base" set
ii  gstreamer1.5-plugins-good:amd64        1.8.1-1kurento5.18.04                           amd64        GStreamer plugins from the "good" set
ii  gstreamer1.5-plugins-good-dbg:amd64    1.8.1-1kurento5.18.04                           amd64        GStreamer plugins from the "good" set
ii  gstreamer1.5-plugins-ugly:amd64        1.8.1-1kurento1.18.04                           amd64        GStreamer plugins from the "ugly" set
ii  gstreamer1.5-plugins-ugly-dbg:amd64    1.8.1-1kurento1.18.04                           amd64        GStreamer plugins from the "ugly" set (debug symbols)
ii  gstreamer1.5-pulseaudio:amd64          1.8.1-1kurento5.18.04                           amd64        GStreamer plugin for PulseAudio
ii  gstreamer1.5-x:amd64                   1.8.1-1kurento2.18.04                           amd64        GStreamer plugins for X11 and Pango
ii  kms-core                               6.15.0-0kurento1.18.04                          amd64        Kurento Core module
ii  kms-core-dbg                           6.15.0-0kurento1.18.04                          amd64        Kurento Core module
ii  kms-elements                           6.15.0-0kurento1.18.04                          amd64        Kurento Elements module
ii  kms-elements-dbg                       6.15.0-0kurento1.18.04                          amd64        Kurento Elements module
ii  kms-filters                            6.15.0-0kurento1.18.04                          amd64        Kurento Filters module
ii  kms-filters-dbg                        6.15.0-0kurento1.18.04                          amd64        Kurento Filters module
ii  kms-jsonrpc                            6.15.0-0kurento1.18.04                          amd64        Kurento JSON-RPC library
ii  kms-jsonrpc-dbg                        6.15.0-0kurento1.18.04                          amd64        Kurento JSON-RPC library
ii  kmsjsoncpp                             1.6.3-1kurento1.18.04                           amd64        Kurento jsoncpp library
ii  kmsjsoncpp-dbg                         1.6.3-1kurento1.18.04                           amd64        Kurento jsoncpp library
ii  kurento-dbg                            6.15.0-0kurento1.18.04                          amd64        Meta-package that installs debug symbols
ii  kurento-media-server                   6.15.0-0kurento1.18.04                          amd64        Kurento Media Server
ii  kurento-media-server-dbg               6.15.0-0kurento1.18.04                          amd64        Kurento Media Server
ii  libgstreamer-plugins-bad1.5-0:amd64    1.8.1-1kurento5.18.04                           amd64        GStreamer development files for libraries from the "bad" set
ii  libgstreamer-plugins-base1.5-0:amd64   1.8.1-1kurento2.18.04                           amd64        GStreamer libraries from the "base" set
ii  libgstreamer1.5-0:amd64                1.8.1-1kurento2.18.04                           amd64        Core GStreamer libraries and elements
ii  libgstreamer1.5-0-dbg:amd64            1.8.1-1kurento2.18.04                           amd64        Core GStreamer libraries and elements
ii  libnice10:amd64                        0.1.18-0kurento1.18.04                          amd64        ICE library (shared library)
ii  libnice10-dbgsym:amd64                 0.1.18-0kurento1.18.04                          amd64        debug symbols for libnice10
ii  libsrtp0:amd64                         1.6.0-0kurento1.18.04                           amd64        Secure RTP (SRTP) and UST Reference Implementations - shared library
ii  libusrsctp                             0.9.2-1kurento1.18.04                           amd64        sctp-refimpl library
ii  openh264                               1.5.0-0kurento1.18.04                           amd64        H.264 Video Codec provided by Cisco Systems, Inc.
ii  openh264-gst-plugins-bad-1.5:amd64     1.8.1-1kurento5.18.04                           amd64        GStreamer plugins from openh264
ii  openwebrtc-gst-plugins                 0.10.0-1kurento1.18.04                          amd64        OpenWebRTC specific GStreamer plugins
ii  openwebrtc-gst-plugins-dbg             0.10.0-1kurento1.18.04                          amd64        OpenWebRTC specific GStreamer plugins

@j1elo
Copy link
Member

j1elo commented Nov 17, 2020

On one hand, the issue is that libnice is asserting the state "DISCOVERED", but the actual state if "FAILED".
On the other, a recent change in libnice doesn't allow me to disable those assertions in libnice, thus making the whole media server to crash, instead of simply failing one connection.

Both issues have been reported so we can work with them and find a solution:

@j1elo
Copy link
Member

j1elo commented Nov 18, 2020

Hi @darrenhp, the libnice dev has replied in our bug report and asked about a libnice debug log to confirm if their solution will actually solve the problem.

Are you able to repeat the issue? If so, please enable libnice debug logs by (assuming a local apt-get installation) adding these lines at the end of /etc/default/kurento-media-server:

export G_MESSAGES_DEBUG="libnice,libnice-stun"
export GST_DEBUG="$GST_DEBUG,glib:5"

@darrenhp
Copy link
Author

Hi @darrenhp, the libnice dev has replied in our bug report and asked about a libnice debug log to confirm if their solution will actually solve the problem.

Are you able to repeat the issue? If so, please enable libnice debug logs by (assuming a local apt-get installation) adding these lines at the end of /etc/default/kurento-media-server:

export G_MESSAGES_DEBUG="libnice,libnice-stun"
export GST_DEBUG="$GST_DEBUG,glib:5"

I have no idea how to reproduce it; But it happen several times every week on the production environment;
I will enable the debug logs this week;

@darrenhp
Copy link
Author

darrenhp commented Dec 14, 2020 •
edited
Loading

I hope it help.
the kms progress started at 2020-11-11 and crashed at 2020-12-11 (and then it restarted);
Crash log;
2020-12-11T000014.00030.pid1251.log

coredump ，https://www.dropbox.com/s/7avbn9wdpewfpb3/CoreDump.zip?dl=0

2020-11-11T02:02:03+08:00 -- New execution
Error loading config: /etc/kurento/modules/kurento/kms_rsa.pem, Unknown file type: kms_rsa.pem
Error loading config: /etc/kurento/modules/kurento/kms_ecdsa.pem, Unknown file type: kms_ecdsa.pem
Error loading config: /etc/kurento/modules/kurento/WebRtcEndpoint.conf.ini.dpkg-dist, Unknown file type: WebRtcEndpoint.conf.ini.dpkg-dist
**
libnice:ERROR:../../agent/conncheck.c:2248:priv_mark_pair_nominated: assertion failed (pair->state == NICE_CHECK_DISCOVERED): (4 == 6)
Aborted (thread 140032120387328, pid 1251)
Stack trace:
[__GI_raise]
sysdeps/unix/sysv/linux/raise.c:51 (discriminator 3)
[__GI_abort]
/build/glibc-S7xCS9/glibc-2.27/stdlib/abort.c:81
[g_assertion_message]
glib/gtestutils.c:2532
[g_assertion_message_cmpnum]
glib/gtestutils.c:2590
[priv_mark_pair_nominated]
agent/conncheck.c:2248 (discriminator 1)
[priv_reply_to_conn_check]
agent/conncheck.c:3248
[agent_recv_message_unlocked]
agent/agent.c:4339
[component_io_cb]
agent/agent.c:5662
[socket_source_dispatch]
gio/gsocket.c:3843
[g_main_dispatch]
glib/gmain.c:3180
[g_main_context_iterate]
glib/gmain.c:3902
[g_main_loop_run]
glib/gmain.c:4097
[gst_nice_src_create]
gst/gstnicesrc.c:292
[gst_base_src_get_range]
./libs/gst/base/gstbasesrc.c:2460
[gst_base_src_loop]
./libs/gst/base/gstbasesrc.c:2737
[gst_task_func]
./gst/gsttask.c:346

2020-12-11T17:21:02+08:00 -- New execution
Error loading config: /etc/kurento/modules/kurento/kms_rsa.pem, Unknown file type: kms_rsa.pem
Error loading config: /etc/kurento/modules/kurento/kms_ecdsa.pem, Unknown file type: kms_ecdsa.pem
Error loading config: /etc/kurento/modules/kurento/WebRtcEndpoint.conf.ini.dpkg-dist, Unknown file type: WebRtcEndpoint.conf.ini.dpkg-dist

@j1elo
Copy link
Member

j1elo commented Dec 14, 2020

Crash log;
2020-12-11T000014.00030.pid1251.log

Hi @darrenhp, I'm afraid your logs don't contain any of the advanced libnice debug messages.

Please review my previous comment where I explained how to enable them:

please enable libnice debug logs by (assuming a local apt-get installation) adding these lines at the end of /etc/default/kurento-media-server:

export G_MESSAGES_DEBUG="libnice,libnice-stun"
export GST_DEBUG="$GST_DEBUG,glib:5"

For completeness, this is a summary of previous messages up until now, with all you need to know in order to provide the requested logs:

  • Please use Kurento 6.15 (6.14 will not work to get libnice logs)

  • Add the requested lines at the end of /etc/default/kurento-media-server

  • Restart the media server so the changes are applied (for example with this command: sudo service kurento-media-server restart or equivalent in your system)

  • Check that logs in /var/log/kurento-media-server/ contain lines like this when doing WebRTC:

2020-12-14T10:56:29,973966 15847 0x00007f43fb7fe700   debug glib                      libnice:0 ()  Agent 0x7f4420030040 : scheduling triggered check with socket=0x7f43f00044f0 and remote cand=0x7f443000bdc0.
2020-12-14T10:56:29,974029 15847 0x00007f43fb7fe700   debug glib                      libnice:0 ()  Agent 0x7f4420030040 : Found a matching pair 0x7f43f0005380 (1:1493399139) (SUCCEEDED) ...
2020-12-14T10:56:29,974057 15847 0x00007f43fb7fe700   debug glib                      libnice:0 ()  Agent 0x7f4420030040 : nothing to do for pair 0x7f43f0005380.
2020-12-14T10:56:29,974079 15847 0x00007f43fb7fe700   debug glib                      libnice:0 ()  Agent 0x7f4420030040 : marking pair 0x7f43f0005380 (1:1493399139) as nominated
2020-12-14T10:56:29,974106 15847 0x00007f43fb7fe700   debug glib                      libnice:0 ()  Agent 0x7f4420030040: Pruning pending checks for s1/c1
2020-12-14T10:56:29,974130 15847 0x00007f43fb7fe700   debug glib                      libnice:0 ()  Agent 0x7f4420030040 : selected pair priority is 782000ff:7e7f1eff:1
2020-12-14T10:56:29,974153 15847 0x00007f43fb7fe700   debug glib                      libnice:0 ()  Agent 0x7f4420030040 : conn.check list status: 1 nominated, 1 valid, c-id 1.
2020-12-14T10:56:29,974175 15847 0x00007f43fb7fe700   debug glib                      libnice:0 ()  agent_recv_message_unlocked: Valid STUN packet received.
2020-12-14T10:56:32,474354 15847 0x00007f43fb7fe700   debug glib                      libnice:0 ()  Agent 0x7f4420030040: inbound STUN packet for 1/1 (stream/component) from [192.168.1.12]:58669 (100 octets) :
2020-12-14T10:56:32,474443 15847 0x00007f43fb7fe700   debug glib                      libnice-stun:0 ()  STUN demux: OK!
2020-12-14T10:56:32,474477 15847 0x00007f43fb7fe700   debug glib                      libnice-stun:0 ()  Comparing username/ufrag of len 9 and 4, equal=0
2020-12-14T10:56:32,474503 15847 0x00007f43fb7fe700   debug glib                      libnice-stun:0 ()    username: 0x726d53333a6e564e39
2020-12-14T10:56:32,474525 15847 0x00007f43fb7fe700   debug glib                      libnice-stun:0 ()    ufrag:    0x726d5333
2020-12-14T10:56:32,474546 15847 0x00007f43fb7fe700   debug glib                      libnice-stun:0 ()  Found valid username, returning password: 'SD25M6caIpbZTk83+3tkBl'
2020-12-14T10:56:32,474578 15847 0x00007f43fb7fe700   debug glib                      libnice-stun:0 ()   Message HMAC-SHA1 fingerprint:
2020-12-14T10:56:32,474600 15847 0x00007f43fb7fe700   debug glib                      libnice-stun:0 ()    key     : 0x534432354d3663614970625a546b38332b33746b426c
2020-12-14T10:56:32,474620 15847 0x00007f43fb7fe700   debug glib                      libnice-stun:0 ()    expected: 0xf42fe67e80699ed53705529d8b3b10a7d4286f05
2020-12-14T10:56:32,474639 15847 0x00007f43fb7fe700   debug glib                      libnice-stun:0 ()    received: 0xf42fe67e80699ed53705529d8b3b10a7d4286f05
2020-12-14T10:56:32,474658 15847 0x00007f43fb7fe700   debug glib                      libnice-stun:0 ()  STUN auth: OK!
2020-12-14T10:56:32,474683 15847 0x00007f43fb7fe700   debug glib                      libnice-stun:0 ()  STUN unknown: 0 mandatory attribute(s)!
2020-12-14T10:56:32,474707 15847 0x00007f43fb7fe700   debug glib                      libnice-stun:0 ()  STUN Reply (buffer size = 1300)...
2020-12-14T10:56:32,474734 15847 0x00007f43fb7fe700   debug glib                      libnice-stun:0 ()   Message HMAC-SHA1 message integrity:
2020-12-14T10:56:32,474757 15847 0x00007f43fb7fe700   debug glib                      libnice-stun:0 ()    key     : 0x534432354d3663614970625a546b38332b33746b426c
2020-12-14T10:56:32,474776 15847 0x00007f43fb7fe700   debug glib                      libnice-stun:0 ()    sent    : 0x8da6620352677f9f5e396e65ae0d2240eb2441d4
2020-12-14T10:56:32,474807 15847 0x00007f43fb7fe700   debug glib                      libnice-stun:0 ()   Message HMAC-SHA1 fingerprint: 0xc22a17ca
2020-12-14T10:56:32,474829 15847 0x00007f43fb7fe700   debug glib                      libnice-stun:0 ()   All done (response size: 80)
2020-12-14T10:56:32,474856 15847 0x00007f43fb7fe700   debug glib                      libnice:0 ()  Agent 0x7f4420030040 : STUN-CC RESP to '192.168.1.12:58669', socket=23, len=80, cand=0x7f443000bdc0 (c-id:1), u

Watch out with the size of logs, it will grow very fast and could take a lot of space. Kurento will by default do log rotation to keep all files under 100 MB, but still, keep an eye on this.

If a crash happens with the libnice debug logs enabled, we can send the logs to the libnice devs and help them find and fix the issue. Thanks for your help!

@darrenhp
Copy link
Author

the last log file;
2021-02-24T191822.00314.pid434.log.tar.gz
image

2021-02-03T21:36:02+08:00 -- New execution
Error loading config: /etc/kurento/modules/kurento/kms_rsa.pem, Unknown file type: kms_rsa.pem
Error loading config: /etc/kurento/modules/kurento/kms_ecdsa.pem, Unknown file type: kms_ecdsa.pem
Error loading config: /etc/kurento/modules/kurento/WebRtcEndpoint.conf.ini.dpkg-dist, Unknown file type: WebRtcEndpoint.conf.ini.dpkg-dist
**
libnice:ERROR:../../agent/conncheck.c:2248:priv_mark_pair_nominated: assertion failed (pair->state == NICE_CHECK_DISCOVERED): (4 == 6)
Aborted (thread 139781230229248, pid 434)
Stack trace:
[__GI_raise]
sysdeps/unix/sysv/linux/raise.c:51 (discriminator 3)
[__GI_abort]
/build/glibc-S7xCS9/glibc-2.27/stdlib/abort.c:81
[g_assertion_message]
glib/gtestutils.c:2532
[g_assertion_message_cmpnum]
glib/gtestutils.c:2590
[priv_mark_pair_nominated]
agent/conncheck.c:2248 (discriminator 1)
[priv_reply_to_conn_check]
agent/conncheck.c:3248
[agent_recv_message_unlocked]
agent/agent.c:4339
[component_io_cb]
agent/agent.c:5662
[socket_source_dispatch]
gio/gsocket.c:3843
[g_main_dispatch]
glib/gmain.c:3180
[g_main_context_iterate]
glib/gmain.c:3902
[g_main_loop_run]
glib/gmain.c:4097
[gst_nice_src_create]
gst/gstnicesrc.c:292
[gst_base_src_get_range]
./libs/gst/base/gstbasesrc.c:2460
[gst_base_src_loop]
./libs/gst/base/gstbasesrc.c:2737
[gst_task_func]
./gst/gsttask.c:346


2021-02-24T19:23:02+08:00 -- New execution
Error loading config: /etc/kurento/modules/kurento/kms_rsa.pem, Unknown file type: kms_rsa.pem
Error loading config: /etc/kurento/modules/kurento/kms_ecdsa.pem, Unknown file type: kms_ecdsa.pem
Error loading config: /etc/kurento/modules/kurento/WebRtcEndpoint.conf.ini.dpkg-dist, Unknown file type: WebRtcEndpoint.conf.ini.dpkg-dist

@darrenhp
Copy link
Author

darrenhp commented Mar 3, 2021

the last log file;
2021-02-24T191822.00314.pid434.log.tar.gz
image

2021-02-03T21:36:02+08:00 -- New execution
Error loading config: /etc/kurento/modules/kurento/kms_rsa.pem, Unknown file type: kms_rsa.pem
Error loading config: /etc/kurento/modules/kurento/kms_ecdsa.pem, Unknown file type: kms_ecdsa.pem
Error loading config: /etc/kurento/modules/kurento/WebRtcEndpoint.conf.ini.dpkg-dist, Unknown file type: WebRtcEndpoint.conf.ini.dpkg-dist
**
libnice:ERROR:../../agent/conncheck.c:2248:priv_mark_pair_nominated: assertion failed (pair->state == NICE_CHECK_DISCOVERED): (4 == 6)
Aborted (thread 139781230229248, pid 434)
Stack trace:
[__GI_raise]
sysdeps/unix/sysv/linux/raise.c:51 (discriminator 3)
[__GI_abort]
/build/glibc-S7xCS9/glibc-2.27/stdlib/abort.c:81
[g_assertion_message]
glib/gtestutils.c:2532
[g_assertion_message_cmpnum]
glib/gtestutils.c:2590
[priv_mark_pair_nominated]
agent/conncheck.c:2248 (discriminator 1)
[priv_reply_to_conn_check]
agent/conncheck.c:3248
[agent_recv_message_unlocked]
agent/agent.c:4339
[component_io_cb]
agent/agent.c:5662
[socket_source_dispatch]
gio/gsocket.c:3843
[g_main_dispatch]
glib/gmain.c:3180
[g_main_context_iterate]
glib/gmain.c:3902
[g_main_loop_run]
glib/gmain.c:4097
[gst_nice_src_create]
gst/gstnicesrc.c:292
[gst_base_src_get_range]
./libs/gst/base/gstbasesrc.c:2460
[gst_base_src_loop]
./libs/gst/base/gstbasesrc.c:2737
[gst_task_func]
./gst/gsttask.c:346


2021-02-24T19:23:02+08:00 -- New execution
Error loading config: /etc/kurento/modules/kurento/kms_rsa.pem, Unknown file type: kms_rsa.pem
Error loading config: /etc/kurento/modules/kurento/kms_ecdsa.pem, Unknown file type: kms_ecdsa.pem
Error loading config: /etc/kurento/modules/kurento/WebRtcEndpoint.conf.ini.dpkg-dist, Unknown file type: WebRtcEndpoint.conf.ini.dpkg-dist

@j1elo Please check whether this log contain some useful message.

@brunsy
Copy link

brunsy commented Aug 21, 2021

@sshsun1990 sorry to ping you on such an old message.
Do you remember how you got that change compiled and into Kurento?

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@j1elo j1elo

Labels
kms Related especifically to the media server libnice stability Crashes or critical bug that affects stability
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@darrenhp @j1elo @brunsy @sshsun1990