libnice cause cpu 100% and don't restore ( after ’attacked' by a spider or security-scaner )

[darrenhp]

Prerequisites

• I have read the SUPPORT document

• I have checked the Troubleshooting Guide

• I have tested with the latest version of Kurento

Issue description

Context

nicesrc thread cost cpu100%

  PID USER      PR  NI    VIRT    RES    SHR S %CPU %MEM     TIME+ COMMAND
13649 kurento   20   0 2620872  50624  25180 R 99.9  0.6   0:11.21 nicesrc29:src

How to reproduce?

Run kurento-tutorial-java/kurento-hello-world，
then open the demo in chrome， and extrace all the port in the console;
then curl on every ports
finally, the nicesrc29:src thread 's cpu goes up to 100%;

Expected & current behavior

(Optional) Possible solution

INFO about Kurento Media Server

• Kurento version: 6.13.2

 Kurento Media Server version: 6.13.2~14.g4e543d0
    Found modules:
	'core' version 6.13.2~11.g2f144b59
	'elements' version 6.13.2~11.ga4c9f35
	'filters' version 6.13.2~2.gd6fea92

• Server OS: 8.04 (Bionic)
• Installation method: apt-get
  • [] apt-get
  • [] Docker
  • [] AWS
  • [] Built from sources

INFO about your Application Server

• Language: Java
• Kurento Client version:

INFO about end-user clients

• Device(s): Mac
• OS(es): MacOs 10.15.5 (19F101)
• Browser(s): Chrome 83

INFO about your environment

Commands output

cat /etc/lsb-release

kurento-media-server --version

dpkg -l | grep -Pi 'kurento|kms-|gst.*1.5|nice'

root@kurento-test-20200408:~# kurento-media-server --version
Kurento Media Server version: 6.13.2~14.g4e543d0
Found modules:
	'core' version 6.13.2~11.g2f144b59
	'elements' version 6.13.2~11.ga4c9f35
	'filters' version 6.13.2~2.gd6fea92
root@kurento-test-20200408:~#
root@kurento-test-20200408:~# dpkg -l | grep -Pi 'kurento|kms-|gst.*1.5|nice'
ii  gstreamer1.5-libav:amd64              1.8.1-1kurento1.18.04~20200506171443.gbpb33143  amd64        libav plugin for GStreamer
ii  gstreamer1.5-nice:amd64               0.1.16-1.18.04~20200506171443.gbp1854ac         amd64        ICE library (GStreamer 1.5 plugin)
ii  gstreamer1.5-plugins-bad:amd64        1.8.1-1kurento4.18.04~20200506171443.gbp17d48d  amd64        GStreamer plugins from the "bad" set
ii  gstreamer1.5-plugins-base:amd64       1.8.1-1kurento2.18.04~20200506171443.gbp7fb324  amd64        GStreamer plugins from the "base" set
ii  gstreamer1.5-plugins-good:amd64       1.8.1-1kurento3.18.04~20200506171443.gbp9c3835  amd64        GStreamer plugins from the "good" set
ii  gstreamer1.5-plugins-ugly:amd64       1.8.1-1kurento1.18.04~20200506171443.gbpbed8b0  amd64        GStreamer plugins from the "ugly" set
ii  gstreamer1.5-pulseaudio:amd64         1.8.1-1kurento3.18.04~20200506171443.gbp9c3835  amd64        GStreamer plugin for PulseAudio
ii  kms-core                              6.13.2-0kurento1.18.04~20200507140307.gbp2f144b amd64        Kurento Core module
ii  kms-elements                          6.13.2-0kurento1.18.04~20200507140307.gbpa4c9f3 amd64        Kurento Elements module
ii  kms-filters                           6.13.2-0kurento1.18.04~20200507140307.gbpd6fea9 amd64        Kurento Filters module
ii  kms-jsonrpc                           6.13.2-0kurento1.18.04~20200507140307.gbpfd864c amd64        Kurento JSON-RPC library
ii  kmsjsoncpp                            1.6.3-1kurento1.18.04~20200506171443.gbp15940b  amd64        Kurento jsoncpp library
ii  kurento-media-server                  6.13.2-0kurento1.18.04~20200507140307.gbp4e543d amd64        Kurento Media Server
ii  libgstreamer-plugins-bad1.5-0:amd64   1.8.1-1kurento4.18.04~20200506171443.gbp17d48d  amd64        GStreamer development files for libraries from the "bad" set
ii  libgstreamer-plugins-base1.5-0:amd64  1.8.1-1kurento2.18.04~20200506171443.gbp7fb324  amd64        GStreamer libraries from the "base" set
ii  libgstreamer1.5-0:amd64               1.8.1-1kurento2.18.04~20200506171443.gbp84c84b  amd64        Core GStreamer libraries and elements
ii  libnice10:amd64                       0.1.16-1.18.04~20200506171443.gbp1854ac         amd64        ICE library (shared library)
ii  libsrtp0:amd64                        1.6.0-0kurento1.18.04~20200506171443.gbp577e0a  amd64        Secure RTP (SRTP) and UST Reference Implementations - shared library
ii  libusrsctp                            0.9.2-1kurento1.18.04~20200506171443.gbpfd1d0e  amd64        sctp-refimpl library
ii  openh264                              1.4.0-1kurento1.18.04~20200506171443.gbpb8cb17  amd64        OpenH264 library
ii  openh264-gst-plugins-bad-1.5:amd64    1.8.1-1kurento4.18.04~20200506171443.gbp17d48d  amd64        GStreamer plugins from openh264
ii  openwebrtc-gst-plugins                0.10.0-1kurento1.18.04~20200506171443.gbp6a0d22 amd64        OpenWebRTC specific GStreamer plugins
root@kurento-test-20200408:~#

[j1elo]

Hi, please detail exactly how you reproduce this:

• How or from where you extract all ports from the browser
• The exact curl command you use
• How you check the nicesrc thread CPU usage

As detailed as possible so we can reproduce with copy&paste commands if possible

[darrenhp]

• How or from where you extract all ports from the browser
  
• The exact curl command you use
  just like this curl xx.xx.xx.xx:48705
• How you check the nicesrc thread CPU usage
  the command top -H

[alexlapa]

I confirm this. To be more exact, you need to curl provided host:port from typ host tcptype passive candidate.

How you check the nicesrc thread CPU usage

That thread is named nicesrc4:src (but that doesnt really means that its libnice fault)

[j1elo]

I think this is an issue in libnice. After running with Callgrind, this is the output graph of where the CPU is being used:

I'm not 100% sure but one possibility is that because the input queue is always empty (because curl doesn't send any data to the socket), the g_main_loop_run is constantly being called, without any time to sleep for a while, here.

Another possibility is that the problem comes from the component_io_cb function:

which is a complex function and does a lot of things, but my first suspect would be the call to agent_recv_message_unlocked.

I'll be opening an issue in their repo, with all this info.

UPDATE: I opened a confidential issue report, will update here with any news.

[alexlapa]

I'm not 100% sure but one possibility is that because the input queue is always empty (because curl doesn't send any data to the socket)

curl ip:port sends HTTP GET request, so it writes to socket immediately.

Ive tested it with netcat: opening socket has no effect, libnice's thread enters loop as soon as first bytes are received.

[j1elo]

This was confirmed to be a bug in libnice, which got reported and promptly fixed by libnice developers. Kurento will update its version of libnice for the next release, so we can all benefit from this improvement.

See libnice bug #114: 100% CPU stuck in nicesrc thread after curl to tcp-passive port