100% Utilisation (lots of sendmsg)

[digipigeon]

Compiled and deployed on a clean Ubuntu 14.04, I ran traffic for 1 hour, then system went to 100% utilisation.

We were using the old RTP Proxy protocol and configured ipv4 only.

Running strace we got the following message (defiantly starting the exact same) continuously repeating. Failed to respond to service stop, had to kill the process and restart.

sendmsg(279, {msg_name(28)={sa_family=AF_INET6, sin6_port=htons(31930), inet_pton(AF_INET6, "::ffff:178.62.XXX.XXX", &sin6_addr), sin6_flowinfo=0, sin6_scope_id=0}, msg_iov(1)=[{"\200\0\241P\16R\377\331\221\2\253\272?OF>;AWVIHFRjlw\337\336\341kX"..., 172}], msg_controllen=32, {cmsg_len=28, cmsg_level=SOL_IP, cmsg_type=, ...}, msg_flags=0}, 0) = 172
recvfrom(275, "\200\0\241\335\16SW\371\221\2\253\272,e\260\243\236\233\232\232\233\235\237\247\262\317>+"\36\33\31"..., 8192, 0, {sa_family=AF_INET6, sin6_port=htons(31934), inet_pton(AF_INET6, "::ffff:178.62.XXX.XXX", &sin6_addr), sin6_flowinfo=0, sin6_scope_id=0}, [28]) = 172

[rfuchs]

You wouldn't happen to have a core dump from this, would you?

Is the address (178.62.XXX.XXX) the local address of the proxy itself by any chance?

[digipigeon]

Hi rfuchs,

I am sorry I don't have a core dump, the program did not crash just max utilisation (and no longer responding to any requests), (I am not sure how to extract one from a running program / if possible).

The 178.62.XXX.XXX is the IPv4 internet address of the proxy, yes.

[rfuchs]

I guess the proxy ended up talking to itself in a loop then. Perhaps an intentional DoS by one of your users? You should be able to find details about who or when in the log. Look for offers and answers advertising the proxy's own IP address

[digipigeon]

We are running a multi-tenant platform currently on standard RTP Proxy, it would be theoretically possible for one customer to send a call through another customer on our platform. It is possible for the same server to be engaged twice. With standard RTP Proxy this has caused no problems.

Is this what you are seeing here?

[rfuchs]

No, not really. The proxy talking to itself isn't a problem per se, this should work as expected. What I mean is the proxy talking to itself in a loop. For example, the proxy sends packets arriving on port X to itself on port Y, and sends packets arriving on port Y to itself on port X. Something like that.

[digipigeon]

Would this not mean that the potential attacker has access to my control ports, this should not be possible as these are fire-walled.

[rfuchs]

No, this could be done through signalling, ie within the SDP body. But deliberate DoS is just a guess, you should really look in your logs and try to identify the call in question. The local ports involved were 31930 and 31934, you should be able to find those in the log.

[digipigeon]

Hi rfuchs,

I have tracked this down and I can see the two commands, comparing this to other lines in the log file, it does not look unusual, it appears that 2 distinct external IP addresses were provided (which upon further checking are my customer and provider).

However if your suspicion is correct (and I have mis-read the data), how can this be prevented, as surely this is a security vulnerability.

[rfuchs]

It would help to know how exactly this came to be, but without additional details I'm afraid that we'll never know for sure.

Detecting these kind of loops (whether caused intentionally or by accident) is tricky. Obviously you can't just keep the proxy from sending packets to itself, as this is required in certain scenarios. I'll try to come up with something.

[rfuchs]

So this is a somewhat crude and experimental attempt to catch forwarding loops, but it seems to do the trick in my tests.

[digipigeon]

I have returned to my rtpengine servers this morning locked up with 100% utilisation and observed the following errors in the log:

Aug 31 03:46:54 rtp-lon-3 rtpengine[10953]: [0533067617c6603a1ea1221e371048e5@107.161.XXX.XXX port 38296] Too many packets in UDP receive queue (more than 50), aborting loop. Dropped packets possible
Aug 31 03:46:54 rtp-lon-3 rtpengine[10953]: [3ada38e9438f61804ebf42752929cf1c@107.161.XXX.XXX port 42782] Too many packets in UDP receive queue (more than 50), aborting loop. Dropped packets possible
Aug 31 03:46:54 rtp-lon-3 rtpengine[10953]: [598eedda6285c1f46c00fbff12f16cce@107.161.XXX.XXX port 30426] Too many packets in UDP receive queue (more than 50), aborting loop. Dropped packets possible
Aug 31 03:46:54 rtp-lon-3 rtpengine[10953]: [3f984c5476921e606938f7e57987db08@107.161.XXX.XXX port 29462] Too many packets in UDP receive queue (more than 50), aborting loop. Dropped packets possible
Aug 31 03:46:54 rtp-lon-3 rtpengine[10953]: [0533067617c6603a1ea1221e371048e5@107.161.XXX.XXX port 38296] Too many packets in UDP receive queue (more than 50), aborting loop. Dropped packets possible

This has continues for many hours through the night.

The version of the compiles binaries is 3.3.0.0+0~mr3.5.0.0 git-master-c0f8196

[digipigeon]

In addition I have just noticed that the lo interface has

RX bytes:1014751848427 (1.0 TB)  TX bytes:1014751848427 (1.0 TB)

[rfuchs]

So your proxy is still sending packets to itself. Can you select one of the call-ids that you posted, and then post the complete rtpengine log for that call please.