Server randomly becomes unresponsive then recovers minutes later

[jrsparks86]

Issue
We have been noticing for a while servers that focus on the webrtc connections randomly becoming unresponsive. This unresponsive state happens anywhere from a couple minutes to ten minutes. If you are looking at the charts the CPU drops off and then once it recovers you see a spike as all the registrations are flooding back in. We have also seen cases where the server never recovers until we restart it. In every case the server seems to keep doing stuff but new registrations are blocked.

I was finally able to recreate something that looks very similar to what we are seeing.

The test environment

• Drachtio Server 0.8.24
• Around 60 SIP connections but able to do it with around 10

How to recreate
Open up 8 terminal windows. In each terminal run the command openssl s_client -connect URL_OF_WEBRTC_SERVER:WEBRTC_PORT right after one another. Once the command stops in one window, run it again. This timing changes and seems to be a perfect timing thing. I had it happen after 9 commands and I had it after 20ish commands. Once it happens all the terminals will be blocked in a "CONNECTED" state besides one which will say read R BLOCK.

At this point you will see on the Drachtio server addAppTransaction and addApiRequest size increase. You will also see registrations will start failing to happen.

To clear it from this state go back to the terminal that says read R BLOCK and end that connection. All the other terminals will start processing again and the registrations will start working again. I was able to recreate this every time I ran the test.

Logs

• Around where the block starts 2024-01-24 22:05:29.120311
• Around where the block ends 2024-01-24 22:07:21.284546

sip-webrtc-redacted.txt