SCTP Abort on new connections from ENB. MME stops processing.

[bhuvaneshne]

Your Environment

• Version:: Currently using v 1.6.1 (We are yet to upgrade to 1.7.X)
• Affected Component:: Access Gateway, or NMS
• Affected Subcomponent: MME, SCTPD
• Deployment Environment: bare-metal (AGW)

Describe the Issue

In our setup, there are about 40+ ENBs connected to a single AGW. In some certain rare occasions, MME would stop processing packets. SCTP Aborts are seen on the wire.

To Reproduce
The issue was getting reproduced rather frequently lately in the lab. The latest additions to the network were scrutinized. The latest addition- A "certain brand" of ENB was sending a spikes of S1AP Partial Resets at constant intervals. We modified open source simulator to send spikes of of S1AP partial resets which then reproduced similar behavior.

Expected behavior
AGW should not stop processing packets.

Additional context
Behavior of MME and SCTPD executables when issue occurs:
1. MME stops processing packets.
2. SCTPD GRPC errors are seen in syslog (GRPC timeout).
3. SCTP Aborts are seen on wire on new connections after the above two happens.

[bhuvaneshne]

Root cause analysis:
Added instrumentation code which added metadata - An integer assigned to the first stage in MME processing (GRPC receive). This metadata is carried across different stages in pipeline (Different MME Tasks). Also modified logs to print the thread ID. Logs showed that the GRPC send downlink was was stopping processing after sometime when S1AP partial reset bursts are sent. The other tasks stops processing soon after (Maybe due to ZMQ buffers getting filled).
Next, we added additional logs at SCTP Send DL processing. sctp_sendmsg was not returning control.

[bhuvaneshne]

Proposed solution:
Instead of waiting indefinitely, add a timeout to sctp_sendmsg

Before fix: sctp_connection.cpp

After proposed fix:

Caveat:
Have not looked into why the sctp_sendmsg was waiting indefinitely.

Please let us know if the fix looks good.

[ushivahs]

s1ap_test.zip
PFA which contains the simulator used for the test.
Usage is : ./s1ap_test mcc mnc tac pps srcIp dstIp
./s1ap_test 315 010 3f 50 192.168.56.93 192.168.56.9

Note: 1) Please give value of tac in hex.
2) pps is the rate at which partial reset has to be sent.

[ssanadhya]

@bhuvaneshne, @ushivahs , thanks for the detailed description of the issue! Adding timeout makes sense to me. Please go ahead and raise a PR with this fix.

[bhuvaneshne]

Thanks @ssanadhya , I have issued a pull request.

[bhuvaneshne]

@ssanadhya : We can close this issue. The code is checked in.

[ssanadhya]

Fixed in #13146