MME: magma@mme.service: Failed with result 'exit-code'.

[javieraubert]

Prior to submitting an issue, check to see if one has already been created. If there is currently an open issue, add a thumbs-up emoji to identify that it is also affecting you.

Your Environment

• **Version:1.8.0-1648772874-62caa3c3
• **Affected Component:AGW
• **Affected Subcomponent:MME Service
• **Deployment Environment:Bare-metal (AGW)

Describe the Issue

After some time of normal working, the mme, pipelined, mobilityd and td-agent suddenly restarts. It happens a few times a week.

To Reproduce
Not sure what the trigger is but it happens after several days of normal working, we do not do anything special.

Screenshots
On Syslog we have this messages:

38 gw2-1 sctpd[3335263]: I0509 00:17:38.029573 3335664 sctp_connection.cpp:195] Unhandled notification type 32777
May 9 00:17:38 gw2-1 sessiond[3335283]: I0509 00:17:38.529649 3335283 LocalEnforcer.cpp:431] Received stats for 56 active sessions and 0 stale sessions
May 9 00:17:39 gw2-1 mme[3335389]: LeakSanitizer:DEADLYSIGNAL
May 9 00:17:39 gw2-1 mme[3335389]: ==3335389==ERROR: LeakSanitizer: SEGV on unknown address 0x000000000000 (pc 0x55b75b876af2 bp 0x7fd9a57dca60 sp 0x7fd9a57dc940 T21)
May 9 00:17:39 gw2-1 mme[3335389]: ==3335389==The signal is caused by a WRITE memory access.
May 9 00:17:39 gw2-1 mme[3335389]: ==3335389==Hint: address points to the zero page.
May 9 00:17:39 gw2-1 sctpd[3335263]: I0509 00:17:39.360821 3335664 sctp_connection.cpp:195] Unhandled notification type 32777
May 9 00:17:39 gw2-1 mme[3335389]: #0 0x55b75b876af2 in s1ap_mme_generate_ue_context_release_command /home/vagrant/magma/lte/gateway/c/core/oai/tasks/s1ap/s1ap_mme_handlers.c:1407
May 9 00:17:39 gw2-1 mme[3335389]: #1 0x55b75b87fa18 in s1ap_mme_handle_uplink_nas_transport /home/vagrant/magma/lte/gateway/c/core/oai/tasks/s1ap/s1ap_mme_nas_procedures.c:336
May 9 00:17:39 gw2-1 mme[3335389]: #2 0x55b75b87c15f in s1ap_mme_handle_message /home/vagrant/magma/lte/gateway/c/core/oai/tasks/s1ap/s1ap_mme_handlers.c:235
May 9 00:17:39 gw2-1 mme[3335389]: #3 0x55b75b822054 in handle_message /home/vagrant/magma/lte/gateway/c/core/oai/tasks/s1ap/s1ap_mme.c:138
May 9 00:17:39 gw2-1 mme[3335389]: #4 0x7fd9b28955c6 in zloop_start (/lib/x86_64-linux-gnu/libczmq.so.4+0x295c6)
May 9 00:17:39 gw2-1 mme[3335389]: #5 0x55b75b821b6e in s1ap_mme_thread /home/vagrant/magma/lte/gateway/c/core/oai/tasks/s1ap/s1ap_mme.c:317
May 9 00:17:39 gw2-1 mme[3335389]: #6 0x7fd9b3756608 in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x8608)
May 9 00:17:39 gw2-1 mme[3335389]: #7 0x7fd9b1f48162 in __clone (/lib/x86_64-linux-gnu/libc.so.6+0x11f162)
May 9 00:17:39 gw2-1 mme[3335389]: LeakSanitizer can not provide additional info.
May 9 00:17:39 gw2-1 mme[3335389]: SUMMARY: LeakSanitizer: SEGV /home/vagrant/magma/lte/gateway/c/core/oai/tasks/s1ap/s1ap_mme_handlers.c:1407 in s1ap_mme_generate_ue_context_release_command
May 9 00:17:39 gw2-1 mme[3335389]: ==3335389==ABORTING
May 9 00:17:39 gw2-1 systemd[1]: magma@mme.service: Main process exited, code=exited, status=23/n/a
May 9 00:17:39 gw2-1 sctpd[3335263]: I0509 00:17:39.658337 3335664 sctpd_uplink_client.cpp:38] sctpul.sendul error
May 9 00:17:39 gw2-1 sctpd[3335263]: I0509 00:17:39.658371 3335664 sctpd_uplink_client.cpp:39] grpc error (14): OS Error
May 9 00:17:39 gw2-1 systemd[1]: magma@mme.service: Failed with result 'exit-code'.

Attached are the logs at the time of the issue

syslog09052022.txt

[javieraubert]

@vikramgreddy @pshelar @ardzoht

[ssanadhya]

@rsarwad , could you please investigate this?

[rsarwad]

@javieraubert Fixes are in master codebase. Could you please confirm whether the issue is fixed or not.

[javieraubert]

Hi @rsarwad we have not seen the same issue after the upgrade. Thanks a lot.