Multi UE Attach getting failed [ESM Failure] and PDN Connectivity Reject with cause 30 [Request rejected by Serving GW or PDN GW] with S1AP Tester (python script) ON LTE Setup and MME use high Memory Usage

[Aniket621]

Prior to submitting an issue, check to see if one has already been created. If there is currently an open issue, add a thumbs-up emoji to identify that it is also affecting you.

Your Environment

1. Magma Master Branch
2. Multi UE (308) attach attempt from S1AP Simulator [magma_test] VM.
   Python Script: make selected_tests TESTS=s1aptests/test_attach_detach_multi_ue.py
3. git status:-
   On branch master
   Your branch is up to date with 'origin/master'.
4. git branch

• master
  v1.6

5. git log -1
   commit 20a6e7a (HEAD -> master, origin/master, origin/HEAD)
   Author: Lars Kreutzer <34488763+LKreutzer@users.noreply.github.com>
   Date: Thu Mar 9 13:43:57 2023 +0100
   chore: Build the Python services with Bazel for CWF Docker images (chore: Build the Python services with Bazel for CWF Docker images #15106)
   Signed-off-by: Lars Kreutzer <lars.kreutzer@tngtech.com>

Describe the Issue

UE attach request+PDN request reject while run multi UE attach test case with ESM Cause 19 ESM Failure and PDN Connectivity request reject with EMM Cause 30 Request rejected by Serving GW or PDN GW.

As per mme.log observe last trace of UE (ue_id:- 277) is of sending session request and response is received exactly after 10 seconds which is failure only EMM_CN_ULA_OR_CSRSP_FAIL (405).
Default EPS bearer Setup failure or an ESM procedure failure and gives EMM Cause 30. As per console logs getting Assertion error, Subscriberdb grpc call failed.
IMSI:- 1010000000277, 274 and ue_id:- 277, 274

243 UE attach with S1AP Tester Magma setup (Master branch) with UE IP Configuration 192.168.128.0/24. After that I change the UE IP configuration from 192.168.128.0/24 to 192.168.128.0/20
as per below document then UE reach 276 with proper attach-detach after that attach request rejected from network. mme logs and wireshark attached here with.

To Reproduce

Issue can reprodue while configure multi UE in test_attach_detach_multi_ue.py script in S1AP Tetser.

Expected behavior

All UE should attach success with PDN Connectivity request success for all UE.

Screenshots

test case output:-

mme logs

syslogs failure screen shot

276 UE successfully attach but not constant with setup.

Additional context

https://magmacore.slack.com/archives/C018J8UMGMR/p1679902713474479
All logs (Wireshark Logs, mme logs, syslogs ) updated slack on above path.

[Aniket621]

MME services is taking high CPU utilization during testing and the service the crash in b/w .

MME Service status :-

MME Debug logs:-

Mar 30 05:08:03 magma-dev mme[30406]: [INFO] sent itti S6A-LOCATION-UPDATE_ANSWER for IMSI: 001010000000236
Mar 30 05:08:13 magma-dev mme[30406]: /home/vagrant/magma/lte/gateway/c/core/oai/tasks/sgw/sgw_handlers.cpp:1462:9: runtime error: null pointer passed as argument 1, which is declared to never be null
Mar 30 05:08:22 magma-dev systemd[1]: Stopping Magma OAI MME service...
Mar 30 05:08:22 magma-dev mme[30406]: Received SIGINT or SIGTERM
Mar 30 05:08:22 magma-dev mme[30406]: [TRACE] Entering shared_log_exit
Mar 30 05:08:22 magma-dev mme[30406]: TASK_S6A terminated
Mar 30 05:08:22 magma-dev mme[30406]: TASK_SERVICE303_SERVER terminated
Mar 30 05:08:22 magma-dev mme[30406]: TASK_SERVICE303 terminated
Mar 30 05:08:22 magma-dev mme[30406]: TASK_SGW_S8 terminated
Mar 30 05:08:22 magma-dev mme[30406]: TASK_GRPC_SERVICE terminated
Mar 30 05:08:22 magma-dev mme[30406]: TASK_ASYNC_GRPC_SERVICE terminated
Mar 30 05:08:22 magma-dev mme[30406]: [TRACE] Leaving shared_log_exit
Mar 30 05:08:22 magma-dev mme[30406]: TASK_SHARED_TS_LOG terminated
Mar 30 05:08:22 magma-dev mme[30406]: TASK_SCTP terminated
Mar 30 05:08:22 magma-dev mme[30406]: TASK_S1AP terminated
Mar 30 05:08:22 magma-dev mme[30406]: TASK_SPGW_APP terminated
Mar 30 05:08:22 magma-dev mme[30406]: TASK_MME_APP terminated
Mar 30 05:08:23 magma-dev mme[30406]: Cleaning up SPGW configs-----------------------------------------------------
Mar 30 05:08:23 magma-dev mme[30406]: Suppressions used:
Mar 30 05:08:23 magma-dev mme[30406]: count bytes template
Mar 30 05:08:23 magma-dev mme[30406]: 1 48 libczmq.so.4
Mar 30 05:08:23 magma-dev mme[30406]: 3 240 fluid_base::BaseOFConnection::OFReadBuffer::read_notify
Mar 30 05:08:23 magma-dev mme[30406]: -----------------------------------------------------
Mar 30 05:08:23 magma-dev systemd[1]: magma@mme.service: Succeeded.
Mar 30 05:08:23 magma-dev systemd[1]: Stopped Magma OAI MME service.
Mar 30 05:08:30 magma-dev systemd[1]: Starting Magma OAI MME service...
Mar 30 05:08:30 magma-dev mme[321097]: ERROR:root:Error retrieving config for mme, key not found: enable_nat
Mar 30 05:08:30 magma-dev mme[321097]: ERROR:root:Error retrieving config for mme, key not found: congestion_control_enabled
Mar 30 05:08:30 magma-dev mme[321097]: ERROR:root:Error retrieving config for mme, key not found: enable5g_features
Mar 30 05:08:30 magma-dev mme[321097]: ERROR:root:Error retrieving config for mme, key not found: amf_default_slice_service_type
Mar 30 05:08:30 magma-dev mme[321097]: ERROR:root:Error retrieving config for mme, key not found: amf_default_slice_differentiator
Mar 30 05:08:30 magma-dev mme[321097]: ERROR:root:Error retrieving config for mme, key not found: amf_name
Mar 30 05:08:30 magma-dev mme[321097]: ERROR:root:Error retrieving config for mme, key not found: amf_region_id
Mar 30 05:08:30 magma-dev mme[321097]: ERROR:root:Error retrieving config for mme, key not found: amf_set_id
Mar 30 05:08:30 magma-dev mme[321097]: ERROR:root:Error retrieving config for mme, key not found: amf_pointer
Mar 30 05:08:30 magma-dev mme[321097]: ERROR:root:Error retrieving config for mme, key not found: default_dnn
Mar 30 05:08:30 magma-dev mme[321097]: ERROR:root:Error retrieving config for mme, key not found: auth_retry_max_count
Mar 30 05:08:30 magma-dev mme[321097]: ERROR:root:Error retrieving config for mme, key not found: auth_retry_interval
Mar 30 05:08:30 magma-dev mme[321097]: ERROR:root:Error retrieving config for spgw, key not found: ovs_internal_sampling_fwd_tbl
Mar 30 05:08:30 magma-dev mme[321097]: WARNING:root:Service (spgw) missing in mconfig
Mar 30 05:08:30 magma-dev mme[321147]: STATELESS mme -> use_stateless
Mar 30 05:08:30 magma-dev mme[321147]: STATELESS pipelined -> clean_restart
Mar 30 05:08:30 magma-dev mme[321147]: STATELESS pipelined -> redis_enabled
Mar 30 05:08:30 magma-dev mme[321147]: STATELESS sessiond -> support_stateless
Mar 30 05:08:30 magma-dev mme[321147]: Check returning return_codes.STATELESS
Mar 30 05:08:30 magma-dev mme[321147]: AGW is stateless, no need to restart Sctpd
Mar 30 05:08:31 magma-dev mme[321149]: Initializing OAI Logging
Mar 30 05:08:31 magma-dev mme[321149]: Initializing OAI Logging to syslog
Mar 30 05:08:31 magma-dev mme[321149]: Initializing shared logging
Mar 30 05:08:31 magma-dev mme[321149]: Initializing shared logging Done
Mar 30 05:08:31 magma-dev sudo[321171]: root : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/usr/bin/ovs-vsctl list Open_vSwitch
Mar 30 05:08:31 magma-dev sudo[321171]: pam_unix(sudo:session): session opened for user root by (uid=0)
Mar 30 05:08:31 magma-dev sudo[321171]: pam_unix(sudo:session): session closed for user root
Mar 30 05:08:31 magma-dev mme[321172]: iface_types : [bareudp, erspan, geneve, gre, gtpu, internal, ip6erspan, ip6gre, lisp, patch, stt, system, tap, vxlan]
Mar 30 05:08:31 magma-dev mme[321149]: Server running (127.0.0.1:6654)
Mar 30 05:08:32 magma-dev systemd[1]: Started Magma OAI MME service.