[TeraVM 5G SA Regression] Core Dump observed for last three runs due to TC:14 (?)

[arun-magma]

Test Scenario: TC-14
SA_TC14_Verify_T3560_expiry_and_Retransmission_of_AUTHENTICATION_REQUEST_211018_120830
Enclosed pcaps of these 3 Runs.

Flow Sequence:
1, UE sends the Registration
2, Authentication Requests
3, Authentication Requests Retry is missing

[Enclosed Pcaps & BT]
Followed by SCTP Aborts found.https://app.zenhub.com/files/170803235/73187523-e0cb-48f1-a3cc-bb8ffa418786/download

Tested Build: 1.7.0-1634577793-14c1cf64
[Core Dump - 10/18]

[New LWP 3404873]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/usr/local/bin/mme -c /var/opt/magma/tmp/mme.conf -s /var/opt/magma/tmp/spgw.co'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 magma5g::get_nas5g_common_procedure (ctxt=0x61d00005b410,
proc_type=magma5g::AMF_COMM_PROC_AUTH)
at /home/vagrant/magma/lte/gateway/c/core/oai/tasks/amf/amf_authentication.cpp:190
[Current thread is 1 (Thread 0x7f11aa3ed700 (LWP 3391673))]
#0 magma5g::get_nas5g_common_procedure (ctxt=0x61d00005b410,
proc_type=magma5g::AMF_COMM_PROC_AUTH)
at /home/vagrant/magma/lte/gateway/c/core/oai/tasks/amf/amf_authentication.cpp:190
#1 0x0000559e6592666d in magma5g::get_nas5g_common_procedure_authentication (
ctxt=0x61d00005b410)
at /home/vagrant/magma/lte/gateway/c/core/oai/tasks/amf/amf_authentication.cpp:215
#2 0x0000559e659282c3 in magma5g::authenthication_t3560_handler (
loop=0x60700001a7f0, timer_id=60, arg=0x0)
at /home/vagrant/magma/lte/gateway/c/core/oai/tasks/amf/amf_authentication.cpp:1007
#3 0x00007f11b89e444a in zloop_start ()
from /lib/x86_64-linux-gnu/libczmq.so.4
#4 0x0000559e65902c0f in magma5g::amf_app_thread (args=0x0)
at /home/vagrant/magma/lte/gateway/c/core/oai/tasks/amf/amf_app_main.cpp:171
#5 0x00007f11b957b609 in start_thread (arg=)
at pthread_create.c:477
#6 0x00007f11b8366293 in clone ()
at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Tested Build: 1.7.0-1634598626-afc2ad9d
[Core Dump - 10/19]

[New LWP 3498638]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/usr/local/bin/mme -c /var/opt/magma/tmp/mme.conf -s /var/opt/magma/tmp/spgw.co'.
Program terminated with signal SIGILL, Illegal instruction.
#0 0x00005615648522cb in ngap_amf_thread (args=)
at /home/vagrant/magma/lte/gateway/c/core/oai/tasks/ngap/ngap_amf.c:209
[Current thread is 1 (Thread 0x7f07d62e8700 (LWP 3499196))]
#0 0x00005615648522cb in ngap_amf_thread (args=)
at /home/vagrant/magma/lte/gateway/c/core/oai/tasks/ngap/ngap_amf.c:209
#1 0x00007f07e7d03609 in start_thread (arg=)
at pthread_create.c:477
#2 0x00007f07e6aee293 in clone ()
at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Tested Build: 1.7.0-1634668331-66d9b7be
[Core Dump - 10/20]

[New LWP 3694399]
[New LWP 3694387]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/usr/local/bin/mme -c /var/opt/magma/tmp/mme.conf -s /var/opt/magma/tmp/spgw.co'.
Program terminated with signal SIGILL, Illegal instruction.
#0 0x0000559d4832a2cb in ngap_amf_thread (args=)
at /home/vagrant/magma/lte/gateway/c/core/oai/tasks/ngap/ngap_amf.c:209
[Current thread is 1 (Thread 0x7f306d5e6700 (LWP 3694426))]
#0 0x0000559d4832a2cb in ngap_amf_thread (args=)
at /home/vagrant/magma/lte/gateway/c/core/oai/tasks/ngap/ngap_amf.c:209
#1 0x00007f307ff4f609 in start_thread (arg=)
at pthread_create.c:477
#2 0x00007f307ed3a293 in clone ()
at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

[Yatheesha-Gubbi]

Tested Build: 1.7.0-1634668331-66d9b7be
[Core Dump - 10/21]

warning: Unexpected size of section `.reg-xstate/4182656' in core file.
#0 magma5g::get_nas5g_common_procedure (ctxt=0x61d0000a8c10, proc_type=magma5g::AMF_COMM_PROC_AUTH)
at /home/vagrant/magma/lte/gateway/c/core/oai/tasks/amf/amf_authentication.cpp:190
190 /home/vagrant/magma/lte/gateway/c/core/oai/tasks/amf/amf_authentication.cpp: No such file or directory.
[Current thread is 1 (Thread 0x7fd46b8ed700 (LWP 4182656))]
(gdb) bt
#0 magma5g::get_nas5g_common_procedure (ctxt=0x61d0000a8c10, proc_type=magma5g::AMF_COMM_PROC_AUTH)
at /home/vagrant/magma/lte/gateway/c/core/oai/tasks/amf/amf_authentication.cpp:190
#1 0x000055640f8bd66d in magma5g::get_nas5g_common_procedure_authentication (ctxt=0x61d0000a8c10)
at /home/vagrant/magma/lte/gateway/c/core/oai/tasks/amf/amf_authentication.cpp:215
#2 0x000055640f8bf2c3 in magma5g::authenthication_t3560_handler (loop=0x60700001a7f0, timer_id=174, arg=0x0)
at /home/vagrant/magma/lte/gateway/c/core/oai/tasks/amf/amf_authentication.cpp:1007
#3 0x00007fd479f7844a in zloop_start () from /lib/x86_64-linux-gnu/libczmq.so.4
#4 0x000055640f899c0f in magma5g::amf_app_thread (args=0x0)
at /home/vagrant/magma/lte/gateway/c/core/oai/tasks/amf/amf_app_main.cpp:171
#5 0x00007fd47ab0f609 in start_thread (arg=) at pthread_create.c:477
#6 0x00007fd4798fa293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
(gdb)

PCAP trace attached for the reference:
https://app.zenhub.com/files/170803235/48ed43e4-a931-4152-80e6-f3ef22502c3b/download

[Kaleem-Wavelabs]

@arun-magma @Yatheesha-Gubbi
From the pcap and core file stack it is not very clear:

1. what is the operation being executed at the time of core
2. pcap shows registration procedure is stopping at initialUEMessage itself, not even reaching to Authentication procedure
3. core file stack points to different locations in different runs
4. please let us know is there any special operation being run at the time of this TC14 case is run?
5. In our 140 TeraVM server, we are able succesfully run this case with the latest build all the time
6. Can you please capture mme.log in debug mode, syslog with grpc log enabled and share them to triage the issue further

[Kaleem-Wavelabs]

For the last 6 regression runs, observed TC is running without any issues, not observed any core files. Let us know if the issue still persist.

[Kaleem-Wavelabs]

Closing this as the issue is not observed anymore. Please do reopen the issue if it is seen with the requested logs.