[Bug]: AMF crash after UE release request #2581
Comments
|
Do you have any experience using GDB? If so, it would be a great help in solving this problem if you could share the crash location using the tool. If you're not familiar with the tool, I can explain how to use it. Please feel free to let me know. Thanks a lot! |
acetcom
added
type:bug
|
Hi @acetcom
or
So yes I'd be glad if you could show me Thanks |
|
Hi @Ma2tGt I hope this is reproducible. Basically, you need to compile and run the source code according to following document. This can be summarized as follows: Then run gdb as follows. open5gs-amfd can be executed in the gdb with If a problem occurs, please backtrace using Please let me know if you have any other questions. Thanks a lot! |
|
Thanks for the details. Here's the result of #0 __pthread_kill_implementation (no_tid=0, signo=6, threadid=140737274422848) at ./nptl/pthread_kill.c:44
#1 __pthread_kill_internal (signo=6, threadid=140737274422848) at ./nptl/pthread_kill.c:78
#2 __GI___pthread_kill (threadid=140737274422848, signo=signo@entry=6) at ./nptl/pthread_kill.c:89
#3 0x00007ffff7112476 in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
#4 0x00007ffff70f87f3 in __GI_abort () at ./stdlib/abort.c:79
#5 0x00007ffff71596f6 in __libc_message (action=action@entry=do_abort, fmt=fmt@entry=0x7ffff72abb8c "%s\n") at ../sysdeps/posix/libc_fatal.c:155
#6 0x00007ffff7170d7c in malloc_printerr (str=str@entry=0x7ffff72ae640 "corrupted size vs. prev_size in fastbins") at ./malloc/malloc.c:5664
#7 0x00007ffff7171a1c in malloc_consolidate (av=av@entry=0x7fffec000030) at ./malloc/malloc.c:4771
#8 0x00007ffff7172f20 in _int_free (av=0x7fffec000030, p=0x7fffec031f70, have_lock=<optimized out>) at ./malloc/malloc.c:4674
#9 0x00007ffff71754d3 in __GI___libc_free (mem=<optimized out>) at ./malloc/malloc.c:3391
#10 0x00007ffff72fc2a4 in ?? () from /lib/x86_64-linux-gnu/libtalloc.so.2
#11 0x00007ffff7f7d7a9 in ogs_talloc_free (ptr=0x7fffec031fe0, location=0x7ffff7f95aa3 "../lib/core/ogs-pkbuf.c:292") at ../lib/core/ogs-memory.c:107
#12 0x00007ffff7f7d14a in ogs_pkbuf_free (pkbuf=0x7fffec031fe0) at ../lib/core/ogs-pkbuf.c:292
#13 0x00007ffff7f6798b in ogs_sctp_senddata (sock=0x7fffec01b690, pkbuf=0x7fffec031fe0, addr=0x0) at ../lib/sctp/ogs-sctp.c:73
#14 0x00007ffff7f67e22 in sctp_write_callback (when=2, fd=12, data=0x7ffff437f040) at ../lib/sctp/ogs-sctp.c:110
#15 0x00007ffff7f947db in epoll_process (pollset=0x555555636a00, timeout=5578823) at ../lib/core/ogs-epoll.c:283
#16 0x000055555555e19a in amf_main (data=0x0) at ../src/amf/init.c:118
#17 0x00007ffff7f8097a in thread_worker (arg=0x5555556ebec0) at ../lib/core/ogs-thread.c:67
#18 0x00007ffff7164b43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#19 0x00007ffff71f6a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81BR, |
|
Can you show the output of git log below? I would like to check which version you are using. |
|
|
I'll look at this problem now. Things that will probably take time. Thank you so much for sharing this problem. Sukchan |
|
Thank you for your help. BR |
|
It's currently difficult to determine the root cause of this issue based on gdb log. Can you turn on the trace level and send me the log file? You can do this as follows: And send me all log files in ./install/var/log/open5gs/*.log. Thanks a lot! |
|
Hi @acetcom, Here gdb log: Here my logs files: DebugCrashAMF.zip BR, |
|
Looking at the log now, I saw that a crash occurred 2 minutes after starting AMF. If so, I guess the pcap size won't be that big. Can you reproduce the crash situation in a short period of time and create a Wireshark pcap and share it with me? If that is possible, I can simulate that situation as is. Thank you so much for your effort. |
|
Hi @acetcom, I already take some PCAP traces. I have 2 cases: 1st: debug-TurnOnAirPlaneMode.pcap: AMF crash when I turn ON air plane mode Thanks |
|
All NAS messages are now encrypted. Probably because I set amf like below: $ diff --git a/configs/open5gs/amf.yaml.in b/configs/open5gs/amf.yaml.in
index b022552db..a73de1040 100644
--- a/configs/open5gs/amf.yaml.in
+++ b/configs/open5gs/amf.yaml.in
@@ -498,7 +498,7 @@ amf:
- sst: 1
security:
integrity_order : [ NIA2, NIA1, NIA0 ]
- ciphering_order : [ NEA0, NEA1, NEA2 ]
+ ciphering_order : [ NEA1, NEA0, NEA2 ]
network_name:
full: Open5GS
amf_name: open5gs-amf0Are crashes occurring even if the NAS messages are not encrypted? If so, could you please share the pcap that the NAS has unencrypted? Thanks a lot! |
|
You're right. I don't have any crash with this configuration In fact, I had changed this configuration in order to connect iPhone. Without encryption iphone canno't be connected. So I had: BR, |
|
I confirmed in the Open5gs simulation environment that changing the NAS security to NIA3 and NEA1 causes a crash. I'll get back to you once I resolve this issue. Thank you so much for raising this issue. Thanks a lot! |
|
Thank you @acetcom BR, |
|
The crash only occurs when the ciphering order starts with NEA1 in Open5GS simulator, as shown below. $ diff --git a/configs/open5gs/amf.yaml.in b/configs/open5gs/amf.yaml.in
index b022552db..a73de1040 100644
--- a/configs/open5gs/amf.yaml.in
+++ b/configs/open5gs/amf.yaml.in
@@ -498,7 +498,7 @@ amf:
- sst: 1
security:
integrity_order : [ NIA2, NIA1, NIA0 ]
- ciphering_order : [ NEA0, NEA1, NEA2 ]
+ ciphering_order : [ NEA1, NEA0, NEA2 ]
network_name:
full: Open5GS
amf_name: open5gs-amf0It works well in NEA0 (No encrypt), NEA2, and NEA3. Can you confirm if this is the case in your environment as well? In other words, it works well if you set it as below. integrity_order doesn't matter.
$ diff --git a/configs/open5gs/amf.yaml.in b/configs/open5gs/amf.yaml.in
index b022552db..450353d53 100644
--- a/configs/open5gs/amf.yaml.in
+++ b/configs/open5gs/amf.yaml.in
@@ -498,7 +498,7 @@ amf:
- sst: 1
security:
integrity_order : [ NIA2, NIA1, NIA0 ]
- ciphering_order : [ NEA0, NEA1, NEA2 ]
+ ciphering_order : [ NEA2, NEA1, NEA0 ]
network_name:
full: Open5GS
amf_name: open5gs-amf0
diff --git a/configs/open5gs/amf.yaml.in b/configs/open5gs/amf.yaml.in
index b022552db..05074071c 100644
--- a/configs/open5gs/amf.yaml.in
+++ b/configs/open5gs/amf.yaml.in
@@ -498,7 +498,7 @@ amf:
- sst: 1
security:
integrity_order : [ NIA2, NIA1, NIA0 ]
- ciphering_order : [ NEA0, NEA1, NEA2 ]
+ ciphering_order : [ NEA3, NEA1, NEA0 ]
network_name:
full: Open5GS
amf_name: open5gs-amf0
Thank you so much! |
There was a memory problem in the encryption using snow_3g_f8, so AMF/MME crashed. To solve this problem, we used the snow-3g encryption library created as below. https://github.com/rcatolino/libressl-snow3g However, it seems that this library cannot be used to create integrity hash like snow_3g_f8. So, we decided to keep both snow-3g libraries for the time being. 1. lib/crypt/snow3g* : for INTEGRITY (NIA1, EIA1) 2. lib/crypt/openssl/snow3g* : for ENCRYPTION (NEA1, EEA1)
There was a memory problem in the encryption using snow_3g_f8, so AMF/MME crashed. To solve this problem, we used the snow-3g encryption library created as below. https://github.com/rcatolino/libressl-snow3g However, it seems that this library cannot be used to create integrity hash like snow_3g_f8. So, we decided to keep both snow-3g libraries for the time being. 1. lib/crypt/snow3g* : for INTEGRITY (NIA1, EIA1) 2. lib/crypt/openssl/snow3g* : for ENCRYPTION (NEA1, EEA1)
|
I've fixed this issue in the main branch. Please let me know if you have any other problem. Thank you so much for your help. |
acetcom
added
Housekeeping:ToClose
|
HI @acetcom, It's works fine with your last commit. Thank you for your help BR |
Open5GS Release, Revision, or Tag
v2.6.4
Steps to reproduce
Hello,
We start to play with Open5GS and gNB Nokia. But sometime when we set UE in airplane mode, AMF crashed and process is killed. Sometime it's appear only after we turned off airplane mode.
Logs
Expected behaviour
AMF doesn't crash
Observed Behaviour
AMF crash after UE release request
eNodeB/gNodeB
Nokia ASOE
UE Models and versions
Iphone 13 / Nokia XR20
The text was updated successfully, but these errors were encountered: