mmed locking up

[theorem6]

I have two issues likely related. In some cases while passing traffic the mmed will lock up and all UE will go offline. Restarting the service restores connectivity.

In another instance only using Open5GS for the control plane and HSS authorization, the MME sends a shutdown packet to the ENB dropping all UE.

Any thoughts are greatly appreciated.

David

[acetcom]

Hi @theorem6

Could you provide the relevant log files or pcap?

For your reference, Open5GS MME does not initiate S1AP-Reset message to eNB first. The Reset ACK is sent only when the eNB sends it.

Thanks a lot!
Sukchan

[bladebeam]

We are seeing a similar issue. Several times per day our UEs go offline, and restarting the Open5Gs services restores service. Here are the logs from this morning:

Sep 5 06:25:30 elvinber open5gs-sgwd[2444]: #33[32m09/05 06:25:30.264#033[0m: [#33[33msock#033[0m] #33[1;33mERROR#033[0m: sendto(len:1508) failed (11:Resource temporarily unavailable) (../lib/core/ogs-socket.c:251)
Sep 5 06:25:30 elvinber open5gs-sgwd[2444]: #33[32m09/05 06:25:30.264#033[0m: [#33[33mgtp#033[0m] #33[1;33mERROR#033[0m: ogs_send() failed (../lib/gtp/path.c:113)
Sep 5 06:25:30 elvinber open5gs-sgwd[2444]: #33[32m09/05 06:25:30.264#033[0m: [#33[33msock#033[0m] #33[1;33mERROR#033[0m: sendto(len:1508) failed (11:Resource temporarily unavailable) (../lib/core/ogs-socket.c:251)

[acetcom]

Hi @bladebeam and @theorem6

There seems to be some problem with MME. Maybe there was an infinite loop or it was dead. During the development of v2.0.0, I found a suspicious part and mme-init.c was modified for safety. However, until this problem is resolved, ogs_expect() has been modified to be killed like assert().

Could you check if you still have this problem using the master branch now? Yeah! To do this, you need to compile source code. Or do you need a package installer? If so, I will make a package and upload it as soon as possible.

Thank you very much for sharing it.
Sukhan!

[cabotek]

@acetcom

Below are some additional errors. A package from a repo would be the best if possible. We use Ubuntu 18.04

09/05 17:21:03.115: [pgw] WARNING: ERROR DIAMETER Result Code(5002) (../src/pgw/pgw-fd-path.c:594)
09/05 17:22:14.160: [pgw] ERROR: [DROP] Cannot find subnet V:6, IPv4:0x7f86a28f4598, IPv6:(nil) (../src/pgw/pgw-gtp-path.c:190)

09/05 17:29:34.264: [sock] ERROR: sendto(len:767) failed (11:Resource temporarily unavailable) (../lib/core/ogs-socket.c:251)
09/05 17:29:34.264: [gtp] ERROR: ogs_send() failed (../lib/gtp/path.c:113)

[diam] ERROR: ../subprojects/freeDiameter/libfdcore/p_psm.c:287 'STATE_OPEN' -> 'STATE_CLOSED' 'pgw.localdomain' (../lib/diameter/common/init.c:116)

[mem] ERROR: 3 in '&pool->pkbuf[16008]' were not released. (../lib/core/ogs-pkbuf.c:150)

[mme] WARNING: ERROR DIAMETER Result Code(3002) (../src/mme/mme-fd-path.c:297)

[diam] ERROR: ../subprojects/freeDiameter/extensions/dbg_msg_dumps/dbg_msg_dumps.c:112 ROUTING ERROR 'No remaining suitable candidate to route the message to' for: (../lib/diameter/common/init.c:116)

09/05 16:30:54.284: [diam] ERROR: ../subprojects/freeDiameter/extensions/dbg_msg_dumps/dbg_msg_dumps.c:113 AVP: 'Session-Id'(263) l=8 f=-M val="mme.localdomain;1599323446;19;app_s6a" (../lib/diameter/common/init.c:116)
09/05 16:30:54.284: [diam] ERROR: ../subprojects/freeDiameter/extensions/dbg_msg_dumps/dbg_msg_dumps.c:113 AVP: 'Auth-Session-State'(277) l=12 f=-M val='NO_STATE_MAINTAINED' (1 (0x1)) (../lib/diameter/common/init.c:116)
09/05 16:30:54.284: [diam] ERROR: ../subprojects/freeDiameter/extensions/dbg_msg_dumps/dbg_msg_dumps.c:113 AVP: 'Vendor-Id'(266) l=12 f=-M val=10415 (0x28af) (../lib/diameter/common/init.c:116)

[bladebeam]

Sukhan, thanks for the quick response! We have 1,000 users on this system and have to restart every few hours. Can you send an installer package for our Ubuntu package manager? Thanks!

…

On Sep 5, 2020, at 11:33 AM, Sukchan Lee ***@***.***> wrote: There seems to be some problem with MME. Maybe there was an infinite loop or it was dead. During the development of v2.0.0, I found a suspicious part and mme-init.c was modified for safety. However, until this problem is resolved, ogs_expect() has been modified to be killed like assert(). Could you check if you still have this problem using the master branch now? Yeah! To do this, you need to compile source code. Or do you need a package installer? If so, I will make a package and upload it as soon as possible. Thank you very much for sharing it. Sukhan!

[theorem6]

Will the master branch update Ubuntu 18.04?

…

On Sat, Sep 5, 2020, 2:33 PM Sukchan Lee ***@***.***> wrote: Hi @bladebeam <https://github.com/bladebeam> and @theorem6 <https://github.com/theorem6> There seems to be some problem with MME. Maybe there was an infinite loop or it was dead. During the development of v2.0.0, I found a suspicious part and mme-init.c was modified for safety. However, until this problem is resolved, ogs_expect() has been modified to be killed like assert(). Could you check if you still have this problem using the master branch now? Yeah! To do this, you need to compile source code. Or do you need a package installer? If so, I will make a package and upload it as soon as possible. Thank you very much for sharing it. Sukhan! — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#472 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AN74XJZMEIW4ZSP5647LIG3SEJ767ANCNFSM4OCX34NQ> .

[acetcom]

To @cabotek

That issue was solved yesterday to solve #548's number 5.

To @bladebeam

1,000 users. That's awesome. The package will be released soon.

To @theorem6
Yeah! All the bugs I know are now fixed in master branch. I'm trying to release new package soon.

Thanks a lot!
Sukchan

[acetcom]

Hi @theorem6, @bladebeam, and @cabotek

Now I have released a new version(v2.0.1). If you're still having issues with this version, please send me the relevant logs.

Thank you very much!
Sukchan

[github-actions]

This issue has been closed automatically due to lack of activity. This has been done to try and reduce the amount of noise. Please do not comment any further. The Open5GS Team may choose to re-open this issue if necessary.