FreeSwitch1.10.5 with postgressql,all registration return system busy,all calls cannot work suddenly

[Qianjianguo]

Today My FreeSwitch1.10.5 cannot work suddently, when Receive Sip Regiter, FS send back "sent 503 System Busy for REGISTER";and all incoming call and outgoing call cannot work also;

I gdb current freeswitch process with command:
bt
bt full
thread apply all bt
thread apply all bt full

I get the gdb thread trace as following
Thread 181 (Thread 0x7fb52d871700 (LWP 5104)):
#0 0x00007fb5ba00e50d in __lll_lock_wait () from /lib64/libpthread.so.0
No symbol table info available.
#1 0x00007fb5ba009e76 in _L_lock_941 () from /lib64/libpthread.so.0
No symbol table info available.
#2 0x00007fb5ba009d6f in pthread_mutex_lock () from /lib64/libpthread.so.0
No symbol table info available.
#3 0x00007fb5bd815cf9 in apr_thread_mutex_lock (mutex=) at locks/unix/thread_mutex.c:92
No locals.
#4 0x00007fb5bd472d55 in switch_mutex_lock (lock=) at src/switch_apr.c:301
No locals.
#5 0x00007fb5a774651c in sofia_glue_execute_sql_now (profile=profile@entry=0xf73100, sqlp=sqlp@entry=0x7fb52d86ced0, sql_already_dynamic=sql_already_dynamic@entry=SWITCH_TRUE) at sofia_glue.c:2708
sql = 0x7fb5982c50d0 "update sip_dialogs set state='confirmed',presence_id='1251@zeji.com',presence_data='' where uuid='1dfd5f09-9737-4906-b8d2-0d3da9f84671';\n"
PRETTY_FUNCTION = "sofia_glue_execute_sql_now"

It seems the connection from freeswitch to Pgsql DB is dead locked.
is it a bug of FS? the same problem also occurs about 10 days ago. my running environment is very simple, a standard call center application, and the customer use WebRTC(JSSIP) or IPPhone(eyebeam) etc.

No suolution, Now I had use SQL-LIte to replace PgSQL, hope it can work well tomorow. however I hope FS Development Team can attention this problem,and fix it as sooner as possible.

Attahed is my gdb trace

Thanks
Freeswitch_bt_log_20201013.txt

[crisonyx]

Also seeing 'all registrations return system busy' deadlocks using MySQL

Our BT isn't exactly the same but the behavior is:

Freeswitch Version 1.10.5 -release.8 64bit

Thread 304 (Thread 0x14baa8ee9700 (LWP 9898)): #0 0x000014bb31bca2ad in __lll_lock_wait () from /usr/lib64/libpthread.so.0 #1 0x000014bb31bc3c6a in pthread_mutex_lock () from /usr/lib64/libpthread.so.0 #2 0x000014bb166bfbce in sofia_glue_execute_sql_callback () from /usr/lib64/freeswitch/mod/mod_sofia.so #3 0x000014bb16687a33 in cmd_status () from /usr/lib64/freeswitch/mod/mod_sofia.so #4 0x000014bb166795b5 in sofia_function () from /usr/lib64/freeswitch/mod/mod_sofia.so #5 0x000014bb341e516e in switch_api_execute () from /usr/lib64/libfreeswitch.so.1 #6 0x000014bb34171050 in switch_console_execute () from /usr/lib64/libfreeswitch.so.1 #7 0x000014bb16912935 in api_exec () from /usr/lib64/freeswitch/mod/mod_event_socket.so #8 0x000014bb16916aef in parse_command.constprop () from /usr/lib64/freeswitch/mod/mod_event_socket.so #9 0x000014bb16917f8a in listener_run () from /usr/lib64/freeswitch/mod/mod_event_socket.so #10 0x000014bb344ff3e0 in dummy_worker () from /usr/lib64/libfreeswitch.so.1 #11 0x000014bb31bc140b in start_thread () from /usr/lib64/libpthread.so.0 #12 0x000014bb31237eef in clone () from /usr/lib64/libc.so.6

We switched to MySQL from sqlite as our sqlite database was getting corrupted under sustained load.

[FredWH]

How is this issue going?

We met this issue too. when running SIPp stress test, only do register, after several days (or hours), the issue happen. no any registrations. fs_cli -x 'status' is stuck.
tcpdump can capture 503 System Busy response.
Debian 10.12, FreeSWITCH 1.10.7.

[FredWH]

This issue can be reproduced with Debian 10.12, FreeSWITCH 1.10.9.

fs_backtrace-20230625_104044_1.log

[Trapulo]

I've same problem: today my PBX has started responding 503 to every registration request.
switch 1.10.10 (64bit)