Deadlock when using mod_sms and mod_callcenter together.

[azzbcc]

FreeSWITCH (Version 1.10.3 -release 64bit) on CentOS 7.6

I want to get agent status by using mod_sms, but sometimes FS is deadlocked.

One thread hold lock(cc.mutex) in frame 9 and wait for lock(loadable_modules.mutex) on src/switch_loadable_module.c:2566.

#0  0x00007f5bc1ed850d in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00007f5bc1ed3e76 in _L_lock_941 () from /lib64/libpthread.so.0
#2  0x00007f5bc1ed3d6f in pthread_mutex_lock () from /lib64/libpthread.so.0
#3  0x00007f5bc4421cdb in apr_thread_mutex_lock (mutex=0x2342368) at locks/unix/thread_mutex.c:92
#4  0x00007f5bc42a64ff in switch_mutex_lock (lock=0x2342368) at src/switch_apr.c:301
#5  0x00007f5bc43509c3 in switch_loadable_module_get_database_interface (name=0x7f5b98f4d9e0 "odbc", modname=0x0) at src/switch_loadable_module.c:2566
#6  0x00007f5bc42d567b in _switch_cache_db_get_db_handle_dsn_ex (dbh=0x7f5b98f4dba0, dsn=0x2567210 "odbc://cc", make_module_no_unloadable=SWITCH_FALSE, file=0x7f5b983f7748 "mod_callcenter/mod_callcenter.c", func=0x7f5b983fe680 <__func__.18047> "cc_get_db_handle", line=520) at src/switch_core_sqldb.c:451
#7  0x00007f5bc42d5562 in _switch_cache_db_get_db_handle_dsn (dbh=0x7f5b98f4dba0, dsn=0x2567210 "odbc://cc", file=0x7f5b983f7748 "mod_callcenter/mod_callcenter.c", func=0x7f5b983fe680 <__func__.18047> "cc_get_db_handle", line=520) at src/switch_core_sqldb.c:431
#8  0x00007f5b983e6aba in cc_get_db_handle () from  /usr/local/freeswitch/mod/mod_callcenter.so
#9  0x00007f5b983e7464 in cc_execute_sql2str () from /usr/local/freeswitch/mod/mod_callcenter.so
#10 0x00007f5b983f0f1c in cc_agent_dispatch_thread_run () from /usr/local/freeswitch/mod/mod_callcenter.so
#11 0x00007f5bc4429db4 in dummy_worker (opaque=0x25df6d0) at threadproc/unix/thread.c:151
#12 0x00007f5bc1ed1e65 in start_thread () from /lib64/libpthread.so.0
#13 0x00007f5bc1bfa88d in clone () from /lib64/libc.so.6

The Other thread hold lock(loadable_modules.mutex) in frame 18 on src/switch_loadable_module.c:860 and wait for lock(cc.mutex).

#0  0x00007f5bc1ed850d in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00007f5bc1ed3e76 in _L_lock_941 () from /lib64/libpthread.so.0
#2  0x00007f5bc1ed3d6f in pthread_mutex_lock () from /lib64/libpthread.so.0
#3  0x00007f5bc4421cdb in apr_thread_mutex_lock (mutex=0x25df350) at locks/unix/thread_mutex.c:92
#4  0x00007f5bc42a64ff in switch_mutex_lock (lock=0x25df350) at src/switch_apr.c:301
#5  0x00007f5b983e745f in cc_execute_sql2str () from /usr/local/freeswitch/mod/mod_callcenter.so
#6  0x00007f5b983e87a8 in cc_agent_get () from /usr/local/freeswitch/mod/mod_callcenter.so
#7  0x00007f5b983f497e in cc_config_api_function () from /usr/local/freeswitch/mod/mod_callcenter.so
#8  0x00007f5bc435239f in switch_api_execute (cmd=0x4186a848 "cc_config", arg=0x417189f8 "agent get system tmp_agent", session=0x0, stream=0x7f5b980e1550) at src/switch_loadable_module.c:3010
#9  0x00007f5bc440e985 in API::execute (this=0x7f5bb8009ca0, cmd=0x4186a848 "cc_config", arg=0x417189f8 "agent get system tmp_agent") at src/switch_cpp.cpp:247
#10 0x00007f5ba7bcb0d3 in _wrap_API_execute (L=0x41859378) at mod_lua_wrap.cpp:3577
#11 0x00007f5ba794bbd6 in lj_BC_FUNCC () from /lib64/libluajit-5.1.so.2
#12 0x00007f5ba798f2d0 in lua_pcall () from /lib64/libluajit-5.1.so.2
#13 0x00007f5ba7bbba7d in docall (L=0x41859378, narg=0, nresults=0, perror=0, fatal=1) at mod_lua.cpp:92
#14 0x00007f5ba7bbc23f in lua_parse_and_execute (L=0x41859378, input_code=0x7f5bb8000900 "lua/chat.lua", session=0x0) at mod_lua.cpp:194
#15 0x00007f5ba7bbd652 in lua_chat_function (message=0x7f5b900048f0, data=0x7f5bb80021c0 "lua/chat.lua") at mod_lua.cpp:512
#16 0x00007f5bc434b3a3 in switch_core_execute_chat_app (message=0x7f5b900048f0, app=0x7f5bb8000ba0 "lua", data=0x7f5bb80021c0 "lua/chat.lua") at src/switch_loadable_module.c:1050
#17 0x00007f5ba7dfd396 in chat_send (message_event=0x7f5b900048f0) at mod_sms.c:468
#18 0x00007f5bc434abcf in do_chat_send (message_event=0x7f5b900048f0) at src/switch_loadable_module.c:866
#19 0x00007f5bc434aedd in chat_process_event (eventp=0x7f5b980e1d78) at src/switch_loadable_module.c:931
#20 0x00007f5bc434af58 in chat_thread_run (thread=0x25f5f60, obj=0x2343890) at src/switch_loadable_module.c:948
#21 0x00007f5bc4429db4 in dummy_worker (opaque=0x25f5f60) at threadproc/unix/thread.c:151
#22 0x00007f5bc1ed1e65 in start_thread () from /lib64/libpthread.so.0
#23 0x00007f5bc1bfa88d in clone () from /lib64/libc.so.6

[azzbcc]

edd3663