FreeSwitch crashs every day

[dxhgq-github]

Hello,

FreeSwitch crashs one or two times every 24 hours , the FS version is 1.10.7 ,the OS is centos 7 ,the only error messages in /var/log/messages is :
Apr 25 12:19:43 localhost systemd: freeswitch.service: main process exited, code=killed, status=8/FPE
Apr 25 12:19:43 localhost systemd: Unit freeswitch.service entered failed state.
Apr 25 12:19:43 localhost systemd: freeswitch.service failed.

Backtrace:

Core was generated by `/usr/local/fs1.10.7/bin/freeswitch -nonat'.
Program terminated with signal SIGFPE, Arithmetic exception.
#0  0x00007f1651eb87e8 in su_block_add () from /usr/local/lib/libsofia-sip-ua.so.0
[Current thread is 1 (LWP 45853)]
(gdb) where
#0  0x00007f36d8db1c3d in su_block_add (b=0x7f36945a3ce0, p=0x7f3694a73170) at su_alloc.c:339
#1  0x00007f36d8db21f2 in sub_alloc (home=0x7f36944eea20, sub=0x7f36945a3ce0, size=80, 
    zero=do_calloc) at su_alloc.c:527
#2  0x00007f36d8db3d2f in su_zalloc (home=0x7f36944eea20, size=80) at su_alloc.c:1574
#3  0x00007f36d8d3dcf9 in outgoing_make_a_aaaa_query (orq=0x7f36945a4a40) at nta.c:10343
#4  0x00007f36d8d3d32e in outgoing_resolve_next (orq=0x7f36945a4a40) at nta.c:10124
#5  0x00007f36d8d3eda7 in outgoing_answer_srv (orq=0x7f36945a4a40, q=0x7f36945a4c20, 
    answers=0x7f369411dfa0) at nta.c:10707
#6  0x00007f36d8da5fd9 in sres_query_report_error (q=0x7f36945a4c20, answers=0x7f369411dfa0)
    at sres.c:2990
#7  0x00007f36d8da6368 in sres_resend_dns_query (res=0x7f3694002960, q=0x7f36945a4c20, 
    timeout=0) at sres.c:3090
#8  0x00007f36d8da725b in sres_resolver_report_error (res=0x7f3694002960, socket=129, 
    errcode=111, remote=0x7f36ab318450, remotelen=16, 
    info=0x7f36ab3183d0 "icmp type=3 code=3 reported by 127.0.0.1") at sres.c:3439
#9  0x00007f36d8da6f29 in sres_resolver_error (res=0x7f3694002960, socket=129) at sres.c:3353
#10 0x00007f36d8db078d in sres_sofia_poll (magic=0x7f36ac001180, w=0x7f3694000ff4, 
    reg=0x7f3694003790) at sresolv.c:357
#11 0x00007f36d8db69ac in su_epoll_port_wait_events (self=0x7f36940008c0, tout=0)
    at su_epoll_port.c:510
#12 0x00007f36d8dbc17f in su_port_wait_events (self=0x7f36940008c0, timeout=0) at su_port.h:318
#13 0x00007f36d8dbd481 in su_root_yield (self=0x7f3694001130) at su_root.c:904
#14 0x00007f36d8d71ffd in nua_stack_timer (nua=0x7f36ac001180, t=0x7f36940011e0, a=0x0)
    at nua_stack.c:739
#15 0x00007f36d8dc69f9 in su_timer_expire (timers=0x7f3694000908, timeout=0x7f36ab318cd8, 
    now=...) at su_timer.c:585
#16 0x00007f36d8db4ee2 in su_base_port_run (self=0x7f36940008c0) at su_base_port.c:339
#17 0x00007f36d8dbc1aa in su_port_run (self=0x7f36940008c0) at su_port.h:326
#18 0x00007f36d8dbd284 in su_root_run (self=0x7f3694001130) at su_root.c:819
#19 0x00007f36d8dbb933 in su_pthread_port_clone_main (varg=0x7f36d8abb4a0)
    at su_pthread_port.c:343
#20 0x00007f36e6ec9e65 in start_thread () from /lib64/libpthread.so.0
#21 0x00007f36e651e88d in __libc_ifunc_impl_list () from /lib64/libc.so.6
#22 0x0000000000000000 in ?? ()

Thanks!

[andywolk]

Please provide a proper gdb log. See Creating A Backtrace With gdb section: https://freeswitch.org/confluence/display/FREESWITCH/Debugging

[andywolk]

What version of the sofia-sip library are you using?

[dxhgq-github]

sofia-sip 1.13.7

[dxhgq-github]

Thread 146 (LWP 63539):
#0 0x00007f1650ef571d in write () from /lib64/libpthread.so.0
#1 0x0000000000000000 in ?? ()

Thread 145 (LWP 45867):
#0 0x00007f1650543e63 in arch_prctl () from /lib64/libc.so.6
#1 0x0000000000000000 in ?? ()

Thread 144 (LWP 64468):
#0 0x00007f1650538bed in fts_read () from /lib64/libc.so.6
#1 0x00007f16542a3607 in rtp_common_read (rtp_session=<error reading variable: Cannot access memory at address 0xfffffe08>, payload_type=<error reading variable: Cannot access memory at address 0xfffffe00>, pmapP=<error reading variable: Cannot access memory at address 0xfffffdf8>, flags=<error reading variable: Cannot access memory at address 0xfffffdf0>, io_flags=<error reading variable: Cannot access memory at address 0xfffffdec>) at src/switch_rtp.c:7571
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 143 (LWP 45890):
#0 0x00007f1650ef29f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f158e8aafe1 in boost::asio::detail::task_io_service::run(boost::system::error_code&) () from /lib64/librocketmq.so
#2 0x00007f158e8ac136 in boost::asio::io_service::run() () from /lib64/librocketmq.so
#3 0x00007f158e98a7cf in thread_proxy () from /lib64/librocketmq.so
#4 0x00007f1650eeee65 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f165054388d in __libc_ifunc_impl_list () from /lib64/libc.so.6
#6 0x0000000000000000 in ?? ()

Thread 142 (LWP 45893):
#0 0x00007f1650ef29f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f158e8aafe1 in boost::asio::detail::task_io_service::run(boost::system::error_code&) () from /lib64/librocketmq.so
#2 0x00007f158e8ac136 in boost::asio::io_service::run() () from /lib64/librocketmq.so
#3 0x00007f158e98a7cf in thread_proxy () from /lib64/librocketmq.so
#4 0x00007f1650eeee65 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f165054388d in __libc_ifunc_impl_list () from /lib64/libc.so.6
#6 0x0000000000000000 in ?? ()

Thread 141 (LWP 65096):
#0 0x00007f165053a953 in setegid () from /lib64/libc.so.6
#1 0x000000000007a120 in ?? ()
#2 0x00007f152a359126 in ?? () from /usr/local/fs1.10.7/lib/freeswitch/mod/mod_callcenter.so
#3 0x0000000002a1be40 in ?? ()
#4 0x00007f1493e53ca0 in ?? ()
#5 0x00007f165431193c in switch_sleep (t=44154432) at src/switch_time.c:644
Backtrace stopped: frame did not save the PC

Thread 140 (LWP 45895):
#0 0x00007f1650ef29f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f158e8aafe1 in boost::asio::detail::task_io_service::run(boost::system::error_code&) () from /lib64/librocketmq.so
#2 0x00007f158e8ac136 in boost::asio::io_service::run() () from /lib64/librocketmq.so
#3 0x00007f158e98a7cf in thread_proxy () from /lib64/librocketmq.so
#4 0x00007f1650eeee65 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f165054388d in __libc_ifunc_impl_list () from /lib64/libc.so.6
#6 0x0000000000000000 in ?? ()

Thread 139 (LWP 45905):
#0 0x00007f165050a80d in sleep () from /lib64/libc.so.6
#1 0x0000000000000000 in ?? ()

Thread 138 (LWP 45876):
#0 0x00007f165053a953 in setegid () from /lib64/libc.so.6
#1 0x00000000000f4240 in ?? ()
#2 0x0000000000000000 in ?? ()

Thread 137 (LWP 45879):
#0 0x00007f1650ef571d in write () from /lib64/libpthread.so.0
#1 0x0000000000000000 in ?? ()

Thread 136 (LWP 45880):
#0 0x00007f1650ef571d in write () from /lib64/libpthread.so.0
#1 0x0000000000000000 in ?? ()

Thread 135 (LWP 45883):
#0 0x00007f1650543e63 in arch_prctl () from /lib64/libc.so.6
#1 0x0000000000000000 in ?? ()

Thread 134 (LWP 64687):
#0 0x00007f1650ef571d in write () from /lib64/libpthread.so.0
#1 0x0000000000000000 in ?? ()

Thread 133 (LWP 45886):
#0 0x00007f1650ef29f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f158e8aafe1 in boost::asio::detail::task_io_service::run(boost::system::error_code&) () from /lib64/librocketmq.so
#2 0x00007f158e8ac136 in boost::asio::io_service::run() () from /lib64/librocketmq.so
#3 0x00007f158e98a7cf in thread_proxy () from /lib64/librocketmq.so
#4 0x00007f1650eeee65 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f165054388d in __libc_ifunc_impl_list () from /lib64/libc.so.6
#6 0x0000000000000000 in ?? ()

Thread 132 (LWP 63929):
#0 0x00007f1650ef2da2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f1654549702 in apr_thread_cond_timedwait (cond=0x7f1654c38f48, mutex=0x7f1654c38ef8, timeout=5000000) at locks/unix/thread_cond.c:89
#2 0x00007f1654540dd9 in apr_queue_pop_timeout (queue=0x7f1654c38eb8, data=0x7f1493fc1d38, timeout=5000000) at misc/apr_queue.c:339
#3 0x00007f16541a8c64 in switch_queue_pop_timeout (queue=0x7f1654c38eb8, data=0x7f1493fc1d38, timeout=5000000) at src/switch_apr.c:1225
#4 0x00007f16541e8709 in switch_core_session_thread_pool_worker (thread=0x7f1628382dc0, obj=0x7f1628382c50) at src/switch_core_session.c:1792
#5 0x00007f165454f770 in dummy_worker (opaque=0x7f1628382dc0) at threadproc/unix/thread.c:151
#6 0x00007f1650eeee65 in start_thread () from /lib64/libpthread.so.0
#7 0x00007f165054388d in __libc_ifunc_impl_list () from /lib64/libc.so.6
#8 0x0000000000000000 in ?? ()

Thread 131 (LWP 64389):
#0 0x00007f1650ef29f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f16541a740f in switch_thread_cond_wait (cond=0x7f16180a1210, mutex=0x7f16180a11c0) at src/switch_apr.c:371
#2 0x00007f16542cf5d2 in recording_thread (thread=0x7f16180a13d0, obj=0x7f16180af708) at src/switch_ivr_async.c:1329
#3 0x00007f165454f770 in dummy_worker (opaque=0x7f16180a13d0) at threadproc/unix/thread.c:151
#4 0x00007f1650eeee65 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f165054388d in __libc_ifunc_impl_list () from /lib64/libc.so.6
#6 0x0000000000000000 in ?? ()

Thread 130 (LWP 64075):
#0 0x00007f1650ef571d in write () from /lib64/libpthread.so.0
#1 0x0000000000000000 in ?? ()

Thread 129 (LWP 64456):
#0 0x00007f16505598c4 in __get_clockfreq () from /lib64/libc.so.6
#1 0x0000000000000000 in ?? ()

Thread 128 (LWP 64839):
#0 0x00007f165053a953 in setegid () from /lib64/libc.so.6
#1 0x000000000007a120 in ?? ()
#2 0x00007f152a359126 in ?? () from /usr/local/fs1.10.7/lib/freeswitch/mod/mod_callcenter.so
#3 0x0000000002a1be40 in ?? ()
#4 0x00007f1492dacca0 in ?? ()
#5 0x00007f165431193c in switch_sleep (t=44154432) at src/switch_time.c:644
Backtrace stopped: frame did not save the PC

Thread 127 (LWP 64774):
#0 0x00007f1650ef29f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f16541a740f in switch_thread_cond_wait (cond=0x7f15ac05ccc8, mutex=0x7f15ac05cc78) at src/switch_apr.c:371
#2 0x00007f16542cf5d2 in recording_thread (thread=0x7f15ac05ce88, obj=0x7f15ac032d78) at src/switch_ivr_async.c:1329
#3 0x00007f165454f770 in dummy_worker (opaque=0x7f15ac05ce88) at threadproc/unix/thread.c:151
#4 0x00007f1650eeee65 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f165054388d in __libc_ifunc_impl_list () from /lib64/libc.so.6
#6 0x0000000000000000 in ?? ()

Thread 126 (LWP 64469):
#0 0x00007f1650ef29f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f16541a740f in switch_thread_cond_wait (cond=0x7f149c02aff0, mutex=0x7f149c02afa0) at src/switch_apr.c:371
#2 0x00007f16542cf5d2 in recording_thread (thread=0x7f149c02b1b0, obj=0x7f149cb4cbe8) at src/switch_ivr_async.c:1329
#3 0x00007f165454f770 in dummy_worker (opaque=0x7f149c02b1b0) at threadproc/unix/thread.c:151
#4 0x00007f1650eeee65 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f165054388d in __libc_ifunc_impl_list () from /lib64/libc.so.6
#6 0x0000000000000000 in ?? ()

Thread 125 (LWP 65053):
#0 0x00007f1650ef571d in write () from /lib64/libpthread.so.0
#1 0x0000000000000000 in ?? ()

Thread 124 (LWP 65073):
#0 0x00007f1650ef571d in write () from /lib64/libpthread.so.0
#1 0x0000000000000000 in ?? ()

Thread 123 (LWP 65095):
#0 0x00007f165053a953 in setegid () from /lib64/libc.so.6
#1 0x0000000003473bc0 in ?? ()
#2 0x0000000000000000 in ?? ()

Thread 122 (LWP 65104):
#0 0x00007f165053a953 in setegid () from /lib64/libc.so.6
#1 0x000000000007a120 in ?? ()
#2 0x00007f152a359126 in ?? () from /usr/local/fs1.10.7/lib/freeswitch/mod/mod_callcenter.so
#3 0x0000000002a1be40 in ?? ()
#4 0x00007f155c145ca0 in ?? ()
#5 0x00007f165431193c in switch_sleep (t=44154432) at src/switch_time.c:644
Backtrace stopped: frame did not save the PC

Thread 121 (LWP 45898):
#0 0x00007f1650543e63 in arch_prctl () from /lib64/libc.so.6
#1 0x0000000000000000 in ?? ()

Thread 120 (LWP 64225):
#0 0x00007f1650ef29f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f16541a740f in switch_thread_cond_wait (cond=0x7f150c041948, mutex=0x7f150c0418f8) at src/switch_apr.c:371
#2 0x00007f16542cf5d2 in recording_thread (thread=0x7f150c041b08, obj=0x7f150c0d0098) at src/switch_ivr_async.c:1329
#3 0x00007f165454f770 in dummy_worker (opaque=0x7f150c041b08) at threadproc/unix/thread.c:151
#4 0x00007f1650eeee65 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f165054388d in __libc_ifunc_impl_list () from /lib64/libc.so.6
#6 0x0000000000000000 in ?? ()

Thread 119 (LWP 64452):
#0 0x00007f1650ef29f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f16541a740f in switch_thread_cond_wait (cond=0x7f1498ab2a80, mutex=0x7f1498ab2a30) at src/switch_apr.c:371
#2 0x00007f16542cf5d2 in recording_thread (thread=0x7f1498ab2c40, obj=0x7f1498730808) at src/switch_ivr_async.c:1329
#3 0x00007f165454f770 in dummy_worker (opaque=0x7f1498ab2c40) at threadproc/unix/thread.c:151
#4 0x00007f1650eeee65 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f165054388d in __libc_ifunc_impl_list () from /lib64/libc.so.6
#6 0x0000000000000000 in ?? ()

Thread 118 (LWP 64451):
#0 0x00007f1650538bed in fts_read () from /lib64/libc.so.6
#1 0x00007f16542a3607 in rtp_common_read (rtp_session=<error reading variable: Cannot access memory at address 0xfffffe08>, payload_type=<error reading variable: Cannot access memory at address 0xfffffe00>, pmapP=<error reading variable: Cannot access memory at address 0xfffffdf8>, flags=<error reading variable: Cannot access memory at address 0xfffffdf0>, io_flags=<error reading variable: Cannot access memory at address 0xfffffdec>) at src/switch_rtp.c:7571
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 117 (LWP 65040):
#0 0x00007f1650ef571d in write () from /lib64/libpthread.so.0
#1 0x0000000000000000 in ?? ()

Thread 116 (LWP 65059):
#0 0x00007f165053a953 in setegid () from /lib64/libc.so.6
#1 0x0000000003473bc0 in ?? ()
#2 0x0000000000000000 in ?? ()

Thread 115 (LWP 45877):
#0 0x00007f1650ef29f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f16541a740f in switch_thread_cond_wait (cond=0x2af4f58, mutex=0x2af4f08) at src/switch_apr.c:371
#2 0x00007f15ccec65cd in timer_thread_run (thread=0x2af5118, obj=0x0) at mod_spandsp_fax.c:223
#3 0x00007f165454f770 in dummy_worker (opaque=0x2af5118) at threadproc/unix/thread.c:151
#4 0x00007f1650eeee65 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f165054388d in __libc_ifunc_impl_list () from /lib64/libc.so.6
#6 0x0000000000000000 in ?? ()

Thread 114 (LWP 45882):
#0 0x00007f1650ef571d in write () from /lib64/libpthread.so.0
#1 0x0000000000000000 in ?? ()

Thread 113 (LWP 65214):
#0 0x00007f1650ef571d in write () from /lib64/libpthread.so.0
#1 0x0000000000000000 in ?? ()

Thread 112 (LWP 45884):
#0 0x00007f1650ef29f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f158e8aafe1 in boost::asio::detail::task_io_service::run(boost::system::error_code&) () from /lib64/librocketmq.so
#2 0x00007f158e8ac136 in boost::asio::io_service::run() () from /lib64/librocketmq.so
#3 0x00007f158e98a7cf in thread_proxy () from /lib64/librocketmq.so
#4 0x00007f1650eeee65 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f165054388d in __libc_ifunc_impl_list () from /lib64/libc.so.6
#6 0x0000000000000000 in ?? ()

Thread 111 (LWP 45885):
#0 0x00007f1650ef29f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f158e8aafe1 in boost::asio::detail::task_io_service::run(boost::system::error_code&) () from /lib64/librocketmq.so
#2 0x00007f158e8ac136 in boost::asio::io_service::run() () from /lib64/librocketmq.so
#3 0x00007f158e98a7cf in thread_proxy () from /lib64/librocketmq.so
#4 0x00007f1650eeee65 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f165054388d in __libc_ifunc_impl_list () from /lib64/libc.so.6
#6 0x0000000000000000 in ?? ()

Thread 110 (LWP 65071):
#0 0x00007f1650ef2da2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f1654549702 in apr_thread_cond_timedwait (cond=0x7f1654c38f48, mutex=0x7f1654c38ef8, timeout=5000000) at locks/unix/thread_cond.c:89
#2 0x00007f1654540dd9 in apr_queue_pop_timeout (queue=0x7f1654c38eb8, data=0x7f1493123d38, timeout=5000000) at misc/apr_queue.c:339
#3 0x00007f16541a8c64 in switch_queue_pop_timeout (queue=0x7f1654c38eb8, data=0x7f1493123d38, timeout=5000000) at src/switch_apr.c:1225
#4 0x00007f16541e8709 in switch_core_session_thread_pool_worker (thread=0x7f14f002fb50, obj=0x7f14f002f9e0) at src/switch_core_session.c:1792
#5 0x00007f165454f770 in dummy_worker (opaque=0x7f14f002fb50) at threadproc/unix/thread.c:151
#6 0x00007f1650eeee65 in start_thread () from /lib64/libpthread.so.0
#7 0x00007f165054388d in __libc_ifunc_impl_list () from /lib64/libc.so.6
#8 0x0000000000000000 in ?? ()

Thread 109 (LWP 65077):
#0 0x00007f1650ef29f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f16541a740f in switch_thread_cond_wait (cond=0x7f14f893b140, mutex=0x7f14f893b0f0) at src/switch_apr.c:371
#2 0x00007f16542cf5d2 in recording_thread (thread=0x7f14f893b300, obj=0x7f14f806ec08) at src/switch_ivr_async.c:1329
#3 0x00007f165454f770 in dummy_worker (opaque=0x7f14f893b300) at threadproc/unix/thread.c:151
#4 0x00007f1650eeee65 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f165054388d in __libc_ifunc_impl_list () from /lib64/libc.so.6
#6 0x0000000000000000 in ?? ()

Thread 108 (LWP 45887):
#0 0x00007f1650ef29f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f158e8aafe1 in boost::asio::detail::task_io_service::run(boost::system::error_code&) () from /lib64/librocketmq.so
#2 0x00007f158e8ac136 in boost::asio::io_service::run() () from /lib64/librocketmq.so
#3 0x00007f158e98a7cf in thread_proxy () from /lib64/librocketmq.so
#4 0x00007f1650eeee65 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f165054388d in __libc_ifunc_impl_list () from /lib64/libc.so.6
#6 0x0000000000000000 in ?? ()

Thread 107 (LWP 45888):
#0 0x00007f1650ef29f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f158e8aafe1 in boost::asio::detail::task_io_service::run(boost::system::error_code&) () from /lib64/librocketmq.so
#2 0x00007f158e8ac136 in boost::asio::io_service::run() () from /lib64/librocketmq.so
#3 0x00007f158e98a7cf in thread_proxy () from /lib64/librocketmq.so
#4 0x00007f1650eeee65 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f165054388d in __libc_ifunc_impl_list () from /lib64/libc.so.6
#6 0x0000000000000000 in ?? ()

Thread 106 (LWP 45889):
#0 0x00007f1650ef29f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f158e8aafe1 in boost::asio::detail::task_io_service::run(boost::system::error_code&) () from /lib64/librocketmq.so
#2 0x00007f158e8ac136 in boost::asio::io_service::run() () from /lib64/librocketmq.so
#3 0x00007f158e98a7cf in thread_proxy () from /lib64/librocketmq.so
#4 0x00007f1650eeee65 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f165054388d in __libc_ifunc_impl_list () from /lib64/libc.so.6
#6 0x0000000000000000 in ?? ()

Thread 105 (LWP 45891):
#0 0x00007f1650ef29f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f158e8aafe1 in boost::asio::detail::task_io_service::run(boost::system::error_code&) () from /lib64/librocketmq.so
#2 0x00007f158e8ac136 in boost::asio::io_service::run() () from /lib64/librocketmq.so
#3 0x00007f158e98a7cf in thread_proxy () from /lib64/librocketmq.so
#4 0x00007f1650eeee65 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f165054388d in __libc_ifunc_impl_list () from /lib64/libc.so.6
#6 0x0000000000000000 in ?? ()

Thread 104 (LWP 45892):
#0 0x00007f1650ef29f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f158e8aafe1 in boost::asio::detail::task_io_service::run(boost::system::error_code&) () from /lib64/librocketmq.so
#2 0x00007f158e8ac136 in boost::asio::io_service::run() () from /lib64/librocketmq.so
#3 0x00007f158e98a7cf in thread_proxy () from /lib64/librocketmq.so
#4 0x00007f1650eeee65 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f165054388d in __libc_ifunc_impl_list () from /lib64/libc.so.6
#6 0x0000000000000000 in ?? ()

Thread 103 (LWP 45894):
#0 0x00007f1650ef29f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f158e8aafe1 in boost::asio::detail::task_io_service::run(boost::system::error_code&) () from /lib64/librocketmq.so
#2 0x00007f158e8ac136 in boost::asio::io_service::run() () from /lib64/librocketmq.so
#3 0x00007f158e98a7cf in thread_proxy () from /lib64/librocketmq.so
#4 0x00007f1650eeee65 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f165054388d in __libc_ifunc_impl_list () from /lib64/libc.so.6
#6 0x0000000000000000 in ?? ()

Thread 102 (LWP 45896):
#0 0x00007f1650ef29f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f158e8aafe1 in boost::asio::detail::task_io_service::run(boost::system::error_code&) () from /lib64/librocketmq.so
#2 0x00007f158e8ac136 in boost::asio::io_service::run() () from /lib64/librocketmq.so
#3 0x00007f158e98a7cf in thread_proxy () from /lib64/librocketmq.so
#4 0x00007f1650eeee65 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f165054388d in __libc_ifunc_impl_list () from /lib64/libc.so.6
#6 0x0000000000000000 in ?? ()

Thread 101 (LWP 45897):
#0 0x00007f1650ef29f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f158e8aafe1 in boost::asio::detail::task_io_service::run(boost::system::error_code&) () from /lib64/librocketmq.so
#2 0x00007f158e932b15 in rocketmq::TcpRemotingClient::boost_asio_work() () from /lib64/librocketmq.so
#3 0x00007f158e98a7cf in thread_proxy () from /lib64/librocketmq.so
#4 0x00007f1650eeee65 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f165054388d in __libc_ifunc_impl_list () from /lib64/libc.so.6
#6 0x0000000000000000 in ?? ()

Thread 100 (LWP 65207):
#0 0x00007f165053a953 in setegid () from /lib64/libc.so.6
#1 0x0000000003473bc0 in ?? ()
#2 0x0000000000000000 in ?? ()

Thread 99 (LWP 65166):
#0 0x00007f165053a953 in setegid () from /lib64/libc.so.6
#1 0x0000000003473bc0 in ?? ()
#2 0x0000000000000000 in ?? ()

Thread 98 (LWP 65060):
#0 0x00007f165053a953 in setegid () from /lib64/libc.so.6
#1 0x000000000007a120 in ?? ()
#2 0x00007f152a359126 in ?? () from /usr/local/fs1.10.7/lib/freeswitch/mod/mod_callcenter.so
#3 0x0000000002a1be40 in ?? ()
#4 0x00007f1529095ca0 in ?? ()
#5 0x00007f165431193c in switch_sleep (t=44154432) at src/switch_time.c:644
Backtrace stopped: frame did not save the PC

Thread 97 (LWP 45944):
#0 0x00007f1650538bed in fts_read () from /lib64/libc.so.6
#1 0x00007f1648f81b49 in read_packet (listener=<error reading variable: Cannot access memory at address 0x15c08d818>, event=<error reading variable: Cannot access memory at address 0x15c08d810>, timeout=<error reading variable: Cannot access memory at address 0x15c08d80c>) at mod_event_socket.c:1496
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 96 (LWP 45986):
#0 0x00007f1650ef29f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f165454969d in apr_thread_cond_wait (cond=, mutex=) at locks/unix/thread_cond.c:68
#2 0x00007f1654540ce1 in apr_queue_pop (queue=0x7f164c6008e8, data=0x7f15296abcf0) at misc/apr_queue.c:276
#3 0x00007f16541a8c37 in switch_queue_pop (queue=0x7f164c6008e8, data=0x7f15296abcf0) at src/switch_apr.c:1220
#4 0x00007f1654288b9e in switch_event_channel_deliver_thread (thread=0x7f1640010b10, obj=0x7f164c6008e8) at src/switch_event.c:3001
#5 0x00007f16541e8786 in switch_core_session_thread_pool_worker (thread=0x7f1640010b10, obj=0x7f16400109a0) at src/switch_core_session.c:1800
#6 0x00007f165454f770 in dummy_worker (opaque=0x7f1640010b10) at threadproc/unix/thread.c:151
#7 0x00007f1650eeee65 in start_thread () from /lib64/libpthread.so.0
#8 0x00007f165054388d in __libc_ifunc_impl_list () from /lib64/libc.so.6
#9 0x0000000000000000 in ?? ()

Thread 95 (LWP 61545):
#0 0x00007f1650ef571d in write () from /lib64/libpthread.so.0
#1 0x0000000000000000 in ?? ()

Thread 94 (LWP 63430):
#0 0x00007f1650ef571d in write () from /lib64/libpthread.so.0
#1 0x0000000000000000 in ?? ()

Thread 93 (LWP 63542):
#0 0x00007f1650ef571d in write () from /lib64/libpthread.so.0
#1 0x0000000000000000 in ?? ()

Thread 92 (LWP 63867):
#0 0x00007f1650ef571d in write () from /lib64/libpthread.so.0
#1 0x0000000000000000 in ?? ()

Thread 91 (LWP 64379):
#0 0x00007f1650ef571d in write () from /lib64/libpthread.so.0
#1 0x0000000000000000 in ?? ()

Thread 90 (LWP 64450):
#0 0x00007f16505598c4 in __get_clockfreq () from /lib64/libc.so.6
#1 0x0000000000000000 in ?? ()

Thread 89 (LWP 63971):
#0 0x00007f1650ef571d in write () from /lib64/libpthread.so.0
#1 0x0000000000000000 in ?? ()

Thread 88 (LWP 63972):
#0 0x00007f1650ef571d in write () from /lib64/libpthread.so.0
#1 0x0000000000000000 in ?? ()

Thread 87 (LWP 64092):
#0 0x00007f1650ef29f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f16541a740f in switch_thread_cond_wait (cond=0x7f15bc4bf400, mutex=0x7f15bc4bf3b0) at src/switch_apr.c:371
#2 0x00007f16542cf5d2 in recording_thread (thread=0x7f15bc4bf5c0, obj=0x7f15bc196e08) at src/switch_ivr_async.c:1329
#3 0x00007f165454f770 in dummy_worker (opaque=0x7f15bc4bf5c0) at threadproc/unix/thread.c:151
#4 0x00007f1650eeee65 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f165054388d in __libc_ifunc_impl_list () from /lib64/libc.so.6
#6 0x0000000000000000 in ?? ()

Thread 86 (LWP 65167):
#0 0x00007f165053a953 in setegid () from /lib64/libc.so.6
#1 0x000000000007a120 in ?? ()
#2 0x00007f152a359126 in ?? () from /usr/local/fs1.10.7/lib/freeswitch/mod/mod_callcenter.so
#3 0x0000000002a1be40 in ?? ()
#4 0x00007f1493282ca0 in ?? ()
#5 0x00007f165431193c in switch_sleep (t=44154432) at src/switch_time.c:644
Backtrace stopped: frame did not save the PC

Thread 85 (LWP 65151):
#0 0x00007f165053a953 in setegid () from /lib64/libc.so.6
#1 0x000000000007a120 in ?? ()
#2 0x00007f152a359126 in ?? () from /usr/local/fs1.10.7/lib/freeswitch/mod/mod_callcenter.so
#3 0x0000000002a1be40 in ?? ()
#4 0x00007f14935bcca0 in ?? ()
#5 0x00007f165431193c in switch_sleep (t=44154432) at src/switch_time.c:644
Backtrace stopped: frame did not save the PC

Thread 84 (LWP 65212):
#0 0x00007f1650ef571d in write () from /lib64/libpthread.so.0
#1 0x0000000000000000 in ?? ()

Thread 83 (LWP 64561):
#0 0x00007f1650ef571d in write () from /lib64/libpthread.so.0
#1 0x0000000000000000 in ?? ()

Thread 82 (LWP 64700):
#0 0x00007f1650ef571d in write () from /lib64/libpthread.so.0
#1 0x0000000000000000 in ?? ()

Thread 81 (LWP 64703):
#0 0x00007f1650ef571d in write () from /lib64/libpthread.so.0
#1 0x0000000000000000 in ?? ()

Thread 80 (LWP 64838):
#0 0x00007f165053a953 in setegid () from /lib64/libc.so.6
#1 0x0000000003473bc0 in ?? ()
#2 0x0000000000000000 in ?? ()

Thread 79 (LWP 64855):
#0 0x00007f165053a953 in setegid () from /lib64/libc.so.6
#1 0x0000000003473bc0 in ?? ()
#2 0x0000000000000000 in ?? ()

Thread 78 (LWP 64856):
#0 0x00007f165053a953 in setegid () from /lib64/libc.so.6
#1 0x000000000007a120 in ?? ()
#2 0x00007f152a359126 in ?? () from /usr/local/fs1.10.7/lib/freeswitch/mod/mod_callcenter.so
#3 0x0000000002dcc570 in ?? ()
#4 0x00007f14939f7ca0 in ?? ()
#5 0x00007f165431193c in switch_sleep (t=48022896) at src/switch_time.c:644
Backtrace stopped: frame did not save the PC

Thread 77 (LWP 64928):
#0 0x00007f1650ef571d in write () from /lib64/libpthread.so.0
#1 0x0000000000000000 in ?? ()

Thread 76 (LWP 64945):
#0 0x00007f165053a953 in setegid () from /lib64/libc.so.6
#1 0x0000000003473bc0 in ?? ()
#2 0x0000000000000000 in ?? ()

Thread 75 (LWP 65048):
#0 0x00007f1650ef571d in write () from /lib64/libpthread.so.0
#1 0x0000000000000000 in ?? ()

Thread 74 (LWP 65062):
#0 0x00007f165053a953 in setegid () from /lib64/libc.so.6
#1 0x0000000003473bc0 in ?? ()
#2 0x0000000000000000 in ?? ()

Thread 73 (LWP 65066):
#0 0x00007f165053a953 in setegid () from /lib64/libc.so.6
#1 0x0000000003473bc0 in ?? ()
#2 0x0000000000000000 in ?? ()

Thread 72 (LWP 65072):
#0 0x00007f16505598c4 in __get_clockfreq () from /lib64/libc.so.6
#1 0x0000000000000000 in ?? ()

Thread 71 (LWP 65063):
#0 0x00007f165053a953 in setegid () from /lib64/libc.so.6
#1 0x000000000007a120 in ?? ()
#2 0x00007f152a359126 in ?? () from /usr/local/fs1.10.7/lib/freeswitch/mod/mod_callcenter.so
#3 0x0000000002a1be40 in ?? ()
#4 0x00007f1528f3fca0 in ?? ()
#5 0x00007f165431193c in switch_sleep (t=44154432) at src/switch_time.c:644
Backtrace stopped: frame did not save the PC

Thread 70 (LWP 65213):
#0 0x00007f1650ef29f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f16541a740f in switch_thread_cond_wait (cond=0x7f157000c780, mutex=0x7f157000c730) at src/switch_apr.c:371
#2 0x00007f16542cf5d2 in recording_thread (thread=0x7f157000c940, obj=0x7f1570a7f758) at src/switch_ivr_async.c:1329
#3 0x00007f165454f770 in dummy_worker (opaque=0x7f157000c940) at threadproc/unix/thread.c:151
#4 0x00007f1650eeee65 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f165054388d in __libc_ifunc_impl_list () from /lib64/libc.so.6
#6 0x0000000000000000 in ?? ()

Thread 69 (Thread 0x7f1654c9e8c0 (LWP 45827)):
#0 0x00007f165053a953 in setegid () from /lib64/libc.so.6
#1 0x00000000000f4240 in ?? ()
#2 0x00007fff556c0910 in ?? ()
#3 0x0000000002df4ca8 in ?? ()
#4 0x00007fff556c0910 in ?? ()
#5 0x00007f165431193c in switch_sleep (t=48188584) at src/switch_time.c:644
Backtrace stopped: frame did not save the PC

Thread 68 (LWP 45836):
#0 0x00007f1650538bed in fts_read () from /lib64/libc.so.6
#1 0x0000000000000005 in ?? ()
#2 0xffffffffffffffff in ?? ()
#3 0x0000000000000000 in ?? ()

Thread 67 (LWP 65087):
#0 0x00007f165053a953 in setegid () from /lib64/libc.so.6
#1 0x0000000003473bc0 in ?? ()
#2 0x0000000000000000 in ?? ()

Thread 66 (LWP 65101):
#0 0x00007f1650ef571d in write () from /lib64/libpthread.so.0
#1 0x0000000000000000 in ?? ()

Thread 65 (LWP 65103):
#0 0x00007f165053a953 in setegid () from /lib64/libc.so.6
#1 0x0000000003473bc0 in ?? ()
#2 0x0000000000000000 in ?? ()

Thread 64 (LWP 65108):
#0 0x00007f165053a953 in setegid () from /lib64/libc.so.6
#1 0x0000000003473bc0 in ?? ()
#2 0x0000000000000000 in ?? ()

Thread 63 (LWP 65113):
#0 0x00007f1650ef571d in write () from /lib64/libpthread.so.0
#1 0x0000000000000000 in ?? ()

Thread 62 (LWP 65161):
#0 0x00007f1650ef29f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f16541a740f in switch_thread_cond_wait (cond=0x7f14d0004300, mutex=0x7f14d00042b0) at src/switch_apr.c:371
#2 0x00007f16542cf5d2 in recording_thread (thread=0x7f14d00044c0, obj=0x7f14d000ca28) at src/switch_ivr_async.c:1329
#3 0x00007f165454f770 in dummy_worker (opaque=0x7f14d00044c0) at threadproc/unix/thread.c:151
#4 0x00007f1650eeee65 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f165054388d in __libc_ifunc_impl_list () from /lib64/libc.so.6
#6 0x0000000000000000 in ?? ()

Thread 61 (LWP 65208):
#0 0x00007f165053a953 in setegid () from /lib64/libc.so.6
#1 0x000000000007a120 in ?? ()
#2 0x00007f152a359126 in ?? () from /usr/local/fs1.10.7/lib/freeswitch/mod/mod_callcenter.so
#3 0x0000000002a1be40 in ?? ()
#4 0x00007f1529b42ca0 in ?? ()
#5 0x00007f165431193c in switch_sleep (t=44154432) at src/switch_time.c:644
Backtrace stopped: frame did not save the PC

Thread 60 (LWP 65150):
#0 0x00007f165053a953 in setegid () from /lib64/libc.so.6
#1 0x0000000003473bc0 in ?? ()
#2 0x0000000000000000 in ?? ()

Thread 59 (LWP 65160):
#0 0x00007f1650ef571d in write () from /lib64/libpthread.so.0
#1 0x0000000000000000 in ?? ()

Thread 58 (LWP 65162):
#0 0x00007f1650ef571d in write () from /lib64/libpthread.so.0
#1 0x0000000000000000 in ?? ()

Thread 57 (LWP 45830):
#0 0x00007f165053a953 in setegid () from /lib64/libc.so.6
#1 0x00000000000f4240 in ?? ()
#2 0x000000000003d000 in ?? ()
#3 0x00007f1654c38c40 in ?? ()
#4 0x00007f1654bd5d20 in ?? ()
#5 0x00007f165431193c in switch_sleep (t=139733888109632) at src/switch_time.c:644
Backtrace stopped: frame did not save the PC

Thread 56 (LWP 45832):
#0 0x00007f1650538bed in fts_read () from /lib64/libc.so.6
#1 0x0000000000000000 in ?? ()

Thread 55 (LWP 65215):
#0 0x00007f1650ef571d in write () from /lib64/libpthread.so.0
#1 0x0000000000000000 in ?? ()

Thread 54 (LWP 65216):
#0 0x00007f1650ef29f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f16541a740f in switch_thread_cond_wait (cond=0x7f15840402e0, mutex=0x7f1584040290) at src/switch_apr.c:371
#2 0x00007f16542cf5d2 in recording_thread (thread=0x7f15840404a0, obj=0x7f1584b08c28) at src/switch_ivr_async.c:1329
#3 0x00007f165454f770 in dummy_worker (opaque=0x7f15840404a0) at threadproc/unix/thread.c:151
#4 0x00007f1650eeee65 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f165054388d in __libc_ifunc_impl_list () from /lib64/libc.so.6
#6 0x0000000000000000 in ?? ()

Thread 53 (LWP 65217):
#0 0x00007f1650ef571d in write () from /lib64/libpthread.so.0
#1 0x0000000000000000 in ?? ()

Thread 52 (LWP 45833):
#0 0x00007f1650ef29f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f165454969d in apr_thread_cond_wait (cond=, mutex=) at locks/unix/thread_cond.c:68
#2 0x00007f1654540ce1 in apr_queue_pop (queue=0x7f164c7dbec8, data=0x7f164c53bd50) at misc/apr_queue.c:276
#3 0x00007f16541a8c37 in switch_queue_pop (queue=0x7f164c7dbec8, data=0x7f164c53bd50) at src/switch_apr.c:1220
#4 0x00007f16543029b3 in log_thread (t=0x7f164c6005a8, obj=0x0) at src/switch_log.c:501
#5 0x00007f165454f770 in dummy_worker (opaque=0x7f164c6005a8) at threadproc/unix/thread.c:151
#6 0x00007f1650eeee65 in start_thread () from /lib64/libpthread.so.0
#7 0x00007f165054388d in __libc_ifunc_impl_list () from /lib64/libc.so.6
#8 0x0000000000000000 in ?? ()

Thread 51 (LWP 45851):
#0 0x00007f1650ef2da2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f1654549702 in apr_thread_cond_timedwait (cond=0x7f1648cb3b68, mutex=0x7f1648cb3b18, timeout=100000) at locks/unix/thread_cond.c:89
#2 0x00007f1654540dd9 in apr_queue_pop_timeout (queue=0x7f1648cb3ad8, data=0x7f162cf08d18, timeout=100000) at misc/apr_queue.c:339
#3 0x00007f16541a8c64 in switch_queue_pop_timeout (queue=0x7f1648cb3ad8, data=0x7f162cf08d18, timeout=100000) at src/switch_apr.c:1225
#4 0x00007f1648cf08fe in sofia_profile_worker_thread_run (thread=0x7f1648295b40, obj=0x29b4ff0) at sofia.c:3010
#5 0x00007f165454f770 in dummy_worker (opaque=0x7f1648295b40) at threadproc/unix/thread.c:151
#6 0x00007f1650eeee65 in start_thread () from /lib64/libpthread.so.0
#7 0x00007f165054388d in __libc_ifunc_impl_list () from /lib64/libc.so.6
#8 0x0000000000000000 in ?? ()

Thread 50 (LWP 45870):
#0 0x00007f1650543e63 in arch_prctl () from /lib64/libc.so.6
#1 0x0000000000000000 in ?? ()

Thread 49 (LWP 64464):
#0 0x00007f1650ef29f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f16541a740f in switch_thread_cond_wait (cond=0x7f14a4020d78, mutex=0x7f14a4020d28) at src/switch_apr.c:371
#2 0x00007f16542cf5d2 in recording_thread (thread=0x7f14a4020f38, obj=0x7f14a4132318) at src/switch_ivr_async.c:1329
#3 0x00007f165454f770 in dummy_worker (opaque=0x7f14a4020f38) at threadproc/unix/thread.c:151
#4 0x00007f1650eeee65 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f165054388d in __libc_ifunc_impl_list () from /lib64/libc.so.6
#6 0x0000000000000000 in ?? ()

Thread 48 (LWP 45834):
#0 0x00007f1650ef2da2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f1654549702 in apr_thread_cond_timedwait (cond=0x2733118, mutex=0x27330c8, timeout=500000) at locks/unix/thread_cond.c:89
#2 0x00007f1654540dd9 in apr_queue_pop_timeout (queue=0x2733088, data=0x7f164c314d58, timeout=500000) at misc/apr_queue.c:339
#3 0x00007f16541a8c64 in switch_queue_pop_timeout (queue=0x2733088, data=0x7f164c314d58, timeout=500000) at src/switch_apr.c:1225
#4 0x00007f16542690a9 in switch_scheduler_task_thread (thread=0x7f164c4fe5d8, obj=0x0) at src/switch_scheduler.c:191
#5 0x00007f165454f770 in dummy_worker (opaque=0x7f164c4fe5d8) at threadproc/unix/thread.c:151
#6 0x00007f1650eeee65 in start_thread () from /lib64/libpthread.so.0
#7 0x00007f165054388d in __libc_ifunc_impl_list () from /lib64/libc.so.6
#8 0x0000000000000000 in ?? ()

Thread 47 (LWP 45850):
#0 0x00007f1650ef2da2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f1654549702 in apr_thread_cond_timedwait (cond=0x7f1648cb3b68, mutex=0x7f1648cb3b18, timeout=100000) at locks/unix/thread_cond.c:89
#2 0x00007f1654540dd9 in apr_queue_pop_timeout (queue=0x7f1648cb3ad8, data=0x7f162cf45d18, timeout=100000) at misc/apr_queue.c:339
#3 0x00007f16541a8c64 in switch_queue_pop_timeout (queue=0x7f1648cb3ad8, data=0x7f162cf45d18, timeout=100000) at src/switch_apr.c:1225
#4 0x00007f1648cf08fe in sofia_profile_worker_thread_run (thread=0x7f1648232b40, obj=0x29c2170) at sofia.c:3010
#5 0x00007f165454f770 in dummy_worker (opaque=0x7f1648232b40) at threadproc/unix/thread.c:151
#6 0x00007f1650eeee65 in start_thread () from /lib64/libpthread.so.0
#7 0x00007f165054388d in __libc_ifunc_impl_list () from /lib64/libc.so.6
#8 0x0000000000000000 in ?? ()

Thread 46 (LWP 64308):
#0 0x00007f1650ef571d in write () from /lib64/libpthread.so.0
#1 0x0000000000000000 in ?? ()

Thread 45 (LWP 45852):
#0 0x00007f1650ef2da2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f1654549702 in apr_thread_cond_timedwait (cond=0x7f1648cb3b68, mutex=0x7f1648cb3b18, timeout=100000) at locks/unix/thread_cond.c:89
#2 0x00007f1654540dd9 in apr_queue_pop_timeout (queue=0x7f1648cb3ad8, data=0x7f162cecbd18, timeout=100000) at misc/apr_queue.c:339
#3 0x00007f16541a8c64 in switch_queue_pop_timeout (queue=0x7f1648cb3ad8, data=0x7f162cecbd18, timeout=100000) at src/switch_apr.c:1225
#4 0x00007f1648cf08fe in sofia_profile_worker_thread_run (thread=0x29be5f8, obj=0x29bb7d0) at sofia.c:3010
#5 0x00007f165454f770 in dummy_worker (opaque=0x29be5f8) at threadproc/unix/thread.c:151
#6 0x00007f1650eeee65 in start_thread () from /lib64/libpthread.so.0
#7 0x00007f165054388d in __libc_ifunc_impl_list () from /lib64/libc.so.6
#8 0x0000000000000000 in ?? ()

Thread 44 (LWP 45878):
#0 0x00007f1650ef571d in write () from /lib64/libpthread.so.0
#1 0x0000000000000000 in ?? ()

Thread 43 (LWP 45881):
#0 0x00007f1650ef571d in write () from /lib64/libpthread.so.0
#1 0x0000000000000000 in ?? ()

Thread 42 (LWP 45932):
#0 0x00007f1650ef29f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f165454969d in apr_thread_cond_wait (cond=, mutex=) at locks/unix/thread_cond.c:68
#2 0x00007f1654540ce1 in apr_queue_pop (queue=0x2dd3720, data=0x7f155c1bfd50) at misc/apr_queue.c:276
#3 0x00007f16541a8c37 in switch_queue_pop (queue=0x2dd3720, data=0x7f155c1bfd50) at src/switch_apr.c:1220
#4 0x00007f165426e3b4 in chat_thread_run (thread=0x2de3250, obj=0x2dd3720) at src/switch_loadable_module.c:911
#5 0x00007f165454f770 in dummy_worker (opaque=0x2de3250) at threadproc/unix/thread.c:151
#6 0x00007f1650eeee65 in start_thread () from /lib64/libpthread.so.0
#7 0x00007f165054388d in __libc_ifunc_impl_list () from /lib64/libc.so.6
#8 0x0000000000000000 in ?? ()

Thread 41 (LWP 45933):
#0 0x00007f1650ef29f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f165454969d in apr_thread_cond_wait (cond=, mutex=) at locks/unix/thread_cond.c:68
#2 0x00007f1654540ce1 in apr_queue_pop (queue=0x2de3280, data=0x7f155c182d50) at misc/apr_queue.c:276
#3 0x00007f16541a8c37 in switch_queue_pop (queue=0x2de3280, data=0x7f155c182d50) at src/switch_apr.c:1220
#4 0x00007f165426e3b4 in chat_thread_run (thread=0x2def5e0, obj=0x2de3280) at src/switch_loadable_module.c:911
#5 0x00007f165454f770 in dummy_worker (opaque=0x2def5e0) at threadproc/unix/thread.c:151
#6 0x00007f1650eeee65 in start_thread () from /lib64/libpthread.so.0
#7 0x00007f165054388d in __libc_ifunc_impl_list () from /lib64/libc.so.6
#8 0x0000000000000000 in ?? ()

Thread 40 (LWP 45938):
#0 0x00007f16505598c4 in __get_clockfreq () from /lib64/libc.so.6
#1 0x0000000000000000 in ?? ()

Thread 39 (LWP 45928):
#0 0x00007f16505598c4 in __get_clockfreq () from /lib64/libc.so.6
#1 0x0000000000000000 in ?? ()

Thread 38 (LWP 45859):
#0 0x00007f1650ef29f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f165454969d in apr_thread_cond_wait (cond=, mutex=) at locks/unix/thread_cond.c:68
#2 0x00007f1654540ce1 in apr_queue_pop (queue=0x29d5558, data=0x7f161fd7bd10) at misc/apr_queue.c:276
#3 0x00007f161fdb0ac2 in apt_consumer_task_run () from /usr/local/fs1.10.7/lib/freeswitch/mod/mod_unimrcp.so
#4 0x00007f161fdb41a7 in apt_task_run (thread_handle=0x29ec528, data=0x29d93e8) at src/apt_task.c:591
#5 0x00007f165454f770 in dummy_worker (opaque=0x29ec528) at threadproc/unix/thread.c:151
#6 0x00007f1650eeee65 in start_thread () from /lib64/libpthread.so.0
#7 0x00007f165054388d in __libc_ifunc_impl_list () from /lib64/libc.so.6
#8 0x0000000000000000 in ?? ()

Thread 37 (LWP 45863):
#0 0x00007f1650543e63 in arch_prctl () from /lib64/libc.so.6
#1 0x0000000000000000 in ?? ()

Thread 36 (LWP 64592):
#0 0x00007f1650ef29f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f16541a740f in switch_thread_cond_wait (cond=0x7f14e0029ab8, mutex=0x7f14e0029a68) at src/switch_apr.c:371
#2 0x00007f16542cf5d2 in recording_thread (thread=0x7f14e0029c78, obj=0x7f14e00eed18) at src/switch_ivr_async.c:1329
#3 0x00007f165454f770 in dummy_worker (opaque=0x7f14e0029c78) at threadproc/unix/thread.c:151
#4 0x00007f1650eeee65 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f165054388d in __libc_ifunc_impl_list () from /lib64/libc.so.6
#6 0x0000000000000000 in ?? ()

Thread 35 (LWP 45860):
#0 0x00007f1650543e63 in arch_prctl () from /lib64/libc.so.6
#1 0x0000000000000000 in ?? ()

Thread 34 (LWP 64949):
#0 0x00007f1650ef571d in write () from /lib64/libpthread.so.0
#1 0x0000000000000000 in ?? ()

Thread 33 (LWP 45856):
#0 0x00007f1650ef29f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f165454969d in apr_thread_cond_wait (cond=, mutex=) at locks/unix/thread_cond.c:68
#2 0x00007f1654540ce1 in apr_queue_pop (queue=0x7f1648c50ad8, data=0x7f162c426d48) at misc/apr_queue.c:276
#3 0x00007f16541a8c37 in switch_queue_pop (queue=0x7f1648c50ad8, data=0x7f162c426d48) at src/switch_apr.c:1220
#4 0x00007f1648cecd58 in sofia_msg_thread_run (thread=0x29b40d0, obj=0x7f1648c50ad8) at sofia.c:2295
#5 0x00007f165454f770 in dummy_worker (opaque=0x29b40d0) at threadproc/unix/thread.c:151
#6 0x00007f1650eeee65 in start_thread () from /lib64/libpthread.so.0
#7 0x00007f165054388d in __libc_ifunc_impl_list () from /lib64/libc.so.6
#8 0x0000000000000000 in ?? ()

Thread 32 (LWP 45855):
#0 0x00007f1650ef2da2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f1654549702 in apr_thread_cond_timedwait (cond=0x7f1648cb3b68, mutex=0x7f1648cb3b18, timeout=100000) at locks/unix/thread_cond.c:89
#2 0x00007f1654540dd9 in apr_queue_pop_timeout (queue=0x7f1648cb3ad8, data=0x7f162c463d18, timeout=100000) at misc/apr_queue.c:339
#3 0x00007f16541a8c64 in switch_queue_pop_timeout (queue=0x7f1648cb3ad8, data=0x7f162c463d18, timeout=100000) at src/switch_apr.c:1225
#4 0x00007f1648cf08fe in sofia_profile_worker_thread_run (thread=0x7f162c68db50, obj=0x29c8950) at sofia.c:3010
#5 0x00007f165454f770 in dummy_worker (opaque=0x7f162c68db50) at threadproc/unix/thread.c:151
#6 0x00007f1650eeee65 in start_thread () from /lib64/libpthread.so.0
#7 0x00007f165054388d in __libc_ifunc_impl_list () from /lib64/libc.so.6
#8 0x0000000000000000 in ?? ()

Thread 31 (LWP 65067):
#0 0x00007f165053a953 in setegid () from /lib64/libc.so.6
#1 0x000000000007a120 in ?? ()
#2 0x00007f152a359126 in ?? () from /usr/local/fs1.10.7/lib/freeswitch/mod/mod_callcenter.so
#3 0x0000000002a1be40 in ?? ()
#4 0x00007f14936b0ca0 in ?? ()
#5 0x00007f165431193c in switch_sleep (t=44154432) at src/switch_time.c:644
Backtrace stopped: frame did not save the PC

Thread 30 (LWP 65076):
#0 0x00007f1650538bed in fts_read () from /lib64/libc.so.6
#1 0x00007f16542a3607 in rtp_common_read (rtp_session=<error reading variable: Cannot access memory at address 0xfffffe08>, payload_type=<error reading variable: Cannot access memory at address 0xfffffe00>, pmapP=<error reading variable: Cannot access memory at address 0xfffffdf8>, flags=<error reading variable: Cannot access memory at address 0xfffffdf0>, io_flags=<error reading variable: Cannot access memory at address 0xfffffdec>) at src/switch_rtp.c:7571
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

Thread 29 (LWP 45837):
#0 0x00007f16505598c4 in __get_clockfreq () from /lib64/libc.so.6
#1 0x0000000000000000 in ?? ()

Thread 28 (LWP 45849):
#0 0x00007f1650ef29f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f16541a740f in switch_thread_cond_wait (cond=0x7f1628028ce8, mutex=0x7f1628028bf8) at src/switch_apr.c:371
#2 0x00007f16541dceeb in switch_user_sql_thread (thread=0x7f162d0846c0, obj=0x7f1628028b30) at src/switch_core_sqldb.c:2296
#3 0x00007f165454f770 in dummy_worker (opaque=0x7f162d0846c0) at threadproc/unix/thread.c:151
#4 0x00007f1650eeee65 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f165054388d in __libc_ifunc_impl_list () from /lib64/libc.so.6
#6 0x0000000000000000 in ?? ()

Thread 27 (LWP 45854):
#0 0x00007f1650ef29f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f16541a740f in switch_thread_cond_wait (cond=0x7f1624002598, mutex=0x7f16240024a8) at src/switch_apr.c:371
#2 0x00007f16541dceeb in switch_user_sql_thread (thread=0x7f162c5656c0, obj=0x7f16240023e0) at src/switch_core_sqldb.c:2296
#3 0x00007f165454f770 in dummy_worker (opaque=0x7f162c5656c0) at threadproc/unix/thread.c:151
#4 0x00007f1650eeee65 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f165054388d in __libc_ifunc_impl_list () from /lib64/libc.so.6
#6 0x0000000000000000 in ?? ()

Thread 26 (LWP 45929):
#0 0x00007f1650ef29f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f16541a740f in switch_thread_cond_wait (cond=0x29f1548, mutex=0x29f14a8) at src/switch_apr.c:371
#2 0x00007f161c1698f4 in mod_verto_runtime () at mod_verto.c:6907
#3 0x00007f165426ac27 in switch_loadable_module_exec (thread=0x273ca38, obj=0x273c618) at src/switch_loadable_module.c:123
#4 0x00007f165454f770 in dummy_worker (opaque=0x273ca38) at threadproc/unix/thread.c:151
#5 0x00007f1650eeee65 in start_thread () from /lib64/libpthread.so.0
#6 0x00007f165054388d in __libc_ifunc_impl_list () from /lib64/libc.so.6
#7 0x0000000000000000 in ?? ()

Thread 25 (LWP 45865):
#0 0x00007f1650543e63 in arch_prctl () from /lib64/libc.so.6
#1 0x0000000000000000 in ?? ()

Thread 24 (LWP 45846):
#0 0x00007f1650543e63 in arch_prctl () from /lib64/libc.so.6
#1 0x0000000000000000 in ?? ()

Thread 23 (LWP 45838):
#0 0x00007f165053a953 in setegid () from /lib64/libc.so.6
#1 0x00000000000f4240 in ?? ()
#2 0x00007f164ad7dd30 in ?? ()
#3 0x00007f16541e1f7d in switch_core_expire_registration (force=0) at src/switch_core_sqldb.c:3557
#4 0x00007f165431193c in switch_sleep (t=1000000) at src/switch_time.c:644
#5 0x00007f16541db054 in switch_core_sql_db_thread (thread=0x7f164c600750, obj=0x0) at src/switch_core_sqldb.c:1541
#6 0x00007f165454f770 in dummy_worker (opaque=0x7f164c600750) at threadproc/unix/thread.c:151
#7 0x00007f1650eeee65 in start_thread () from /lib64/libpthread.so.0
#8 0x00007f165054388d in __libc_ifunc_impl_list () from /lib64/libc.so.6
#9 0x0000000000000000 in ?? ()

Thread 22 (LWP 45930):
#0 0x00007f1650ef599d in fcntl () from /lib64/libpthread.so.0
#1 0x00007f15180029a0 in ?? ()
#2 0x0000000000002687 in ?? ()
#3 0x0000000000000000 in ?? ()

Thread 21 (LWP 45840):
#0 0x00007f1650543e63 in arch_prctl () from /lib64/libc.so.6
#1 0x0000000000000000 in ?? ()

Thread 20 (LWP 45841):
#0 0x00007f1650543e63 in arch_prctl () from /lib64/libc.so.6
#1 0x0000000000000000 in ?? ()

Thread 19 (LWP 45931):
#0 0x00007f1650ef571d in write () from /lib64/libpthread.so.0
#1 0x0000000000000000 in ?? ()

Thread 18 (LWP 45842):
#0 0x00007f1650543e63 in arch_prctl () from /lib64/libc.so.6
#1 0x0000000000000000 in ?? ()

Thread 17 (LWP 45927):
#0 0x00007f16505598c4 in __get_clockfreq () from /lib64/libc.so.6
#1 0x0000000000000000 in ?? ()

Thread 16 (LWP 45845):
#0 0x00007f1650543e63 in arch_prctl () from /lib64/libc.so.6
#1 0x0000000000000000 in ?? ()

Thread 15 (LWP 45844):
#0 0x00007f1650543e63 in arch_prctl () from /lib64/libc.so.6
#1 0x0000000000000000 in ?? ()

Thread 14 (LWP 45847):
#0 0x00007f1650ef29f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f16541a740f in switch_thread_cond_wait (cond=0x7f1638028d08, mutex=0x7f1638028c18) at src/switch_apr.c:371
#2 0x00007f16541dceeb in switch_user_sql_thread (thread=0x7f162d20e6c0, obj=0x7f1638028b40) at src/switch_core_sqldb.c:2296
#3 0x00007f165454f770 in dummy_worker (opaque=0x7f162d20e6c0) at threadproc/unix/thread.c:151
#4 0x00007f1650eeee65 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f165054388d in __libc_ifunc_impl_list () from /lib64/libc.so.6
#6 0x0000000000000000 in ?? ()

Thread 13 (LWP 45873):
#0 0x00007f1650ef2da2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f15a0fa7939 in ks_cond_timedwait (cond=0x2a1fb58, ms=10000) at /home/freeswitch/tmp/libks/src/ks_mutex.c:563
#2 0x00007f15cf1a7c4e in __manager_loop () at /home/freeswitch/tmp/20220405/signalwire-c/src/handle_manager.c:255
#3 0x00007f15cf1a7d0f in __manager_thread_wrapper (thread=0x2a1fcc8, data=0x0) at /home/freeswitch/tmp/20220405/signalwire-c/src/handle_manager.c:275
#4 0x00007f15a0fb5036 in thread_launch (args=0x2a1fcc8) at /home/freeswitch/tmp/libks/src/ks_thread.c:248
#5 0x00007f1650eeee65 in start_thread () from /lib64/libpthread.so.0
#6 0x00007f165054388d in __libc_ifunc_impl_list () from /lib64/libc.so.6
#7 0x0000000000000000 in ?? ()

Thread 12 (LWP 45872):
#0 0x00007f1650543e63 in arch_prctl () from /lib64/libc.so.6
#1 0x0000000000000000 in ?? ()

Thread 11 (LWP 45843):
#0 0x00007f1650543e63 in arch_prctl () from /lib64/libc.so.6
#1 0x0000000000000000 in ?? ()

Thread 10 (LWP 65206):
#0 0x00007f1650ef29f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f16541a740f in switch_thread_cond_wait (cond=0x7f14a812e0f0, mutex=0x7f14a812e0a0) at src/switch_apr.c:371
#2 0x00007f16542cf5d2 in recording_thread (thread=0x7f14a812e2b0, obj=0x7f14a80a8bc8) at src/switch_ivr_async.c:1329
#3 0x00007f165454f770 in dummy_worker (opaque=0x7f14a812e2b0) at threadproc/unix/thread.c:151
#4 0x00007f1650eeee65 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f165054388d in __libc_ifunc_impl_list () from /lib64/libc.so.6
#6 0x0000000000000000 in ?? ()

Thread 9 (LWP 45831):
#0 0x00007f1650ef29f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f165454969d in apr_thread_cond_wait (cond=, mutex=) at locks/unix/thread_cond.c:68
#2 0x00007f1654540ce1 in apr_queue_pop (queue=0x24a1d98, data=0x7f164c751d48) at misc/apr_queue.c:276
#3 0x00007f16541a8c37 in switch_queue_pop (queue=0x24a1d98, data=0x7f164c751d48) at src/switch_apr.c:1220
#4 0x00007f16542819cd in switch_event_dispatch_thread (thread=0x7f164c7dbc40, obj=0x24a1d98) at src/switch_event.c:331
#5 0x00007f165454f770 in dummy_worker (opaque=0x7f164c7dbc40) at threadproc/unix/thread.c:151
#6 0x00007f1650eeee65 in start_thread () from /lib64/libpthread.so.0
#7 0x00007f165054388d in __libc_ifunc_impl_list () from /lib64/libc.so.6
#8 0x0000000000000000 in ?? ()

Thread 8 (LWP 45848):
#0 0x00007f1650ef29f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f16541a740f in switch_thread_cond_wait (cond=0x7f16200280a8, mutex=0x7f1620027fb8) at src/switch_apr.c:371
#2 0x00007f16541dceeb in switch_user_sql_thread (thread=0x7f162d1496c0, obj=0x7f1620027ee0) at src/switch_core_sqldb.c:2296
#3 0x00007f165454f770 in dummy_worker (opaque=0x7f162d1496c0) at threadproc/unix/thread.c:151
#4 0x00007f1650eeee65 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f165054388d in __libc_ifunc_impl_list () from /lib64/libc.so.6
#6 0x0000000000000000 in ?? ()

Thread 7 (LWP 45861):
#0 0x00007f165053a953 in setegid () from /lib64/libc.so.6
#1 0x0000000000000000 in ?? ()

Thread 6 (LWP 45874):
#0 0x00007f1650ef29f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f165454969d in apr_thread_cond_wait (cond=, mutex=) at locks/unix/thread_cond.c:68
#2 0x00007f1654540ce1 in apr_queue_pop (queue=0x29a4628, data=0x7f162c113d50) at misc/apr_queue.c:276
#3 0x00007f16541a8c37 in switch_queue_pop (queue=0x29a4628, data=0x7f162c113d50) at src/switch_apr.c:1220
#4 0x00007f1648d308fa in sofia_presence_event_thread_run (thread=0x29a54e0, obj=0x0) at sofia_presence.c:1625
#5 0x00007f165454f770 in dummy_worker (opaque=0x29a54e0) at threadproc/unix/thread.c:151
#6 0x00007f1650eeee65 in start_thread () from /lib64/libpthread.so.0
#7 0x00007f165054388d in __libc_ifunc_impl_list () from /lib64/libc.so.6
#8 0x0000000000000000 in ?? ()

Thread 5 (LWP 45862):
#0 0x00007f1650543e63 in arch_prctl () from /lib64/libc.so.6
#1 0x0000000000000000 in ?? ()

Thread 4 (LWP 45900):
#0 0x00007f1650ef29f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f165454969d in apr_thread_cond_wait (cond=, mutex=) at locks/unix/thread_cond.c:68
#2 0x00007f1654540ce1 in apr_queue_pop (queue=0x7f153001b958, data=0x7f15c406d468) at misc/apr_queue.c:276
#3 0x00007f16541a8c37 in switch_queue_pop (queue=0x7f153001b958, data=0x7f15c406d468) at src/switch_apr.c:1220
#4 0x00007f165433b0ee in EventConsumer::pop (this=, block=block@entry=1, timeout=) at src/switch_cpp.cpp:110
#5 0x00007f155c1ebaf5 in _wrap_EventConsumer_pop (L=0x7f15300008c8) at mod_lua_wrap.cpp:5547
#6 0x00007f155c1f48a0 in luaD_precall () from /usr/local/fs1.10.7/lib/freeswitch/mod/mod_lua.so
#7 0x00007f155c1ff29f in luaV_execute () from /usr/local/fs1.10.7/lib/freeswitch/mod/mod_lua.so
#8 0x00007f155c1f4c9c in luaD_call () from /usr/local/fs1.10.7/lib/freeswitch/mod/mod_lua.so
#9 0x00007f155c1ff0ac in luaV_execute () from /usr/local/fs1.10.7/lib/freeswitch/mod/mod_lua.so
#10 0x00007f155c1f4c9c in luaD_call () from /usr/local/fs1.10.7/lib/freeswitch/mod/mod_lua.so
#11 0x00007f155c1f425f in luaD_rawrunprotected () from /usr/local/fs1.10.7/lib/freeswitch/mod/mod_lua.so
#12 0x00007f155c1f4f0d in luaD_pcall () from /usr/local/fs1.10.7/lib/freeswitch/mod/mod_lua.so
#13 0x00007f155c1f275e in lua_pcallk () from /usr/local/fs1.10.7/lib/freeswitch/mod/mod_lua.so
#14 0x00007f155c1d1b2d in docall (L=L@entry=0x7f15300008c8, narg=narg@entry=0, nresults=nresults@entry=0, perror=perror@entry=0, fatal=fatal@entry=1) at mod_lua.cpp:92
#15 0x00007f155c1d213f in lua_parse_and_execute (L=L@entry=0x7f15300008c8, input_code=, session=session@entry=0x0) at mod_lua.cpp:194
#16 0x00007f155c1d21d7 in lua_thread_run (thread=, obj=0x2c06050) at mod_lua.cpp:221
#17 0x00007f165454f770 in dummy_worker (opaque=0x2c061d8) at threadproc/unix/thread.c:151
#18 0x00007f1650eeee65 in start_thread () from /lib64/libpthread.so.0
#19 0x00007f165054388d in __libc_ifunc_impl_list () from /lib64/libc.so.6
#20 0x0000000000000000 in ?? ()

Thread 3 (LWP 45875):
#0 0x00007f1650ef29f5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00007f16541a740f in switch_thread_cond_wait (cond=0x2a8d800, mutex=0x2a8d710) at src/switch_apr.c:371
#2 0x00007f16541dceeb in switch_user_sql_thread (thread=0x7f161c1496c0, obj=0x2a8d660) at src/switch_core_sqldb.c:2296
#3 0x00007f165454f770 in dummy_worker (opaque=0x7f161c1496c0) at threadproc/unix/thread.c:151
#4 0x00007f1650eeee65 in start_thread () from /lib64/libpthread.so.0
#5 0x00007f165054388d in __libc_ifunc_impl_list () from /lib64/libc.so.6
#6 0x0000000000000000 in ?? ()

Thread 2 (LWP 45864):
#0 0x00007f1650543e63 in arch_prctl () from /lib64/libc.so.6
#1 0x0000000000000000 in ?? ()

Thread 1 (LWP 45853):
#0 0x00007f1651eb87e8 in su_block_add () from /usr/local/lib/libsofia-sip-ua.so.0
#1 0x00007f1651eb8d9d in sub_alloc () from /usr/local/lib/libsofia-sip-ua.so.0
#2 0x00007f1651eba8b1 in su_zalloc () from /usr/local/lib/libsofia-sip-ua.so.0
#3 0x00007f1651e42d4b in outgoing_make_a_aaaa_query () from /usr/local/lib/libsofia-sip-ua.so.0
#4 0x00007f1651e42350 in outgoing_resolve_next () from /usr/local/lib/libsofia-sip-ua.so.0
#5 0x00007f1651e43e88 in outgoing_answer_srv () from /usr/local/lib/libsofia-sip-ua.so.0
#6 0x00007f1651ead296 in sres_query_report_error () from /usr/local/lib/libsofia-sip-ua.so.0
#7 0x00007f1651ead643 in sres_resend_dns_query () from /usr/local/lib/libsofia-sip-ua.so.0
#8 0x00007f1651eae590 in sres_resolver_report_error () from /usr/local/lib/libsofia-sip-ua.so.0
#9 0x00007f1651eae24f in sres_resolver_error () from /usr/local/lib/libsofia-sip-ua.so.0
#10 0x00007f1651eb3eee in sres_sofia_poll () from /usr/local/lib/libsofia-sip-ua.so.0
#11 0x00007f1651ec5768 in su_epoll_port_wait_events () from /usr/local/lib/libsofia-sip-ua.so.0
#12 0x00007f1651ec1b68 in su_base_port_run () from /usr/local/lib/libsofia-sip-ua.so.0
#13 0x00007f1651ebde18 in su_port_run () from /usr/local/lib/libsofia-sip-ua.so.0
#14 0x00007f1651ebeef2 in su_root_run () from /usr/local/lib/libsofia-sip-ua.so.0
#15 0x00007f1651ec2924 in su_pthread_port_clone_main () from /usr/local/lib/libsofia-sip-ua.so.0
#16 0x00007f1650eeee65 in start_thread () from /lib64/libpthread.so.0
#17 0x00007f165054388d in __libc_ifunc_impl_list () from /lib64/libc.so.6
#18 0x0000000000000000 in ?? ()

[andywolk]

You need to install sofia sip debugging symbols.
Also I don't think your sofia is the latest.
And please attach the log, do not paste it.

[andywolk]

Sounds like it is not a bug but some memory issues on your box.
Information you provided does not show me that a crash happened.
Line numbers do not match sofia-sip version you mentioned.

[dxhgq-github]

Ok, I will take a check at sofia-sip version , Thanks!

[dxhgq-github]

Hello
I have downloaded and rebuilt the latest version 1.13.7 of sofia-sip , after running 4 days , the FreeSwitch still crashs. error message is :
Core was generated by `/usr/local/fs1.10.7/bin/freeswitch -nonat'.
Program terminated with signal SIGFPE, Arithmetic exception.
#0 0x00007f2c9e58381a in su_block_add (b=0x7f2c542282f0, p=0x7f2c542547f0) at su_alloc.c:342
342 h = (size_t)((uintptr_t)p % b->sub_n);
[Current thread is 1 (LWP 92647)]
(gdb)
(gdb) bt
#0 0x00007f2c9e58381a in su_block_add (b=0x7f2c542282f0, p=0x7f2c542547f0) at su_alloc.c:342
#1 0x00007f2c9e583dcf in sub_alloc (home=0x7f2c54250ef0, sub=0x7f2c542282f0, size=80,
zero=do_calloc) at su_alloc.c:530
#2 0x00007f2c9e5858e3 in su_zalloc (home=0x7f2c54250ef0, size=80) at su_alloc.c:1577
#3 0x00007f2c9e50dd4b in outgoing_make_a_aaaa_query (orq=0x7f2c54254690) at nta.c:10463
#4 0x00007f2c9e50d350 in outgoing_resolve_next (orq=0x7f2c54254690) at nta.c:10244
#5 0x00007f2c9e50ee88 in outgoing_answer_srv (orq=0x7f2c54254690, q=0x7f2c5410b2c0,
answers=0x7f2c541167b0) at nta.c:10830
#6 0x00007f2c9e5782c8 in sres_query_report_error (q=0x7f2c5410b2c0, answers=0x7f2c541167b0)
at sres.c:2991
#7 0x00007f2c9e578675 in sres_resend_dns_query (res=0x7f2c54002990, q=0x7f2c5410b2c0,
timeout=0) at sres.c:3091
#8 0x00007f2c9e5795c2 in sres_resolver_report_error (res=0x7f2c54002990, socket=590,
errcode=111, remote=0x7f2c7d58c560, remotelen=16,
info=0x7f2c7d58c4e0 "icmp type=3 code=3 reported by 127.0.0.1") at sres.c:3440
#9 0x00007f2c9e579281 in sres_resolver_error (res=0x7f2c54002990, socket=590) at sres.c:3354
#10 0x00007f2c9e57ef20 in sres_sofia_poll (magic=0x7f2c68001180, w=0x7f2c54001084,
reg=0x7f2c540037c0) at sresolv.c:357
#11 0x00007f2c9e59079a in su_epoll_port_wait_events (self=0x7f2c540008c0, tout=690)
at su_epoll_port.c:510
#12 0x00007f2c9e58cb9a in su_base_port_run (self=0x7f2c540008c0) at su_base_port.c:349
#13 0x00007f2c9e588e4a in su_port_run (self=0x7f2c540008c0) at su_port.h:326
#14 0x00007f2c9e589f24 in su_root_run (self=0x7f2c54001130) at su_root.c:819
#15 0x00007f2c9e58d956 in su_pthread_port_clone_main (varg=0x7f2c91233460)
at su_pthread_port.c:343
#16 0x00007f2c9d5b9e65 in start_thread () from /lib64/libpthread.so.0
#17 0x00007f2c9cc0e88d in __libc_ifunc_impl_list () from /lib64/libc.so.6
#18 0x0000000000000000 in ?? ()
(gdb)

backtrace1.log

[dxhgq-github]

Please help , thanks

[andywolk]

Please wrap backtraces into a code block when you comment.

[andywolk]

Based on your last backtrace please print some debugging information:

frame 2
p *home

[dxhgq-github]

thank you for replay !

[dxhgq-github]

(gdb)
(gdb)
(gdb) frame 2
#2 0x00007ff892b5992d in su_zalloc (home=0x7ff86028d060, size=80) at su_alloc.c:1577
1577 data = sub_alloc(home, MEMLOCK(home), size, (enum sub_zero)1);
(gdb) p *home
$1 = {suh_size = 1620230464, suh_blocks = 0x7ff8613c3900, suh_lock = 0x0}
(gdb)

thank you !

[dxhgq-github]

whether it is a IPv6 resolve problem ? but I do not configure any call flow about local resolve, and ping 127.0.0.1 is alive , and /etc/resolv.conf is empty.

[dxhgq-github]

backtrace.log

[dxhgq-github]

last week ,i disabled the ipv6 profile ,mv internal-ipv6.xml internal-ipv6.xml.bak ,mv external-ipv6 external-ipv6.bak , but problem is still unsolved。

[dxhgq-github]

yesterday ,i disabled the ipv6 on box ,add tow lines :
net.ipv6.conf.all.disable_ipv6 =1
net.ipv6.conf.default.disable_ipv6 =1
to /etc/sysctl.conf.
by the end of this time ,i have tested for about 30 hours.

[andywolk]

This frame 2 is not from a backtrace mentioned in this github issue. From another crash?

[dragos-oancea]

@dxhgq-github , #1659 - you should not open the same issue in two places, sofia-sip and FS, just keep the discussion in one place.

[dxhgq-github]

ok , This frame 2 is from another crash, same problem ,above attachment is backtrace