Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Application crash due to intensive "dispather.add" using #3424

Closed
NikolayShakin opened this issue Apr 22, 2023 · 5 comments
Closed

Application crash due to intensive "dispather.add" using #3424

NikolayShakin opened this issue Apr 22, 2023 · 5 comments

Comments

@NikolayShakin
Copy link

NikolayShakin commented Apr 22, 2023 •
edited

Description

I want to add addresses to dispatcher.list dynamically

modparam("dispatcher", "db_url", DBURL)
modparam("dispatcher", "table_name", "dispatcher")
modparam("dispatcher", "ds_ping_interval", 10)
modparam("dispatcher", "ds_probing_mode", 1)
modparam("dispatcher", "ds_inactive_threshold", 1)
modparam("dispatcher", "ds_probing_threshold", 1)
modparam("dispatcher", "reload_delta", 0)
modparam("dispatcher", "ds_timer_mode", 1)
...	
     if(!ds_is_from_list("1","0","$var(contact)")) {
		jsonrpc_exec(
			'{
				"jsonrpc": "2.0",
				"method": "dispatcher.add",
				"params": [1, "$var(contact)"],
				"id": 1
			}'
		);
	}

this scheme works with low load(one or two UAs) but it doesn't work with dozens of requests, kamailio crashes

Troubleshooting

Apr 22 22:18:09 kamailio /usr/sbin/kamailio[6832]: CRITICAL: {1 12016 REGISTER db88353a-66cf-4ca6-91e2-6f306e43cee9} <core> [core/mem/q_malloc.c:519]: qm_free(): BUG: freeing already freed pointer (0x7f2f964578b8), called from dispatcher: dispatch.c: reindex_dests(790), first free dispatcher: dispatch.c: ds_avl_destroy(4013) - ignoring
Apr 22 22:18:09 kamailio /usr/sbin/kamailio[6832]: INFO: {1 12016 REGISTER db88353a-66cf-4ca6-91e2-6f306e43cee9} dispatcher [dispatch.c:682]: dp_init_relative_weights(): extra rweight 100 for last active destination in group 1
Apr 22 22:18:09 kamailio /usr/sbin/kamailio[6827]: INFO: {1 16898 REGISTER eebaeca9-c642-465f-bc09-67d1448ba6b7} dispatcher [dispatch.c:682]: dp_init_relative_weights(): extra rweight 100 for last active destination in group 1
Apr 22 22:18:09 kamailio /usr/sbin/kamailio[6825]: ERROR: {1 27217 REGISTER 885dc562-c8f3-477a-bc4b-ba3718cdf8db} dispatcher [dispatch.c:407]: pack_dest(): bad uri []
Apr 22 22:18:09 kamailio /usr/sbin/kamailio[6825]: WARNING: {1 27217 REGISTER 885dc562-c8f3-477a-bc4b-ba3718cdf8db} dispatcher [dispatch.c:2624]: ds_add_dest_cb(): failed to add destination in group 1 -
Apr 22 22:18:09 kamailio /usr/sbin/kamailio[6825]: ERROR: {1 27217 REGISTER 885dc562-c8f3-477a-bc4b-ba3718cdf8db} dispatcher [dispatch.c:407]: pack_dest(): bad uri []
Apr 22 22:18:09 kamailio /usr/sbin/kamailio[6825]: WARNING: {1 27217 REGISTER 885dc562-c8f3-477a-bc4b-ba3718cdf8db} dispatcher [dispatch.c:2624]: ds_add_dest_cb(): failed to add destination in group 1 -
Apr 22 22:18:09 kamailio /usr/sbin/kamailio[6825]: ERROR: {1 27217 REGISTER 885dc562-c8f3-477a-bc4b-ba3718cdf8db} dispatcher [dispatch.c:407]: pack_dest(): bad uri []
Apr 22 22:18:09 kamailio /usr/sbin/kamailio[6825]: WARNING: {1 27217 REGISTER 885dc562-c8f3-477a-bc4b-ba3718cdf8db} dispatcher [dispatch.c:2624]: ds_add_dest_cb(): failed to add destination in group 1 -
Apr 22 22:18:09 kamailio /usr/sbin/kamailio[6829]: ERROR: {1 40473 REGISTER bd8d332f-d896-4dab-b4ab-3904f8a29dda} dispatcher [dispatch.c:407]: pack_dest(): bad uri []
Apr 22 22:18:09 kamailio /usr/sbin/kamailio[6829]: WARNING: {1 40473 REGISTER bd8d332f-d896-4dab-b4ab-3904f8a29dda} dispatcher [dispatch.c:2624]: ds_add_dest_cb(): failed to add destination in group 1 -
Apr 22 22:18:09 kamailio /usr/sbin/kamailio[6829]: ERROR: {1 40473 REGISTER bd8d332f-d896-4dab-b4ab-3904f8a29dda} dispatcher [dispatch.c:407]: pack_dest(): bad uri []
Apr 22 22:18:09 kamailio /usr/sbin/kamailio[6829]: WARNING: {1 40473 REGISTER bd8d332f-d896-4dab-b4ab-3904f8a29dda} dispatcher [dispatch.c:2624]: ds_add_dest_cb(): failed to add destination in group 1 -
Apr 22 22:18:09 kamailio /usr/sbin/kamailio[6829]: ERROR: {1 40473 REGISTER bd8d332f-d896-4dab-b4ab-3904f8a29dda} dispatcher [dispatch.c:407]: pack_dest(): bad uri []
Apr 22 22:18:09 kamailio /usr/sbin/kamailio[6829]: WARNING: {1 40473 REGISTER bd8d332f-d896-4dab-b4ab-3904f8a29dda} dispatcher [dispatch.c:2624]: ds_add_dest_cb(): failed to add destination in group 1 -
Apr 22 22:18:09 kamailio /usr/sbin/kamailio[6832]: ERROR: {1 55015 REGISTER bfc85ce0-6aee-40c3-9b73-d0f64e835dc1} dispatcher [dispatch.c:407]: pack_dest(): bad uri []
Apr 22 22:18:09 kamailio /usr/sbin/kamailio[6832]: WARNING: {1 55015 REGISTER bfc85ce0-6aee-40c3-9b73-d0f64e835dc1} dispatcher [dispatch.c:2624]: ds_add_dest_cb(): failed to add destination in group 1 -
Apr 22 22:18:09 kamailio /usr/sbin/kamailio[6832]: ERROR: {1 55015 REGISTER bfc85ce0-6aee-40c3-9b73-d0f64e835dc1} dispatcher [dispatch.c:407]: pack_dest(): bad uri []
Apr 22 22:18:09 kamailio /usr/sbin/kamailio[6832]: WARNING: {1 55015 REGISTER bfc85ce0-6aee-40c3-9b73-d0f64e835dc1} dispatcher [dispatch.c:2624]: ds_add_dest_cb(): failed to add destination in group 1 -
Apr 22 22:18:09 kamailio /usr/sbin/kamailio[6832]: ERROR: {1 55015 REGISTER bfc85ce0-6aee-40c3-9b73-d0f64e835dc1} dispatcher [dispatch.c:407]: pack_dest(): bad uri []
Apr 22 22:18:09 kamailio /usr/sbin/kamailio[6832]: WARNING: {1 55015 REGISTER bfc85ce0-6aee-40c3-9b73-d0f64e835dc1} dispatcher [dispatch.c:2624]: ds_add_dest_cb(): failed to add destination in group 1 -
Apr 22 22:18:09 kamailio /usr/sbin/kamailio[6808]: ALERT: <core> [main.c:774]: handle_sigs(): child process 6825 exited by a signal 11
Apr 22 22:18:09 kamailio /usr/sbin/kamailio[6808]: ALERT: <core> [main.c:777]: handle_sigs(): core was not generated
Apr 22 22:18:09 kamailio /usr/sbin/kamailio[6808]: INFO: <core> [main.c:799]: handle_sigs(): terminating due to SIGCHLD
Apr 22 22:18:09 kamailio /usr/sbin/kamailio[6827]: INFO: <core> [main.c:854]: sig_usr(): signal 15 received
Apr 22 22:18:10 kamailio /usr/sbin/kamailio[6808]: INFO: <core> [core/sctp_core.c:53]: sctp_core_destroy(): SCTP API not initialized
Apr 22 22:18:10 kamailio systemd[1]: kamailio.service: Main process exited, code=exited, status=1/FAILURE
Apr 22 22:18:10 kamailio systemd[1]: kamailio.service: Failed with result 'exit-code'.
Apr 22 22:18:10 kamailio systemd[1]: kamailio.service: Consumed 1.304s CPU time.

Reproduction

use code from Description section on request route

Debugging Data

(gdb) bt full
#0  0x000055945e0ccd20 in ?? ()
No symbol table info available.
#1  0x000055945e0cd6ee in qm_malloc ()
No symbol table info available.
#2  0x000055945e0da9c9 in qm_shm_malloc ()
No symbol table info available.
#3  0x00007f325a05e921 in pack_dest () from /usr/lib/x86_64-linux-gnu/kamailio/modules/dispatcher.so
No symbol table info available.
#4  0x00007f325a060ad4 in add_dest2list () from /usr/lib/x86_64-linux-gnu/kamailio/modules/dispatcher.so
No symbol table info available.
#5  0x00007f325a087cf9 in ds_filter_dest_cb () from /usr/lib/x86_64-linux-gnu/kamailio/modules/dispatcher.so
No symbol table info available.
#6  0x00007f325a05affb in ds_iter_set () from /usr/lib/x86_64-linux-gnu/kamailio/modules/dispatcher.so
No symbol table info available.
#7  0x00007f325a088706 in ds_remove_dst () from /usr/lib/x86_64-linux-gnu/kamailio/modules/dispatcher.so
No symbol table info available.
#8  0x00007f325a0af7a8 in ?? () from /usr/lib/x86_64-linux-gnu/kamailio/modules/dispatcher.so
No symbol table info available.
#9  0x00007f325e7e82f5 in jsonrpc_exec_ex () from /usr/lib/x86_64-linux-gnu/kamailio/modules/jsonrpcs.so
No symbol table info available.
#10 0x00007f325e7e875e in ?? () from /usr/lib/x86_64-linux-gnu/kamailio/modules/jsonrpcs.so
No symbol table info available.
#11 0x000055945ddce3f3 in do_action ()
No symbol table info available.
#12 0x000055945dddd391 in run_actions ()
No symbol table info available.
#13 0x000055945ddce2d3 in do_action ()
No symbol table info available.
#14 0x000055945dddd391 in run_actions ()
No symbol table info available.
#15 0x000055945ddce2d3 in do_action ()
No symbol table info available.
#16 0x000055945dddd391 in run_actions ()
No symbol table info available.
#17 0x000055945ddca244 in do_action ()
No symbol table info available.
--Type <RET> for more, q to quit, c to continue without paging--
#18 0x000055945dddd391 in run_actions ()
No symbol table info available.
#19 0x000055945ddce2d3 in do_action ()
No symbol table info available.
#20 0x000055945dddd391 in run_actions ()
No symbol table info available.
#21 0x000055945ddddc0b in run_top_route ()
No symbol table info available.
#22 0x000055945df3b1b2 in receive_msg ()
No symbol table info available.
#23 0x000055945e09ea5d in udp_rcv_loop ()
No symbol table info available.
#24 0x000055945ddb709b in main_loop ()
No symbol table info available.
#25 0x000055945ddc420c in main ()
No symbol table info available.

(gdb) info locals
No symbol table info available.

(gdb) list
1       ../sysdeps/unix/sysv/linux/dl-vdso-setup.c: No such file or directory.

SIP Traffic

kamailio.txt

Possible Solutions

no idea

Additional Information

root@kamailio:[/etc]: kamailio -v
version: kamailio 5.6.3 (x86_64/linux) 
flags: USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MMAP, PKG_MALLOC, Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLOCKLIST, HAVE_RESOLV_RES, TLS_PTHREAD_MUTEX_SHARED
ADAPTIVE_WAIT_LOOPS 1024, MAX_RECV_BUFFER_SIZE 262144, MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB
poll method support: poll, epoll_lt, epoll_et, sigio_rt, select.
id: unknown 
compiled with gcc 10.2.1

root@kamailio:[/etc]: uname -a
Linux kamailio 5.10.0-21-amd64 #1 SMP Debian 5.10.162-1 (2023-01-21) x86_64 GNU/Linux
@henningw
Copy link
Contributor

What you are trying to achieve, using dispatcher on REGISTER messages to add the contact as destination? This is probably not a good idea at all. Why you are not using the usrloc module for this kind of functionality?
As you are also using an old, unsupported version of Kamailio, I am closing this issue. If you have questions about how to use the usrloc module, please contact our sr-users mailing list.

@henningw
Copy link
Contributor

Sorry, I misread the version - the 5.6.3 is of course supported. But anyway, please lets discuss on the users lists first, what you are trying to achieve, thank you.

@NikolayShakin
Copy link
Author

I'm making a mid-registrar configuration, when reg comes I authenticate it on Kamailio at first, then save it to usrloc, then relay it to the main registrar making authentication with uac there. I use the dispatcher module to monitor registered users, usrloc's keep alive doesn't suit this purpose because there is no event route if the user went offline. I want to use the dispatcher module to trigger and send unregister to the main registrar when the user died.

@NikolayShakin
Copy link
Author

Sorry, I misread the version - the 5.6.3 is of course supported. But anyway, please lets discuss on the users lists first, what you are trying to achieve, thank you.

In fact, my use case doesn't matter, if you add addresses to dispatcher list intensively, Kamailio crashes due to a memory leak. I reckon this is a bug

@miconda
Copy link
Member

miconda commented Apr 24, 2023

As it was mentioned above, this is for discussing on sr-users mailing list. There are limitations to use of rpc commands, some addressed in the latest version. If it proves something wrong in the design and code, then an issue can be opened for the right problem identified for it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@miconda @henningw @NikolayShakin