http_async_client: segfault at 0

[negbie]

Description

Send async http post request to multiple servers.

Troubleshooting

Reproduction

Kamailio runs for about 1-2 min till the segfault happens. Sadly there are not many infos in the coredump. I use this config for the http modules:
https://github.com/negbie/heplify/blob/master/kamailio5/http.cfg
With #!define DO_ASYNC

Debugging Data

gdb kamailio core.kamailio.114.1503484559.29804
GNU gdb (Debian 7.7.1+dfsg-5) 7.7.1
Copyright (C) 2014 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from kamailio...(no debugging symbols found)...done.
[New LWP 29804]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/usr/sbin/kamailio -P /var/run/kamailio/kamailio.pid -f /etc/kamailio/kamailio.'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x00007f4b0a1c23e9 in notification_socket_cb () from /usr/lib/x86_64-linux-gnu/kamailio/modules/http_async_client.so
(gdb) bt full
#0  0x00007f4b0a1c23e9 in notification_socket_cb () from /usr/lib/x86_64-linux-gnu/kamailio/modules/http_async_client.so
No symbol table info available.
#1  0x00007f4b09cf23dc in event_base_loop () from /usr/lib/x86_64-linux-gnu/libevent-2.0.so.5
No symbol table info available.
#2  0x00007f4b0a1bec74 in async_http_run_worker () from /usr/lib/x86_64-linux-gnu/kamailio/modules/http_async_client.so
No symbol table info available.
#3  0x00007f4b0a1a886e in ?? () from /usr/lib/x86_64-linux-gnu/kamailio/modules/http_async_client.so
No symbol table info available.
#4  0x000000000053e25a in ?? ()
No symbol table info available.
#5  0x000000000053df78 in ?? ()
No symbol table info available.
#6  0x000000000053e588 in init_child ()
No symbol table info available.
#7  0x0000000000423913 in main_loop ()
No symbol table info available.
#8  0x000000000042a308 in main ()
No symbol table info available.
(gdb) 

Log Messages

kernel:[698859.513811] kamailio[29804]: segfault at 0 ip 00007f4b0a1c23e9 sp 00007ffeb48bdc00 error 4 in http_async_client.so[7f4b0a19e000+38000]
/usr/sbin/kamailio[29777]: ALERT: <core> [main.c:743]: handle_sigs(): child process 29804 exited by a signal 11
/usr/sbin/kamailio[29777]: ALERT: <core> [main.c:746]: handle_sigs(): core was generated
systemd[1]: kamailio.service: main process exited, code=exited, status=1/FAILURE
systemd[1]: Unit kamailio.service entered failed state.

Additional Information

• Kamailio Version - output of kamailio -v

kamailio 5.0.2 (x86_64/linux) 

• Operating System:

Linux 3.16.0-4-amd64 #1 SMP Debian 3.16.43-2+deb8u2

[grumvalski]

Could you, by any chance, reproduce it with a kamailio with debug symbols?
One question to try to reproduce it on my side: I had a look at the config, are you sending to the different backends (so different urls) at the same time?

[negbie]

Hi Federico,
since this machine is part of a production setup i can replace kamailio compiled with dbg symbols only during night hours. Will see if i can do it tonight.

Yes I send LOTS of post requests to 3 different urls at the same time.

[negbie]

Hi Federico,
due tue better batch processing i was able to reduce the post request from over 1500/s to about 200/s. So far no crashes. So i close this issue and will open it again if it happens again.