KH1 crashes

[Matthias78]

Hi Karl,

for testing I put a KH1 in production (as a slave - without having own sources).
As it seems it's not stable.
Yesterday four crashes. Today already one.
Cause or initiator for the crashes yesterday was, that the connection to the master server was broken (or better: the master server itself was not running for some minutes) while the crashed server was under user load (around 3k connections).
Interesting is, that another KH1, which was connected to the same master server, did not crash. But this second server had no client connections on it. So it's a issue while listerners are connected and the source goes down (my guess).
The crash today was caused (AFAICS) because a single source got disconnected. So basicly it should be the same error.
Here are the 5 stack traces (the first 4 yesterday - the 5th today):

Crash number 3 looks different. So we have two issues?

Crash1:

Thread 1 (Thread 15449):
#0  signal_segv (signum=11, info=0x7ff2ef6309b0, ptr=0x7ff2ef630880) at ./Common/SIGSEGVHandler.cpp:128
#1  <signal handler called>
#2  0x00007ff2f44027f0 in memcpy () from /lib/libc.so.6
#3  0x0000000000418d18 in refbuf_copy (orig=0x7ff2c437a1a0) at refbuf.c:74
#4  0x0000000000414a5f in source_listener_detach (source=0x12e5480, client=<value optimized out>) at source.c:885
#5  0x0000000000414d2f in source_listener_release (client=0x7ff2d5abb9d0) at source.c:1861
#6  send_to_listener (client=0x7ff2d5abb9d0) at source.c:958
#7  0x000000000041945b in worker (arg=<value optimized out>) at client.c:557
#8  0x00000000004309e9 in _start_routine (arg=0x12af7f0) at thread.c:664
#9  0x00007ff2f416d8ba in start_thread () from /lib/libpthread.so.0
#10 0x00007ff2f445202d in clone () from /lib/libc.so.6
#11 0x0000000000000000 in ?? ()


Crash2:

Thread 1 (Thread 20238):
#0  signal_segv (signum=11, info=0x7f4026b6e9b0, ptr=0x7f4026b6e880) at ./Common/SIGSEGVHandler.cpp:128
#1  <signal handler called>
#2  0x00007f402b9407f0 in memcpy () from /lib/libc.so.6
#3  0x0000000000418d18 in refbuf_copy (orig=0x1dd09a0) at refbuf.c:74
#4  0x0000000000414a5f in source_listener_detach (source=0x7f40144c2830, client=<value optimized out>) at source.c:885
#5  0x0000000000414d2f in source_listener_release (client=0x7f401415ef50) at source.c:1861
#6  send_to_listener (client=0x7f401415ef50) at source.c:958
#7  0x000000000041945b in worker (arg=<value optimized out>) at client.c:557
#8  0x00000000004309e9 in _start_routine (arg=0x15457f0) at thread.c:664
#9  0x00007f402b6ab8ba in start_thread () from /lib/libpthread.so.0
#10 0x00007f402b99002d in clone () from /lib/libc.so.6
#11 0x0000000000000000 in ?? ()


Crash3:

Thread 1 (Thread 20804):
#0  signal_segv (signum=11, info=0x7ff6a9d7e9b0, ptr=0x7ff6a9d7e880) at ./Common/SIGSEGVHandler.cpp:128
#1  <signal handler called>
#2  0x0000000000423e2f in send_iceblock_to_client (client=0x7ff6a674d180) at format_mp3.c:558
#3  write_mpeg_buf_to_client (client=0x7ff6a674d180) at format_mp3.c:599
#4  0x0000000000414f1d in send_listener (client=0x7ff6a674d180) at source.c:1079
#5  send_to_listener (client=0x7ff6a674d180) at source.c:954
#6  0x000000000041945b in worker (arg=<value optimized out>) at client.c:557
#7  0x00000000004309e9 in _start_routine (arg=0xeb77f0) at thread.c:664
#8  0x00007ff6ae8bb8ba in start_thread () from /lib/libpthread.so.0
#9  0x00007ff6aeba002d in clone () from /lib/libc.so.6
#10 0x0000000000000000 in ?? ()


Crash4:

Thread 1 (Thread 22120):
#0  signal_segv (signum=11, info=0x7f8ef41b59b0, ptr=0x7f8ef41b5880) at ./Common/SIGSEGVHandler.cpp:128
#1  <signal handler called>
#2  0x00007f8ef8f877f0 in memcpy () from /lib/libc.so.6
#3  0x0000000000418d18 in refbuf_copy (orig=0x7f8ec99a2a00) at refbuf.c:74
#4  0x0000000000414a5f in source_listener_detach (source=0x7f8ed8004020, client=<value optimized out>) at source.c:885
#5  0x0000000000414d2f in source_listener_release (client=0x3d51c80) at source.c:1861
#6  send_to_listener (client=0x3d51c80) at source.c:958
#7  0x000000000041945b in worker (arg=<value optimized out>) at client.c:557
#8  0x00000000004309e9 in _start_routine (arg=0x24ec7f0) at thread.c:664
#9  0x00007f8ef8cf28ba in start_thread () from /lib/libpthread.so.0
#10 0x00007f8ef8fd702d in clone () from /lib/libc.so.6
#11 0x0000000000000000 in ?? ()


Crash5:

Thread 1 (Thread 25085):
#0  signal_segv (signum=11, info=0x0, ptr=<value optimized out>) at ./Common/SIGSEGVHandler.cpp:128
#1  <signal handler called>
#2  0x00007ff0a69797f0 in memcpy () from /lib/libc.so.6
#3  0x0000000000418d18 in refbuf_copy (orig=0x7ff06f977210) at refbuf.c:74
#4  0x0000000000414a5f in source_listener_detach (source=0xb7f330, client=<value optimized out>) at source.c:885
#5  0x0000000000414d2f in source_listener_release (client=0x7ff034c1d800) at source.c:1861
#6  send_to_listener (client=0x7ff034c1d800) at source.c:958
#7  0x000000000041945b in worker (arg=<value optimized out>) at client.c:557
#8  0x00000000004309e9 in _start_routine (arg=0xa1b270) at thread.c:664
#9  0x00007ff0a66e48ba in start_thread () from /lib/libpthread.so.0
#10 0x00007ff0a69c902d in clone () from /lib/libc.so.6
#11 0x0000000000000000 in ?? ()

Matthias

[karlheyes]

The refbuf_copy crashes will be down to the clearing of the queue occurring too early on a relay, should of been fixed on Jun 27. The send_iceblock_to_client crash (which is master feeding a relay), should of been fixed on Jul 6.

karl.

[Matthias78]

Great to hear.
As written in the other stats.c issue - I recompiled using your latest "slave.c".
So it should not crash anymore. Can't wait to see.. ;)

Matthias

[karlheyes]

still waiting? kh3 has been out for a little while now.

karl.

[Matthias78]

Hi Karl,

last week we switched to KH3.
So let's see what happens.
I'll close this issue.

Thank you!

Matthias