worker process exited on signal 11 #470
Comments
|
I made some progress with gdb: Program received signal SIGSEGV, Segmentation fault. |
|
which nginx version ? |
|
/usr/local/nginx/sbin/nginx -V |
|
Got the same shit on 1.7.5 |
|
I am using 1.7.5, everything seems OK! |
|
Same problem on 1.7.5 , OS: Centos 6 |
|
The problem seems to occur with random static cam like these: |
|
Just FYI. See also comments in #472. |
|
I have recompiled nginx with debug option to get some more info: 2014/09/19 12:31:13 [debug] 2503#0: posted event 0000000000000000 |
|
Please update the code from master. It should be fixed now. |
|
I rebuild the server with nginx 1.6.2 and git://github.com/arut/nginx-rtmp-module.git |
|
@arut Awesome! Just tested, works like a charm in both scenarios described above (HTTP only and with RTMP config). Thanks again a ton for fixing this so quickly. |
|
The config what I use for testing and, still get these errors: |
|
Without the pull section I don't get these errors, but with the static cams the worker process stops approximately in every 10 min. |
|
@gmarcz what if you recompile with nginx 1.7.5? |
|
Strange because I got the same errors with 1.7.5 nginx version: nginx/1.7.5 2014/09/22 13:23:19 [alert] 24#0: worker process 3841 exited on signal 11 Program received signal SIGSEGV, Segmentation fault. |
|
@gmarcz please set up debug logging and post more lines from error.log |
|
@gmarcz Didn't wait long enough during my tests... I get the signal 11 error too after about 20min. |
|
These are the last few lines from the exiting process: 2014/09/22 15:43:00 [debug] 353#0: *15 nhandlers: 5 |
|
Same problem in 1.7.5 |
|
Please try Thanks everyone in this thread for the reports. I'm waiting for feedback. |
|
I've tested your fix playing 2 random streams from the configuration for 20min+; videos are still playing, no entries in error log. I'll leave them running for longer, but it looks good to me. My compile configuration (with branch static-relay-cleanup): nginx version: nginx/1.7.5 |
|
Update: 1h40m and the videos are still streaming with above configuration. I guess this works. |
|
@arut thank you for your effort, I will test it with 1.6.2 and let you know if I found anything. |
|
Server is running about 2h 49m seems to be stable for now. |
|
We have the same problem with pulling, but our pulls aren't static. 2014/09/24 04:13:59 [alert] 15065#0: worker process 29253 exited on signal 11 nginx 1.7.5, latest nginx-rtmp version. The static-relay-cleanup didn't help. It's hard to debug because of heavy traffic. Will the debug log be enough, or I should also run gdb? |
|
@developer222 yes, please report gdb backtrace |
|
Hello, We have a similar problem with dying worker on a server with high number of active streams (1000-1500). The traffic isn't so big, about 300 Mbps to 1.5 Gbps. I've recompiled static-cleanup-branch and nginx 1.6.2 with debug enabled and here is what we have in gdb output:Core was generated by `nginx: worker process '. ccf = 0x2236258nginx -V nginx config: user nginx; for debug only:worker_rlimit_core 2000M; error_log /var/log/nginx/error.log warn; events { rtmp { ... Do you have any insights how to mitigate this issue? Thanks! |
|
@arut , sorry for the long delay. Here's the gdb output with backtrace. What else may I provide to clarify the problem? Thanks. gdb /usr/sbin/nginx.debug core warning: Can't read pathname for load map: Input/output error. |
|
Hi, I have came same error in nginx 1.10 version. Can you explain where is exact problem. |
|
Still there today on Nginx 1.11.1 |
|
try add |
Hi All!
From time to time I see this error in error.log
2014/09/15 11:34:24 [alert] 27#0: worker process 28 exited on signal 11
2014/09/15 11:44:28 [alert] 27#0: worker process 21539 exited on signal 11
2014/09/15 12:34:48 [alert] 27#0: worker process 9652 exited on signal 11
2014/09/15 12:44:53 [alert] 27#0: worker process 15638 exited on signal 11
2014/09/15 12:49:20 [alert] 27#0: worker process 32588 exited on signal 11
2014/09/15 12:53:49 [alert] 27#0: worker process 4974 exited on signal 11
2014/09/15 13:12:06 [alert] 27#0: worker process 10578 exited on signal 11
2014/09/15 13:28:17 [alert] 27#0: worker process 2807 exited on signal 11
2014/09/15 13:35:06 [alert] 27#0: worker process 30753 exited on signal 11
I don't have to much experience to debug such problem, what I tried was to attache strace to the worker process, the last few lines from the log:
kill(26701, SIGKILL) = 0
--- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_KILLED, si_pid=26701, si_status=SIGKILL, si_utime=1, si_stime=0} ---
gettimeofday({1410787697, 694141}, NULL) = 0
wait4(-1, [{WIFSIGNALED(s) && WTERMSIG(s) == SIGKILL}], WNOHANG, NULL) = 26701
write(4, "2014/09/15 13:28:17 [alert] 2807"..., 77) = 77
wait4(-1, 0x7fff044965fc, WNOHANG, NULL) = 0
rt_sigreturn() = 0
sendto(63, "\2\0\0\0\0\0\6\4\0\0\0\0\0\1\0\0\0\1", 18, 0, NULL, 0) = -1 EPIPE (Broken pipe)
--- SIGPIPE {si_signo=SIGPIPE, si_code=SI_USER, si_pid=2807, si_uid=65534} ---
close(16) = 0
close(63) = 0
--- SIGSEGV {si_signo=SIGSEGV, si_code=SEGV_MAPERR, si_addr=0x10} ---
+++ killed by SIGSEGV +++
Can someone help to identify the root case of this problem ?
Thank you for your time.
Regards
Gabor
The text was updated successfully, but these errors were encountered: