segfault at 0 ip 00007ffbbfa58646 sp 00007fff7c0c0d38 error 4 in libc-2.27.so #2216
Comments
|
That is the output of: "valgrind --leak-check=full --show-leak-kinds=all -v nginx" ==19276== Memcheck, a memory error detector |
|
And this is my nginx.conf file: |
|
Do you happens to have the core dump? |
|
Hi @zimmerle This is the output of: root@varnish62:~# gdb --pid 6938 This is the output of 6950 pid: Hope this will helps. |
|
It will be way better to have the debug symbols and the gdb-backtrace by the time that the process crash. |
|
And if I rebuild NGINX with "--widht-debug" flag you think it will behave the same as the package installed with apt? |
|
Hi @KfirManSnunit , The flag '--with-debug' can, in some cases, make a segfault go away (depending on the type of bug), but I don't think that's what he means. Have a look at: Enabling Core Dumps (https://docs.nginx.com/nginx/admin-guide/monitoring/debugging/#enabling-core-dumps) This won't alter the resulting executable in the same way that '--with-debug' can. |
|
Sill not manage to build nginx with debug, but a new error appear today: terminate called after throwing an instance of 'std::runtime_error' When the exited on signal 6 happens the number of active connections is up to 1000-4000+ In fact, there are only 350+- 2019/12/18 08:45:18 [alert] 12902#12902: worker process 12905 exited on signal 11 (core dumped) When the exited on signal 11 happens the number of reading connections is between 2-8 And the active connections is up to 1000-2000+ that in fact, there are only 350+- Hope this information can help, |
|
@martinhsv @zimmerle GNU gdb (Ubuntu 8.1-0ubuntu3.2) 8.1.0.20180409-git nginx log: syslog: I also checked other core dump files and the output looks the same... Thanks, |
|
Hi @KfirManSnunit Although it's not clear to me what combination of configuration and request is resulting in that execution path, there is at least a fairly obvious robustness issue in the code that we could mitigate. I'll do a bit more investigation sometime over the next couple of days and get back to you. |
|
Hi @martinhsv Another thing that may be related, The error logs of all websites was full of warnings like: Thanks in advance, |
|
Hi @KfirManSnunit , Thank you very much for the update. Based on that information, I'm not very likely to do further investigation. It sounds likely that there was an underlying bug in nginx code (possibly only in the non-stable version 1.17) that they have since corrected. Per my comment last week, in the coming days a small change may get merged into the ModSecurity nginx connector to exit more gracefully if that use case should arise again in future. |
|
@martinhsv patch is now merged! thank you guys. |
|
is this the same issue? ubuntu 18.04 LTS installed with kickstart.sh |
|
Hi @mailinglists35 , If you are using ModSecurity-nginx code that includes the above-mentioned fix ('Exit more gracefully if uri length is zero'), then what you have just mentioned shouldn't be the same issue. If you do not have that ModSecurity-nginx change, it could be the same issue if you are using versions of nginx mentioned by @KfirManSnunit . Beyond that, it's hard to conclude anything without a backtrace. |
|
I have no clue about what you're asking related to nginx. |
|
I don't even have nginx installed on that machine |
|
Hi @mailinglists35 , If you don't have nginx installed on your machine, then you are not encountering issue 2216. |
Hello,
I've been trying for a while to figure out what cause the problem for unkilled worker, without success.
It started to happen since I installed ModSecurity-nginx according to this guide:
https://www.nginx.com/blog/compiling-and-installing-modsecurity-for-open-source-nginx/
I've seen these cases:
#19
#29
And it did not work for me.
It happens randomly and without any action on our part like "restart\reload\nginx -t"... And usually not during rush hours.
I've open this issue: owasp-modsecurity/ModSecurity-nginx#136
That set as "duplicate/invalid" buy way?
My env info:
Ubuntu 18.04.3
Uname Output: Linux varnish62 4.15.0-70-generic #79-Ubuntu SMP Tue Nov 12 10:36:11 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
Nginx v1.17.6
Latest libmodsecurity
Latest ModSecurity-nginx
COMODO WAF set rules v3
syslog:
Nov 25 13:41:59 varnish62 kernel: [485324.474746] nginx[10347]: segfault at 0 ip 00007ffbbfa58646 sp 00007fff7c0c0d38 error 4 in libc-2.27.so[7ffbbf9a7000+1e7000]
Nov 25 13:43:00 varnish62 kernel: [485385.358807] nginx[10346]: segfault at 0 ip 00007ffbbfa58646 sp 00007fff7c0c0d38 error 4 in libc-2.27.so[7ffbbf9a7000+1e7000]
nginx log:
2019/11/25 13:41:59 [alert] 10345#10345: worker process 10347 exited on signal 11 (core dumped)
2019/11/25 13:43:00 [alert] 10345#10345: worker process 10346 exited on signal 11 (core dumped)
This is the stak trace from valgrind: "valgrind --leak-check=full -v nginx"
To Reproduce
None
Rule Set
Any help will be appreciated!
The text was updated successfully, but these errors were encountered: