New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

mod_http2(v1.10.6) causes segmentation fault #142

Closed
j-ed opened this Issue Jun 23, 2017 · 22 comments

Comments

Projects
None yet
5 participants
@j-ed

j-ed commented Jun 23, 2017

Apache2: 2.4.26
mod_http2: 1.10.6
nghttp2: 1.23.1
openssl: 1.0.2l

After activating HTTP2 on my Apache2 server I wasn't able to use Nextcloud 12.0.0 anymore, because of regular segmentation faults (AH00052: child pid 3822 exit signal Segmentation fault (11)) happening when trying to access the web gui.
As soon as I disable http2 everything works like a charm again. Therefore mod_http2 might most likely be responsible for this problem. Unfortunately I don't now what kind of request Nextcloud is sending but I would expect that mod_http2 should handle all of them without crashing. The Nexcloud developers don't believe that the problem (nextcloud/server#4706) is in their responsibility therefore I want to point your attention to this problem.

@elukey

This comment has been minimized.

elukey commented Jun 23, 2017

It would be really helpful if you could provide a coredump with https://httpd.apache.org/docs/2.4/mod/mpm_common.html#coredumpdirectory, this will likely help a lot Stefan and others to track down the issue.

Thanks!

@icing

This comment has been minimized.

Owner

icing commented Jun 23, 2017

Thanks for the ticket and the link. What I could not see is if there is anything in the Apache error logs when this crash occurs. Also, a core file with a full stackdump would be really helpful.

It looks as if nextcloud is sending h2 frames in a way that mod_http2 has a problem with. This could be a bug or an assertion that was made to protect against "unforeseen" behaviour. The log/dump would tell.

In case you need help with creating this, there is (https://httpd.apache.org/dev/debugging.html)

@icing

This comment has been minimized.

Owner

icing commented Jun 23, 2017

@elukey hey, thanks!

@j-ed

This comment has been minimized.

j-ed commented Jun 23, 2017

@icing How can I send you the core file? Do you have a central upload server or email address?

@icing

This comment has been minimized.

Owner

icing commented Jun 23, 2017

stefan dot eissing add greenbytes dot de should work unless you have gigabytes...

@Sp1l

This comment has been minimized.

Contributor

Sp1l commented Jun 23, 2017

Successfully running Nextcloud 12.0 here with mod_http2 from Apache 2.4.26

  • FreeBSD 11-p9 amd64
  • LibreSSL 2.5.4
  • PHP 7.1.6 (as mod_php)
  • Apache 2.4.26 mod_event

I'd expect issues with PHP

@icing

This comment has been minimized.

Owner

icing commented Jun 23, 2017

Thanks @Sp1l !

Ok, easy questions for @j-ed:

  • what mpm are you running
  • what is the php setup?
@j-ed

This comment has been minimized.

j-ed commented Jun 23, 2017

I'm running a non-standard distribution, called Eisfair (http://www.eisfair.org/) on my server. PHP v5.6.30 is used together with the Apache 2.4.26 server. Independently of this environment some other users have reported that problem in the Nextcloud repository too, who are using other Linux distributions.

@icing

This comment has been minimized.

Owner

icing commented Jun 23, 2017

Hmm, I have trouble finding a matching httpd for the core file. Seems to be from a 32bit installation?
Do you have a gdb and can

 gdb /<path>/httpd core
 thread apply all bt

on it?

@icing

This comment has been minimized.

Owner

icing commented Jun 23, 2017

See that you are using prefork. Hmm, well. As @Sp1l already suggested, you probably see an issue with php.

The problem here is that php is not thread safe. That is why people run mpm-prefork as this, in http/1.1, processes only one request at a time. mod_http2 however opens a minimum of 4 worker threads by default. If 2 parallel php request come in, php probably explodes.

You can try H2MaxWorkers 1 to limit this, but then your http2 performance might be bad. It's not what it is intended to do.

My advice here would be: find a php setup that works nicely with mpm-event. Your Apache will be so much more powerful then.

@j-ed

This comment has been minimized.

j-ed commented Jun 23, 2017

Here's the requested gdb core dump (which seems to underpin your assumption?):

# gdb /usr/local/apache2/bin/httpd core
GNU gdb (GDB) SUSE (7.5.1-3.7)
Copyright (C) 2012 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "i586-pc-linux".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /usr/local/apache2/bin/httpd...(no debugging symbols found)...done.
[New LWP 7138]
[New LWP 7136]
[New LWP 7140]
[New LWP 7141]
[New LWP 7139]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/libthread_db.so.1".
Core was generated by `/usr/local/apache2/bin/httpd -k start'.
Program terminated with signal 11, Segmentation fault.
#0  0xb685b53e in ?? () from /usr/local/apache2/modules/libphp5.so
thread apply all bt

Thread 5 (Thread 0xa7bffb40 (LWP 7139)):
#0  0xb73efc3e in send () from /lib/libpthread.so.0
#1  0xb679c356 in ?? () from /usr/local/apache2/modules/libphp5.so
#2  0xb4f61226 in ?? () from /usr/lib/php5/extensions/openssl.so
#3  0xb678e326 in ?? () from /usr/local/apache2/modules/libphp5.so
#4  0xb46bdddb in redis_sock_write () from /usr/lib/php5/extensions/redis.so
#5  0xb46a95bb in zim_Redis_get () from /usr/lib/php5/extensions/redis.so
#6  0xb687cd18 in ?? () from /usr/local/apache2/modules/libphp5.so
#7  0xb6841896 in execute_ex () from /usr/local/apache2/modules/libphp5.so
#8  0xb67dbd42 in zend_execute_scripts () from /usr/local/apache2/modules/libphp5.so
#9  0xb687f83d in ?? () from /usr/local/apache2/modules/libphp5.so
#10 0x08096b43 in ap_run_handler ()
#11 0x08097019 in ap_invoke_handler ()
#12 0x080ad34a in ap_process_async_request ()
#13 0x080ad615 in ap_process_request ()
#14 0xb71c7ca0 in ?? () from /usr/local/apache2/modules/mod_http2.so
#15 0x080a1283 in ap_run_process_connection ()
#16 0xb71c92d1 in ?? () from /usr/local/apache2/modules/mod_http2.so
#17 0xb71cd52e in ?? () from /usr/local/apache2/modules/mod_http2.so
#18 0xb7474a14 in ?? () from /usr/lib/libapr-1.so.0
#19 0xb73e8bc7 in start_thread () from /lib/libpthread.so.0
#20 0xb733c4ae in clone () from /lib/libc.so.6

Thread 4 (Thread 0xa6bffb40 (LWP 7141)):
#0  0xb73ec882 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
#1  0xb7466bf9 in apr_thread_cond_wait () from /usr/lib/libapr-1.so.0
#2  0xb71cd4fe in ?? () from /usr/local/apache2/modules/mod_http2.so
#3  0xb7474a14 in ?? () from /usr/lib/libapr-1.so.0
#4  0xb73e8bc7 in start_thread () from /lib/libpthread.so.0
#5  0xb733c4ae in clone () from /lib/libc.so.6

Thread 3 (Thread 0xa73ffb40 (LWP 7140)):
#0  0xb73ec882 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
#1  0xb7466bf9 in apr_thread_cond_wait () from /usr/lib/libapr-1.so.0
#2  0xb71cd4fe in ?? () from /usr/local/apache2/modules/mod_http2.so
#3  0xb7474a14 in ?? () from /usr/lib/libapr-1.so.0
#4  0xb73e8bc7 in start_thread () from /lib/libpthread.so.0
#5  0xb733c4ae in clone () from /lib/libc.so.6

Thread 2 (Thread 0xb725f700 (LWP 7136)):
#0  0xb73ecc15 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib/libpthread.so.0
#1  0xb7466ca0 in apr_thread_cond_timedwait () from /usr/lib/libapr-1.so.0
#2  0xb71b583c in ?? () from /usr/local/apache2/modules/mod_http2.so
#3  0xb71c02cd in ?? () from /usr/local/apache2/modules/mod_http2.so
#4  0xb71aa33c in ?? () from /usr/local/apache2/modules/mod_http2.so
#5  0xb71b1293 in ?? () from /usr/local/apache2/modules/mod_http2.so
#6  0x080a1283 in ap_run_process_connection ()
#7  0xb72592bd in ?? () from /usr/local/apache2/modules/mod_mpm_prefork.so
#8  0xb7259542 in ?? () from /usr/local/apache2/modules/mod_mpm_prefork.so
#9  0xb72595bc in ?? () from /usr/local/apache2/modules/mod_mpm_prefork.so
#10 0xb725a2b8 in ?? () from /usr/local/apache2/modules/mod_mpm_prefork.so
#11 0x0807828f in ap_run_mpm ()
#12 0x08070635 in main ()

Thread 1 (Thread 0xb0557b40 (LWP 7138)):
#0  0xb685b53e in ?? () from /usr/local/apache2/modules/libphp5.so
#1  0xb6841896 in execute_ex () from /usr/local/apache2/modules/libphp5.so
#2  0xb67dbd42 in zend_execute_scripts () from /usr/local/apache2/modules/libphp5.so
#3  0xb677748b in php_execute_script () from /usr/local/apache2/modules/libphp5.so
#4  0xb687fc88 in ?? () from /usr/local/apache2/modules/libphp5.so
#5  0x08096b43 in ap_run_handler ()
#6  0x08097019 in ap_invoke_handler ()
#7  0x080ad34a in ap_process_async_request ()
#8  0x080ad615 in ap_process_request ()
#9  0xb71c7ca0 in ?? () from /usr/local/apache2/modules/mod_http2.so
#10 0x080a1283 in ap_run_process_connection ()
#11 0xb71c92d1 in ?? () from /usr/local/apache2/modules/mod_http2.so
#12 0xb71cd52e in ?? () from /usr/local/apache2/modules/mod_http2.so
#13 0xb7474a14 in ?? () from /usr/lib/libapr-1.so.0
#14 0xb73e8bc7 in start_thread () from /lib/libpthread.so.0
#15 0xb733c4ae in clone () from /lib/libc.so.6
(gdb)
@elukey

This comment has been minimized.

elukey commented Jun 23, 2017

My advice here would be: find a php setup that works nicely with mpm-event. Your Apache will be so much more powerful then.

https://wiki.apache.org/httpd/php is a good guide on the subject :)

@oerdnj

This comment has been minimized.

oerdnj commented Jun 29, 2017

I have reported similar issue to apache bugzilla: https://bz.apache.org/bugzilla/show_bug.cgi?id=61237

@oerdnj

This comment has been minimized.

oerdnj commented Jun 29, 2017

And it is a regression from 2.4.25 to 2.4.26

@icing

This comment has been minimized.

Owner

icing commented Jun 29, 2017

@oerdnj could try if

H2MaxWorkers 1 

brings back the old behaviour?

@oerdnj

This comment has been minimized.

oerdnj commented Jun 29, 2017

@icing One of the affected users has reported that it indeed fixes the segfault.

@icing

This comment has been minimized.

Owner

icing commented Jun 29, 2017

In hindsight, I should not have changed the default on this. 1 Worker in perfork was the behaviour before, but it causes other troubles. A browser will only use 1 connection and all streams will, since there is only 1 worker, be processed one after the other. Not very snappy.

@oerdnj

This comment has been minimized.

oerdnj commented Jun 29, 2017

@icing I guess there would be a simple patch for that? People that want better performance are using proxy_fcgi and php-fpm, so using libphp.so apache module is only for simple setups anyway.

@icing

This comment has been minimized.

Owner

icing commented Jun 30, 2017

@oerdnj I think the best patch would be to auto-disable h2* in prefork setups and put a WARNING in the logs. Servers will continue to work, but h2 will no longer be negotiated.

Thoughts? Comments?

@oerdnj

This comment has been minimized.

oerdnj commented Jun 30, 2017

If http2 module breaks the basic assumption of being forked (and no threaded) then I think this would be best course of action.

@icing

This comment has been minimized.

Owner

icing commented Jul 6, 2017

release v1.10.7 implements the proposed change and will be part of the upcoming 2.4.27 Apache release. If you find the time to test and report back this week, I'd appreciate it. Thanks!

@icing icing added the fix-available label Jul 6, 2017

@icing

This comment has been minimized.

Owner

icing commented Jul 31, 2017

Fixed in v1.10.7

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment