Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

zend_signal_handler_defer crashes on apache shutdown #10015

Closed
psumbera opened this issue Nov 28, 2022 · 5 comments
Closed

zend_signal_handler_defer crashes on apache shutdown #10015

psumbera opened this issue Nov 28, 2022 · 5 comments
Labels
Bug SAPI: apache2handler Status: Needs Triage

Comments

@psumbera
Copy link
Contributor

Description

Core dump is generated during Apache shutdown.

Seems to be the same incarnation of:
https://bugs.php.net/bug.php?id=78619

This time on Solaris:

   7fc096c1fac61 libphp8.0.so`zend_signal_handler_defer+0x50(1, 7fc096c1fbb00, 7fc096c1fb820, 7fc097d02db70, 0, 7fc09776180a0)
   7fc096c1fad11 libc.so.1`__sighndlr+0xc(1, 7fc096c1fbb00, 7fc096c1fb820, 7fc097741d424, 0, 7fc097c9247c0)
   7fc096c1fadc1 libc.so.1`call_user_handler+0x400(fffffff7, 7fc096c1fbb00, 8, 7fc097d02da40, 1, fffffff7)
   7fc096c1faeb1 libc.so.1`sigacthandler+0xd0(1, 7fc096c1fbb00, 7fc096c1fb820, 34, 1, 7fc097c9247c0)
   7fc096c1fb401 libc.so.1`_portfs+8(8, e4aa7e2068, 34, 7fc096c1fbd70, 0, 0)
   7fc096c1fb4c1 libapr-1.so.0.7.0`impl_pollset_poll+0x284(e4aa7e1f58, ffffffffffffffff, 7fc096c1fbea4, 7fc096c1fbe80, e4aa7e1fb0, e4aa7e2068)
   7fc096c1fb591 mod_mpm_event.so`listener_thread+0x6b8(3, 7fc097bf00100, 7fc097be175fc, 7fc097be16000, 4e2e, 1117f)
   7fc096c1fb6f1 libc.so.1`_lwp_start(0, 0, 0, 0, 0, 0)

PHP Version

PHP 8.0

Operating System

Solaris 11.4
@cmb69
Copy link
Member

cmb69 commented Nov 28, 2022

Indeed, possibly a duplicate of https://bugs.php.net/bug.php?id=78619.

Anyhow, since PHP 8.0 is no longer actively supported, does this happen on PHP 8.1 too?

@cmb69
Copy link
Member

cmb69 commented Nov 30, 2022

Might also be related to #8789.

@psumbera
Copy link
Contributor Author

psumbera commented Dec 1, 2022

I can reproduce it with PHP 8.1 too. It's actually very easy:

cp /etc/apache2/2.4/samples-conf.d/php8.1.conf /etc/apache2/2.4/conf.d/
/usr/apache2/2.4/bin/apachectl start
cat > /var/apache2/2.4/htdocs/phpinfo.php <<EOF
<?php
phpinfo();
?>
EOF
wget localhost/phpinfo.php& /usr/apache2/2.4/bin/apachectl graceful

# cat /var/apache2/2.4/logs/error_log
[Thu Dec 01 15:05:16.261105 2022] [mpm_event:notice] [pid 7077:tid 1] AH00489: Apache/2.4.54 (Unix) PHP/8.1.9 configured -- resuming normal operations
[Thu Dec 01 15:05:16.261856 2022] [core:notice] [pid 7077:tid 1] AH00094: Command line: '/usr/apache2/2.4/bin/httpd'
[Thu Dec 01 15:05:25.824091 2022] [mpm_event:notice] [pid 7077:tid 1] AH00493: SIGUSR1 received.  Doing graceful restart
[Thu Dec 01 15:05:25.880343 2022] [mpm_event:notice] [pid 7077:tid 1] AH00489: Apache/2.4.54 (Unix) PHP/8.1.9 configured -- resuming normal operations
[Thu Dec 01 15:05:25.880432 2022] [core:notice] [pid 7077:tid 1] AH00094: Command line: '/usr/apache2/2.4/bin/httpd'
[Thu Dec 01 15:05:32.888902 2022] [core:notice] [pid 7077:tid 1] AH00051: child pid 7079 exit signal Segmentation fault (11), possible coredump in /usr/apache2/2.4

@arnaud-lb
Copy link
Member

Seems to be the same issue as #9649: the signal is delivered to a thread that didn't execute PHP yet. It crashes when zend_signal_handler_defer access thread globals.

0x00007f338867185e in zend_signal_handler_defer (signo=1, siginfo=0x7f3341ffa0b0, context=0x7f3341ff9f80) at /php-src/Zend/zend_signal.c:96
96		if (EXPECTED(SIGG(active))) {
(gdb) bt
#0  0x00007f338867185e in zend_signal_handler_defer (signo=1, siginfo=0x7f3341ffa0b0, context=0x7f3341ff9f80) at /php-src/Zend/zend_signal.c:96
#1  <signal handler called>
#2  0x00007f338961cfde in epoll_wait (epfd=8, events=0x7f33889a9620, maxevents=52, timeout=5000) at ../sysdeps/unix/sysv/linux/epoll_wait.c:30
#3  0x00007f33897476b0 in ?? () from /lib/x86_64-linux-gnu/libapr-1.so.0
#4  0x00007f3389383be8 in ?? () from /usr/lib/apache2/modules/mod_mpm_event.so
#5  0x00007f338958bb43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#6  0x00007f338961da00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
(gdb) list
91			zend_signal_handler(signo, siginfo, context);
92			return;
93		}
94	#endif
95	
96		if (EXPECTED(SIGG(active))) {
97			if (UNEXPECTED(SIGG(depth) == 0)) { /* try to handle signal */
98				if (UNEXPECTED(SIGG(blocked))) {
99					SIGG(blocked) = 0;
100				}

This has been fixed in master by #9766, however in hindsight this may not be ideal in this context.

jtojnar added a commit to jtojnar/nixfiles that referenced this issue Jan 14, 2023
@jtojnar
theo: Use php-fpm instead of mod_php
76aa389 
mod_php uses ZTS (thread-safe) build of PHP which is buggy.
E.g. causing Apache to crash php/php-src#10015.
Let’s switch to FastCGI instead.

I had to loosen the systemd service hardening to allow it access my home directory.
Took me a while to figure out too.

---

For debugging, I added “.mon” suffix to the handler socket and ran the following to monitor it:

    sudo socat -t100 -x -v UNIX-LISTEN:/run/phpfpm/dev.sock.mon,mode=777,user=wwwrun,group=wwwrun,reuseaddr,fork UNIX-CONNECT:/run/phpfpm/dev.sock

Source: https://superuser.com/a/576404/39197

Also used the following for more info:

    services.phpfpm.settings."log_level" = "debug";
    services.phpfpm.pools.dev.settings."access.log" = "/var/log/httpd/phpfpm-$pool.access";
@psumbera
Copy link
Contributor Author

This has been fixed in master by #9766, however in hindsight this may not be ideal in this context.

I have applied above pull request and I indeed cannot reproduce the issue (with PHP 8.2).

Will this be back ported to versions 8.1 and 8.2?

nielsdos added a commit to nielsdos/php-src that referenced this issue Mar 15, 2023
@nielsdos
Fix ZTS zend signal crashes due to NULL globals
d869ead 
Fixes phpGH-8789.
Fixes phpGH-10015.

This is one small part of the underlying bug for phpGH-10737, as in my
attempts to reproduce the issue I constantly hit this crash easily.
(The fix for the other underlying issue for that bug will follow soon.)

It's possible that a signal arrives at a thread that never handled a PHP
request before. This causes the signal globals to dereference a NULL
pointer because the TSRM pointers for the thread aren't set up to point
to the thread resources yet.

PR phpGH-9766 previously fixed this for master by ignoring the signal if
the thread didn't handle a PHP request yet. While this fixes the crash
bug, I think the solution is suboptimal for 3 reasons:

1) The signal is ignored and a message is printed saying there is a bug.
   However, this is not a bug at all. For example in Apache, the signal
   set up happens on child process creation, and the thread resource
   creation happens lazily when the first request is handled by the
   thread. Hence, the fact that the thread resources aren't set up yet
   is not actually buggy behaviour.

2) I believe since it was believed to be buggy behaviour, that fix was
   only applied to master, so 8.1 & 8.2 keep on crashing.

3) We can do better than ignoring the signal. By just initialising the
   resources if they don't exist, in the very same way the request
   handler in Apache works, we can gracefully handle the signal.
@nielsdos nielsdos mentioned this issue Mar 15, 2023
Closed
nielsdos added a commit to nielsdos/php-src that referenced this issue Mar 17, 2023
@nielsdos
Fix ZTS zend signal crashes due to NULL globals
9d9a3fb 
Fixes phpGH-8789.
Fixes phpGH-10015.

This is one small part of the underlying bug for phpGH-10737, as in my
attempts to reproduce the issue I constantly hit this crash easily.
(The fix for the other underlying issue for that bug will follow soon.)

It's possible that a signal arrives at a thread that never handled a PHP
request before. This causes the signal globals to dereference a NULL
pointer because the TSRM pointers for the thread aren't set up to point
to the thread resources yet.

PR phpGH-9766 previously fixed this for master by ignoring the signal if
the thread didn't handle a PHP request yet. While this fixes the crash
bug, I think the solution is suboptimal for 3 reasons:

1) The signal is ignored and a message is printed saying there is a bug.
   However, this is not a bug at all. For example in Apache, the signal
   set up happens on child process creation, and the thread resource
   creation happens lazily when the first request is handled by the
   thread. Hence, the fact that the thread resources aren't set up yet
   is not actually buggy behaviour.

2) I believe since it was believed to be buggy behaviour, that fix was
   only applied to master, so 8.1 & 8.2 keep on crashing.

3) We can do better than ignoring the signal. By just acting in the
   same way as if the signals aren't active. This means we need to
   take the same path as if the TSRM had already shut down.
nielsdos added a commit that referenced this issue Mar 18, 2023
@nielsdos
Merge branch 'PHP-8.1' into PHP-8.2
d479b5a 
* PHP-8.1:
  Fix GH-8789 and GH-10015: Fix ZTS zend signal crashes due to NULL globals
nielsdos added a commit that referenced this issue Mar 18, 2023
@nielsdos
Merge branch 'PHP-8.2'
33dd226 
* PHP-8.2:
  Fix GH-8789 and GH-10015: Fix ZTS zend signal crashes due to NULL globals
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Bug SAPI: apache2handler Status: Needs Triage
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@arnaud-lb @cmb69 @psumbera and others