Polling readiness for PHP-FPM, NGINX, and Apache

[jaydrogers]

⏰ Background

• I am building "PHP-FPM + NGINX" and "PHP + Apache2" images with S6 Overlay 3.1.2.1

👉 Problem I am trying to solve

• Some users are reporting that NGINX is starting before PHP-FPM: https://community.serversideup.net/t/502-bad-gateway-on-google-cloud-run/700/3

🤔 My Approach to a Solution

Below details how I am approaching to solve this problem:

Upgrade & Convert "Longrun" services to 3.x standards

In my source code, I converted my "longrun" services to the new standard. This went well, thanks!

I need to create "readiness" script

According @jprjr to a post in 2018 (#245 (comment)), PHP-FPM needs to be polled because of the poor design of PHP-FPM.

I found the documentation on notifywhenup: https://skarnet.org/software/s6/notifywhenup.html

It talks about creating a polling script:

Create an executable script /run/service/foo/data/check that calls foo-check.

I am confused if this directory structure is with the new 3.x standards and where foo-check comes from.

⚡️ S6 Overlay Version

3.1.2.1

👨‍💻 My Source Code

All on GitHub: https://github.com/serversideup/docker-php/tree/dev/dist/8.1

Note: The fpm-nginx image is built from the fpm image (so you understand the image dependency and what's included).

Easily pull and inspect any Docker image with bash

# FPM
docker run --rm -it serversideup/php:beta-8.1-fpm bash

# FPM-NGINX
docker run --rm -it serversideup/php:beta-8.1-fpm-nginx bash

# FPM-Apache
docker run --rm -it serversideup/php:beta-8.1-fpm-apache bash

❓ Questions

1. Are there any examples available that already poll things like PHP-FPM, Apache, or NGINX?
2. Is the documentation I'm reading the latest in regards to S6 Overlay 3.x?

Thanks for your help!

[skarnet]

On vacation at the moment, sorry for the brief answer.

The data subdirectory of a s6-rc longrun source definition directory will be copied verbatim to the service directory it creates. So in your case, you should add your check script in
/etc/s6-overlay/s6-rc.d/php-fpm/data/check, and make it executable. And change your /etc/s6-overlay/s6-rc.d/php-fpm/run script to the following:

#!/command/execlineb -P
with-contenv
s6-notifyoncheck
s6-setuidgid webuser
/usr/sbin/php-fpm8.1 --nodaemonize

(this is a translation of your current run script that will work with s6-notifyoncheck)

[jaydrogers]

Thank you very much! This helps out a ton.

I have some examples working locally that I will push up once I figure out how to have PHP-FPM tell me what I need it to 😅

I will loop back with my examples once I have them complete. Feel free to use them to help others if needed.

I'll keep you posted, enjoy your vacation! ☀️🏖

[jaydrogers]

Update: I got it working 🎉

Thanks for your help @skarnet!

You pushed me in the right direction and I was able to get everything functioning.

I'm currently beta testing my changes, but here's the most important notes for others that are coming here for PHP-FPM healthchecks:

How I am defining "healthy" for PHP-FPM

We all know PHP-FPM isn't the most modern daemon out there, so defining "healthy" can be challenging. I was able to do it with php-fpm-healthcheck. This essentially checks against the /status page built-in with PHP-FPM.

After I had things configured, I ran into another error

PHP-FPM was throwing this error on container start up:

ALERT: oops, unknown child (95) exited with code 0. Please open a bug report (https://github.com/php/php-src/issues).

I assumed it had something to do with the timing of s6-notifyoncheck. Adjusting the delay had no effect on this. It just delayed when this error would appear.

Thankfully I noticed this in the docs:

-d : doublefork. s6-notifyoncheck will run as the grandchild of prog... instead of its direct child. This is useful if prog... never reaps zombies it does not know it has.

Taking a wild guess (because I know PHP-FPM has a parent and child processes), enabling -d prevented this message.

Here is an example of my run script:

#!/command/execlineb -P
with-contenv
s6-notifyoncheck -d
exec /usr/sbin/php-fpm8.1 --nodaemonize

Source code

For anyone who needs to reference how I solved the full solution, here's all of my source code: https://github.com/serversideup/docker-php/tree/dev/dist/8.1

I am very grateful for this project by all of it's contributors. 🙌

Feel free to use my source code as an example for others or submit any improvements if you see any areas that could use work.

Thanks again!

[skarnet]

Congrats on getting it to work! And yes, the -d option to s6-notifyoncheck is the correct workaround to the spurious php-fpm error message. (Spurious because on Unix it is normal to have children you don't know you have, so no program should treat this as an error condition.)

Note that you can still use s6-setuidgid webuser if you want php-fpm to run as webuser instead of root, you just need to add the call to s6-setuidgid after the call to s6-notifyoncheck. Also, if your run script is an execline script (as opposed to a shell script), you don't need the exec keyword.

Cheers!

[jaydrogers]

Thanks for your feedback!

If your run script is an execline script (as opposed to a shell script), you don't need the exec keyword.

I removed that extra exec command and it works great, thanks!

Note that you can still use s6-setuidgid webuser if you want php-fpm to run as webuser instead of root, you just need to add the call to s6-setuidgid after the call to s6-notifyoncheck

When I made this change, I get this error:

[30-Sep-2022 14:14:31] ERROR: FPM initialization failed
[30-Sep-2022 14:14:32] ERROR: failed to open error_log (/proc/self/fd/2): Permission denied (13)

Probable Cause

I set a setting in PHP-FPM to redirect output to stdout/stderr:
https://github.com/serversideup/docker-php/blob/dev/src/fpm/etc/php/fpm/pool.d/y-override-php-defaults.conf#L11-L12

(my workaround to run as webuser and NOT root has been a config within PHP-FPM: https://github.com/serversideup/docker-php/blob/dev/src/fpm/etc/php/fpm/pool.d/y-override-php-defaults.conf#L25-L29)

Question

1. Is there a better way to redirect output with PHP-FPM + S6 Overlay (instead of /proc/self/fd/2)?

I am grateful for your advice and expertise 🙌

[skarnet]

If fpm closes its stdout/stderr and can only reopen them by specifying /proc/self/fd/1 and /proc/self/fd/2 in its config file, then no, there's no better way and you're doing the right thing by also dropping privileges in the config file.

Ideally, fpm would not do that, and you could just tell it to write its access logs to stdout and its error logs to stderr, without doing the close/reopen dance; in which case you could drop privileges before running it. But it looks like it's neither the first nor the last of fpm's misdesigns, and we just have to live with it.

[jaydrogers]

We are definitely on the same page about PHP-FPM's design! 😆

I greatly appreciate your comments and help on PHP-FPM (even though it's completely out of scope of this project).

Thank you so much for this incredible project! 👍