SaltStack (using Python subprocess) hangs when running php-fpm 8.1.11

[bgdnlp]

Description

I am cross-posting saltstack/salt#62891 because I'm not sure what's happening, but it is related to PHP 8.1.11 and 8.0.24, previous versions worked fine.

I tested on Ubuntu with Python 3.10 and FreeBSD with Python 3.9. Running

salt-call -l all cmd.run 'php-fpm8.1'

on Ubuntu or the equivalent on FreeBSD just hangs. It didn't use to do that.

To reproduce:

• start an Ubuntu instance
• install PHP 8.1:
  • apt-get install software-properties-common
  • sudo apt-get install software-properties-common
  • sudo add-apt-repository ppa:ondrej/php
  • sudo apt-get update
  • sudo apt install php8.1-fpm
• install SaltStack:
  • sudo apt install python3-pip
  • sudo pip3 install --upgrade salt
  • sudo salt-minion <- run once. it will set up /etc, then it's not needed any more
• run some random command with SaltStack, just to see how it should work:
  • sudo salt-call -l all cmd.run 'ls'
• run php-fpm with SaltStack, this should just hang, although php-fpm does start:
  • sudo salt-call -l all cmd.run 'php-fpm8.1'

SaltStack hangs when it gets to this line: https://github.com/saltstack/salt/blob/427718c5aed40b376785ebc4813f5ccd18e07272/salt/utils/timed_subprocess.py#L90 . At that point php-fpm is already running, Salt is just trying to interact with php-fpm.

I wasn't able to reproduce in Python directly.

PHP Version

PHP 8.1.11 and PHP 8.0.24

Operating System

FreeBSD 13.1 and Ubuntu 22.04

[bukka]

I think you might want to run FPM in foreground in this case so maybe try to run FPM as php-fpm8.1 -F (basically just add -F option to run it in foreground). If you need it run as a root, you might also need to add -R option... See php-fpm8.1 --help for more details.

[bgdnlp]

I think you might want to run FPM in foreground

That's not an option, what I'm actually trying to do is start the php-fpm service, which means -D. That's the actual problem, I can no longer set up PHP through config management. The cmd.run line is just the easiest way to reproduce that I could find. If I ran it in the foreground, Salt would still hang, but then I wouldn't know if it is because of the bug, or because it's just waiting for the process to exit.

There are some other details in the related issue I opened in SaltStack repo.

[bukka]

Ok this might need some deeper investigation and might also need more looking to the SaltStack integration. I added that to my big TODO list but there are bunch of other higher priority tasks in front of it so it will likely take some time to get into it.

[bukka]

Actually I put a bit higher on my TODO list in case it's a regression. It shouldn't hopefully take that much time to look into it. So should be hopefully able to look more in the next few weeks.

[bukka]

Just a quick update that I spent some time on this today and did some debugging. I managed to recreate the issue under the Salt stack and then extract it to the subprocess call that hangs with PHP-8.0 but works with PHP-7.4:

import subprocess

kwargs = {
    'stdout': subprocess.PIPE,
    'stderr': subprocess.STDOUT,
    'executable': '/bin/bash',
    'shell': True,
}
proc = subprocess.Popen(['php-fpm8.0'], **kwargs)
out, err = proc.communicate()
print(out, err)

Now I will need to figure out what changed in 8.0 and why it's not working.

[bgdnlp]

It worked with the previous minor version of PHP too. PHP 8.1.10 didn't have this problem. Should help narrow it down.

[bukka]

Thanks that helped me to find the cause a bit quicker - the fix is in #9852 and should be hopefully part of PHP 8.0.26 and 8.1.13. I managed to put together a test for this so it won't hopefully happen again.