New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

stream_select warning FD_SETSIZE #1631

Closed
PDFCoder opened this Issue Jan 7, 2019 · 18 comments

Comments

Projects
None yet
4 participants
@PDFCoder
Copy link

PDFCoder commented Jan 7, 2019

We run into a problem on our live server (debian, PHP 7.1) with phpmailer 6.0.6 – everything has been working fine there with version 5.2.x.

If we try to send an email (can even be a simple text, or the smtp_check of the sample files!) we get a php warning and the server is running into a timeout:

PHP Warning: stream_select(): You MUST recompile PHP with a larger value of FD_SETSIZE.
It is set to 1024, but you have descriptors numbered at least as high as 1314.
--enable-fd-setsize=2048 is recommended, but you may want to set it
to equal the maximum number of open files supported by your system,
in order to avoid seeing this error again at a later date. in /home/www/xxx/phpmailer/src/SMTP.php on line 1125

Our hoster won't recompile PHP, so the suggested solution won't work.

We made some futher tests and found out, that the stream_select if condition in line 1125 didn't exist in phpmailer 5.2.x. If we delete this condition, everythings seems to work fine again (as far as we could test it now; with bigger mails, attachments).

if (!stream_select($selR, $selW, $selW, $this->Timelimit)) { …

If we set the timelimit of the function to a small value (like 3 seconds) the server loops a warning to the error log until the script is terminated:
PHP Warning: stream_select(): No stream arrays were passed in /home/www/xxx/phpmailer/src/SMTP.php

So what is the idea behind the stream_select condition? Is there a fix to this problem?

On other servers/accounts (same setup) the problem does not appear …!? Any idea?

@PDFCoder

This comment has been minimized.

Copy link
Author

PDFCoder commented Jan 7, 2019

OK, as I see now, this might be a duplicate to #1618.

But do you see any disadvantages in deleting the stream_select if condition? As I understand, it should prevent a timeout under certain circumstances?

@Synchro

This comment has been minimized.

Copy link
Member

Synchro commented Jan 7, 2019

This doesn't require recompiling PHP, but it does require being able to alter runtime file descriptor limits, usually via sysctl and ulimit, which you may also not have access to, but your ISP may be able to alter that. The origin of this use of stream select is here. You can see that it replaced a simpler timeout mechanism that failed in other ways. You could create a child class and override that usage (e.g. by reverting to the old implementation) as a workaround, but be aware that doing that may create other problems elsewhere.

@Synchro Synchro closed this Jan 7, 2019

@dpeca

This comment has been minimized.

Copy link

dpeca commented Feb 5, 2019

Very interesting thing is that:

  • When PHP is running on Apache (mod_php) - bug occur
  • When PHP is running on PHP-FPM - there is no bug
  • When PHP is running from SSH console - there is no bug

Tested on the same machine.
Debian9, php packages from sury.org

@dpeca

This comment has been minimized.

Copy link

dpeca commented Feb 5, 2019

And, btw, in sysctl.conf we have
fs.file-max = 200000
And in limits.conf we have

* soft     nofile         200000
* hard     nofile         200000

So file descriptors ar not limited at all...

@dpeca

This comment has been minimized.

Copy link

dpeca commented Feb 5, 2019

Btw, solution that @PDFCoder found (to comment if (!stream_select) solved a problem.

@Synchro

This comment has been minimized.

Copy link
Member

Synchro commented Feb 5, 2019

I'd recommend double-checking what the apache user's file limits actually are, for example run sudo -u www-data bash and then ulimit -n. IME it's common to find large global settings, but then to find that individual processes/users have the default 1024 handle limit.

@chriscroome

This comment has been minimized.

Copy link

chriscroome commented Feb 11, 2019

I've come across this issue and it doesn't appear to depend on the user that is running Apache and their file limits, it also doesn't appear to depend on PHP, Apache and OpenSSL being recompiled to support more than 1024 file descriptors, I have tried to address this by re-building the Debian packages and on a server running these rebuilt packages the test script posted to this issue can be run and will open more that 1024 sockets if edited to do so (it won't open more than 1024 sockets with the versions of Apache and PHP Debian provides).

This Plesk guide suggests that libc-client2002edebian also needs rebuilding, this is now the libc-client2007e package and it provides the uw-imap library, I couldn't get this to recompile, could this be where the problem lies?

@dpeca

This comment has been minimized.

Copy link

dpeca commented Feb 13, 2019

@Synchro

root@server:~# sudo -u www-data bash
www-data@server:/root$ ulimit -n
200000

@Synchro

This comment has been minimized.

Copy link
Member

Synchro commented Feb 13, 2019

OK, so it's not that. Can you see if your PHP is built with its own FD limit, as @PDFCoder mentioned?

@dpeca

This comment has been minimized.

Copy link

dpeca commented Feb 13, 2019

I'm pretty sure we would have much larger issues if ulimit is only 1024...

@Synchro
How I can check it?

@chriscroome

This comment has been minimized.

Copy link

chriscroome commented Feb 13, 2019

This is how I checked the enable-fd-setsize variable on Debian Stretch:

apt install -y php-dev
php-config --configure-options | sed 's/\s\+/\n/g' | grep enable-fd-setsize

But having it set to a value higher than 1024 doesn't actually appear to help…

@dpeca

This comment has been minimized.

Copy link

dpeca commented Feb 13, 2019

I got just empty output with it, @chriscroome

@dpeca

This comment has been minimized.

Copy link

dpeca commented Feb 13, 2019

# php-config7.0 --configure-options
--includedir=/usr/include --mandir=/usr/share/man --infodir=/usr/share/info --disable-silent-rules --libdir=/usr/lib/x86_64-linux-gnu --libexecdir=/usr/lib/x86_64-linux-gnu --disable-maintainer-mode --disable-dependency-tracking --prefix=/usr --enable-cli --disable-cgi --disable-phpdbg --with-config-file-path=/etc/php/7.0/cli --with-config-file-scan-dir=/etc/php/7.0/cli/conf.d --build=x86_64-linux-gnu --host=x86_64-linux-gnu --config-cache --cache-file=/build/php7.0-7.0.33/config.cache --libdir=${prefix}/lib/php --libexecdir=${prefix}/lib/php --datadir=${prefix}/share/php/7.0 --program-suffix=7.0 --sysconfdir=/etc --localstatedir=/var --mandir=/usr/share/man --disable-all --disable-debug --disable-rpath --disable-static --with-pic --with-layout=GNU --without-pear --enable-filter --with-openssl=yes --with-pcre-regex=/usr --enable-hash --with-mhash=/usr --enable-libxml --enable-session --with-system-tzdata --with-zlib=/usr --with-zlib-dir=/usr --enable-dtrace --enable-pcntl --with-libedit=shared,/usr build_alias=x86_64-linux-gnu host_alias=x86_64-linux-gnu CFLAGS=-g -O2 -fdebug-prefix-map=/build/php7.0-7.0.33=. -fstack-protector-strong -Wformat -Werror=format-security -O2 -Wall -pedantic -fsigned-char -fno-strict-aliasing -g

There is no enable-fd-setsize

@chriscroome

This comment has been minimized.

Copy link

chriscroome commented Feb 13, 2019

Empty output means it has the default of 1024 as far as I could work out.

@chriscroome

This comment has been minimized.

Copy link

chriscroome commented Feb 13, 2019

I you have a dev / test Stretch server feel free to try the debs I built which are available here https://deb.webarch.net/ they have --enable-fd-setsize 2048 but servers running them still hit the limit, it appears to make no difference…

@Synchro

This comment has been minimized.

Copy link
Member

Synchro commented Feb 13, 2019

I'm not sure at what point the limit is applied - for example if it's per-process and you're running under a non-threaded SAPI like mod_php, you're less likely to run into it than if you're using a threaded single process like PHP-FPM.

I'd also expect that if you're running into a limit like this, it's likely to occur in at least slightly random random places, rather than consistently the same place. It would be useful to get someone with PHP internals knowledge on this.

@dpeca

This comment has been minimized.

Copy link

dpeca commented Feb 13, 2019

Both Apache (with mod_php) and PHP-FPM are configured to switch process UID and GID to user - so we do not run scripts as www-data, for sure.
For Apache we are using mod_ruid2 to switch UID and GID to script owner user...
I'm testing PHPMailer as user that only runs phpmailer test script.

And as I previously said, there is no issues if script is run through PHP-FPM.
Hard to believe that mod_ruid2 is not doing proper UID and GID switching... we are running 50 servers with mod_ruid2, everything works fine with switching process to UID and GID of script owner...

Again, I expect pretty much issues if scripts are reaching ulimits... but we don't have any issues, even we hosts 200 sites on that server.
This is the first time to see error about FD_SETSIZE...

@chriscroome

This comment has been minimized.

Copy link

chriscroome commented Feb 13, 2019

I'm using Apache 2 ITK MPM, it is in Debian, I haven't come across mod_ruid2 before, I see that this is also in Debian, I guess they do more-or-less that same thing…

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment