Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

stream_select warning FD_SETSIZE #1631

Closed
PDFCoder opened this issue Jan 7, 2019 · 21 comments
Closed

stream_select warning FD_SETSIZE #1631

PDFCoder opened this issue Jan 7, 2019 · 21 comments

Comments

@PDFCoder
Copy link

@PDFCoder PDFCoder commented Jan 7, 2019

We run into a problem on our live server (debian, PHP 7.1) with phpmailer 6.0.6 – everything has been working fine there with version 5.2.x.

If we try to send an email (can even be a simple text, or the smtp_check of the sample files!) we get a php warning and the server is running into a timeout:

PHP Warning: stream_select(): You MUST recompile PHP with a larger value of FD_SETSIZE.
It is set to 1024, but you have descriptors numbered at least as high as 1314.
--enable-fd-setsize=2048 is recommended, but you may want to set it
to equal the maximum number of open files supported by your system,
in order to avoid seeing this error again at a later date. in /home/www/xxx/phpmailer/src/SMTP.php on line 1125

Our hoster won't recompile PHP, so the suggested solution won't work.

We made some futher tests and found out, that the stream_select if condition in line 1125 didn't exist in phpmailer 5.2.x. If we delete this condition, everythings seems to work fine again (as far as we could test it now; with bigger mails, attachments).

if (!stream_select($selR, $selW, $selW, $this->Timelimit)) { …

If we set the timelimit of the function to a small value (like 3 seconds) the server loops a warning to the error log until the script is terminated:
PHP Warning: stream_select(): No stream arrays were passed in /home/www/xxx/phpmailer/src/SMTP.php

So what is the idea behind the stream_select condition? Is there a fix to this problem?

On other servers/accounts (same setup) the problem does not appear …!? Any idea?

@PDFCoder
Copy link
Author

@PDFCoder PDFCoder commented Jan 7, 2019

OK, as I see now, this might be a duplicate to #1618.

But do you see any disadvantages in deleting the stream_select if condition? As I understand, it should prevent a timeout under certain circumstances?

@Synchro
Copy link
Member

@Synchro Synchro commented Jan 7, 2019

This doesn't require recompiling PHP, but it does require being able to alter runtime file descriptor limits, usually via sysctl and ulimit, which you may also not have access to, but your ISP may be able to alter that. The origin of this use of stream select is here. You can see that it replaced a simpler timeout mechanism that failed in other ways. You could create a child class and override that usage (e.g. by reverting to the old implementation) as a workaround, but be aware that doing that may create other problems elsewhere.

@Synchro Synchro closed this Jan 7, 2019
@dpeca
Copy link

@dpeca dpeca commented Feb 5, 2019

Very interesting thing is that:

  • When PHP is running on Apache (mod_php) - bug occur
  • When PHP is running on PHP-FPM - there is no bug
  • When PHP is running from SSH console - there is no bug

Tested on the same machine.
Debian9, php packages from sury.org

@dpeca
Copy link

@dpeca dpeca commented Feb 5, 2019

And, btw, in sysctl.conf we have
fs.file-max = 200000
And in limits.conf we have

* soft     nofile         200000
* hard     nofile         200000

So file descriptors ar not limited at all...

@dpeca
Copy link

@dpeca dpeca commented Feb 5, 2019

Btw, solution that @PDFCoder found (to comment if (!stream_select) solved a problem.

@Synchro
Copy link
Member

@Synchro Synchro commented Feb 5, 2019

I'd recommend double-checking what the apache user's file limits actually are, for example run sudo -u www-data bash and then ulimit -n. IME it's common to find large global settings, but then to find that individual processes/users have the default 1024 handle limit.

@chriscroome
Copy link

@chriscroome chriscroome commented Feb 11, 2019

I've come across this issue and it doesn't appear to depend on the user that is running Apache and their file limits, it also doesn't appear to depend on PHP, Apache and OpenSSL being recompiled to support more than 1024 file descriptors, I have tried to address this by re-building the Debian packages and on a server running these rebuilt packages the test script posted to this issue can be run and will open more that 1024 sockets if edited to do so (it won't open more than 1024 sockets with the versions of Apache and PHP Debian provides).

This Plesk guide suggests that libc-client2002edebian also needs rebuilding, this is now the libc-client2007e package and it provides the uw-imap library, I couldn't get this to recompile, could this be where the problem lies?

@dpeca
Copy link

@dpeca dpeca commented Feb 13, 2019

@Synchro

root@server:~# sudo -u www-data bash
www-data@server:/root$ ulimit -n
200000

@Synchro
Copy link
Member

@Synchro Synchro commented Feb 13, 2019

OK, so it's not that. Can you see if your PHP is built with its own FD limit, as @PDFCoder mentioned?

@dpeca
Copy link

@dpeca dpeca commented Feb 13, 2019

I'm pretty sure we would have much larger issues if ulimit is only 1024...

@Synchro
How I can check it?

@chriscroome
Copy link

@chriscroome chriscroome commented Feb 13, 2019

This is how I checked the enable-fd-setsize variable on Debian Stretch:

apt install -y php-dev
php-config --configure-options | sed 's/\s\+/\n/g' | grep enable-fd-setsize

But having it set to a value higher than 1024 doesn't actually appear to help…

@dpeca
Copy link

@dpeca dpeca commented Feb 13, 2019

I got just empty output with it, @chriscroome

@dpeca
Copy link

@dpeca dpeca commented Feb 13, 2019

# php-config7.0 --configure-options
--includedir=/usr/include --mandir=/usr/share/man --infodir=/usr/share/info --disable-silent-rules --libdir=/usr/lib/x86_64-linux-gnu --libexecdir=/usr/lib/x86_64-linux-gnu --disable-maintainer-mode --disable-dependency-tracking --prefix=/usr --enable-cli --disable-cgi --disable-phpdbg --with-config-file-path=/etc/php/7.0/cli --with-config-file-scan-dir=/etc/php/7.0/cli/conf.d --build=x86_64-linux-gnu --host=x86_64-linux-gnu --config-cache --cache-file=/build/php7.0-7.0.33/config.cache --libdir=${prefix}/lib/php --libexecdir=${prefix}/lib/php --datadir=${prefix}/share/php/7.0 --program-suffix=7.0 --sysconfdir=/etc --localstatedir=/var --mandir=/usr/share/man --disable-all --disable-debug --disable-rpath --disable-static --with-pic --with-layout=GNU --without-pear --enable-filter --with-openssl=yes --with-pcre-regex=/usr --enable-hash --with-mhash=/usr --enable-libxml --enable-session --with-system-tzdata --with-zlib=/usr --with-zlib-dir=/usr --enable-dtrace --enable-pcntl --with-libedit=shared,/usr build_alias=x86_64-linux-gnu host_alias=x86_64-linux-gnu CFLAGS=-g -O2 -fdebug-prefix-map=/build/php7.0-7.0.33=. -fstack-protector-strong -Wformat -Werror=format-security -O2 -Wall -pedantic -fsigned-char -fno-strict-aliasing -g

There is no enable-fd-setsize

@chriscroome
Copy link

@chriscroome chriscroome commented Feb 13, 2019

Empty output means it has the default of 1024 as far as I could work out.

@chriscroome
Copy link

@chriscroome chriscroome commented Feb 13, 2019

I you have a dev / test Stretch server feel free to try the debs I built which are available here https://deb.webarch.net/ they have --enable-fd-setsize 2048 but servers running them still hit the limit, it appears to make no difference…

@Synchro
Copy link
Member

@Synchro Synchro commented Feb 13, 2019

I'm not sure at what point the limit is applied - for example if it's per-process and you're running under a non-threaded SAPI like mod_php, you're less likely to run into it than if you're using a threaded single process like PHP-FPM.

I'd also expect that if you're running into a limit like this, it's likely to occur in at least slightly random random places, rather than consistently the same place. It would be useful to get someone with PHP internals knowledge on this.

@dpeca
Copy link

@dpeca dpeca commented Feb 13, 2019

Both Apache (with mod_php) and PHP-FPM are configured to switch process UID and GID to user - so we do not run scripts as www-data, for sure.
For Apache we are using mod_ruid2 to switch UID and GID to script owner user...
I'm testing PHPMailer as user that only runs phpmailer test script.

And as I previously said, there is no issues if script is run through PHP-FPM.
Hard to believe that mod_ruid2 is not doing proper UID and GID switching... we are running 50 servers with mod_ruid2, everything works fine with switching process to UID and GID of script owner...

Again, I expect pretty much issues if scripts are reaching ulimits... but we don't have any issues, even we hosts 200 sites on that server.
This is the first time to see error about FD_SETSIZE...

@chriscroome
Copy link

@chriscroome chriscroome commented Feb 13, 2019

I'm using Apache 2 ITK MPM, it is in Debian, I haven't come across mod_ruid2 before, I see that this is also in Debian, I guess they do more-or-less that same thing…

@eldy
Copy link

@eldy eldy commented Feb 6, 2020

Same trouble here on ubuntu 18.04, apache2 and thousands of virtual hosts using their own UID and GID with mpm_itk.
The limit is reached only when using phpmailer. Other applications and other features are working correctly.
Note that the trouble occurs when doing an action as simple as sending 1 email and only 1 email from 1 instance. It is strange that the library needs to open so many files, just to send 1 email. The default limit of 1024 was never a trouble in several years of activity.
Nevermind, I confirm that solution that @PDFCoder found (to comment if (!stream_select) solved the problem on our side too.

@Synchro
Copy link
Member

@Synchro Synchro commented Feb 6, 2020

It's not the library opening "so many" files. It's the library opening 1 file, and PHP not having enough file descriptors available to let it happen.

@mj-mehdizadeh
Copy link

@mj-mehdizadeh mj-mehdizadeh commented Nov 25, 2020

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
6 participants