Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Comments included as part of minfds value #1230

Closed
mnaberez opened this issue Apr 17, 2019 · 19 comments

Comments

4 participants
@mnaberez
Copy link
Member

commented Apr 17, 2019

@BillMedernach wrote in #1229 (comment):

I'm running into the same error on the minfds config param where the comment is getting parsed as part of the configuration value. If I downgrade from 4.0.0 or 4.0.1 to 3.3.5 the issue goes away.

@BillMedernach

This comment has been minimized.

Copy link

commented Apr 17, 2019

Error: Error: invalid literal for long() with base 10: '1024 ; (min. avail startup file descriptors;default 1024)'

Configuration excerpt:

[supervisord]
logfile=/mnt/md0/server/logs/supervisord.log ; (main log file;default $CWD/supervisord.log)
logfile_maxbytes=50MB                              ; (max main logfile bytes b4 rotation;default 50MB)
logfile_backups=10                                 ; (num of main logfile rotation backups;default 10)
loglevel=info                                      ; (log level;default info; others: debug,warn,trace)
pidfile=/mnt/md0/server/logs/supervisord.pid ; (supervisord pidfile;default supervisord.pid)
nodaemon=false                                     ; (start in foreground if true;default false)
minfds=1024                                        ; (min. avail startup file descriptors;default 1024)
minprocs=200                                       ; (min. avail process descriptors;default 200)

Removing the comment or downgrading to 3.3.5 fixes the problem.

@mnaberez

This comment has been minimized.

Copy link
Member Author

commented Apr 17, 2019

@BillMedernach What Python version are you using (python -V)?

mnaberez added a commit that referenced this issue Apr 17, 2019

@BillMedernach

This comment has been minimized.

Copy link

commented Apr 17, 2019

@mnaberez We're running 2.7.15

@mnaberez

This comment has been minimized.

Copy link
Member Author

commented Apr 17, 2019

I added tests for this case (minfds) and it passes on CI. The CI environment is using 2.7.14. I was not able to reproduce this locally yet.

Are you able to consistently reproduce it with the config above?

@BillMedernach

This comment has been minimized.

Copy link

commented Apr 17, 2019

Are you able to consistently reproduce it with the config above?

Yes, that config always produces that error on v4.0.0+.

@mnaberez

This comment has been minimized.

Copy link
Member Author

commented Apr 17, 2019

Could you please attach the actual supervisord.conf file where you're seeing this happen? You can drag it into the GitHub comment box, or alternatively paste the output of either hexdump supervisord.conf or base64 supervisord.conf. The tests that I added cover both spaces and tabs before the semicolon, so I don't know what is wrong. I will try to reproduce it using your byte-for-byte exact config file. Hopefully that reproduces it and then we look into what's wrong.

@BillMedernach

This comment has been minimized.

Copy link

commented Apr 17, 2019

[unix_http_server]
file=/tmp/supervisor.sock   ; (the path to the socket file)
chmod=0700                 ; sockef file mode (default 0700)
;chown=nobody:nogroup       ; socket file uid:gid owner
;username=REDACTED              ; (default is no username (open server))
;password=REDACTED               ; (default is no password (open server))

[supervisord]
logfile=/mnt/md0/server/logs/supervisord.log ; (main log file;default $CWD/supervisord.log)
logfile_maxbytes=50MB                              ; (max main logfile bytes b4 rotation;default 50MB)
logfile_backups=10                                 ; (num of main logfile rotation backups;default 10)
loglevel=info                                      ; (log level;default info; others: debug,warn,trace)
pidfile=/mnt/md0/server/logs/supervisord.pid ; (supervisord pidfile;default supervisord.pid)
nodaemon=false                                     ; (start in foreground if true;default false)
minfds=1024                                        ; (min. avail startup file descriptors;default 1024)
minprocs=200                                       ; (min. avail process descriptors;default 200)

[rpcinterface:supervisor]
supervisor.rpcinterface_factory = supervisor.rpcinterface:make_main_rpcinterface

[supervisorctl]
serverurl=unix:///tmp/supervisor.sock ; use a unix:// URL  for a unix socket
history_file=~/.sc_history            ; use readline history if available

[program:dyno]
command=/srv/server/bin/dyno foreground
directory=/srv/server
user=root
startsecs=10
autorestart=true
stopsignal=TERM
stopwaitsecs=3600
redirect_stderr=true
stdout_logfile=/mnt/md0/server/logs/subprocess.log
environment=PATH=/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin,HOME=/root

[program:check_nginx]
command=/srv/server/bin/check_nginx_keepalive.py
directory=/srv/server
user=root
startsecs=10
autorestart=true
stopsignal=TERM
stopwaitsecs=3600
redirect_stderr=true
stdout_logfile=/mnt/md0/server/logs/subprocess-check-nginx.log
environment=PATH=/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin,HOME=/root

[program:check_server]
command=/srv/server/bin/check_server.py
directory=/srv/server
user=root
startsecs=10
autorestart=true
stopsignal=TERM
stopwaitsecs=3600
redirect_stderr=true
stdout_logfile=/mnt/md0/server/logs/subprocess-check-server.log
environment=PATH=/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin,HOME=/root
@mnaberez

This comment has been minimized.

Copy link
Member Author

commented Apr 17, 2019

Here's what I get with that config. This is Supervisor 4.0.1 running on Python 2.7.15:

2019-04-17 10:12:48,177 CRIT Supervisor is running as root.  Privileges were not dropped because no user is specified in the config file.  If you intend to run as root, you can set user=root in the config file to avoid this message.
2019-04-17 10:12:48,177 INFO Increased RLIMIT_NOFILE limit to 1024
2019-04-17 10:12:48,190 INFO RPC interface 'supervisor' initialized
2019-04-17 10:12:48,190 CRIT Server 'unix_http_server' running without any HTTP authentication checking
2019-04-17 10:12:48,190 INFO supervisord started with pid 3011
2019-04-17 10:12:49,194 INFO spawnerr: can't find command '/srv/server/bin/dyno'
2019-04-17 10:12:49,195 INFO spawnerr: can't find command '/srv/server/bin/check_server.py'
2019-04-17 10:12:49,195 INFO spawnerr: can't find command '/srv/server/bin/check_nginx_keepalive.py'
2019-04-17 10:12:50,196 INFO spawnerr: can't find command '/srv/server/bin/dyno'
2019-04-17 10:12:50,197 INFO spawnerr: can't find command '/srv/server/bin/check_server.py'
2019-04-17 10:12:50,197 INFO spawnerr: can't find command '/srv/server/bin/check_nginx_keepalive.py'
2019-04-17 10:12:52,200 INFO spawnerr: can't find command '/srv/server/bin/dyno'
2019-04-17 10:12:52,201 INFO spawnerr: can't find command '/srv/server/bin/check_server.py'
2019-04-17 10:12:52,201 INFO spawnerr: can't find command '/srv/server/bin/check_nginx_keepalive.py'
2019-04-17 10:12:55,208 INFO spawnerr: can't find command '/srv/server/bin/dyno'
2019-04-17 10:12:55,209 INFO gave up: dyno entered FATAL state, too many start retries too quickly
2019-04-17 10:12:55,209 INFO spawnerr: can't find command '/srv/server/bin/check_server.py'
2019-04-17 10:12:55,209 INFO gave up: check_server entered FATAL state, too many start retries too quickly
2019-04-17 10:12:55,209 INFO spawnerr: can't find command '/srv/server/bin/check_nginx_keepalive.py'
2019-04-17 10:12:55,209 INFO gave up: check_nginx entered FATAL state, too many start retries too quickly

It is not halting with Error: invalid literal for long() with base 10: '1024 ; (min. avail startup file descriptors;default 1024)'. There are errors from programs not starting but the midfds error would have occurred before it tried to spawn them.

Maybe something is getting lost with copy/paste. Can you please hexdump supervisord.conf or base64 supervisord.conf? I will try it with your byte-for-byte exact config file. Please be certain this is exact config file where you are consistently seeing the error.

@BillMedernach

This comment has been minimized.

Copy link

commented Apr 17, 2019

Here's the base64. I did redact the username and password we use though.

W3VuaXhfaHR0cF9zZXJ2ZXJdCmZpbGU9L3RtcC9zdXBlcnZpc29yLnNvY2sgICA7ICh0aGUgcGF0aCB0byB0aGUgc29ja2V0IGZpbGUpCmNobW9kPTA3MDAgICAgICAgICAgICAgICAgIDsgc29ja2VmIGZpbGUgbW9kZSAoZGVmYXVsdCAwNzAwKQo7Y2hvd249bm9ib2R5Om5vZ3JvdXAgICAgICAgOyBzb2NrZXQgZmlsZSB1aWQ6Z2lkIG93bmVyCjt1c2VybmFtZT1SRURBQ1RFRCAgICAgICAgICAgICAgOyAoZGVmYXVsdCBpcyBubyB1c2VybmFtZSAob3BlbiBzZXJ2ZXIpKQo7cGFzc3dvcmQ9UkVEQUNURUQgICAgICAgICAgICAgICA7IChkZWZhdWx0IGlzIG5vIHBhc3N3b3JkIChvcGVuIHNlcnZlcikpCgpbc3VwZXJ2aXNvcmRdCmxvZ2ZpbGU9L21udC9tZDAvc2VydmVyL2xvZ3Mvc3VwZXJ2aXNvcmQubG9nIDsgKG1haW4gbG9nIGZpbGU7ZGVmYXVsdCAkQ1dEL3N1cGVydmlzb3JkLmxvZykKbG9nZmlsZV9tYXhieXRlcz01ME1CICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgOyAobWF4IG1haW4gbG9nZmlsZSBieXRlcyBiNCByb3RhdGlvbjtkZWZhdWx0IDUwTUIpCmxvZ2ZpbGVfYmFja3Vwcz0xMCAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIDsgKG51bSBvZiBtYWluIGxvZ2ZpbGUgcm90YXRpb24gYmFja3VwcztkZWZhdWx0IDEwKQpsb2dsZXZlbD1pbmZvICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICA7IChsb2cgbGV2ZWw7ZGVmYXVsdCBpbmZvOyBvdGhlcnM6IGRlYnVnLHdhcm4sdHJhY2UpCnBpZGZpbGU9L21udC9tZDAvc2VydmVyL2xvZ3Mvc3VwZXJ2aXNvcmQucGlkIDsgKHN1cGVydmlzb3JkIHBpZGZpbGU7ZGVmYXVsdCBzdXBlcnZpc29yZC5waWQpCm5vZGFlbW9uPWZhbHNlICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIDsgKHN0YXJ0IGluIGZvcmVncm91bmQgaWYgdHJ1ZTtkZWZhdWx0IGZhbHNlKQptaW5mZHM9MTAyNCAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICA7IChtaW4uIGF2YWlsIHN0YXJ0dXAgZmlsZSBkZXNjcmlwdG9ycztkZWZhdWx0IDEwMjQpCm1pbnByb2NzPTIwMCAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgICAgIDsgKG1pbi4gYXZhaWwgcHJvY2VzcyBkZXNjcmlwdG9ycztkZWZhdWx0IDIwMCkKCltycGNpbnRlcmZhY2U6c3VwZXJ2aXNvcl0Kc3VwZXJ2aXNvci5ycGNpbnRlcmZhY2VfZmFjdG9yeSA9IHN1cGVydmlzb3IucnBjaW50ZXJmYWNlOm1ha2VfbWFpbl9ycGNpbnRlcmZhY2UKCltzdXBlcnZpc29yY3RsXQpzZXJ2ZXJ1cmw9dW5peDovLy90bXAvc3VwZXJ2aXNvci5zb2NrIDsgdXNlIGEgdW5peDovLyBVUkwgIGZvciBhIHVuaXggc29ja2V0Cmhpc3RvcnlfZmlsZT1+Ly5zY19oaXN0b3J5ICAgICAgICAgICAgOyB1c2UgcmVhZGxpbmUgaGlzdG9yeSBpZiBhdmFpbGFibGUKCltwcm9ncmFtOmR5bm9dCmNvbW1hbmQ9L3Nydi9zZXJ2ZXIvYmluL2R5bm8gZm9yZWdyb3VuZApkaXJlY3Rvcnk9L3Nydi9zZXJ2ZXIKdXNlcj1yb290CnN0YXJ0c2Vjcz0xMAphdXRvcmVzdGFydD10cnVlCnN0b3BzaWduYWw9VEVSTQpzdG9wd2FpdHNlY3M9MzYwMApyZWRpcmVjdF9zdGRlcnI9dHJ1ZQpzdGRvdXRfbG9nZmlsZT0vbW50L21kMC9zZXJ2ZXIvbG9ncy9zdWJwcm9jZXNzLmxvZwplbnZpcm9ubWVudD1QQVRIPS91c3IvbG9jYWwvYmluOi9zYmluOi9iaW46L3Vzci9zYmluOi91c3IvYmluLEhPTUU9L3Jvb3QKCltwcm9ncmFtOmNoZWNrX25naW54XQpjb21tYW5kPS9zcnYvc2VydmVyL2Jpbi9jaGVja19uZ2lueF9rZWVwYWxpdmUucHkKZGlyZWN0b3J5PS9zcnYvc2VydmVyCnVzZXI9cm9vdApzdGFydHNlY3M9MTAKYXV0b3Jlc3RhcnQ9dHJ1ZQpzdG9wc2lnbmFsPVRFUk0Kc3RvcHdhaXRzZWNzPTM2MDAKcmVkaXJlY3Rfc3RkZXJyPXRydWUKc3Rkb3V0X2xvZ2ZpbGU9L21udC9tZDAvc2VydmVyL2xvZ3Mvc3VicHJvY2Vzcy1jaGVjay1uZ2lueC5sb2cKZW52aXJvbm1lbnQ9UEFUSD0vdXNyL2xvY2FsL2Jpbjovc2JpbjovYmluOi91c3Ivc2JpbjovdXNyL2JpbixIT01FPS9yb290CgpbcHJvZ3JhbTpjaGVja19zZXJ2ZXJdCmNvbW1hbmQ9L3Nydi9zZXJ2ZXIvYmluL2NoZWNrX3NlcnZlci5weQpkaXJlY3Rvcnk9L3Nydi9zZXJ2ZXIKdXNlcj1yb290CnN0YXJ0c2Vjcz0xMAphdXRvcmVzdGFydD10cnVlCnN0b3BzaWduYWw9VEVSTQpzdG9wd2FpdHNlY3M9MzYwMApyZWRpcmVjdF9zdGRlcnI9dHJ1ZQpzdGRvdXRfbG9nZmlsZT0vbW50L21kMC9zZXJ2ZXIvbG9ncy9zdWJwcm9jZXNzLWNoZWNrLXNlcnZlci5sb2cKZW52aXJvbm1lbnQ9UEFUSD0vdXNyL2xvY2FsL2Jpbjovc2JpbjovYmluOi91c3Ivc2JpbjovdXNyL2JpbixIT01FPS9yb290Cg==

@mnaberez

This comment has been minimized.

Copy link
Member Author

commented Apr 17, 2019

$ base64 --decode supervisord.b64 > supervisord.conf

$ openssl sha1 supervisord.conf 
SHA1(supervisord.conf)= 2976fd7b3661f0f49c636a311cd899f2ff18882b

$ python -V
Python 2.7.15

$ supervisord --version 
4.0.1

$ sudo supervisord -n -c supervisord.conf 
2019-04-17 10:23:48,381 CRIT Supervisor is running as root.  Privileges were not dropped because no user is specified in the config file.  If you intend to run as root, you can set user=root in the config file to avoid this message.
2019-04-17 10:23:48,381 INFO Increased RLIMIT_NOFILE limit to 1024
2019-04-17 10:23:48,391 INFO RPC interface 'supervisor' initialized
2019-04-17 10:23:48,391 CRIT Server 'unix_http_server' running without any HTTP authentication checking
2019-04-17 10:23:48,392 INFO supervisord started with pid 3205
2019-04-17 10:23:49,397 INFO spawnerr: can't find command '/srv/server/bin/dyno'
2019-04-17 10:23:49,397 INFO spawnerr: can't find command '/srv/server/bin/check_server.py'
2019-04-17 10:23:49,397 INFO spawnerr: can't find command '/srv/server/bin/check_nginx_keepalive.py'
2019-04-17 10:23:50,402 INFO spawnerr: can't find command '/srv/server/bin/dyno'

I believe I'm using the same Supervisor version (4.0.1), same Python version (2.7.15), and a byte-for-byte identical config file as you. I can't get the error to happen. I'd appreciate if you could please confirm all this on your setup again. After that, I'm not sure, any ideas would be appreciated.

@BillMedernach

This comment has been minimized.

Copy link

commented Apr 17, 2019

@mnaberez My apologies. I didn't have a way to get the actual host this was running on before but now I have an exact replica of it. The issue is still occurring but we're actually running Python 2.7.16.

Small difference, but perhaps the issue lies there.

I can also confirm this does not happen with v3.4.0 by the way.

@mnaberez

This comment was marked as resolved.

Copy link
Member Author

commented Apr 17, 2019

The issue is still occurring but we're actually running Python 2.7.16.

I don't have access to 2.7.16 at the moment. The changelog shows a fix for IDLE since 2.7.15 and no other changes. I checked the git history for ConfigParser and don't see any changes there either.

Could you please try progressively removing things from your config file until you arrive at the minimum required to reproduce it?

You can test the config easily in the foreground using -n (--nodaemon):

$ supervisord -n -c supervisord.conf

It should either start or display the minfds error, then you can Control-C to exit.

For example, remove the [program:check_server] section and then try to start it, then also remove [program:check_nginx], and progressively delete most of the config file. I'm hoping that you discover removing some lines eventually fixes the problem, and then we can try to puzzle out why. The only requirement for the config file is that it has a [supervisord] line. All the rest is optional.

It is useful to know that it doesn't seem to happen on 3.4.0. There were some changes to the config parsing code between the versions. However, I can't get this to happen on either version and I don't have an idea why this might be happening.

@jenyak

This comment has been minimized.

Copy link

commented Apr 17, 2019

I have same error
supervisord -n -c supervisord.conf
Error: invalid literal for long() with base 10: '1024 ; min. avail startup file descriptors; default 1024'
Python 2.7.16 supervisord 4.0.1

I remove everything, my supervisord.conf only has

[supervisord]
minfds=1024                  ; min. avail startup file descriptors; default 1024
@mnaberez

This comment has been minimized.

Copy link
Member Author

commented Apr 17, 2019

I was able to reproduce it. Here's what I tried:

  • Fresh VM running Ubuntu 18.04.2 Bionic, with Python 2.7.15rc1 installed via apt install python-minimal python-pip and Supervisor 4.0.1 installed via pip install supervisor. The error does not occur.

  • Fresh VM running Ubuntu 19.04 Disco (development branch), with Python 2.7.16 installed via apt install python-minimal python-pip and Supervisor 4.0.1 installed via pip install supervisor. The error occurs.

This is the exact config file that I used in both cases:

$ cat supervisord.conf
[supervisord]
minfds=1024			; min. avail startup file descriptors; default 1024

$ hexdump -C supervisord.conf
00000000  5b 73 75 70 65 72 76 69  73 6f 72 64 5d 0a 6d 69  |[supervisord].mi|
00000010  6e 66 64 73 3d 31 30 32  34 09 09 09 3b 20 6d 69  |nfds=1024...; mi|
00000020  6e 2e 20 61 76 61 69 6c  20 73 74 61 72 74 75 70  |n. avail startup|
00000030  20 66 69 6c 65 20 64 65  73 63 72 69 70 74 6f 72  | file descriptor|
00000040  73 3b 20 64 65 66 61 75  6c 74 20 31 30 32 34 0a  |s; default 1024.|
00000050

$ base64 supervisord.conf
W3N1cGVydmlzb3JkXQptaW5mZHM9MTAyNAkJCTsgbWluLiBhdmFpbCBzdGFydHVwIGZpbGUgZGVz
Y3JpcHRvcnM7IGRlZmF1bHQgMTAyNAo=

$ sha1sum supervisord.conf
SHA1(supervisord.conf)= 021443d5a887781b2cdc3e73f4f5bd1de0f59ce4
@mnaberez

This comment was marked as resolved.

Copy link
Member Author

commented Apr 17, 2019

@BillMedernach @jenyak What operating system and version are you running, and did you install Python via the system package manager?

@mnaberez

This comment has been minimized.

Copy link
Member Author

commented Apr 17, 2019

This error occurs when a configparser package is available on a Python 2 installation. On the fresh Ubuntu 19.04 install noted above:

Python 2.7.16 (default, Apr  6 2019, 01:42:57)
[GCC 8.3.0] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import configparser
>>> configparser.__doc__
u'Convenience module importing everything from backports.configparser.'

In Python 3, the ConfigParser module was renamed to configparser. Starting with Supervisor 4.0.0, we try to import one or the other for 2/3 compatibility. Our code is not expecting to find a configparser module on Python 2. On the Ubuntu 19.04 system that I tried, and apparently your systems, it is importing the wrong module. If you run the Supervisor test suite (python setup.py test) on such a system, there are many test failures from config parsing.

It can be fixed with this patch against master (d3f48a8):

diff --git a/supervisor/compat.py b/supervisor/compat.py
index 25b5e15..ef78033 100644
--- a/supervisor/compat.py
+++ b/supervisor/compat.py
@@ -62,9 +62,9 @@ except ImportError: # pragma: no cover
     syslog = None
 
 try: # pragma: no cover
-    import configparser as ConfigParser
+    import ConfigParser
 except ImportError: # pragma: no cover
-   import ConfigParser
+    import configparser as ConfigParser

I will test it further and then make a point release with this fix.

@mnaberez mnaberez closed this in 2c2cd35 Apr 17, 2019

@BillMedernach

This comment has been minimized.

Copy link

commented Apr 17, 2019

Fantastic. Thanks for all your help!

mnaberez added a commit that referenced this issue Apr 17, 2019

@mnaberez

This comment has been minimized.

Copy link
Member Author

commented Apr 17, 2019

Supervisor 4.0.2 (PyPI package; includes this fix)

@BillMedernach @jenyak @sdanbury Thank you!

mnaberez added a commit that referenced this issue Apr 17, 2019

@sdanbury

This comment has been minimized.

Copy link

commented May 3, 2019

Great find! Good work.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.