sudo supervisorctl returns Errno (2): unix:///tmp/supervisor.sock no such file

[pisymbol]

RHEL 7.4 x86_64
supervisor-3.3.1-1.noarch

The issue is the following:

When our web application written in php does something like this:

exec("sudo /path/to/some/script", $stdout, $stderr);

The script gets executed but the call to 'supervisorctl status ' within it fails with Errno 2 complaining the UNIX domain socket doesn't exist. However, it most certainly exists and is listening:

$ netstat -a | grep supervisor:
unix  2      [ ACC ]     STREAM     LISTENING     16823    /tmp/supervisor.sock.692
$ stat /tmp/supervisor.sock
  File: ‘/tmp/supervisor.sock’
 ..
Access: (0777/srwxrwxrwx)  Uid: (    0/    root)   Gid: (    0/    root)
..

It's 0777 because I set it to 0777. It was original 0700 but the problem persists either way.

Also note that the script in question is in the suders file, part of a cmd alias (bunch of commands grouped with the same rules):

Defaults!OURCMDS !requiretty, visiblepw

Looking at source, it seems the following lines are failing:

    517 class UnixStreamHTTPConnection(httplib.HTTPConnection):
    518     def connect(self):
    519         self.sock = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
    520         # we abuse the host parameter as the socketname
    521         self.sock.connect(self.socketfile)

Note that L520 should be removed but a minor nit. L521 though is failing and for the life of me I don't understand why. The error message itself is strange since self.socketfile I verified is indeed the string '/tmp/supervisor.sock' but the exception seems to think it's trying to open 'unix:///tmp/supervisor.sock'.

What's even more strange is if I do the following:

$ sudo -u apache bash -c 'php -a'
> exec('/bin/sudo /path/to/script/that/calls/supervisorctl', $a, $b);
> var_dump($a);

Everything looks good!

Why oh why does executing from php a script under sudo as apache cause supervisorctl not to be able to open the UNIX domain socket that is clearly sitting in /tmp and listening?

EDIT: EL6 seems to work with zero code changes. But EL7 fails. The only difference I can think of is python 2.6 vs 2.7.

[pisymbol]

#121

I changed the socket file location to /var/run and everything worked. Can someone explain to me why?

[mnaberez]

I changed the socket file location to /var/run and everything worked. Can someone explain to me why?

I don't see any code in our tree that uses /var/run so either that path is specified in one of your configuration files or the RPM version of Supervisor you installed may be patched to use it. Those packages are created by others so we usually don't know what's been changed.

I noticed above that you aren't specifying an explicit path to a config file with the -c option. Some difference in environments may be causing this, e.g. the PATH environment variable or the current working directory, as supervisorctl uses these to search when no path to the config file is specified.

This seems to be an issue related to how this particular system is configured. I don't think there is anything we can do here so I am going to close this issue for now. There may be others who can help with installation and usage questions on the supervisor-users mailing list. We use this issue tracker mainly for bugs in Supervisor itself.

[pisymbol]

I said I CHANGED the default configuration file for the sock file from /tmp/supervisor.sock to /var/run/supervisor.sock.

My initial comment was with the default CentOS config file.

There is definitely a bug here - I just don't know where.

[mnaberez]

There is definitely a bug here - I just don't know where.

It's possible but I don't have enough information to confirm that. If you can provide specific instructions to reproduce a bug, we can certainly reopen.

Since it works as you expect from some environments on your machine and not others, please look at the difference between those environments and in particular, try the -c suggestion above.

As far the CentOS package goes, nobody that works on this code is familiar with it. However, you may find other CentOS users on the mailing list.

[pisymbol]

Here is just a general question: Does the default supervisor.conf file use /tmp? That seems very dangerous.

[mnaberez]

Here is just a general question: Does the default supervisor.conf file use /tmp? That seems very dangerous.

There isn't such a default file. There's only a command, echo_supervisord_conf, that outputs sample configuration to stdout. The person installing or packaging the software is responsible for adapting it to what is appropriate for their system and writing it to a file.

For other general questions, please ask them on the mailing list. That's the usual place and there are more people there that can answer as well.

[aimensasi]

he is talking about the following section in the config file

[unix_http_server]
file=/tmp/supervisor.sock   ; the path to the socket file ******* this one
;chmod=0700                 ; socket file mode (default 0700)
;chown=nobody:nogroup       ; socket file uid:gid owner
;username=user              ; default is no username (open server)
;password=123               ; default is no password (open server)

after a while supervisor stop running and start giving an error

unix:///tmp/supervisor.sock no such file

I had the same problem multiple times on different computers, but all of them running Centos (6, 7)

is this a bug, or are we missing something in the config file.

[pisymbol]

The fix or me was to move the sock file to /var/run. But of course, that was a royal pain in the butt because there is really no way to restart supervisor when you change the sock file location gracefully.

[aimensasi]

I did change thr location of the file. And yes I am having trouble restarting supervisor.

Did you manage to find a workaround

[pisymbol]

That's unfortunately a design limitation in supervisor I believe.

Supervisor is actually an http RPC server that accepts "commands" over a socket via supervisorctl. For localhost connections, supervisor uses a Unix domain socket. But if the running server is already listening on one Unix domain socket located in /tmp and you change the configuration file to now zonk it under /var/run without restarting supervisor, the next invocation of the client will pick up /var/run and think supervisor is down because it is temporarily connecting to the wrong socket.

The only way I can see to change this gracefully is to shutdown supervisor (i.e. systemctl stop supervisord/service supervisord stop), edit /etc/supervisor.conf, and then start it again.

What we really need is a way to tell supervisor to reload its config WITHOUT using supervisorctl (so reload is not I want). Something even to the tune of kill -SIGHUP or USER1?

[aimensasi]

Currently the problem I am having is a way to setup supervisor to start after a reboot of the system.
Have you faced a problem like this before.

when the file was at /tmp, it was working fine but after I switched to /var . I can't seem to get it to work

[pisymbol]

Yes, I just enabled it via systemd (systemctl enable supervisord). What error are you actually seeing then?

If you change the location of the sock file you must change it both in the "file" line you cited above as well the "serverurl" line under [supervisorctl] section.

[aimensasi]

so if I placed the sock file in /var/run, where should the serverurl be .

currently it is pointed to serverurl=unix:///tmp/supervisor.sock

[pisymbol]

That would be your problem then. Try:

serverurl=unix:///var/run/supervisor.sock

[aimensasi]

anything else I might be missing.

The config file

; Sample supervisor config file.
;
; For more information on the config file, please see:
; http://supervisord.org/configuration.html
;
; Notes:
;  - Shell expansion ("~" or "$HOME") is not supported.  Environment
;    variables can be expanded using this syntax: "%(ENV_HOME)s".
;  - Quotes around values are not supported, except in the case of
;    the environment= options as shown below.
;  - Comments must have a leading space: "a=b ;comment" not "a=b;comment".
;  - Command will be truncated if it looks like a config file comment, e.g.
;    "command=bash -c 'foo ; bar'" will truncate to "command=bash -c 'foo ".

[unix_http_server]
file=/var/run/supervisor.sock   ; the path to the socket file
;chmod=0700                 ; socket file mode (default 0700)
;chown=nobody:nogroup       ; socket file uid:gid owner
;username=user              ; default is no username (open server)
;password=123               ; default is no password (open server)

;[inet_http_server]         ; inet (TCP) server disabled by default
;port=127.0.0.1:9001        ; ip_address:port specifier, *:port for all iface
;username=user              ; default is no username (open server)
;password=123               ; default is no password (open server)

[supervisord]
logfile=/tmp/supervisord.log ; main log file; default $CWD/supervisord.log
logfile_maxbytes=50MB        ; max main logfile bytes b4 rotation; default 50MB
logfile_backups=10           ; # of main logfile backups; 0 means none, default 10
loglevel=info                ; log level; default info; others: debug,warn,trace
pidfile=/tmp/supervisord.pid ; supervisord pidfile; default supervisord.pid
nodaemon=false               ; start in foreground if true; default false
minfds=1024                  ; min. avail startup file descriptors; default 1024
minprocs=200                 ; min. avail process descriptors;default 200
;umask=022                   ; process file creation umask; default 022
;user=chrism                 ; default is current user, required if root
;identifier=supervisor       ; supervisord identifier, default is 'supervisor'
;directory=/tmp              ; default is not to cd during start
;nocleanup=true              ; don't clean up tempfiles at start; default false
;childlogdir=/tmp            ; 'AUTO' child log dir, default $TEMP
;environment=KEY="value"     ; key value pairs to add to environment
;strip_ansi=false            ; strip ansi escape codes in logs; def. false

; The rpcinterface:supervisor section must remain in the config file for
; RPC (supervisorctl/web interface) to work.  Additional interfaces may be
; added by defining them in separate [rpcinterface:x] sections.

[rpcinterface:supervisor]
supervisor.rpcinterface_factory = supervisor.rpcinterface:make_main_rpcinterface

; The supervisorctl section configures how supervisorctl will connect to
; supervisord.  configure it match the settings in either the unix_http_server
; or inet_http_server section.

[supervisorctl]
serverurl=unix:///var/run/supervisor.sock ; use a unix:// URL  for a unix socket
;serverurl=http://127.0.0.1:9001 ; use an http:// url to specify an inet socket
;username=chris              ; should be same as in [*_http_server] if set
;password=123                ; should be same as in [*_http_server] if set
;prompt=mysupervisor         ; cmd line prompt (default "supervisor")
;history_file=~/.sc_history  ; use readline history if available

; The sample program section below shows all possible program subsection values.
; Create one or more 'real' program: sections to be able to control them under
; supervisor.

;[program:theprogramname]
;command=/bin/cat              ; the program (relative uses PATH, can take args)
;process_name=%(program_name)s ; process_name expr (default %(program_name)s)
;numprocs=1                    ; number of processes copies to start (def 1)
;directory=/tmp                ; directory to cwd to before exec (def no cwd)
;umask=022                     ; umask for process (default None)
;priority=999                  ; the relative start priority (default 999)
;autostart=true                ; start at supervisord start (default: true)
;startsecs=1                   ; # of secs prog must stay up to be running (def. 1)
;startretries=3                ; max # of serial start failures when starting (default 3)
;autorestart=unexpected        ; when to restart if exited after running (def: unexpected)
;exitcodes=0,2                 ; 'expected' exit codes used with autorestart (default 0,2)
;stopsignal=QUIT               ; signal used to kill process (default TERM)
;stopwaitsecs=10               ; max num secs to wait b4 SIGKILL (default 10)
;stopasgroup=false             ; send stop signal to the UNIX process group (default false)
;killasgroup=false             ; SIGKILL the UNIX process group (def false)
;user=chrism                   ; setuid to this UNIX account to run the program
;redirect_stderr=true          ; redirect proc stderr to stdout (default false)
;stdout_logfile=/a/path        ; stdout log path, NONE for none; default AUTO
;stdout_logfile_maxbytes=1MB   ; max # logfile bytes b4 rotation (default 50MB)
;stdout_logfile_backups=10     ; # of stdout logfile backups (0 means none, default 10)
;stdout_capture_maxbytes=1MB   ; number of bytes in 'capturemode' (default 0)
;stdout_events_enabled=false   ; emit events on stdout writes (default false)
;stderr_logfile=/a/path        ; stderr log path, NONE for none; default AUTO
;stderr_logfile_maxbytes=1MB   ; max # logfile bytes b4 rotation (default 50MB)
;stderr_logfile_backups=10     ; # of stderr logfile backups (0 means none, default 10)
;stderr_capture_maxbytes=1MB   ; number of bytes in 'capturemode' (default 0)
;stderr_events_enabled=false   ; emit events on stderr writes (default false)
;environment=A="1",B="2"       ; process environment additions (def no adds)
;serverurl=AUTO                ; override serverurl computation (childutils)

; The sample eventlistener section below shows all possible eventlistener
; subsection values.  Create one or more 'real' eventlistener: sections to be
; able to handle event notifications sent by supervisord.

;[eventlistener:theeventlistenername]
;command=/bin/eventlistener    ; the program (relative uses PATH, can take args)
;process_name=%(program_name)s ; process_name expr (default %(program_name)s)
;numprocs=1                    ; number of processes copies to start (def 1)
;events=EVENT                  ; event notif. types to subscribe to (req'd)
;buffer_size=10                ; event buffer queue size (default 10)
;directory=/tmp                ; directory to cwd to before exec (def no cwd)
;umask=022                     ; umask for process (default None)
;priority=-1                   ; the relative start priority (default -1)
;autostart=true                ; start at supervisord start (default: true)
;startsecs=1                   ; # of secs prog must stay up to be running (def. 1)
;startretries=3                ; max # of serial start failures when starting (default 3)
;autorestart=unexpected        ; autorestart if exited after running (def: unexpected)
;exitcodes=0,2                 ; 'expected' exit codes used with autorestart (default 0,2)
;stopsignal=QUIT               ; signal used to kill process (default TERM)
;stopwaitsecs=10               ; max num secs to wait b4 SIGKILL (default 10)
;stopasgroup=false             ; send stop signal to the UNIX process group (default false)
;killasgroup=false             ; SIGKILL the UNIX process group (def false)
;user=chrism                   ; setuid to this UNIX account to run the program
;redirect_stderr=false         ; redirect_stderr=true is not allowed for eventlisteners
;stdout_logfile=/a/path        ; stdout log path, NONE for none; default AUTO
;stdout_logfile_maxbytes=1MB   ; max # logfile bytes b4 rotation (default 50MB)
;stdout_logfile_backups=10     ; # of stdout logfile backups (0 means none, default 10)
;stdout_events_enabled=false   ; emit events on stdout writes (default false)
;stderr_logfile=/a/path        ; stderr log path, NONE for none; default AUTO
;stderr_logfile_maxbytes=1MB   ; max # logfile bytes b4 rotation (default 50MB)
;stderr_logfile_backups=10     ; # of stderr logfile backups (0 means none, default 10)
;stderr_events_enabled=false   ; emit events on stderr writes (default false)
;environment=A="1",B="2"       ; process environment additions
;serverurl=AUTO                ; override serverurl computation (childutils)

; The sample group section below shows all possible group values.  Create one
; or more 'real' group: sections to create "heterogeneous" process groups.

;[group:thegroupname]
;programs=progname1,progname2  ; each refers to 'x' in [program:x] definitions
;priority=999                  ; the relative start priority (default 999)

; The [include] section can just contain the "files" setting.  This
; setting can list multiple files (separated by whitespace or
; newlines).  It can also contain wildcards.  The filenames are
; interpreted as relative to this file.  Included files *cannot*
; include files themselves.

;[include]
;files = relative/directory/*.ini

[aimensasi]

and thanks a lot for the help, I really appreciate it

[pisymbol]

What is github for? :-)

I would put the pid file in /var/run as well and the logfile in /var/log but that's just me! Anyway, it should come up now without issue. Let me know.

[aimensasi]

you are awesome, you know that

[pisymbol]

Again, any time. Happy Supervisoring!

[guruvishnuvardan]

Folks,

Can you please help me, I have the following issue(unix:///var/run/supervisor.sock refused connection) and I see the supervisor.sock is empty

Please find the supervisord.conf @ /etc/

; Sample supervisor config file.
;
; For more information on the config file, please see:
; http://supervisord.org/configuration.html
;
; Notes:
; - Shell expansion ("~" or "$HOME") is not supported. Environment
; variables can be expanded using this syntax: "%(ENV_HOME)s".
; - Comments must have a leading space: "a=b ;comment" not "a=b;comment".

[unix_http_server]
file=/var/run/supervisor.sock ; (the path to the socket file)
;chmod=0700 ; socket file mode (default 0700)
;chown=nobody:nogroup ; socket file uid:gid owner
;username=user ; (default is no username (open server))
;password=123 ; (default is no password (open server))

;[inet_http_server] ; inet (TCP) server disabled by default
;port=127.0.0.1:9001 ; (ip_address:port specifier, *:port for all iface)
;username=user ; (default is no username (open server))
;password=123 ; (default is no password (open server))

[supervisord]
logfile=/tmp/supervisord.log ; (main log file;default $CWD/supervisord.log)
logfile_maxbytes=50MB ; (max main logfile bytes b4 rotation;default 50MB)
logfile_backups=10 ; (num of main logfile rotation backups;default 10)
loglevel=info ; (log level;default info; others: debug,warn,trace)
pidfile=/tmp/supervisord.pid ; (supervisord pidfile;default supervisord.pid)
nodaemon=false ; (start in foreground if true;default false)
minfds=1024 ; (min. avail startup file descriptors;default 1024)
minprocs=200 ; (min. avail process descriptors;default 200)
;umask=022 ; (process file creation umask;default 022)
;user=chrism ; (default is current user, required if root)
;identifier=supervisor ; (supervisord identifier, default is 'supervisor')
;directory=/tmp ; (default is not to cd during start)
;nocleanup=true ; (don't clean up tempfiles at start;default false)
;childlogdir=/tmp ; ('AUTO' child log dir, default $TEMP)
;environment=KEY="value" ; (key value pairs to add to environment)
;strip_ansi=false ; (strip ansi escape codes in logs; def. false)

; the below section must remain in the config file for RPC
; (supervisorctl/web interface) to work, additional interfaces may be
; added by defining them in separate rpcinterface: sections
[rpcinterface:supervisor]
supervisor.rpcinterface_factory = supervisor.rpcinterface:make_main_rpcinterface

[supervisorctl]
serverurl=unix:///var/run/supervisor.sock ; use a unix:// URL for a unix socket
;serverurl=http://127.0.0.1:9001 ; use an http:// url to specify an inet socket
;username=chris ; should be same as http_username if set
;password=123 ; should be same as http_password if set
;prompt=mysupervisor ; cmd line prompt (default "supervisor")
;history_file=~/.sc_history ; use readline history if available

; The below sample program section shows all possible program subsection values,
; create one or more 'real' program: sections to be able to control them under
; supervisor.

;[program:theprogramname]
;command=/bin/cat ; the program (relative uses PATH, can take args)
;process_name=%(program_name)s ; process_name expr (default %(program_name)s)
;numprocs=1 ; number of processes copies to start (def 1)
;directory=/tmp ; directory to cwd to before exec (def no cwd)
;umask=022 ; umask for process (default None)
;priority=999 ; the relative start priority (default 999)
;autostart=true ; start at supervisord start (default: true)
;startsecs=1 ; # of secs prog must stay up to be running (def. 1)
;startretries=3 ; max # of serial start failures when starting (default 3)
;autorestart=unexpected ; when to restart if exited after running (def: unexpected)
;exitcodes=0,2 ; 'expected' exit codes used with autorestart (default 0,2)
;stopsignal=QUIT ; signal used to kill process (default TERM)
;stopwaitsecs=10 ; max num secs to wait b4 SIGKILL (default 10)
;stopasgroup=false ; send stop signal to the UNIX process group (default false)
;killasgroup=false ; SIGKILL the UNIX process group (def false)
;user=chrism ; setuid to this UNIX account to run the program
;redirect_stderr=true ; redirect proc stderr to stdout (default false)
;stdout_logfile=/a/path ; stdout log path, NONE for none; default AUTO
;stdout_logfile_maxbytes=1MB ; max # logfile bytes b4 rotation (default 50MB)
;stdout_logfile_backups=10 ; # of stdout logfile backups (default 10)
;stdout_capture_maxbytes=1MB ; number of bytes in 'capturemode' (default 0)
;stdout_events_enabled=false ; emit events on stdout writes (default false)
;stderr_logfile=/a/path ; stderr log path, NONE for none; default AUTO
;stderr_logfile_maxbytes=1MB ; max # logfile bytes b4 rotation (default 50MB)
;stderr_logfile_backups=10 ; # of stderr logfile backups (default 10)
;stderr_capture_maxbytes=1MB ; number of bytes in 'capturemode' (default 0)
;stderr_events_enabled=false ; emit events on stderr writes (default false)
;environment=A="1",B="2" ; process environment additions (def no adds)
;serverurl=AUTO ; override serverurl computation (childutils)

; The below sample eventlistener section shows all possible
; eventlistener subsection values, create one or more 'real'
; eventlistener: sections to be able to handle event notifications
; sent by supervisor.

;[eventlistener:theeventlistenername]
;command=/bin/eventlistener ; the program (relative uses PATH, can take args)
;process_name=%(program_name)s ; process_name expr (default %(program_name)s)
;numprocs=1 ; number of processes copies to start (def 1)
;events=EVENT ; event notif. types to subscribe to (req'd)
;buffer_size=10 ; event buffer queue size (default 10)
;directory=/tmp ; directory to cwd to before exec (def no cwd)
;umask=022 ; umask for process (default None)
;priority=-1 ; the relative start priority (default -1)
;autostart=true ; start at supervisord start (default: true)
;startsecs=1 ; # of secs prog must stay up to be running (def. 1)
;startretries=3 ; max # of serial start failures when starting (default 3)
;autorestart=unexpected ; autorestart if exited after running (def: unexpected)
;exitcodes=0,2 ; 'expected' exit codes used with autorestart (default 0,2)
;stopsignal=QUIT ; signal used to kill process (default TERM)
;stopwaitsecs=10 ; max num secs to wait b4 SIGKILL (default 10)
;stopasgroup=false ; send stop signal to the UNIX process group (default false)
;killasgroup=false ; SIGKILL the UNIX process group (def false)
;user=chrism ; setuid to this UNIX account to run the program
;redirect_stderr=false ; redirect_stderr=true is not allowed for eventlisteners
;stdout_logfile=/a/path ; stdout log path, NONE for none; default AUTO
;stdout_logfile_maxbytes=1MB ; max # logfile bytes b4 rotation (default 50MB)
;stdout_logfile_backups=10 ; # of stdout logfile backups (default 10)
;stdout_events_enabled=false ; emit events on stdout writes (default false)
;stderr_logfile=/a/path ; stderr log path, NONE for none; default AUTO
;stderr_logfile_maxbytes=1MB ; max # logfile bytes b4 rotation (default 50MB)
;stderr_logfile_backups=10 ; # of stderr logfile backups (default 10)
;stderr_events_enabled=false ; emit events on stderr writes (default false)
;environment=A="1",B="2" ; process environment additions
;serverurl=AUTO ; override serverurl computation (childutils)

; The below sample group section shows all possible group values,
; create one or more 'real' group: sections to create "heterogeneous"
; process groups.

;[group:thegroupname]
;programs=progname1,progname2 ; each refers to 'x' in [program:x] definitions
;priority=999 ; the relative start priority (default 999)

; The [include] section can just contain the "files" setting. This
; setting can list multiple files (separated by whitespace or
; newlines). It can also contain wildcards. The filenames are
; interpreted as relative to this file. Included files cannot
; include files themselves.

;[include]
;files = relative/directory/*.ini

and I have also replaced the supervisord.conf @ /etc/supervisor to the above format

[khacnha]

Error the same: unix:///tmp/supervisor.sock no such file

i have created config file with command: echo_supervisord_conf > /etc/supervisord.conf

This error in Amazon Linux 2 AMI server.

Any help? thanks.