Supervisor won't start using %(ENV_env_var_here)s in environment

[crisbratu]

Hey folks I'm trying to run a node process via supervisord (supervisor-3.0-0.1.a8.fc13.noarch) and I've noticed some strange behaviour.

Supervisor won't start when %(ENV_VAR)s is set. The config, sensitive info redacted:

[program:node-studio] environment=PATH="/data/shared/blender:/data/shared/bin:%(ENV_PATH)s" numprocs=1 command=/usr/bin/node /data/web/studio/current/app.js directory=/data/web/studio/current

When running 'supervisorctl update':

supervisorctl update

error: <class 'xmlrpclib.Fault'>, <Fault 92: 'CANT_REREAD: Format string PATH="/data/shared/blender:/data/shared/bin:%(ENV_PATH)s" for \'environment\' contains names which cannot be expanded'>: file: /usr/lib/python2.6/site-packages/supervisor/supervisorctl.py line: 933

However, if I set the path manually based on 'echo $PATH':

environment=PATH="/data/shared/blender:/data/shared/bin:/data/rvm/gems/ruby-2.0.0-p576/bin:/data/rvm/gems/ruby-2.0.0-p576@global/bin:/data/rvm/rubies/ruby-2.0.0-p576/bin:/usr/pgsql-9.3/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/data/rvm/bin"

Supervisord starts successfully as expected. If I modify the config again with environment as:

environment=PATH="/data/shared/blender:/data/shared/bin:%(ENV_PATH)s"

The path is now correct for the process, as grabbed from /proc/$PID/environ. The correct $PATH variable is only present when the PATH= variable is set to $PATH manually, then to %(ENV_PATH)s.

[mnaberez]

error: <class 'xmlrpclib.Fault'>, <Fault 92: 'CANT_REREAD: Format string PATH="/data/shared/blender:/data/shared/bin:%(ENV_PATH)s" for \'environment\' contains names which cannot be expanded'>: file: /usr/lib/python2.6/site-packages/supervisor/supervisorctl.py line: 933

This error indicates that the %(ENV_PATH)s expansion is not supported in the environment= option for the version of Supervisor that you are using.

This expansion should be supported since 2d6ca34, which will be released in Supervisor 3.2. I added a test for this specific case in 3fe078a. Since it is passing, I am going to close this issue.

[mrcoles]

Hey @mnaberez, looking at the supervisord release log, the last release was 3.1.3 on 2014-10-28. Do you know when 3.2 might release?

Separately, do you know any workarounds in the short-term to pass non-hardcoded environment variables into the environment of a supervisord process?

EDIT: in my particular scenario I discovered that by running service supervisor start, I was not getting access to environment variables that I needed, while running supervisord -n directly would make them available to supervisor processes. So, not quite as pressing of a problem as I thought 😊

[mnaberez]

Hey @mnaberez, looking at the supervisord release log, the last release was 3.1.3 on 2014-10-28. Do you know when 3.2 might release?

Very soon, probably this week or next.

Edit: 3.2 is pending fix of #677.

[citizen-stig]

I've got the same problem. Mainly I use supervisor in amazon elastic beanstalk

[jbank]

So is 3.2 still coming out?

[mnaberez]

I actually tried to release it a few days ago, but found another significant problem just before uploading the package. I'll probably release it next week if I don't find anything else broken.

[arjenveenhuizen]

mrcoles comment on Sep 24

EDIT: in my particular scenario I discovered that by running service supervisor start, I was not getting access to environment variables that I needed, while running supervisord -n directly would make them available to supervisor processes. So, not quite as pressing of a problem as I thought

The same holds for me. This is the only way I got (custom) environment variables to work in a supervisor program (3.0b2, 3.2). Using -n, I no longer have to specify the environment= variable in the .conf file.

[mnaberez]

Supervisor 3.2.0 (November 30, 2015) supports using %(ENV_x)s expansions in environment= and there is a unit test for it specifically.

[asennoussi]

3.3.1 and same issue...

[sacheth003]

Seeing the same issue in 3.3.1

[mnaberez]

For the commenters above saying that %(ENV_x)s expansions in environment= do not work on 3.3.1, please see the following example.

Supervisor version:

$ supervisord --version
3.3.1

Minimal supervisord.conf config file:

[supervisord]
loglevel=debug 
nodaemon=true

[program:env]
environment = PETS="cats,%(ENV_PETS)s"
command = bash -c 'env | grep PETS'
startsecs = 0
autorestart = false

Create an environment variable PETS:

$ export PETS=dogs

Run supervisord in the foreground with that config:

$ supervisord -n -c supervisord.conf
2018-12-16 15:21:07,247 INFO Increased RLIMIT_NOFILE limit to 1024
2018-12-16 15:21:07,250 INFO supervisord started with pid 3748
2018-12-16 15:21:08,253 INFO spawned: 'env' with pid 3751
2018-12-16 15:21:08,269 DEBG 'env' stdout output:
PETS=cats,dogs

2018-12-16 15:21:08,269 INFO success: env entered RUNNING state, process has stayed up for > than 0 seconds (startsecs)
2018-12-16 15:21:08,269 DEBG fd 9 closed, stopped monitoring <POutputDispatcher at 4424125328 for <Subprocess at 4424125112 with name env in state RUNNING> (stderr)>
2018-12-16 15:21:08,269 DEBG fd 7 closed, stopped monitoring <POutputDispatcher at 4424125184 for <Subprocess at 4424125112 with name env in state RUNNING> (stdout)>
2018-12-16 15:21:08,269 INFO exited: env (exit status 0; expected)
2018-12-16 15:21:08,270 DEBG received SIGCHLD indicating a child quit

As you can see above, the environment when supervisord is started contains PETS=dogs. When the subprocess is run, its environment contains PETS=cats,dogs. This shows that the environment expansion (%(ENV_PETS)s) in environment= is working, which is this issue report.

Please see Subprocess Environment for more information about how Supervisor handles environment variables.

[markroxor]

@mnaberez I am getting -

Jan 20 18:10:30 raspberrypi supervisord[3183]: Error: Format string 'PETS="cats,%(ENV_PETS)s"' for 'environment' contains names ('ENV_PETS') which cannot be expanded. Available names: ENV_INVOCATION_ID, E

and yes I have the same version and have exported the variable as well.

PS - I am using supervisor as a service. Can that be an issue?

[mnaberez]

and yes I have the same version and have exported the variable as well.

PS - I am using supervisor as a service. Can that be an issue?

The environment variable must exist in the environment from which the supervisord process is started. See Subprocess Environment for more information. The error message (Available names: ...) is showing the environment variables that are present in supervisord's environment.

[cocobuster]

I am running in to the same issue:
<Fault 92: 'CANT_REREAD: Format string 'MY_VAR="%(MY_VAR)"' for 'environment' contains names ('MY_VAR') which cannot be expanded.

My supervisor version is: supervisor:all/bionic 3.3.1-1.1 uptodate

[arjunvkurup]

Hi @mnaberez,
I want to reopen this issue with Supervisor 4.1.0.
I have a configuration that was working on the earlier version with user=%(ENV_USER)s and now I am getting an error ERROR: CANT_REREAD: Format string '%(ENV_USER)s' for 'program:celery-beat.user' contains names ('ENV_USER') which cannot be expanded. Available names: ENV_INVOCATION_ID, ENV_JOURNAL_STREAM, ENV_LANG, ENV_PATH, group_name, here, host_node_name, program_name in section 'program:celery-beat' (file: '/etc/supervisor/conf.d/celery.conf').

[mnaberez]

I want to reopen this issue with Supervisor 4.1.0.
I have a configuration that was working on the earlier version with user=%(ENV_USER)s and now I am getting an error

No version of supervisord has ever created an environment variable USER. Therefore, to reproduce this, I started supervisord from an environment that had USER in it:

$ echo $USER
mnaberez

$ cat supervisord.conf 
[supervisord]
user = %(ENV_USER)s

[program:cat]
command = /bin/cat
user = %(ENV_USER)s

$ supervisord --version
4.1.0

$ supervisord -n -c supervisord.conf 
2022-03-11 07:35:59,065 INFO Set uid to user 501 succeeded
2022-03-11 07:35:59,079 INFO supervisord started with pid 25101
2022-03-11 07:36:00,084 INFO spawned: 'cat' with pid 25102
2022-03-11 07:36:01,090 INFO success: cat entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)

The above shows that on 4.1.0, user = %(ENV_USER)s works in both the [supervisord] and a [program:x]section, when USER exists in the environment from which supervisord was started.

Looking again at the error message:

Available names: ENV_INVOCATION_ID, ENV_JOURNAL_STREAM, ENV_LANG, ENV_PATH, group_name, here, host_node_name, program_name in section 'program:celery-beat' (file: '/etc/supervisor/conf.d/celery.conf').

The only variables present in the environment at the time supervisord was started were INVOCATION_ID, JOURNAL_STREAM, LANG, and PATH. Notice that USER is not in this list. Adding USER to the environment before supervisord starts will allow %(ENV_USER)s to be used.

[nagri]

Facing the same issue, on supervisor 4.1.0.
I am trying environment=SERVER_ID="%(SERVER_ID)s" which fails with
supervisor.service: Main process exited, code=exited, status=2/INVALIDARGUMENT;
but,
environment=SERVER_ID="ABCD" passes.

[mnaberez]

I am trying environment=SERVER_ID="%(SERVER_ID)s" which fails with

The syntax for an environment variable called SERVER_ID would be %(ENV_SERVER_ID)s. Similarly, an environment variable called HOME would be %(ENV_HOME)s.

supervisor/supervisor/skel/sample.conf

Lines 7 to 8 in 602d6f5

	; - Shell expansion ("~" or "$HOME") is not supported. Environment
	; variables can be expanded using this syntax: "%(ENV_HOME)s".

Note also that the variable must be present in the environment before supervisord is started.

[nagri]

I am trying environment=SERVER_ID="%(SERVER_ID)s" which fails with

The syntax for an environment variable called SERVER_ID would be %(ENV_SERVER_ID)s. Similarly, an environment variable called HOME would be %(ENV_HOME)s.

supervisor/supervisor/skel/sample.conf

Lines 7 to 8 in 602d6f5

	; - Shell expansion ("~" or "$HOME") is not supported. Environment
	; variables can be expanded using this syntax: "%(ENV_HOME)s".

Note also that the variable must be present in the environment before supervisord is started.

Tried this, still getting the same error.

[mnaberez]

Tried this, still getting the same error.

Here's what I tried to reproduce this.

$ supervisord --version
4.1.0

$ cat supervisord.conf 
[supervisord]
loglevel = debug
nodaemon = true

[program:echo]
command = /bin/bash -c 'echo $FOO'
environment = FOO=%(ENV_SERVER_ID)s
autostart = true 
startsecs = 0
autorestart = false 

$ env | grep SERVER_ID  

$ supervisord -c supervisord.conf 
Error: Format string 'FOO=%(ENV_SERVER_ID)s' for 'environment' contains names ('ENV_SERVER_ID') which cannot be expanded. Available names: ...

Above, supervisord refused to start because SERVER_ID was not present in the environment from which supervisord was started. When SERVER_ID is added to the environment:

$ export SERVER_ID=ABCD 

$ supervisord -c supervisord.conf 
2022-04-19 10:29:36,830 INFO supervisord started with pid 8446
2022-04-19 10:29:37,837 INFO spawned: 'echo' with pid 8447
2022-04-19 10:29:37,847 DEBG 'echo' stdout output:
ABCD

2022-04-19 10:29:37,847 INFO success: echo entered RUNNING state, process has stayed up for > than 0 seconds (startsecs)

Based on the above, Supervisor 4.1.0 is behaving as expected.

[nagri]

@mnaberez I was using systemd to start supervisor as a service.
When I run /usr/bin/supervisord -n -c /etc/supervisor/supervisord.conf this environment variable expansion works just fine. But when I do a systemctl restart supervisor I still get supervisor.service: Main process exited, code=exited, status=2/INVALIDARGUMENT;

journalctl -xe showed me this

supervisord[3105685]: Error: Format string 'SERVER_ID=%(ENV_SERVER_ID)s' for 'supervisord.environment' contains names ('ENV_SERVER_ID') which cannot be expanded. Available names: ENV_INVOCATION_ID, ENV_JOURNAL_STREAM, ENV_LANG, ENV_PATH

SERVER_ID is set in /etc/environment and therefore all the system users will have access SERVER_ID

Something weird is happening between systemd and supervisord.

[mnaberez]

Format string 'SERVER_ID=%(ENV_SERVER_ID)s' for 'supervisord.environment' contains names ('ENV_SERVER_ID') which cannot be expanded. Available names

supervisord will emit this error when no SERVER_ID variable is present in the environment of the shell from which it is started.

Something weird is happening between systemd and supervisord.

The Supervisor project itself does not provide any integration with systemd or other init systems. However, the information above and on this page applies to any installation, so hopefully it will be helpful. The Supervisor developers do not have enough bandwidth to assist with troubleshooting individual installations, sorry.

[nagri]

supervisord will emit this error when no SERVER_ID variable is present in the environment of the shell from which it is started.

I appreciate you answering. Thanks.

[nabhosal]

supervisord[3105685]: Error: Format string 'SERVER_ID=%(ENV_SERVER_ID)s' for 'supervisord.environment' contains names ('ENV_SERVER_ID') which cannot be expanded. Available names: ENV_INVOCATION_ID, ENV_JOURNAL_STREAM, ENV_LANG, ENV_PATH

SERVER_ID is set in /etc/environment and therefore all the system users will have access SERVER_ID

Facing the same issue, @nagri could you please share your resolution here?

[nagri]

supervisord[3105685]: Error: Format string 'SERVER_ID=%(ENV_SERVER_ID)s' for 'supervisord.environment' contains names ('ENV_SERVER_ID') which cannot be expanded. Available names: ENV_INVOCATION_ID, ENV_JOURNAL_STREAM, ENV_LANG, ENV_PATH
SERVER_ID is set in /etc/environment and therefore all the system users will have access SERVER_ID

Facing the same issue, @nagri could you please share your resolution here?

I gave up the idea of using the variable. I manipulated the conf file and feeded the value directly into the conf file like this "environment=SERVER_ID=" and then restarted supervisor.