Singularity and environment variables and 3.6.0 proposal

[cclerget]

To resume how things works actually

Singularity forward host user environment variables to container process in various inconsistent and not reproducible ways.

Variables always forwarded from host user environment to container process no matter if --cleanenv is specified or not:

• term (case insensitive)
• http_proxy (case insensitive)
• https_proxy (case insensitive)
• no_proxy (case insensitive)
• ftp_proxy (case insensitive)

Variables never forwarded from host user environment to container process:

• path (case insensitive)
• ld_library_path (case insensitive)

Default variables always forwarded but impossible to override from CLI:

• HOME by default uses home directory defined in user database or the source part from --home option
• PATH always set to "/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin:/usr/local/sbin"
• LANG set to "C" when --cleanenv is specified

Variables prefixed with SINGULARITYENV_ see their prefix trimmed and are forwarded to container process no matter if --cleanenv is specified or not:

• SINGULARITYENV_FOO become FOO in container process :
  • Important consideration: before fix https://github.com/sylabs/singularity/pull/5021 those variables may or may not take precedence over host user environment variables when --cleanenv is not set leading to inconsistencies

There is few exceptions for SINGULARITYENV_ variables :

• SINGULARITYENV_PREPEND_PATH become SING_USER_DEFINED_PREPEND_PATH
• SINGULARITYENV_APPEND_PATH become SING_USER_DEFINED_APPEND_PATH
• SINGULARITYENV_PATH become SING_USER_DEFINED_PATH

All variables forwarded to container process goes through an environment setup before executing the real container process and it's done with sourced scripts located in /.singularity.d/env within the container image:

• /.singularity.d/env/01-base.sh (nothing done in this script)
• /.singularity.d/env/10-docker2singularity.sh (contains environment variables defined in Dockerfile if any)
• /.singularity.d/env/90-environment.sh (content of %environment section if build from a definition file)
• /.singularity.d/env/91-environment.sh (optional and depend of $SINGULARITY_ENVIRONMENT use)
• /.singularity.d/env/94-appsbase.sh (contains related app environment variables)
• /.singularity.d/env/95-apps.sh (append bin/lib apps path in PATH and LD_LIBRARY_PATH)
  • /scif/apps/${SINGULARITY_APPNAME}/scif/env/01-base.sh
  • /scif/apps/${SINGULARITY_APPNAME}/scif/env/90-environment.sh (content of app %appenv section)
• /.singularity.d/env/99-base.sh (adds /singularity.d/libs to LD_LIBRARY_PATH and force PS1="Singularity> ")
• /.singularity.d/env/99-runtimevars.sh (handle special SING_USER_DEFINED_ path variables)

Important consideration: until Singularity 3.1.0 all images bootstrapped from docker translated ENV KEY val Dockerfile lines to the export form export KEY=VAL, since 3.1.0 they are translated to export KEY=${KEY:-"VAL"} to be overridable, those exported variables are located in container image at:

• /.singularity.d/env/10-docker2singularity.sh for v3
• /.singularity.d/env/10-docker.sh for v2

With all the above in mind let's take an example with an image run from docker://golang:1.13, the
the Dockerfile of this image has defined environment variables:

GOLANG_VERSION=1.14rc1
GOPATH=/go
PATH=/go/bin:/usr/local/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

When running with Singularity prior to 3.1.0:

$ GOPATH=/fake singularity exec docker://golang:1.13 go env | grep GOPATH
GOPATH="/go"
$ SINGULARITYENV_GOPATH=/fake singularity exec --cleanenv docker://golang:1.13 go env | grep GOPATH
GOPATH="/go"

Because it's always forced by /.singularity.d/env/10-docker2singularity.sh containing export GOPATH=/go

When running with Singularity >= 3.1.0:

$ GOPATH=/fake singularity exec docker://golang:1.13 go env | grep GOPATH
GOPATH="/fake"
$ SINGULARITYENV_GOPATH=/fake singularity exec --cleanenv docker://golang:1.13 go env | grep GOPATH
GOPATH="/fake"

Because /.singularity.d/env/10-docker2singularity.sh contains export GOPATH=${GOPATH:-"/go"} allowing to override GOPATH from the forwarded variable.

Now if we build the image with the following definition file:

cat > /tmp/docker.def <<EOF
bootstrap: docker
from: docker://golang:1.13

%environment
	# override docker GOPATH and set it to a writable location
	export GOPATH="/tmp/go"

EOF

The container image now contains a /.singularity.d/env/90-environment.sh with the content:

export GOPATH="/tmp/go"

Obviously with any Singularity version GOPATH is not overridable anymore

$ singularity build -f /tmp/go.sif /tmp/docker.def
$ GOPATH=/fake singularity exec /tmp/go.sif go env | grep GOPATH
GOPATH="/tmp/go"
$ SINGULARITYENV_GOPATH=/fake singularity exec --cleanenv /tmp/go.sif go env | grep GOPATH
GOPATH="/tmp/go"

Proposal for consistency and rules in 3.6.0

No changes regarding forwarding of user host environment variables

1. Only variables prefixed with SINGULARITYENV_ are passed to:

• /.singularity.d/env/10-docker2singularity.sh or /.singularity.d/env/10-docker.sh
• /.singularity.d/env/90-environment.sh
• /.singularity.d/env/91-environment.sh

2. Add --env option to pass environment variables by simply prefixing them with SINGULARITYENV_

• and allow to pass environment variable as literal like this:
  --env PYTHONPATH=/python/path:\${PYTHONPATH}

Rules:

1. Docker inherited environment variables are always overridable (stay aligned with docker behaviour)
2. Environment variables defined in %environment section are overridable only if the image author allow it:

• Example for an overridable variable:

  %environment
      # if PYTHONPATH is already set use it otherwise use
      # default value "/usr/local/python3"
      export PYTHONPATH=${PYTHONPATH:-"/usr/local/python3"}
  

• Example for not overridable variable:

  %environment
      export PYTHONPATH="/usr/local/python3"
  

[vsoch]

A few questions!

Envar Preference

• SINGULARITYENV_FOO become FOO in container process

Does this mean that a previously defined FOO is overwritten (this would be my expectation, if I explicitly set a SINGULARITYENV_* variable I'd want it to override. Also, does this mean I can unset a defined variable by leaving it empty (e.g., SINGULARITYENV_FOO= singularity run...) that seems like it would be useful too.

Environment external files

It's fairly common to want to set envars at runtime, especially with secrets and what not. What about instead of requiring a bunch of custom SINGULARITYENV_* there is some simple support for an environment file? For example, if you just bind a file to be inside the env folder, this should work as expected. This could have a nice wrapper to do this, so for the client something like:

singularity run --env-file myenv.sh container.sif

And actually you could mirror Docker and add the --env flag (this would be very intuitive)

singularity run --env MINNIE=MOUSE container.sif

Feeback

Okay taking a look at last section (there is already overlap with my suggestions above!) For this bit:

--env PYTHONPATH=/python/path:${PYTHONPATH}

We'd just want to make sure that you can also do:

--env PYTHONPATH=

To unset the variable.

Environment variables defined in %environment section are overridable only if the image author allow it:

This won't come naturally to writing recipes, so this should be all over the docs, and even current recipes/example updated to explain it.

Question : You mentioned /.singularity.d/env/10-docker2singularity.sh or /.singularity.d/env/10-docker.sh (two docker environment files) in the later section, but not the first. What is the difference? I'd expect docker2singularity to be associated with the docker2singularity tool, and docker.sh something else?

Overall, it's really great that attention is being placed on hardening this up! I am reviewing this from the standpoint of a typical user, that likely won't even delve into the tricks of using SINGULARITYENV_*, but perhaps would be comfortable with the idea of a file of environment variables (--env-file) or one off flags (--env).

[cclerget]

Does this mean that a previously defined FOO is overwritten (this would be my expectation, if I explicitly set a SINGULARITYENV_* variable I'd want it to override

It depends of the workflow in the runtime and need to be defined to establish clear rules.

Also, does this mean I can unset a defined variable by leaving it empty (e.g., SINGULARITYENV_FOO= singularity run...) that seems like it would be useful too.

Can't answer right now, also depends of the workflow

Environment external files

--env-file is a good idea, it would be possible and without involving binding with changes coming in https://github.com/sylabs/singularity/pull/5028

You mentioned /.singularity.d/env/10-docker2singularity.sh or /.singularity.d/env/10-docker.sh (two docker environment files) in the later section, but not the first. What is the difference? I'd expect docker2singularity to be associated with the docker2singularity tool, and docker.sh something else?

• /.singularity.d/env/10-docker2singularity.sh is the file containing the docker environment metadata in images built/pulled from docker/oci source since v3, I never used docker2singularity and didn't code this part but it was certainly name like this to be aligned with docker2singularity tool
• /.singularity.d/env/10-docker.sh is the file containing the docker environment metadata in images built/pulled from docker/oci source with v2 (since 2.3)

[cclerget]

@vsoch #5028 is ready for experiment, I think it addresses your comments, so your feedbacks would be much appreciated, thanks !

In its current status :

• host user environment variables couldn't override environment variables defined by the image unless variables are set with --env or --env-file or by passing with SINGULARITYENV_ prefix
• environment variables specified by --env or --env-file are automatically prefixed with SINGULARITYENV_ to distinguish them from host user environment

The action script sourcing do :

• clear the environment and just keep HOME, SINGULARITY_APPNAME while sourcing scripts in /.singularity.d/env directory by alphanum order
• source script steps:
  1. take docker defined environment variables if any
  2. • if PATH is not defined set the Singularity default PATH
     • if PATH is defined: add missing path from Singularity default PATH
  3. take environment defined in the image so they can override previous variables and/or create new
  4. setup app (--app) environment variables
  5. setup base environment (PS1 and LD_LIBRARY_PATH)
  6. inject SINGULARITYENV_ variables so they can override or modify any of previous
  7. source remaining scripts
• finally restore user environment variables which are not defined by above sourcing, also empty variables are unset

[dtrudg]

I'm reopening this as I'm referring to it in discussion in various places. We can close once we are both stable and documented.

[smarocchi]

I am really happy with the idea of putting --env-file !! Currently there is some other way to do the same trick for many variables ?

The only solutions I found so far is writing a long script full of SINGULARITYENV_* one after the other.

Thanks for the help.

[dtrudg]

Hi @alchemroz - there isn't a way to do this on the current stable release. The 3.6 release with then --env-file option is expected in May though, and you can try this out if you build Singularity from the master branch.