Environment variables not available from inside the definition file

[fmorency]

Version of Singularity:

2.3

Expected behavior

$ export BLA=/path/to/file
$ singularity bootstrap container.img container.def
...
I expect to be able to access $BLA from inside the various sections (%setup, %post, etc) of the definition file. This was working properly in 2.2.1

Actual behavior

The environment variable is not accessible from inside the definition file. I tried using SINGULARITY_BLA and setting the variable in %environment without success

Steps to reproduce behavior

Export an environment variable on your shell and try using it from inside the definition file

[gmkurtzer]

Hi @fmorency,

Where are you exporting the variables within the container? Can you post an example deffile?

Thanks!

[fmorency]

Hi @gmkurtzer

Simply do

$ sudo su
# export BLA=TESTING
# export BLA2=TESTING2
# singularity create --size 1000 test.img
# singularity bootstrap test.img test.def

with the following test.def file

BootStrap: debootstrap
OSVersion: trusty
MirrorURL: http://ca.archive.ubuntu.com/ubuntu/

%setup
    touch $SINGULARITY_ROOTFS/$BLA

%post
    touch $BLA2

Note that neither files are created. The log shows

I: Base system installed successfully.
+ touch /var/lib/singularity/mnt/container/
Running post scriptlet
+ touch
touch: missing file operand
Try 'touch --help' for more information.

[gmkurtzer]

Ahhh, yes, that actually is a "feature" not a bug. lol

This is because the intention of the bootstrap command is to try and be as reproducible as possible which means it should not depend on any variables from the "outside world". Singularity thus purposefully cleans all variables from the environment when bootstrapping.

[vsoch]

ah ok. So @fmorency I think wyou have a few options:

1. You can have the variables set in %post (or %setup, wherever you need them). If it's the case that the user of your script needs to set them, then provide this instruction to do so in the docs. Anyone that finds a bootstrap file lying around will see the variables written, and it won't be "lost information."
2. If it's something that can be done during runtime, you could have the user define it in the way we mentioned (eg, export SINGULARITY_VAR) and then the container will find it. I don't know your use case, but this makes sense for (most) applications.
3. If it's something for the install that you want to also keep with the container, then export it in %post/%setup (where needed) and also add it to the %environment section.

[fmorency]

You can close this issue :)

[vsoch]

werd!

[HonzaSi]

Can you confirm that the variables have to be set in each block separately? E.g. in %post and %environment. I'd like to be able to set them in one block, should it be %setup or %env, and then use them in others.

[vsoch]

hey @HonzaSi ! The different blocks serve different purposes, so it would probably be too general to take a "one block for them all" approach.

• %setup": is where you might do actions for your container communicating with the host. Your host environment variables will be available here, as would any other variables you define in the section itself. The variables won't be passed beyond that.
• %environment: This will be variables that are used by your container at runtime, not during the building (bootstrap) process. So if you need the same variables available at both times, you need to set them here and in...
• %post: is the section that builds your container, and any environment variables you define in it are available. Variables in %environment are not, nor are %setup.

@GodloveD added an environment variable that you can echo variables to export (that are defined at runtime) and I think this might be in development branch? He can chime in.

[HonzaSi]

Hey @vsoch, thanks for prompt response. Could you please explain the concept of "environment variable that you can echo variables to export" in more detail?

[GodloveD]

Hi Honza, Sorry for the delay. In the development branch it is now possible to add env vars to the runtime within the %post section using the following syntax: %post echo "export FOO=bar" >> $SINGULARITY_ENVIRONMENT This takes the place of the old deprecated syntax: %post echo "export FOO=bar" >> /environment Note that variables added this way take precedence over variables added with the %environment section. For example, let's say you had these lines in your definition file. %environment export ANAKIN=jedi %post echo "export ANAKIN=sith" >> $SINGULARITY_ENVIRONMENT In the container ANAKIN=sith. The actual order of the commands within the def file does not matter. $SINGULARITY_ENVIRONMENT will take precedence over other variables. Does that make sense. Not sure if it helps you with your original question though of making the environment available across sections.

…

On Thu, Aug 17, 2017 at 9:41 AM, HonzaSi ***@***.***> wrote: Hey @vsoch <https://github.com/vsoch>, thanks for prompt response. Could you please explain the concept of "environment variable that you can echo variables to export" in more detail? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#732 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AHUUXOIMBQf1WXizYBqg3M_cMrYacxioks5sZEL5gaJpZM4NzDkV> .

[vsoch]

+1! Here is the original discussion, if needed #800

[HonzaSi]

As far as I understand correctly, instead of doing something like:

% environment
export WORKDIR="/workdir"

% post
export WORKDIR="/workdir"
mkdir -p $WORKDIR

I can do the following:

% environment

% post
export WORKDIR="/workdir"
echo "export WORKDIR=$WORKDIR" >> $SINGULARITY_ENVIRONMENT
mkdir -p $WORKDIR

Is it correct? If so, then yes, this helps with my original concern about specifying one value at multiple places which I found error prone.

[GodloveD]

Yes that should work. Under the hood $SINGULARITY_ENVIRONMENT points to /.singularity.d/env/95-environment.sh (if I remember correctly).

…

On Thu, Aug 17, 2017 at 11:36 AM, HonzaSi ***@***.***> wrote: As far as I understand correctly, instead of doing something like: % environment export WORKDIR="/workdir" % post export WORKDIR="/workdir" mkdir -p $WORKDIR I can do the following: % environment % post export WORKDIR="/workdir" echo "export WORKDIR=$WORKDIR" >> $SINGULARITY_ENVIRONMENT mkdir -p $WORKDIR Is it correct? If so, then yes, this helps with my original concern about specifying one value at multiple places which I found error prone. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#732 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AHUUXIqnDUocPdEsFXH6jope9L2B7s01ks5sZF3xgaJpZM4NzDkV> .

[HonzaSi]

Thanks

[NinjMenon]

Sorry to dig up an old issue, but I can't seem to figure out why environment variables defined in the %environment section are available only when I shell into the container and are not recognized by/available with exec using Singularity version 2.6.1.

For example, with a definition file excerpt as such:

%environment
      FSLDIR=/usr/local/fsl
      export FSLDIR

I get the following output when I shell into the container:

Singularity fsl414.simg:~> echo $FSLDIR
/usr/local/fsl

But with exec I get the value of the variable that is set on the host (something I really don't want):

singularity exec --cleanenv fsl414.simg echo $FSLDIR
/data/mcr/fsl5

This is similar and potentially related to #734 and #1053 but these were for older versions of singularity and I'm not sure if this should be an issue with 2.6.1.
definition file just pulls a base Ubuntu image from Docker and installs vim in the %post section, so there really isn't much going on there.
Is there something that I need to do to prevent the variable from being evaluated on the host?