New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Same container, different results #476

Closed
sysmso opened this Issue Feb 1, 2017 · 18 comments

Comments

Projects
None yet
6 participants
@sysmso

sysmso commented Feb 1, 2017

Hi
I've created an singularity container for a simple python program on a Centos 7 server. When i run this container on my laptop, i have an error. On other Centos 7 server in the cluster it works well.

Laptop : Linux 4.9.6-1-ARCH #1 SMP PREEMPT Thu Jan 26 09:22:26 CET 2017 x86_64 GNU/Linux
Server : Linux 3.10.0-327.4.5.el7.x86_64 #1 SMP Mon Jan 25 22:07:14 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

On the server :
[sysmso@server singularity]$ singularity --version
2.2
[sysmso@server singularity]$ singularity exec ubuntu.img python tenso.py
(0, array([ 0.63393396], dtype=float32), array([ 0.02993923], dtype=float32))

On my laptop :
[sysmso@laptop singularity]$ singularity --version
2.2
[sysmso@laptop singularity]$ singularity exec ubuntu.img python tenso.py
Traceback (most recent call last):
File "tenso.py", line 21, in
init = tf.global_variables_initializer()
AttributeError: 'module' object has no attribute 'global_variables_initializer'

If i understand well the purpose of Singularity i should have the same result on every hosts ?
If i'm right, how to debug this ?

Thanks for your help

@yarikoptic

This comment has been minimized.

Show comment
Hide comment
@yarikoptic

yarikoptic Feb 1, 2017

Contributor

singularity

  1. mounts your home directory as well -- if you have some Python modules installed under ~/.local etc, your tenso.py might be picking those up, thus making execution dependent on the host if you have those differ among hosts.
    I guess for complete reproducibility you would need to point to some sanitized $HOME in both cases. smth like (-H available since 2.2 seems to me)
rm -fr /tmp/temp_home && mkdir -p /tmp/temp_home; singularity exec -H /tmp/temp_home dbic-bids-env.img  /bin/bash
  1. Singularity passes your environment variables inside -- so if you have those differ on two hosts and something in your script relies on them (PYTHONPATH etc) -- results could differ. I do not see any setting to avoid passing environment variables.

Would be interesting to know if not passing environment is possible and it better become possible in the long run ;)

Contributor

yarikoptic commented Feb 1, 2017

singularity

  1. mounts your home directory as well -- if you have some Python modules installed under ~/.local etc, your tenso.py might be picking those up, thus making execution dependent on the host if you have those differ among hosts.
    I guess for complete reproducibility you would need to point to some sanitized $HOME in both cases. smth like (-H available since 2.2 seems to me)
rm -fr /tmp/temp_home && mkdir -p /tmp/temp_home; singularity exec -H /tmp/temp_home dbic-bids-env.img  /bin/bash
  1. Singularity passes your environment variables inside -- so if you have those differ on two hosts and something in your script relies on them (PYTHONPATH etc) -- results could differ. I do not see any setting to avoid passing environment variables.

Would be interesting to know if not passing environment is possible and it better become possible in the long run ;)

@sysmso

This comment has been minimized.

Show comment
Hide comment
@sysmso

sysmso Feb 1, 2017

Ok it works with -H, i didn't know this option (not in the man).
Thanks for your help

sysmso commented Feb 1, 2017

Ok it works with -H, i didn't know this option (not in the man).
Thanks for your help

@vsoch

This comment has been minimized.

Show comment
Hide comment
@vsoch

vsoch Feb 1, 2017

Collaborator

I'll add this to the main help (and singularity site) - let's keep this issue open.

Collaborator

vsoch commented Feb 1, 2017

I'll add this to the main help (and singularity site) - let's keep this issue open.

@vsoch

This comment has been minimized.

Show comment
Hide comment
@vsoch

vsoch Feb 1, 2017

Collaborator

ok! I have a PR in to add the -H and --home detail to the --help output, and I've also updated our main docs to have more clear links to the Troubleshooting section, which is hidden under FAQ and was hard to find. Specifically, now the user can find troubleshooting via:

  • Quick links
  • Getting Help sidebar on main page
  • User guide under getting started

I've also added our recent discussion above here. Let us know if the issue is resolved and you are happy with the docs!

Collaborator

vsoch commented Feb 1, 2017

ok! I have a PR in to add the -H and --home detail to the --help output, and I've also updated our main docs to have more clear links to the Troubleshooting section, which is hidden under FAQ and was hard to find. Specifically, now the user can find troubleshooting via:

  • Quick links
  • Getting Help sidebar on main page
  • User guide under getting started

I've also added our recent discussion above here. Let us know if the issue is resolved and you are happy with the docs!

@sysmso

This comment has been minimized.

Show comment
Hide comment
@sysmso

sysmso Feb 2, 2017

Yes it's great thank you !

sysmso commented Feb 2, 2017

Yes it's great thank you !

@yarikoptic

This comment has been minimized.

Show comment
Hide comment
@yarikoptic

yarikoptic Feb 2, 2017

Contributor

re:

rm -rf /tmp/homie 
mkdir -p /tmp/homie
singularity exec -H /tmp/homie analysis.img /bin/bash

I would strongly recommend to either use '&&' to join all the actions, i.e.

rm -rf /tmp/homie && mkdir -p /tmp/homie && \
singularity exec -H /tmp/homie analysis.img /bin/bash

or use some directory under $HOME, e.g. $HOME/singularity_home.
Reasons are numerous, as an example I, as a malicious user, could create /tmp/homie with .bashrc containing just rm -rf $HOME/*. Your first and second command would fail since you wouldn't own that directory, but overall it would get to the 3rd line and then get to run that .bashrc.

edit 1: having such ~/singularity_home might in general be advised as to provide custom "sanitized" environment within singularity containers

Contributor

yarikoptic commented Feb 2, 2017

re:

rm -rf /tmp/homie 
mkdir -p /tmp/homie
singularity exec -H /tmp/homie analysis.img /bin/bash

I would strongly recommend to either use '&&' to join all the actions, i.e.

rm -rf /tmp/homie && mkdir -p /tmp/homie && \
singularity exec -H /tmp/homie analysis.img /bin/bash

or use some directory under $HOME, e.g. $HOME/singularity_home.
Reasons are numerous, as an example I, as a malicious user, could create /tmp/homie with .bashrc containing just rm -rf $HOME/*. Your first and second command would fail since you wouldn't own that directory, but overall it would get to the 3rd line and then get to run that .bashrc.

edit 1: having such ~/singularity_home might in general be advised as to provide custom "sanitized" environment within singularity containers

@vsoch

This comment has been minimized.

Show comment
Hide comment
@vsoch

vsoch Feb 2, 2017

Collaborator

fixed per your suggestion:

image

thanks @yarikoptic !

Collaborator

vsoch commented Feb 2, 2017

fixed per your suggestion:

image

thanks @yarikoptic !

@chrisfilo

This comment has been minimized.

Show comment
Hide comment
@chrisfilo

chrisfilo Feb 2, 2017

I think this situation speaks to my suggestion not to mount $HOME by default (see #445). A few other people also run into this (for example @russpold).

chrisfilo commented Feb 2, 2017

I think this situation speaks to my suggestion not to mount $HOME by default (see #445). A few other people also run into this (for example @russpold).

@vsoch

This comment has been minimized.

Show comment
Hide comment
@vsoch

vsoch Feb 2, 2017

Collaborator

I think Russ bot 🤖 is @poldrack on here

...too many username spaces... cannot compute... beepbeep boooooooooooop /death

Collaborator

vsoch commented Feb 2, 2017

I think Russ bot 🤖 is @poldrack on here

...too many username spaces... cannot compute... beepbeep boooooooooooop /death

@truatpasteurdotfr

This comment has been minimized.

Show comment
Hide comment
@truatpasteurdotfr

truatpasteurdotfr Feb 8, 2017

Contributor

here's my chrome launcher which create a one-time $HOME each time... off course ymmv :P
chrome-private.sh.txt

Contributor

truatpasteurdotfr commented Feb 8, 2017

here's my chrome launcher which create a one-time $HOME each time... off course ymmv :P
chrome-private.sh.txt

@jsmedmar

This comment has been minimized.

Show comment
Hide comment
@jsmedmar

jsmedmar Mar 15, 2018

Hey what if the image has the software installed in the container's /home? see https://hub.docker.com/r/ensemblorg/ensembl-vep/~/dockerfile/

jsmedmar commented Mar 15, 2018

Hey what if the image has the software installed in the container's /home? see https://hub.docker.com/r/ensemblorg/ensembl-vep/~/dockerfile/

@vsoch

This comment has been minimized.

Show comment
Hide comment
@vsoch

vsoch Mar 15, 2018

Collaborator

If there is software in the container in home it will be mounted over by the users home unless you add an argument to prevent this (specifying another home with --home) or --containall)

Collaborator

vsoch commented Mar 15, 2018

If there is software in the container in home it will be mounted over by the users home unless you add an argument to prevent this (specifying another home with --home) or --containall)

@jsmedmar

This comment has been minimized.

Show comment
Hide comment
@jsmedmar

jsmedmar Mar 15, 2018

@vsoch thank you using --home helped!

jsmedmar commented Mar 15, 2018

@vsoch thank you using --home helped!

@vsoch

This comment has been minimized.

Show comment
Hide comment
@vsoch

vsoch Mar 15, 2018

Collaborator

Glad to help! This issue is kind of like...

image

Collaborator

vsoch commented Mar 15, 2018

Glad to help! This issue is kind of like...

image

@yarikoptic

This comment has been minimized.

Show comment
Hide comment
@yarikoptic

yarikoptic Mar 15, 2018

Contributor

Given that I had similar gotchas due to /tmp being bind mounted, it seems like might be a good idea to be able to say to not bind mount anything during build. Is it possible?

Contributor

yarikoptic commented Mar 15, 2018

Given that I had similar gotchas due to /tmp being bind mounted, it seems like might be a good idea to be able to say to not bind mount anything during build. Is it possible?

@vsoch

This comment has been minimized.

Show comment
Hide comment
@vsoch

vsoch Mar 15, 2018

Collaborator

Have you tried --isolated?

Collaborator

vsoch commented Mar 15, 2018

Have you tried --isolated?

@yarikoptic

This comment has been minimized.

Show comment
Hide comment
@yarikoptic

yarikoptic Mar 15, 2018

Contributor

I guess I didn't. I will from now on ;-)

Contributor

yarikoptic commented Mar 15, 2018

I guess I didn't. I will from now on ;-)

@vsoch

This comment has been minimized.

Show comment
Hide comment
@vsoch

vsoch Mar 15, 2018

Collaborator

It's an awesome feature, @cclerget is amazing!

Collaborator

vsoch commented Mar 15, 2018

It's an awesome feature, @cclerget is amazing!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment