New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

singularity shell segfault with docker://centos:6 #845

Closed
jbd opened this Issue Jul 31, 2017 · 11 comments

Comments

Projects
None yet
4 participants
@jbd

jbd commented Jul 31, 2017

Hi,

I'm running Singularity 2.3.1-development.ga78190c8 on a Debian testing using kernel 4.11.0-1-amd64.

$ singularity --version
2.3.1-development.ga78190c8
$ singularity selftest
 + sh -c test -f /usr/local/etc/singularity/singularity.conf                           (retval=0) OK
 + test -u /usr/local/libexec/singularity/bin/action-suid                              (retval=0) OK
 + test -u /usr/local/libexec/singularity/bin/create-suid                              (retval=0) OK
 + test -u /usr/local/libexec/singularity/bin/expand-suid                              (retval=0) OK
 + test -u /usr/local/libexec/singularity/bin/export-suid                              (retval=0) OK
 + test -u /usr/local/libexec/singularity/bin/import-suid                              (retval=0) OK
 + test -u /usr/local/libexec/singularity/bin/mount-suid                               (retval=0) OK

I've got a segfault with a centos6 docker image:

$ singularity shell docker://centos:6 
Docker image path: index.docker.io/library/centos:6
Cache folder set to /home/jbdenis/.singularity/docker
Creating container runtime...
Singularity: Invoking an interactive shell within container...

Segmentation fault

I've got no problem with a centos7 docker image:

$ singularity --version
2.3.1-development.ga78190c8
$ singularity shell docker://centos:7
Docker image path: index.docker.io/library/centos:7
Cache folder set to /home/jbdenis/.singularity/docker
Creating container runtime...
Singularity: Invoking an interactive shell within container...

Singularity centos:7:~> 

Here is the tail of the debug output with the centos6 container:

[...]
DEBUG   [U=21489,P=13302]  singularity_registry_get()                Returning NULL on 'TARGET_PWD'
DEBUG   [U=21489,P=13302]  main()                                    Attempting to chdir to CWD: /home/jbdenis
DEBUG   [U=21489,P=13302]  envar_set()                               Setting environment variable: 'SINGULARITY_CONTAINER' = 'centos:6'
DEBUG   [U=21489,P=13302]  envar_set()                               Setting environment variable: 'SINGULARITY_NAME' = 'centos:6'
DEBUG   [U=21489,P=13302]  singularity_registry_get()                Returning NULL on 'SHELL'
DEBUG   [U=21489,P=13302]  envar_set()                               Unsetting environment variable: SINGULARITY_SHELL
DEBUG   [U=21489,P=13302]  singularity_registry_get()                Returning value from registry: 'COMMAND' = 'shell'
DEBUG   [U=21489,P=13302]  singularity_registry_get()                Returning value from registry: 'COMMAND' = 'shell'
LOG     [U=21489,P=13302]  main()                                    USER=jbdenis, IMAGE='centos:6', COMMAND='shell'
DEBUG   [U=21489,P=13302]  action_shell()                            Exec'ing /.singularity.d/actions/shell
VERBOSE [U=21489,P=24203]  main()                                    Cleaning directory: /tmp/.singularity-runtime.1U1fKtq0
DEBUG   [U=21489,P=24203]  s_rmdir()                                 Removing directory: /tmp/.singularity-runtime.1U1fKtq0
Segmentation fault (core dumped)

(full output https://gist.github.com/jbd/6b6983514e6c430d2c8db147af025f66)

@jbd

This comment has been minimized.

Show comment
Hide comment
@jbd

jbd Jul 31, 2017

Seems to be related to docker/for-linux#58, see this comment in particular.

jbd commented Jul 31, 2017

Seems to be related to docker/for-linux#58, see this comment in particular.

@jbd

This comment has been minimized.

Show comment
Hide comment
@jbd

jbd Jul 31, 2017

I confirm passing vsyscall=emulate to the kernel resolves it for me.

In /etc/default/grub:

GRUB_CMDLINE_LINUX_DEFAULT="vsyscall=emulate"
# update-grub && reboot

I let you close the issue if you want to add something.

jbd commented Jul 31, 2017

I confirm passing vsyscall=emulate to the kernel resolves it for me.

In /etc/default/grub:

GRUB_CMDLINE_LINUX_DEFAULT="vsyscall=emulate"
# update-grub && reboot

I let you close the issue if you want to add something.

@gmkurtzer

This comment has been minimized.

Show comment
Hide comment
@gmkurtzer

gmkurtzer Aug 1, 2017

Member

Ahh, great catch! We should update our docs. @vsoch, do you want to add a troubleshooting entry for this or shall I?

Thanks!

Member

gmkurtzer commented Aug 1, 2017

Ahh, great catch! We should update our docs. @vsoch, do you want to add a troubleshooting entry for this or shall I?

Thanks!

@vsoch

This comment has been minimized.

Show comment
Hide comment
@vsoch

vsoch Aug 1, 2017

Member

@gmkurtzer I'll get us started here, and if you want to edit / adjust I can add to our troubleshooting docs. Here is a first go not following this issue closely:

Segfault on Bootstrap of Centos Image

If you are bootstrapping a centos 6 docker image, you might hit a segfault:

$ singularity shell docker://centos:6 
Docker image path: index.docker.io/library/centos:6
Cache folder set to /home/jbdenis/.singularity/docker
Creating container runtime...
Singularity: Invoking an interactive shell within container...

Segmentation fault

@gmkurtzer add something here for why this is happening?

The fix is to pass the variable vsyscall=emulate to the kernel, meaning in the file /etc/default/grub, add the following

GRUB_CMDLINE_LINUX_DEFAULT="vsyscall=emulate"

and then update grub and reboot:

update-grub && reboot

@gmkurtzer and @jbd were these commands on the host, or somehow executed to the image? Thanks for the help!

Member

vsoch commented Aug 1, 2017

@gmkurtzer I'll get us started here, and if you want to edit / adjust I can add to our troubleshooting docs. Here is a first go not following this issue closely:

Segfault on Bootstrap of Centos Image

If you are bootstrapping a centos 6 docker image, you might hit a segfault:

$ singularity shell docker://centos:6 
Docker image path: index.docker.io/library/centos:6
Cache folder set to /home/jbdenis/.singularity/docker
Creating container runtime...
Singularity: Invoking an interactive shell within container...

Segmentation fault

@gmkurtzer add something here for why this is happening?

The fix is to pass the variable vsyscall=emulate to the kernel, meaning in the file /etc/default/grub, add the following

GRUB_CMDLINE_LINUX_DEFAULT="vsyscall=emulate"

and then update grub and reboot:

update-grub && reboot

@gmkurtzer and @jbd were these commands on the host, or somehow executed to the image? Thanks for the help!

@jbd

This comment has been minimized.

Show comment
Hide comment
@jbd

jbd Aug 1, 2017

These commands were on the host. The /etc/default/grub is debian specific, you might want to mention that.

It would be interesting to know the vsyscall status for the main GNU/Linux distributions.

On the top of my head, I would suggest:

$ grep VSYSCALL /boot/config*

or

$  zgrep VSYSCALL /proc/config.gz # if it exists of course

From the debian kernel changelog:

linux (4.10~rc6-1~exp1) experimental; urgency=medium
[amd64] Enable LEGACY_VSYSCALL_NONE instead of LEGACY_VSYSCALL_EMULATE
    (Closes: #852620).  This breaks (e)glibc versions < 2.14 and dietlibc
    versions < 0.33.  It can be reverted using the kernel parameter:
    vsyscall=emulate

jbd commented Aug 1, 2017

These commands were on the host. The /etc/default/grub is debian specific, you might want to mention that.

It would be interesting to know the vsyscall status for the main GNU/Linux distributions.

On the top of my head, I would suggest:

$ grep VSYSCALL /boot/config*

or

$  zgrep VSYSCALL /proc/config.gz # if it exists of course

From the debian kernel changelog:

linux (4.10~rc6-1~exp1) experimental; urgency=medium
[amd64] Enable LEGACY_VSYSCALL_NONE instead of LEGACY_VSYSCALL_EMULATE
    (Closes: #852620).  This breaks (e)glibc versions < 2.14 and dietlibc
    versions < 0.33.  It can be reverted using the kernel parameter:
    vsyscall=emulate

vsoch added a commit to singularityware/singularityware.github.io that referenced this issue Aug 2, 2017

@vsoch

This comment has been minimized.

Show comment
Hide comment
@vsoch
Member

vsoch commented Aug 2, 2017

@jbd

This comment has been minimized.

Show comment
Hide comment
@jbd

jbd Aug 2, 2017

Seems OK to me, thank you for the update !

Also, from the kernel-parameters.txt of linux kernel source tree:

vsyscall=	[X86-64]
			Controls the behavior of vsyscalls (i.e. calls to
			fixed addresses of 0xffffffffff600x00 from legacy
			code).  Most statically-linked binaries and older
			versions of glibc use these calls.  Because these
			functions are at fixed addresses, they make nice
			targets for exploits that can control RIP.

			emulate     [default] Vsyscalls turn into traps and are
			            emulated reasonably safely.

			native      Vsyscalls are native syscall instructions.
			            This is a little bit faster than trapping
			            and makes a few dynamic recompilers work
			            better than they would in emulation mode.
			            It also makes exploits much easier to write.

			none        Vsyscalls don't work at all.  This makes
			            them quite hard to use for exploits but
			            might break your system.

The change to "emulate" has security implications. Given that emulate is the default mode of the upstream. I guess it is OK to change it. I think Singularity users should be clearly aware of that. Maybe it should appears somewhere in the Security section of the documentation or this linked issue is enough for the moment ? Your call ! =)

jbd commented Aug 2, 2017

Seems OK to me, thank you for the update !

Also, from the kernel-parameters.txt of linux kernel source tree:

vsyscall=	[X86-64]
			Controls the behavior of vsyscalls (i.e. calls to
			fixed addresses of 0xffffffffff600x00 from legacy
			code).  Most statically-linked binaries and older
			versions of glibc use these calls.  Because these
			functions are at fixed addresses, they make nice
			targets for exploits that can control RIP.

			emulate     [default] Vsyscalls turn into traps and are
			            emulated reasonably safely.

			native      Vsyscalls are native syscall instructions.
			            This is a little bit faster than trapping
			            and makes a few dynamic recompilers work
			            better than they would in emulation mode.
			            It also makes exploits much easier to write.

			none        Vsyscalls don't work at all.  This makes
			            them quite hard to use for exploits but
			            might break your system.

The change to "emulate" has security implications. Given that emulate is the default mode of the upstream. I guess it is OK to change it. I think Singularity users should be clearly aware of that. Maybe it should appears somewhere in the Security section of the documentation or this linked issue is enough for the moment ? Your call ! =)

@vsoch

This comment has been minimized.

Show comment
Hide comment
@vsoch

vsoch Aug 2, 2017

Member

okay, just added a note about it to the same troubleshooting page. Thanks!

Member

vsoch commented Aug 2, 2017

okay, just added a note about it to the same troubleshooting page. Thanks!

@jbd

This comment has been minimized.

Show comment
Hide comment
@jbd

jbd Aug 3, 2017

I think this should not appear on the troubleshooting section:

@gmkurtzer I’ll get us started here, and if you want to edit / adjust I can add to our troubleshooting docs. Here is a first go not following this issue closely:

jbd commented Aug 3, 2017

I think this should not appear on the troubleshooting section:

@gmkurtzer I’ll get us started here, and if you want to edit / adjust I can add to our troubleshooting docs. Here is a first go not following this issue closely:
@vsoch

This comment has been minimized.

Show comment
Hide comment
@vsoch

vsoch Aug 3, 2017

Member

I have no idea what you are talking about! 😉

image

Member

vsoch commented Aug 3, 2017

I have no idea what you are talking about! 😉

image

vsoch added a commit to singularityware/singularityware.github.io that referenced this issue Aug 22, 2017

Sanity check for merging updated master into docs/2.4 (#102)
* adding troubleshooting for debian issue singularityware/singularity#845

* adding note about security implications

* Typo: dummpy.img -> dummy.img (#94)

Not sure if this was a typo or a snarky insider joke by a Perl/Ruby programmer....

* Typo: One -> On & wordsmithing (#95)

* Type: One -> On

* Wordsmithing

* Fix inaccurate docker command description (#96)

There is no `docker -ps` command.  The docker `ps` subcommand lists containers.  The docker `history` subcommand will list the layers, like so:

```
hartzelg@blah:~$ docker history hartzelg/cime
IMAGE               CREATED             CREATED BY                                      SIZE                COMMENT
7185c64fa2a5        5 months ago        /bin/sh -c #(nop) VOLUME [/cime_config.yaml]    0 B
fd53e85b356b        5 months ago        /bin/sh -c #(nop) VOLUME [/var/jenkins_home]    0 B
b50f22ae6ec1        5 months ago        /bin/sh -c #(nop) COPY file:fb917d778dfa959bb   10.11 kB
[...]
```

* Typo: maybe -> may be (#97)

* Oops: "pids" -> "numeric userid" (#98)

"pids" is process ids, I think?

What you mean is the users numeric id, e.g. on machine A `hartzell` is `3245` so files are owned by `3245`.  But, when I get to machine B where `hartzell` is `22453`, those same files appear to be someone else.

Right?

* nothing to see here...

* fixing environment variable example

* Fix typo (#99)

* $SINGULARITY_ENVIRONMENT, --nv, and random cleanup

* removed Usage sections from commands

* removed $SINGULARITY_ENVIRONMENT jazz from current docs

* reverted docs-usage to old version

* cleaning up the merge a bit more

* finalizing and approving PR to close #100!
@dctrud

This comment has been minimized.

Show comment
Hide comment
Contributor

dctrud commented Feb 22, 2018

@dctrud dctrud closed this Feb 22, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment