Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ERROR: build: failed to make environment files: open /tmp/sbuild-468665265/fs/etc/resolv.conf: permission denied #4532

Closed
soichih opened this issue Sep 26, 2019 · 14 comments · Fixed by #4646
Assignees
Labels
Milestone

Comments

@soichih
Copy link
Contributor

@soichih soichih commented Sep 26, 2019

I am using singularity version 3.4.0-1.2.el7

I used to be able to run my container with v2 singularity with the following command.

$ singularity exec -e docker://brainlife/hcppipeline:0.1 whoami
..
hayashis

However, when I run it with singularity v3 (3.4.0-1.2.el7). I run into a strange error message.

$ singularity exec -e docker://brainlife/hcppipeline:0.1 whoami
INFO:    Converting OCI blobs to SIF format
INFO:    Starting build...
...
2019/09/26 14:20:29  warn rootless{dev/tty0} creating empty file in place of device 4:0
2019/09/26 14:20:29  warn rootless{dev/tty1} creating empty file in place of device 4:1
2019/09/26 14:20:29  warn rootless{dev/tty2} creating empty file in place of device 4:2
...
2019/09/26 14:20:29  warn rootless{dev/tty9} creating empty file in place of device 4:9
2019/09/26 14:20:29  warn rootless{dev/urandom} creating empty file in place of device 1:9
2019/09/26 14:20:29  warn rootless{dev/zero} creating empty file in place of device 1:5
2019/09/26 14:20:33  info unpack layer: sha256:755da0cdb7d25b74b205ff1eccd26ea4eede693ec7cf2150ae4c1caafe6394b1
2019/09/26 14:20:33  info unpack layer: sha256:969d017f67e62ae323a3e8077e3ac4a5b1bf4a27c349148c1f6c28bd6ca3bbb8
2019/09/26 14:20:33  info unpack layer: sha256:37c9a911359525fa28aa16715d36954723a8924492b5216cc97d1099251a5023
...
2019/09/26 14:21:40  info unpack layer: sha256:1822309a1099a548079350dfc7837fe8cc905498e9a6181f72ea791accf07e6a
...
ERROR:   build: failed to make environment files: open /tmp/sbuild-468665265/fs/etc/resolv.conf: permission denied
FATAL:   Unable to handle docker://brainlife/hcppipeline:0.1 uri: unable to build: packer failed to pack: while inserting base environment: build: failed to make environment files: open /tmp/sbuild-468665265/fs/etc/resolv.conf: permission denied

What OS/distro are you running

$ cat /etc/os-release
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"

How did you install Singularity

I did yum install singularity

@dctrud

This comment has been minimized.

Copy link
Contributor

@dctrud dctrud commented Sep 26, 2019

This is likely related to #4524

@dctrud

This comment has been minimized.

Copy link
Contributor

@dctrud dctrud commented Sep 26, 2019

Not sure what is going on here yet...

When building, in the failed rootfs at /tmp/sbuild-442286694/fs, the permissions on /etc seem to be fine...

drwxr-xr-x 79 dave dave 4.0K Jan 27  2018 etc/

... and etc/resolv/conf is there, but a 0 byte file

dave@piran/t/s/f/etc> pwd
/tmp/sbuild-442286694/fs/etc
dave@piran/t/s/f/etc> ls -lah resolv.conf
-r--r--r-- 1 dave dave 0 Jan 18  2017 resolv.conf

@ikaneshiro @cclerget - are we writing into resolv.confin the build, rather than just binding host one over it?

@jmstover

This comment has been minimized.

Copy link
Collaborator

@jmstover jmstover commented Sep 26, 2019

It looks like the build is attempting to open it for write... To write an empty string into it.

 f, err := os.OpenFile(name, os.O_WRONLY|os.O_CREATE|os.O_TRUNC, perm)

If we're ending up adding user write perm to all files/directories in #4524 ... then this should be fixed by that as well.

@dctrud

This comment has been minimized.

Copy link
Contributor

@dctrud dctrud commented Sep 26, 2019

@jmstover - Currently #4524 is ensuring w on directories (so dir contents can be removed) but only r on files (so they can be read during a copy or inter-device move). thoughts?

@jmstover

This comment has been minimized.

Copy link
Collaborator

@jmstover jmstover commented Sep 26, 2019

If in makeFile we're writing to files to write empty strings into them... then unfortunately directories can't be the only thing we're touching...

@dctrud

This comment has been minimized.

Copy link
Contributor

@dctrud dctrud commented Sep 26, 2019

yeah - we used to do 0600 in the image-tools patch... I was trying to be a little tighter.

dctrud added a commit to dctrud/singularity that referenced this issue Sep 26, 2019
@dctrud

This comment has been minimized.

Copy link
Contributor

@dctrud dctrud commented Sep 26, 2019

The above #4524 commit 12e7860 fixes the error for me

2019/09/26 11:21:25  info unpack layer: sha256:e4ae2934f55b87e6bb491d7f47ed754c8e9671e6d6878ee0778b29ab85734d8c
2019/09/26 11:21:25  info unpack layer: sha256:1822309a1099a548079350dfc7837fe8cc905498e9a6181f72ea791accf07e6a
INFO:    Creating sandbox directory...
INFO:    Build performed with no clean up option, build bundle(s) located at: [/tmp/sbuild-665988338]
INFO:    Build complete: hcppipeline
@soichih

This comment has been minimized.

Copy link
Contributor Author

@soichih soichih commented Oct 2, 2019

Is there a workaround for this issue? Is there an easy way to downgrade to singularity v2 for CentOS7?

@dctrud

This comment has been minimized.

Copy link
Contributor

@dctrud dctrud commented Oct 2, 2019

Hi @soichih - The Sylabs team doesn't create the EPEL packages. They are kindly created for the community by @DrDaveD. I presume there may be an archive somewhere you can download and yum downgrade.

Note that downgrading to 3.3.x is sufficient to avoid this issue - and that we have a PR merged into the release-3.4 branch now that addresses it. We will prepare a release candidate, and then a 3.4.2 release in the near future.

(also Dave Dykstra has indicated he may backport the patch into his EPEL 3.4.1)

@DrDaveD

This comment has been minimized.

Copy link
Contributor

@DrDaveD DrDaveD commented Oct 3, 2019

Yes patch #4522 will be included in EPEL singularity-3.4.1-1.2. I just submitted the requests to get it into epel-testing, which usually are fulfilled within 24 hours.

@verdurin

This comment has been minimized.

Copy link

@verdurin verdurin commented Oct 17, 2019

The EPEL build fixes this problem for me.

@dctrud

This comment has been minimized.

Copy link
Contributor

@dctrud dctrud commented Oct 17, 2019

This will also need to be mopped up in master with the new sandbox handling that doesn't do permission mangling. I believe the staging to tmpfs suggested in #4579 will address this.

@dctrud dctrud added the Release 3.5 label Oct 17, 2019
@dctrud dctrud added this to the 3.5 milestone Oct 17, 2019
@dctrud

This comment has been minimized.

Copy link
Contributor

@dctrud dctrud commented Oct 18, 2019

This is not fixed by #4629. Still exists on master.

dctrud added a commit to dctrud/singularity that referenced this issue Oct 18, 2019
Fixes: sylabs#4532

In base_environment we try to create or truncate /etc/hosts and
/etc/resolv.conf in the container, with a specified permission. The
OpenFile call will create a new file with the correct permission, but
we need to chmod an existing file, to avoid a fatal error when building
from a base image where these files do not have write permission.

Signed-off-by: David Trudgian <dave@trudgian.net>
@dctrud

This comment has been minimized.

Copy link
Contributor

@dctrud dctrud commented Oct 18, 2019

This is fixed on master by #4646

dave@piran~/S/G/s/builddir> singularity exec -e docker://brainlife/hcppipeline:0.1 whoami
...
2019/10/18 16:08:25  info unpack layer: sha256:1822309a1099a548079350dfc7837fe8cc905498e9a6181f72ea791accf07e6a
INFO:    Creating SIF file...
dave
@mem mem closed this in #4646 Oct 18, 2019
mem added a commit that referenced this issue Oct 18, 2019
Fixes: #4532

In base_environment we try to create or truncate /etc/hosts and
/etc/resolv.conf in the container, with a specified permission. The
OpenFile call will create a new file with the correct permission, but
we need to chmod an existing file, to avoid a fatal error when building
from a base image where these files do not have write permission.

Signed-off-by: David Trudgian <dave@trudgian.net>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
9 participants
You can’t perform that action at this time.