Singularity contains built from bioconda have broken /etc/resolv.conf

[pvanheus]

A test with a container built using singularity build from the Docker images on quay.io using Singularity 2.5.2.

$ singularity exec /tools/containers/sra-tools-2.9.1_1--h470a237_0.simg ping www.google.com
ping: bad address 'www.google.com'
$ singularity exec /tools/containers/sra-tools-2.9.1_1--h470a237_0.simg cat /etc/resolv.conf
cat: can't open '/etc/resolv.conf': No such file or directory
$ singularity exec /tools/containers/sra-tools-2.9.1_1--h470a237_0.simg ls -ld /etc/resolv.conf
lrwxrwxrwx    1 root     root            18 Feb 27  2014 /etc/resolv.conf -> ../tmp/resolv.conf

This is mentioned by @vsoch as a Weirdo File. It is not clear where the link to ../tmp/resolv.conf comes from.

[vsoch]

resolv.conf should be mapped from the host to resolve dns names, see here. Is there either something different going on with binds, or permissions here? Also, I believe that Singularity 2.5* family has (runtime?) security issues, so I would consider upgrading and seeing if you reproduce the error there.

[vsoch]

This still might be the busybox issue, e.g.,

there is still the issue of the missing /etc/resolv.conf (noted in #749 (comment)) in biocontainers images pulled from quay.io due to their use of a version of busybox that is incompatible with how singularity exposes the host resolv.conf (unless a privileged user subsequently modifies the image with the workaround described in #749 (comment)).

[pvanheus]

I think this is a busybox issue rather than a 2.5.* issue. I get the same result with 3.0:

$ /var/tmp/src/github.com/sylabs/singularity/builddir/singularity exec /var/tmp/diamond-0.9.21--1.simg ls -ld /etc/resolv.conf
lrwxrwxrwx    1 root     root            18 Oct 25 16:48 /etc/resolv.conf -> ../tmp/resolv.conf
$ /var/tmp/src/github.com/sylabs/singularity/builddir/singularity --version
singularity version v3.0.0

Apologies for the long-winded path to singularity - it is a local build referring to an imagine built from a bioconda container.

[vsoch]

Can you just use a different container that doesn't have a busybox base? What about:

$ singularity build sra-tools.simg docker://inutano/sra-toolkit
Docker image path: index.docker.io/inutano/sra-toolkit:latest
Cache folder set to /home/vanessa/.singularity/docker
[5/5] |===================================| 100.0% 
Importing: base Singularity environment
Exploding layer: sha256:0423802a3f4956844710db206c0022c8509834ec3f84659f5908c80c99dbeb10.tar.gz
Exploding layer: sha256:91c010ec7be39a0e05f16d0fb06b9d6e8a4ea62b293f47ee380033598df6dd30.tar.gz
Exploding layer: sha256:a0a1ed9583752de17fe62bb9c599d56eb7101007e97e02c473ebce5ca737a58e.tar.gz
Exploding layer: sha256:d3d8062ca11ad83415399ce117c4a8de8a9f44a89ba7d908b6e6668621a57b4e.tar.gz
Exploding layer: sha256:55bc265397581a10565dad308fabe95b06310ef186cffcae1c1e7d44c8f85ffb.tar.gz
Exploding layer: sha256:c6a9ef4b9995d615851d7786fbc2fe72f72321bee1a87d66919b881a0336525a.tar.gz
WARNING: Building container as an unprivileged user. If you run this container as root
WARNING: it may be missing some functionality.
Building Singularity image...
Singularity container built: sra-tools.simg
Cleaning up...

$ singularity exec sra-tools.simg sra-search

ERROR: Missing arguments

Usage:
  sra-search [Options] query accession ...

Summary:
  Searches all reads in the accessions and prints Ids of all the fragments that contain a match.

Example:
  sra-search ACGT SRR000001 SRR000002
  sra-search "CGTA||ACGT" -e -a NucStrstr SRR000002

Options:
  -h|--help                 Output brief explanation of the program.
  -a|--algorithm <alg>      Search algorithm, one of:
      FgrepStandard (default)
      FgrepBoyerMoore
      FgrepAho
      AgrepDP
      AgrepWuManber
      AgrepMyers
      AgrepMyersUnltd
      NucStrstr
      SmithWaterman
  -e|--expression <expr>    Query is an expression (currently only supported for NucStrstr)
  -S|--score <number>       Minimum match score (0..100), default 100 (perfect match);
                            supported for all variants of Agrep and SmithWaterman.
  -T|--threads <number>     The number of threads to use; 2 by deafult
  --threadperacc            One thread per accession mode (by default, multiple threads per accession)
  --sort                    Sort output by accession/read/fragment
  --reference [refName,...] Scan reference(s) for potential matches; all references if none specified
  -m|--max <number>         Stop after N matches
  -U|--unaligned            Search in unaligned and partially aligned reads only
  --fasta [ <lineWidth> ]   Output in FASTA format with specified line width (default 70 bases)


sra-search : 2.9.0 ( 2.9.2 )

[vsoch]

Not to say that is the "best" one, just a randomly chosen one off Docker Hub that minimally has an associated repository and build recipe :) https://hub.docker.com/r/inutano/sra-toolkit/~/dockerfile/

[pvanheus]

The base container question is deeper than I can answer - perhaps @bgruening or @jmchilton can advise?

[bgruening]

The base image was chosen to have a minimal as possible container that still has libc included and not some libc derivative.

[pvanheus]

@bgruening very often resolv.conf doesn't matter for bioinformatics tools but for some that use the network (like sra-tools) this is an issue.

[vsoch]

Maybe a different base could be considered for the tools that rely on webby places?

[nathanweeks]

Another place where we have encountered issues with busybox-base-imaged biocontainers breaking singularity is that the /var/tmp -> /tmp symlink causes issues when "mount tmp = yes" in singularity.conf; specifically, the host /tmp is mounted at container /tmp, then the host /var/tmp ends up being mounted over the container /tmp... So both the container /tmp and /var/tmp are host /var/tmp. Things like Open-MPI and PMIX put, e.g., unix domain sockets in /tmp, so this breaks containers that want to use MPI.

However... the busybox base image used for biocontainers is quite old (almost 5 years!):

$ $ docker run -it --rm quay.io/biocontainers/samtools:1.9--h8571acd_11 busybox | head -n 1
BusyBox v1.22.1 (2014-05-23 01:24:27 UTC) multi-call binary.

Newer busybox images don't have a /var/tmp -> /tmp symlink, and so shouldn't break Singularity in this regard (and could be beneficial to use for other reasons, such as hundreds of bug fixes and enhancements).

Could the base busybox image for biocontainers be updated, at least going forward for newly-created biocontainers images?

[bgruening]

It tried this one year ago and it broke so many workflows from users. But I should probably try now again. The problem is glibc or the variants. I see that there is now a proper glibc variant, I will give this a try.

[vsoch]

You could try doing a PR to update the base image first here (or better, create a new tag and preserve the older one) and then the container here can be rebuilt.

I can't really tell which is the correct base image (the tags don't line up) but probably someone knows. You could also just build a different samtools image.

[bgruening]

@vsoch these are not the base images for the automatically created containers. For this we used a busybox image with a static compiled bash - to please some workflow engines.

@nathanweeks I created a new version here. Would be great if you can test this image:
https://cloud.docker.com/repository/docker/bgruening/busybox-bash/tags

But this was the easy part, testing a sufficient amount of package builds before we put this into production is the harder part. @nathanweeks are you familiar with the way how we build those containers? Could you help with this?

[vsoch]

@bgruening can you make the base recipes available to the users so they know what they are dealing with?

[bgruening]

They are here: https://github.com/bgruening/docker-busybox-bash

[vsoch]

cool thanks! And it looks like you have an updated image to test. Is there a way (in the future) for the user to be able to find this repo (or will it eventually be in the biocontainers repo?)

[nathanweeks]

@bgruening : is it the mulled-build described in the Bioconda Build system docs? Otherwise, if you could link to some documentation (including on how to change the base image), I could probably figure it out & test at least a few (but sufficiently complicated) recipes I'm interested in.

[bgruening]

@vsoch is it not linked on the dockerhub page? We should add it to some documentation if not.

@nathanweeks yes exactly. That should still work I hope: https://galaxy-lib.readthedocs.io/en/latest/topics/mulled.html?highlight=mulled#building-docker-containers-for-local-conda-packages

You should be able to specify the base image via command line args. If not this should get you started: galaxyproject/galaxy-lib#148

Sorry, in a train with terrible internet.
A nice Easter holiday for you all!

[vsoch]

Omg it's easter!! :D I totally forgot. I'm going to paint some tiny avocados like eggs this weekend :)

Happy Easter!

[nathanweeks]

@bgruening : When attempting to build a container image for a bioconda package that requires perl (maker), the following error was emitted during the mulled-build stage:

SERR /usr/local/bin/perl: error while loading shared libraries: libdl.so.2: cannot open shared object file: No such file or directory

Possibly related?: docker-library/busybox#46

[nathanweeks]

FWIW, I tried alpine:3.9.3 + libc6-dev, and while it has libdl.so.2, it's missing a number of glibc-related symbols:

/source # ldd /usr/local/bin/perl
        /lib64/ld-linux-x86-64.so.2 (0x7f87a267d000)
        libpthread.so.0 => /lib64/ld-linux-x86-64.so.2 (0x7f87a267d000)
        libdl.so.2 => /lib64/ld-linux-x86-64.so.2 (0x7f87a267d000)
        libm.so.6 => /lib64/ld-linux-x86-64.so.2 (0x7f87a267d000)
        libc.so => /lib64/ld-linux-x86-64.so.2 (0x7f87a267d000)
Error relocating /usr/local/bin/perl: __snprintf_chk: symbol not found
Error relocating /usr/local/bin/perl: getgrent_r: symbol not found
Error relocating /usr/local/bin/perl: __vfprintf_chk: symbol not found
Error relocating /usr/local/bin/perl: __isnan: symbol not found
Error relocating /usr/local/bin/perl: gethostent_r: symbol not found
Error relocating /usr/local/bin/perl: getnetbyname_r: symbol not found
Error relocating /usr/local/bin/perl: getpwent_r: symbol not found
Error relocating /usr/local/bin/perl: __isinf: symbol not found
Error relocating /usr/local/bin/perl: getservent_r: symbol not found
Error relocating /usr/local/bin/perl: getnetent_r: symbol not found
Error relocating /usr/local/bin/perl: __register_atfork: symbol not found
Error relocating /usr/local/bin/perl: __memmove_chk: symbol not found
Error relocating /usr/local/bin/perl: __memcpy_chk: symbol not found
Error relocating /usr/local/bin/perl: getprotoent_r: symbol not found
Error relocating /usr/local/bin/perl: __vsnprintf_chk: symbol not found
Error relocating /usr/local/bin/perl: __longjmp_chk: symbol not found
Error relocating /usr/local/bin/perl: __finitel: symbol not found
Error relocating /usr/local/bin/perl: getnetbyaddr_r: symbol not found
Error relocating /usr/local/bin/perl: getprotobynumber_r: symbol not found
Error relocating /usr/local/bin/perl: getprotobyname_r: symbol not found
Error relocating /usr/local/bin/perl: _LIB_VERSION: symbol not found

[bgruening]

Oh fun, this reminds me on my tries from last year :(

[bgruening]

@nathanweeks let me know if there is any busybox container which we can use.

[nathanweeks]

@bgruening :

alpine + alpine-pkg-glibc might be close enough. I substituted bgruening/busybox-bash:0.2 with an image created from the following Dockerfile, and the result was almost enough to run maker (which failed for other reasons):

FROM alpine:3.9.3

RUN wget -q -O /etc/apk/keys/sgerrand.rsa.pub https://alpine-pkgs.sgerrand.com/sgerrand.rsa.pub && \
    wget https://github.com/sgerrand/alpine-pkg-glibc/releases/download/2.29-r0/glibc-2.29-r0.apk && \
    apk add --no-cache glibc-2.29-r0.apk && \
    rm glibc-2.29-r0.apk /etc/apk/keys/sgerrand.rsa.pub

The resulting docker image clocks in at 9.66MB (compared to 6.65MB for bgruening/busybox-bash:0.2), though adding bash increases the size about 40% (!) to 13.7MB. Still fairly minimal compared to other images like debian:9.8-slim (55.3MB).

Of course, this would need more testing.

[bgruening]

@nathanweeks I push 0.3 which is doing a very similar thing than what you mentioned, indeed this is what miniconda is now switching to as well. Maybe we can give the 0.3 a try ...?

[nathanweeks]

@bgruening : Out of the box, mulled-build build-and-test still ends with this error using the bgruening/busybox-bash:latest image.

However, it looks like the glibc libraries are in a conda-specific location. Committing the stopped container resulting from mulled-build into a new image, shelling into a container from that image, and poking around, it looks like the glibc libraries are in a conda-specific location. Setting LD_LIBRARY_PATH, I was able to at least get perl to work:

bash-4.4# LD_LIBRARY_PATH=/lib:/usr/local/lib:/usr/local/x86_64-conda_cos6-linux-gnu/sysroot/lib/ perl --version

This is perl 5, version 26, subversion 2 (v5.26.2) built for x86_64-linux-thread-multi

However, maker (a perl script) itself segfaulted:

bash-4.4# LD_LIBRARY_PATH=/lib:/usr/local/lib:/usr/local/x86_64-conda_cos6-linux-gnu/sysroot/lib/ maker -h                                                    
Segmentation fault

[bgruening]

@pvanheus does Singularity 3 help here? We have rebuild all containers with Singularity 3 in the last weeks.

[nathanweeks]

@bgruening the problem persists:

$ singularity version
3.4.1-1.el7
$ curl -LO https://depot.galaxyproject.org/singularity/sra-tools%3A2.9.6--hf484d3e_0
...
$ singularity exec sra-tools%3A2.9.6--hf484d3e_0 ping www.google.com
ping: bad address 'www.google.com'
$ singularity exec sra-tools%3A2.9.6--hf484d3e_0 ls -l /etc/resolv.conf
lrwxrwxrwx    1 root     root            18 Aug 25 17:04 /etc/resolv.conf -> ../tmp/resolv.conf
$ singularity exec sra-tools%3A2.9.6--hf484d3e_0 ls -l /tmp
total 0
drwxr-xr-x    2 173      173              6 Jul 13  2018 abrt
drwx------    3 root     root            17 Jun 11 16:40 systemd-private-0cd1954d757443d69908297e3f763b1b-ntpd.service-unzAlP
$ singularity inspect sra-tools%3A2.9.6--hf484d3e_0 
==labels==
org.label-schema.schema-version: 1.0
org.label-schema.usage.singularity.deffile.bootstrap: docker
org.label-schema.usage.singularity.deffile.from: quay.io/biocontainers/sra-tools:2.9.6--hf484d3e_0
org.label-schema.usage.singularity.version: 3.3.0-rc.2
org.label-schema.build-date: Sunday_25_August_2019_21:4:41_UTC

[pcm32]

Hi! Has this problem evolved in any way? I have similar issues with:

singularity shell docker://quay.io/biocontainers/wget:1.20.1
INFO:    Using cached SIF image
WARNING: Skipping mount /var/singularity/mnt/session/etc/resolv.conf [files]: /etc/resolv.conf doesn't exist in container
Singularity> ping www.google.com
ping: bad address 'www.google.com'
Singularity> cd /etc/
Singularity> ls -ltr
total 41
-rw-rw-r--    1 root     root         10873 Feb 27  2014 services
-rw-rw-r--    1 root     root           386 Feb 27  2014 securetty
lrwxrwxrwx    1 root     root            18 Feb 27  2014 resolv.conf -> ../tmp/resolv.conf

although there is no /tmp/resolv.conf.

singularity version
3.6.4-1.el7

thanks!

[pvanheus]

Thanks @mbargull the singularity config global --set 'mount tmp' no. works on Singularity 3.5.3, on Singularity 3.4 I had to edit the singularity.conf file manually.

[mbargull]

Thanks @mbargull the singularity config global --set 'mount tmp' no. works on Singularity 3.5.3, on Singularity 3.4 I had to edit the singularity.conf file manually.

Happy to hear that works for you!

The workaround I described works with Singularity 3.6.x (x=4 I think) since I was told that 3.7 wasn't available yet, in case this is useful for anyone. Works well for all the cases I have tried so far.

Right, overriding the tmp mount is also an option! (Sorry, overlooked your message before.)
(And of course, you can cp /etc/resolv.conf /any/path/ (e.g., /any/path be $( pwd )/tmp) and then --bind /any/path:/tmp if you don't want the host's /tmp to be mounted.)

[pvanheus]

@mbargull downstream of this I discovered that some tools (e.g. R ones) write to /tmp - and then setting mount tmp to no led to them having a readonly /tmp. So I'm still working on the perfect workaround.

[fgypas]

I started using biocontainer images for a snakemake pipeline and it seems that all of them give a similar warning:
WARNING: Skipping mount /opt/conda/envs/zarp/var/singularity/mnt/session/etc/resolv.conf [files]: /etc/resolv.conf doesn't exist in container
I am using singularity version 3.5.3-1.1.el7

Do we have any updates on the issue?

[nathanweeks]

@fgypas : it seems recent Biocontainers images use an updated base image (I'm guessing this one is now the default, but I don't know that for a fact---can anyone confirm or refute?) that is not affected by this issue.

e.g.:

$ singularity exec docker://quay.io/biocontainers/csvtk:0.23.0--h9ee0642_0 csvtk version
...
INFO:    Creating SIF file...
csvtk v0.23.0
$ singularity exec docker://quay.io/biocontainers/csvtk:0.23.0--h9ee0642_0 cat /etc/os-release
INFO:    Using cached SIF image
PRETTY_NAME="Debian GNU/Linux 10 (buster)"
NAME="Debian GNU/Linux"
VERSION_ID="10"
VERSION="10 (buster)"
VERSION_CODENAME=buster
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"

[mbargull]

Yes, we now use quay.io/bioconda/base-glibc-busybox-bash:2.1.0 (and quay.io/bioconda/base-glibc-debian-bash:2.1.0 for the extended base image).
I just checked and the *busybox* variant does not have etc/resolv.conf but the *debian* one has etc/resolv.conf
(with contents

# https://1.1.1.1 (privacy-focused, highly-available DNS service)
nameserver 1.1.1.1
nameserver 1.0.0.1

-- the content is not very important since the bind mounted one gets used, but just wanted to record it here.)
We can update the base images now much easier, so I can add the same file to the smaller image to get consistent behavior.
(Leaving this ticket open until I have done that.)

[fgypas]

So what I am using at the moment are the following images:

• bedtools: quay.io/biocontainers/bedtools:2.27.1--h9a82719_5
• bedgraphtobigwig: quay.io/biocontainers/ucsc-bedgraphtobigwig:377--h0b8a92a_2
• cutadapt: quay.io/biocontainers/cutadapt:1.16--py35_2
• kallisto: quay.io/biocontainers/kallisto:0.46.1--h4f7b962_0
• samtools: quay.io/biocontainers/samtools:1.10--h2e538c0_3
• star: quay.io/biocontainers/star:2.7.3a--0
• salmon: quay.io/biocontainers/salmon:1.1.0--hf69c8f4_0
• gffread: quay.io/biocontainers/gffread:0.11.7--h8b12597_0

Does that mean that I can find similar versions of the tools with the latest base image, or should I use the latest (updated versions of the tools)?

[mvdbeek]

Yes, we now use quay.io/bioconda/base-glibc-busybox-bash:2.1.0 (and quay.io/bioconda/base-glibc-debian-bash:2.1.0 for the extended base image).

Thanks for the update @mbargull ! Any chance those images could be published under a repo name we can track, such as bioconda/default-base:latest and bioconda/extended-base:latest ? That would make it easier for projects like https://github.com/BioContainers/multi-package-containers/ to stay in sync with what bioconda does.

[bgruening]

I think this one can now be closed, please reopen if this is still an issue.

[nfancy]

@pcm32 hi I am encountering the same issue. Can you please explain a bit more on how to use this command. Sorry for my ignorance, how do I use it as a singularity command? I am using the following command to run nf-core/scrnaseq pipeline which uses one of the python image from galaxy and I get the following error.

Command error:
  WARNING: Skipping mount /var/singularity/mnt/session/etc/resolv.conf [files]: /etc/resolv.conf doesn't exist in container
  FATAL:   container creation failed: mount /rds/general/user/nfancy/ephemeral/tmp_cellranger/28/b8c34d34d92c874f630f7a89b802f3->/rds/general/user/nfancy/ephemeral/tmp_cellranger/28/b8c34d34d92c874f630f7a89b802f3 error: while mounting /rds/general/user/nfancy/ephemeral/tmp_cellranger/28/b8c34d34d92c874f630f7a89b802f3: destination /rds/general/user/nfancy/ephemeral/tmp_cellranger/28/b8c34d34d92c874f630f7a89b802f3 doesn't exist in container

#!/bin/bash
#PBS -l walltime=72:00:00
#PBS -l select=1:ncpus=2:mem=8gb
#PBS -o /rds/general/user/nfancy/home/log/
#PBS -e /rds/general/user/nfancy/home/log/

cd $PBS_O_WORKDIR
module load gcc/8.2.0
NXF_OPTS='-Xms1g -Xmx4g'

resource_dir=/rds/general/project/ukdrmultiomicsproject/live/Users/Nurun
launch_dir=/rds/general/user/nfancy/home/ROSMAP_snRNA_micro
export PATH=$resource_dir/scflow_resources/jdk-11.0.12/bin:$PATH
export JAVA_HOME=$resource_dir/scflow_resources/jdk-11.0.12

$resource_dir/scflow_resources/nextflow run \
$launch_dir/scrnaseq/main.nf \
--input $launch_dir/samplesheet.csv \
--outdir $launch_dir/result \
-c $launch_dir/conf/imperial.config \
-c $launch_dir/conf/run_param.config \
-c $launch_dir/conf/resource.config

Thanks in advance. Nurun

[pcm32]

@nfancy , not sure how you would do it on nextflow, but I presume there must be somewhere there where you can specify the singularity binary, then you could create a bash script that wraps the singularity call and adds the requirement to mount the resolv.conf file in the appropiate place inside the container. Then tell nextflow to call that as a singularity binary and it should work. Or maybe you can tell nextflow what additional files should be mounted on singularity. They key is to provision the resolv.conf file from the host machine in the required place inside the container.

[nathanweeks]

@nfancy Regarding the /etc/resolv.conf warning: I've seen this error in old container images that were built before the Biocontainers base image update mentioned in #11583 (comment). I don't have practical experience with Nextflow; does your workflow define an old container image(s)?

The "FATAL" error looks like it could be a site-specific Singularity configuration issue; I'd suggest reaching out to your local sysadmin and/or Nextflow experts for guidance. This issue could be related.