Singularity container cannot find files after being created from a Docker image

[Deepakgthomas]

Version of Apptainer

What version of Apptainer (or Singularity) are you using? Run

apptainer --version (or singularity --version).

singularity-ce version 4.0.2

Description of issue
I have posted the same question on SO too - https://stackoverflow.com/questions/77781660/singularity-container-cannot-find-files-after-being-created-from-a-docker-image

I am creating a Singularity container from a Docker image. However, the Singularity container gives me errors that weren't present while I used Docker.

This is how I create my Docker image -

FROM ubuntu:22.04

WORKDIR /app

SHELL ["/bin/bash", "-c"] 

RUN echo "Hello World!"
RUN apt-get update && apt-get install -y \
    libosmesa6-dev \
    sudo \
    wget \
    curl \
    unzip \
    gcc \
    g++ \
    &&  apt-get install \
    libosmesa6-dev \
    && rm -rf /var/lib/apt/lists/*
    
ENV DEBIAN_FRONTEND=noninteractive

ENV PATH="/miniconda3/bin:${PATH}"
ARG PATH="/miniconda3/bin:${PATH}"
RUN cd / \
    && mkdir -p /miniconda3 \
    && wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -O /miniconda3/miniconda.sh \
    && bash /miniconda3/miniconda.sh -b -u -p /miniconda3 \
    && /miniconda3/bin/conda init bash \
    && source ~/.bashrc \
    && conda init \
    && conda create -y -n myenv python=3.8 \
    && conda update -y conda
    
WORKDIR /~
RUN wget https://roboti.us/download/mjpro150_linux.zip \
    && unzip mjpro150_linux.zip \
    && mkdir ~/.mujoco \
    && mv mjpro150 ~/.mujoco \
    && wget https://roboti.us/file/mjkey.txt \
    && mv mjkey.txt ~/.mujoco \
    && rm mjpro150_linux.zip


RUN cd /miniconda3/envs/myenv/lib/ && mv libstdc++.so.6 libstdc++.so.6.old && ln -s /usr/lib/x86_64-linux-gnu/libstdc++.so.6 libstdc++.so.6
SHELL ["conda", "run", "-n", "myenv", "/bin/bash", "-c"]
EXPOSE 5003


RUN pip install --no-cache-dir "Cython<3" \
    && pip install mujoco-py==1.50.1.0 \
    && pip install gym

This is how I build, run and test my Docker image -

docker build -t dockercontainer:latest .
docker run -it dockercontainer:latest 

Once inside the container -

conda run -n myenv python -c"import gym; gym.make('Humanoid-v3')"

This works (except for a few warnings).

Then, I build a Singularity container from the Docker image -

singularity build singularity_container.sif docker-daemon://dockercontainer:latest

This is how I run the Singularity container -

singularity exec singularity_container.sif/ conda run -n myenv python -c"import gym; gym.make('Humanoid-v3')"

But then I get this error over here -

File "/home/thoma/.local/lib/python3.8/site-packages/mujoco_py/config.py", line 37, in init_config
    raise error.MujocoDependencyError('Found your MuJoCo license key but not binaries. Please put your binaries into ~/.mujoco/mjpro131 or set MUJOCO_PY_MJPRO_PATH. Follow the instructions on https://github.com/openai/mujoco-py for setup.')
mujoco_py.error.MujocoDependencyError: Found your MuJoCo license key but not binaries. Please put your binaries into ~/.mujoco/mjpro131 or set MUJOCO_PY_MJPRO_PATH. Follow the instructions on https://github.com/openai/mujoco-py for setup.

ERROR conda.cli.main_run:execute(124): `conda run python -cimport gym; gym.make('Humanoid-v3')` failed. (See above for error)

What OS/distro are you running

PRETTY_NAME="Ubuntu 22.04.3 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.3 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy

How did you install Apptainer

I installed Singularity. I followed the documentation given over here - https://docs.sylabs.io/guides/3.0/user-guide/installation.html

[dtrudg]

Converted to a discussion as this is not a bug or feature request.

Your Dockerfile installs various things into the default home directory in the container. This will be /root by default, unless you use a USER statement in your Dockerfile.

Singularity is different to Docker with home directory handling:

1. Inside the container, you are the user who started the container. Therefore the home directory inside the container is unlikely to match that under Docker. It won't be /root unless you start the container as root, or with --fakeroot.

2. Additionally, Singularity binds your home directory from the host into the container, by default. This will mask any files inside the container image at the home directory PATH.

There are flags to stop Singularity binding the home directory for the host, and to override the home directory in the container. Search --no-mount and --home in the documentation.

Generally images that depend on things that are in the home directory of the container image will not work correctly under Singularity without some of these additional flags being specified. If you need to use a Docker container with Singularity it is best, instead, to write it so that no software / configuration is places in the home directory in the Docker build.

I would advise reviewing the Docker compatibility section of the user guide, or investigating the new OCI-mode in Singularity 4.0+ - which behaves more like Docker.

https://docs.sylabs.io/guides/4.0/user-guide/singularity_and_docker.html#differences-and-limitations-vs-docker

[Deepakgthomas]

Thank you for your reply.

Currently, I am running Docker and Singularity (sequentially) on the same computer. Therefore, won't the home directory be the same for both containers?

[dtrudg]

Currently, I am running Docker and Singularity (sequentially) on the same computer. Therefore, won't the home directory be the same for both containers?

No. In default operation Singularity does not attempt to behave identically to Docker.

With Docker, you are the root user in the container by default, so your home directory will be /root. With Singularity, you are the same user in the container as the user that you ran singularity with on the host, and your home directory will be the home directory path of that user.

With Docker, the home directory from the host computer does not get bind mounted into the container. With Singularity it will be by default.

As above, you can either investigate the Docker compatibility (--compat and related) flags, --no-mount and --home, or you may wish to try the new OCI-mode in 4.0

I suggest working through all of the following sections of the user guide to understand how Singularity differs from Docker (intentionally), and how the OCI-mode offers a

https://docs.sylabs.io/guides/4.0/user-guide/singularity_and_docker.html
https://docs.sylabs.io/guides/4.0/user-guide/oci_runtime.html#oci-mode-oci