diff --git a/appendix.rst b/appendix.rst index f326d8e..5c65110 100755 --- a/appendix.rst +++ b/appendix.rst @@ -1124,7 +1124,7 @@ where both ```` and ```` are mandatory fields that must be written explicitly. -.. _docker-daemon: +.. _docker-archive: ``docker-archive`` bootstrap agent ================================== diff --git a/build_a_container.rst b/build_a_container.rst index baab36d..53b897a 100755 --- a/build_a_container.rst +++ b/build_a_container.rst @@ -270,6 +270,246 @@ containers. The containers are decrypted at runtime entirely in kernel space, meaning that no intermediate decrypted data is ever written to disk. See :ref:`encrypted containers ` for more details. +************************* +Building from Dockerfiles +************************* + +Starting with version 4.1, {Singularity} can build OCI-SIF images directly from +`Dockerfiles +`__, +creating images that can be run using {Singularity}'s :ref:`OCI mode +`. + +.. code:: console + + $ cat ./Dockerfile + FROM debian + CMD cat /etc/os-release + + $ singularity build --oci ./debian.oci.sif ./Dockerfile + INFO: Did not find usable running buildkitd daemon; spawning our own. + INFO: cfg.Root for buildkitd: /home/omer/.local/share/buildkit + INFO: Using crun runtime for buildkitd daemon. + INFO: running buildkitd server on /run/user/1000/buildkit/buildkitd-0179484509442521.sock + [+] Building 4.3s (5/5) + [+] Building 4.4s (5/5) FINISHED + => [internal] load build definition from Dockerfile 0.0s + => => transferring dockerfile: 131B 0.0s + => [internal] load metadata for docker.io/library/debian:latest 1.2s + => [internal] load .dockerignore 0.0s + => => transferring context: 2B 0.0s + => [1/1] FROM docker.io/library/debian:latest@sha256:fab22df3737 2.9s + => => resolve docker.io/library/debian:latest@sha256:fab22df3737 0.0s + => => sha256:8457fd5474e70835e4482983a5662355d 49.58MB / 49.58MB 2.8s + => exporting to docker image format 3.1s + => => exporting layers 0.0s + => => exporting manifest sha256:9fec77672dfa11e5eb28e3fe9377cd6c 0.0s + => => exporting config sha256:4243e816256d45bb137ff40bafe396da5f 0.0s + => => sending tarball 0.2s + Getting image source signatures + Copying blob 8457fd5474e7 done | + Copying config 46c53efffd done | + Writing manifest to image destination + INFO: Converting OCI image to OCI-SIF format + INFO: Squashing image to single layer + INFO: Writing OCI-SIF image + INFO: Cleaning up. + INFO: Build complete: ./debian.oci.sif + + $ singularity run --oci ./debian.oci.sif + PRETTY_NAME="Debian GNU/Linux 12 (bookworm)" + NAME="Debian GNU/Linux" + VERSION_ID="12" + VERSION="12 (bookworm)" + VERSION_CODENAME=bookworm + ID=debian + HOME_URL="https://www.debian.org/" + SUPPORT_URL="https://www.debian.org/support" + BUG_REPORT_URL="https://bugs.debian.org/" + +The resulting containers can be used with all the :ref:`action commands +` (`exec +`__ +/ `shell +`__ +/ `run +`__) +in the expected way. + +.. code:: console + + $ singularity exec --oci ./debian.oci.sif uname -a + Linux nueve 5.14.0-284.30.1.el9_2.x86_64 #1 SMP PREEMPT_DYNAMIC Fri Aug 25 09:13:12 EDT 2023 x86_64 GNU/Linux + + $ singularity shell --oci ./debian.oci.sif uname + Singularity> uname -a + Linux nueve 5.14.0-284.30.1.el9_2.x86_64 #1 SMP PREEMPT_DYNAMIC Fri Aug 25 09:13:12 EDT 2023 x86_64 GNU/Linux + Singularity> + +.. note:: + + If the `exec + `__ + or `shell + `__ + commands are used, the ``CMD`` / ``ENTRYPOINT`` directives in the Dockerfile + will be ignored. + +The resulting containers also accept command-line arguments, as well as input +that is piped through ``stdin``. The following example demonstrates both: + +.. code:: console + + $ cat ./Dockerfile + FROM debian + + SHELL ["/bin/bash", "-c"] + + RUN apt-get update + RUN apt-get install -y cowsay lolcat + + RUN echo $'#! /bin/bash \n\ + echo from cmdline: $@ | /usr/games/cowsay | /usr/games/lolcat \n\ + sed "s/^/from stdin: /g" | /usr/games/cowsay | /usr/games/lolcat' > /myscript.sh + + RUN chmod +x /myscript.sh + + ENTRYPOINT ["/myscript.sh"] + + $ singularity build --oci ./lolcow.oci.sif ./Dockerfile + INFO: Did not find usable running buildkitd daemon; spawning our own. + INFO: cfg.Root for buildkitd: /home/omer/.local/share/buildkit + INFO: Using crun runtime for buildkitd daemon. + INFO: running buildkitd server on /run/user/1000/buildkit/buildkitd-8961170237105250.sock + [+] Building 15.1s (9/9) + [+] Building 15.2s (9/9) FINISHED + => [internal] load build definition from Dockerfile 0.0s + => => transferring dockerfile: 549B 0.0s + => [internal] load metadata for docker.io/library/debian:latest 0.5s + => [internal] load .dockerignore 0.0s + => => transferring context: 2B 0.0s + => [1/5] FROM docker.io/library/debian:latest@sha256:fab22df3737 1.0s + => => resolve docker.io/library/debian:latest@sha256:fab22df3737 0.0s + => => extracting sha256:8457fd5474e70835e4482983a5662355d892d5f6 1.0s + => [2/5] RUN apt-get update 2.2s + => [3/5] RUN apt-get install -y cowsay lolcat 7.9s + => [4/5] RUN echo $'#! /bin/bash \necho from cmdline: $@ | /usr/ 0.1s + => [5/5] RUN chmod +x /myscript.sh 0.1s + => exporting to docker image format 3.3s + => => exporting layers 2.7s + => => exporting manifest sha256:fc7222347c207c35165ccd2fee562af9 0.0s + => => exporting config sha256:74c5da659e8504e4be283ad6d82774194e 0.0s + => => sending tarball 0.5s + Getting image source signatures + Copying blob 8457fd5474e7 done | + Copying blob 4769fe2f22da done | + Copying blob 173d009c20af done | + Copying blob 7ec86debbe9b done | + Copying blob 491c7ee403c2 done | + Copying config 74b69e878e done | + Writing manifest to image destination + INFO: Converting OCI image to OCI-SIF format + INFO: Squashing image to single layer + INFO: Writing OCI-SIF image + INFO: Cleaning up. + INFO: Build complete: ./lolcow.oci.sif + + $ echo "world" | singularity run --oci ./lolcow.oci.sif hello + _____________________ + < from cmdline: hello > + --------------------- + \ ^__^ + \ (oo)\_______ + (__)\ )\/\ + ||----w | + || || + ___________________ + < from stdin: world > + ------------------- + \ ^__^ + \ (oo)\_______ + (__)\ )\/\ + ||----w | + || || + +{Singularity} uses `buildkit `__ to +build an OCI image from a Dockerfile. It checks if there is a ``buildkitd`` +daemon that is already running on the system (and whose permissions allow access +by the current user), and if so, that daemon is used for the build process. If a +usable ``buildkitd`` daemon is not found, {Singularity} will launch an ephemeral +build daemon of its own, inside a :ref:`user namespace `, +that will be used for the build process and torn down when the build is +complete. This ephemeral build daemon is based on `moby/buildkit +`__, but is embedded within +{Singularity} and runs as part of the same process. + +.. note:: + + Launching the ephemeral ``buildkitd`` daemon requires a system with + :ref:`user namespace support ` as well as ``crun`` / + ``runc`` installed. These are independently required for using + {Singularity}'s :ref:`OCI mode `. See the `Admin Guide + `__ for more + information on these system requirements. + +Additional features +=================== + +Build from Dockerfiles supports many of the same command-line options as regular +(non-OCI-SIF) ``build``, including: + +* ``--build-arg KEY=VAL`` / ``--build-arg-file ``: pass value for + Dockerfile variables at build time (see `Dockerfile ARG documentation + `__). + +* ``--docker-login`` / Docker credential-related environment variables / + ``--authfile``: see the documentation on :ref:`authenticating with Docker/OCI + registries ` and on the :ref:`authfile flag `. + +* ``--arch``: build a container for a different CPU architecture than that of + the running host. + +As an example, if you are running on an ``amd64`` machine, you can run the +following to build a container image for the 64-bit ARM architecure: + +.. code:: console + + $ singularity build --arch arm64 --oci ./alpine.oci.sif ./Dockerfile.alpine + INFO: Did not find usable running buildkitd daemon; spawning our own. + INFO: cfg.Root for buildkitd: /home/omer/.local/share/buildkit + INFO: Using "crun" runtime for buildkitd daemon. + INFO: running buildkitd server on /run/user/1000/buildkit/buildkitd-4747966236261602.sock + [+] Building 0.6s (1/2) + [+] Building 0.7s (5/5) FINISHED + => [internal] load build definition from Dockerfile.alpine 0.0s + => => transferring dockerfile: 142B 0.0s + => [internal] load metadata for docker.io/library/alpine:latest 0.6s + => [internal] load .dockerignore 0.0s + => => transferring context: 2B 0.0s + => CACHED [1/1] FROM docker.io/library/alpine:latest@sha256:eece 0.0s + => => resolve docker.io/library/alpine:latest@sha256:eece025e432 0.0s + => exporting to docker image format 0.0s + => => exporting layers 0.0s + => => exporting manifest sha256:b799c38cef1756bcc55b0684617fda7d 0.0s + => => exporting config sha256:5118299610d621e305a9153753a52e2f9e 0.0s + => => sending tarball 0.0s + Getting image source signatures + Copying blob 579b34f0a95b done | + Copying config 5a13726077 done | + Writing manifest to image destination + INFO: Converting OCI image to OCI-SIF format + INFO: Squashing image to single layer + INFO: Writing OCI-SIF image + INFO: Cleaning up. + INFO: Build complete: ./alpine.oci.sif + +.. note:: + + In order to use Dockerfile directives like ``RUN`` in a cross-architecture + build, you will have to have ``qemu-static`` / ``binfmt_misc`` emulation + installed. See the discussion of :ref:`CPU emulation ` for more + information. + ************* Build options ************* diff --git a/definition_files.rst b/definition_files.rst index 6f9a21a..f4bb31c 100755 --- a/definition_files.rst +++ b/definition_files.rst @@ -106,9 +106,9 @@ Other bootstrap agents Scientific Linux) - :ref:`debootstrap ` (apt-based systems such as Debian and Ubuntu) -- :ref:`oci ` (bundle compliant with OCI Image +- :ref:`oci ` (bundle compliant with OCI Image Specification) -- :ref:`oci-archive ` (tar files +- :ref:`oci-archive ` (tar files obeying the OCI Image Layout Specification) - :ref:`docker-daemon ` (images managed by the locally running docker daemon) diff --git a/oci_runtime.rst b/oci_runtime.rst index 2f10bb4..6015158 100644 --- a/oci_runtime.rst +++ b/oci_runtime.rst @@ -69,6 +69,8 @@ Users are encouraged to employ OCI-mode when their primary use-case for registries. Behavior will more closely match that described for Docker than with {Singularity}'s native runtime. +.. _oci_sysreq: + System Requirements =================== @@ -167,7 +169,7 @@ registry in question, anonymous authentication will be used instead. However, the ``run / shell / exec`` and ``pull`` commands can also use credentials stored in a different file of the user's choosing, by specifying the -``--authfile `` flag. See the :ref:`documentation of the --authfile flag +``--authfile `` flag. See the :ref:`documentation of the authfile flag ` for details on how to create and use custom credential files. .. _oci_compat: diff --git a/registry.rst b/registry.rst index 87a504c..80afc59 100644 --- a/registry.rst +++ b/registry.rst @@ -10,9 +10,8 @@ images. Some registries require credentials to access certain images or even the registry itself. Previously, the only method in {Singularity} to supply credentials to registries was to supply credentials for each command or set environment variables to contain the credentials for a single registry. See -:ref:`Authentication via Interactive Login -` and :ref:`Authentication via Environment -Variables `. +:ref:`Authentication via Interactive Login ` and +:ref:`Authentication via Environment Variables `. Starting with {Singularity} 4.0, users can supply credentials on a per-registry basis with the ``registry`` command. diff --git a/security.rst b/security.rst index cbb507c..e1d69ed 100644 --- a/security.rst +++ b/security.rst @@ -57,6 +57,8 @@ and processes apply. In a default installation, {Singularity} uses a setuid starter binary to perform only the specific tasks needed to setup the container. +.. _setuid_and_userns: + ************************ Setuid & User Namespaces ************************ diff --git a/singularity_and_docker.rst b/singularity_and_docker.rst index 3899d64..6b2614c 100755 --- a/singularity_and_docker.rst +++ b/singularity_and_docker.rst @@ -160,6 +160,8 @@ SIF, and then always run from the SIF file, rather than using Alternatively, if you have signed up for a Docker Hub account, make sure that you authenticate before using ``docker://`` container URIs. +.. _docker_auth: + Authentication / Private Containers =================================== @@ -215,6 +217,8 @@ login. ``~/.docker/config.json``. It cannot read credentials from external Docker credential helpers. +.. _sec:docker_login: + Interactive Login ----------------- @@ -497,6 +501,8 @@ Ubuntu image for a 64-bit ARM system: $ singularity pull --arch arm64 docker://ubuntu +.. _qemu: + CPU emulation ============= @@ -640,7 +646,7 @@ ways: ``--authfile `` flag to the ``build`` command. Note, however, that this will store the relevant credentials unencrypted in the specified file, so appropriate care must be taken concerning the location, ownership, and - permissions of this file. See the :ref:`documentation of the --authfile flag + permissions of this file. See the :ref:`documentation of the authfile flag ` for more information. If you are running the build under your account via the ``--fakeroot``