containerize: pin the Spack version used in a container #21910

alalazo · 2021-02-24T14:35:46Z

This PR permits to specify the url and ref of the Spack instance used in a container recipe simply by expanding the YAML schema as outlined in #20442:

container:
  images:
    os: amazonlinux:2
    spack:
      ref: develop
      resolve_sha: true

The resolve_sha option, if true, verifies the ref by cloning the Spack repository in a temporary directory and transforming any tag or branch name to a commit sha. When this new ability is leveraged an additional "bootstrap" stage is added, which builds an image with Spack setup and ready to install software. The Spack repository to be used can be customized with the url keyword under spack.

Modifications:

Permit to pin the version of Spack, either by branch or tag or sha
Added a few new OSes (centos:8, amazonlinux:2, ubuntu:20.04, alpine:3, cuda:11.2.1)
Permit to print the bootstrap image as a standalone
Add documentation on the new part of the schema
Add unit tests for different use cases

haampie · 2021-03-03T15:16:03Z

Some issues I found:

spack:
  view: true
  specs:
  - sirius +cuda ^cuda@:10.2 ^mpich ^openblas@0.3.10 threads=openmp target=haswell
  container:
    images:
      os: amazonlinux:2
      spack:
        ref: develop
        enforce-sha: true

I'm used to clingo and I'm missing it here :p normally i wouldnt have to specify the openblas version and let the concretizer figure that one out

strip fails because it is applied to powerpc shared libs that apparently ship with the cuda package

Step 16/23 : RUN find -L /opt/view/* -type f -exec readlink -f '{}' \; |     xargs file -i |     grep 'charset=binary' |     grep 'x-executable\|x-archive\|x-sharedlib' |     awk -F: '{print $1}' | xargs strip -s
 ---> Running in 257649bd6d9a
strip: Unable to recognise the format of the input file `/opt/software/linux-amzn2-haswell/gcc-7.3.1/cuda-10.2.89-mao6gttdiesgc5ff4uvjdlk5iqodjubz/nsight-compute-2019.5.0/target/linux-desktop-glibc_2_19_0-ppc64le/libcuda-injection.so'
strip: Unable to recognise the format of the input file `/opt/software/linux-amzn2-haswell/gcc-7.3.1/cuda-10.2.89-mao6gttdiesgc5ff4uvjdlk5iqodjubz/nsight-compute-2019.5.0/target/linux-desktop-glibc_2_19_0-ppc64le/TreeLauncherSubreaper'
strip: Unable to recognise the format of the input file `/opt/software/linux-amzn2-haswell/gcc-7.3.1/cuda-10.2.89-mao6gttdiesgc5ff4uvjdlk5iqodjubz/nsight-compute-2019.5.0/target/linux-desktop-glibc_2_19_0-ppc64le/libInterceptorInjectionTarget.so'
strip: Unable to recognise the format of the input file `/opt/software/linux-amzn2-haswell/gcc-7.3.1/cuda-10.2.89-mao6gttdiesgc5ff4uvjdlk5iqodjubz/nsight-compute-2019.5.0/target/linux-desktop-glibc_2_19_0-ppc64le/libTreeLauncherTargetInjection.so'
strip: Unable to recognise the format of the input file `/opt/software/linux-amzn2-haswell/gcc-7.3.1/cuda-10.2.89-mao6gttdiesgc5ff4uvjdlk5iqodjubz/nsight-compute-2019.5.0/target/linux-desktop-glibc_2_19_0-ppc64le/libnvperf_target.so'
strip: Unable to recognise the format of the input file `/opt/software/linux-amzn2-haswell/gcc-7.3.1/cuda-10.2.89-mao6gttdiesgc5ff4uvjdlk5iqodjubz/nsight-compute-2019.5.0/target/linux-desktop-glibc_2_19_0-ppc64le/libnvperfapi64.so'
strip: Unable to recognise the format of the input file `/opt/software/linux-amzn2-haswell/gcc-7.3.1/cuda-10.2.89-mao6gttdiesgc5ff4uvjdlk5iqodjubz/nsight-compute-2019.5.0/target/linux-desktop-glibc_2_19_0-ppc64le/nv-nsight-cu-cli'
strip: Unable to recognise the format of the input file `/opt/software/linux-amzn2-haswell/gcc-7.3.1/cuda-10.2.89-mao6gttdiesgc5ff4uvjdlk5iqodjubz/nsight-compute-2019.5.0/target/linux-desktop-glibc_2_19_0-ppc64le/libnvperf_host.so'
strip: Unable to recognise the format of the input file `/opt/software/linux-amzn2-haswell/gcc-7.3.1/cuda-10.2.89-mao6gttdiesgc5ff4uvjdlk5iqodjubz/nsight-compute-2019.5.0/target/linux-desktop-glibc_2_19_0-ppc64le/libNvSwCounterApi.so'
strip: Unable to recognise the format of the input file `/opt/software/linux-amzn2-haswell/gcc-7.3.1/cuda-10.2.89-mao6gttdiesgc5ff4uvjdlk5iqodjubz/nsight-compute-2019.5.0/target/linux-desktop-glibc_2_19_0-ppc64le/libTreeLauncherPlaceholder.so'
strip: Unable to recognise the format of the input file `/opt/software/linux-amzn2-haswell/gcc-7.3.1/cuda-10.2.89-mao6gttdiesgc5ff4uvjdlk5iqodjubz/nsight-compute-2019.5.0/target/linux-desktop-glibc_2_19_0-ppc64le/TreeLauncherTargetLdPreloadHelper'
strip: Unable to recognise the format of the input file `/opt/software/linux-amzn2-haswell/gcc-7.3.1/cuda-10.2.89-mao6gttdiesgc5ff4uvjdlk5iqodjubz/nsight-compute-2019.5.0/target/linux-desktop-glibc_2_19_0-ppc64le/libTreeLauncherTargetUpdatePreloadInjection.so'

setting LD_LIBRARY_PATH makes system executables fail to work:

bash-4.2# yum install libgomp
There was a problem importing one of the Python modules
required to run yum. The error leading to this problem was:
   /opt/view/lib/liblzma.so.5: version `XZ_5.1.2alpha' not found (required by /lib64/librpmio.so.3)
Please install a package which provides this module, or
verify that the module is installed correctly.
It's possible that the above module doesn't match the
current version of Python, which is:
2.7.18 (default, Feb 18 2021, 06:07:59) 
[GCC 7.3.1 20180712 (Red Hat 7.3.1-12)]
If you cannot solve this problem yourself, please go to 
the yum faq at:
  http://yum.baseurl.org/wiki/Faq

alalazo · 2021-03-03T15:18:11Z

Thanks for test-driving the PR @haampie

alalazo · 2021-03-03T16:58:01Z

@victorusu

tgamblin

A few minor requests and some arg tweaks, and one larger question about bootstrapping images.

lib/spack/docs/containers.rst

lib/spack/llnl/util/filesystem.py

share/spack/templates/container/amazonlinux_2.dockerfile

lib/spack/spack/container/images.json

lib/spack/spack/schema/container.py

share/spack/templates/container/Dockerfile

share/spack/templates/container/centos_7.dockerfile

alalazo · 2021-05-25T08:30:26Z

@tgamblin This is ready for another review

alalazo · 2021-06-01T07:47:11Z

@tgamblin ping

alalazo · 2021-06-30T09:40:20Z

@tgamblin Do you have some time for a second review?

kniec · 2021-09-09T16:08:39Z

I updated the containers-on-pcluster workshop to reflect that. Do you also are going to kick out the CUDA binary problem from above?

alalazo · 2021-09-09T16:10:11Z

@kniec Maybe in a smaller, follow up PR? Would that work?

kniec · 2021-09-09T16:14:36Z

@alalazo sure thing - I just kick that part out with

spack containerize \
  | sed -e '/RUN find -L /,+4d' > Dockerfile

alalazo · 2021-09-09T19:27:03Z

@vsoch With respect to Ubuntu 20.04 you can try:

spack:
  specs:
  - zlib

  container:
    images:
      os: ubuntu:20.04
      spack:
        ref: develop

with this PR

vsoch · 2021-09-09T19:45:26Z

will do!

vsoch · 2021-09-09T20:10:52Z

Okay testing for 20.04! The Dockerfile looks great, builds without issue:

FROM ubuntu:20.04 as bootstrap

ENV SPACK_ROOT=/opt/spack \
    CURRENTLY_BUILDING_DOCKER_IMAGE=1 \
    container=docker

ENV DEBIAN_FRONTEND=noninteractive   \
    LANGUAGE=en_US.UTF-8 \
    LANG=en_US.UTF-8 \
    LC_ALL=en_US.UTF-8

RUN apt-get -yqq update \
 && apt-get -yqq install --no-install-recommends \
        build-essential \
        ca-certificates \
        curl \
        file \
        g++ \
        gcc \
        gfortran \
        git \
        gnupg2 \
        iproute2 \
        locales \
        lua-posix \
        make \
        python3 \
        python3-pip \
        python3-setuptools \
        unzip \
 && locale-gen en_US.UTF-8 \
 && pip3 install boto3 \
 && rm -rf /var/lib/apt/lists/*

RUN mkdir $SPACK_ROOT && cd $SPACK_ROOT && \
    git clone https://github.com/spack/spack.git . && git checkout develop  && \
    mkdir -p $SPACK_ROOT/opt/spack

RUN ln -s $SPACK_ROOT/share/spack/docker/entrypoint.bash \
          /usr/local/bin/docker-shell \
 && ln -s $SPACK_ROOT/share/spack/docker/entrypoint.bash \
          /usr/local/bin/interactive-shell \
 && ln -s $SPACK_ROOT/share/spack/docker/entrypoint.bash \
          /usr/local/bin/spack-env

RUN mkdir -p /root/.spack \
 && cp $SPACK_ROOT/share/spack/docker/modules.yaml \
        /root/.spack/modules.yaml \
 && rm -rf /root/*.* /run/nologin $SPACK_ROOT/.git

# [WORKAROUND]
# https://superuser.com/questions/1241548/
#     xubuntu-16-04-ttyname-failed-inappropriate-ioctl-for-device#1253889
RUN [ -f ~/.profile ]                                               \
 && sed -i 's/mesg n/( tty -s \\&\\& mesg n || true )/g' ~/.profile \
 || true


WORKDIR /root
SHELL ["docker-shell"]

# Creates the package cache
RUN spack spec hdf5+mpi

ENTRYPOINT ["/bin/bash", "/opt/spack/share/spack/docker/entrypoint.bash"]
CMD ["interactive-shell"]

# Build stage with Spack pre-installed and ready to be used
FROM bootstrap as builder


# What we want to install and how we want to install it
# is specified in a manifest file (spack.yaml)
RUN mkdir /opt/spack-environment \
&&  (echo "spack:" \
&&   echo "  specs:" \
&&   echo "  - zlib" \
&&   echo "  concretization: together" \
&&   echo "  config:" \
&&   echo "    install_tree: /opt/software" \
&&   echo "  view: /opt/view") > /opt/spack-environment/spack.yaml

# Install the software, remove unnecessary deps
RUN cd /opt/spack-environment && \
    spack env activate . && \
    spack install --fail-fast && \
    spack gc -y

# Strip all the binaries
RUN find -L /opt/view/* -type f -exec readlink -f '{}' \; | \
    xargs file -i | \
    grep 'charset=binary' | \
    grep 'x-executable\|x-archive\|x-sharedlib' | \
    awk -F: '{print $1}' | xargs strip -s

# Modifications to the environment that are necessary to run
RUN cd /opt/spack-environment && \
    spack env activate --sh -d . >> /etc/profile.d/z10_spack_environment.sh

# Bare OS image to run the installed executables
FROM ubuntu:20.04

COPY --from=builder /opt/spack-environment /opt/spack-environment
COPY --from=builder /opt/software /opt/software
COPY --from=builder /opt/view /opt/view
COPY --from=builder /etc/profile.d/z10_spack_environment.sh /etc/profile.d/z10_spack_environment.sh

ENTRYPOINT ["/bin/bash", "--rcfile", "/etc/profile", "-l"]

Shelling in is very straight forward:

$ docker run -it test

I think my one question is if spack should be already on the path? As a user I wouldn't know what to do after shelling in and not seeing it.

root@a6cc2bd49210:/# which spack

alalazo · 2021-09-09T20:26:28Z

@vsoch The Dockerfile, by default, is a 3 stage build where Spack is in the first two. The last stage is just the software required in the environment. You can:

$ spack containerize --last-stage build

If you want to stop before the last stage.

vsoch · 2021-09-09T20:29:48Z

ah okay, that does make sense. Sorry didn't look closely enough!

vsoch · 2021-09-14T15:39:08Z

@davidbeckingsale see this PR here for ubuntu 20.04 support

kniec · 2021-09-15T07:44:46Z

@alalazo docker build defines the stage to build for with --target - I am voting to use the same option.

alalazo · 2021-09-22T14:21:35Z

@kniec I think "target" would be confusing for people using Spack since in specs it's used to identify the target microarchitecture.

kniec · 2021-09-22T14:26:14Z

yeah, that is true.

This commit permits to specify the url and ref of the Spack instance to be used in a container recipe. If this is done an initial bootstrap stage is added, which builds an image with Spack setup and ready to install software. A few new OSes have been added: centos:8, amazonlinux:2, ubuntu:20.04, alpine:3.

alalazo added feature containers labels Feb 24, 2021

alalazo force-pushed the features/containerize_fix_spack_sha branch 6 times, most recently from 2c246d7 to 3355414 Compare March 3, 2021 15:10

alalazo force-pushed the features/containerize_fix_spack_sha branch from 3355414 to e4ad7dc Compare March 3, 2021 16:17

alalazo marked this pull request as ready for review March 3, 2021 16:57

alalazo added this to In progress in Spack 0.17.0 Release via automation Mar 3, 2021

alalazo force-pushed the features/containerize_fix_spack_sha branch from e4ad7dc to 690e8a5 Compare March 3, 2021 18:51

tgamblin requested changes Mar 16, 2021

View reviewed changes

Spack 0.17.0 Release automation moved this from In progress to Review in progress Mar 16, 2021

alalazo force-pushed the features/containerize_fix_spack_sha branch from 690e8a5 to 313b581 Compare April 28, 2021 15:54

alalazo mentioned this pull request Apr 29, 2021

Docker: spack/ubuntu 20.04 container recipe #23322

Closed

s-sajid-ali mentioned this pull request Jun 10, 2021

Installation issue: intel-oneapi-tbb in a container #24227

Open

alalazo mentioned this pull request Jun 11, 2021

Add New Build Containers Workflow #24257

Merged

alalazo force-pushed the features/containerize_fix_spack_sha branch from ab4533c to 2517ceb Compare June 18, 2021 09:35

alalazo force-pushed the features/containerize_fix_spack_sha branch from 2517ceb to 1b0ff7a Compare July 28, 2021 07:55

spackbot-app bot added commands docker documentation Improvements or additions to documentation tests General test capability(ies) labels Jul 28, 2021

alalazo added 13 commits October 1, 2021 15:15

Permit to print only the bootstrap phase

8aafdd4

Added documentation

31350f2

Add unit tests

1b89093

Improve wording in docs and fix typos

440e39d

Rename enforce-sha to resolve_sha

9028038

Use a single option to select stages

c047557

Make render_phase a named tuple

6eb260c

Update completion file

00b2f17

Trim a few unneeded packages from the bootstrap stage

ca8581a

Automatically fix import style

ec0c1a0

Fix issue with epel install on amazonlinux

b40f5a6

Remove support for CentOS 6

6942705

alalazo force-pushed the features/containerize_fix_spack_sha branch from d78d65b to 6942705 Compare October 1, 2021 13:36

tgamblin approved these changes Oct 25, 2021

View reviewed changes

tgamblin merged commit 6063600 into spack:develop Oct 25, 2021

Spack 0.17.0 Release automation moved this from Review in progress to Done Oct 25, 2021

alalazo deleted the features/containerize_fix_spack_sha branch October 25, 2021 20:29

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

containerize: pin the Spack version used in a container #21910

containerize: pin the Spack version used in a container #21910

alalazo commented Feb 24, 2021 •

edited

Loading

haampie commented Mar 3, 2021 •

edited

Loading

alalazo commented Mar 3, 2021

alalazo commented Mar 3, 2021

tgamblin left a comment

alalazo commented May 25, 2021

alalazo commented Jun 1, 2021

alalazo commented Jun 30, 2021

kniec commented Sep 9, 2021

alalazo commented Sep 9, 2021

kniec commented Sep 9, 2021 •

edited

Loading

alalazo commented Sep 9, 2021

vsoch commented Sep 9, 2021

vsoch commented Sep 9, 2021 •

edited

Loading

alalazo commented Sep 9, 2021

vsoch commented Sep 9, 2021

vsoch commented Sep 14, 2021

kniec commented Sep 15, 2021

alalazo commented Sep 22, 2021

kniec commented Sep 22, 2021

containerize: pin the Spack version used in a container #21910

containerize: pin the Spack version used in a container #21910

Conversation

alalazo commented Feb 24, 2021 • edited Loading

haampie commented Mar 3, 2021 • edited Loading

alalazo commented Mar 3, 2021

alalazo commented Mar 3, 2021

tgamblin left a comment

Choose a reason for hiding this comment

alalazo commented May 25, 2021

alalazo commented Jun 1, 2021

alalazo commented Jun 30, 2021

kniec commented Sep 9, 2021

alalazo commented Sep 9, 2021

kniec commented Sep 9, 2021 • edited Loading

alalazo commented Sep 9, 2021

vsoch commented Sep 9, 2021

vsoch commented Sep 9, 2021 • edited Loading

alalazo commented Sep 9, 2021

vsoch commented Sep 9, 2021

vsoch commented Sep 14, 2021

kniec commented Sep 15, 2021

alalazo commented Sep 22, 2021

kniec commented Sep 22, 2021

alalazo commented Feb 24, 2021 •

edited

Loading

haampie commented Mar 3, 2021 •

edited

Loading

kniec commented Sep 9, 2021 •

edited

Loading

vsoch commented Sep 9, 2021 •

edited

Loading