diff --git a/llvm/docs/Docker.rst b/llvm/docs/Docker.rst index e606e1b71a2c0..7862c5ab7a3de 100644 --- a/llvm/docs/Docker.rst +++ b/llvm/docs/Docker.rst @@ -53,24 +53,15 @@ serve as a basis for anyone who wants to create their own Docker image with LLVM components, compiled from sources. The sources are checked out from the upstream svn repository when building the image. -Inside each subfolder we host Dockerfiles for two images: - -- ``build/`` image is used to compile LLVM, it installs a system compiler and all - build dependencies of LLVM. After the build process is finished, the build - image will have an archive with compiled components at ``/tmp/clang.tar.gz``. -- ``release/`` image usually only contains LLVM components, compiled by the - ``build/`` image, and also libstdc++ and binutils to make image minimally - useful for C++ development. The assumption is that you usually want clang to - be one of the provided components. - -To build both of those images, use ``build_docker_image.sh`` script. -It will checkout LLVM sources and build clang in the ``build`` container, copy results -of the build to the local filesystem and then build the ``release`` container using -those. The ``build_docker_image.sh`` accepts a list of LLVM repositories to -checkout, and arguments for CMake invocation. +The resulting image contains only the requested LLVM components and a few extra +packages to make the image minimally useful for C++ development, e.g. libstdc++ +and binutils. + +The interface to run the build is ``build_docker_image.sh`` script. It accepts a +list of LLVM repositories to checkout and arguments for CMake invocation. If you want to write your own docker image, start with an ``example/`` subfolder. -It provides incomplete Dockerfiles with (very few) FIXMEs explaining the steps +It provides an incomplete Dockerfile with (very few) FIXMEs explaining the steps you need to take in order to make your Dockerfiles functional. Usage @@ -110,10 +101,10 @@ this command will do that: -DBOOTSTRAP_CMAKE_BUILD_TYPE=Release \ -DCLANG_ENABLE_BOOTSTRAP=ON -DCLANG_BOOTSTRAP_TARGETS="install-clang;install-clang-headers" -This will produce two images, a release image ``clang-debian8:staging`` and a -build image ``clang-debian8-build:staging`` from the latest upstream revision. -After the image is built you can run bash inside a container based on your -image like this: +This will produce a new image ``clang-debian8:staging`` from the latest +upstream revision. +After the image is built you can run bash inside a container based on your image +like this: .. code-block:: bash @@ -181,19 +172,14 @@ debian8-based image using the latest ``google/stable`` sources for you: Minimizing docker image size ============================ -Due to Docker restrictions we use two images (i.e., build and release folders) -for the release image to be as small as possible. It's much easier to achieve -that using two images, because Docker would store a filesystem layer for each -command in the Dockerfile, i.e. if you install some packages in one command, -then remove those in a separate command, the size of the resulting image will -still be proportinal to the size of an image with installed packages. -Therefore, we strive to provide a very simple release image which only copies -compiled clang and does not do anything else. - -Docker 1.13 added a ``--squash`` flag that allows to flatten the layers of the -image, i.e. remove the parts that were actually deleted. That is an easier way -to produce the smallest images possible by using just a single image. We do not -use it because as of today the flag is in experimental stage and not everyone -may have the latest docker version available. When the flag is out of -experimental stage, we should investigate replacing two images approach with -just a single image, built using ``--squash`` flag. +Due to how Docker's filesystem works, all intermediate writes are persisted in +the resulting image, even if they are removed in the following commands. +To minimize the resulting image size we use `multi-stage Docker builds +`_. +Internally Docker builds two images. The first image does all the work: installs +build dependencies, checks out LLVM source code, compiles LLVM, etc. +The first image is only used during build and does not have a descriptive name, +i.e. it is only accessible via the hash value after the build is finished. +The second image is our resulting image. It contains only the built binaries +and not any build dependencies. It is also accessible via a descriptive name +(specified by -d and -t flags). diff --git a/llvm/utils/docker/build_docker_image.sh b/llvm/utils/docker/build_docker_image.sh index 9b0ba46fe4b34..0e129636ca40e 100755 --- a/llvm/utils/docker/build_docker_image.sh +++ b/llvm/utils/docker/build_docker_image.sh @@ -163,19 +163,9 @@ if [ "$DOCKER_TAG" != "" ]; then DOCKER_TAG=":$DOCKER_TAG" fi -echo "Building from $IMAGE_SOURCE" -echo "Building $DOCKER_REPOSITORY-build$DOCKER_TAG" -docker build -t "$DOCKER_REPOSITORY-build$DOCKER_TAG" \ +echo "Building ${DOCKER_REPOSITORY}${DOCKER_TAG} from $IMAGE_SOURCE" +docker build -t "${DOCKER_REPOSITORY}${DOCKER_TAG}" \ --build-arg "buildscript_args=$BUILDSCRIPT_ARGS" \ - -f "$BUILD_DIR/$IMAGE_SOURCE/build/Dockerfile" \ + -f "$BUILD_DIR/$IMAGE_SOURCE/Dockerfile" \ "$BUILD_DIR" - -echo "Copying clang installation to release image sources" -docker run -v "$BUILD_DIR/$IMAGE_SOURCE:/workspace" "$DOCKER_REPOSITORY-build$DOCKER_TAG" \ - cp /tmp/clang.tar.gz /workspace/release - -echo "Building release image" -docker build -t "${DOCKER_REPOSITORY}${DOCKER_TAG}" \ - "$BUILD_DIR/$IMAGE_SOURCE/release" - echo "Done" diff --git a/llvm/utils/docker/debian8/build/Dockerfile b/llvm/utils/docker/debian8/Dockerfile similarity index 80% rename from llvm/utils/docker/debian8/build/Dockerfile rename to llvm/utils/docker/debian8/Dockerfile index 3f42f2ad5916a..fd3bf9f49f06e 100644 --- a/llvm/utils/docker/debian8/build/Dockerfile +++ b/llvm/utils/docker/debian8/Dockerfile @@ -6,22 +6,18 @@ # License. See LICENSE.TXT for details. # #===----------------------------------------------------------------------===// -# Produces an image that compiles and archives clang, based on debian8. -FROM launcher.gcr.io/google/debian8:latest - +# Stage 1. Check out LLVM source code and run the build. +FROM launcher.gcr.io/google/debian8:latest as builder LABEL maintainer "LLVM Developers" - # Install build dependencies of llvm. # First, Update the apt's source list and include the sources of the packages. RUN grep deb /etc/apt/sources.list | \ sed 's/^deb/deb-src /g' >> /etc/apt/sources.list - # Install compiler, python and subversion. RUN apt-get update && \ apt-get install -y --no-install-recommends ca-certificates gnupg \ build-essential python wget subversion unzip && \ rm -rf /var/lib/apt/lists/* - # Install a newer ninja release. It seems the older version in the debian repos # randomly crashes when compiling llvm. RUN wget "https://github.com/ninja-build/ninja/releases/download/v1.8.2/ninja-linux.zip" && \ @@ -29,10 +25,8 @@ RUN wget "https://github.com/ninja-build/ninja/releases/download/v1.8.2/ninja-li | sha256sum -c && \ unzip ninja-linux.zip -d /usr/local/bin && \ rm ninja-linux.zip - # Import public key required for verifying signature of cmake download. RUN gpg --keyserver hkp://pgp.mit.edu --recv 0x2D2CEF1034921684 - # Download, verify and install cmake version that can compile clang into /usr/local. # (Version in debian8 repos is is too old) RUN mkdir /tmp/cmake-install && cd /tmp/cmake-install && \ @@ -47,9 +41,18 @@ RUN mkdir /tmp/cmake-install && cd /tmp/cmake-install && \ ADD checksums /tmp/checksums ADD scripts /tmp/scripts - # Arguments passed to build_install_clang.sh. ARG buildscript_args - -# Run the build. Results of the build will be available as /tmp/clang.tar.gz. +# Run the build. Results of the build will be available at /tmp/clang-install/. RUN /tmp/scripts/build_install_llvm.sh ${buildscript_args} + + +# Stage 2. Produce a minimal release image with build results. +FROM launcher.gcr.io/google/debian8:latest +LABEL maintainer "LLVM Developers" +# Install packages for minimal useful image. +RUN apt-get update && \ + apt-get install -y --no-install-recommends libstdc++-4.9-dev binutils && \ + rm -rf /var/lib/apt/lists/* +# Copy build results of stage 1 to /usr/local. +COPY --from=builder /tmp/clang-install/ /usr/local/ diff --git a/llvm/utils/docker/debian8/release/Dockerfile b/llvm/utils/docker/debian8/release/Dockerfile deleted file mode 100644 index 3a44a7d411667..0000000000000 --- a/llvm/utils/docker/debian8/release/Dockerfile +++ /dev/null @@ -1,21 +0,0 @@ -#===- llvm/utils/docker/debian8/release/Dockerfile -----------------------===// -# -# The LLVM Compiler Infrastructure -# -# This file is distributed under the University of Illinois Open Source -# License. See LICENSE.TXT for details. -# -#===----------------------------------------------------------------------===// -# A release image, containing clang installation, produced by the 'build/' image -# and adding libstdc++ and binutils. -FROM launcher.gcr.io/google/debian8:latest - -LABEL maintainer "LLVM Developers" - -# Install packages for minimal useful image. -RUN apt-get update && \ - apt-get install -y --no-install-recommends libstdc++-4.9-dev binutils && \ - rm -rf /var/lib/apt/lists/* - -# Unpack clang installation into this image. -ADD clang.tar.gz /usr/local/ diff --git a/llvm/utils/docker/example/build/Dockerfile b/llvm/utils/docker/example/Dockerfile similarity index 58% rename from llvm/utils/docker/example/build/Dockerfile rename to llvm/utils/docker/example/Dockerfile index be077f59f48a4..bb42a4df6bfcc 100644 --- a/llvm/utils/docker/example/build/Dockerfile +++ b/llvm/utils/docker/example/Dockerfile @@ -9,20 +9,29 @@ # This is an example Dockerfile to build an image that compiles clang. # Replace FIXMEs to prepare your own image. +# Stage 1. Check out LLVM source code and run the build. # FIXME: Replace 'ubuntu' with your base image -FROM ubuntu - +FROM ubuntu as builder # FIXME: Change maintainer name LABEL maintainer "Maintainer " - -# FIXME: Install llvm/clang build dependencies. Including compiler to +# FIXME: Install llvm/clang build dependencies here. Including compiler to # build stage1, cmake, subversion, ninja, etc. ADD checksums /tmp/checksums ADD scripts /tmp/scripts - # Arguments passed to build_install_clang.sh. ARG buildscript_args - -# Run the build. Results of the build will be available as /tmp/clang.tar.gz. +# Run the build. Results of the build will be available as /tmp/clang-install. RUN /tmp/scripts/build_install_llvm.sh ${buildscript_args} + + +# Stage 2. Produce a minimal release image with build results. +# FIXME: Replace 'ubuntu' with your base image. +FROM ubuntu +# FIXME: Change maintainer name. +LABEL maintainer "Maintainer " +# FIXME: Install all packages you want to have in your release container. +# A minimal useful installation should include at least libstdc++ and binutils. + +# Copy build results of stage 1 to /usr/local. +COPY --from=builder /tmp/clang-install/ /usr/local/ diff --git a/llvm/utils/docker/example/release/Dockerfile b/llvm/utils/docker/example/release/Dockerfile deleted file mode 100644 index b088ad885ac5a..0000000000000 --- a/llvm/utils/docker/example/release/Dockerfile +++ /dev/null @@ -1,24 +0,0 @@ -#===- llvm/utils/docker/example/release/Dockerfile -----------------------===// -# -# The LLVM Compiler Infrastructure -# -# This file is distributed under the University of Illinois Open Source -# License. See LICENSE.TXT for details. -# -#===----------------------------------------------------------------------===// -# An image that unpacks a clang installation, compiled by the 'build/' -# container. -# Replace FIXMEs to prepare your own image. - -# FIXME: Replace 'ubuntu' with your base image. -FROM ubuntu - -# FIXME: Change maintainer name. -LABEL maintainer "Maintainer " - -# FIXME: Install all packages you want to have in your release container. -# A minimal useful installation must include libstdc++ and binutils. - -# Unpack clang installation into this container. -# It is copied to this directory by build_docker_image.sh script. -ADD clang.tar.gz /usr/local/ diff --git a/llvm/utils/docker/nvidia-cuda/build/Dockerfile b/llvm/utils/docker/nvidia-cuda/Dockerfile similarity index 60% rename from llvm/utils/docker/nvidia-cuda/build/Dockerfile rename to llvm/utils/docker/nvidia-cuda/Dockerfile index cd353a2578bde..6a354c88c7c9b 100644 --- a/llvm/utils/docker/nvidia-cuda/build/Dockerfile +++ b/llvm/utils/docker/nvidia-cuda/Dockerfile @@ -6,26 +6,26 @@ # License. See LICENSE.TXT for details. # #===----------------------------------------------------------------------===// -# Produces an image that compiles and archives clang, based on nvidia/cuda -# image. -FROM nvidia/cuda:8.0-devel - +# Stage 1. Check out LLVM source code and run the build. +FROM nvidia/cuda:8.0-devel as builder LABEL maintainer "LLVM Developers" - -# Arguments to pass to build_install_clang.sh. -ARG buildscript_args - # Install llvm build dependencies. RUN apt-get update && \ apt-get install -y --no-install-recommends ca-certificates cmake python \ - subversion ninja-build && \ + subversion ninja-build && \ rm -rf /var/lib/apt/lists/* ADD checksums /tmp/checksums ADD scripts /tmp/scripts - # Arguments passed to build_install_clang.sh. ARG buildscript_args - -# Run the build. Results of the build will be available as /tmp/clang.tar.gz. +# Run the build. Results of the build will be available at /tmp/clang-install/. RUN /tmp/scripts/build_install_llvm.sh ${buildscript_args} + + +# Stage 2. Produce a minimal release image with build results. +FROM nvidia/cuda:8.0-devel +LABEL maintainer "LLVM Developers" +# Copy clang installation into this container. +COPY --from=builder /tmp/clang-install/ /usr/local/ +# C++ standard library and binutils are already included in the base package. diff --git a/llvm/utils/docker/nvidia-cuda/release/Dockerfile b/llvm/utils/docker/nvidia-cuda/release/Dockerfile deleted file mode 100644 index a30d7d7e91ee1..0000000000000 --- a/llvm/utils/docker/nvidia-cuda/release/Dockerfile +++ /dev/null @@ -1,23 +0,0 @@ -#===- llvm/utils/docker/nvidia-cuda/release/Dockerfile -------------------===// -# -# The LLVM Compiler Infrastructure -# -# This file is distributed under the University of Illinois Open Source -# License. See LICENSE.TXT for details. -# -#===----------------------------------------------------------------------===// -# This is an example Dockerfile that copies a clang installation, compiled -# by the 'build/' container into a fresh docker image to get a container of -# minimal size. -# Replace FIXMEs to prepare a new Dockerfile. - -# FIXME: Replace 'ubuntu' with your base image. -FROM nvidia/cuda:8.0-devel - -# FIXME: Change maintainer name. -LABEL maintainer "LLVM Developers" - -# Unpack clang installation into this container. -ADD clang.tar.gz /usr/local/ - -# C++ standard library and binutils are already included in the base package. diff --git a/llvm/utils/docker/scripts/build_install_llvm.sh b/llvm/utils/docker/scripts/build_install_llvm.sh index 67e537fad2783..7e5ac1e580722 100755 --- a/llvm/utils/docker/scripts/build_install_llvm.sh +++ b/llvm/utils/docker/scripts/build_install_llvm.sh @@ -16,8 +16,8 @@ Usage: build_install_llvm.sh [options] -- [cmake-args] Checkout svn sources and run cmake with the specified arguments. Used inside docker container. -Passes additional -DCMAKE_INSTALL_PREFIX and archives the contents of -the directory to /tmp/clang.tar.gz. +Passes additional -DCMAKE_INSTALL_PREFIX and puts the build results into +/tmp/clang-install/ directory. Available options: -h|--help show this help message @@ -244,12 +244,7 @@ ninja $CMAKE_INSTALL_TARGETS popd -# Pack the installed clang into an archive. -echo "Archiving clang installation to /tmp/clang.tar.gz" -cd "$CLANG_INSTALL_DIR" -tar -czf /tmp/clang.tar.gz * - # Cleanup. -rm -rf "$CLANG_BUILD_DIR" "$CLANG_INSTALL_DIR" +rm -rf "$CLANG_BUILD_DIR" echo "Done"