From 7ef30c4363eb6f304fd86ec239d7af0c4ea895ce Mon Sep 17 00:00:00 2001 From: David Sanchez Date: Mon, 29 Apr 2024 19:55:40 +0200 Subject: [PATCH 01/31] conan runner docs added --- reference.rst | 1 + reference/runners.rst | 17 + reference/runners/docker.rst | 717 +++++++++++++++++++++++++++++++++++ 3 files changed, 735 insertions(+) create mode 100644 reference/runners.rst create mode 100644 reference/runners/docker.rst diff --git a/reference.rst b/reference.rst index 9c303c0f857..8ab96ceaa07 100644 --- a/reference.rst +++ b/reference.rst @@ -17,3 +17,4 @@ Reference reference/environment reference/binary_model reference/conan_server + reference/runners diff --git a/reference/runners.rst b/reference/runners.rst new file mode 100644 index 00000000000..060c853f491 --- /dev/null +++ b/reference/runners.rst @@ -0,0 +1,17 @@ +.. _reference_runners: + +Runners +======= + +Runners are the way to run conan remotely and transparently just by modifying the host profile. There are three requirements to be able to use the requirements: + +- Installing a version of conan with runner dependencies ``pip install conan[runners]``. +- Install the tools to run each of the runners (``docker``). +- Add to the ``host profile`` the ``[runner]`` section defined in the documentation of each runner. + +Runners: + +.. toctree:: + :maxdepth: 1 + + runners/docker diff --git a/reference/runners/docker.rst b/reference/runners/docker.rst new file mode 100644 index 00000000000..c131da6da7d --- /dev/null +++ b/reference/runners/docker.rst @@ -0,0 +1,717 @@ +.. _reference_runners_docker: + +Docker runner +============= + +How to use a docker runner +-------------------------- + +To run conan inside a docker container you need to define a ``[runner]`` section in your host profile file using the following fields. + +- ``type`` **(mandatory)**: define the ruuner we want to use, in that case ``docker``. +- ``dockerfile`` **(optional, default None)**: define the Dockerfile absolute path in case you want to build a docker image. +- ``image`` **(optional, default conan-runner-default)**: docker image name you want to download from a docker registry or the name of the builded image in case you define a dockerfile path. +- ``cache`` **(optional, default clean)**: how docker container uses (or not) the host cache. + + - ``clean``: use an empty cache. + - ``copy``: copy the host cache inside the container using the :ref:`conan cache save/restore` command. + - ``shared``: mount the host conan cache as a shared volume. + +- ``remove`` **(optional, default false)**: ``true`` or ``false``. Remove the container after running the conan command. +- ``configfile`` **(optional, default None)**: configuration file absolute path with extra parameters (see **extra configuration** section for more info). + +Extra configuration +------------------- + +If you need more controll over the container build and execution, you can define more parameters inside a ``configfile`` yaml. + +.. code-block:: yaml + + image: image_name # The image to build or run. + build: + dockerfile: /dockerfile/path # Dockerfile path. + build_context: /build/context/path # Path within the build context to the Dockerfile. + build_args: # A dictionary of build arguments + foo: bar + cacheFrom: # A list of images used for build cache resolution + - image_1 + run: + name: container_name # The name for this container. + containerEnv: # Environment variables to set inside the container. + env_var_1: env_valie + containerUser: user_name # Username or UID to run commands as inside the container. + privileged: False # Run as privileged + capAdd: # Add kernel capabilities. + - SYS_ADMIN + - MKNOD + securityOpt: # A list of string values to customize labels for MLS systems, such as SELinux. + - opt_1 + mount: # A dictionary to configure volumes mounted inside the container. + /home/user1/: # The host path or a volume name + bind: /mnt/vol2 # The path to mount the volume inside the container + mode: rw # rw to mount the volume read/write, or ro to mount it read-only. + +Basic example +------------- + +In this example we are going to see how to compile zlib 1.3.1 inside docker. Let’s create two profiles and a Dockerfile inside our project folder. + +.. code-block:: bash + + $ cd + $ tree + . + ├── Dockerfile + ├── docker_example_build + └── docker_example_host + +``docker_example_host`` profile + +.. code-block:: text + + [settings] + arch=x86_64 + build_type=Release + compiler=gcc + compiler.cppstd=gnu17 + compiler.libcxx=libstdc++11 + compiler.version=11 + os=Linux + [runner] + type=docker + dockerfile= + cache=copy + remove=true + +``docker_example_build`` profile + +.. code-block:: text + + [settings] + arch=x86_64 + build_type=Release + compiler=gcc + compiler.cppstd=gnu17 + compiler.libcxx=libstdc++11 + compiler.version=11 + os=Linux + +``Dockerfile`` + +.. code-block:: docker + + FROM ubuntu:22.04 + RUN apt-get update \ + && DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends \ + build-essential \ + cmake \ + python3 \ + python3-pip \ + python3-venv \ + && rm -rf /var/lib/apt/lists/* + RUN pip install conan + +In this example we are going to start from a totally clean docker, without containers or images. In addition, we are going to have the conan cache also completely empty. + +.. code-block:: bash + + $ conan list "*:*" + Found 0 pkg/version recipes matching * in local cache + + $ docker ps --all + CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES + + $ docker images + REPOSITORY TAG IMAGE ID CREATED SIZE + + +Now, we are going to clone and build zlib from conan-center-index and create it using our new runner definition. + +.. code-block:: bash + + $ git clone https://github.com/conan-io/conan-center-index.git + $ conan create ./conan-center-index/recipes/zlib/all --version 1.3.1 -pr:h /docker_example_host -pr:b /docker_example_build + +If we split and analyze the command output, we can see what is happening and where the commands are being executed. + +1. Standard conan execution. + +.. code-block:: bash + + ======== Exporting recipe to the cache ======== + zlib/1.3.1: Exporting package recipe: /conan-center-index/recipes/zlib/all/conanfile.py + zlib/1.3.1: exports: File 'conandata.yml' found. Exporting it... + zlib/1.3.1: Calling export_sources() + zlib/1.3.1: Copied 1 '.py' file: conanfile.py + zlib/1.3.1: Copied 1 '.yml' file: conandata.yml + zlib/1.3.1: Copied 1 '.patch' file: 0001-fix-cmake.patch + zlib/1.3.1: Exported to cache folder: /Users/conan/.conan2/p/zlib95420566fc0dd/e + zlib/1.3.1: Exported: zlib/1.3.1#e20364c96c45455608a72543f3a53133 (2024-04-29 17:03:44 UTC) + + ======== Input profiles ======== + Profile host: + [settings] + arch=x86_64 + build_type=Release + compiler=gcc + compiler.cppstd=gnu17 + compiler.libcxx=libstdc++11 + compiler.version=11 + os=Linux + + Profile build: + [settings] + arch=x86_64 + build_type=Release + compiler=gcc + compiler.cppstd=gnu17 + compiler.libcxx=libstdc++11 + compiler.version=11 + os=Linux + +2. Build docker image + +.. code-block:: bash + + ┌────────────────────────────────────────────┐ + | Building the Docker image: my-conan-runner | + └────────────────────────────────────────────┘ + + Dockerfile path: '/Dockerfile' + Docker build context: '' + + Step 1/4 : FROM ubuntu:22.04 + + ... + + ---> dba927bb0517 + Successfully built dba927bb0517 + Successfully tagged my-conan-runner:latest + +3. Save the local cache running ``conan cache save``. + +.. code-block:: bash + + ┌────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐ + | Save host cache in: /conan-center-index/recipes/zlib/all/.conanrunner/local_cache_save.tgz | + └────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘ + + Found 1 pkg/version recipes matching * in local cache + Saving zlib/1.3.1: p/zlib95420566fc0dd + +4. Create and initialize the docker container. + +.. code-block:: bash + + ┌───────────────────────────────┐ + | Creating the docker container | + └───────────────────────────────┘ + + ┌───────────────────────────────────────┐ + | Container conan-runner-docker running | + └───────────────────────────────────────┘ + +5. Check if the container has a conan version with the runner feature. + +.. code-block:: bash + + ┌─────────────────────────────────────────┐ + | Running in container: "conan --version" | + └─────────────────────────────────────────┘ + + Conan version 2.3.0 + +6. Initialize the container conan cache using the host copy running ``conan cache restore``. + +.. code-block:: bash + + ┌───────────────────────────────────────────────────────────────────────────────────────────────────────┐ + | Running in container: "conan cache restore "/root/conanrunner/all/.conanrunner/local_cache_save.tgz"" | + └───────────────────────────────────────────────────────────────────────────────────────────────────────┘ + + Restore: zlib/1.3.1 in p/zlib95420566fc0dd + Local Cache + zlib + zlib/1.3.1 + revisions + e20364c96c45455608a72543f3a53133 (2024-04-29 17:19:32 UTC) + packages + recipe_folder: p/zlib95420566fc0dd + +7. Run the ``conan create`` inside the container and build zlib. + +.. code-block:: bash + + ┌───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐ + | Running in container: "conan create /root/conanrunner/all --version 1.3.1 -pr:h /root/conanrunner/all/.conanrunner/profiles/docker_example_host_1 -pr:b /root/conanrunner/all/.conanrunner/profiles/docker_example_build_0 -f json > create.json" | + └───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘ + + + ======== Exporting recipe to the cache ======== + zlib/1.3.1: Exporting package recipe: /root/conanrunner/all/conanfile.py + zlib/1.3.1: exports: File 'conandata.yml' found. Exporting it... + zlib/1.3.1: Calling export_sources() + zlib/1.3.1: Copied 1 '.yml' file: conandata.yml + zlib/1.3.1: Copied 1 '.py' file: conanfile.py + zlib/1.3.1: Copied 1 '.patch' file: 0001-fix-cmake.patch + zlib/1.3.1: Exported to cache folder: /root/.conan2/p/zlib95420566fc0dd/e + zlib/1.3.1: Exported: zlib/1.3.1#e20364c96c45455608a72543f3a53133 (2024-04-29 17:19:32 UTC) + + ======== Input profiles ======== + Profile host: + [settings] + arch=x86_64 + build_type=Release + compiler=gcc + compiler.cppstd=gnu17 + compiler.libcxx=libstdc++11 + compiler.version=11 + os=Linux + + Profile build: + [settings] + arch=x86_64 + build_type=Release + compiler=gcc + compiler.cppstd=gnu17 + compiler.libcxx=libstdc++11 + compiler.version=11 + os=Linux + + + ======== Computing dependency graph ======== + Graph root + cli + Requirements + zlib/1.3.1#e20364c96c45455608a72543f3a53133 - Cache + + ======== Computing necessary packages ======== + zlib/1.3.1: Forced build from source + Requirements + zlib/1.3.1#e20364c96c45455608a72543f3a53133:b647c43bfefae3f830561ca202b6cfd935b56205 - Build + + ======== Installing packages ======== + zlib/1.3.1: Calling source() in /root/.conan2/p/zlib95420566fc0dd/s/src + + -------- Installing package zlib/1.3.1 (1 of 1) -------- + zlib/1.3.1: Building from source + zlib/1.3.1: Package zlib/1.3.1:b647c43bfefae3f830561ca202b6cfd935b56205 + zlib/1.3.1: Copying sources to build folder + zlib/1.3.1: Building your package in /root/.conan2/p/b/zlib8dd8e27348e8c/b + zlib/1.3.1: Calling generate() + zlib/1.3.1: Generators folder: /root/.conan2/p/b/zlib8dd8e27348e8c/b/build/Release/generators + zlib/1.3.1: CMakeToolchain generated: conan_toolchain.cmake + zlib/1.3.1: CMakeToolchain generated: /root/.conan2/p/b/zlib8dd8e27348e8c/b/build/Release/generators/CMakePresets.json + zlib/1.3.1: CMakeToolchain generated: /root/.conan2/p/b/zlib8dd8e27348e8c/b/src/CMakeUserPresets.json + zlib/1.3.1: Generating aggregated env files + zlib/1.3.1: Generated aggregated env files: ['conanbuild.sh', 'conanrun.sh'] + zlib/1.3.1: Calling build() + zlib/1.3.1: Apply patch (conan): separate static/shared builds, disable debug suffix + zlib/1.3.1: Running CMake.configure() + zlib/1.3.1: RUN: cmake -G "Unix Makefiles" -DCMAKE_TOOLCHAIN_FILE="generators/conan_toolchain.cmake" -DCMAKE_INSTALL_PREFIX="/root/.conan2/p/b/zlib8dd8e27348e8c/p" -DCMAKE_POLICY_DEFAULT_CMP0091="NEW" -DCMAKE_BUILD_TYPE="Release" "/root/.conan2/p/b/zlib8dd8e27348e8c/b/src" + -- Using Conan toolchain: /root/.conan2/p/b/zlib8dd8e27348e8c/b/build/Release/generators/conan_toolchain.cmake + -- Conan toolchain: Setting CMAKE_POSITION_INDEPENDENT_CODE=ON (options.fPIC) + -- Conan toolchain: Setting BUILD_SHARED_LIBS = OFF + -- The C compiler identification is GNU 11.4.0 + -- Detecting C compiler ABI info + -- Detecting C compiler ABI info - done + -- Check for working C compiler: /usr/bin/cc - skipped + -- Detecting C compile features + -- Detecting C compile features - done + -- Looking for sys/types.h + -- Looking for sys/types.h - found + -- Looking for stdint.h + -- Looking for stdint.h - found + -- Looking for stddef.h + -- Looking for stddef.h - found + -- Check size of off64_t + -- Check size of off64_t - done + -- Looking for fseeko + -- Looking for fseeko - found + -- Looking for unistd.h + -- Looking for unistd.h - found + -- Renaming + -- /root/.conan2/p/b/zlib8dd8e27348e8c/b/src/zconf.h + -- to 'zconf.h.included' because this file is included with zlib + -- but CMake generates it automatically in the build directory. + -- Configuring done + -- Generating done + -- Build files have been written to: /root/.conan2/p/b/zlib8dd8e27348e8c/b/build/Release + zlib/1.3.1: Running CMake.build() + zlib/1.3.1: RUN: cmake --build "/root/.conan2/p/b/zlib8dd8e27348e8c/b/build/Release" -- -j16 + [ 12%] Building C object CMakeFiles/zlib.dir/adler32.c.o + [ 12%] Building C object CMakeFiles/zlib.dir/compress.c.o + [ 18%] Building C object CMakeFiles/zlib.dir/deflate.c.o + [ 25%] Building C object CMakeFiles/zlib.dir/crc32.c.o + [ 31%] Building C object CMakeFiles/zlib.dir/gzlib.c.o + [ 37%] Building C object CMakeFiles/zlib.dir/gzread.c.o + [ 43%] Building C object CMakeFiles/zlib.dir/gzclose.c.o + [ 56%] Building C object CMakeFiles/zlib.dir/infback.c.o + [ 56%] Building C object CMakeFiles/zlib.dir/gzwrite.c.o + [ 62%] Building C object CMakeFiles/zlib.dir/inflate.c.o + [ 68%] Building C object CMakeFiles/zlib.dir/inffast.c.o + [ 75%] Building C object CMakeFiles/zlib.dir/trees.c.o + [ 81%] Building C object CMakeFiles/zlib.dir/zutil.c.o + [ 87%] Building C object CMakeFiles/zlib.dir/uncompr.c.o + [ 93%] Building C object CMakeFiles/zlib.dir/inftrees.c.o + [100%] Linking C static library libz.a + [100%] Built target zlib + zlib/1.3.1: Package 'b647c43bfefae3f830561ca202b6cfd935b56205' built + zlib/1.3.1: Build folder /root/.conan2/p/b/zlib8dd8e27348e8c/b/build/Release + zlib/1.3.1: Generating the package + zlib/1.3.1: Packaging in folder /root/.conan2/p/b/zlib8dd8e27348e8c/p + zlib/1.3.1: Calling package() + zlib/1.3.1: Running CMake.install() + zlib/1.3.1: RUN: cmake --install "/root/.conan2/p/b/zlib8dd8e27348e8c/b/build/Release" --prefix "/root/.conan2/p/b/zlib8dd8e27348e8c/p" + -- Install configuration: "Release" + -- Installing: /root/.conan2/p/b/zlib8dd8e27348e8c/p/lib/libz.a + -- Installing: /root/.conan2/p/b/zlib8dd8e27348e8c/p/include/zconf.h + -- Installing: /root/.conan2/p/b/zlib8dd8e27348e8c/p/include/zlib.h + + zlib/1.3.1: package(): Packaged 1 file: LICENSE + zlib/1.3.1: package(): Packaged 2 '.h' files: zlib.h, zconf.h + zlib/1.3.1: package(): Packaged 1 '.a' file: libz.a + zlib/1.3.1: Created package revision fd85b1346d5377ae2465645768e62bf2 + zlib/1.3.1: Package 'b647c43bfefae3f830561ca202b6cfd935b56205' created + zlib/1.3.1: Full package reference: zlib/1.3.1#e20364c96c45455608a72543f3a53133:b647c43bfefae3f830561ca202b6cfd935b56205#fd85b1346d5377ae2465645768e62bf2 + zlib/1.3.1: Package folder /root/.conan2/p/b/zlib8dd8e27348e8c/p + WARN: deprecated: Usage of deprecated Conan 1.X features that will be removed in Conan 2.X: + WARN: deprecated: 'cpp_info.names' used in: zlib/1.3.1 + + ======== Launching test_package ======== + + ======== Computing dependency graph ======== + Graph root + zlib/1.3.1 (test package): /root/conanrunner/all/test_package/conanfile.py + Requirements + zlib/1.3.1#e20364c96c45455608a72543f3a53133 - Cache + + ======== Computing necessary packages ======== + Requirements + zlib/1.3.1#e20364c96c45455608a72543f3a53133:b647c43bfefae3f830561ca202b6cfd935b56205#fd85b1346d5377ae2465645768e62bf2 - Cache + + ======== Installing packages ======== + zlib/1.3.1: Already installed! (1 of 1) + WARN: deprecated: Usage of deprecated Conan 1.X features that will be removed in Conan 2.X: + WARN: deprecated: 'cpp_info.names' used in: zlib/1.3.1 + + ======== Testing the package ======== + Removing previously existing 'test_package' build folder: /root/conanrunner/all/test_package/build/gcc-11-x86_64-gnu17-release + zlib/1.3.1 (test package): Test package build: build/gcc-11-x86_64-gnu17-release + zlib/1.3.1 (test package): Test package build folder: /root/conanrunner/all/test_package/build/gcc-11-x86_64-gnu17-release + zlib/1.3.1 (test package): Writing generators to /root/conanrunner/all/test_package/build/gcc-11-x86_64-gnu17-release/generators + zlib/1.3.1 (test package): Generator 'CMakeToolchain' calling 'generate()' + zlib/1.3.1 (test package): CMakeToolchain generated: conan_toolchain.cmake + zlib/1.3.1 (test package): CMakeToolchain generated: /root/conanrunner/all/test_package/build/gcc-11-x86_64-gnu17-release/generators/CMakePresets.json + zlib/1.3.1 (test package): CMakeToolchain generated: /root/conanrunner/all/test_package/CMakeUserPresets.json + zlib/1.3.1 (test package): Generator 'CMakeDeps' calling 'generate()' + zlib/1.3.1 (test package): CMakeDeps necessary find_package() and targets for your CMakeLists.txt + find_package(ZLIB) + target_link_libraries(... ZLIB::ZLIB) + zlib/1.3.1 (test package): Generator 'VirtualRunEnv' calling 'generate()' + zlib/1.3.1 (test package): Generating aggregated env files + zlib/1.3.1 (test package): Generated aggregated env files: ['conanrun.sh', 'conanbuild.sh'] + + ======== Testing the package: Building ======== + zlib/1.3.1 (test package): Calling build() + zlib/1.3.1 (test package): Running CMake.configure() + zlib/1.3.1 (test package): RUN: cmake -G "Unix Makefiles" -DCMAKE_TOOLCHAIN_FILE="generators/conan_toolchain.cmake" -DCMAKE_INSTALL_PREFIX="/root/conanrunner/all/test_package" -DCMAKE_POLICY_DEFAULT_CMP0091="NEW" -DCMAKE_BUILD_TYPE="Release" "/root/conanrunner/all/test_package" + -- Using Conan toolchain: /root/conanrunner/all/test_package/build/gcc-11-x86_64-gnu17-release/generators/conan_toolchain.cmake + -- Conan toolchain: C++ Standard 17 with extensions ON + -- The C compiler identification is GNU 11.4.0 + -- Detecting C compiler ABI info + -- Detecting C compiler ABI info - done + -- Check for working C compiler: /usr/bin/cc - skipped + -- Detecting C compile features + -- Detecting C compile features - done + -- Conan: Target declared 'ZLIB::ZLIB' + -- Configuring done + -- Generating done + -- Build files have been written to: /root/conanrunner/all/test_package/build/gcc-11-x86_64-gnu17-release + zlib/1.3.1 (test package): Running CMake.build() + zlib/1.3.1 (test package): RUN: cmake --build "/root/conanrunner/all/test_package/build/gcc-11-x86_64-gnu17-release" -- -j16 + [ 50%] Building C object CMakeFiles/test_package.dir/test_package.c.o + [100%] Linking C executable test_package + [100%] Built target test_package + + ======== Testing the package: Executing test ======== + zlib/1.3.1 (test package): Running test() + zlib/1.3.1 (test package): RUN: ./test_package + Compressed size is: 21 + Compressed string is: Conan Package Manager + Compressed size is: 22 + Compressed string is: xsKHLNLOUMRE + ZLIB VERSION: 1.3.1 + +8. Copy just the package created inside the container using the ``pkglist.json`` info from the previous ``conan create``, restore this new package inside the host cache running a ``conan cache save`` and remove the container. + +.. code-block:: bash + + ┌────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐ + | Running in container: "conan cache save --list=pkglist.json --file "/root/conanrunner/all"/.conanrunner/docker_cache_save.tgz" | + └────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘ + + Saving zlib/1.3.1: p/zlib95420566fc0dd + Saving zlib/1.3.1:b647c43bfefae3f830561ca202b6cfd935b56205: p/b/zlib8dd8e27348e8c/p + Saving zlib/1.3.1:b647c43bfefae3f830561ca202b6cfd935b56205 metadata: p/b/zlib8dd8e27348e8c/d/metadata + Local Cache + zlib + zlib/1.3.1 + revisions + e20364c96c45455608a72543f3a53133 (2024-04-29 17:19:32 UTC) + packages + b647c43bfefae3f830561ca202b6cfd935b56205 + revisions + fd85b1346d5377ae2465645768e62bf2 + package_folder: p/b/zlib8dd8e27348e8c/p + metadata_folder: p/b/zlib8dd8e27348e8c/d/metadata + info + settings + os: Linux + arch: x86_64 + compiler: gcc + compiler.version: 11 + build_type: Release + options + fPIC: True + shared: False + recipe_folder: p/zlib95420566fc0dd + + + ┌──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐ + | Restore host cache from: /conan-center-index/recipes/zlib/all/.conanrunner/docker_cache_save.tgz | + └──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘ + + Restore: zlib/1.3.1 in p/zlib95420566fc0dd + Restore: zlib/1.3.1:b647c43bfefae3f830561ca202b6cfd935b56205 in p/b/zlib8dd8e27348e8c/p + Restore: zlib/1.3.1:b647c43bfefae3f830561ca202b6cfd935b56205 metadata in p/b/zlib8dd8e27348e8c/d/metadata + + ┌────────────────────┐ + | Stopping container | + └────────────────────┘ + + + ┌────────────────────┐ + | Removing container | + └────────────────────┘ + +If we now check the status of our conan and docker cache, we will see the new zlib package compile for Linux and the new docker image. We don't have any container because we define ``remove=true`` + +.. code-block:: bash + + $ conan list "*:*" + Found 1 pkg/version recipes matching * in local cache + Local Cache + zlib + zlib/1.3.1 + revisions + e20364c96c45455608a72543f3a53133 (2024-04-29 17:18:07 UTC) + packages + b647c43bfefae3f830561ca202b6cfd935b56205 + info + settings + arch: x86_64 + build_type: Release + compiler: gcc + compiler.version: 11 + os: Linux + options + fPIC: True + shared: False + + $ docker ps --all + CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES + + $ docker images + REPOSITORY TAG IMAGE ID CREATED SIZE + my-conan-runner latest 383b905f352e 22 minutes ago 531MB + ubuntu 22.04 437ec753bef3 12 days ago 77.9MB + +Config file example +------------------- + +In this example we are going to see how to use a docker runner configfile. Let’s create two profiles and a Dockerfile inside our project folder. + +.. code-block:: bash + + $ cd + $ tree + . + ├── Dockerfile + ├── configfile + ├── docker_example_build + └── docker_example_host + +``docker_example_host`` profile + +.. code-block:: text + + [settings] + arch=x86_64 + build_type=Release + compiler=gcc + compiler.cppstd=gnu17 + compiler.libcxx=libstdc++11 + compiler.version=11 + os=Linux + [runner] + type=docker + configfile=/configfile + cache=copy + remove=false + +``docker_example_build`` profile + +.. code-block:: text + + [settings] + arch=x86_64 + build_type=Release + compiler=gcc + compiler.cppstd=gnu17 + compiler.libcxx=libstdc++11 + compiler.version=11 + os=Linux + +``configfile`` + +.. code-block:: yaml + + image: my-conan-runner-image + build: + dockerfile: + build_context: + build_args: + BASE_IMAGE: ubuntu:22.04 + run: + name: my-conan-runner-container + +``Dockerfile`` + +.. code-block:: docker + + ARG BASE_IMAGE + FROM $BASE_IMAGE + RUN apt-get update \ + && DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends \ + build-essential \ + cmake \ + python3 \ + python3-pip \ + python3-venv \ + && rm -rf /var/lib/apt/lists/* + RUN pip install conan + +In this example we are going to start from a totally clean docker, without containers or images. In addition, we are going to have the conan cache also completely empty. + +.. code-block:: bash + + $ conan list "*:*" + Found 0 pkg/version recipes matching * in local cache + + $ docker ps --all + CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES + + $ docker images + REPOSITORY TAG IMAGE ID CREATED SIZE + + +Now, we are going to clone and build zlib from conan-center-index and create it using our new runner definition. + +.. code-block:: bash + + $ git clone https://github.com/conan-io/conan-center-index.git + $ conan create ./conan-center-index/recipes/zlib/all --version 1.3.1 -pr:h /docker_example_host -pr:b /docker_example_build + + ... + + ┌──────────────────────────────────────────────────┐ + | Building the Docker image: my-conan-runner-image | + └──────────────────────────────────────────────────┘ + + Dockerfile path: '/Dockerfile' + Docker build context: '' + + Step 1/5 : ARG BASE_IMAGE + + Step 2/5 : FROM $BASE_IMAGE + + ... + + Successfully built 286df085400f + Successfully tagged my-conan-runner-image:latest + + ... + + ┌───────────────────────────────┐ + | Creating the docker container | + └───────────────────────────────┘ + + + ┌─────────────────────────────────────────┐ + | Container my-conan-runner-image running | + └─────────────────────────────────────────┘ + + + ┌─────────────────────────────────────────┐ + | Running in container: "conan --version" | + └─────────────────────────────────────────┘ + + ┌──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐ + | Restore host cache from: /conan-center-index/recipes/zlib/all/.conanrunner/docker_cache_save.tgz | + └──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘ + + Restore: zlib/1.3.1 in p/zlib95420566fc0dd + Restore: zlib/1.3.1:b647c43bfefae3f830561ca202b6cfd935b56205 in p/zlibd59462fc4358e/p + Restore: zlib/1.3.1:b647c43bfefae3f830561ca202b6cfd935b56205 metadata in p/zlibd59462fc4358e/d/metadata + + ┌────────────────────┐ + | Stopping container | + └────────────────────┘ + +If we now check the status of our conan and docker cache, we will see the zlib package compile for Linux and the new docker image and container. + +.. code-block:: bash + + $ conan list "*:*" + Found 1 pkg/version recipes matching * in local cache + Local Cache + zlib + zlib/1.3.1 + revisions + e20364c96c45455608a72543f3a53133 (2024-04-29 17:18:07 UTC) + packages + b647c43bfefae3f830561ca202b6cfd935b56205 + info + settings + arch: x86_64 + build_type: Release + compiler: gcc + compiler.version: 11 + os: Linux + options + fPIC: True + shared: False + + $ docker ps --all + CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES + 1379072ae424 my-conan-runner-image "/bin/bash -c 'while…" 17 seconds ago Exited (137) 2 seconds ago my-conan-runner-image + + $ docker images + REPOSITORY TAG IMAGE ID CREATED SIZE + my-conan-runner latest 383b905f352e 22 minutes ago 531MB + ubuntu 22.04 437ec753bef3 12 days ago 77.9MB + +If we run the conan create command again we will see how conan reuses the previous container beacause we set ``remove=False``. + +.. code-block:: bash + + $ conan create ./conan-center-index/recipes/zlib/all --version 1.3.1 -pr:h /docker_example_host -pr:b /docker_example_build + + ... + + ┌───────────────────────────────┐ + | Starting the docker container | + └───────────────────────────────┘ + + ... \ No newline at end of file From 3b6a0ce4c43d3f3fb0897583e95a437114f35840 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?David=20S=C3=A1nchez=20Falero?= Date: Mon, 6 May 2024 16:36:24 +0200 Subject: [PATCH 02/31] Update reference/runners.rst MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-authored-by: Rubén Rincón Blanco --- reference/runners.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/reference/runners.rst b/reference/runners.rst index 060c853f491..e744f6e1437 100644 --- a/reference/runners.rst +++ b/reference/runners.rst @@ -3,7 +3,7 @@ Runners ======= -Runners are the way to run conan remotely and transparently just by modifying the host profile. There are three requirements to be able to use the requirements: +Runners are the way to run Conan remotely and transparently just by modifying the host profile. There are three requirements to be able to use the feature: - Installing a version of conan with runner dependencies ``pip install conan[runners]``. - Install the tools to run each of the runners (``docker``). From 52afd3a1bc7e604a2922defcf4b9d340f4df5c5a Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?David=20S=C3=A1nchez=20Falero?= Date: Mon, 6 May 2024 16:36:52 +0200 Subject: [PATCH 03/31] Update reference/runners.rst MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-authored-by: Rubén Rincón Blanco --- reference/runners.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/reference/runners.rst b/reference/runners.rst index e744f6e1437..1a723910b88 100644 --- a/reference/runners.rst +++ b/reference/runners.rst @@ -7,7 +7,7 @@ Runners are the way to run Conan remotely and transparently just by modifying th - Installing a version of conan with runner dependencies ``pip install conan[runners]``. - Install the tools to run each of the runners (``docker``). -- Add to the ``host profile`` the ``[runner]`` section defined in the documentation of each runner. +- Add the ``[runner]`` section defined in the documentation of each runner to the host profile. Runners: From 493133b74045efddcd7591e2c9be9fd9d1aa5ce5 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?David=20S=C3=A1nchez=20Falero?= Date: Mon, 6 May 2024 16:37:06 +0200 Subject: [PATCH 04/31] Update reference/runners/docker.rst MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-authored-by: Rubén Rincón Blanco --- reference/runners/docker.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/reference/runners/docker.rst b/reference/runners/docker.rst index c131da6da7d..a87fd25957d 100644 --- a/reference/runners/docker.rst +++ b/reference/runners/docker.rst @@ -6,7 +6,7 @@ Docker runner How to use a docker runner -------------------------- -To run conan inside a docker container you need to define a ``[runner]`` section in your host profile file using the following fields. +To run Conan inside a docker container you need to define a ``[runner]`` section in your host profile using the following fields: - ``type`` **(mandatory)**: define the ruuner we want to use, in that case ``docker``. - ``dockerfile`` **(optional, default None)**: define the Dockerfile absolute path in case you want to build a docker image. From cf7e0d03c09597fb64a6028b246a46da579ffd36 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?David=20S=C3=A1nchez=20Falero?= Date: Mon, 6 May 2024 16:37:13 +0200 Subject: [PATCH 05/31] Update reference/runners/docker.rst MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-authored-by: Rubén Rincón Blanco --- reference/runners/docker.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/reference/runners/docker.rst b/reference/runners/docker.rst index a87fd25957d..3832028a21d 100644 --- a/reference/runners/docker.rst +++ b/reference/runners/docker.rst @@ -8,7 +8,7 @@ How to use a docker runner To run Conan inside a docker container you need to define a ``[runner]`` section in your host profile using the following fields: -- ``type`` **(mandatory)**: define the ruuner we want to use, in that case ``docker``. +- ``type`` **(mandatory)**: define the runner we want to use, in this case ``docker``. - ``dockerfile`` **(optional, default None)**: define the Dockerfile absolute path in case you want to build a docker image. - ``image`` **(optional, default conan-runner-default)**: docker image name you want to download from a docker registry or the name of the builded image in case you define a dockerfile path. - ``cache`` **(optional, default clean)**: how docker container uses (or not) the host cache. From a8af193ee1329ef14b943aa5ac47efa6fb2251d4 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?David=20S=C3=A1nchez=20Falero?= Date: Mon, 6 May 2024 16:37:21 +0200 Subject: [PATCH 06/31] Update reference/runners/docker.rst MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-authored-by: Rubén Rincón Blanco --- reference/runners/docker.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/reference/runners/docker.rst b/reference/runners/docker.rst index 3832028a21d..698c4f37f6b 100644 --- a/reference/runners/docker.rst +++ b/reference/runners/docker.rst @@ -10,7 +10,7 @@ To run Conan inside a docker container you need to define a ``[runner]`` section - ``type`` **(mandatory)**: define the runner we want to use, in this case ``docker``. - ``dockerfile`` **(optional, default None)**: define the Dockerfile absolute path in case you want to build a docker image. -- ``image`` **(optional, default conan-runner-default)**: docker image name you want to download from a docker registry or the name of the builded image in case you define a dockerfile path. +- ``image`` **(optional, default conan-runner-default)**: docker image name you want to download from a docker registry or the name of the built image in case you define a dockerfile path. - ``cache`` **(optional, default clean)**: how docker container uses (or not) the host cache. - ``clean``: use an empty cache. From 1a0f7627ce5ffba6d997d2205ce3530a7667e568 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?David=20S=C3=A1nchez=20Falero?= Date: Mon, 6 May 2024 16:37:37 +0200 Subject: [PATCH 07/31] Update reference/runners/docker.rst MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-authored-by: Rubén Rincón Blanco --- reference/runners/docker.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/reference/runners/docker.rst b/reference/runners/docker.rst index 698c4f37f6b..7f682d8eff8 100644 --- a/reference/runners/docker.rst +++ b/reference/runners/docker.rst @@ -586,9 +586,9 @@ In this example we are going to see how to use a docker runner configfile. Let run: name: my-conan-runner-container -``Dockerfile`` .. code-block:: docker + :caption: Dockerfile ARG BASE_IMAGE FROM $BASE_IMAGE From 51e4a1a68909d36b098c621790f9b637327f44db Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?David=20S=C3=A1nchez=20Falero?= Date: Mon, 6 May 2024 16:37:44 +0200 Subject: [PATCH 08/31] Update reference/runners/docker.rst MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-authored-by: Rubén Rincón Blanco --- reference/runners/docker.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/reference/runners/docker.rst b/reference/runners/docker.rst index 7f682d8eff8..099fa1a75c1 100644 --- a/reference/runners/docker.rst +++ b/reference/runners/docker.rst @@ -129,7 +129,7 @@ Now, we are going to clone and build zlib from conan-center-index and create it .. code-block:: bash - $ git clone https://github.com/conan-io/conan-center-index.git + $ git clone https://github.com/conan-io/conan-center-index.git --depth=1 $ conan create ./conan-center-index/recipes/zlib/all --version 1.3.1 -pr:h /docker_example_host -pr:b /docker_example_build If we split and analyze the command output, we can see what is happening and where the commands are being executed. From eb22c6a6c37a2d8b5e8f5c5b3cbf05065bcbc0e6 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?David=20S=C3=A1nchez=20Falero?= Date: Mon, 6 May 2024 16:37:51 +0200 Subject: [PATCH 09/31] Update reference/runners/docker.rst MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-authored-by: Rubén Rincón Blanco --- reference/runners/docker.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/reference/runners/docker.rst b/reference/runners/docker.rst index 099fa1a75c1..ca08c1d18a3 100644 --- a/reference/runners/docker.rst +++ b/reference/runners/docker.rst @@ -669,7 +669,7 @@ Now, we are going to clone and build zlib from conan-center-index and create it | Stopping container | └────────────────────┘ -If we now check the status of our conan and docker cache, we will see the zlib package compile for Linux and the new docker image and container. +If we now check the status of our Conan and docker cache, we will see the zlib package compiled for Linux and the new docker image and container. .. code-block:: bash From b6c2387c41db8caeac1321016cfb0cc2902c51be Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?David=20S=C3=A1nchez=20Falero?= Date: Mon, 6 May 2024 16:38:06 +0200 Subject: [PATCH 10/31] Update reference/runners/docker.rst MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-authored-by: Rubén Rincón Blanco --- reference/runners/docker.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/reference/runners/docker.rst b/reference/runners/docker.rst index ca08c1d18a3..b4ae514ded6 100644 --- a/reference/runners/docker.rst +++ b/reference/runners/docker.rst @@ -620,7 +620,7 @@ Now, we are going to clone and build zlib from conan-center-index and create it .. code-block:: bash - $ git clone https://github.com/conan-io/conan-center-index.git + $ git clone https://github.com/conan-io/conan-center-index.git --depth 1 $ conan create ./conan-center-index/recipes/zlib/all --version 1.3.1 -pr:h /docker_example_host -pr:b /docker_example_build ... From 6b27ee674a5f3d21036807e2a15d005bc941fac7 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?David=20S=C3=A1nchez=20Falero?= Date: Mon, 6 May 2024 16:38:14 +0200 Subject: [PATCH 11/31] Update reference/runners/docker.rst MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-authored-by: Rubén Rincón Blanco --- reference/runners/docker.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/reference/runners/docker.rst b/reference/runners/docker.rst index b4ae514ded6..dfc6ee36c9a 100644 --- a/reference/runners/docker.rst +++ b/reference/runners/docker.rst @@ -702,7 +702,7 @@ If we now check the status of our Conan and docker cache, we will see the zlib p my-conan-runner latest 383b905f352e 22 minutes ago 531MB ubuntu 22.04 437ec753bef3 12 days ago 77.9MB -If we run the conan create command again we will see how conan reuses the previous container beacause we set ``remove=False``. +If we run the ``conan create`` command again we will see how Conan reuses the previous container because we have set ``remove=False``. .. code-block:: bash From f8ad316488b2928a1b97d48e16ac87d3a3ee2c21 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?David=20S=C3=A1nchez=20Falero?= Date: Mon, 6 May 2024 16:38:30 +0200 Subject: [PATCH 12/31] Update reference/runners/docker.rst Co-authored-by: Carlos Zoido --- reference/runners/docker.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/reference/runners/docker.rst b/reference/runners/docker.rst index dfc6ee36c9a..685de297ab6 100644 --- a/reference/runners/docker.rst +++ b/reference/runners/docker.rst @@ -38,7 +38,7 @@ If you need more controll over the container build and execution, you can define run: name: container_name # The name for this container. containerEnv: # Environment variables to set inside the container. - env_var_1: env_valie + env_var_1: env_value containerUser: user_name # Username or UID to run commands as inside the container. privileged: False # Run as privileged capAdd: # Add kernel capabilities. From 9b0df238213be8d0525e8f400704da2f73ad1756 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?David=20S=C3=A1nchez=20Falero?= Date: Mon, 6 May 2024 16:38:44 +0200 Subject: [PATCH 13/31] Update reference/runners/docker.rst Co-authored-by: Carlos Zoido --- reference/runners/docker.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/reference/runners/docker.rst b/reference/runners/docker.rst index 685de297ab6..1f138c92e83 100644 --- a/reference/runners/docker.rst +++ b/reference/runners/docker.rst @@ -130,7 +130,7 @@ Now, we are going to clone and build zlib from conan-center-index and create it .. code-block:: bash $ git clone https://github.com/conan-io/conan-center-index.git --depth=1 - $ conan create ./conan-center-index/recipes/zlib/all --version 1.3.1 -pr:h /docker_example_host -pr:b /docker_example_build + $ conan create ./conan-center-index/recipes/zlib/all --version 1.3.1 -pr:h /docker_example_host -pr:b /docker_example_build If we split and analyze the command output, we can see what is happening and where the commands are being executed. From 24b94fbc431f3d24f4db9cea53494ef1865b5948 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?David=20S=C3=A1nchez=20Falero?= Date: Mon, 6 May 2024 16:39:05 +0200 Subject: [PATCH 14/31] Update reference/runners/docker.rst Co-authored-by: Carlos Zoido --- reference/runners/docker.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/reference/runners/docker.rst b/reference/runners/docker.rst index 1f138c92e83..bac939a171f 100644 --- a/reference/runners/docker.rst +++ b/reference/runners/docker.rst @@ -139,7 +139,7 @@ If we split and analyze the command output, we can see what is happening and whe .. code-block:: bash ======== Exporting recipe to the cache ======== - zlib/1.3.1: Exporting package recipe: /conan-center-index/recipes/zlib/all/conanfile.py + zlib/1.3.1: Exporting package recipe: /conan-center-index/recipes/zlib/all/conanfile.py zlib/1.3.1: exports: File 'conandata.yml' found. Exporting it... zlib/1.3.1: Calling export_sources() zlib/1.3.1: Copied 1 '.py' file: conanfile.py From 050a5897c573d846ae5397deefae65a298ab80b6 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?David=20S=C3=A1nchez=20Falero?= Date: Mon, 6 May 2024 16:39:11 +0200 Subject: [PATCH 15/31] Update reference/runners/docker.rst Co-authored-by: Carlos Zoido --- reference/runners/docker.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/reference/runners/docker.rst b/reference/runners/docker.rst index bac939a171f..20e47028520 100644 --- a/reference/runners/docker.rst +++ b/reference/runners/docker.rst @@ -177,8 +177,8 @@ If we split and analyze the command output, we can see what is happening and whe | Building the Docker image: my-conan-runner | └────────────────────────────────────────────┘ - Dockerfile path: '/Dockerfile' - Docker build context: '' + Dockerfile path: '/Dockerfile' + Docker build context: '' Step 1/4 : FROM ubuntu:22.04 From 973f649945d0d2eaf6e111707727f238c52c5a9f Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?David=20S=C3=A1nchez=20Falero?= Date: Mon, 6 May 2024 16:39:21 +0200 Subject: [PATCH 16/31] Update reference/runners/docker.rst Co-authored-by: Carlos Zoido --- reference/runners/docker.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/reference/runners/docker.rst b/reference/runners/docker.rst index 20e47028520..9ade4db6e91 100644 --- a/reference/runners/docker.rst +++ b/reference/runners/docker.rst @@ -193,7 +193,7 @@ If we split and analyze the command output, we can see what is happening and whe .. code-block:: bash ┌────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐ - | Save host cache in: /conan-center-index/recipes/zlib/all/.conanrunner/local_cache_save.tgz | + | Save host cache in: /conan-center-index/recipes/zlib/all/.conanrunner/local_cache_save.tgz | └────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘ Found 1 pkg/version recipes matching * in local cache From 4e14d9f776a5ceef2434b5f6dbbd00b3b9ee0804 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?David=20S=C3=A1nchez=20Falero?= Date: Mon, 6 May 2024 16:39:28 +0200 Subject: [PATCH 17/31] Update reference/runners/docker.rst Co-authored-by: Carlos Zoido --- reference/runners/docker.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/reference/runners/docker.rst b/reference/runners/docker.rst index 9ade4db6e91..cd8de27affd 100644 --- a/reference/runners/docker.rst +++ b/reference/runners/docker.rst @@ -658,7 +658,7 @@ Now, we are going to clone and build zlib from conan-center-index and create it └─────────────────────────────────────────┘ ┌──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐ - | Restore host cache from: /conan-center-index/recipes/zlib/all/.conanrunner/docker_cache_save.tgz | + | Restore host cache from: /conan-center-index/recipes/zlib/all/.conanrunner/docker_cache_save.tgz | └──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘ Restore: zlib/1.3.1 in p/zlib95420566fc0dd From bed82328cad345883da88b5ad42ad0222a5a78be Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?David=20S=C3=A1nchez=20Falero?= Date: Mon, 6 May 2024 16:39:33 +0200 Subject: [PATCH 18/31] Update reference/runners/docker.rst Co-authored-by: Carlos Zoido --- reference/runners/docker.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/reference/runners/docker.rst b/reference/runners/docker.rst index cd8de27affd..2305b079f94 100644 --- a/reference/runners/docker.rst +++ b/reference/runners/docker.rst @@ -706,7 +706,7 @@ If we run the ``conan create`` command again we will see how Conan reuses the pr .. code-block:: bash - $ conan create ./conan-center-index/recipes/zlib/all --version 1.3.1 -pr:h /docker_example_host -pr:b /docker_example_build + $ conan create ./conan-center-index/recipes/zlib/all --version 1.3.1 -pr:h /docker_example_host -pr:b /docker_example_build ... From 4474a1ef2947a469892ac8d7e92eeaaca0def4ed Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?David=20S=C3=A1nchez=20Falero?= Date: Mon, 6 May 2024 16:39:50 +0200 Subject: [PATCH 19/31] Update reference/runners/docker.rst Co-authored-by: Carlos Zoido --- reference/runners/docker.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/reference/runners/docker.rst b/reference/runners/docker.rst index 2305b079f94..9ab28cd10c5 100644 --- a/reference/runners/docker.rst +++ b/reference/runners/docker.rst @@ -579,8 +579,8 @@ In this example we are going to see how to use a docker runner configfile. Let image: my-conan-runner-image build: - dockerfile: - build_context: + dockerfile: + build_context: build_args: BASE_IMAGE: ubuntu:22.04 run: From de77e9003a608e7d1c1681afae3783fd9cb4c5b3 Mon Sep 17 00:00:00 2001 From: David Sanchez Date: Mon, 6 May 2024 17:10:36 +0200 Subject: [PATCH 20/31] docker runners examples added --- examples.rst | 1 + examples/runners.rst | 10 + examples/runners/docker/basic.rst | 477 +++++++++++++ .../runners/docker/configfile_build_args.rst | 190 +++++ reference/runners/docker.rst | 667 +----------------- 5 files changed, 683 insertions(+), 662 deletions(-) create mode 100644 examples/runners.rst create mode 100644 examples/runners/docker/basic.rst create mode 100644 examples/runners/docker/configfile_build_args.rst diff --git a/examples.rst b/examples.rst index f6a29957e6c..e236ff722a0 100644 --- a/examples.rst +++ b/examples.rst @@ -15,3 +15,4 @@ Examples examples/graph examples/dev_flow examples/commands + examples/runners diff --git a/examples/runners.rst b/examples/runners.rst new file mode 100644 index 00000000000..1c90a813686 --- /dev/null +++ b/examples/runners.rst @@ -0,0 +1,10 @@ +.. _examples_runners: + +Conan runners examples +====================== + +.. toctree:: + :maxdepth: 2 + + runners/docker/basic + runners/docker/configfile_build_args diff --git a/examples/runners/docker/basic.rst b/examples/runners/docker/basic.rst new file mode 100644 index 00000000000..5ee977c7100 --- /dev/null +++ b/examples/runners/docker/basic.rst @@ -0,0 +1,477 @@ +.. _examples_runners_docker_basic: + +Using docker runner +=================== + +In this example we are going to see how to compile zlib 1.3.1 inside docker. Let’s create two profiles and a Dockerfile inside our project folder. + +.. code-block:: bash + + $ cd + $ tree + . + ├── Dockerfile + ├── docker_example_build + └── docker_example_host + +``docker_example_host`` profile + +.. code-block:: text + + [settings] + arch=x86_64 + build_type=Release + compiler=gcc + compiler.cppstd=gnu17 + compiler.libcxx=libstdc++11 + compiler.version=11 + os=Linux + [runner] + type=docker + dockerfile= + cache=copy + remove=true + +``docker_example_build`` profile + +.. code-block:: text + + [settings] + arch=x86_64 + build_type=Release + compiler=gcc + compiler.cppstd=gnu17 + compiler.libcxx=libstdc++11 + compiler.version=11 + os=Linux + +``Dockerfile`` + +.. code-block:: docker + + FROM ubuntu:22.04 + RUN apt-get update \ + && DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends \ + build-essential \ + cmake \ + python3 \ + python3-pip \ + python3-venv \ + && rm -rf /var/lib/apt/lists/* + RUN pip install conan + +In this example we are going to start from a totally clean docker, without containers or images. In addition, we are going to have the conan cache also completely empty. + +.. code-block:: bash + + $ conan list "*:*" + Found 0 pkg/version recipes matching * in local cache + + $ docker ps --all + CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES + + $ docker images + REPOSITORY TAG IMAGE ID CREATED SIZE + + +Now, we are going to clone and build zlib from conan-center-index and create it using our new runner definition. + +.. code-block:: bash + + $ git clone https://github.com/conan-io/conan-center-index.git --depth=1 + $ conan create ./conan-center-index/recipes/zlib/all --version 1.3.1 -pr:h /docker_example_host -pr:b /docker_example_build + +If we split and analyze the command output, we can see what is happening and where the commands are being executed. + +1. Standard conan execution. + +.. code-block:: bash + + ======== Exporting recipe to the cache ======== + zlib/1.3.1: Exporting package recipe: /conan-center-index/recipes/zlib/all/conanfile.py + zlib/1.3.1: exports: File 'conandata.yml' found. Exporting it... + zlib/1.3.1: Calling export_sources() + zlib/1.3.1: Copied 1 '.py' file: conanfile.py + zlib/1.3.1: Copied 1 '.yml' file: conandata.yml + zlib/1.3.1: Copied 1 '.patch' file: 0001-fix-cmake.patch + zlib/1.3.1: Exported to cache folder: /Users/conan/.conan2/p/zlib95420566fc0dd/e + zlib/1.3.1: Exported: zlib/1.3.1#e20364c96c45455608a72543f3a53133 (2024-04-29 17:03:44 UTC) + + ======== Input profiles ======== + Profile host: + [settings] + arch=x86_64 + build_type=Release + compiler=gcc + compiler.cppstd=gnu17 + compiler.libcxx=libstdc++11 + compiler.version=11 + os=Linux + + Profile build: + [settings] + arch=x86_64 + build_type=Release + compiler=gcc + compiler.cppstd=gnu17 + compiler.libcxx=libstdc++11 + compiler.version=11 + os=Linux + +2. Build docker image + +.. code-block:: bash + + ┌────────────────────────────────────────────┐ + | Building the Docker image: my-conan-runner | + └────────────────────────────────────────────┘ + + Dockerfile path: '/Dockerfile' + Docker build context: '' + + Step 1/4 : FROM ubuntu:22.04 + + ... + + ---> dba927bb0517 + Successfully built dba927bb0517 + Successfully tagged my-conan-runner:latest + +3. Save the local cache running ``conan cache save``. + +.. code-block:: bash + + ┌────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐ + | Save host cache in: /conan-center-index/recipes/zlib/all/.conanrunner/local_cache_save.tgz | + └────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘ + + Found 1 pkg/version recipes matching * in local cache + Saving zlib/1.3.1: p/zlib95420566fc0dd + +4. Create and initialize the docker container. + +.. code-block:: bash + + ┌───────────────────────────────┐ + | Creating the docker container | + └───────────────────────────────┘ + + ┌───────────────────────────────────────┐ + | Container conan-runner-docker running | + └───────────────────────────────────────┘ + +5. Check if the container has a conan version with the runner feature. + +.. code-block:: bash + + ┌─────────────────────────────────────────┐ + | Running in container: "conan --version" | + └─────────────────────────────────────────┘ + + Conan version 2.3.0 + +6. Initialize the container conan cache using the host copy running ``conan cache restore``. + +.. code-block:: bash + + ┌───────────────────────────────────────────────────────────────────────────────────────────────────────┐ + | Running in container: "conan cache restore "/root/conanrunner/all/.conanrunner/local_cache_save.tgz"" | + └───────────────────────────────────────────────────────────────────────────────────────────────────────┘ + + Restore: zlib/1.3.1 in p/zlib95420566fc0dd + Local Cache + zlib + zlib/1.3.1 + revisions + e20364c96c45455608a72543f3a53133 (2024-04-29 17:19:32 UTC) + packages + recipe_folder: p/zlib95420566fc0dd + +7. Run the ``conan create`` inside the container and build zlib. + +.. code-block:: bash + + ┌───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐ + | Running in container: "conan create /root/conanrunner/all --version 1.3.1 -pr:h /root/conanrunner/all/.conanrunner/profiles/docker_example_host_1 -pr:b /root/conanrunner/all/.conanrunner/profiles/docker_example_build_0 -f json > create.json" | + └───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘ + + + ======== Exporting recipe to the cache ======== + zlib/1.3.1: Exporting package recipe: /root/conanrunner/all/conanfile.py + zlib/1.3.1: exports: File 'conandata.yml' found. Exporting it... + zlib/1.3.1: Calling export_sources() + zlib/1.3.1: Copied 1 '.yml' file: conandata.yml + zlib/1.3.1: Copied 1 '.py' file: conanfile.py + zlib/1.3.1: Copied 1 '.patch' file: 0001-fix-cmake.patch + zlib/1.3.1: Exported to cache folder: /root/.conan2/p/zlib95420566fc0dd/e + zlib/1.3.1: Exported: zlib/1.3.1#e20364c96c45455608a72543f3a53133 (2024-04-29 17:19:32 UTC) + + ======== Input profiles ======== + Profile host: + [settings] + arch=x86_64 + build_type=Release + compiler=gcc + compiler.cppstd=gnu17 + compiler.libcxx=libstdc++11 + compiler.version=11 + os=Linux + + Profile build: + [settings] + arch=x86_64 + build_type=Release + compiler=gcc + compiler.cppstd=gnu17 + compiler.libcxx=libstdc++11 + compiler.version=11 + os=Linux + + + ======== Computing dependency graph ======== + Graph root + cli + Requirements + zlib/1.3.1#e20364c96c45455608a72543f3a53133 - Cache + + ======== Computing necessary packages ======== + zlib/1.3.1: Forced build from source + Requirements + zlib/1.3.1#e20364c96c45455608a72543f3a53133:b647c43bfefae3f830561ca202b6cfd935b56205 - Build + + ======== Installing packages ======== + zlib/1.3.1: Calling source() in /root/.conan2/p/zlib95420566fc0dd/s/src + + -------- Installing package zlib/1.3.1 (1 of 1) -------- + zlib/1.3.1: Building from source + zlib/1.3.1: Package zlib/1.3.1:b647c43bfefae3f830561ca202b6cfd935b56205 + zlib/1.3.1: Copying sources to build folder + zlib/1.3.1: Building your package in /root/.conan2/p/b/zlib8dd8e27348e8c/b + zlib/1.3.1: Calling generate() + zlib/1.3.1: Generators folder: /root/.conan2/p/b/zlib8dd8e27348e8c/b/build/Release/generators + zlib/1.3.1: CMakeToolchain generated: conan_toolchain.cmake + zlib/1.3.1: CMakeToolchain generated: /root/.conan2/p/b/zlib8dd8e27348e8c/b/build/Release/generators/CMakePresets.json + zlib/1.3.1: CMakeToolchain generated: /root/.conan2/p/b/zlib8dd8e27348e8c/b/src/CMakeUserPresets.json + zlib/1.3.1: Generating aggregated env files + zlib/1.3.1: Generated aggregated env files: ['conanbuild.sh', 'conanrun.sh'] + zlib/1.3.1: Calling build() + zlib/1.3.1: Apply patch (conan): separate static/shared builds, disable debug suffix + zlib/1.3.1: Running CMake.configure() + zlib/1.3.1: RUN: cmake -G "Unix Makefiles" -DCMAKE_TOOLCHAIN_FILE="generators/conan_toolchain.cmake" -DCMAKE_INSTALL_PREFIX="/root/.conan2/p/b/zlib8dd8e27348e8c/p" -DCMAKE_POLICY_DEFAULT_CMP0091="NEW" -DCMAKE_BUILD_TYPE="Release" "/root/.conan2/p/b/zlib8dd8e27348e8c/b/src" + -- Using Conan toolchain: /root/.conan2/p/b/zlib8dd8e27348e8c/b/build/Release/generators/conan_toolchain.cmake + -- Conan toolchain: Setting CMAKE_POSITION_INDEPENDENT_CODE=ON (options.fPIC) + -- Conan toolchain: Setting BUILD_SHARED_LIBS = OFF + -- The C compiler identification is GNU 11.4.0 + -- Detecting C compiler ABI info + -- Detecting C compiler ABI info - done + -- Check for working C compiler: /usr/bin/cc - skipped + -- Detecting C compile features + -- Detecting C compile features - done + -- Looking for sys/types.h + -- Looking for sys/types.h - found + -- Looking for stdint.h + -- Looking for stdint.h - found + -- Looking for stddef.h + -- Looking for stddef.h - found + -- Check size of off64_t + -- Check size of off64_t - done + -- Looking for fseeko + -- Looking for fseeko - found + -- Looking for unistd.h + -- Looking for unistd.h - found + -- Renaming + -- /root/.conan2/p/b/zlib8dd8e27348e8c/b/src/zconf.h + -- to 'zconf.h.included' because this file is included with zlib + -- but CMake generates it automatically in the build directory. + -- Configuring done + -- Generating done + -- Build files have been written to: /root/.conan2/p/b/zlib8dd8e27348e8c/b/build/Release + zlib/1.3.1: Running CMake.build() + zlib/1.3.1: RUN: cmake --build "/root/.conan2/p/b/zlib8dd8e27348e8c/b/build/Release" -- -j16 + [ 12%] Building C object CMakeFiles/zlib.dir/adler32.c.o + [ 12%] Building C object CMakeFiles/zlib.dir/compress.c.o + [ 18%] Building C object CMakeFiles/zlib.dir/deflate.c.o + [ 25%] Building C object CMakeFiles/zlib.dir/crc32.c.o + [ 31%] Building C object CMakeFiles/zlib.dir/gzlib.c.o + [ 37%] Building C object CMakeFiles/zlib.dir/gzread.c.o + [ 43%] Building C object CMakeFiles/zlib.dir/gzclose.c.o + [ 56%] Building C object CMakeFiles/zlib.dir/infback.c.o + [ 56%] Building C object CMakeFiles/zlib.dir/gzwrite.c.o + [ 62%] Building C object CMakeFiles/zlib.dir/inflate.c.o + [ 68%] Building C object CMakeFiles/zlib.dir/inffast.c.o + [ 75%] Building C object CMakeFiles/zlib.dir/trees.c.o + [ 81%] Building C object CMakeFiles/zlib.dir/zutil.c.o + [ 87%] Building C object CMakeFiles/zlib.dir/uncompr.c.o + [ 93%] Building C object CMakeFiles/zlib.dir/inftrees.c.o + [100%] Linking C static library libz.a + [100%] Built target zlib + zlib/1.3.1: Package 'b647c43bfefae3f830561ca202b6cfd935b56205' built + zlib/1.3.1: Build folder /root/.conan2/p/b/zlib8dd8e27348e8c/b/build/Release + zlib/1.3.1: Generating the package + zlib/1.3.1: Packaging in folder /root/.conan2/p/b/zlib8dd8e27348e8c/p + zlib/1.3.1: Calling package() + zlib/1.3.1: Running CMake.install() + zlib/1.3.1: RUN: cmake --install "/root/.conan2/p/b/zlib8dd8e27348e8c/b/build/Release" --prefix "/root/.conan2/p/b/zlib8dd8e27348e8c/p" + -- Install configuration: "Release" + -- Installing: /root/.conan2/p/b/zlib8dd8e27348e8c/p/lib/libz.a + -- Installing: /root/.conan2/p/b/zlib8dd8e27348e8c/p/include/zconf.h + -- Installing: /root/.conan2/p/b/zlib8dd8e27348e8c/p/include/zlib.h + + zlib/1.3.1: package(): Packaged 1 file: LICENSE + zlib/1.3.1: package(): Packaged 2 '.h' files: zlib.h, zconf.h + zlib/1.3.1: package(): Packaged 1 '.a' file: libz.a + zlib/1.3.1: Created package revision fd85b1346d5377ae2465645768e62bf2 + zlib/1.3.1: Package 'b647c43bfefae3f830561ca202b6cfd935b56205' created + zlib/1.3.1: Full package reference: zlib/1.3.1#e20364c96c45455608a72543f3a53133:b647c43bfefae3f830561ca202b6cfd935b56205#fd85b1346d5377ae2465645768e62bf2 + zlib/1.3.1: Package folder /root/.conan2/p/b/zlib8dd8e27348e8c/p + WARN: deprecated: Usage of deprecated Conan 1.X features that will be removed in Conan 2.X: + WARN: deprecated: 'cpp_info.names' used in: zlib/1.3.1 + + ======== Launching test_package ======== + + ======== Computing dependency graph ======== + Graph root + zlib/1.3.1 (test package): /root/conanrunner/all/test_package/conanfile.py + Requirements + zlib/1.3.1#e20364c96c45455608a72543f3a53133 - Cache + + ======== Computing necessary packages ======== + Requirements + zlib/1.3.1#e20364c96c45455608a72543f3a53133:b647c43bfefae3f830561ca202b6cfd935b56205#fd85b1346d5377ae2465645768e62bf2 - Cache + + ======== Installing packages ======== + zlib/1.3.1: Already installed! (1 of 1) + WARN: deprecated: Usage of deprecated Conan 1.X features that will be removed in Conan 2.X: + WARN: deprecated: 'cpp_info.names' used in: zlib/1.3.1 + + ======== Testing the package ======== + Removing previously existing 'test_package' build folder: /root/conanrunner/all/test_package/build/gcc-11-x86_64-gnu17-release + zlib/1.3.1 (test package): Test package build: build/gcc-11-x86_64-gnu17-release + zlib/1.3.1 (test package): Test package build folder: /root/conanrunner/all/test_package/build/gcc-11-x86_64-gnu17-release + zlib/1.3.1 (test package): Writing generators to /root/conanrunner/all/test_package/build/gcc-11-x86_64-gnu17-release/generators + zlib/1.3.1 (test package): Generator 'CMakeToolchain' calling 'generate()' + zlib/1.3.1 (test package): CMakeToolchain generated: conan_toolchain.cmake + zlib/1.3.1 (test package): CMakeToolchain generated: /root/conanrunner/all/test_package/build/gcc-11-x86_64-gnu17-release/generators/CMakePresets.json + zlib/1.3.1 (test package): CMakeToolchain generated: /root/conanrunner/all/test_package/CMakeUserPresets.json + zlib/1.3.1 (test package): Generator 'CMakeDeps' calling 'generate()' + zlib/1.3.1 (test package): CMakeDeps necessary find_package() and targets for your CMakeLists.txt + find_package(ZLIB) + target_link_libraries(... ZLIB::ZLIB) + zlib/1.3.1 (test package): Generator 'VirtualRunEnv' calling 'generate()' + zlib/1.3.1 (test package): Generating aggregated env files + zlib/1.3.1 (test package): Generated aggregated env files: ['conanrun.sh', 'conanbuild.sh'] + + ======== Testing the package: Building ======== + zlib/1.3.1 (test package): Calling build() + zlib/1.3.1 (test package): Running CMake.configure() + zlib/1.3.1 (test package): RUN: cmake -G "Unix Makefiles" -DCMAKE_TOOLCHAIN_FILE="generators/conan_toolchain.cmake" -DCMAKE_INSTALL_PREFIX="/root/conanrunner/all/test_package" -DCMAKE_POLICY_DEFAULT_CMP0091="NEW" -DCMAKE_BUILD_TYPE="Release" "/root/conanrunner/all/test_package" + -- Using Conan toolchain: /root/conanrunner/all/test_package/build/gcc-11-x86_64-gnu17-release/generators/conan_toolchain.cmake + -- Conan toolchain: C++ Standard 17 with extensions ON + -- The C compiler identification is GNU 11.4.0 + -- Detecting C compiler ABI info + -- Detecting C compiler ABI info - done + -- Check for working C compiler: /usr/bin/cc - skipped + -- Detecting C compile features + -- Detecting C compile features - done + -- Conan: Target declared 'ZLIB::ZLIB' + -- Configuring done + -- Generating done + -- Build files have been written to: /root/conanrunner/all/test_package/build/gcc-11-x86_64-gnu17-release + zlib/1.3.1 (test package): Running CMake.build() + zlib/1.3.1 (test package): RUN: cmake --build "/root/conanrunner/all/test_package/build/gcc-11-x86_64-gnu17-release" -- -j16 + [ 50%] Building C object CMakeFiles/test_package.dir/test_package.c.o + [100%] Linking C executable test_package + [100%] Built target test_package + + ======== Testing the package: Executing test ======== + zlib/1.3.1 (test package): Running test() + zlib/1.3.1 (test package): RUN: ./test_package + Compressed size is: 21 + Compressed string is: Conan Package Manager + Compressed size is: 22 + Compressed string is: xsKHLNLOUMRE + ZLIB VERSION: 1.3.1 + +8. Copy just the package created inside the container using the ``pkglist.json`` info from the previous ``conan create``, restore this new package inside the host cache running a ``conan cache save`` and remove the container. + +.. code-block:: bash + + ┌────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐ + | Running in container: "conan cache save --list=pkglist.json --file "/root/conanrunner/all"/.conanrunner/docker_cache_save.tgz" | + └────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘ + + Saving zlib/1.3.1: p/zlib95420566fc0dd + Saving zlib/1.3.1:b647c43bfefae3f830561ca202b6cfd935b56205: p/b/zlib8dd8e27348e8c/p + Saving zlib/1.3.1:b647c43bfefae3f830561ca202b6cfd935b56205 metadata: p/b/zlib8dd8e27348e8c/d/metadata + Local Cache + zlib + zlib/1.3.1 + revisions + e20364c96c45455608a72543f3a53133 (2024-04-29 17:19:32 UTC) + packages + b647c43bfefae3f830561ca202b6cfd935b56205 + revisions + fd85b1346d5377ae2465645768e62bf2 + package_folder: p/b/zlib8dd8e27348e8c/p + metadata_folder: p/b/zlib8dd8e27348e8c/d/metadata + info + settings + os: Linux + arch: x86_64 + compiler: gcc + compiler.version: 11 + build_type: Release + options + fPIC: True + shared: False + recipe_folder: p/zlib95420566fc0dd + + + ┌──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐ + | Restore host cache from: /conan-center-index/recipes/zlib/all/.conanrunner/docker_cache_save.tgz | + └──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘ + + Restore: zlib/1.3.1 in p/zlib95420566fc0dd + Restore: zlib/1.3.1:b647c43bfefae3f830561ca202b6cfd935b56205 in p/b/zlib8dd8e27348e8c/p + Restore: zlib/1.3.1:b647c43bfefae3f830561ca202b6cfd935b56205 metadata in p/b/zlib8dd8e27348e8c/d/metadata + + ┌────────────────────┐ + | Stopping container | + └────────────────────┘ + + + ┌────────────────────┐ + | Removing container | + └────────────────────┘ + +If we now check the status of our conan and docker cache, we will see the new zlib package compile for Linux and the new docker image. We don't have any container because we define ``remove=true`` + +.. code-block:: bash + + $ conan list "*:*" + Found 1 pkg/version recipes matching * in local cache + Local Cache + zlib + zlib/1.3.1 + revisions + e20364c96c45455608a72543f3a53133 (2024-04-29 17:18:07 UTC) + packages + b647c43bfefae3f830561ca202b6cfd935b56205 + info + settings + arch: x86_64 + build_type: Release + compiler: gcc + compiler.version: 11 + os: Linux + options + fPIC: True + shared: False + + $ docker ps --all + CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES + + $ docker images + REPOSITORY TAG IMAGE ID CREATED SIZE + my-conan-runner latest 383b905f352e 22 minutes ago 531MB + ubuntu 22.04 437ec753bef3 12 days ago 77.9MB diff --git a/examples/runners/docker/configfile_build_args.rst b/examples/runners/docker/configfile_build_args.rst new file mode 100644 index 00000000000..3b74f557e90 --- /dev/null +++ b/examples/runners/docker/configfile_build_args.rst @@ -0,0 +1,190 @@ +.. _examples_runners_docker_configfile_build_args: + +Using a docker runner configfile to parameterize a Dockerfile +============================================================= + +In this example we are going to see how to use a docker runner configfile to define our Dockerfile base image. Let’s create two profiles and a Dockerfile inside our project folder. + +.. code-block:: bash + + $ cd + $ tree + . + ├── Dockerfile + ├── configfile + ├── docker_example_build + └── docker_example_host + +``docker_example_host`` profile + +.. code-block:: text + + [settings] + arch=x86_64 + build_type=Release + compiler=gcc + compiler.cppstd=gnu17 + compiler.libcxx=libstdc++11 + compiler.version=11 + os=Linux + [runner] + type=docker + configfile=/configfile + cache=copy + remove=false + +``docker_example_build`` profile + +.. code-block:: text + + [settings] + arch=x86_64 + build_type=Release + compiler=gcc + compiler.cppstd=gnu17 + compiler.libcxx=libstdc++11 + compiler.version=11 + os=Linux + +``configfile`` + +.. code-block:: yaml + + image: my-conan-runner-image + build: + dockerfile: + build_context: + build_args: + BASE_IMAGE: ubuntu:22.04 + run: + name: my-conan-runner-container + + +.. code-block:: docker + :caption: Dockerfile + + ARG BASE_IMAGE + FROM $BASE_IMAGE + RUN apt-get update \ + && DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends \ + build-essential \ + cmake \ + python3 \ + python3-pip \ + python3-venv \ + && rm -rf /var/lib/apt/lists/* + RUN pip install conan + +In this example we are going to start from a totally clean docker, without containers or images. In addition, we are going to have the conan cache also completely empty. + +.. code-block:: bash + + $ conan list "*:*" + Found 0 pkg/version recipes matching * in local cache + + $ docker ps --all + CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES + + $ docker images + REPOSITORY TAG IMAGE ID CREATED SIZE + + +Now, we are going to clone and build zlib from conan-center-index and create it using our new runner definition. + +.. code-block:: bash + + $ git clone https://github.com/conan-io/conan-center-index.git --depth 1 + $ conan create ./conan-center-index/recipes/zlib/all --version 1.3.1 -pr:h /docker_example_host -pr:b /docker_example_build + + ... + + ┌──────────────────────────────────────────────────┐ + | Building the Docker image: my-conan-runner-image | + └──────────────────────────────────────────────────┘ + + Dockerfile path: '/Dockerfile' + Docker build context: '' + + Step 1/5 : ARG BASE_IMAGE + + Step 2/5 : FROM $BASE_IMAGE + + ... + + Successfully built 286df085400f + Successfully tagged my-conan-runner-image:latest + + ... + + ┌───────────────────────────────┐ + | Creating the docker container | + └───────────────────────────────┘ + + + ┌─────────────────────────────────────────┐ + | Container my-conan-runner-image running | + └─────────────────────────────────────────┘ + + + ┌─────────────────────────────────────────┐ + | Running in container: "conan --version" | + └─────────────────────────────────────────┘ + + ┌──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐ + | Restore host cache from: /conan-center-index/recipes/zlib/all/.conanrunner/docker_cache_save.tgz | + └──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘ + + Restore: zlib/1.3.1 in p/zlib95420566fc0dd + Restore: zlib/1.3.1:b647c43bfefae3f830561ca202b6cfd935b56205 in p/zlibd59462fc4358e/p + Restore: zlib/1.3.1:b647c43bfefae3f830561ca202b6cfd935b56205 metadata in p/zlibd59462fc4358e/d/metadata + + ┌────────────────────┐ + | Stopping container | + └────────────────────┘ + +If we now check the status of our Conan and docker cache, we will see the zlib package compiled for Linux and the new docker image and container. + +.. code-block:: bash + + $ conan list "*:*" + Found 1 pkg/version recipes matching * in local cache + Local Cache + zlib + zlib/1.3.1 + revisions + e20364c96c45455608a72543f3a53133 (2024-04-29 17:18:07 UTC) + packages + b647c43bfefae3f830561ca202b6cfd935b56205 + info + settings + arch: x86_64 + build_type: Release + compiler: gcc + compiler.version: 11 + os: Linux + options + fPIC: True + shared: False + + $ docker ps --all + CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES + 1379072ae424 my-conan-runner-image "/bin/bash -c 'while…" 17 seconds ago Exited (137) 2 seconds ago my-conan-runner-image + + $ docker images + REPOSITORY TAG IMAGE ID CREATED SIZE + my-conan-runner latest 383b905f352e 22 minutes ago 531MB + ubuntu 22.04 437ec753bef3 12 days ago 77.9MB + +If we run the ``conan create`` command again we will see how Conan reuses the previous container because we have set ``remove=False``. + +.. code-block:: bash + + $ conan create ./conan-center-index/recipes/zlib/all --version 1.3.1 -pr:h /docker_example_host -pr:b /docker_example_build + + ... + + ┌───────────────────────────────┐ + | Starting the docker container | + └───────────────────────────────┘ + + ... \ No newline at end of file diff --git a/reference/runners/docker.rst b/reference/runners/docker.rst index 9ab28cd10c5..a26dfb5a887 100644 --- a/reference/runners/docker.rst +++ b/reference/runners/docker.rst @@ -51,667 +51,10 @@ If you need more controll over the container build and execution, you can define bind: /mnt/vol2 # The path to mount the volume inside the container mode: rw # rw to mount the volume read/write, or ro to mount it read-only. -Basic example -------------- +How to run a `conan create` in a runner +--------------------------------------- -In this example we are going to see how to compile zlib 1.3.1 inside docker. Let’s create two profiles and a Dockerfile inside our project folder. +In the following links you can find some examples about how to use a conan docker runner: -.. code-block:: bash - - $ cd - $ tree - . - ├── Dockerfile - ├── docker_example_build - └── docker_example_host - -``docker_example_host`` profile - -.. code-block:: text - - [settings] - arch=x86_64 - build_type=Release - compiler=gcc - compiler.cppstd=gnu17 - compiler.libcxx=libstdc++11 - compiler.version=11 - os=Linux - [runner] - type=docker - dockerfile= - cache=copy - remove=true - -``docker_example_build`` profile - -.. code-block:: text - - [settings] - arch=x86_64 - build_type=Release - compiler=gcc - compiler.cppstd=gnu17 - compiler.libcxx=libstdc++11 - compiler.version=11 - os=Linux - -``Dockerfile`` - -.. code-block:: docker - - FROM ubuntu:22.04 - RUN apt-get update \ - && DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends \ - build-essential \ - cmake \ - python3 \ - python3-pip \ - python3-venv \ - && rm -rf /var/lib/apt/lists/* - RUN pip install conan - -In this example we are going to start from a totally clean docker, without containers or images. In addition, we are going to have the conan cache also completely empty. - -.. code-block:: bash - - $ conan list "*:*" - Found 0 pkg/version recipes matching * in local cache - - $ docker ps --all - CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES - - $ docker images - REPOSITORY TAG IMAGE ID CREATED SIZE - - -Now, we are going to clone and build zlib from conan-center-index and create it using our new runner definition. - -.. code-block:: bash - - $ git clone https://github.com/conan-io/conan-center-index.git --depth=1 - $ conan create ./conan-center-index/recipes/zlib/all --version 1.3.1 -pr:h /docker_example_host -pr:b /docker_example_build - -If we split and analyze the command output, we can see what is happening and where the commands are being executed. - -1. Standard conan execution. - -.. code-block:: bash - - ======== Exporting recipe to the cache ======== - zlib/1.3.1: Exporting package recipe: /conan-center-index/recipes/zlib/all/conanfile.py - zlib/1.3.1: exports: File 'conandata.yml' found. Exporting it... - zlib/1.3.1: Calling export_sources() - zlib/1.3.1: Copied 1 '.py' file: conanfile.py - zlib/1.3.1: Copied 1 '.yml' file: conandata.yml - zlib/1.3.1: Copied 1 '.patch' file: 0001-fix-cmake.patch - zlib/1.3.1: Exported to cache folder: /Users/conan/.conan2/p/zlib95420566fc0dd/e - zlib/1.3.1: Exported: zlib/1.3.1#e20364c96c45455608a72543f3a53133 (2024-04-29 17:03:44 UTC) - - ======== Input profiles ======== - Profile host: - [settings] - arch=x86_64 - build_type=Release - compiler=gcc - compiler.cppstd=gnu17 - compiler.libcxx=libstdc++11 - compiler.version=11 - os=Linux - - Profile build: - [settings] - arch=x86_64 - build_type=Release - compiler=gcc - compiler.cppstd=gnu17 - compiler.libcxx=libstdc++11 - compiler.version=11 - os=Linux - -2. Build docker image - -.. code-block:: bash - - ┌────────────────────────────────────────────┐ - | Building the Docker image: my-conan-runner | - └────────────────────────────────────────────┘ - - Dockerfile path: '/Dockerfile' - Docker build context: '' - - Step 1/4 : FROM ubuntu:22.04 - - ... - - ---> dba927bb0517 - Successfully built dba927bb0517 - Successfully tagged my-conan-runner:latest - -3. Save the local cache running ``conan cache save``. - -.. code-block:: bash - - ┌────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐ - | Save host cache in: /conan-center-index/recipes/zlib/all/.conanrunner/local_cache_save.tgz | - └────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘ - - Found 1 pkg/version recipes matching * in local cache - Saving zlib/1.3.1: p/zlib95420566fc0dd - -4. Create and initialize the docker container. - -.. code-block:: bash - - ┌───────────────────────────────┐ - | Creating the docker container | - └───────────────────────────────┘ - - ┌───────────────────────────────────────┐ - | Container conan-runner-docker running | - └───────────────────────────────────────┘ - -5. Check if the container has a conan version with the runner feature. - -.. code-block:: bash - - ┌─────────────────────────────────────────┐ - | Running in container: "conan --version" | - └─────────────────────────────────────────┘ - - Conan version 2.3.0 - -6. Initialize the container conan cache using the host copy running ``conan cache restore``. - -.. code-block:: bash - - ┌───────────────────────────────────────────────────────────────────────────────────────────────────────┐ - | Running in container: "conan cache restore "/root/conanrunner/all/.conanrunner/local_cache_save.tgz"" | - └───────────────────────────────────────────────────────────────────────────────────────────────────────┘ - - Restore: zlib/1.3.1 in p/zlib95420566fc0dd - Local Cache - zlib - zlib/1.3.1 - revisions - e20364c96c45455608a72543f3a53133 (2024-04-29 17:19:32 UTC) - packages - recipe_folder: p/zlib95420566fc0dd - -7. Run the ``conan create`` inside the container and build zlib. - -.. code-block:: bash - - ┌───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐ - | Running in container: "conan create /root/conanrunner/all --version 1.3.1 -pr:h /root/conanrunner/all/.conanrunner/profiles/docker_example_host_1 -pr:b /root/conanrunner/all/.conanrunner/profiles/docker_example_build_0 -f json > create.json" | - └───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘ - - - ======== Exporting recipe to the cache ======== - zlib/1.3.1: Exporting package recipe: /root/conanrunner/all/conanfile.py - zlib/1.3.1: exports: File 'conandata.yml' found. Exporting it... - zlib/1.3.1: Calling export_sources() - zlib/1.3.1: Copied 1 '.yml' file: conandata.yml - zlib/1.3.1: Copied 1 '.py' file: conanfile.py - zlib/1.3.1: Copied 1 '.patch' file: 0001-fix-cmake.patch - zlib/1.3.1: Exported to cache folder: /root/.conan2/p/zlib95420566fc0dd/e - zlib/1.3.1: Exported: zlib/1.3.1#e20364c96c45455608a72543f3a53133 (2024-04-29 17:19:32 UTC) - - ======== Input profiles ======== - Profile host: - [settings] - arch=x86_64 - build_type=Release - compiler=gcc - compiler.cppstd=gnu17 - compiler.libcxx=libstdc++11 - compiler.version=11 - os=Linux - - Profile build: - [settings] - arch=x86_64 - build_type=Release - compiler=gcc - compiler.cppstd=gnu17 - compiler.libcxx=libstdc++11 - compiler.version=11 - os=Linux - - - ======== Computing dependency graph ======== - Graph root - cli - Requirements - zlib/1.3.1#e20364c96c45455608a72543f3a53133 - Cache - - ======== Computing necessary packages ======== - zlib/1.3.1: Forced build from source - Requirements - zlib/1.3.1#e20364c96c45455608a72543f3a53133:b647c43bfefae3f830561ca202b6cfd935b56205 - Build - - ======== Installing packages ======== - zlib/1.3.1: Calling source() in /root/.conan2/p/zlib95420566fc0dd/s/src - - -------- Installing package zlib/1.3.1 (1 of 1) -------- - zlib/1.3.1: Building from source - zlib/1.3.1: Package zlib/1.3.1:b647c43bfefae3f830561ca202b6cfd935b56205 - zlib/1.3.1: Copying sources to build folder - zlib/1.3.1: Building your package in /root/.conan2/p/b/zlib8dd8e27348e8c/b - zlib/1.3.1: Calling generate() - zlib/1.3.1: Generators folder: /root/.conan2/p/b/zlib8dd8e27348e8c/b/build/Release/generators - zlib/1.3.1: CMakeToolchain generated: conan_toolchain.cmake - zlib/1.3.1: CMakeToolchain generated: /root/.conan2/p/b/zlib8dd8e27348e8c/b/build/Release/generators/CMakePresets.json - zlib/1.3.1: CMakeToolchain generated: /root/.conan2/p/b/zlib8dd8e27348e8c/b/src/CMakeUserPresets.json - zlib/1.3.1: Generating aggregated env files - zlib/1.3.1: Generated aggregated env files: ['conanbuild.sh', 'conanrun.sh'] - zlib/1.3.1: Calling build() - zlib/1.3.1: Apply patch (conan): separate static/shared builds, disable debug suffix - zlib/1.3.1: Running CMake.configure() - zlib/1.3.1: RUN: cmake -G "Unix Makefiles" -DCMAKE_TOOLCHAIN_FILE="generators/conan_toolchain.cmake" -DCMAKE_INSTALL_PREFIX="/root/.conan2/p/b/zlib8dd8e27348e8c/p" -DCMAKE_POLICY_DEFAULT_CMP0091="NEW" -DCMAKE_BUILD_TYPE="Release" "/root/.conan2/p/b/zlib8dd8e27348e8c/b/src" - -- Using Conan toolchain: /root/.conan2/p/b/zlib8dd8e27348e8c/b/build/Release/generators/conan_toolchain.cmake - -- Conan toolchain: Setting CMAKE_POSITION_INDEPENDENT_CODE=ON (options.fPIC) - -- Conan toolchain: Setting BUILD_SHARED_LIBS = OFF - -- The C compiler identification is GNU 11.4.0 - -- Detecting C compiler ABI info - -- Detecting C compiler ABI info - done - -- Check for working C compiler: /usr/bin/cc - skipped - -- Detecting C compile features - -- Detecting C compile features - done - -- Looking for sys/types.h - -- Looking for sys/types.h - found - -- Looking for stdint.h - -- Looking for stdint.h - found - -- Looking for stddef.h - -- Looking for stddef.h - found - -- Check size of off64_t - -- Check size of off64_t - done - -- Looking for fseeko - -- Looking for fseeko - found - -- Looking for unistd.h - -- Looking for unistd.h - found - -- Renaming - -- /root/.conan2/p/b/zlib8dd8e27348e8c/b/src/zconf.h - -- to 'zconf.h.included' because this file is included with zlib - -- but CMake generates it automatically in the build directory. - -- Configuring done - -- Generating done - -- Build files have been written to: /root/.conan2/p/b/zlib8dd8e27348e8c/b/build/Release - zlib/1.3.1: Running CMake.build() - zlib/1.3.1: RUN: cmake --build "/root/.conan2/p/b/zlib8dd8e27348e8c/b/build/Release" -- -j16 - [ 12%] Building C object CMakeFiles/zlib.dir/adler32.c.o - [ 12%] Building C object CMakeFiles/zlib.dir/compress.c.o - [ 18%] Building C object CMakeFiles/zlib.dir/deflate.c.o - [ 25%] Building C object CMakeFiles/zlib.dir/crc32.c.o - [ 31%] Building C object CMakeFiles/zlib.dir/gzlib.c.o - [ 37%] Building C object CMakeFiles/zlib.dir/gzread.c.o - [ 43%] Building C object CMakeFiles/zlib.dir/gzclose.c.o - [ 56%] Building C object CMakeFiles/zlib.dir/infback.c.o - [ 56%] Building C object CMakeFiles/zlib.dir/gzwrite.c.o - [ 62%] Building C object CMakeFiles/zlib.dir/inflate.c.o - [ 68%] Building C object CMakeFiles/zlib.dir/inffast.c.o - [ 75%] Building C object CMakeFiles/zlib.dir/trees.c.o - [ 81%] Building C object CMakeFiles/zlib.dir/zutil.c.o - [ 87%] Building C object CMakeFiles/zlib.dir/uncompr.c.o - [ 93%] Building C object CMakeFiles/zlib.dir/inftrees.c.o - [100%] Linking C static library libz.a - [100%] Built target zlib - zlib/1.3.1: Package 'b647c43bfefae3f830561ca202b6cfd935b56205' built - zlib/1.3.1: Build folder /root/.conan2/p/b/zlib8dd8e27348e8c/b/build/Release - zlib/1.3.1: Generating the package - zlib/1.3.1: Packaging in folder /root/.conan2/p/b/zlib8dd8e27348e8c/p - zlib/1.3.1: Calling package() - zlib/1.3.1: Running CMake.install() - zlib/1.3.1: RUN: cmake --install "/root/.conan2/p/b/zlib8dd8e27348e8c/b/build/Release" --prefix "/root/.conan2/p/b/zlib8dd8e27348e8c/p" - -- Install configuration: "Release" - -- Installing: /root/.conan2/p/b/zlib8dd8e27348e8c/p/lib/libz.a - -- Installing: /root/.conan2/p/b/zlib8dd8e27348e8c/p/include/zconf.h - -- Installing: /root/.conan2/p/b/zlib8dd8e27348e8c/p/include/zlib.h - - zlib/1.3.1: package(): Packaged 1 file: LICENSE - zlib/1.3.1: package(): Packaged 2 '.h' files: zlib.h, zconf.h - zlib/1.3.1: package(): Packaged 1 '.a' file: libz.a - zlib/1.3.1: Created package revision fd85b1346d5377ae2465645768e62bf2 - zlib/1.3.1: Package 'b647c43bfefae3f830561ca202b6cfd935b56205' created - zlib/1.3.1: Full package reference: zlib/1.3.1#e20364c96c45455608a72543f3a53133:b647c43bfefae3f830561ca202b6cfd935b56205#fd85b1346d5377ae2465645768e62bf2 - zlib/1.3.1: Package folder /root/.conan2/p/b/zlib8dd8e27348e8c/p - WARN: deprecated: Usage of deprecated Conan 1.X features that will be removed in Conan 2.X: - WARN: deprecated: 'cpp_info.names' used in: zlib/1.3.1 - - ======== Launching test_package ======== - - ======== Computing dependency graph ======== - Graph root - zlib/1.3.1 (test package): /root/conanrunner/all/test_package/conanfile.py - Requirements - zlib/1.3.1#e20364c96c45455608a72543f3a53133 - Cache - - ======== Computing necessary packages ======== - Requirements - zlib/1.3.1#e20364c96c45455608a72543f3a53133:b647c43bfefae3f830561ca202b6cfd935b56205#fd85b1346d5377ae2465645768e62bf2 - Cache - - ======== Installing packages ======== - zlib/1.3.1: Already installed! (1 of 1) - WARN: deprecated: Usage of deprecated Conan 1.X features that will be removed in Conan 2.X: - WARN: deprecated: 'cpp_info.names' used in: zlib/1.3.1 - - ======== Testing the package ======== - Removing previously existing 'test_package' build folder: /root/conanrunner/all/test_package/build/gcc-11-x86_64-gnu17-release - zlib/1.3.1 (test package): Test package build: build/gcc-11-x86_64-gnu17-release - zlib/1.3.1 (test package): Test package build folder: /root/conanrunner/all/test_package/build/gcc-11-x86_64-gnu17-release - zlib/1.3.1 (test package): Writing generators to /root/conanrunner/all/test_package/build/gcc-11-x86_64-gnu17-release/generators - zlib/1.3.1 (test package): Generator 'CMakeToolchain' calling 'generate()' - zlib/1.3.1 (test package): CMakeToolchain generated: conan_toolchain.cmake - zlib/1.3.1 (test package): CMakeToolchain generated: /root/conanrunner/all/test_package/build/gcc-11-x86_64-gnu17-release/generators/CMakePresets.json - zlib/1.3.1 (test package): CMakeToolchain generated: /root/conanrunner/all/test_package/CMakeUserPresets.json - zlib/1.3.1 (test package): Generator 'CMakeDeps' calling 'generate()' - zlib/1.3.1 (test package): CMakeDeps necessary find_package() and targets for your CMakeLists.txt - find_package(ZLIB) - target_link_libraries(... ZLIB::ZLIB) - zlib/1.3.1 (test package): Generator 'VirtualRunEnv' calling 'generate()' - zlib/1.3.1 (test package): Generating aggregated env files - zlib/1.3.1 (test package): Generated aggregated env files: ['conanrun.sh', 'conanbuild.sh'] - - ======== Testing the package: Building ======== - zlib/1.3.1 (test package): Calling build() - zlib/1.3.1 (test package): Running CMake.configure() - zlib/1.3.1 (test package): RUN: cmake -G "Unix Makefiles" -DCMAKE_TOOLCHAIN_FILE="generators/conan_toolchain.cmake" -DCMAKE_INSTALL_PREFIX="/root/conanrunner/all/test_package" -DCMAKE_POLICY_DEFAULT_CMP0091="NEW" -DCMAKE_BUILD_TYPE="Release" "/root/conanrunner/all/test_package" - -- Using Conan toolchain: /root/conanrunner/all/test_package/build/gcc-11-x86_64-gnu17-release/generators/conan_toolchain.cmake - -- Conan toolchain: C++ Standard 17 with extensions ON - -- The C compiler identification is GNU 11.4.0 - -- Detecting C compiler ABI info - -- Detecting C compiler ABI info - done - -- Check for working C compiler: /usr/bin/cc - skipped - -- Detecting C compile features - -- Detecting C compile features - done - -- Conan: Target declared 'ZLIB::ZLIB' - -- Configuring done - -- Generating done - -- Build files have been written to: /root/conanrunner/all/test_package/build/gcc-11-x86_64-gnu17-release - zlib/1.3.1 (test package): Running CMake.build() - zlib/1.3.1 (test package): RUN: cmake --build "/root/conanrunner/all/test_package/build/gcc-11-x86_64-gnu17-release" -- -j16 - [ 50%] Building C object CMakeFiles/test_package.dir/test_package.c.o - [100%] Linking C executable test_package - [100%] Built target test_package - - ======== Testing the package: Executing test ======== - zlib/1.3.1 (test package): Running test() - zlib/1.3.1 (test package): RUN: ./test_package - Compressed size is: 21 - Compressed string is: Conan Package Manager - Compressed size is: 22 - Compressed string is: xsKHLNLOUMRE - ZLIB VERSION: 1.3.1 - -8. Copy just the package created inside the container using the ``pkglist.json`` info from the previous ``conan create``, restore this new package inside the host cache running a ``conan cache save`` and remove the container. - -.. code-block:: bash - - ┌────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐ - | Running in container: "conan cache save --list=pkglist.json --file "/root/conanrunner/all"/.conanrunner/docker_cache_save.tgz" | - └────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘ - - Saving zlib/1.3.1: p/zlib95420566fc0dd - Saving zlib/1.3.1:b647c43bfefae3f830561ca202b6cfd935b56205: p/b/zlib8dd8e27348e8c/p - Saving zlib/1.3.1:b647c43bfefae3f830561ca202b6cfd935b56205 metadata: p/b/zlib8dd8e27348e8c/d/metadata - Local Cache - zlib - zlib/1.3.1 - revisions - e20364c96c45455608a72543f3a53133 (2024-04-29 17:19:32 UTC) - packages - b647c43bfefae3f830561ca202b6cfd935b56205 - revisions - fd85b1346d5377ae2465645768e62bf2 - package_folder: p/b/zlib8dd8e27348e8c/p - metadata_folder: p/b/zlib8dd8e27348e8c/d/metadata - info - settings - os: Linux - arch: x86_64 - compiler: gcc - compiler.version: 11 - build_type: Release - options - fPIC: True - shared: False - recipe_folder: p/zlib95420566fc0dd - - - ┌──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐ - | Restore host cache from: /conan-center-index/recipes/zlib/all/.conanrunner/docker_cache_save.tgz | - └──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘ - - Restore: zlib/1.3.1 in p/zlib95420566fc0dd - Restore: zlib/1.3.1:b647c43bfefae3f830561ca202b6cfd935b56205 in p/b/zlib8dd8e27348e8c/p - Restore: zlib/1.3.1:b647c43bfefae3f830561ca202b6cfd935b56205 metadata in p/b/zlib8dd8e27348e8c/d/metadata - - ┌────────────────────┐ - | Stopping container | - └────────────────────┘ - - - ┌────────────────────┐ - | Removing container | - └────────────────────┘ - -If we now check the status of our conan and docker cache, we will see the new zlib package compile for Linux and the new docker image. We don't have any container because we define ``remove=true`` - -.. code-block:: bash - - $ conan list "*:*" - Found 1 pkg/version recipes matching * in local cache - Local Cache - zlib - zlib/1.3.1 - revisions - e20364c96c45455608a72543f3a53133 (2024-04-29 17:18:07 UTC) - packages - b647c43bfefae3f830561ca202b6cfd935b56205 - info - settings - arch: x86_64 - build_type: Release - compiler: gcc - compiler.version: 11 - os: Linux - options - fPIC: True - shared: False - - $ docker ps --all - CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES - - $ docker images - REPOSITORY TAG IMAGE ID CREATED SIZE - my-conan-runner latest 383b905f352e 22 minutes ago 531MB - ubuntu 22.04 437ec753bef3 12 days ago 77.9MB - -Config file example -------------------- - -In this example we are going to see how to use a docker runner configfile. Let’s create two profiles and a Dockerfile inside our project folder. - -.. code-block:: bash - - $ cd - $ tree - . - ├── Dockerfile - ├── configfile - ├── docker_example_build - └── docker_example_host - -``docker_example_host`` profile - -.. code-block:: text - - [settings] - arch=x86_64 - build_type=Release - compiler=gcc - compiler.cppstd=gnu17 - compiler.libcxx=libstdc++11 - compiler.version=11 - os=Linux - [runner] - type=docker - configfile=/configfile - cache=copy - remove=false - -``docker_example_build`` profile - -.. code-block:: text - - [settings] - arch=x86_64 - build_type=Release - compiler=gcc - compiler.cppstd=gnu17 - compiler.libcxx=libstdc++11 - compiler.version=11 - os=Linux - -``configfile`` - -.. code-block:: yaml - - image: my-conan-runner-image - build: - dockerfile: - build_context: - build_args: - BASE_IMAGE: ubuntu:22.04 - run: - name: my-conan-runner-container - - -.. code-block:: docker - :caption: Dockerfile - - ARG BASE_IMAGE - FROM $BASE_IMAGE - RUN apt-get update \ - && DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends \ - build-essential \ - cmake \ - python3 \ - python3-pip \ - python3-venv \ - && rm -rf /var/lib/apt/lists/* - RUN pip install conan - -In this example we are going to start from a totally clean docker, without containers or images. In addition, we are going to have the conan cache also completely empty. - -.. code-block:: bash - - $ conan list "*:*" - Found 0 pkg/version recipes matching * in local cache - - $ docker ps --all - CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES - - $ docker images - REPOSITORY TAG IMAGE ID CREATED SIZE - - -Now, we are going to clone and build zlib from conan-center-index and create it using our new runner definition. - -.. code-block:: bash - - $ git clone https://github.com/conan-io/conan-center-index.git --depth 1 - $ conan create ./conan-center-index/recipes/zlib/all --version 1.3.1 -pr:h /docker_example_host -pr:b /docker_example_build - - ... - - ┌──────────────────────────────────────────────────┐ - | Building the Docker image: my-conan-runner-image | - └──────────────────────────────────────────────────┘ - - Dockerfile path: '/Dockerfile' - Docker build context: '' - - Step 1/5 : ARG BASE_IMAGE - - Step 2/5 : FROM $BASE_IMAGE - - ... - - Successfully built 286df085400f - Successfully tagged my-conan-runner-image:latest - - ... - - ┌───────────────────────────────┐ - | Creating the docker container | - └───────────────────────────────┘ - - - ┌─────────────────────────────────────────┐ - | Container my-conan-runner-image running | - └─────────────────────────────────────────┘ - - - ┌─────────────────────────────────────────┐ - | Running in container: "conan --version" | - └─────────────────────────────────────────┘ - - ┌──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐ - | Restore host cache from: /conan-center-index/recipes/zlib/all/.conanrunner/docker_cache_save.tgz | - └──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘ - - Restore: zlib/1.3.1 in p/zlib95420566fc0dd - Restore: zlib/1.3.1:b647c43bfefae3f830561ca202b6cfd935b56205 in p/zlibd59462fc4358e/p - Restore: zlib/1.3.1:b647c43bfefae3f830561ca202b6cfd935b56205 metadata in p/zlibd59462fc4358e/d/metadata - - ┌────────────────────┐ - | Stopping container | - └────────────────────┘ - -If we now check the status of our Conan and docker cache, we will see the zlib package compiled for Linux and the new docker image and container. - -.. code-block:: bash - - $ conan list "*:*" - Found 1 pkg/version recipes matching * in local cache - Local Cache - zlib - zlib/1.3.1 - revisions - e20364c96c45455608a72543f3a53133 (2024-04-29 17:18:07 UTC) - packages - b647c43bfefae3f830561ca202b6cfd935b56205 - info - settings - arch: x86_64 - build_type: Release - compiler: gcc - compiler.version: 11 - os: Linux - options - fPIC: True - shared: False - - $ docker ps --all - CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES - 1379072ae424 my-conan-runner-image "/bin/bash -c 'while…" 17 seconds ago Exited (137) 2 seconds ago my-conan-runner-image - - $ docker images - REPOSITORY TAG IMAGE ID CREATED SIZE - my-conan-runner latest 383b905f352e 22 minutes ago 531MB - ubuntu 22.04 437ec753bef3 12 days ago 77.9MB - -If we run the ``conan create`` command again we will see how Conan reuses the previous container because we have set ``remove=False``. - -.. code-block:: bash - - $ conan create ./conan-center-index/recipes/zlib/all --version 1.3.1 -pr:h /docker_example_host -pr:b /docker_example_build - - ... - - ┌───────────────────────────────┐ - | Starting the docker container | - └───────────────────────────────┘ - - ... \ No newline at end of file +- :ref:`Compile zlib using a docker runner` +- :ref:`Using a docker runner configfile to parameterize the Dockerfile base image` \ No newline at end of file From 23095dc0baded13b1e442d62dbaa2cfd27b12411 Mon Sep 17 00:00:00 2001 From: David Sanchez Date: Mon, 6 May 2024 17:12:57 +0200 Subject: [PATCH 21/31] basic docker runner example updated --- examples/runners/docker/basic.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/examples/runners/docker/basic.rst b/examples/runners/docker/basic.rst index 5ee977c7100..0eeed3693c1 100644 --- a/examples/runners/docker/basic.rst +++ b/examples/runners/docker/basic.rst @@ -1,7 +1,7 @@ .. _examples_runners_docker_basic: -Using docker runner -=================== +Compile zlib/1.3.1 using docker runner +====================================== In this example we are going to see how to compile zlib 1.3.1 inside docker. Let’s create two profiles and a Dockerfile inside our project folder. From da32acb97702c31547237e25ae92bda124f60163 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?David=20S=C3=A1nchez=20Falero?= Date: Mon, 6 May 2024 17:23:42 +0200 Subject: [PATCH 22/31] Update reference/runners.rst MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-authored-by: Rubén Rincón Blanco --- reference/runners.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/reference/runners.rst b/reference/runners.rst index 1a723910b88..bf1a2001f69 100644 --- a/reference/runners.rst +++ b/reference/runners.rst @@ -5,7 +5,7 @@ Runners Runners are the way to run Conan remotely and transparently just by modifying the host profile. There are three requirements to be able to use the feature: -- Installing a version of conan with runner dependencies ``pip install conan[runners]``. +- Installing a version of Conan with runner dependencies ``pip install conan[runners]``. - Install the tools to run each of the runners (``docker``). - Add the ``[runner]`` section defined in the documentation of each runner to the host profile. From c9fb3263f1f67e5c8d6b9005898a9bad6d62bbbd Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?David=20S=C3=A1nchez=20Falero?= Date: Mon, 6 May 2024 17:23:49 +0200 Subject: [PATCH 23/31] Update reference/runners/docker.rst MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-authored-by: Rubén Rincón Blanco --- reference/runners/docker.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/reference/runners/docker.rst b/reference/runners/docker.rst index a26dfb5a887..d9ecf761304 100644 --- a/reference/runners/docker.rst +++ b/reference/runners/docker.rst @@ -9,7 +9,7 @@ How to use a docker runner To run Conan inside a docker container you need to define a ``[runner]`` section in your host profile using the following fields: - ``type`` **(mandatory)**: define the runner we want to use, in this case ``docker``. -- ``dockerfile`` **(optional, default None)**: define the Dockerfile absolute path in case you want to build a docker image. +- ``dockerfile`` **(optional, default None)**: absolute path to a Dockerfile in case you want to build a docker image. - ``image`` **(optional, default conan-runner-default)**: docker image name you want to download from a docker registry or the name of the built image in case you define a dockerfile path. - ``cache`` **(optional, default clean)**: how docker container uses (or not) the host cache. From 453ae1bb3d9614afef1e40f44fc0fbfe6baf0af5 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?David=20S=C3=A1nchez=20Falero?= Date: Mon, 6 May 2024 17:23:56 +0200 Subject: [PATCH 24/31] Update reference/runners/docker.rst MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-authored-by: Rubén Rincón Blanco --- reference/runners/docker.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/reference/runners/docker.rst b/reference/runners/docker.rst index d9ecf761304..7cc4f682e9b 100644 --- a/reference/runners/docker.rst +++ b/reference/runners/docker.rst @@ -11,7 +11,7 @@ To run Conan inside a docker container you need to define a ``[runner]`` section - ``type`` **(mandatory)**: define the runner we want to use, in this case ``docker``. - ``dockerfile`` **(optional, default None)**: absolute path to a Dockerfile in case you want to build a docker image. - ``image`` **(optional, default conan-runner-default)**: docker image name you want to download from a docker registry or the name of the built image in case you define a dockerfile path. -- ``cache`` **(optional, default clean)**: how docker container uses (or not) the host cache. +- ``cache`` **(optional, default clean)**: how docker container uses (or not) the host's Conan cache. - ``clean``: use an empty cache. - ``copy``: copy the host cache inside the container using the :ref:`conan cache save/restore` command. From 03ce453647cbc0722668a202f9455e2271f40a9a Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?David=20S=C3=A1nchez=20Falero?= Date: Mon, 6 May 2024 17:24:05 +0200 Subject: [PATCH 25/31] Update reference/runners/docker.rst MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-authored-by: Rubén Rincón Blanco --- reference/runners/docker.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/reference/runners/docker.rst b/reference/runners/docker.rst index 7cc4f682e9b..5e110911bfa 100644 --- a/reference/runners/docker.rst +++ b/reference/runners/docker.rst @@ -18,7 +18,7 @@ To run Conan inside a docker container you need to define a ``[runner]`` section - ``shared``: mount the host conan cache as a shared volume. - ``remove`` **(optional, default false)**: ``true`` or ``false``. Remove the container after running the conan command. -- ``configfile`` **(optional, default None)**: configuration file absolute path with extra parameters (see **extra configuration** section for more info). +- ``configfile`` **(optional, default None)**: Absolute path to a configuration file with extra parameters (see **extra configuration** section for more info). Extra configuration ------------------- From ff3b5f40d1a4795876ca7e652accd71a0204d950 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?David=20S=C3=A1nchez=20Falero?= Date: Mon, 6 May 2024 17:24:32 +0200 Subject: [PATCH 26/31] Update reference/runners/docker.rst MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-authored-by: Rubén Rincón Blanco --- reference/runners/docker.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/reference/runners/docker.rst b/reference/runners/docker.rst index 5e110911bfa..f6a87028aa8 100644 --- a/reference/runners/docker.rst +++ b/reference/runners/docker.rst @@ -23,7 +23,7 @@ To run Conan inside a docker container you need to define a ``[runner]`` section Extra configuration ------------------- -If you need more controll over the container build and execution, you can define more parameters inside a ``configfile`` yaml. +If you need more control over the build and execution of the container, you can define more parameters inside a ``configfile`` yaml. .. code-block:: yaml From e604cf8c5823fe1806b067106aaea58daec2588d Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?David=20S=C3=A1nchez=20Falero?= Date: Mon, 6 May 2024 17:24:49 +0200 Subject: [PATCH 27/31] Update reference/runners/docker.rst MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-authored-by: Rubén Rincón Blanco --- reference/runners/docker.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/reference/runners/docker.rst b/reference/runners/docker.rst index f6a87028aa8..399d1b3f867 100644 --- a/reference/runners/docker.rst +++ b/reference/runners/docker.rst @@ -15,7 +15,7 @@ To run Conan inside a docker container you need to define a ``[runner]`` section - ``clean``: use an empty cache. - ``copy``: copy the host cache inside the container using the :ref:`conan cache save/restore` command. - - ``shared``: mount the host conan cache as a shared volume. + - ``shared``: mount the host's Conan cache as a shared volume. - ``remove`` **(optional, default false)**: ``true`` or ``false``. Remove the container after running the conan command. - ``configfile`` **(optional, default None)**: Absolute path to a configuration file with extra parameters (see **extra configuration** section for more info). From 7036998726ec052a2b9d38c4c94f473bd52094b6 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?David=20S=C3=A1nchez=20Falero?= Date: Mon, 6 May 2024 17:24:56 +0200 Subject: [PATCH 28/31] Update reference/runners/docker.rst MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-authored-by: Rubén Rincón Blanco --- reference/runners/docker.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/reference/runners/docker.rst b/reference/runners/docker.rst index 399d1b3f867..a427055447a 100644 --- a/reference/runners/docker.rst +++ b/reference/runners/docker.rst @@ -17,7 +17,7 @@ To run Conan inside a docker container you need to define a ``[runner]`` section - ``copy``: copy the host cache inside the container using the :ref:`conan cache save/restore` command. - ``shared``: mount the host's Conan cache as a shared volume. -- ``remove`` **(optional, default false)**: ``true`` or ``false``. Remove the container after running the conan command. +- ``remove`` **(optional, default False)**: ``True`` or ``False``. Remove the container after running the Conan command. - ``configfile`` **(optional, default None)**: Absolute path to a configuration file with extra parameters (see **extra configuration** section for more info). Extra configuration From 05235e00824eed9d171803bb7f6eda45f95700ce Mon Sep 17 00:00:00 2001 From: David Sanchez Date: Mon, 6 May 2024 17:33:22 +0200 Subject: [PATCH 29/31] note about package id added --- reference/runners.rst | 2 +- reference/runners/docker.rst | 6 +++++- 2 files changed, 6 insertions(+), 2 deletions(-) diff --git a/reference/runners.rst b/reference/runners.rst index bf1a2001f69..27ec21b61e0 100644 --- a/reference/runners.rst +++ b/reference/runners.rst @@ -3,7 +3,7 @@ Runners ======= -Runners are the way to run Conan remotely and transparently just by modifying the host profile. There are three requirements to be able to use the feature: +Runners provide a seamless method to execute Conan on remote build environments like Docker ones, directly from your local setup by simply configuring your host profile. - Installing a version of Conan with runner dependencies ``pip install conan[runners]``. - Install the tools to run each of the runners (``docker``). diff --git a/reference/runners/docker.rst b/reference/runners/docker.rst index a427055447a..a7269263e9d 100644 --- a/reference/runners/docker.rst +++ b/reference/runners/docker.rst @@ -17,9 +17,13 @@ To run Conan inside a docker container you need to define a ``[runner]`` section - ``copy``: copy the host cache inside the container using the :ref:`conan cache save/restore` command. - ``shared``: mount the host's Conan cache as a shared volume. -- ``remove`` **(optional, default False)**: ``True`` or ``False``. Remove the container after running the Conan command. +- ``remove`` **(optional, default false)**: ``true`` or ``false``. Remove the container after running the Conan command. - ``configfile`` **(optional, default None)**: Absolute path to a configuration file with extra parameters (see **extra configuration** section for more info). +.. note:: + + The runner profile section doesn't affect the package id. + Extra configuration ------------------- From 5af6f74794be0ee2e5bbb15c9bf63fb0afa5dc9a Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?David=20S=C3=A1nchez=20Falero?= Date: Mon, 6 May 2024 17:35:49 +0200 Subject: [PATCH 30/31] Update examples/runners/docker/basic.rst Co-authored-by: Carlos Zoido --- examples/runners/docker/basic.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/examples/runners/docker/basic.rst b/examples/runners/docker/basic.rst index 0eeed3693c1..8070a5399be 100644 --- a/examples/runners/docker/basic.rst +++ b/examples/runners/docker/basic.rst @@ -3,7 +3,7 @@ Compile zlib/1.3.1 using docker runner ====================================== -In this example we are going to see how to compile zlib 1.3.1 inside docker. Let’s create two profiles and a Dockerfile inside our project folder. +In this example we are going to see how to create the ``zlib/1.3.1`` Conan packge inside Docker using a runner. Let’s create two profiles and a Dockerfile inside our project folder. .. code-block:: bash From 1a53d9aef20fd73676532f20e3f18f0078080a1f Mon Sep 17 00:00:00 2001 From: David Sanchez Date: Mon, 6 May 2024 17:36:39 +0200 Subject: [PATCH 31/31] conan docker runner basic example updated --- examples/runners/docker/basic.rst | 4 ++-- reference/runners/docker.rst | 2 +- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/examples/runners/docker/basic.rst b/examples/runners/docker/basic.rst index 8070a5399be..9136d6f9602 100644 --- a/examples/runners/docker/basic.rst +++ b/examples/runners/docker/basic.rst @@ -1,7 +1,7 @@ .. _examples_runners_docker_basic: -Compile zlib/1.3.1 using docker runner -====================================== +Creating a Conan package using a Docker runner +============================================== In this example we are going to see how to create the ``zlib/1.3.1`` Conan packge inside Docker using a runner. Let’s create two profiles and a Dockerfile inside our project folder. diff --git a/reference/runners/docker.rst b/reference/runners/docker.rst index a7269263e9d..9a51dc57733 100644 --- a/reference/runners/docker.rst +++ b/reference/runners/docker.rst @@ -60,5 +60,5 @@ How to run a `conan create` in a runner In the following links you can find some examples about how to use a conan docker runner: -- :ref:`Compile zlib using a docker runner` +- :ref:`Creating a Conan package using a Docker runner` - :ref:`Using a docker runner configfile to parameterize the Dockerfile base image` \ No newline at end of file