Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

conan runner docs added #3699

Merged
merged 31 commits into from
May 6, 2024
Merged
Show file tree
Hide file tree
Changes from 21 commits
Commits
Show all changes
31 commits
Select commit Hold shift + click to select a range
7ef30c4
conan runner docs added
davidsanfal Apr 29, 2024
3b6a0ce
Update reference/runners.rst
davidsanfal May 6, 2024
52afd3a
Update reference/runners.rst
davidsanfal May 6, 2024
493133b
Update reference/runners/docker.rst
davidsanfal May 6, 2024
cf7e0d0
Update reference/runners/docker.rst
davidsanfal May 6, 2024
a8af193
Update reference/runners/docker.rst
davidsanfal May 6, 2024
1a0f762
Update reference/runners/docker.rst
davidsanfal May 6, 2024
51e4a1a
Update reference/runners/docker.rst
davidsanfal May 6, 2024
eb22c6a
Update reference/runners/docker.rst
davidsanfal May 6, 2024
b6c2387
Update reference/runners/docker.rst
davidsanfal May 6, 2024
6b27ee6
Update reference/runners/docker.rst
davidsanfal May 6, 2024
f8ad316
Update reference/runners/docker.rst
davidsanfal May 6, 2024
9b0df23
Update reference/runners/docker.rst
davidsanfal May 6, 2024
24b94fb
Update reference/runners/docker.rst
davidsanfal May 6, 2024
050a589
Update reference/runners/docker.rst
davidsanfal May 6, 2024
973f649
Update reference/runners/docker.rst
davidsanfal May 6, 2024
4e14d9f
Update reference/runners/docker.rst
davidsanfal May 6, 2024
bed8232
Update reference/runners/docker.rst
davidsanfal May 6, 2024
4474a1e
Update reference/runners/docker.rst
davidsanfal May 6, 2024
de77e90
docker runners examples added
davidsanfal May 6, 2024
23095dc
basic docker runner example updated
davidsanfal May 6, 2024
da32acb
Update reference/runners.rst
davidsanfal May 6, 2024
c9fb326
Update reference/runners/docker.rst
davidsanfal May 6, 2024
453ae1b
Update reference/runners/docker.rst
davidsanfal May 6, 2024
03ce453
Update reference/runners/docker.rst
davidsanfal May 6, 2024
ff3b5f4
Update reference/runners/docker.rst
davidsanfal May 6, 2024
e604cf8
Update reference/runners/docker.rst
davidsanfal May 6, 2024
7036998
Update reference/runners/docker.rst
davidsanfal May 6, 2024
05235e0
note about package id added
davidsanfal May 6, 2024
5af6f74
Update examples/runners/docker/basic.rst
davidsanfal May 6, 2024
1a53d9a
conan docker runner basic example updated
davidsanfal May 6, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
1 change: 1 addition & 0 deletions examples.rst
Original file line number Diff line number Diff line change
Expand Up @@ -15,3 +15,4 @@ Examples
examples/graph
examples/dev_flow
examples/commands
examples/runners
10 changes: 10 additions & 0 deletions examples/runners.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
.. _examples_runners:

Conan runners examples
======================

.. toctree::
:maxdepth: 2

runners/docker/basic
runners/docker/configfile_build_args
477 changes: 477 additions & 0 deletions examples/runners/docker/basic.rst

Large diffs are not rendered by default.

190 changes: 190 additions & 0 deletions examples/runners/docker/configfile_build_args.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,190 @@
.. _examples_runners_docker_configfile_build_args:

Using a docker runner configfile to parameterize a Dockerfile
=============================================================

In this example we are going to see how to use a docker runner configfile to define our Dockerfile base image. Let’s create two profiles and a Dockerfile inside our project folder.

.. code-block:: bash

$ cd </my/runner/folder>
$ tree
.
├── Dockerfile
├── configfile
├── docker_example_build
└── docker_example_host

``docker_example_host`` profile

.. code-block:: text

[settings]
arch=x86_64
build_type=Release
compiler=gcc
compiler.cppstd=gnu17
compiler.libcxx=libstdc++11
compiler.version=11
os=Linux
[runner]
type=docker
configfile=</my/runner/folder>/configfile
cache=copy
remove=false

``docker_example_build`` profile

.. code-block:: text

[settings]
arch=x86_64
build_type=Release
compiler=gcc
compiler.cppstd=gnu17
compiler.libcxx=libstdc++11
compiler.version=11
os=Linux

``configfile``

.. code-block:: yaml

image: my-conan-runner-image
build:
dockerfile: </my/runner/folder>
build_context: </my/runner/folder>
build_args:
BASE_IMAGE: ubuntu:22.04
run:
name: my-conan-runner-container


.. code-block:: docker
:caption: Dockerfile

ARG BASE_IMAGE
FROM $BASE_IMAGE
RUN apt-get update \
&& DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends \
build-essential \
cmake \
python3 \
python3-pip \
python3-venv \
&& rm -rf /var/lib/apt/lists/*
RUN pip install conan

In this example we are going to start from a totally clean docker, without containers or images. In addition, we are going to have the conan cache also completely empty.

.. code-block:: bash

$ conan list "*:*"
Found 0 pkg/version recipes matching * in local cache

$ docker ps --all
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES

$ docker images
REPOSITORY TAG IMAGE ID CREATED SIZE


Now, we are going to clone and build zlib from conan-center-index and create it using our new runner definition.

.. code-block:: bash

$ git clone https://github.com/conan-io/conan-center-index.git --depth 1
$ conan create ./conan-center-index/recipes/zlib/all --version 1.3.1 -pr:h </my/runner/folder>/docker_example_host -pr:b </my/runner/folder>/docker_example_build

...

┌──────────────────────────────────────────────────┐
| Building the Docker image: my-conan-runner-image |
└──────────────────────────────────────────────────┘

Dockerfile path: '</my/runner/folder>/Dockerfile'
Docker build context: '</my/runner/folder>'

Step 1/5 : ARG BASE_IMAGE

Step 2/5 : FROM $BASE_IMAGE

...

Successfully built 286df085400f
Successfully tagged my-conan-runner-image:latest

...

┌───────────────────────────────┐
| Creating the docker container |
└───────────────────────────────┘


┌─────────────────────────────────────────┐
| Container my-conan-runner-image running |
└─────────────────────────────────────────┘


┌─────────────────────────────────────────┐
| Running in container: "conan --version" |
└─────────────────────────────────────────┘

┌──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
| Restore host cache from: </my/runner/folder>/conan-center-index/recipes/zlib/all/.conanrunner/docker_cache_save.tgz |
└──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘

Restore: zlib/1.3.1 in p/zlib95420566fc0dd
Restore: zlib/1.3.1:b647c43bfefae3f830561ca202b6cfd935b56205 in p/zlibd59462fc4358e/p
Restore: zlib/1.3.1:b647c43bfefae3f830561ca202b6cfd935b56205 metadata in p/zlibd59462fc4358e/d/metadata

┌────────────────────┐
| Stopping container |
└────────────────────┘

If we now check the status of our Conan and docker cache, we will see the zlib package compiled for Linux and the new docker image and container.

.. code-block:: bash

$ conan list "*:*"
Found 1 pkg/version recipes matching * in local cache
Local Cache
zlib
zlib/1.3.1
revisions
e20364c96c45455608a72543f3a53133 (2024-04-29 17:18:07 UTC)
packages
b647c43bfefae3f830561ca202b6cfd935b56205
info
settings
arch: x86_64
build_type: Release
compiler: gcc
compiler.version: 11
os: Linux
options
fPIC: True
shared: False

$ docker ps --all
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
1379072ae424 my-conan-runner-image "/bin/bash -c 'while…" 17 seconds ago Exited (137) 2 seconds ago my-conan-runner-image

$ docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
my-conan-runner latest 383b905f352e 22 minutes ago 531MB
ubuntu 22.04 437ec753bef3 12 days ago 77.9MB

If we run the ``conan create`` command again we will see how Conan reuses the previous container because we have set ``remove=False``.

.. code-block:: bash

$ conan create ./conan-center-index/recipes/zlib/all --version 1.3.1 -pr:h </my/runner/folder>/docker_example_host -pr:b </my/runner/folder>/docker_example_build

...

┌───────────────────────────────┐
| Starting the docker container |
└───────────────────────────────┘

...
1 change: 1 addition & 0 deletions reference.rst
Original file line number Diff line number Diff line change
Expand Up @@ -17,3 +17,4 @@ Reference
reference/environment
reference/binary_model
reference/conan_server
reference/runners
17 changes: 17 additions & 0 deletions reference/runners.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
.. _reference_runners:

Runners
=======

Runners are the way to run Conan remotely and transparently just by modifying the host profile. There are three requirements to be able to use the feature:

davidsanfal marked this conversation as resolved.
Show resolved Hide resolved
- Installing a version of conan with runner dependencies ``pip install conan[runners]``.
davidsanfal marked this conversation as resolved.
Show resolved Hide resolved
- Install the tools to run each of the runners (``docker``).
- Add the ``[runner]`` section defined in the documentation of each runner to the host profile.

Runners:

.. toctree::
:maxdepth: 1

runners/docker
60 changes: 60 additions & 0 deletions reference/runners/docker.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,60 @@
.. _reference_runners_docker:

Docker runner
=============

How to use a docker runner
--------------------------

To run Conan inside a docker container you need to define a ``[runner]`` section in your host profile using the following fields:

- ``type`` **(mandatory)**: define the runner we want to use, in this case ``docker``.
- ``dockerfile`` **(optional, default None)**: define the Dockerfile absolute path in case you want to build a docker image.
davidsanfal marked this conversation as resolved.
Show resolved Hide resolved
- ``image`` **(optional, default conan-runner-default)**: docker image name you want to download from a docker registry or the name of the built image in case you define a dockerfile path.
- ``cache`` **(optional, default clean)**: how docker container uses (or not) the host cache.
davidsanfal marked this conversation as resolved.
Show resolved Hide resolved

- ``clean``: use an empty cache.
davidsanfal marked this conversation as resolved.
Show resolved Hide resolved
- ``copy``: copy the host cache inside the container using the :ref:`conan cache save/restore<reference_commands_cache>` command.
- ``shared``: mount the host conan cache as a shared volume.
davidsanfal marked this conversation as resolved.
Show resolved Hide resolved

- ``remove`` **(optional, default false)**: ``true`` or ``false``. Remove the container after running the conan command.
davidsanfal marked this conversation as resolved.
Show resolved Hide resolved
- ``configfile`` **(optional, default None)**: configuration file absolute path with extra parameters (see **extra configuration** section for more info).
davidsanfal marked this conversation as resolved.
Show resolved Hide resolved

davidsanfal marked this conversation as resolved.
Show resolved Hide resolved
Extra configuration
-------------------

If you need more controll over the container build and execution, you can define more parameters inside a ``configfile`` yaml.
davidsanfal marked this conversation as resolved.
Show resolved Hide resolved

.. code-block:: yaml

image: image_name # The image to build or run.
build:
dockerfile: /dockerfile/path # Dockerfile path.
build_context: /build/context/path # Path within the build context to the Dockerfile.
build_args: # A dictionary of build arguments
foo: bar
cacheFrom: # A list of images used for build cache resolution
- image_1
run:
name: container_name # The name for this container.
containerEnv: # Environment variables to set inside the container.
env_var_1: env_value
containerUser: user_name # Username or UID to run commands as inside the container.
privileged: False # Run as privileged
capAdd: # Add kernel capabilities.
- SYS_ADMIN
- MKNOD
securityOpt: # A list of string values to customize labels for MLS systems, such as SELinux.
- opt_1
mount: # A dictionary to configure volumes mounted inside the container.
/home/user1/: # The host path or a volume name
bind: /mnt/vol2 # The path to mount the volume inside the container
mode: rw # rw to mount the volume read/write, or ro to mount it read-only.

How to run a `conan create` in a runner
---------------------------------------

In the following links you can find some examples about how to use a conan docker runner:

- :ref:`Compile zlib using a docker runner<examples_runners_docker_basic>`
- :ref:`Using a docker runner configfile to parameterize the Dockerfile base image<examples_runners_docker_configfile_build_args>`