[question] Building using Conan CMakePresets via Jenkins jobs in parallel

[amachado-pie]

What is your question?

I'm having a problem with building multiple targets with Jenkins jobs in parallel using CMakePresets.
The source code is checkout once, and a build folder for each target job is created.
I had to create a CONAN_HOME folder per job since the Conan cache failed when trying multiple jobs using the same folder. But that's okay.

The issue I'm having is with the root file CMakePresets.json which Conan adds as the preset's include path in this file.
As there are several jobs, each one compiling a target, each one edits the CMakePresets.json adding the preset path.
But when the job ends and the build folder is deleted, CMakePresets.json becomes with a dangling path that the other build will give an error when invoking cmake --preset android-release saying that another preset --linux-release no longer exists.

The pipeline is something like this:

stage('Build Setup Packages') {
    steps {
        script {
        def buildMatrix = [:]
        def failedBuilds = []

        def versions = ['light', 'pro']
        def targets = ['linux-x86', 'windows', 'android']

        for (def version in versions) {
          for (def target in targets) {
            def jobName = "${target}-${version}"
            def buildDir = "build_${target}_${version}"
            def thisTarget = target
            def thisVersion = version

            buildMatrix[jobName] = {
            stage("Build ${jobName}") {
              // Run make_setup.sh with BUILD_FOLDER set to job-specific directory a local Conan cache
              sh """#!/bin/bash
                cd /workdir
                mkdir -p ${buildDir}                                
                export BUILD_FOLDER="${buildDir}"
                export CONAN_HOME="/workdir/.conan_${jobName}"

                mkdir -p "\${CONAN_HOME}/profiles"
                cp -r /home/builder/.conan2/profiles/* "\${CONAN_HOME}/profiles"

                conan remote disable conancenter 1> /dev/null
                conan remote add artifactory-packages "${artifactory_pkg_url}" 1> /dev/null

                echo "Building ${jobName} with \$(nproc) parallel jobs"
                scripts/make_setup.sh "${thisTarget}" "${thisVersion}"
              """
            }
          }
        }
        def results = parallel(buildMatrix)
        ...
  }

The script make_setup.sh basically does something like this:

eval "conan install .  -pr:h ./conan_profiles/{arch} -pr:b default -s build_type=${BUILD_TYPE^} -b missing -b outdated -c tools.cmake.cmake_layout:build_folder_vars='[\"const.${BUILD_PRESET}\"]'"
cmake --preset conan-${BUILD_PRESET}-${BUILD_TYPE,,}
cmake --build --preset conan-${BUILD_PRESET}-${BUILD_TYPE,,} -j$(nproc)

Would it be possible for Conan to edit and CMake to use CMakePresets.json defined in a build directory instead of the source directory to avoid collision? I wanted to avoid having to do code checkouts or source copy to be able to launch compilation jobs for each target.

I was able to reduce the occurrence of dangling paths in CMakePresets.json by deleting the file before invoking conan install ... & cmake --preset conan-xxx.
But even so, I've had problems with a job failing because it couldn't find the preset because it was deleted by another one:

16:05:44  conanfile.py (project/0.1): Generating aggregated env files
16:05:44  conanfile.py (project/0.1): Generated aggregated env files: ['conanbuild.sh', 'conanrun.sh']
16:05:44  Install finished successfully
16:05:44  CMake Error: Could not read presets from /workdir:
16:05:44  File not found: /workdir/CMakePresets.json

I know it's a bit confusing, but any suggestions on how to implement parallel CI/CD pipeline Jenkins jobs with Conan and CMake Presets?

Thanks!

Have you read the CONTRIBUTING guide?

• I've read the CONTRIBUTING guide

[memsharded]

Hi @amachado-pie

Thanks for your question

Let me try to see if I understood correctly:

• Several parallel jobs are building something
• The jobs use different Conan caches, because they parameterize the CONAN_HOME using the job number
• But the jobs seems to use the same buildDir, so basically there is just one single common source checkout?
• The Conan creates presets, Conan might create different subfolders, with different names for different configurations
• But the root CMakeUserPresets.json is including() those presets
• The job is removing the different subfolders so the inclusion from the CMakeUserPresets.json is broken

Did I get the issue correctly?

If this is the case, some quick questions/possible approaches:

• Why not using different buildDirs or different source checkouts for the parallel jobs? In general every parallel job in jenkins should have its independent and isolated workspace with its own checkout?
• Why removing the build subfolders in the first place, why not leaving them alone, they will typically be garbage collected later?

[amachado-pie]

Hi @memsharded thanks for your feedback!

Did I get the issue correctly?

Exactly!

Why not using different buildDirs or different source checkouts for the parallel jobs? In general every parallel job in jenkins should have its independent and isolated workspace with its own checkout?

The source code checkout is done in the Jenkins pipeline workspace, a Docker container with the necessary toolchains is executed by mounting the Jenkins workspace as a /workdir volume and within the container the Jenkins stages and parallel jobs are executed.

If we had to create a clean checkout in a workspace per target then we would have to launch as many Docker containers as targets, or serialize the builds. Or doing source checkouts inside the container instead of in the Jenkins workspace would imply that the container has credentials to access the repos, etc.

We found this approach to be more appropriate because in practice the source code directory (source_dir) should be immutable across builds. What changes are the files in the build directories (build_dir), so having parallel compilation jobs using a single source_dir workspace was expected.
But what deviates from this assumption is that CMakeUserPresets.json is actually being modified at build-time.

But I understand that Conan does it this way, and I understand that it is the only way because unfortunately, from what I have read, CMake does not allow you to indicate the path of the CMakeUserPresets.json file to use and then generate it in the root of the build_dir of the target in question.

Why removing the build subfolders in the first place, why not leaving them alone, they will typically be garbage collected later?

Good question :)
The target does conan install . to install and build the main project but at the end it does conan install --requires pkg/1.x. to copy other artifacts from this dependency and the latter is deleting the build folder so as not to have files from the main project build.
But we will have to see here, not to delete the build and create a sub-folder for the dependencies and not to use the main one to keep the CMakePreset path that was added to the root's CMakeUserPresets.json.

[memsharded]

But I understand that Conan does it this way, and I understand that it is the only way because unfortunately, from what I have read, CMake does not allow you to indicate the path of the CMakeUserPresets.json file to use and then generate it in the root of the build_dir of the target in question.

Yes, exactly, CMake mandates that the presets file is only besides the root CMakeLists.txt, it cannot be parameterized to be elsewhere.

The target does conan install . to install and build the main project but at the end it does conan install --requires pkg/1.x. to copy other artifacts from this dependency and the latter is deleting the build folder so as not to have files from the main project build.

With a deployer or something like that?

Yes, maybe everything, the buildDir, the deployer folder, etc can also be parameterized or use the build number

From the Conan side what could be done and I think it is feasible and might make sense is that for every conan install, it could check the paths included in the CMakeUserPresets.json, and if they are not existing, they could be removed. What do you think, would that alleviate the issue?

[amachado-pie]

From the Conan side what could be done and I think it is feasible and might make sense is that for every conan install, it could check the paths included in the CMakeUserPresets.json, and if they are not existing, they could be removed. What do you think, would that alleviate the issue?

May be an idea to explore, but how you know if that include is "managed" by Conan or not to delete it if the path doesn't exist?
Something like creating a CMakeConanPresets.json that is managed by conan?

Current: CMakeUserPresets.json:

{
    "version": 4,
    "vendor": {
        "conan": {}
    },
    "include": [
        "build_android_light/android-light/Release/generators/CMakePresets.json",
        "build_android_pro/android-pro/Release/generators/CMakePresets.json"
    ]
}

Proposal: CMakeUserPresets.json:

{
    "version": 4,
    "vendor": {
        "conan": {}
    },
    "include": [
        "CMakeConanPresets.json"
    ]
}

CMakeConanPresets.json

{
    "version": 4,
    "vendor": {
        "conan": {}
    },
    "include": [
        "build_android_light/android-light/Release/generators/CMakePresets.json",
        "build_android_pro/android-pro/Release/generators/CMakePresets.json"
    ]
}

Then for every conan install, it could check whether or not CMakeUserPresets.json already has CMakeConanPresets.json and add it if necessary, and then validate the "managed CMakeConanPresets.json, check the paths included in and if they are not existing, they could be removed.

[memsharded]

The CMakeUserPresets.json and the CMakePresets.json generated by Conan contain:

"vendor": {
        "conan": {}
    },

So we know it is managed by Conan, so we are free to modify, remove or whatever is necessary in that file. That is exactly the goal of that field, it shouldn't be necessary to redirect to other file.

[GavinNL]

Hi @amachado-pie ,

I just came across your post while looking for "preset" issues. I'm sort of in the same boat where I'm dealing with minor inconveniences with the presets that being created by conan. I personally don't like my tools/compilers modifying my source folder and generating files that I don't want. Only the build directory should be modified.

The conan install command has a way to set which preset file to generate. You can point it to some random location in the /tmp folder, and then call cmake without using presets.

conan install conanfile.py \
        -s:h "&:build_type=Debug" \
        -of /path/to/build \
        -c tools.cmake.cmaketoolchain:user_presets=/tmp/TestFile.json

cmake ..  -B /path/to/build \
               -DCMAKE_TOOLCHAIN_FILE=conan_toolchain.cmake  \
               -DCMAKE_POLICY_DEFAULT_CMP0091=NEW \
               -DCMAKE_BUILD_TYPE=Debug

It's not the most convenient, but this is what I'm doing. I created my own conan_install.sh script that handles all of this for me.

[amachado-pie]

Hi @GavinNL

I personally don't like my tools/compilers modifying my source folder and generating files that I don't want. Only the build directory should be modified.

Unfortunately, it is CMake that does not allow you to indicate the location of the preset file and forces it to be in the root of the source code directory. I think this decision by Kitware/CMake team is an issue.

The conan install command has a way to set which preset file to generate. You can point it to some random location in the /tmp folder, and then call cmake without using presets.

conan install conanfile.py
-s:h "&:build_type=Debug"
-of /path/to/build
-c tools.cmake.cmaketoolchain:user_presets=/tmp/TestFile.json

cmake .. -B /path/to/build
-DCMAKE_TOOLCHAIN_FILE=conan_toolchain.cmake
-DCMAKE_POLICY_DEFAULT_CMP0091=NEW
-DCMAKE_BUILD_TYPE=Debug
It's not the most convenient, but this is what I'm doing. I created my own conan_install.sh script that handles all of this for me.

If you don't want to generate the preset file, you can simply set this setting to empty to disable it as per the documentation:

tools.cmake.cmaketoolchain:user_presets: 
(Experimental) Select a different name instead of CMakeUserPresets.json, empty to disable

The use of cmake presets, despite this annoying limitation, is very cool because all the parameters that are dynamically defined by the profile (debug, release, options, etc.) are transparent and you don't have to explicitly define them on the cmake command line.

[memsharded]

Hi @GavinNL

Are you aware that if you do -c tools.cmake.cmaketoolchain:user_presets="", it directly disables the creation of the file? no need to redirect to a random folder.

You might as well write that in the generate() method of your recipe, like:

def generate(self):
      t = CMakeToolchain(self)
      t.user_presets_path = False
      t.generate()

And then forget to have to do in the command line.

Likewise, if you add the cmake_layout(self) in your conanfile.py layout() method, you can forget about the need of -of=build argument, and that further simplifies your command line and UX to:

conan install . -s:h &:build_type=Debug

[GavinNL]

Hi @memsharded ,

Thanks for the info. I now know about setting user_preset="".

I am aware you can set a lot of the configuration in the conanfile.py but I honestly don't like having too many things in my conanfile.py, the only thing I want there are the dependences it uses. Any other "configuration" for my build (build location, build folder structure, preset-name, etc) should come from the user and be modifiable from the commandline.

There was a big change with the behaviour of the CLI between conan1 and 2 which really messed with my workflow, unfortunately.

With Conan1, the CLI behaved like Cmake, you pass the location of the conanfile into conan via first arg, and it dumps everything into CWD. With Conan2, the default is to dump everything into same folder as the conanfile. I really didn't like this default behaviour as it polluted by src folder way too many times.

I ended up creating a cc_configure shell script, which wraps conan/cmake, but does a whole bunch of additional checks and configuration (like figure out the correct versions of compilers, etc) without having the need for profiles. I have this working for all the compilers I tend to use (gcc/clang/emcc/zig), eg below.

cd SRC_FOLDER

# Configure a GCC build
mkdir build-gcc && cd build-gcc
CC=$(which gcc-12) CXX=$(which g++-12) cc_configure ..
make

# Configure a web assembly build 
cd ..
mkdir build-emcc && cd build-emcc
CC=$(which emcc) CXX=$(which em++)  cc_configure ..
make

# Configure a static build with musl using zig and build for ARM
# (zcc and zc++ are wrapper scripts that call "zig cc" and "zig c++"
cd ..
mkdir build-zig && cd build-zig
CC=$(which zcc) CXX=$(which zc++) BUILD_TYPE=Release ARCH=armv8 cc_configure ..
make

[memsharded]

I am aware you can set a lot of the configuration in the conanfile.py but I honestly don't like having too many things in my conanfile.py, the only thing I want there are the dependences it uses. Any other "configuration" for my build (build location, build folder structure, preset-name, etc) should come from the user and be modifiable from the commandline.

But most of it is configurable already. Having it in the conanfile.py is having some nice defaults, but that doesn't mean that users can't configure it.

For example if you have the cmake_layout(self) defined in the layout() method, you have nice defaults that allow to do things like:

conan install .
cmake --preset conan-default
# open IDE

which is super nice and convenient.

Still users that want to change the default configuration have many confs to do it, for example:

• tools.cmake.cmake_layout:build_folder can change the root build folder in the same way -of=xxxx does
• tools.cmake.cmake_layout:build_folder_vars allows to have multiple build folder nicely named after settings/options, to work simultaneously on different configurations of the project

With Conan1, the CLI behaved like Cmake, you pass the location of the conanfile into conan via first arg, and it dumps everything into CWD. With Conan2, the default is to dump everything into same folder as the conanfile. I really didn't like this default behaviour as it polluted by src folder way too many times.

And this was one of the main reasons why the layout() was created and recommended, because Conan 1 was polluting the user folder too many times, too many users complaining about this. Now the layout allows to not pollute the user folder by default while being also configurable.

[GavinNL]

If I have the cmake_layout(self) in the conanfile.py, can I choose to ignore that from the command line when I call conan install?

I tried adding that into my conanfile.py and ran conan install conanfile.py -of /tmp/build, I was expecting all the generated files to be in /tmp/build, but instead its in /tmp/build/build/Release/generators, (note the double build/build folder) which is what I don't want.

If I am compiling software using cmake only, my workflow is:

mkdir /tmp/build && cd /tmp/build

cmake /path/to/source/folder

make

If I was using conan, I expect my workflow to be the same

mkdir /tmp/build && cd /tmp/build

# builds packages in release mode
conan install /path/to/source/folder

# defaults to debug build
cmake /path/to/source/folder

make

This was how it worked with conan 1, but conan 2 changed all that. I can work around it if I don't use the cmake_layout(self)