[question] conan install followed by cmake build

[tiolan]

My question is about using conan with cmake.
I generate a conan package that consists of source code, a CMake project and a conanfile.py. It depends on several other conan packages.

When using conan create ..., it pulls the dependencies, compiles the code and everything works fine.

Let’s assume I want to further develop the source code e.g. in Visual Studio (Code) without actually building the package.
There I can normally integrate the CMake project directly, that gives me more flexibility and functionality (intellisense, debugger, etc) in the IDE.
My idea was to manually conan install... followed by the CMake build instructions (e.g triggered by the IDE).

The problem here is that conan build... invokes CMake with additional parameter such as CMAKE_MODULE_PATH that are package specific. They are missing in the normal CMake build.

As a workaround I can add these parameters via the IDE such that CMake is invoked properly.

Any idea how to improve this?

I was checking the conan-cmake wrapper.
But failed until now.

• I've read the CONTRIBUTING guide.

[memsharded]

Hi @tiolan

You have hit a very interesting case. The good news is that we have been working for this very hard in the last releases, and it is one of our major focus right now:

• Please check the new CMakeDeps, CMakeToolchain integrations in the new conan.tools section (note the singular "conan.tools"): https://docs.conan.io/en/latest/reference/conanfile/tools.html
• These new conan.tools is the future for Conan 2.0, the other integrations will be removed.
• The idea is that CMakeToolchain and other generators run at generate() time, in the conan install command. They generate everything that the build needs, specifically a conantoolchain.cmake file that can be passed in the command line very easily. https://docs.conan.io/en/latest/reference/conanfile/methods.html#generate
• The amount of configurability of generators with this new approach is much higher. Custom configurations are possible both in CMakeDeps and MSBuildDeps. Also, CMakeToolchain can be extended and customized to a high degree (https://docs.conan.io/en/latest/reference/conanfile/tools/cmake.html#cmaketoolchain)

The conan new hello/0.1 -m v2_cmake command will generate an example template that should work out of the box. You can check the test_package/conanfile.py and test_package/CMakeLists.txt too.

Besides that, we are also working heavily in the "layouts" problem, to have things correctly defined for both the local developer flow and when building in the cache with conan create. Hopefully we can introduce this new "layout" approach in next Conan 1.37.

Please have a look to these new features, I think this is what you are looking for. They are still experimental, but the idea is to stabilize them soon so it is possible to use them for a smooth transition to Conan 2.0.

[tiolan]

@memsharded
Thanks for the fast response! Thats great to hear.
I've tried it.
Once with just generators = "cmake_find_package", "cmake", "CMakeToolchain"
Which leads to:

ERROR: Error in generator 'CMakeToolchain': Invalid setting 'apple-clang' is not a valid 'settings.compiler' value.
Possible values are ['Visual Studio', 'clang', 'gcc', 'msvc']
Read "http://docs.conan.io/en/latest/faq/troubleshooting.html#error-invalid-setting"

Also this does not help:

from conan.tools.cmake import CMakeToolchain
...
    generators = "cmake_find_package", "cmake"
...
    def generate(self):
        print(self.settings.compiler)
        tc = CMakeToolchain(self)
        if self.options.unittests:
            tc.variables["BUILD_TESTING"] = True
        tc.generate()

conanfile.py (grpcclient/0.1): Calling generate()
gcc
ERROR: conanfile.py (grpcclient/0.1): Error in generate() method, line 27
        tc = CMakeToolchain(self)
        ConanException: Invalid setting 'apple-clang' is not a valid 'settings.compiler' value.
Possible values are ['Visual Studio', 'clang', 'gcc', 'msvc']
Read "http://docs.conan.io/en/latest/faq/troubleshooting.html#error-invalid-setting"

You can see self.settings.compiler being set to gcc, not apple-clang.

I invoke the build like this:
conan install .. --profile ubuntu -s compiler.version=9 -o unittests=True

My settings.yaml has actually been adapted and apple-clang has been removed, plus some custom settings were added.
Is it expected, that the generator does not take this into account?

I tried with conan 1.35.2 and conan 1.36.0
The conan new hello/0.1 -m v2_cmake together with my custom settings brings the same.

[memsharded]

Hi @tiolan

Does your recipe declare settings = "os", "compiler", "arch", "build_type". These helpers will expect to have those settings in the recipe to do their thing.

Also it seems you might have a quite old settings.yml in your cache? Is it possible? Can you please have a look and share what you have in your /.conan/settings.yml? To get a new modern one, you could just remove it, it will be generated fresh next Conan command.

[tiolan]

@memsharded
Yes, the settings = "custom_os", "os", "compiler", "arch", "build_type" are set in my class. Note that I added custom_os.

This actually prints "gcc":

    def generate(self):
        print(self.settings.compiler)

The settings.yml has been generated just before testing everything today by doing this:

rm ~/.conan/settings.yml
rm ~/.conan/remotes.json
rm ~/.conan/profiles/default

conan config install $(MY_CONF_TEMPLATE)/config
conan profile new --detect --force default

conan profile remove settings.arch_build default
conan profile remove settings.os_build default

cat ~/.conan/settings.yml is:

os:
    Windows:
        subsystem: [None, cygwin, msys, msys2, wsl]
    WindowsStore:
        version: ["8.1", "10.0"]
    Linux:
    Android:
        api_level: ANY

custom_os:
    MyCustomOs1:
    MyCustomOs2:
        toolchain: ["variant1", "variant2"]
        #version: ANY
    Ubuntu:
    Alpine:
    Windows:

arch: [x86_64, armv8]

compiler:
    gcc:
        version: ["7", "7.3", "7.5",
                  "9", "9.3",
                  "10", "10.2"]
        libcxx: [libstdc++, libstdc++11]
        threads: [None, posix, win32] #  Windows MinGW
        exception: [None, dwarf2, sjlj, seh] # Windows MinGW
        cppstd: [None, 98, gnu98, 11, gnu11, 14, gnu14, 17, gnu17, 20, gnu20]
    Visual Studio: 
        runtime: [MD, MT, MTd, MDd]
        version: ["14", "15", "16"]
        toolset: [None, v90, v100, v110, v110_xp, v120, v120_xp,
                  v140, v140_xp, v140_clang_c2, LLVM-vs2012, LLVM-vs2012_xp,
                  LLVM-vs2013, LLVM-vs2013_xp, LLVM-vs2014, LLVM-vs2014_xp,
                  LLVM-vs2017, LLVM-vs2017_xp, v141, v141_xp, v141_clang_c2, v142,
                  llvm, ClangCL]
        cppstd: [None, 14, 17, 20]
    msvc:
        version: ["19.0",
                  "19.1", "19.10", "19.11", "19.12", "19.13", "19.14", "19.15", "19.16",
                  "19.2", "19.20", "19.21", "19.22", "19.23", "19.24", "19.25", "19.26", "19.27", "19.28"]
        runtime: [static, dynamic]
        runtime_type: [Debug, Release]
        cppstd: [14, 17, 20]
    clang:
        version: ["8"]
        libcxx: [None, libstdc++, libstdc++11, libc++, c++_shared, c++_static]
        cppstd: [None, 98, gnu98, 11, gnu11, 14, gnu14, 17, gnu17, 20, gnu20]
        runtime: [None, MD, MT, MTd, MDd]

build_type: [None, Debug, Release, RelWithDebInfo, MinSizeRel]

(Note that I needed to put some dummy values for custom_os here)

So, it should not be an old settings file, right?

[memsharded]

Oh, I see, you are installing a custom setings.yml with conan config install $(MY_CONF_TEMPLATE)/config, there will be a settings.yml inside there.

That custom settings.yml has removed the apple-clang compiler.

Now I see the issue. The new CMakeToolchain has a hard check on the compiler. Lets relax it, we'll do it for the next 1.37 version, or for the next 1.36.X patch if there is one.

Thanks very much for reporting!

[memsharded]

Hint: If you want to workaround it to keep trying right now, please add manually apple-clang to your settings.yml (only the compiler should be enough, no need to add the sub-settings, like version, etc)

[tiolan]

@memsharded
I have now added

compiler:
    apple-clang:
    sun-cc:

to my settings as a workaround (sun-cc was also required).

My conanfile contains this (as a workaround the version is intentionally set to 1.36):

    def generate(self):
        tc = CMakeToolchain(self)
        if self.options.unittests:
            tc.variables["BUILD_TESTING"] = True
        # TODO: Re-Test once 1.37 has been released
        if conan_version < Version("1.36"):
            self.output.warn(
                "Conan < 1.37 does not support generating toolchains files with custom settings")
        else:
            tc.generate()

the generated conan_toolchain.cmake contains this:

set(CMAKE_FIND_PACKAGE_PREFER_CONFIG ON)
# To support the cmake_find_package generators
set(CMAKE_MODULE_PATH ${CMAKE_CURRENT_LIST_DIR} ${CMAKE_MODULE_PATH})
set(CMAKE_PREFIX_PATH ${CMAKE_CURRENT_LIST_DIR} ${CMAKE_PREFIX_PATH})

I am missing, that this toolchain file sets the CMAKE_PREFIX_PATH to also contain the path for dependent Conan packages.

If building with conan install .. -pr ubuntu -s compiler=gcc -s compiler.version=9 and cmake -DCMAKE_TOOLCHAIN_FILE=build/conan_toolchain.cmake it fails finding packages. It does not fail when building with conan create...

[cmake] CMake Error at /usr/share/cmake-3.16/Modules/FindPackageHandleStandardArgs.cmake:146 (message):
[cmake]   Could NOT find Protobuf (missing: Protobuf_INCLUDE_DIR)

[memsharded]

I am proposing a fix for the issue in #8900 (for 1.36.1 or 1.37, it depends).

The CMakeToolchain includes:

set(CMAKE_MODULE_PATH ${CMAKE_CURRENT_LIST_DIR} ${CMAKE_MODULE_PATH})

because it is intended to locate the generated xxxx-config.cmake files created by the CMakeDeps generator, not the ones inside the packages.

Listing the package folders directly and trying to use the scripts within the packages have 2 known limitations:

• It doesn't work for multi-configuration, which is a stopper for many people using multi-config environments like Visual Studio or XCode. CMake find_package cannot locate more than 1 location at a time, so it cannot be used to switch among Debug/Release (and possible others like 32/64bits or custom configurations).
• Their implementation of transitivity can be problematic, and they can locate dependencies in the system, rather than the transitive Conan package (thus using an older system library compared to the newer one contained in the Conan package).

The CMakeDeps generator doesn't have these limitations, and it also has the added advantage that it works irrespective of the build system of the dependency. You can have dependencies built with MSBuild, or Meson or Makefiles, and you will still get a xxxx-config.cmake to consume it. The only limitation is that the usage of custom cmake scripts, like you can find in protobuf is a bit more tricky (still possible, using the cpp_info.build_modules feature).

[tiolan]

Sorry, please ignore my last (hidden) comment, was my fault.

[memsharded]

#8900 was merged, this will be released in 1.37