[feature] Configurable and distributable toolchains

[DoDoENT]

This proposal was originally created as tribe pull request, but after some conversation there I decided to put it here in form of feature request:

Summary

Define a special type of conan package (profile-package) that can be used only in conan profiles or pose instead of conan profile, but have the ability to impose settings on the dependency graph and ability to update settings.yml file.

Motivation

Conan profiles are a great way of sharing common settings and build requirements in your development team. However, the feature currently does not have enough flexibility, such as enforcing specific compile flags on all packages in the dependency graph. Also, profiles cannot be distributed as a conan package, so different channels must be use to distribute profile to your development team.
The motivation originated from this Github issue comment.

Proposal

Define a special "profile-package" type of the conan package, as special conan package that:

• can impose settings on the dependency graph
• may have build_requires and python_requires, optionally even requires
  • for using common python utilities and for depending on already-existing conan packages, such as cmake_installer, emsdk_installer, android_ndk, ...
  • if we don't allow requires for such packages, then the build procedure will have to ensure that all tools are repackaged into the toolchain (which makes sense for repackaging emsdk_installer, android_ndk and similar, but I'm not so sure about the purpose of repackaging cmake - for such case it may make sense to allow requires, but in some kind of "restricted" form. We need to discuss this further).
• are optionally configurable
  • for example, a specific toolchain may have an option to enable/disable sanitizers, LTO, etc. The changed option would require different package IDs of the entire graph built using the toolchain-profile. Alternatively, we could require different toolchains for each change. This is how currently @vector-of-bool's dds does it, but I find this pretty annoying - I would prefer configurability.
• should be able to update settings.yml
  • this is a bit tricky, but it may be required for some toolchains to add specific architectures, OS options or compiler versions that do not exist in the default settings.yml
  • for example, let's consider an iOS toolchain that ensures that produced binaries are all fat. It's obviously wrong to advertise package architecture as x86_64 as it will also contain arm64 and other slices. However, it's possible to use the ios_fat architecture, which does not exist in the default settings.yml, so, an iOS toolchain should be able to register new architecture to settings.yml during its installation.
  • another example: Emscripten. Emscripten does not allow for link-compatibility between their minor versions, while the used clang compiler advertises the same version. For example, binaries built with emsdk 1.39.16 are not link-compatible with binaries built with emsdk 1.39.11, while both emsdk versions advertise clang 6.0.2 for their fastcomp backend and clang 11.0.0 for their upstream backend. This could be addressed by introducing special clang versions 6.0-emsdk-1.39.11, 6.0-emsdk-1.39.16, 11-emsdk-1.39.11, 11-emsdk-1.39.16, ..., in settings.yml on installation. Additionally, the emscripten toolchain may allow for choosing whether you want to enable emscripten threads and SIMD support, which require imposing specific compile flags on all packages in the dependency graph.
• can be distributed via Artifactory
  • a toolchain should be deployed to a conan repository on Artifactory and installed with conan config install or similar command. The installation would download the profile-package from the conan repository, create profiles associated with the package and trigger hooks that will allow for custom updates of settings.yml as described above.

Alternative Approaches

Alternative approach, using only currently available conan features, is completely described in this Github comment.

Open issues

• conan issue #7493
• conan issue #3573
• possibly conan issue #4753, if observed from the point of view of build and host profiles. This would be replaced with build and host toolchains (profile-packages).

Future Extensions

• the issue of minumum version of Visual Studio, discussed in conan-tribe PR-5 would be completely avoided, as specific versions of MSVC could be handled with custom toolchains, as described in this comment

[memsharded]

Hi @DoDoENT

I am reviewing this issue, as we are preparing a package types proposal, and I am struggling to understand some of the use cases and put some reasonable scope to the above comments. It seems there are too many different features in the above, and many of them are a challenge on their own:

can impose settings on the dependency graph

Something that can impose settings on a dependency graph is a profile, I am not fully sure what you mean here.

for using common python utilities and for depending on already-existing conan packages, such as cmake_installer, emsdk_installer, android_ndk, ...

What does it mean to define so many build-requires at the same time? It is not possible to use them, at the same time, so this would be only for installing, but how to specify that some specific build tools should be applied to a given conan install?

should be able to update settings.yml

Anything that is not replacing the whole settings.yml is challenging, as merging/updating yaml is not an easy problem. Adding some values seems doable, but updating things is a different story, specially if removal of things is desired. Seems a relatively minor UX improvement compared to the challenges, instead of updating, replacing the settings.yml with a new one seems more explicit and manageable. The recommendation is try to use the same settings.yml company wise, it should be possible and avoid lots of confusion.

a toolchain should be deployed to a conan repository on Artifactory and installed with conan config install

This is an idea that we like and have considered before, having conan config install pkg/version@user/channel. This could package settings.yml, profiles, conan.conf, hooks, etc, all the conan config install things, and install them as usual, but from a Conan package. But I fail to see the connection to the above build_requires for example, wouldn't it be enough to have the specific build_requires defined in each profile? Considering that conan config install is something you do at the beginning of each CI job, if it install all possible build_requires, it would be very inefficient, if it is going to use only some of them.

are optionally configurable
for example, a specific toolchain may have an option to enable/disable sanitizers, LTO, etc.

Yes, the new [conf] mechanism could package a global.conf file, the configuration defined there would be affecting the build system integrations, but it is not possible to say to at the toolchain level: enable LTO, and have it enabled transparently to consumers. Same with sanitizers, injecting these things to the builds, transparently, is not possible in the general case, and it requires explicit action in each package build scripts.

In summary: I can understand the value of a potential conan config install <pkg-ref>, that will put configuration items in a Conan package, and will install the configuration as usual. The other features, I struggle to understand the proposal. It would be worth trying to focus on one item at a time, and also specifying the consumption user story, how the UI, command line, recipes would look like. Also very important, to compare the current way to do that specific task, what are the limitations and how the current proposal improves over it or fixes it.

[DoDoENT]

Hi @memsharded.

Let me try to explain further my idea:

can impose settings on the dependency graph
Something that can impose settings on a dependency graph is a profile, I am not fully sure what you mean here.

This is exactly why I call this a "profile-package" - think of it as a configurable and executable profile. Just like what conanfile.py is to conanfile.txt, a profile-package would be to what currently a "profile" is.

For example, consider what currently needs to be done in order to introduce a new toolchain (i.e. a new version of Android NDK) to a company's ecosystem:

• first I need to download the NDK separately so I can execute its clang --version in order to discover the version of clang that is shipped with the NDK
• then I need to add that version of clang to settings.yml, if it does not already exist
• then I need to create a Conan package that contains the actual NDK (i.e. contains the compilers, linkers, sysroot files, etc.)
  • the recipe for this package also need to have correctly implemented the package_info method, so that all environment variables are set correctly, such as CONAN_CMAKE_TOOLCHAIN_FILE, CC, CXX, AR, LD, SYSROOT, etc.
• then I need to create a new profile for the new NDK
  • the new profile must ensure that it build_requires the package created in the previous step
• then I need to package the new settings.yml and new profile into a zip file and distribute that to my users

So, as you can see, I need to keep two different things in sync - the profile and settings.yml on the one side and the Conan package containing the NDK on the other. Not to mention that the package is distributed via Artifactory and the configuration zip via a different method (in our case it's via Jenkins' archiveArtifact facility).

The above proposal would allow me to create a single Conan package that will be both the profile and the toolchain package. Imagine something like following method within the recipe of such package:

def update_settings(self):
    cc_path = os.path.join(self.package_folder, 'toolchains', 'llvm', 'prebuilt', self._host_os + '-' + self._host_arch, 'bin', 'clang')
    clang_version = self.run([cc_path, '--version'])
    self.info.settings.compiler = 'clang'
    self.info.settings.compiler.version = clang_version
    self.info.settings.compiler.libcxx = 'libc++'
    self.info.settings.os = 'Android'
    self.info.settings.os.api_level = self.options.android_api_level  # assumes package has android_api_level option

for using common python utilities and for depending on already-existing conan packages, such as cmake_installer, emsdk_installer, android_ndk, ...

What does it mean to define so many build-requires at the same time? It is not possible to use them, at the same time, so this would be only for installing, but how to specify that some specific build tools should be applied to a given conan install?

Here I meant primarily that such packages would be allowed to depend on another package. For example, consider the Android NDK package from the previous example. Imagine that our NDK package does not download prebuilt binaries from Google's server, but instead clones the NDK source code from git and compiles it. In order to compile it, it may require some dependencies, such as cmake, system compiler (or even different profile-package needed to build the NDK package for the host system). Such dependencies would be a build_requires of the NDK package and would need to be installed in the build context (i.e. host system). In a similar way, multiple profile-packages may need to reuse common python code, which could then be put into a separate conan package that profile-packages would python_require.
I've already said that the usual requires may not be needed for profile-packages as I'm not completely sure what it would mean (a profile-package depending on another profile-package? It could be possibly useful if we have, e.g. a base profile-package with a common configuration that would be overridden by a derived profile-package - similar to how currently profiles can be composed. But, as I said, let's discuss this further - there may be more elegant solutions for that).

Anything that is not replacing the whole settings.yml is challenging, as merging/updating yaml is not an easy problem. Adding some values seems doable, but updating things is a different story, specially if removal of things is desired. Seems a relatively minor UX improvement compared to the challenges, instead of updating, replacing the settings.yml with a new one seems more explicit and manageable. The recommendation is try to use the same settings.yml company wise, it should be possible and avoid lots of confusion.

I'm aware of that, but I think it should be enough just to be able to add new values if they don't already exist. Modifying and deleting could be problematic and currently, I don't see use cases for this.

But I fail to see the connection to the above build_requires for example, wouldn't it be enough to have the specific build_requires defined in each profile? Considering that conan config install is something you do at the beginning of each CI job, if it install all possible build_requires, it would be very inefficient, if it is going to use only some of them.

Please let me know if the above NDK example still doesn't make this clearer to you.

but it is not possible to say to at the toolchain level: enable LTO, and have it enabled transparently to consumers. Same with sanitizers, injecting these things to the builds, transparently, is not possible in the general case, and it requires explicit action in each package build scripts.

Well, that's a problem. This is exactly what I would like to achieve in order to be able to tell conan to rebuild packages when the compiler changes. For example, apple-clang 12.0.0 (Xcode 12.0-12.4) and apple-clang 12.0.5 (Xcode 12.5 and newer) are link-compatible (i.e. binaries have the same ABI), only if built without sanitisers and without LTO. If you enable LTO on a specific package, it needs to be rebuilt. A similar problem is also with MSVC, which caused deprecation of the 'Visual Studio' compiler in favour of MSVC. However, although it's possible to together link binaries built with LTO and binaries built without LTO (thus advocating for letting the package script have control over the final build flags), it's not possible to link together binaries build with address sanitiser and binaries built without the address sanitiser. The problem is that the container overflow detector within the address sanitiser changes the ABI of the STL by adding some runtime checks into the common STL containers. So, when you link together two binaries where one was built with Address Sanitizer and the other without, you end up with an ODR violation in your program.

I'm aware that conan cannot strictly enforce the correct compile flags on all packages as, e.g. dds does, but it should be able to affect common build system integrations like CMake, Xcode, MSBuild and Autotools. You said that the conf mechanism in global.conf already does that, but this needs to be done at the toolchain level - it doesn't make sense to have globally enabled sanitisers - you need them only in toolchains that actually support them.

Currently, it's possible to introduce LTO and Sanitiser settings into the settings.yml under a specific compiler, but this is not enough, as conan does not know that with those enabled the ABI compatibility rules change. Even having that possibility would make ABI management easier. At the moment I'm relying on custom compiler versioning in settings.yml (i.e. clang 11.0.0, 11.1.0, 12.0.0, 12.0.1) in order to force Conan to recompile packages every time when compiler version changes, even if for some slices (i.e. non-LTO, no-sanitiser build) the recompilation is not needed - but this is still better than having to debug ODR violations or linker errors for slices where recompilation is needed.

I hope I made some things clearer. Please let me know if there are still things you struggle to understand.

[memsharded]

The above proposal would allow me to create a single Conan package that will be both the profile and the toolchain package. Imagine something like following method within the recipe of such package:

But this is a chicken and egg problem, this is why the configuration as profiles need to be separately from the actual package.
How would you be able to install the AndroidNDK that you want, if you don't specify the correct settings (that should already be in settings.yml and in your profile)?

Lets focus on the consumer user story: You want to build a Conan package for Android, lets call it mylib/1.0. Lets assume that we have an existing AndroidNDK package android_ndk/1.0 and we have our settings and profiles defined:

android_profile1

[settings]
compiler=clang
compiler.version=7.1

[build_requires]
android_ndk/1.0

We can execute this to create my package for this configuration

conan create . mylib/1.0@ -pr=android_profile1

Now, there is a new AndroidNDK, with a new compiler version 8.1, etc. Lets say that our devops team does the following steps for us:

• Create the android_ndk/2.0 package for us and upload to Artifactory
• Create a new android_profile2, that build_requires=android_ndk/2.0 and update settings.yml if necessary and put it in the git repo that contains the configuration.

Now user story to use the new version is:

conan config install  # No need to pass the URL, it is cached, and will update from there
conan create . mylib/1.0@ -pr=android_profile2

Note that when the package mylib/1.0 is being created, it is already passing the new compiler.version 8.1 and the new build requires. I think the flow is well defined and straightforward, and I struggle to see the user story on both sides: The "devops" story for creating the android_ndk/2.0 is still necessary, and it is already putting the package in Artifactory for consumption, and the new profile has to be written anyway. Could you please clarify about how you envision the user stories, specially on the consumer side? I still struggle to understand it.

[DoDoENT]

The user story should stay the same, but my proposal advocates for a simpler "DevOps" story, as the DevOps team will not require creating two different things (android_profile2 and android_ndk/2.0), distribute them via different mechanisms (Artifactory and zip) and manually make sure that settings within the profile match the actual compiler version in android_ndk/2.0. For example, how can DevOps know that the new NDK has compiler 8.1 and not 11.0?

Currently, the only possible way is to create a temporary package or side-install the NDK, check the version of the compiler and hardcode that into the profile file. With the proposed approach, the compiler version would be defined automatically, directly from the actual version that the compiler binary reports.

But this is a chicken and egg problem, this is why the configuration as profiles need to be separately from the actual package.
How would you be able to install the AndroidNDK that you want, if you don't specify the correct settings (that should already be in settings.yml and in your profile)?

It's not - the installation of the AndroidNDK should be done in build context, i.e. for the host system. I need macOS x86_64 version of the AndroidNDK package which then defines the compiler, compiler version, OS, Android API level, etc. for all packages that use the AndroidNDK for building them for Android (i.e. host context, i.e. target system).

Also, for cases when you need profile configurability, like Android API level in this case, but also LTO, sanitiser, etc., that influences the ABI and require the recompilation of the entire graph, it's much easier to have options on profile-package than having a separate profile for each and every combination.

What does like more convenient for you?
This:

conan create . mylib/1.0@ -pr=android-ndk-r22b-arm64-v8a-no-lto-no-asan-no-ubsan
conan create . mylib/1.0@ -pr=android-ndk-r22b-arm64-v8a-no-lto-no-asan-ubsan
conan create . mylib/1.0@ -pr=android-ndk-r22b-arm64-v8a-no-lto-asan-ubsan
conan create . mylib/1.0@ -pr=android-ndk-r22b-arm64-v8a-lto-no-asan-no-ubsan

Or this?

conan create . mylib/1.0@ -pr=android_ndk/r22b@company/stable -pr:o=abi=arm64-v8a -pr:o=lto=False -pr:o=asan=False -pr:0=ubsan=False
conan create . mylib/1.0@ -pr=android_ndk/r22b@company/stable -pr:o=abi=arm64-v8a -pr:o=lto=False -pr:o=asan=False -pr:0=ubsan=True
conan create . mylib/1.0@ -pr=android_ndk/r22b@company/stable -pr:o=abi=arm64-v8a -pr:o=lto=False -pr:o=asan=True -pr:0=ubsan=True
conan create . mylib/1.0@ -pr=android_ndk/r22b@company/stable -pr:o=abi=arm64-v8a -pr:o=lto=True -pr:o=asan=False -pr:0=ubsan=False

Here I use -pr:o as an imaginary --profile-option flag that defines an option for profile-package, which then influences the build settings via the proposed update_settings method described in my previous comment.

The number of combinations that need to be built is the same, but the number of files in ~/.conan/profiles is much more manageable. Furthermore, if the parameter to -pr flag is in package-like format (name/version@user/channel), Conan can automatically fetch that profile from Artifactory - you don't even need separate conan install anymore. This can make -pr flag backwards compatible with the current state - if the parameter is not in package-like format, it will simply refer to the textual profile as today. In that case, the profile needs to exist in ~/.conan/profiles folder and should be previously installed with conan config install.

Of course, my proposal is not to replace the current profile mechanism, but to extend it with above features.

[memsharded]

I still not see it. The advantage that you are describing as user story is doable as today with:

conan create . mylib/1.0@ -pr=android_ndk_r22b -o android_ndk:abi=arm64-v8a -o android_ndk:lto=True ...

You don't need to have a combinaric explosion of profiles. You can fine tune with added settings and options on the command line, you can include() them, you can compose them in command line as well:

conan create . mylib/1.0@ -pr=android_ndk_r22b -pr=v8_lto_asan

From Conan 1.38, you can also define jinja templates for profiles, that could define values on the fly based on environment or platform conditions.

Having the profile directly fetching and defining configuration could be really problematic:

Furthermore, if the parameter to -pr flag is in package-like format (name/version@user/channel), Conan can automatically fetch that profile from Artifactory - you don't even need separate conan install anymore.

The problem there could be exactly that, that the proposal is that such package doesn't contain only the profile, but also the package itself. To resolve the package itself, the evaluation of settings, options, etc is necessary. No package is downloaded from the server without evaluating first its package_id, that is a result of settings + options + dependencies. If the settings and options of this package are defined inside the package itself, then it is directly not possible to evaluate it without errors.

[DoDoENT]

I still not see it. The advantage that you are describing as user story is doable as today with:
conan create . mylib/1.0@ -pr=android_ndk_r22b -o android_ndk:abi=arm64-v8a -o android_ndk:lto=True ...

🤔, but this still requires having android_ndk_r22b profile to be in sync with android_ndk/r22b@ package, i.e. it's easy to make an error to put, e.g. compiler.version=12 in profile, while the actual version of the compiler in the android_ndk/r22b@ package would be different. Also, every package should be aware of the android_ndk:lto option in its package_id function to ensure that the package ID changes whenever this option in android_ndk changes (this makes it essentially a setting, not an option).

I wasn't aware of the possibility to compose the profiles - that looks like it could help a lot with the combinatoric explosion of the profiles. It could basically provide the feature mentioned in your NDK example whilst ensuring different package ID for every package due to tweaking of custom settings, without the need for every single package to be aware of the android_ndk package. Thank you for the suggestion!

Also, I haven't played yet with jinja templates for profiles - I'll definitely explore that.

The problem there could be exactly that, that the proposal is that such package doesn't contain only the profile, but also the package itself. To resolve the package itself, the evaluation of settings, options, etc is necessary. No package is downloaded from the server without evaluating first its package_id, that is a result of settings + options + dependencies. If the settings and options of this package are defined inside the package itself, then it is directly not possible to evaluate it without errors.

I understand 😞. My idea with the proposal was mostly to make the DevOps story simpler and more similar to the user story, but I see now that the build configuration complexity must be expressed somewhere and why it's not possible to have that in one place (package recipe) as my proposal suggested as this would make implementation on Conan side too complex.

I'll close this issue so that it doesn't confuse people. We can still discuss the possibility of conan config install <pkg-ref>, but without the features in my proposal, I don't see the value of having this. Even today it's possible to deploy the configuration zip to Artifactory's generic repository or use Jenkins' archiveArtifact feature. Even if you need configuration versioning, it's easy to achieve with different zip names/deployment locations. The <pkg-ref> would only standardize the deployment layout - that remains as the sole benefit of this proposal. But, as I said, let's close this issue and eventually open a new one focusing only on having configuration files as a conan package in the Conan repository and see if there is any real value in that.