Use always build and host contexts and profiles

[memsharded]

Use always, for all install/create commands both the "build" and the "host" profiles,
being the "build" one the configuration for the current building machine, and "host" the configuration
for the system in which the final built application will run.

If one or both of the build and host profiles is not specified, the "default" one will be used.
These commands will be equivalent and do the same:

$ conan install .
$ conan install . --profile=default
$ conan install . --profile:host=default --profile:build=default

The behavior is currently available in Conan 1.X, as opt-in (specifying --profile:build), the proposal is to make this the standard behavior, having always both build and host contexts.

---

• Upvote 👍 or downvote 👎 to show acceptance or not to the proposal (other reactions will be ignored)
  • Please, use 👀 to acknowledge you've read it, but it doesn't affect your workflow
• Comment and reviews to suggest changes to all (or part) of the proposal.

[petermbauer]

May i suggest to introduce separate default profiles then for build and host. Developers always cross-compiling (embedded) would use this so that there is no need to specify a profile when building for the default target platform.

[steinerthomas]

So one UC for the flag force_host_context=True would for example be: gtest as a build require (e.g. build_requires = "gtest/1.8.1")? Am I right?

[ytimenkov]

May i suggest to introduce separate default profiles then for build and host. Developers always cross-compiling (embedded) would use this so that there is no need to specify a profile when building for the default target platform.

There is already setting default_profile. I guess it feels natural to split it into 2.

[Da-LiFe]

May i suggest to introduce separate default profiles then for build and host. Developers always cross-compiling (embedded) would use this so that there is no need to specify a profile when building for the default target platform.

There is already setting default_profile. I guess it feels natural to split it into 2.

Furthermore, I propose to also provide an additional env var such as CONAN_DEFAULT_BUILD_PROFILE_PATH (cp CONAN_DEFAULT_PROFILE_PATH)

[memsharded]

@petermbauer @ytimenkov @Da-LiFe

I initially was thinking about having both a default and default_build profile to be automatically created in the cache. But sounds that it could be confusing, because changing the default_build one, that by nature should be different in each platform, would make it useless for distribution with conan config install for example.

So I think that @ytimenkov suggestion of providing default_profile configuration in "conf" is better, as it will easily allow to define a named profile naturally like default_profile=arm-cross-buildxxx in conf, that will not affect the default_profile_build that could remain by default default_profile_build=default or could be changed independently. I suggest to add this idea and integrate it in the proposal.

@Da-LiFe

Regarding CONAN_DEFAULT_BUILD_PROFILE_PATH, at the moment we are reconsidering the Conan configuration mechanisms. The new [conf] experimental mechanism does not have env-vars counterparts (https://docs.conan.io/en/latest/reference/config_files/global_conf.html), and we wouldn't like to have an env-var for every possible configuration, because they are highly increasing, and not manageable, while [conf] will be able to be managed by conan config install too. We will wait a bit until this mechanism is matured to decide which env-vars would make sense.

[memsharded]

@steinerthomas

It is not very clear what you mean with the gtest and force_host_context=True. By definition, the "test" requires like gtest should live in the same context as the application being built, as it links with the package code, that is in the host context. So the idea is that "test_requires" will receive the "host" profile and live in the "host" context. This is the current implementation in Conan 1.X.

It is possible that we might introduce a more explicit definition like test_requires = "gtest/...", to make it clearly different from build_requires as cmake, because they are wildly different things in essence, and it is a bit confusing to have them under the same name.

[ytimenkov]

I think this is somewhat similar to code generators (like protobuf with generator running in build context and library being linked in host context), but slightly different: there is a library and a runner (for example), but library is linked into test code and shouldn't really affect the package id. However I'm not sure if it's too different: test framework library could be a private host requirement and runner a build one.

Although running tests in such way with cross-compiled code is tricky. IIRC there were examples which included emulators as build requirement.

[memsharded]

I think this is somewhat similar to code generators (like protobuf with generator running in build context and library being linked in host context), but slightly different: there is a library and a runner (for example), but library is linked into test code and shouldn't really affect the package id. However I'm not sure if it's too different: test framework library could be a private host requirement and runner a build one.

Yes, that is the idea. The test_requires would have the same "private" trait that build_requires have, but will live in the "host" context, and not affecting the final package-id.

Although running tests in such way with cross-compiled code is tricky. IIRC there were examples which included emulators as build requirement.

That is the case that would be supported, even if not easy or very common, we were reported in the past of people doing this, so we want to have a model for it.

[Da-LiFe]

Regarding CONAN_DEFAULT_BUILD_PROFILE_PATH, at the moment we are reconsidering the Conan configuration mechanisms. The new [conf] experimental mechanism does not have env-vars counterparts (https://docs.conan.io/en/latest/reference/config_files/global_conf.html), and we wouldn't like to have an env-var for every possible configuration, because they are highly increasing, and not manageable, while [conf] will be able to be managed by conan config install too. We will wait a bit until this mechanism is matured to decide which env-vars would make sense.

We are not using the default profile in our setup, because our devs do not have a default platform. We are leveraging the fact that our machines have set the profile to use in the shell env already since ages. The env is called HOST_PLATFORM and is set e.g. to osl15-x86_64. We are distributing matching conan profiles via conan config install, i.e. we have a conan profile also called osl15-x86_64. A tiny wrapper around conan then translates HOST_PLATFORM to CONAN_DEFAULT_PROFILE_PATH=<path_to_cache>/profiles/$HOST_PLATFORM. This way, users can use conan without having to specify any profile and without knowing the current HOST_PLATFORM of their machine by themselfs.

However, I'm not sure how we would be able to map the HOST_PLATFORM env var to the new [conf] section. Maybe you could elaborate some suggestions?

[steinerthomas]

@steinerthomas

It is not very clear what you mean with the gtest and force_host_context=True. By definition, the "test" requires like gtest should live in the same context as the application being built, as it links with the package code, that is in the host context. So the idea is that "test_requires" will receive the "host" profile and live in the "host" context. This is the current implementation in Conan 1.X.

It is possible that we might introduce a more explicit definition like test_requires = "gtest/...", to make it clearly different from build_requires as cmake, because they are wildly different things in essence, and it is a bit confusing to have them under the same name.

@memsharded
In the "Conan Advanced Training" (ConanTrainingAdvanced.pdf) you have shown us in "Exercise 20 – Gtest as build-require [/gtest]" that we should use gtest as build_require, so that we do not pollute our package-id.

And so we did in our Conan 1.X packages, so yeah we'll need something like a test_requires in Conan 2.X. Can you mention this test-requires feature in this PR? Otherwise I can't agree with that solution.

[memsharded]

@memsharded
In the "Conan Advanced Training" (ConanTrainingAdvanced.pdf) you have shown us in "Exercise 20 – Gtest as build-require [/gtest]" that we should use gtest as build_require, so that we do not pollute our package-id.

Yes, the training is defined for single "everything is host" context.

When the build/host profiles are used now in Conan 1.X, it is necessary to define that gtest is in the host context. This can be done with the syntax:

def build_requirements(self):
     self.build_requires("gtest/1.1.1", force_host_context=True)

This can be tested right now, just passing a --profile:build to your command line.
And the above code is necessary, otherwise, it will identify it as a "build tool" like cmake, and put it in the build context, which means that it will not link in cross-building scenarios.
This is not related to the package_id, both "cmake" and "gtest" will still have no effect by default on the package_id, but when the 2 contexts are used, it is necessary to differentiate which "build-requires" goes to which context:

• build-requires in the "build" context, like "cmake" typically inject environment, like putting itself in the PATH env-var, so an application like "cmake" can be found and run. But they do not provide includes or libraries, because they are not linked.
• build-requires in the "host" context, like "gtest" can define headers, libraries to link, and they can be translated to "gtest-config.cmake" or equivalent build-system files.

The above idea of test_requires is simplified syntax for the above "build_requires + host context" syntax, that has been being used in Conan 1.X already for a while.

Please let me know if this makes sense.

[steinerthomas]

Please let me know if this makes sense.

self.build_requires("gtest/1.1.1", force_host_context=True) exactly that was, what I meant with my initial question.
Makes sense, thanks for clarification. Here you also mentioned the feature: conan-io/conan#8292 (comment)

[DoDoENT]

I am strongly in favour of this proposal. However, it's important for the model to also support use cases described in this bug report - notably that the tool, used in the build context, may provide some modules/scripts/whatever to be used in the host context.

[memsharded]

I am strongly in favour of this proposal. However, it's important for the model to also support use cases described in this bug report - notably that the tool, used in the build context, may provide some modules/scripts/whatever to be used in the host context.

Absolutely yes. We are iterating on CMakeDeps to provide a better experience, including providing build modules for the build context (that are used to build a package in the "host" context, but that run in the build context, as they belong to a build-requires in the build context). We will evolve that in Conan 1.X next releases.

[memsharded]

@Da-LiFe

I see, yes, that means that CONAN_DEFAULT_PROFILE_PATH might be one of the few env-vars that we could define in Conan, to have that configured per-machine, something that cannot be done with the conan config install configuration.

[ytimenkov]

Doesn't conan config install merge config file? Then just make sure that default_build_profile is not defined in the shared settings.

In the absence of default build profile we have a convention to have a profile named "local" which should contain all build machine-specific settings. It may just include other named profile if there's a suitable one.

[memsharded]

Doesn't conan config install merge config file? Then just make sure that default_build_profile is not defined in the shared settings.

Yes, but if I understood @Da-LiFe that would still require to execute some conan config set ... default_build_profile=something in their machines, something different in each machine, and then it is easier to automate the provision of the machines with an env-var, that will be valid for all Conan caches there, instead of having to run that conan config set for each new cache.

In any case, we are talking here about a relative minor UX thing, iiuc, it doesn't alter the core of the proposal.

[foundry-markf]

My question is just about clarification.

As we never use the default profile (instead having very specific profiles, stored in a git repo, for various desktop OSs), and we don't cross compile, can I confirm that the intent is that

conan install . --profile=customprofile

will set both host and build profiles to 'customprofile'? i.e. unchanged behaviour from 1.x.

Asking as I was concerned by the statement

If one or both of the build and host profiles is not specified, the "default" one will be used.

if we only specify the one --profile argument.

[memsharded]

Hi @foundry-markf

Yep, there are 2 possibilities:

• If profile build is not explicit, then it fallback to --profile:build=default
• If profile build is not explicit, then it fallback to --profile:build=<same-as-profile-host>

I can see pros and cons for both:

• For people that usually cross-compile the first one makes more sense, as the current machine default is a better option than the cross-compiled target 100% of cases
• For people that usually build native, then the second use case might make more sense.

At the moment, it seemed a bit more natural to use something closer to the current behavior (now in Conan 1.X, if you specify for example -s:b compiler=gcc it will also use the "default" profile as base build profile.

Whatever the default is, we could implement a conf item that will opt-in into the other behavior as default. I am fine with either, whatever the majority wants, my only concern is that we might not get enough attention from the tribe to this specific comment, and creating a whole new proposal for this seems a bit overkill. But lets try here:

---

• If you want: If profile build is not explicit, then it fallback to --profile:build=default Upvote 👍 THIS COMMENT
• If you want: If profile build is not explicit, then it fallback to --profile:build=<same-as-profile-host> downvote 👎 THIS COMMENT
• Please, use 👀 to acknowledge you've read it, but it doesn't affect your workflow

[datalogics-kam]

I like

If profile build is not explicit, then it fallback to --profile:build=<same-as-profile-host>

Because in a lot of cases, they're going to be the same, and it's less typing.

There are also other places where one option defaults to another, so this is a pattern that's already there. As in conan build:

  -if INSTALL_FOLDER, --install-folder INSTALL_FOLDER
                        Directory containing the conaninfo.txt and conanbuildinfo.txt files (from
                        previous 'conan install'). Defaulted to --build-folder

For us, default is almost never what we want, it's just convenient to have something. Almost everything we build is from a specific profile, because it's probably customized, and it's more explicit than the default that Conan created up front.

[Daniel-Roberts-Bose]

May i suggest to introduce separate default profiles then for build and host. Developers always cross-compiling (embedded) would use this so that there is no need to specify a profile when building for the default target platform.

This, please. If we are going to make this behavior the new normal, then I feel we need to be able to specify different defaults for the host and build contexts. We are actually currently in the process of trying to move away from simply doing conan install --profile in favour of conan install -pr:h -pr:b but it is proving challenging because if a user still does conan install -pr it will try to build things like CMake for our cross compiled target rather than the build system

As another side note, this is unlikely to change but calling them host and build really confuses many people on my team. In our minds, host is the current computer we are compiling on and not the cross compiled target we are compiling for. A name like target would make a lot more sense in my opinion. Something like conan install -pr:target

[ytimenkov]

Even if not cross compiling I would like to have build_type==release always for build profile and debug for host (or at least vary).

Would be unpleasant surprise to get non-optimized tool or even compiler. And build profile is not what I'd like to specify on the command line or change often.

[a4z]

What ever you decide to use, please do not implement a conf item that will opt-in into the other behavior.
I would hate to not know what goes on on other machines and that you might need to query, or forget about the default behaviour just to be on the right site (what would mean, a default does not exist, so you do not need to implement one)
We have already enough complexity, and making everything configurable sounds nice, but makes the number of execution paths sky rock

[memsharded]

Hi @Daniel-Roberts-Bose

This, please. If we are going to make this behavior the new normal, then I feel we need to be able to specify different defaults for the host and build contexts. We are actually currently in the process of trying to move away from simply doing conan install --profile in favour of conan install -pr:h -pr:b but it is proving challenging because if a user still does conan install -pr it will try to build things like CMake for our cross compiled target rather than the build system

Yes, this will be possible, you will be able to configure different defaults for build and host context, this was agreed above.

As another side note, this is unlikely to change but calling them host and build really confuses many people on my team. In our minds, host is the current computer we are compiling on and not the cross compiled target we are compiling for. A name like target would make a lot more sense in my opinion. Something like conan install -pr:target

I agree with that, and I found it myself a bit counter intuitive when I first saw it. But we were pushed by the community to adopt this convention, because it is the GNU one: https://www.gnu.org/software/automake/manual/html_node/Cross_002dCompilation.html

[memsharded]

What ever you decide to use, please do not implement a conf item that will opt-in into the other behavior.
I would hate to not know what goes on on other machines and that you might need to query, or forget about the default behaviour just to be on the right site (what would mean, a default does not exist, so you do not need to implement one)
We have already enough complexity, and making everything configurable sounds nice, but makes the number of execution paths sky rock

Ok, that could make sense, lets try to come up with a default and try to respect and go with it without defining an alternate configuration, (unless later it becomes a big pain for a big amount of users).

[memsharded]

@ytimenkov

Even if not cross compiling I would like to have build_type==release always for build profile and debug for host (or at least vary).

Would be unpleasant surprise to get non-optimized tool or even compiler. And build profile is not what I'd like to specify on the command line or change often.

This would be the behavior that you achieve with the first proposal --profile:build=default, as the default profile is Release.

THIS IS IMPORTANT, TRIBE, the second approach will by default put build-requires in the build context in the Debug mode when building for host build_type=Debug, because unless specified, it will take the same profile as the host one. For me this would be an important "con" for the second approach.

[foundry-markf]

Thanks for considering my question in #23 (comment). Sorry if it's turned into more of a problem to solve though.

As a related aside, and putting it out there for discussion, it has occurred to me (long before this discussion), that as we never use the default profile, whether it would be possible to put Conan in a mode to never create a default profile. If all profile switches were omitted, then it would be an error in that mode. (This is a problem we have now, devs forgetting to set the profile, and seeing different behaviour to others who do set it.)

It may not work for everyone reading the above comments, but I think I'd be happy if Conan had that mode (which we would enable), and would thus say always use the same profile for both host and build if only one was specified. (Obviously, you can still be explicit with both profiles.)

If the mode wasn't enabled (the default Conan behaviour perhaps?), then specifying only one profile, the build profile would map to the default profile.

[puetzk]

whether it would be possible to put Conan in a mode to never create a default profile

ACK this. I usually intentionally create an incomplete default profile so if conan ends up using it, it just fails fast.

Furthermore, I propose to also provide an additional env var such as CONAN_DEFAULT_BUILD_PROFILE_PATH (cp CONAN_DEFAULT_PROFILE_PATH)

This too, it would also be really nice to have a good way for IDE things like Qt creator kit or CMake presets to provide the new separated build/host profiles (could be support in cmake-conan for getting them out of the CMake cache variables, or in conan itself for environment variables - conan has CONAN_DEFAULT_PROFILE_PATH, but not equivalent for host/build profile (and memsharded seemed pretty down on this in conan-io/conan#6476 and conan-io/cmake-conan#323 hasn't gone anywhere either). In general when conan runs the build the profile drives the toolchain; when the IDE runs the build I'd like its choice of toolchain to drive conan, so that in both cases they match up.

[Da-LiFe]

THIS IS IMPORTANT, TRIBE, the second approach will by default put build-requires in the build context in the Debug mode when building for host build_type=Debug, because unless specified, it will take the same profile as the host one. For me this would be an important "con" for the second approach.

We are distributing some common optimization profiles to our devs, e.g.

opt-<arch>-<compiler>-standard, which 95% of our machines are capable
opt-<arch>-<compiler>-edge, which <20% of our machines are capable

When using ...-edge as host profile together with the second approach, the build tools would get highly optimized and potentionally whould not run anymore on the build machine. Therefore, I upvoted --profile:build=default (hoping to have CONAN_DEFAULT_BUILD_PROFILE_PATH to dynamically specifiy the default based on the env).

[DoDoENT]

whether it would be possible to put Conan in a mode to never create a default profile

ACK this. I usually intentionally create an incomplete default profile so if conan ends up using it, it just fails fast.

+1 for this from my side as well.

This too, it would also be really nice to have a good way for IDE things like Qt creator kit or CMake presets to provide the new separated build/host profiles...

I solved this by putting explicit invocation of -pr:b=default in my wrapper around cmake-conan. This is then IDE agnostic - can be used from any IDE that supports CMake (VSCode, VS, QtCreator, vim, ...)

[ansutremel]

As a related aside, and putting it out there for discussion, it has occurred to me (long before this discussion), that as we never use the default profile, whether it would be possible to put Conan in a mode to never create a default profile. If all profile switches were omitted, then it would be an error in that mode. (This is a problem we have now, devs forgetting to set the profile, and seeing different behaviour to others who do set it.)

Agreed, you can quickly face random issues across different dev/build setups only caused by auto-created default profiles, and to avoid trapping into this again we disabled the default profile across the board to fail quickly as mentioned above already:

[DoDoENT]

@ansutremel , we tried that solution as well, but it didn't work for us, as we are often cross-building and our cross-build profiles rely on arch_build and os_build from the default profile. So, we ended up distributing default profiles for Mac, Windows and Linux to our devs, which is far from perfect.

We are currently not able to use build and host profiles in Conan 1.X due to this issue.

[memsharded]

This is very interested feedback!

Indeed we have thought of NEVER creating a default profile, and if users don't provide just raise: "HEY YOU NEED TO CREATE A DEFAULT PROFILE".

It was done this way to try to ease the live of onboarding people, newbies, etc., but we are now more about being super-explicit in everything, and having a default profile auto-generated that magically tries to guess your configuration doesn't really help the users, I think it is better if we make this step explicit.

This is probably another UX big enough change to submit it to the Tribe as a separate issue, lets do it.

[foundry-markf]

I've upvoted the PR, as I see the benefits on making it clearer for cross compilation.
But I also acknowledge that our current usage of Conan (no default profile, native only builds) will have a knock on effect with the proposal as it is. (Always knew we'd have to change something for Conan 2.)

[ytimenkov]

but we are now more about being super-explicit in everything, and having a default profile auto-generated that magically tries to guess your configuration doesn't really help the users,

Maybe it's worth providing a special hook for config install command which overrides default behavior.

I assume that then one don't want to have a default profile there is a central configuration.
What I did is set default profile in config to empty to prevent autogeneration or let conan use it automatically.

But for open source projects out may be valuable to have default profile generated. Having per project overrides makes it less attractive and more complex to start working on something.

[memsharded]

Maybe it's worth providing a special hook for config install command which overrides default behavior.

The problem with such a hook, is that you need to install it too, and once you are doing it, it is easy to do whatever you want with the default profile.

But for open source projects out may be valuable to have default profile generated. Having per project overrides makes it less attractive and more complex to start working on something.

Yes, sure, this has been our default so far. But it is also true that open source users are also been biten by the default libcxx=libstdc++ configuration, as the default profile automatic generation is typically not being read, even if printed to stdout. Of course that would be the major drawback of such approach, but on the other hand, having open source users learn about the profile, what it is, what it means, instead of doing a blind "conan install" and then having binary compatibility issues, could probably alleviate some of the current onboarding issues.

[memsharded]

This PR is being approved and merged, 46 upvotes, no downvotes.
Feedback has been incorporated in the document, with UX details about how the default profile would be defined if not explicitly defined by the user in command line or config.

[ytimenkov]

The problem with such a hook, is that you need to install it too, and once you are doing it, it is easy to do whatever you want with the default profile.

Thinking of it a bit more I think it's a good idea. For example with "please-don't-use-default-profile" setting people may have troubles figuring out what to do.

Also if settings.yml are extended default profile detection may not work as new settings might need to be taken into account and hook could do this. Or (as in my case) most developers end up in build failures with obscure errors and have to find that they need to define those extra settings themselves.

I'm striving for "Just works (tm)" for ordinary developers. So if your environment has customized settings / profiles then it's the same conan config install command is used to get environment ready. Maybe even post-config-install hooks to check configuration.