Propagate exec_properties from toolchains

[moroten]

Description of the feature request:

The field exec_properties in a toolchain does not propagate to remote execution.

As a rule can depend on multiple toolchains, I would expect all the exec_properties from all toolchains to be merged to the action. Manually propagating exec_properties to ctx.actions.run would be difficult because many rule developers would most probably forget to do that and ctx.actions.run is also missing the field exec_properties.

What underlying problem are you trying to solve with this feature?

From https://groups.google.com/g/bazel-discuss/c/YeEsT4q037c/m/CoZmpaR1AgAJ:
FWIW: My particular use case is to inform the remote execution about tests built by thread/address/memory sanitizer C++ toolchains since such tests normally require more resources to be allocated; one can invent a suitable cc_test wrapper macro and inject the desired test.xxx exec_properties, but I would prefer to avoid such hassle.

Another use case is that some toolchains depend on installed licenses, which would be nice to propagate this way.

The workaround is to create a number of platforms that aims for a collection of toolchains for different languages and then pinpointing the toolchain resolution to connect the platforms with the specific toolchains.

Which operating system are you running Bazel on?

Linux

What is the output of bazel info release?

release 5.3.2

If bazel info release returns development version or (@non-git), tell us how you built Bazel.

No response

What's the output of git remote get-url origin; git rev-parse master; git rev-parse HEAD ?

No response

Have you found anything relevant by searching the web?

The only relevant GitHub issue with the word exec_properties is #16742 for propagating èxec_properties` constraint values.

One relevant thread on bazel-discuss without any answers so far: Toolchain specific exec_properties

Any other information, logs, or outputs that you want to share?

No response

[katre]

It is an accident that the rule toolchain has an exec_properties attribute, I plan to remove it in the near future. I don't see any specific use cases that can't be handled differently, and this feels very confusing.

[weedless]

Hi @katre!
I am struggling with a similar problem where we build a product for several architectures. When building, the dependency tree contains transitions into different platforms with corresponding toolchain resolutions. Some of the toolchains are license limited and only available on a remote execution cluster.
Is there a way to direct actions in a build to different remote execution workers depending on selected toolchain? It is not supported by the spawn_strategy, I have not found a way to add it to the transition and the exec_properties of the toolchain rule does not propagate.
Is there another obvious way of handling this use case?

[katre]

I discussed this with @moroten at BazelCon (it was great to talk to you), and I think this is probably the proper solution:

First, define constraints and execution platforms for your different execution clusters:

constraint_setting(name = "licenses")
constraint_value(name = "foobar_license", constraint_setting = ":licenses")
# repeat as needed
platform(
    name = " cluster1",
    constraint_values = [
        "@platforms//os:linux",
        "@platforms//cpu:x86_64",
        ":foobar_license",
    ],
    exec_properties = { ... },
)

Then, define platforms for your specific target hardware:

platform(
    name = "device_1",
    constraint_values = [
        ":custom_cpu_constraint",
    ],
)

Finally, define the toolchains you need, giving them the licensing constraints:

toolchain(
    name = "special_licensed_toolchain",
    toolchain_type = "@bazel_tools//tools/cpp:toolchain_type",
    toolchain = ":licensed_cc_toolchain",
    exec_compatible_with = [
        ":foobar_license",
    ],
    target_compatible_with = [
        ":custom_cpu_constraint",
    ],
)

Then, when you bazel build --platforms=:device_1 //target, the only available toolchain for that CPU (:special_licensed_toolchain) will cause the execution platform to be :cluster1 and the proper exec properties will be forwarded.

[weedless]

Thanks for the input @katre! I followed your example and managed to map my targets to different remote workers depending on what {toolchain, execution platform} is resolved. However, I do not see how it can enable running some combinations of {toolchain, execution platform} remote and all rest locally. Is that use case supported by bazel?

I tried to solve it using --remote_local_fallback where some execution platforms are not available remote hence forcing it to fallback to run locally. This seems to work but has the serious drawback that local and remote builds will not share cache (since different execution platforms invalidate cache).

I would have expected the local/remote problem to be solved using spawn strategies. The parameters {strategy, spawn_strategy} enables mapping of actions and files to be build locally or remote. This is great but is also complicating things since the spawning is unrelated to the toolchain resolution and what execution platform is chosen. You could end up in a situation where an action is mapped remote and where there is no matching execution platform. I believe a better (or an add-on) solution would be to implement strategy-support for mapping actions based on {toolchain, execution platform/execution group}. What is your thoughts on this?

[katre]

Unfortunately, at the current time there is a divergence in support of execution platforms and execution strategies: while logically some execution platforms are only compatible with certain strategies (ie, a remote platform only works with the remote strategy, the host platform only works with the local or sandbox strategy, etc), this isn't part of Bazel's internal decision making and so it's required to keep the flags and execution platforms in sync manually. This is especially problematic with tests that are tagged as local, since that ends up either being ignored or just causing failures.

We have plans to deal with this but unfortunately not a schedule.