Bazel CI: rules_haskell failing with Bazel@HEAD in downstream

[meteorcloudy]

After the fixes of #1653 and #1655, there are new failures with Bazel@HEAD

https://buildkite.com/bazel/bazel-at-head-plus-disabled/builds/1135#bd2706c7-71d5-4ff3-ba18-0db702a7456a

(10:47:02) ERROR: /var/lib/buildkite-agent/.cache/bazel/_bazel_buildkite-agent/1a898b3d70076f020535b91eba1e94e2/external/linux_amd64_asterius/BUILD:793:18: While resolving toolchains for target @linux_amd64_asterius//:toolchain-impl: no matching toolchains found for types @build_bazel_rules_nodejs//toolchains/node:toolchain_type

This is reproducible with Bazel 5.0.0rc2:

export USE_BAZEL_VERSION=5.0.0rc2
bazelisk build --build_tag_filters=-requires_nix,-requires_lz4,-requires_shellcheck,-requires_threaded_rts,-dont_test_with_bindist,-dont_test_on_bazelci //tests/...

[meteorcloudy]

/cc @aherrmann

[aherrmann]

@meteorcloudy Thank you for notifying!

I can reproduce this locally:

$ USE_BAZEL_VERSION=5.0.0rc2 bazel test //... --nobuild --toolchain_resolution_debug=nodejs.*
...
... Rejected toolchain @nodejs_linux_amd64_config//:toolchain; mismatching values: linux, x86_64
...
ERROR: ... While resolving toolchains for target @linux_amd64_asterius//:toolchain-impl: no matching toolchains found for types @build_bazel_rules_nodejs//toolchains/node:toolchain_type
$ USE_BAZEL_VERSION=4.2.2 bazel test //... --nobuild --toolchain_resolution_debug
...
... Selected execution platform @local_config_platform//:host, type @build_bazel_rules_nodejs//toolchains/node:toolchain_type -> toolchain @nodejs_linux_amd64_config//:toolchain, ...
...

I think this has to do with the Asterius toolchain depending on the NodeJS toolchain and the change of the default of incompatible_override_toolchain_transition in Bazel 5.
In particular, I can get Bazel 5 to pass with

$ USE_BAZEL_VERSION=5.0.0rc2 bazel test //... --nobuild --toolchain_resolution_debug=.* --noincompatible_override_toolchain_transition

and Bazel 4 to fail with

$ USE_BAZEL_VERSION=4.2.2 bazel test //... --nobuild --toolchain_resolution_debug --incompatible_override_toolchain_transition

@ylecornec could you take a look at this? A good fix may be to obtain the NodeJS toolchain not via a direct toolchain dependency but through a tool dependency with cfg = "exec".

[meteorcloudy]

rules_haskell is now green with Bazel@HEAD: https://buildkite.com/bazel/bazel-at-head-plus-disabled/builds/1142#8012be70-45bd-4ffb-8628-d249462fff44

I'll re-enable it in downstreaml