wrapped_clang is not built with the requested Xcode versions

[keith]

Description of the problem / feature request:

Currently wrapped_clang is built as part of the local_config_cc setup:

bazel/tools/cpp/osx_cc_configure.bzl

Lines 111 to 121 in 0b5f341

	xcrun_result = repository_ctx.execute([
	"env",
	"-i",
	"xcrun",
	"clang",
	"-std=c++11",
	"-lc++",
	"-o",
	"wrapped_clang",
	wrapped_clang_src_path,
	], 30)

Currently it doesn't take into account what you pass for --xcode_version. Because of this if you change your xcode-select on macOS and build with a different version of Xcode than is requested with this flag, you produce a different wrapped_clang. This ends up breaking remote cache compatibility with other versions even if the build would otherwise be done with the correct Xcode version.

Bugs: what's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

• sudo xcode-select -s /Applications/Xcode-10.2.1.app
• bazel build some_ios_target --xcode_version=10.2.1 --remote_http_cache=...
• bazel clean --expunge
• sudo xcode-select -s /Applications/Xcode-11beta3.app
• bazel build some_ios_target --xcode_version=10.2.1 --remote_http_cache=...

With the second build, you will get nearly no remote cache hits because wrapped_clang differs as an input to every action, even though you requested the same Xcode version from the command line.

What operating system are you running Bazel on?

macOS

What's the output of bazel info release?

release 0.28.0

[keith]

This would be less of an issue if changing Xcode versions globally with xcode-select invalidated wrapped_clang, but this requires a clean --expunge to recover from

[keith]

@scentini do you have a suggestion for how to fix this? Having to clean --expunge whenever switching versions here is a pain for our large team

[scentini]

cc/ @hlopko

[keith]

@hlopko do you have any suggestion for how to fix this?

[brentleyjones]

This should also potentially consider the DEVELOPER_DIR environment variable, per my duped issue: #11716.

[keith]

Would passing that through cause invalidation when the user's Xcode version changed? Maybe if we correctly accessed it in the workspace rule it could help?

[brentleyjones]

If the user set their Xcode version that way, then yes. But if all they did was change xcode-select -s, I don't believe so.

[keith]

Yea, I was at least thinking that in your bazel wrapper you could always specify that var if it would fix this

[brentleyjones]

Ohh, yeah, it could read the set value and then export it. The workspace rule can state that it should invalidate based on the environment variable. I think that would work nicely.

It would differ from --xcode_version though.

[oquenchil]

If the PR above didn't fix this issue, please re-open.

[brentleyjones]

We can specify the DEVELOPER_DIR in a bazel wrapper now after #11719, but I'm not sure if @keith's original issue was fixed.

[keith]

Using DEVELOPER_DIR in a wrapper around bazel definitely works around the original issue, but it would be nice if it was solved for everyone without having to manually do that.

[keith]

One thing I realized with the DEVELOPER_DIR workaround is that if the user overwrites their Xcode.app with a newer version, the DEVELOPER_DIR will not change, but the version of Xcode will. So that's a case where this workaround isn't enough 😞

[jerrymarino]

@keith one way to sidestep this is to just use a custom local_config_cc. e.g.

bazel fetch :*  && ditto $(bazel info execution_root)/../../external/local_config_cc tools/local_config_cc

# WORKSPACE
local_repository(name="local_config_cc", path="tools/local_config_cc")

YMMV for different host arch than x86_64 🤔

It might be possible to factor out the binaries into their own input to the cc rules rather than compiling in the repo rule, but I don't know the long term plans for this / overlap with rules_cc off the top of my head!

[keith]

copying that in is definitely an interesting idea, we also run some things in our build on Linux so I think we'd have to do that in a cross platform way though. Definitely handling the arch problem (although you could check fat binaries in there instead if you only supported macOS) would be necessary too

I think doing the input route is easier said than done because it'd be a bit of a circular dependency

[jerrymarino]

@keith in order to make the binary an input to the toolchain w/o a circle you'll need another rule to compile it: e.g. a rule that consumes only the relevant xcode config bits and not the custom toolchain.

As the toolchain will be specific to macOS, you might need to override local_config_cc conditionally when on macOS to use the hardcoded one. You might be able to get away with switching the entire generated toolchain with --override_repository instead of having to create a cross platform one.

[jerrymarino]

For what it's worth, it seems like making wrapped_clang binaries an input to the toolchain seems like good way to fix this outside of forking the generated one

[jerrymarino]

I've put up a WIP commit to test out compiling the wrappers with a rule that depends on the Xcode env. It looks like we can compile this with the apple_common.apple_host_system_env( and then point the tool path at the output of the rule.

It seems like this is the right approach - plus fixing the last TODO is addressed on that commit. Does anyone see any downsides to compiling this with a rule? If not I can prepare that commit for a PR cc @keith @hlopko

[brentleyjones]

apple_host_system_env is nice, since it includes XCODE_VERSION_OVERRIDE, would would be different for different Xcode versions, even with the same DEVELOPER_DIR.

[jerrymarino]

@brentleyjones yep passing the entire Xcode env is how I was looking to solve it for good. This is similar to how other C compiler rules work 👍 #11716 alone won't help the use case where people use Xcode mac app store to install Xcode to `/Applications/Xcode.app

The 3 ways I know of to fix this are:

• clean the repo cache when the Xcode version changes
• manage the toolchain binaries in another way ( e.g. commit them ) - likely this is what others are doing
• compile them with the apple_host_system_env and pass paths into the toolchain

[brentleyjones]

To recap since this has bitten me again: if you have --xcode_version set, you need to make sure DEVELOPER_DIR or xcode-select -s is set to the same version. Otherwise your wrapped_clang binary might mismatch. On a mismatch you probably won't get any remote cache hits.

We probably want a fix that uses apple_host_system_env as mentioned above.

[github-actions]

Thank you for contributing to the Bazel repository! This issue has been marked as stale since it has not had any activity in the last 1+ years. It will be closed in the next 90 days unless any other activity occurs or one of the following labels is added: "not stale", "awaiting-bazeler". Please reach out to the triage team (@bazelbuild/triage) if you think this issue is still relevant or you are interested in getting the issue resolved.

[brentleyjones]

@bazelbuild/triage not stale