remote/correctness: Bazel doesn't track system libraries.

[buchgr]

Bazel currently does not track tools outside a workspace. This can be a problem if, for example, an action uses a compiler from /usr/bin/. Then, two users with different
compilers installed will wrongly share cache hits because the outputs are different but they have the same action hash.

[BenTheElder]

Is there a workaround for this currently?
Can we just do something like --action_env=COMPILER=<compiler version>?

[buchgr]

@BenTheElder Yep exactly. It's not just compilers though it's any tools that your build uses from your system that you did not explicitly specify really. So you might want to include some more info than just your compiler.

[edbaunton]

Would the use of toolchains help with this?

[ulfjack]

Our naive proposal is to allow each toolchain to add a toolchain identifier into the cache key(s). The risk is that a bug (or a design flaw) in the toolchain might cause incorrect caching, but, on the other hand, it leaves the door open to toolchains that want to cache across platforms. E.g., a Java 7 compiler on Mac and Linux outputs essentially the same files, so caching those might be safe even if the binaries are not bit identical.

[BenTheElder]

I forgot to follow up here: setting --action_env doesn't seem to be sufficient since not all actions consume this (?) I'm currently working around this by just making sure different caches are used for different toolchain versions.

[buchgr]

@BenTheElder It’s my understanding that this would be a bug in Bazel, correct
@ulfjack? Do you have a reproducer?

Something that works for sure with every action is —experimental_remote_platform_override. It takes as a value the text representation of the Platform message defined in the remote_execution.proto. It’s effectively just a key value pair.

[BenTheElder]

I don't have a great reproducer currently, in https://github.com/kubernetes/test-infra I modify the bazelrc to point at a local copy of bazel-remote-cache from within a debian+bazel docker container and then I build again from another container after upgrading the GCC version to gcc-7.

When doing this I noticed stale headers from gcc-4.9 being included as part of compiling https://github.com/google/protobuf as a dependency despite setting an --action_env=CC_VERSION=<exact version of the gcc package> each time so I ran the bulid with -s and didn't see the env being passed to the failed compilation step.

@buchgr where is --experimental_remote_platform_override documented?

[ulfjack]

See #3320. :-/

[buchgr]

@BenTheElder ohh I apologize for the misinformation - it was my understanding that --action_env is supported by all actions. Please use the platform override then - I shall document it on our website.

Essentially, you can pass it the following protobuf:

--experimental_remote_platform_override='properties:{name:"key1" value:"value1" name:"key2" value:"value2"}'

This is passed through to the computation of the action key for remote caching.