Bazel remote-caching does not properly treat volatile-status as if it never changes

[Panfilwk]

Description of the problem / feature request:

When a target depends on the file volatile-status.txt as an input, that target will not be rebuilt if volatile-status.txt changes and the target is cached locally. However, the remote cache does not behave in the same way, and a change in the hash of volatile-status.txt will cause the target not to read from the remote cache.

Bugs: what's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

Clone the repo test_volatile_status and set up a bazel remote cache. Then run the following commands at the root of the repo:

bazel build --remote_cache=<your_remote_cache> //:artifact
bazel build --remote_cache=<your_remote_cache> //:artifact 
# the above will read from the local cache, as expected

bazel clean
bazel build --remote_cache=<your_remote_cache> //:artifact
# this will not read from the remote cache

What operating system are you running Bazel on?

Ubuntu 16.04.6 LTS

What's the output of bazel info release?

release 0.29.0

What's the output of git remote get-url origin ; git rev-parse master ; git rev-parse HEAD ?

https://github.com/Panfilwk/test_volatile_status.git
da002118b6c79bd7e33b017221c772fbfba4f009
da002118b6c79bd7e33b017221c772fbfba4f009

Have you found anything relevant by searching the web?

This issue touches on this concept for local caching, but nothing about how the behavior of this file works with remote caching.

Any other information, logs, or outputs that you want to share?

exec1.log.txt and exec2.log.txt are the resulting parsed experimental execution logs from the first and third build commands following the debug instructions here

[buchgr]

The behavior you are seeing is the expected behavior. Bazel's local cache will not rerun an action if only volatile-status.txt changed but will rerun it otherwise. The remote cache does not know about volatile-status.txt and it shouldn't. volatile-status.txt is mostly used to for timestamping (=this binary was built at XX.XX.XXXX XX:XX).

What exactly is the use case you are trying to make work?

[Panfilwk]

For our CI builds that use Bazel, we have an issue that docker containers spun up for tests sometimes linger after the CI job is finished. We want to tag these containers with the CI build information (using volatile-status.txt) to determine which jobs are causing containers to linger and hopefully discover the root cause, or to allow each job to clean up it's own containers without interfering with other jobs.

We don't want the tests using these containers to rerun if only the build information has changed and we want to cache across multiple workers that could potentially run a job, hence the use of remote-caching.

[buchgr]

For our CI builds that use Bazel, we have an issue that docker containers spun up for tests sometimes linger after the CI job is finished.

That sounds like your machines aren't recycled after each run? Would it be an option to keep the Bazel server running between builds? This would give you the volatile-status.txt behavior you are looking for.

[buchgr]

To clarify: I haven't thought deeply about this problem nor do I have any experience with such a use case. So I am neither for nor against having this feature. However, implementing this in Bazel would require protocol support and so would need discussions with the API community.

@ulfjack @lberki @ericfelly might have more insights on how we handle this internally.

[lberki]

I vote for "Will not fix; too much effort" here.

What you are saying is certainly true, but bazel clean; bazel build --remote_cache=<stuff> is a rare use case and it's arguably not a "cached" build anyway (depending on your definition of "cached").

This, coupled with the fact that fixing this would require changes to the cache protocol makes the cost outweigh the benefits IMHO.

One thing you could do is to read the identifier (non-hermetically) from a local file as opposed to through volatile-status.txt. That way, you wouldn't get cache hit but still be able to tag the test invocation with the ID.

[ulfjack]

We're very reluctant to change the behavior of volatile-status wrt. remote caching or execution, because doing so can potentially cause incorrect cross-user caching. Better to have remote caching / execution be strict and loose a bit of performance there.

Now, if you need to tag actions in order to figure out where they're coming from, I think your best bet is to use a side channel, instead of trying to do it with action inputs / env variables. That is, whatever spins up those docker containers should add metadata to the container in a way that the action can't inspect, so it doesn't affect caching, but that you can inspect if you find a hanging container after the fact. I don't have much experience with containers, but I'm sure there's a way to do that. If you're using Bazel to spin up the container, we could add some code to Bazel to do that by default.

[Panfilwk]

That sounds like your machines aren't recycled after each run? Would it be an option to keep the Bazel server running between builds? This would give you the volatile-status.txt behavior you are looking for.

Machines are recycled, but there's no guarantee about which of several worker machines a CI job will run on, so we want the cache to be shared between all of them.

What you are saying is certainly true, but bazel clean; bazel build --remote_cache= is a rare use case and it's arguably not a "cached" build anyway (depending on your definition of "cached").

This is how official bazel documentation recommends debugging remote-cache hits

If this would require large changes to the structure of remote caching though, I think reading from a local file or other side channel as @ulfjack and @lberki suggested would work for our use case. I'll close this, but would appreciate any suggestions on how best to use a side channel in this way; current ideas are reading from an expected absolute path on CI workers, or maybe using the --action_env flag?

[buchgr]

@Panfilwk --action_env is taken into account for remote cache key computation. An absolute path from where your action happens to read from should work.

[lberki]

+1 for what @ulfjack said; suboptimal performance is not great, but cache poisoning is even worse.

As for the side channel to use, as long as you can tag a container with some sort of unique ID you can use to find the container again, the easiest thing that comes to mind is exactly what you suggested: reading the ID from a file at some fixed path, then printing it on the stdout of the test.

[mariusgrigoriu]

Not sure I understand something. Wouldn't the stamped output files be stored in the remote cache, and couldn't bazel just ignore volatile-status.txt and just download the previously built output file?

[mariusgrigoriu]

Or at the very least update the docs to state that bazel only pretends that the volatile file doesn't change in the context of a local incremental build.

[joeljeske]

Yea I agree, it seems like unless the volatile status can actually be omitted from the actions inputs during action digest for the remote cache, then volatile keys is very misleading; it appears to work during local development of stamping rules, but then fails in a primary way people expect to leverage it.

I also ended up side loading a status file for my stamping needs, but it seems like extra work that is prone to error that really should be handled by bazel.

[alanfalloon]

@Panfilwk Can we re-open this please? Once you add remote-execution, there is literally no way to "sneak" the version information to the remote builders in a way that avoids the action hash. This was the one and only way to have a remotely built target that embeds version info that doesn't cause it to rebuild every time.

[mariusgrigoriu]

I submitted some PR / issues to various rules (k8s, docker, etc.) that are affected by this to disable volatile status stamping but feels wrong to me. This really should be addressed here. I understand the concerns about correctness higher in the thread, but Bazel claims both Fast and Correct. This issue seriously impairs "Fast."

[alanfalloon]

@ulfjack @lberki Can you please reconsider fixing this. The listed work-around for this issue is to use no-remote-exec on the stamped action so that it can sneak access to volatile-status.txt without listing it as an input. But in combination with #9412 the no-remote-exec action will not get cached and will always re-run anyway.
So the combination of this issue and #9412 means that you have to choose between remote execution and stamping. The work-around was already a problem and killing build performance, but now it doesn't even work.

[ulfjack]

I don't work at Google anymore. That said, there is no technical way to fix this with a dumb remote cache, or with the current remote execution protocol.

Avoid stamping unless you absolutely positively must have stamped binaries - ideally, never enable stamping for developer builds, and require that everything that is deployed to production (or shipped to an app store or whatever) is built by an automated system that explicitly enables stamping.

[joeljeske]

Are you suggesting that stamping should occur in an artifact pipeline after bazel? How does google perform their stamping?

[lberki]

(Oh hello, @ulfjack !)

At Google we:

1. Try to inject build stamping information into the build as late as possible (so that changing build stamp information invalidates as few actions as possible)
2. Only used stamped builds for outputs that need it (e.g. ones that are to be released). This is reflected by the value of the --stamp command line flag being false by default.

[joeljeske]

Yea crane seems like a good solution. My setup includes more than just containers, and I wanted to have a unified solution for all my applications.

I’d love to hear how you end up solving this problem and how it could be used in a broader audience.

[siddharthab]

For all who are still struggling with this issue, consider just making your volatile status files constant through your workspace status command. You can use an environment variable to control this behavior, which can be set or unset depending on whether you are testing or deploying. If the status files themselves are constant, you don't have to worry about how they are handled by bazel and any language rules.

You can override bazel's default values for status vars as in this example here.

[dieend]

Hi, we use bazel and use the volatile-status to generate the Kubernetes YAML, and add the commit info to the Kubernetes object metadata. If the generated container image doesn't change on a newer commit, we don't expect the YAML info to change. It's important to have volatile-status work as documented when using remote-cache for us.

Even if we stamp it very late, it'll trigger unnecessary rollout if a change in the hash of volatile-status.txt will cause the target not to read from the remote cache.

How do we work around this?

[pauldraper]

The behavior you are seeing is the expected behavior.

Why do people want different caching behavior local vs remote? What the case that is trying to solve?

Seems like they should work similarly, no?

[burdiyan]

I think this issue definitely needs to be reopened, unless there's a dupe already. All the workarounds to pass information to build actions that would not affect the action key are very annoying, and error-prone.

[lbcjbb]

We are also facing this issue, which impact the efficiency of our continuous delivery and load and fill, unnecessarily, our remote cache. I have just spent a little time to find the origin of the problem, and if it can help, here a minimal example to reproduce the behaviour.

Example

WORKSPACE

BUILD.bazel

load(":myrule.bzl", "myrule")

myrule(name = "dummy")

myrule.bzl

def _myrule_impl(ctx):
    output_file = ctx.actions.declare_file(ctx.attr.name)

    ctx.actions.run_shell(
        outputs = [output_file],
        inputs = [ctx.version_file],
        command = "cp -f $1 $2",
        arguments = [
            ctx.version_file.path,
            output_file.path,
        ],
    )

    return DefaultInfo(
        files = depset([output_file]),
    )

myrule = rule(
    implementation = _myrule_impl,
)

A simple rule which copy the file volatile-status.txt to bazel-bin/dummy. Exactly the same example than test_volatile_status.

Scenarios

First build

We start from an empty environment. The Bazel local cache and the remote cache are fully cleaned.

bazel build \
    --stamp \
    --remote_cache=grpc://localhost:9092 \
    --experimental_remote_grpc_log=/tmp/grpc.log \
    //:dummy

bazel-bin/dummy:

BUILD_TIMESTAMP 1654449958

bazel-out/volatile-status.txt:

BUILD_TIMESTAMP 1654449958

As expected, the file, not present in the local cache and in the remote cache, is built and the remote cache is filled.
We can see all executed gRPC functions from the log file:

[05 Jun 2022 19:25:58.229] build.bazel.remote.execution.v2.ActionCache::GetActionResult - NotFound (2ms)
    Message: 1e8a3a53556eb21863ddd36e4e6dc6b4b40e1e5308c489130334dca96e77f399 not found in AC
    -->
    	|- ActionDigest: 1e8a3a53556eb21863ddd36e4e6dc6b4b40e1e5308c489130334dca96e77f399/143
    <--

[05 Jun 2022 19:25:58.245] build.bazel.remote.execution.v2.ContentAddressableStorage::FindMissingBlobs - OK (2ms)
    -->
    	|- BlobDigests
    	   - 405ee2251a1467910dd9163e6991d0a6df935920484e9759e6ecc728b980bd7b/27
    	   - e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855/0
    	   - d543d2b229eec2c9cfff4ef2a4795f50d53944628193efb2b27cec40da964ddb/131
    	   - 1e8a3a53556eb21863ddd36e4e6dc6b4b40e1e5308c489130334dca96e77f399/143
    <--
    	|- MissingBlobDigests
    	   - 405ee2251a1467910dd9163e6991d0a6df935920484e9759e6ecc728b980bd7b/27
    	   - d543d2b229eec2c9cfff4ef2a4795f50d53944628193efb2b27cec40da964ddb/131
    	   - 1e8a3a53556eb21863ddd36e4e6dc6b4b40e1e5308c489130334dca96e77f399/143

[05 Jun 2022 19:25:58.253] google.bytestream.ByteStream::Write - OK (20ms)
    -->
    	|- ResourceNames: [uploads/935da4f7-4095-4492-a43b-7bab3b2b147d/blobs/1e8a3a53556eb21863ddd36e4e6dc6b4b40e1e5308c489130334dca96e77f399/143]

[05 Jun 2022 19:25:58.252] google.bytestream.ByteStream::Write - OK (22ms)
    -->
    	|- ResourceNames: [uploads/25a22748-a5e1-4294-a15d-03766431a14a/blobs/405ee2251a1467910dd9163e6991d0a6df935920484e9759e6ecc728b980bd7b/27]

[05 Jun 2022 19:25:58.253] google.bytestream.ByteStream::Write - OK (28ms)
    -->
    	|- ResourceNames: [uploads/26a4e52c-eac7-4839-a0b7-abb5994bbc38/blobs/d543d2b229eec2c9cfff4ef2a4795f50d53944628193efb2b27cec40da964ddb/131]

[05 Jun 2022 19:25:58.283] build.bazel.remote.execution.v2.ActionCache::UpdateActionResult - OK (10ms)
    -->
    	|- ActionDigest: 1e8a3a53556eb21863ddd36e4e6dc6b4b40e1e5308c489130334dca96e77f399/143
    	|- ActionResult:
    		|- OutputFiles:
    		|-   - x bazel-out/k8-fastbuild/bin/dummy
    		|-     |- 405ee2251a1467910dd9163e6991d0a6df935920484e9759e6ecc728b980bd7b/27
    <--

Second build without clean cache

bazel build \
    --stamp \
    --remote_cache=grpc://localhost:9092 \
    --experimental_remote_grpc_log=/tmp/grpc.log \
    //:dummy

bazel-bin/dummy:

BUILD_TIMESTAMP 1654449958

bazel-out/volatile-status.txt:

BUILD_TIMESTAMP 1654450956

Not surprisingly, the file, already in the local cache is not rebuilt even if the volatile stamp file has changed.
No gRPC requests are executed.
This behaviour is conform with the documentation.

Third build with a clean of the local cache

bazel clean --expunge
bazel build \
    --stamp \
    --remote_cache=grpc://localhost:9092 \
    --experimental_remote_grpc_log=/tmp/grpc.log \
    //:dummy

bazel-bin/dummy:

BUILD_TIMESTAMP 1654451141

bazel-out/volatile-status.txt:

BUILD_TIMESTAMP 1654451141

The file, with a different digestKey than the first build, is searched from the remote cache and is not found.
The file is then rebuilt and pushed again in the remote cache. We can see all similary executed gRPC requests than the first build.

The computation of this digestKey should not take into account the updated content of the volatile-status.txt, as is the case when we remove or manually update the volatile stamp file between two builds.

It's the definition of the volatile file:

... In order to avoid rerunning stamped actions all the time though, Bazel pretends that the volatile file never changes. In other words, if the volatile status file is the only file whose contents has changed, Bazel will not invalidate actions that depend on it. If other inputs of the actions have changed, then Bazel reruns that action, and the action will see the updated volatile status, but just the volatile status changing alone will not invalidate the action.

(from the user-manual 5.1.1)

This behaviour should be the same even if we use a remote cache.

Workaround

To avoid this issue, we are going to remove the use of the --stamp option and:

• Find a dirty and not hermetic way to inject our volatile values.
  or
• Use an another tool or another build system to finish the build of our final artifacts.

If we can reopen this issue that would be really great and appreciated ❤️ !