In the issue description you mention unexpected "downloads", but the reproducer description doesn't mention any downloads. The "actual result" for the reproducer actually looks correct to me: Bazel has to rerun the actions to refresh the metadata.

If you really do see unexpected downloads of file contents as in the other issue I fixed, could you describe that in more detail for your reproducing example?

Hi @fmeum Yes, I was in a hurry with the description of problem, I meant invalidating and rebuilding the actions, not downloading itself. I will fix the description.
Initially the problem was observed with Bazel 7.4.0, and we have seen the real downloads of files from remote cache. When we have found your fix, we tried the same scenario with Bazel 7.6.1. And we still saw the rebuilds for the actions, even though nothing has been changed in environment. This produces several issues.
Let's consider a case when software developer works on a code locally and incrementally builds the results with Bazel. The whole solution may be large (let's say, 50 000 actions to build), but only a small part of code is subject to change. Most of actions are taken from remote cache. In this case:

• when TTL expires, Bazel rechecks all that actions, although the source code for those actions has not been changed. This increases the build time.
• checking the AC with remote cache produces the unnecessary load on a cache server.
• in case when AC is missing in remote cache, the action will be rebuilt locally although it might not be needed actually.
• in case when locally rebuilt action produces non-deterministic output (such as .pdb, .pch files), this invalidates the whole tree of actions which depend on it. This might significantly increase the build time.
  The described behavior may look as unexpected for users, because not all people know about the TTL feature, which is turned on by default with 3h setting.
  The workaround for issues above is disabling the TTL functionality with --experimental_remote_cache_ttl option.
  We would expect that TTL feature was implemented without invalidating the actions just because they are taken from remote cache. If it is not technically feasible, then we could consider disabling the feature by default, or removing the feature at all.

cc @coeuvre

The described behavior may look as unexpected for users, because not all people know about the TTL feature, which is turned on by default with 3h setting.

While the default TTL is 3h, it doesn't mean the cache entry will be expired after 3h. Bazel will extend the TTL in the background each time the build is running. In theory, it means if the next incremental build happens within 3h, the TTL of the cache entry should be extended for another 3h.

We would expect that TTL feature was implemented without invalidating the actions just because they are taken from remote cache.

This is not true. REAPI doesn't have a way for the server to communicate TTL with Bazel. So we have to implement it the current way.

If it is not technically feasible, then we could consider disabling the feature by default, or removing the feature at all.

One way to improve this is to not invalidate the cache entry when the TTL expires. We can rely on build rewinding to recover from cache eviction, because technically, the cost of build rewinding is roughly the same as invalidating cache entries. We still want to keep TTL because we can extend the lease in the background to help remote server understand that these blobs are still in use. WDTY @fmeum?

Short-term relying on build rewinding seems reasonable to me. It avoids eager invalidation but should be equally capable at resolving any issues that arise due to eviction.

Mid- to long-term action rewinding should avoid this wholesale invalidation altogether and the default TTL value would become less relevant, which is good as many users won't be aware of it.