Insufficient remote Merkle tree size causes slow builds

[lukaciko]

Description of the bug:

After updating our version of Bazel we saw a significant regression in some of our builds. We saw that a couple of actions towards the end of the build that have a lot of inputs took significantly longer. When looking at trace report we saw that the CPU usage is low when executing those actions while the memory usage of the main Bazel process is constantly going up and down:

Using Git bisect, we found #18015 to be the change that lead to the biggest regression.

We figured out that the builds which have regressed are using --experimental_remote_merkle_tree_cache and we could fix it by increasing --experimental_remote_merkle_tree_cache_size. With an insufficient size, Bazel will keep allocating and deallocating the Merklee trees. Presumably we saw a regression after that change because it keeps Merkle trees around for longer.

While we can work around the issue by increasing the size, a warning or error when this starts happening would be appreciated.

What's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

Haven't tried this myself, but I'm assuming this can be replicated on a decently size Java project with enabling --experimental_remote_merkle_tree_cache and setting --experimental_remote_merkle_tree_cache_size to a low value.

Which operating system are you running Bazel on?

MacOS 13.4 & Ubuntu Focal Fossa

What is the output of bazel info release?

/

If bazel info release returns development version or (@non-git), tell us how you built Bazel.

We built on top of commit 286306e from the 6.x branch with some additional patches.

What's the output of git remote get-url origin; git rev-parse master; git rev-parse HEAD ?

git@github.com:bazelbuild/bazel.git
a7b96f45df00b4024de4e70b90989956904ca4fb
286306e8358542ce272f7442075bf157a2a62ec7

Is this a regression? If yes, please try to identify the Bazel commit where the bug was introduced.

1641fa8

Have you found anything relevant by searching the web?

No response

Any other information, logs, or outputs that you want to share?

No response

[coeuvre]

cc @tjgq

[tjgq]

I didn't have access to a good repro at the time this issue was filed, but in light of recent developments in #20862 (thanks again for the repro, @DavidANeil) I now have a theory as to why this occurs, which is explained in #20862 (comment).

The wider discussion in #20862 is also making us consider scrapping the Merkle tree cache optimization entirely (or, more likely, scale it back to cover only the cases where we're able to consistently see a benefit - large tree artifacts and runfile trees). We don't have a lot of evidence that it provides a performance benefit for most builds, and quite a bit of evidence that it makes things worse (sometimes dramatically, as #20862 demonstrates) in some cases, which doesn't look like a good tradeoff.

If anyone in the community could point us to a reproducible use case (ideally a real one, but synthetic would work as long as it's representative of a real one) where enabling the Merkle tree cache provides a significant benefit, that would be very helpful to guide future work in this area.