Allow sharing caches across actions when sandboxing is in use

[ktf]

Description of the problem / feature request:

Extremely low performance of bazel when building a large project on macOS when compared to running CMake + make.

Bugs: what's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

Build a large bazel project on macOS. I see that a lot of fsevents for files in /private/var/tmp/_bazel_user triggered and fseventsd seems to crawl at 100% - 200% CPU usage after them, slowing down the builds. I do have /private/var/tmp masked out in my spotlight configuration, but it does not seem to be enough.

What operating system are you running Bazel on?

Darwin grisu.dyndns.cern.ch 18.2.0 Darwin Kernel Version 18.2.0: Mon Nov 12 20:24:46 PST 2018; root:xnu-4903.231.4~2/RELEASE_X86_64 x86_64

What's the output of bazel info release?

release 0.22.0

[irengrig]

Hello @ktf, thank you for your feedback.
Do you some particular project in mind that we can experiment with?

[ktf]

What I have is:

https://github.com/ktf/AliceO2/tree/bazel-support

where you can for example do bazel build @root//:cling. Given this is a subset of tensorflow (i.e. the part which builds llvm), plus some extra stuff to build parts of clang, I am surprised this is not a common issue. Notice I build on my local laptop (2015 MBP) with no external cache.

[irengrig]

Thank you for the example project!
Assigning to Apple team.

[keith]

It might be worth trying to repro with this commit 0877340

[jmmv]

Indeed; that commit may help and I'd be very interested to know if it does.

But I have also noticed fseventsd consuming lots of CPU (and, incidentally, this service runs at default priority -- so that commit actually makes Bazel run at a lower priority which is probably good). Haven't had a chance to dig into this. Maybe setting --nowatchfs would prevent the fseventsd activity, though then Bazel will be slower to detect file changes of course... (just thinking aloud). It just feels stupid that we have to waste so much CPU just to watch for file changes...

[ktf]

Thanks. I will try that when I have a bit of time. Notice also that the fsevents seem to be relative to stuff in /var/tmp/_bazel_user/, not in ~/src/myproject. Maybe I simply do not understand the bazel internals, but is that what you'd expect from the file watching process?

[jmmv]

Yes. That's where the build artifacts are -- they are not stored in your source tree -- and Bazel has to watch them for out-of-band changes.