Use bazel cache when building a python extension with pip

[TendTo]

One of the promises of bazel is that it tends to make recompilations fast thanks to its ingenious caching system. While generally true, I find myself putting this statement to the test.

I have a project where I compile a CPP python extension using pybind11 (more specifically, I use the bazel specific extension). I can build my packages just fine. But, to make everything seamless to someone who does not care about bazel, I also wanted to ensure the package can be installed with a simple pip install .. My efforts lead me to this setup.py + pyproject.toml combination. While a bit convoluted (I'm not sure if there is a better way of doing things) it gets the job done, with an annoying caveat: every time I make a change in the source code and I need to re-install the package, bazel appears to recompile everything from scratch (including the dependencies). If I use the standard rules_py rules from the repository (e.g., bazel run //bindings:python .....), I get way faster results (only the changes are actually recompiled).

My guess is that pip moves all files in a temporary folder and bazel does not recognize it already has cached the project. Since I'm not an expert, I would like to ask: is there a way of connecting the two caches? And are you aware of a better way to structure the installation scripts?

[Gooh456]

Took a look at your actual setup.py/BuildBazelExtension.bazel_build() to see what's actually happening rather than guessing generically. Two things stand out:

1. bazel shutdown after every build is probably the main culprit for the "feels like a full recompile" experience. run() calls self.spawn(["bazel", "shutdown"]) right after the build. That kills the persistent Bazel server, so the next invocation (the next pip install .) has to cold-start a new server and redo the full loading/analysis phase for the whole graph before it can even check the action cache — for a project pulling in pybind11 + its bazel extension + your third_party deps, that alone can look and feel like "everything recompiled," even if the actual compile actions were cache hits. Worth checking directly: run bazel build --config=py //bindings/pylucid:_pylucid twice in a row without the bazel shutdown in between and see if the second run is fast — if it is, you've confirmed the action cache itself is fine and the shutdown call is the actual cost, not a cache-identity problem. If you don't need the shutdown for some resource-cleanup reason specific to your CI, dropping it (or only doing it conditionally, e.g. if os.environ.get("CI")) should get you the fast local-dev loop bazel run already gives you.

2. Bazel's default local cache is keyed to the output_base, which is not necessarily identical to the workspace path — so if anything in your install flow ever does end up invoking bazel against a different working directory (a temp copy, a different checkout, CI vs local), the default cache silently won't be shared even though your source is unchanged. The fix that's actually designed for this — a cache that's portable across different output_bases/workspace identities — is an explicit --disk_cache=<fixed absolute path> (e.g. build --disk_cache=~/.cache/bazel-disk-cache in a .bazelrc, or via BAZEL_DISK_CACHE/pass it through get_bazel_target_args). Unlike the default cache, --disk_cache is a plain content-addressed blob store at a path you choose, so it stays valid and shared no matter what output_base a given invocation happens to pick. I'd try this in combination with #1 rather than instead of it — if #1 alone fixes it, you don't need the extra .bazelrc complexity; if not, --disk_cache should make even a "new" output_base/workspace identity reuse prior action results.

(Didn't run either of these myself — no easy way to verify against your actual C++ build graph — but both are grounded in what your setup.py is actually doing and in how those two Bazel caching mechanisms are documented to behave, so they should be quick to test.)

[TendTo]

Those are great suggestions, but after quickly applying both and running pip install . two times (with no changes to the codebase), the build time was almost the same, around 7 min. Unless I have made some silly mistake (but I do see that the disk_cache folder has been created), I'm afraid there is something else we are missing.

[Gooh456]

Since both #1 and #2 landed and made zero difference (same ~7min twice in a row, disk_cache dir non-empty), that rules out "cache unreachable" — the actions themselves aren't cache-hitting, which means the action keys differ between runs, not that the cache can't be found.

Most likely cause given your setup.py: pip install . uses build isolation by default, which builds inside a fresh, randomly-named temp virtualenv on every invocation. If anything in the //bindings/pylucid:pylucid_files graph — hermetic/autodetected Python toolchain, interpreter path, headers — resolves off that venv's sys.executable, every action touching it gets a different absolute path baked into its command line each run. Bazel's disk cache keys on command line + input digests, so a changing absolute path anywhere in there is a guaranteed miss on every action that references it, independent of source changes.

To confirm: run bazel build --disk_cache=~/.cache/bazel-disk-cache --config=py --python_version=<ver> //bindings/pylucid:pylucid_files --explain=/tmp/explain.log --verbose_explanations twice directly (no pip in the loop, so the interpreter path is fixed). If the second run is fast, that confirms it — try pip install . --no-build-isolation next, which reuses your existing interpreter/venv path instead of a fresh one each time. If it's still slow even without pip, /tmp/explain.log will name the exact input/flag that changed per action.