feat(pixi-build-cmake): track exact build inputs from ninja instead of broad globs#5972
Merged
baszalmstra merged 3 commits intoprefix-dev:mainfrom May 5, 2026
Merged
Conversation
61e3d30 to
4980ebb
Compare
…file list
Query ninja's view of the build graph after the build runs to recover the
exact set of input files instead of returning broad globs. Two ninja
invocations cover everything:
* 'ninja -t deps' — sources and the transitive header chain (loaded
from the depfile DB the compiler emits).
* 'ninja -t targets all' — surfaces CMakeLists.txt and *.cmake files
that CMake registers as inputs to its regen rule (they appear as
'<path>: phony' entries).
Both ninja calls run in parallel via std::thread::scope so neither can
block on a full pipe buffer waiting for the other. Source root comes from
CMakeCache.txt's CMAKE_HOME_DIRECTORY entry; paths outside it are dropped
(those are tracked by pixi's environment hash).
If anything fails (build dir missing, ninja exits non-zero, parse error)
we fall back to the original broad globs. The fallback also fixes a typo:
'**/CMakeFiles.txt' was meant to be '**/CMakeLists.txt'.
4980ebb to
3854460
Compare
Bundle (source_dir, build_dir) into a `Layout` struct that owns the relativize / relativize-glob rules, and wrap the ninja invocation in a small `Ninja` struct so the parallel-query plumbing isn't repeated. Each parser now takes `&str` (decoded once via `str::lines()`, which handles both `\n` and `\r\n`) instead of `&[u8]` with manual `\r` trimming.
hunger
approved these changes
May 4, 2026
Contributor
hunger
left a comment
There was a problem hiding this comment.
I was thinking about doing something like this just this WE :-)
I had opted for the file path thing though: That works independent of build backend configured, but indeed has the disadvantage of not listing headers (at least not reliably, many project list them in their sources though to make them show up in IDEs).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
The
pixi-build-cmakebackend currently reports a coarse set of glob patterns as the build's inputs:Pixi expands those globs against the source tree to decide whether the build cache is still valid. There are two problems with this:
node_modules-shaped subtrees, the walk dominates the time spent deciding whether anything actually changed. This is the main reason for the change..cppor.hsomewhere in the tree (a vendored dep, a sibling experiment, a file the user copied in for reference) is treated as a build input. Editing it triggers a rebuild that does not need to happen, and lockfile churn can include files CMake never looked at.CMake and Ninja already know exactly which files the build consumed: declared sources, transitively included headers, the
CMakeLists.txtchain, and anything aninclude()pulled in viaCMAKE_MODULE_PATH. We have that information for free after every successful build.This PR has the backend ask Ninja directly which files participated in the build, after the build runs. The exact set is forwarded to pixi's caching layer. Cache validation now reads a small enumerated list of files instead of walking the project tree.
How the input set is computed
Three Ninja sub-commands cover the build graph:
ninja -t inputs all.cc/.cppfiles that targets list as their sources). These do not show up in-t deps.ninja -t depsninja -t targets allCMakeLists.txtand the*.cmakefiles that CMake registers on its regen rule (they appear as<path>: phony).Anything outside the project root is dropped (system headers, conda environment files, files in unrelated source trees). Anything under
<project>/.pixi/is also dropped, since pixi's own cache and conda envs are tracked through the environment hash, not the input set.file(GLOB CONFIGURE_DEPENDS …)If a
CMakeLists.txtcollects sources viafile(GLOB … CONFIGURE_DEPENDS …), the backend recovers the original glob patterns from CMake'sVerifyGlobs.cmakeand forwards them to pixi. Adding a new file matching one of those patterns invalidates the build cache and triggers a reconfigure, the same way it does for an in-treecmake --buildcycle.Plain
file(GLOB)withoutCONFIGURE_DEPENDSfollows CMake's own semantics: only the files that matched at configure time are tracked. Adding a new file does not invalidate the cache. This mirrors what plainfile(GLOB)does inside CMake itself, the documented footgun where you have to re-run CMake by hand. Users who want auto-detection should useCONFIGURE_DEPENDS, and they get correct invalidation in exchange.Why not the CMake File API?
Both the File API (
cmakeFiles-v1,codemodel-v2) and Ninja-based extraction give the same set of files. The File API is mildly cleaner to parse, but it would need a query-marker injection step before configure, and it does not expose discovered headers (-t depsis still required). Crucially, neither approach exposes the original glob patterns; for those we have to readVerifyGlobs.cmakeeither way. So this PR sticks with Ninja, which does not require any pre-configure setup.Fallback
If any step fails (build dir wiped, Ninja exits non-zero, CMakeCache.txt unparseable), the backend logs a warning and falls back to a coarse glob set.
How Has This Been Tested?
crates/pixi_build_cmake/src/inputs.rscover the parsers, path canonicalization (relative-t depspaths, Windows backslashes, mixed slashes), the.pixi/filter, the absolute-path filter for-t targets all(drops Ninja's synthetic phony aliases), andVerifyGlobs.cmakeextraction includingGLOB_RECURSEtranslation to gitignore-style**patterns.examples/pixi-build/cpp-sdlwithPIXI_BUILD_BACKEND_OVERRIDE. The cache went from broad globs to two exact entries:["src/main.cc", "CMakeLists.txt"].CMakeLists.txtviaadd_subdirectory, aninclude()'d helper fromCMAKE_MODULE_PATH). All 8 expected files were captured, nothing else.file(GLOB)andfile(GLOB CONFIGURE_DEPENDS)together: cold build captured the recoveredsrc_configure_deps/*.ccpattern alongside the resolved files. After adding a new file matching that pattern, the nextpixi installcorrectly invalidated the cache and rebuilt; a third invocation reported "up-to-date" again.AI Disclosure
Tools: Claude Code
Checklist: