feat(build): scope the dependency cache per package, and make hits actually skip work (2026.7.30.2) - #317
Merged
Merged
Conversation
…ding), justify the recursive F axis
…tually skip work (2026.7.30.2) The global dependency cache existed but its net benefit was zero, for two independent reasons. It never hit across projects. The key was the whole-project fingerprint, whose flags field serializes every package in the graph INCLUDING the root — its name, its version, its [build] flags. So bumping a project's own version invalidated every dependency it had, and two projects with identical dependencies and toolchain shared nothing. Measured on one machine: 26 GB across 1198 fingerprint directories, compat.zlib@1.3.2 stored 162 times, 15 distinct std module identities occupying 1014 directories (16.1 GB where ~0.5 GB was needed). And when it did hit, nothing was saved. Artifacts were copied into the build dir from inside prepare_build while those paths stayed declared as compile edge outputs — and ninja treats an output it has no command line for in .ninja_log as dirty, which a fresh build dir always is. Every "cached" unit was recompiled while the CLI printed "Cached". ninja explain: command line not found in log for obj/zutil.o ninja explain: obj/zutil.o is dirty Keying is now per package (new mcpp.build.cache_key): toolchain identity, language/dialect, profile, package identity, the package's own build config, and — recursively — the keys of its direct dependencies. Nothing about the consumer, which is sound because the root's [build] flags verifiably do not reach dependency translation units. The recursion is not conservatism: GCC embeds a CRC of an imported module's BMI into the importer's BMI, so an importer's artifacts are bound to the exact upstream artifacts they read. Hits now emit stage_file edges instead of compile edges (and no scan/dyndep edges), so ninja has a command-line record for the staged outputs. Artifacts land where a compile edge would have put them, leaving link edges, BMI implicit inputs and runtime deployment unchanged. The status line carries the unit count it saved, because a bare "Cached" was printed for months while every unit was recompiled behind it. Three correctness defects had to land with it, all of them harmless only while the cache was a no-op: - The profile was not an invalidation axis. --dev, --release and --profile dist shared one fingerprint, one build dir and one cache entry, so a release build would have been served -O0 -g objects. - .build_cache keyed fast-path entries by target triple alone, and the fast path only refuses to run for an EXPLICIT profile flag. `mcpp build --release` followed by a bare `mcpp build` reported success in 0.00s and left the release artifacts in place. This one was live regardless of the cache. - Transitively reached path/git dependencies were cached: the exclusion predicate consulted the root manifest's dependency maps, where a transitive package does not appear. Their sources can change without name@version changing. Also: --cache=global|local|off (with --no-cache as a deprecated alias for off, and its inaccurate help text corrected), and mcpp cache grown into something operable — dir / gc with a real LRU / clean --deps|--std|--all|--legacy / list --json / verify, with each entry now describing itself in entry.json so a suspected wrong hit can be audited. The cache root is $MCPP_HOME/build-cache/v1, deliberately not $MCPP_HOME/cache: that name belongs to the index metadata cache, whose reset path removes the whole directory. Design: .agents/docs/2026-07-30-dep-build-cache-scoping-design.md Plan: .agents/docs/2026-07-30-dep-build-cache-implementation-plan.md
A key covers an upstream package by folding in that package's KEY, and a local package's key covers its file list but not its file contents — nothing could, without hashing a tree that may change between the hash and the compile. So a cached downstream entry would keep looking valid after a local upstream's source was edited. No index descriptor can declare a path dependency today, which makes the shape unreachable in practice. Enforced structurally anyway: "unreachable today" is exactly how the transitive path-dep leak got in. Also asserts the invariant that cache-served units keep their compile_commands.json entries — they stay in the plan on purpose so clangd does not lose the dependency's sources.
…ding std entries Three fixes from the first CI round. Module partitions. Replacing a package's compile edges with stage edges also removes the ordering those compile edges carried. A consumer that imports `pkg` has `pkg`'s BMI in its dyndep and nothing else — the partition BMI `pkg:part` was reached only because `pkg`'s own compile edge depended on it. With independent stage edges ninja may start the consumer while the partition is still unstaged: error: failed to find module file for module 'mcpplibs.cmdline:options' macOS CI hit it; Linux won the race, which is why it is now an invariant rather than a scheduling accident. Every staged artifact becomes an ORDER-ONLY prerequisite of every non-staged edge, aggregated through one phony so no edge repeats the list (mcpp#274: long ninja lines are how a 50781-character command line blew past cmd.exe's 8191 limit). Order-only is the right strength — the real content dependencies are still declared where they always were, so a changed BMI still invalidates its consumers; this adds sequencing, not dirtiness. Verified locally on both GCC and Clang against mcpplibs.cmdline, which has a `:options` partition. Ages were computed against the wrong epoch. file_time_type is std::chrono::file_clock, whose epoch is not the Unix epoch, so `cache list` printed "74509d ago". Converted through clock_cast. Test fallout, all of it real: - 22_doctor_cache_publish asserted `cache list` was empty after `mcpp self doctor`, which precompiles a std module. That assertion only held because the old `cache list` walked dep entries and skipped std ones — and hiding them is how 16 GB of duplicated std BMIs went unnoticed. The empty-cache check moved ahead of doctor; the std entry is now asserted to be visible, with a bound on the age column that would have caught the epoch bug. - 40_llvm_bmi_cache used `--no-cache` to force a cold first build, then expected the second build to reuse it. `--no-cache` is now an alias for `--cache=off`, which means neither read NOR write — so it left nothing to reuse. The test's actual intent (fresh MCPP_HOME is already cold) is now what it says, and it additionally asserts zero compile edges and the partition sequencing. The semantic tightening is called out in the CHANGELOG: a mode named `off` that still writes the cache would not be defensible. - 98_reflection_import_std grepped $MCPP_HOME/bmi, the pre-v1 root. It passed locally purely because this machine still holds 26 GB of legacy entries, one of which happened to record -freflection — a false green of exactly the kind the plan warned about. It and two siblings now select the std entry by CONTENT instead of `find | head -1`, which is no longer well-defined: one MCPP_HOME can hold several std identities now, and that is the point.
…poch libc++ does not provide std::chrono::clock_cast, so the previous fix broke every Clang build. Measure the mtime as an offset from now IN THE FILE CLOCK and apply that offset to the system clock instead: no shared epoch, no conversion trait, and "how long ago" is all any caller wants. Verified compiling and behaving under llvm@22.1.8 + libc++, which is the configuration that broke.
Package roots can nest — a workspace member lives under the workspace root — and taking the first matching root filed the member's sources under the outer package, i.e. into the wrong cache key. Index payloads live in the xpkgs store and cannot be shadowed this way, so no cached entry is affected today; resolving by specificity rather than by iteration order is what keeps that true if roots ever move.
gc deliberately never evicts std entries — one is shared by every project on the machine and costs ~30 s to rebuild, so trading it for a little disk is the wrong trade. But its summary reported the remaining PACKAGE bytes as the cache size, so a run that freed everything in scope printed "cache now 0.0 B" with tens of MB of std BMIs sitting right next to it. e2e now pins both halves: the budget is met, std survives, and the figure says what it measured.
A reader hits the design before the plan, and the design's layout section still said $MCPP_HOME/cache/v1 — a path that would nest the build cache inside the index metadata cache, whose reset removes the whole directory.
Same defect shape as the profile one this PR already fixes, in the switch this PR adds. A build.ninja generated under `--cache=global` contains stage_file edges reading the global cache. `.build_cache` did not record the mode, and a bare `mcpp build` (no CLI flag, so nothing bypasses the fast path) replayed that graph even when the manifest said `cache = "local"` — using the cache the project just declared it did not want. Ruling the cache out is `local`'s entire purpose, so this defeated the feature silently. Reproduced with a warm cache and an untouched manifest, which is what makes it reachable: editing mcpp.toml would have invalidated the fast path on mtime, so only the recorded mode can settle it. build --cache=global → 15 stage edges build → 15 stage edges [build] cache = "local" ignored after → 0 stage edges Mode selection moves into resolve_cache_mode, a pure function of (manifest, override, environment), so prepare_build and both fast paths settle it from one rule — the same treatment resolve_profile_name got. prepare_build keeps sole ownership of the diagnostics: an unparseable value must be reported once, by the invocation that actually resolves the build, and must fall through to the next source rather than silently meaning "global". `.build_cache` records the mode alongside the profile; a mismatch is a miss, and entries predating the field have an empty mode that matches nothing. No migration, self-healing on first rebuild.
The fast path replays build.ninja on the premise that this request would generate the same graph. This PR pins two axes (profile, cache mode) into .build_cache, but MACOSX_DEPLOYMENT_TARGET, MCPP_VERIFY_MODGRAPH and MCPP_SCANNER also change the graph and are recorded nowhere — all three predate this PR, and closing them needs a cheap graph identity rather than re-deriving the fingerprint, which is exactly what the fast path exists to avoid. Written down instead of half-fixed.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
依赖的全局缓存此前净收益为零,原因有两个,互相独立。
它从来不跨工程命中。 缓存键是整个工程的 fingerprint,而它的 flags 字段序列化了图里
每一个包,含 root —— root 的包名、版本、
[build]flags 都在里面。所以改自己的版本号就让全部依赖缓存失效,两个工程只要包名不同就一条都不共享:
本机实测:26 GB / 1198 个指纹目录,
compat.zlib@1.3.2存了 162 份,compat.x11@1.8.1373 份共 1.49 GB;std 侧 1014 个目录里只有 15 个真身份,16.10 GB(需要 ~0.5 GB)。
而且命中了也不省事。 命中的产物由
prepare_build直接拷进 build dir,但这些路径在build.ninja里仍然是 compile edge 的输出;ninja 对「输出存在但.ninja_log里没有该输出的命令行记录」一律判脏,而新 build dir 天然没有
.ninja_log。ninja 自己的话:改动
键:全工程指纹 → 每包 Merkle 键(新模块
mcpp.build.cache_key)。七个轴:工具链身份 /语言与方言 / profile / 包身份 / 该包自身的构建配置 / 每个直接依赖的键(递归) /
cache epoch。不含 root 的身份与 flags —— 实测 root 的
[build] cflags/cxxflags不会下发到依赖 TU,这正是跨工程共享成立的机制依据。
F 轴递归而不是「枚举上游 public 接口」,因为 GCC 把被导入模块 BMI 的 CRC 烙进导入者的 BMI
(手工三组对照:只改函数体通过;改签名、
-D改布局都module 'B' CRC mismatch+Bad import dependency)。可枚举清单必然漏项(上游 re-export 的传递模块接口在上游自己的manifest 里看不出来),而失败模态不对称 —— BMI 轴取窄是编译器硬报错,
.o轴取窄是静默错对象。std 缓存键 → std 自己的身份。
stdmod.cppm那 15 字段 metadata 一直就是正确完整的 std身份,
metadata_matches一直在按它校验 —— 唯一错的是目录名用的不是它。命中改发
stage_file边,不发 compile / scan / dyndep 边。 产物落在 compile edge 本来会产生的同一路径,所以链接行、消费者 TU 的 BMI implicit input、运行期部署边全不变,变的只是
「这些文件由谁产生」—— ninja 从此有了可比对的命令行记录。复用 #312 的
mcpp stage原语 +restat = 1。状态行带上省下的 TU 数(Cached compat.zlib v1.3.2 (15 units))。三个正确性缺陷必须同批(它们只在缓存是 no-op 时无害):
--dev/--release/--profile dist共用同一个 fingerprint、同一个 build dir、同一条缓存条目。修好上面那条之后,
--release就会吃到-O0 -g的依赖对象。--release之后裸mcpp build会 0.00s「成功」并交付 release 产物 —— 与缓存无关的独立缺陷:
.build_cache只按 target triple 去重,而 fast path 的准入只挡显式--profile。path/git依赖被写进全局缓存 —— 排除谓词查的是 root manifest 的依赖表,而传递到达的包两个表里都没有;
indexName还回落到默认索引(workspace 成员B以mcpplibs/B@0.1.0落盘)。顺带:
Finished release [optimized]是硬编码的(--dev也这么打);缓存条目现在写entry.json自描述,命中判定从「目录存在 + 文件齐」升级为「键匹配 ∧ 记录的输入逐字段相等 ∧清单齐」——永不因 hash 相等就放行。
新增用户面
--cache global|local|off(--no-cache保留为off的兼容别名,旧 help 文案两处不准已改),MCPP_BUILD_CACHE,[build] cache;mcpp run/mcpp test一并补上。mcpp cache补齐dir/gc --max-size|--older-than(真 LRU,此前prune按目录 mtime排序,那只记录写入时间) /
clean --deps|--std|--all|--legacy/list --json/verify。缓存根
$MCPP_HOME/build-cache/v1/,刻意不放$MCPP_HOME/cache—— 那个名字归索引元数据缓存所有,它的 reset 路径会整目录删除。旧的
$MCPP_HOME/bmi/不读不写不自动删:doctor报体积,
cache clean --legacy回收。效果
mcpp version bump之后--release复用--dev的依赖对象e2e
172_build_cache_cross_project.sh的核心断言:第二个工程(不同包名、不同版本、同一依赖)的
build.ninja里依赖的 compile 边数 = 0。读build.ninja而不是 grep 日志 ——状态行正是此前说谎的那个东西。
测试
test_cache_key18 项、test_build_profile10 项、test_bmi_cache重写 16 项、
test_ninja_backend+6 项)172(跨工程,含「版本 bump 不失效」与「依赖换版本必须重编」两侧)、173(profile 隔离 +--release后裸 build 的回归闸)、174(三模式 + 全部 cache 子命令)19(升级为传递 path 依赖的负例)、49(改读entry.json,并断言无compile 边)、
10/59/100/170(路径)可感知的行为变化(已写 CHANGELOG)
target/<triple>/下从此每个 profile 一个哈希目录。好处是 dev↔release 来回切从「每次全量」变成增量、两个 profile 二进制可并存;代价是磁盘随实际使用的 profile 数增长。
设计与实施计划:
.agents/docs/2026-07-30-dep-build-cache-scoping-design.md、.agents/docs/2026-07-30-dep-build-cache-implementation-plan.md(后者含「实施结果与计划的偏差」一节)。
后续批次(未在本 PR 承诺):
-ffile-prefix-map归一依赖对象路径、零拷贝(直接-fmodule-file=/ 绝对路径.o进链接行,与 #312 §7 合并)。