fix(cache): a package's object layout must not depend on the consumer (#344) - #345
Conversation
…#344) A dependency's objects live in the global build cache under a key that deliberately excludes the consuming project — that exclusion is what makes an entry shareable across projects. Their LAYOUT inside the entry did not exclude it: the address was the consumer's build-dir path with `obj/` stripped, and build-dir object paths are decided by #233's basename disambiguation, whose census spans the WHOLE build directory. So `compat.zlib`'s compress.o was obj/compress.o alone obj/compat_zlib/zlib-1.3.2/compress.o beside compat.bzip2 under one key. Whichever project ran second asked the entry for a path the first had never written, and ninja rejected the graph before running any command: ninja: error: '<cache>/…/obj/compress.o', needed by 'obj/compress.o', missing and no known rule to make it one line after the CLI printed `Cached compat.zlib v1.3.2 (15 units)`. 32 of 47 mcpp-index workspace members across three platforms. #233 (compile edges collided), #240 (link inputs didn't follow the rename) and this are three products of one machine: a layout decided by a global census. The fix is not a fourth place to keep in sync — it is to take dependencies out of the census. * plan.cppm now emits TWO addresses from ONE derivation: `object` (build dir) and `packageObjectRel` (inside a cache entry, a pure function of the owning package). Dependency objects go to `obj/<pkg-slug>/<mirrored relDir>/<name>.o` unconditionally — cross-package collisions are structurally impossible, so dependencies need no census at all. The root project, never cached, keeps its flat `obj/<name>.o` and now censuses only its own sources. * prepare.cppm's `object_cache_path` — the second derivation — is deleted, along with its `filename()` fallback, which silently mapped distinct sources onto one entry address. * `is_cached(key)` validated the entry's OWN file list while the caller went on to read addresses it computed itself; the two answers were never compared. It is now `probe_cached(key, requested)`. Any divergence is a MISS, never a build failure, and is reported — a systematic recurrence would otherwise show up only as "the cache never hits", with no signal at all. * Cacheability is judged by where the payload is on disk, not by the `sourceKind` label. Multi-version mangling re-anchors a consumer package at `target/.mangled/…` and rewrites its sources while keeping that label; it stays out of the cache today only because axis F happens to differ. Containment is judged LEXICALLY (`path_is_under_any`): stores built out of symlinks are ordinary, and canonicalizing drops every such package out of the cache silently. * All-or-nothing per package: one unaddressable unit takes the whole package out, rather than leaving it half staged. * kCacheEpoch 1 → 2 (the artifact layout changed) and `cache verify` now reports obj addresses that escape their entry. Tests: unit ObjectAddress pins "a dependency's addresses are immune to the rest of the graph"; e2e 184 runs both orderings and asserts ZERO compile edges for the reused dependency — asserting only "it builds" would be satisfied by a build that quietly degraded every hit to a miss. e2e 123's assertion is generalized from "the consumer's main must be renamed" (the shape of #240's first fix) to "every link input is produced by some edge" (the property).
…Windows Discovered by running the real mcpp-index workspace against the #344 branch: `opencv-module`, `opencv-module-dnn` and `opencv-module-unifont` all died with ninja: fatal: posix_spawn: Argument list too long naming no edge, no file and no cause. #344 lengthened dependency object paths (they now carry a per-package directory) and that was enough to cross a ceiling nothing was watching. The rule emitter said rsp was needed only where commands spawn through CreateProcess, and that "POSIX driver-style keeps the inline form: ARG_MAX is ample". Both halves are wrong. ninja runs `sh -c "<whole command>"` on POSIX, so the command is a SINGLE argv entry and the limit is MAX_ARG_STRLEN — 32 pages, 128 KiB — not the 2 MiB ARG_MAX anyone would think to check. And it was never ample: measured on opencv-module, the inline link line was already 56 840 bytes before this branch, 43% of the ceiling. With #344's paths it reached 161 687. So: rsp always, every platform, every dialect. clang/gcc drivers, link.exe, GNU ar and llvm-ar all accept @rspfile, so this is one rule shape instead of two, and the response file next to the output reads better than a 160 KB command line anyway. A build system may not have a maximum project size that it discovers by crashing, and "how long is this command" must not be something anyone carries in their head when choosing an object path. The unit test asserted the falsified premise for POSIX ("must NOT use rspfile"); it now asserts the invariant on every platform, plus that `$in` appears exactly once per rule — as rspfile_content, never inlined into the command. Note this was invisible to all 18 CI jobs: none of them builds an opencv-class package. CI has no large-link coverage at all.
追加提交
|
| 内联链接行 | 占上限 | |
|---|---|---|
| 修复前 | 56 840 B | 43% |
| 带 #344 | 161 687 B | 123% |
上限不是 ARG_MAX(2 MiB)。ninja 在 POSIX 上是 sh -c "<整条命令>",整条命令是
一个 argv 项,撞的是 MAX_ARG_STRLEN = 32 页 = 128 KiB。
规则发射器原本写着「rsp 只在 CreateProcess 的地方需要;POSIX 内联即可,ARG_MAX 很宽裕」。
两半都错:撞的不是那个限制,而且它从来就不宽裕 —— 本分支之前就已经用掉 43%。
#344 让依赖对象路径多一层包目录,把剩下的 74 KB 余量一次用完。
修法
link/archive/shared 全平台、全方言一律走 response file。clang/gcc driver、link.exe、
GNU ar、llvm-ar 全都认 @rspfile,所以这是把两种规则形态收敛成一种,不是加分支。
构建系统不能有一个「靠崩溃才被发现的项目规模上限」,
「这条命令有多长」也不应该是选对象路径时需要有人记在脑子里的事。
原单测把被证伪的前提钉死了(POSIX 必须不用 rspfile),已改为断言全平台不变量 +
$in 每条规则恰好出现一次(只作 rspfile_content,绝不内联进命令)。
验证
| 面 | 结果 |
|---|---|
| unit | 54 passed / 0 failed |
| e2e | 175 passed / 0 failed / 8 skipped |
| mcpp-index workspace(本地 34 成员) | 34/34 ok,43 passed / 0 failed(此前 31/34) |
#344 本身的生态复验:pre-fix 二进制在真实 mcpp-index 上复现出与 issue 完全一致的报错文本;
修复后 A/B 双向都过,且 libpng 的 build.ninja 里 zlib 有 15 条 stage 边 / 0 条 compile 边 ——
是真命中,不是「全都降级成 miss」的假绿。
一条值得单独跟进的缺口(不在本 PR 扩大范围)
这个回归对 18 个 CI job 全部不可见 —— 没有任何一个 job 构建 opencv 这一量级的包,
CI 完全没有大链接覆盖。建议单开 issue。
Fixes #344. 版本
2026.8.3.4。设计文档:
.agents/docs/2026-08-03-issue344-cache-object-address-design.md根因不止一层
issue 归因正确,但只到第一层。代码核实后是四层,其中两层是同一根因的其它实例:
.o地址直接复用消费方 build dir 相对路径plan.cppm的消歧普查扫topoOrder= 全图所有包,而 cache 条目是 per-packageis_cached校验条目自述的文件表,消费方按自己算的地址去取 —— 两处推导从不比对sourceKind)而非出处.mangled重锚包被判可缓存(静默)两个关键机制事实(都在实现里被利用了):
stage.cppm:185那条staging source does not exist运行期守卫永远到不了。issue 建议的「stage 时回落到 compile 边」是死代码,修复只能落在 plan 阶段。prepare.cppm在调is_cached前已经算好本次真正要取的arts,然后扔了。改动
R1 拆掉生成器。 依赖包的对象无条件落
obj/<pkg-slug>/<mirrored relDir>/<name>.o—— 跨包撞名结构性不可能,依赖包因此完全退出普查。根工程(永不入 cache)保持扁平obj/<name>.o,普查范围收窄到自己的源。不保留「无冲突则平铺」的条件化:那个条件判断就是生成这一族 bug 的机器,省下的只是路径长度。R2 单一真源。
CompileUnit::packageObjectRel(条目内地址)与object(build 路径)由同一处object_for产出。删除prepare.cppm的object_cache_path及其filename()兜底 —— 那个兜底会把两个不同源静默映射到同一条目地址。DepArtifacts::objFiles拆成{cacheRel, buildRel},entry.json 只记 cacheRel。R3 命中判据校验「本次要取的那批」。
is_cached(key)→probe_cached(key, requested)。任何分歧一律 miss,永不构建失败,并打一行 warning —— 系统性分歧否则会表现为「cache 永远不命中」而无任何信号,正是 v2026.7.30.2 之前假Cached那个失败模态的镜像。cache verify同步报告逃出条目的地址。R4
kCacheEpoch1 → 2。R5 可缓存性改判磁盘出处 + 全有全无。两处实施中被现实证伪的假设
① store root 不是一个目录,是一组。 自定义 git 索引把 payload 装进项目本地数据根(
config::project_xlings_data_roots)—— e2e 172 正是这个形态。单一 storeRoot 判据会让它「cold build did not populate a cache entry」。② 包含判定必须按字面路径,不能用
std::filesystem::relative。 它对两侧跑weakly_canonical,会解析符号链接;而「store 条目是指向另一个 store 的符号链接」是常态(_inherit_toolchain.sh与 CI 缓存都这么干)。canonical 化后该包静默退出缓存 —— 被 e2e40_llvm_bmi_cache抓到(第二次构建打印Compiling而非Cached)。收敛为path_is_under_any:字面为主 + canonical 重试兜住 Windows 路径拼写差异,拼写不同最坏退化成慢,不会退化成错。测试(先红后绿,两条都验过)
tests/unit/test_object_address.cpp—— 钉住 L1:同一个依赖包在「图里有无一个刻意撞名的包」两种 plan 下,地址必须逐字节相同。临时把object_for换回全局普查后精确变红(不能靠 checkout main 得到:packageObjectRel在 main 上不存在,那是编译失败不是断言失败)。tests/e2e/184_build_cache_object_layout.sh—— A/B 双向 + 断言「零 compile 边」。用真正的 pre-fix 二进制跑,复现出与 issue 完全一致的报错文本。只断言「构建成功」会被 R3 的 miss 悄悄满足,变假绿。tests/e2e/123_same_named_main_across_dep.sh的断言从「消费方 main 必须被改名」([bug] #233 消歧后链接输入未跟随改名:依赖包与消费者同名源(如双方都有 main.cpp)即 'obj/main.o missing and no known rule' #240 首个修复的形状)泛化为「链接输入必须由某条边产出」(性质)。新架构下依赖住在自己子树里,消费方 main 正确地保持扁平 —— 旧断言是过度指定。本地验证
发布后需在 mcpp-index 全量 workspace 三 leg 复验:修复前 linux 13 / macos 11 / windows 8 成员失败,须回到 47/47,且核对第二个成员的
build.ninja确有 stage 边(否则可能是「全都变 miss」的假绿)。