fix(cache): a package's object layout must not depend on the consumer (#344)

CHANGELOG.md

@@ -3,6 +3,35 @@
 > 本文件追踪 `mcpp-community/mcpp` 公开仓的版本演进。
 > 格式参考 [Keep a Changelog](https://keepachangelog.com/zh-CN/1.1.0/)。
 
+## [2026.8.3.4] — 2026-08-03
+
+### 修复
+
+- **全局 build cache:同一个 key 下第二个消费者必挂 `missing and no known rule to make it`(#344)。** 一个依赖的 `.o` 存在 cache 条目里的**路径**,此前取自消费方 build dir 的相对路径。而 build dir 里的对象布局由 #233 的 basename 消歧决定,消歧的判据是一次**跨越整个 build dir**的普查 —— 也就是说,它取决于消费方还拉了哪些别的包:
+
+  - 同时拉了 `compat.zlib` 与 `compat.bzip2`(两者上游各有一个 `compress.c`)→ `obj/compat_zlib/zlib-1.3.2/compress.o`
+  - 只拉了 `compat.zlib` → `obj/compress.o`
+
+  cache key **刻意不含消费方**(这正是跨工程共享成立的前提),于是两种布局落进同一个条目,后跑的那个工程按自己的布局去取,必然缺一个文件。失败发生在 ninja 的 **graph 加载**阶段 —— 一条命令都还没跑,上一行却刚打印过 `Cached … (15 units)`。mcpp-index 全量 workspace 在三个平台上共 32 个成员因此失败。
+
+  条目内的对象地址现在是**包自身的纯函数**:镜像源文件相对它**自己**包根的路径,不含任何消费方信息。构建目录里,依赖包的对象一律落在 `obj/<pkg-slug>/…` 下 —— 跨包撞名结构性地不可能发生,依赖包因此完全退出消歧普查。根工程(永不入 cache)保持沿用至今的扁平 `obj/<name>.o`。
+
+  #233(编译边撞名)、#240(链接输入未跟改名)与本条是同一台机器的三个产物:**布局由一次全局普查决定**。所以修的不是再补一处同步,而是把依赖包从普查里彻底拿掉。
+
+- **链接/归档命令一律走 response file,不再有「项目大到一定程度就崩」的隐形上限。** 此前只有 Windows(CreateProcess 32 KiB)与 msvc 方言用 rspfile,POSIX 走内联 `$in`,理由写的是「ARG_MAX 很宽裕」。两半都错:ninja 在 POSIX 上是 `sh -c "<整条命令>"`,整条命令是**一个 argv 项**,撞的是 `MAX_ARG_STRLEN`(32 页 = 128 KiB)而不是 2 MiB 的 `ARG_MAX`;而且它从来就不宽裕 —— 实测 mcpp-index 的 `opencv-module`,内联链接行**本来就已经 56840 字节**,占那条无人看守的上限的 43%。
+
+  上面 #344 让依赖对象路径变长(多一层包目录),同一条边到了 161687 字节,于是 ninja 直接 `ninja: fatal: posix_spawn: Argument list too long` —— **不报是哪条边、哪个文件、什么原因**。构建系统不能有一个「靠崩溃才被发现的项目规模上限」,「这条命令有多长」也不应该是选对象路径时需要有人记在脑子里的事。clang/gcc driver、link.exe、GNU ar、llvm-ar 全都认 `@rspfile`,现在全平台一个规则形态。
+
+### 改进
+
+- **cache 条目与本次构建的布局分歧,现在降级为 miss 并明确报告,而不是让 ninja 崩在图加载阶段。** `is_cached` 此前校验的是条目**自述的**文件表,而消费方随后按**自己算的**地址去取 —— 两处独立推导,从不比对。命中判据现在校验「本次实际要读的那批产物」,任何不匹配都只是一次重编。同时新增一行 warning:一个系统性的分歧否则会表现为「cache 永远不命中」而毫无信号,这正是 v2026.7.30.2 之前那个假 `Cached` 骗了三个月的失败模态。
+
+- **可缓存性改判磁盘出处,不再只认标签。** 规则一直写着「无法证明来自不可变的 xpkgs store 就不准入」,但代码实现的是更弱的代理(`sourceKind == "version"`)。多版本共存(mangling)会把消费方包的根重锚到 `<project>/target/.mangled/…` 并**改写其源码**,而标签仍是 `"version"` —— 它今天不出错只靠轴 F 侥幸。现在按包根的实际位置判定。同一批还加了「全有全无」:任何一个单元拿不到与机器无关的条目地址,整个包退出缓存,不留下半 staged 的包(那会表现为三条边之后的 BMI CRC mismatch)。
+
+- **`mcpp cache verify` 现在会报告逃出条目的对象地址。** 让「条目地址必须是包内相对路径」这条不变量可以离线审计,而不是只能通过复现一次双工程构建才看得见。
+
+- **`kCacheEpoch` 1 → 2。** 产物布局变了,旧条目描述的是本版本不会去要的布局。它们本来就会被判为 miss,但让两套布局共用一个目录会让 `cache gc` 的体积统计和 `cache verify` 的输出失去意义。用户侧表现为一次全量重建,无需任何手工步骤。
+
 ## [2026.8.3.3] — 2026-08-03
 
 ### 修复

mcpp.toml

@@ -1,6 +1,6 @@
 [package]
 name        = "mcpp"
-version     = "2026.8.3.3"
+version     = "2026.8.3.4"
 description = "Modern C++ build & package management tool"
 license     = "Apache-2.0"
 authors     = ["mcpp-community"]

src/bmi_cache.cppm

@@ -5,7 +5,16 @@
 //   (the cache root is $MCPP_HOME/build-cache/v1 — see mcpp.home::cache_root)
 //     entry.json                       sentinel + self-description + file list
 //     bmi/<module>.{gcm,pcm}
-//     obj/<relative>.o
+//     obj/<package-internal path>.o
+//
+// The obj address is PACKAGE-INTERNAL: it mirrors the source's path relative to
+// its own package root and contains nothing about the consuming project
+// (mcpp#344). It used to be the consumer's build-dir path with `obj/` stripped,
+// which made the layout depend on which OTHER packages the consumer happened to
+// pull in — #233's basename disambiguation is triggered by a census over the
+// whole build dir — while the key deliberately excludes the consumer. Two
+// consumers then wrote and read incompatible layouts under one key and the
+// second one died in ninja's graph phase.
 //
 // <key16> comes from mcpp.build.cache_key: a per-package Merkle key over the
 // toolchain, the language/dialect settings, the resolved profile, the package's
@@ -75,16 +84,63 @@ struct CacheKey {
     std::filesystem::path objDir()    const { return dir() / "obj"; }
 };
 
-// File names (basenames for BMIs, output-relative paths for objects) belonging
-// to one package's cache entry.
+// One cached object file. The two addresses are deliberately separate
+// (mcpp#344):
+//
+//   cacheRel — where it lives INSIDE the entry (`<entry>/obj/<cacheRel>`).
+//              Must be a pure function of the package, because the key
+//              deliberately excludes the consumer. plan.cppm derives it.
+//   buildRel — where THIS build produces it (`<buildDir>/<buildRel>`).
+//              Consumer-side and therefore not recordable: two consumers of
+//              one entry may legitimately place the same object at different
+//              build-dir paths.
+//
+// Collapsing the two — recording the consumer's path as the entry's address —
+// is exactly what made the second consumer of an entry fail with ninja's
+// "missing and no known rule to make it". Only `cacheRel` ever reaches
+// entry.json; `buildRel` is populate-time input and is empty on read-back.
+struct ObjArtifact {
+    std::string           cacheRel;
+    std::filesystem::path buildRel;
+};
+
+// The artifacts belonging to one package's cache entry: BMI basenames plus the
+// objects above.
 struct DepArtifacts {
-    std::vector<std::string> bmiFiles;
-    std::vector<std::string> objFiles;
+    std::vector<std::string>  bmiFiles;
+    std::vector<ObjArtifact>  objFiles;
 };
 
-// True when entry.json exists, its schema matches, its recorded inputs equal
-// `key.inputs` field for field, and every listed file is present on disk.
-bool is_cached(const CacheKey& key);
+// Why an entry could not serve this build.
+struct CacheProbe {
+    bool ok = false;
+    // Non-empty ONLY when the entry itself validated (schema, key, inputs) but
+    // does not carry the artifacts THIS build asked for. After mcpp#344 that
+    // shape should be unreachable, which is precisely why it must be reported
+    // rather than silently folded into "miss": a systematic recurrence would
+    // otherwise present as "the cache simply never hits", with no signal at
+    // all — the same failure mode as the fake `Cached` that went unnoticed for
+    // three months.
+    std::vector<std::string> layoutMismatch;
+};
+
+// Validate an entry AGAINST WHAT THIS BUILD WILL ACTUALLY READ.
+//
+// A hit requires all of: entry.json exists, its schema matches, its recorded
+// key matches, its recorded inputs equal `key.inputs` field for field, and
+// every artifact in `requested` is both listed by the entry and present on
+// disk. Checking only the entry's OWN file list — which is what this used to
+// do — validates a different question than the one the caller goes on to ask,
+// and the two answers diverged the moment the object layout stopped being a
+// function of the package alone.
+//
+// Anything short of a full match is a MISS. This function must never be the
+// reason a build fails: an unusable entry costs a recompile, and the staging
+// edges that would read it are never emitted.
+CacheProbe probe_cached(const CacheKey& key, const DepArtifacts& requested);
+
+// probe_cached(...).ok
+bool is_cached(const CacheKey& key, const DepArtifacts& requested);
 
 // The artifact list of a validated entry. Does NOT copy anything: the ninja
 // backend stages cached files through its own `stage_file` edges, so that a
@@ -137,12 +193,15 @@ std::optional<nlohmann::json> read_entry(const std::filesystem::path& p) {
     return j;
 }
 
+// Read-back fills `cacheRel` only: `buildRel` is consumer-side and is not — and
+// must not be — recorded in the entry.
 DepArtifacts artifacts_from(const nlohmann::json& j) {
     DepArtifacts a;
     if (auto it = j.find("bmi"); it != j.end() && it->is_array())
         for (auto& v : *it) if (v.is_string()) a.bmiFiles.push_back(v.get<std::string>());
     if (auto it = j.find("obj"); it != j.end() && it->is_array())
-        for (auto& v : *it) if (v.is_string()) a.objFiles.push_back(v.get<std::string>());
+        for (auto& v : *it)
+            if (v.is_string()) a.objFiles.push_back({v.get<std::string>(), {}});
     return a;
 }
 
@@ -199,21 +258,39 @@ std::filesystem::path cached_obj_path(const CacheKey& key, std::string_view rel)
     return key.objDir() / std::filesystem::path(std::string(rel));
 }
 
-bool is_cached(const CacheKey& key) {
+CacheProbe probe_cached(const CacheKey& key, const DepArtifacts& requested) {
+    CacheProbe probe;
     auto j = read_entry(key.entryFile());
-    if (!j) return false;
-    if (j->value("schema", 0) != kEntrySchema) return false;
-    if (j->value("key", std::string{}) != key.keyHex) return false;
+    if (!j) return probe;
+    if (j->value("schema", 0) != kEntrySchema) return probe;
+    if (j->value("key", std::string{}) != key.keyHex) return probe;
     auto it = j->find("inputs");
-    if (it == j->end() || !inputs_match(*it, key.inputs)) return false;
+    if (it == j->end() || !inputs_match(*it, key.inputs)) return probe;
+
+    // The entry itself is valid. From here on, every remaining check is about
+    // whether it holds what THIS build is going to read.
+    auto recorded = artifacts_from(*j);
+    std::set<std::string> haveBmi(recorded.bmiFiles.begin(), recorded.bmiFiles.end());
+    std::set<std::string> haveObj;
+    for (auto& o : recorded.objFiles) haveObj.insert(o.cacheRel);
 
-    auto arts = artifacts_from(*j);
     std::error_code ec;
-    for (auto& g : arts.bmiFiles)
-        if (!std::filesystem::exists(cached_bmi_path(key, g), ec)) return false;
-    for (auto& o : arts.objFiles)
-        if (!std::filesystem::exists(cached_obj_path(key, o), ec)) return false;
-    return true;
+    for (auto& g : requested.bmiFiles) {
+        if (!haveBmi.contains(g)
+            || !std::filesystem::exists(cached_bmi_path(key, g), ec))
+            probe.layoutMismatch.push_back(g);
+    }
+    for (auto& o : requested.objFiles) {
+        if (!haveObj.contains(o.cacheRel)
+            || !std::filesystem::exists(cached_obj_path(key, o.cacheRel), ec))
+            probe.layoutMismatch.push_back(o.cacheRel);
+    }
+    probe.ok = probe.layoutMismatch.empty();
+    return probe;
+}
+
+bool is_cached(const CacheKey& key, const DepArtifacts& requested) {
+    return probe_cached(key, requested).ok;
 }
 
 std::expected<DepArtifacts, std::string> resolve_cached(const CacheKey& key) {
@@ -250,7 +327,6 @@ populate_from(const CacheKey& key,
     std::filesystem::create_directories(cacheObj, ec);
 
     auto projectBmi = projectTargetDir / key.bmiDirName;
-    auto projectObj = projectTargetDir / "obj";
 
     for (auto& g : arts.bmiFiles) {
         auto from = projectBmi / g;
@@ -263,15 +339,20 @@ populate_from(const CacheKey& key,
                 "populate bmi '{}': {}", g, ec.message()));
         }
     }
+    // Read from `buildRel`, write at `cacheRel`. These are NOT the same path in
+    // general (mcpp#344): the build-dir layout partitions objects by package,
+    // the entry's layout is package-internal, and a source that sits outside its
+    // package root is re-anchored for the entry. Deriving one from the other
+    // here is what this split exists to prevent.
     for (auto& o : arts.objFiles) {
-        auto from = projectObj / o;
-        if (!std::filesystem::exists(from)) {
+        auto from = projectTargetDir / o.buildRel;
+        if (o.buildRel.empty() || !std::filesystem::exists(from)) {
             return std::unexpected(std::format(
                 "expected build output missing: {}", from.string()));
         }
-        if (!copy_one(from, cached_obj_path(key, o), ec)) {
+        if (!copy_one(from, cached_obj_path(key, o.cacheRel), ec)) {
             return std::unexpected(std::format(
-                "populate obj '{}': {}", o, ec.message()));
+                "populate obj '{}': {}", o.cacheRel, ec.message()));
         }
     }
 
@@ -289,7 +370,13 @@ populate_from(const CacheKey& key,
     j["tag"]      = key.manifestTag;
     j["inputs"]   = key.inputs;
     j["bmi"]      = arts.bmiFiles;
-    j["obj"]      = arts.objFiles;
+    // Only the entry-internal addresses. Recording the consumer's build path
+    // here is mcpp#344 in one line.
+    {
+        auto objs = nlohmann::json::array();
+        for (auto& o : arts.objFiles) objs.push_back(o.cacheRel);
+        j["obj"] = std::move(objs);
+    }
     j["accessed"] = now_iso8601();
     return write_entry(key.entryFile(), j);
 }

src/bmi_cache/maintenance.cppm

@@ -177,9 +177,24 @@ void check_pkg_files(Entry& e, const nlohmann::json& j) {
     if (auto it = j.find("obj"); it != j.end() && it->is_array()) {
         for (auto& v : *it) {
             if (!v.is_string()) continue;
-            if (missing(e.dir / "obj" / v.get<std::string>())) {
+            auto rel = v.get<std::string>();
+            // mcpp#344: an entry's obj addresses must be package-internal —
+            // downward, relative, no drive letter. An address that escapes the
+            // entry is one that was derived from some consumer's build tree,
+            // which is precisely the defect that made two consumers of one key
+            // disagree about the layout. Report it here so the invariant is
+            // auditable offline rather than only observable as a build that
+            // dies in ninja's graph phase.
+            if (rel.empty() || rel.starts_with("/") || rel.starts_with("..")
+                || rel.find(':') != std::string::npos) {
                 e.complete = false;
-                e.problem = std::format("missing obj/{}", v.get<std::string>());
+                e.problem = std::format(
+                    "obj address is not package-internal: '{}'", rel);
+                return;
+            }
+            if (missing(e.dir / "obj" / rel)) {
+                e.complete = false;
+                e.problem = std::format("missing obj/{}", rel);
                 return;
             }
         }

src/build/cache_key.cppm

@@ -64,7 +64,13 @@ export namespace mcpp::build::cache_key {
 // Deliberately NOT the mcpp release number: folding the whole version in
 // orphaned every entry on every release, including plain C object files whose
 // validity has nothing to do with mcpp's version.
-inline constexpr int kCacheEpoch = 1;
+// 2 (mcpp#344): the artifact layout changed. An entry's obj addresses are now
+// package-internal instead of "the first consumer's build-dir path minus
+// `obj/`", so entries written by an older mcpp describe a layout this one does
+// not ask for. They would all miss anyway (probe_cached compares the REQUESTED
+// artifacts), but sharing a directory between two layouts makes `cache gc`'s
+// size accounting and `cache verify`'s output meaningless.
+inline constexpr int kCacheEpoch = 2;
 
 // Axes A/B/C — identical for every package in one build, computed once.
 struct BuildAxes {

src/build/ninja_backend.cppm

@@ -749,15 +749,33 @@ std::string emit_ninja_string(const BuildPlan& plan) {
     // link/archive command; revisit the first-match replace if a dialect
     // ever grows another).
     //
-    // rsp is used when the command spawns through CreateProcess (32 KiB
-    // command-line ceiling): always for the separate-linker msvc dialect,
-    // and on Windows for driver-style too (#247 — ffmpeg/opencv-class
-    // packages link thousands of objects; clang/gcc drivers and GNU/llvm ar
-    // all accept @rspfile). POSIX driver-style keeps the inline form
-    // byte-identical: ARG_MAX is ample and the plain command is easier to
-    // reproduce by hand.
+    // rsp is used ALWAYS, on every platform and every dialect. The objects of
+    // one link edge are unbounded — an ecosystem package like opencv or ffmpeg
+    // contributes thousands — and every way of spawning a command has a ceiling:
+    //
+    //   Windows   CreateProcess, 32 KiB command line (#247)
+    //   POSIX     ninja spawns `sh -c "<whole command>"`, so the command is a
+    //             SINGLE argv entry and hits MAX_ARG_STRLEN — 32 pages, 128 KiB
+    //             — long before the 2 MiB ARG_MAX anyone would think to check.
+    //
+    // This used to read "POSIX keeps the inline form; ARG_MAX is ample and the
+    // plain command is easier to reproduce by hand". Both halves were wrong.
+    // ARG_MAX is the wrong limit, and it was never ample: measured on
+    // mcpp-index's opencv-module, the inline link line was already 56 840 bytes
+    // — 43% of a ceiling nothing was watching. mcpp#344 lengthened dependency
+    // object paths (they now carry a per-package directory) and the same edge
+    // reached 161 687 bytes, at which point ninja dies with
+    //
+    //     ninja: fatal: posix_spawn: Argument list too long
+    //
+    // naming no edge, no file and no cause. A build system may not have a
+    // maximum project size that it discovers by crashing, and "how long is this
+    // command" must not be a thing anyone has to keep in their head when
+    // choosing an object path. clang/gcc drivers, link.exe, GNU ar and llvm-ar
+    // all accept @rspfile, so there is one rule shape everywhere; the response
+    // file sits next to the output and `cat`ing it beats reading a 160 KB line.
     {
-        const bool useRsp = separateLinker || mcpp::platform::is_windows;
+        constexpr bool useRsp = true;
         auto link_rule = [&](std::string_view name, std::string cmd,
                              std::string_view desc) {
             append(std::format("rule {}\n", name));