perf(build): fw-lib (framework library) object compiles still miss zccache cross-project

[zackees]

Context

Follow-up to #966 (PR #985). #985 made ESP32 sketch + core compiles hit zccache across project directories (warm compile-sketch 94.5 s → 0.73 s, core 12.9 s → 0.55 s) by fixing two fbuild-side determinism leaks: the object filename hashed the absolute source path, and library include order came from unsorted read_dir.

Framework-lib (fw-libs) object compiles still miss cross-project — warm fw-libs stays ~150 s. The 126 fw-libs TUs report verdict=Cold in the per-compile zccache DIAG (their context key is not found), while the 126 sketch/core TUs now verdict=Hit.

What's ruled out

• Args: a fw-lib compile's full argv (Esp.cpp, 435 tokens) is byte-identical between the /tmp/nds and /tmp/nds2 checkouts after normalizing the project-dir prefix — including -o, all -I, defines, and order. So it is not an arg-level path/order leak.
• Object filename: fw-lib sources live in the global framework cache (~/.fbuild/…), so fix(build): ESP32 sketch+core compiles hit zccache cross-project (#966) #985's object_hash_key keeps their absolute (already project-independent) path — object names match across checkouts.
• worktree_root: fix(build): ESP32 sketch+core compiles hit zccache cross-project (#966) #985 also wires ZCCACHE_WORKTREE_ROOT on the zccache wrap CLI path fw-libs use; it did not change the outcome.

Hypothesis

Since the args are identical yet the context key differs (verdict=Cold), the leak is a non-arg context input specific to the zccache wrap CLI path (fw-libs use the CLI wrapper → daemon IPC, not the in-process embedded service that sketch/core use). Candidates: per-session system_includes discovery differing on the CLI path, or a session/key_root resolution difference between the CLI-wrap session and the embedded ephemeral session.

Next diagnostic

Dump the fw-lib compile's context-key inputs (not just the hash) for cold vs warm — reuse the [FB966DIAG] tracing approach (patch a tracing::info! at pipeline/mod.rs verdict, [patch] fbuild's zccache to the local clone). Diff the normalized source/includes/defines/system-includes fed to compute_context_key between the two checkouts; the one differing input is the leak. Then relativize/normalize it (fbuild-side) or exclude it (zccache-side), matching #985's approach.

Acceptance

• A clean cold→warm harness run shows warm fw-libs hitting zccache (drops from ~150 s to seconds), completing cross-project caching for all firmware compile phases.

Part of #974 / follow-up to #966.

[zackees]

Corrected root cause: fw-libs bypass zccache entirely (not a key leak)

Re-diagnosed. fw-lib (framework builtin library) compiles are not cached by zccache at all — they run gcc directly.

• crates/fbuild-build/src/esp32/orchestrator/build.rs:46: let compiler_cache: Option<PathBuf> = None; (set to None by the perf(zccache) Phase 4 stage 2: embedded mandatory — delete FBUILD_ZCCACHE_EMBEDDED + wrapper fallback #800 embedded-service migration).
• That None flows to compile_framework_builtin_libs → library_compiler::compile_library_with_jobs → compile_one_source, which calls wrap_compiler_args(&args, compiler_cache). With compiler_cache = None, wrap_compiler_args returns the args unwrapped, and run_command spawns gcc directly — no zccache wrap, no embedded service.
• So fw-libs never touch zccache. Same-project rebuilds only skip them via the object-exists staleness check; a fresh checkout has no objects → full recompile (~150 s). That's why the earlier args-diff was empty and the key looked fine — the real fw-lib compiles aren't in the zccache pipeline at all. (The Esp.cpp I diffed is a core file compiled via the embedded service, which is why it hit.)

This is a #800 migration gap: #800 moved every compile to the in-process embedded ZccacheService and deleted the zccache wrap binary path — but library compilation in fbuild-packages::library::library_compiler was never migrated; it still shells out via run_command.

Fix options

1. Dispatch callback (preferred, aligns with perf(zccache) Phase 4 stage 2: embedded mandatory — delete FBUILD_ZCCACHE_EMBEDDED + wrapper fallback #800): fbuild-packages can't depend on fbuild-build (dependency direction), so thread a compile-dispatch trait object / closure from fbuild-build (which owns the embedded compile_backend) into compile_library_with_jobs, so lib TUs dispatch through the same embedded service as sketch/core. Then fix(build): ESP32 sketch+core compiles hit zccache cross-project (#966) #985's object-name + worktree_root fixes make them hit cross-project automatically.
2. Re-enable zccache wrap for libraries: pass a resolved zccache CLI path as compiler_cache. Simpler, but routes lib compiles to the standalone zccache daemon (a different cache store than the embedded service) and contradicts perf(zccache) Phase 4 stage 2: embedded mandatory — delete FBUILD_ZCCACHE_EMBEDDED + wrapper fallback #800's embedded-only intent.

Scope: this is an architectural change (compile-path migration), not a determinism tweak like #985. Its payoff is the ~150 s one-time-per-checkout fw-lib cost; the interactive sketch/core win already shipped in #985.
EOF