Skip to content

v0.7.5

v0.7.5: ChunkCache — content-hash-validated pipeline cache (议 C)
@Hosuke Hosuke tagged this 20 Apr 07:11 
Downstream chunking pipelines (siwen wenguan etc.) cache expensive
per-chunk LLM outputs keyed by slot position (``chunks/{idx:02d}.md``).
When splitter boundaries shift, the same slot holds new content but
the positional key still resolves — cache returns stale output and
the pipeline stitches together LLM answers for chunks that no longer
exist. siwen hit this on 2026-04-20: 法華、仁王、宗體論 three books
corrupted in one run.

Fix: key by ``(cid, content_hash)``. Both must match on read; any
content change produces a new hash so the cache correctly misses.

Upstream stays domain-agnostic: cid and content_hash are opaque
strings the caller supplies. Downstream picks the strategy (siwen
wenguan: chunk H3 title + sha256 of chunk text; other pipelines
could use line-range or slug+idx). Parallels the customization
contract of ``normalize_heads`` rule packs.

Security / correctness:
- Full sha256 (not truncated) for both cid and content_hash paths,
  so distinct inputs never alias (Codex MEDIUM round 1).
- Atomic writes via ``atomic_write_text`` — concurrent readers
  never see a torn file.
- ``clear(cid)`` suppresses only ``FileNotFoundError`` (idempotent);
  real I/O errors propagate so stale entries can't silently linger
  (Codex MEDIUM round 1).
- cid / content_hash are sha-hashed before hitting disk, so
  ``../../etc/passwd`` or CJK slashes can't escape the cache dir.

Reviewed by Codex: LGTM after 1 round of fixes. 20 tests.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Assets 2
Loading