fix(dream): reject refinements that describe the memory, not its subject - #52
Conversation
1.4.0 fixed the token budget behind the empty and truncated definitions, and that fix holds. It did not fix the bug underneath, which was never the budget: nothing checked whether what consolidation wrote was ABOUT anything. Found by running dream against a copy of a store that had just been repaired to 0 damaged definitions. Of 25 refinements it produced two regressions - one rewrote a specific, grounded definition about SQLite-backed semantic indexes into "The memory concept involves rapid embedding of observations through optimized processing", the other stored its own prompt scaffolding, referring to "Concept A" and "Concept B" as though they were the subject. Both were accepted by the gate. GROUNDING CANNOT CATCH THIS CLASS. The failure is a paraphrase of the definition it replaces, so it retains that definition's vocabulary and scores well - the case above scored 0.32 with zero generic-marker hits. groundingScore measures whether a thought derives from its evidence; it says nothing about whether the thought has a subject at all. So these are now form failures, rejected unconditionally alongside truncation and markdown leakage, rather than weighed against grounding. The opener check is anchored to the START of the text on purpose. A legitimate memory may quote this phrasing mid-sentence - a finding about memory corruption necessarily cites the corrupt text - and matching anywhere would stop the engine ever recording its own failure modes. The test corpus keeps the real 2026-07-31 texts verbatim, including the legitimate row that must continue to pass. 312 tests green, build clean. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DGqQGN6tcEQcE9bX1VAboL
There was a problem hiding this comment.
Pull request overview
This PR strengthens the dream thought-quality gate to reject refinements that are structurally “about the memory” (meta-text) rather than about the memory’s subject, preventing grounded-but-subjectless paraphrases from being stored during consolidation.
Changes:
- Add unconditional “form failure” rejections for self-referential meta-text openers and leaked
Concept A/Concept Bscaffolding inassessThought. - Add a verbatim regression corpus test suite covering the 2026-07-31 incident, including a case that must continue to pass due to mid-sentence quoting.
- Document the incident and rationale in
CHANGELOG.md.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
| src/engines/thought-quality.ts | Adds new unconditional rejection checks for meta-text openers and placeholder leakage inside assessThought. |
| src/engines/thought-quality.test.ts | Adds regression tests using verbatim incident texts to ensure the new gate rejects the failures while preserving legitimate corruption-reporting memories. |
| CHANGELOG.md | Records the fix rationale and verification details under Unreleased. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| * Internal comparison scaffolding leaking into stored text. The refine and | ||
| * connect prompts label their inputs "Concept A" / "Concept B"; when those | ||
| * labels survive into the definition, the row describes the prompt rather than |
End-to-end verification: same dream, same store, fixed gateRe-ran the full
Zero regressions. Nothing that was clean got worse. The rejections fired in the real path, not just in unit tests: Each of those would have been a silent corruption on 1.4.0. The single remaining "damaged" row is a false positive in the auditing harness, not in this gate — a legitimate memory about the corruption, flagged only because it quotes the boilerplate as evidence. This branch's anchored check accepts it correctly. The harness has been corrected to anchor too; a checker less precise than the gate it audits is exactly the self-reported result this work exists to distrust. Why the placeholder rejections are so frequentWorth recording, because it explains the rejection rate and points at a second bug this PR does not fix. The So this gate is catching contamination whose source is upstream. Blocking it at the write is correct and should stay, but the better fix is for Not fixed here, found while verifyingThe |
1.4.0 was named for silent failures and shipped with one of its own. It fixed the token budget behind the empty and truncated definitions, and that fix holds. It did not fix the failure underneath, which was never the budget: nothing checked whether what consolidation wrote was about anything. Ships #52. Verified end-to-end on a real store: same dream, same model, only the gate changed, regressions 2 -> 0. Claude-Session: https://claude.ai/code/session_01DGqQGN6tcEQcE9bX1VAboL Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
The bug 1.4.0 didn't fix
1.4.0 fixed the token budget that let
refinewrite empty and truncated definitions, and that fix holds — no truncation appeared in this run. It did not fix the failure underneath it, which was never the budget: nothing checked whether what consolidation wrote was about anything.Found by running
dreamagainst a copy of a store that had just been repaired from 301 damaged definitions to 0. Of 25 refinements it produced two regressions:One rewrote a specific, grounded definition into meta-text:
The other stored its own prompt scaffolding:
Both were accepted by
assessThought.Why grounding can never catch this
This is the part that matters for the design, not just the patch.
The failure is a paraphrase of the definition it replaces. It therefore retains that definition's vocabulary and scores well — the first case scored 0.32 grounding with zero generic-marker hits, comfortably above the 0.25 floor.
groundingScoreanswers "is this thought derived from its evidence?" It has nothing to say about "does this thought have a subject?" A text can be faithfully derived from its source and still be about nothing. No threshold tuning fixes that, because the signal isn't weak — it's measuring a different property.So self-referential meta-text openers and leaked
Concept A/Concept Bscaffolding are now form failures, rejected unconditionally alongside truncation and markdown leakage, rather than weighed against grounding.The opener check is anchored, deliberately
A legitimate memory may quote this phrasing mid-sentence — a finding about memory corruption necessarily cites the corrupt text. Matching anywhere in the string would stop the engine ever recording its own failure modes, which is precisely the knowledge worth keeping.
The test corpus keeps the real 2026-07-31 texts verbatim, including a legitimate row that must continue to pass:
That row quotes the boilerplate as evidence. It passes. A naive substring check would have rejected it.
Verification
npm run buildKnown limitation, stated rather than hidden
A rejected refinement leaves the existing row untouched. The gate stops new damage; it does not repair old. A definition that is already meta-text stays that way until repaired separately.
Also unrelated and still open:
refine's edge-revalidate step emitsDocument not found: edges/…for edges that exist in the store. A lookup or scoping fault, not missing data — worth its own issue.