Skip to content

fix(dream): reject refinements that describe the memory, not its subject - #52

Merged
idapixl merged 1 commit into
masterfrom
fix/dream-meta-text-gate
Aug 1, 2026
Merged

fix(dream): reject refinements that describe the memory, not its subject#52
idapixl merged 1 commit into
masterfrom
fix/dream-meta-text-gate

Conversation

@idapixl

@idapixl idapixl commented Aug 1, 2026

Copy link
Copy Markdown
Collaborator

The bug 1.4.0 didn't fix

1.4.0 fixed the token budget that let refine write empty and truncated definitions, and that fix holds — no truncation appeared in this run. It did not fix the failure underneath it, which was never the budget: nothing checked whether what consolidation wrote was about anything.

Found by running dream against a copy of a store that had just been repaired from 301 damaged definitions to 0. Of 25 refinements it produced two regressions:

One rewrote a specific, grounded definition into meta-text:

Before: Embedding observations now occurs in seconds due to migration from SQLite-backed semantic indexes derived from markdown, with structured reflection loops and entity-aware retrieval, as demonstrated by systems like OpenClaw's workspace memory v2…

After: The memory concept involves rapid embedding of observations through optimized processing…

The other stored its own prompt scaffolding:

The memory concept encompasses two distinct yet interconnected intellectual pursuits… Concept A emphasizes the capacity of the memory concept to support diverse endeavors, while Concept B explores…

Both were accepted by assessThought.

Why grounding can never catch this

This is the part that matters for the design, not just the patch.

The failure is a paraphrase of the definition it replaces. It therefore retains that definition's vocabulary and scores well — the first case scored 0.32 grounding with zero generic-marker hits, comfortably above the 0.25 floor.

groundingScore answers "is this thought derived from its evidence?" It has nothing to say about "does this thought have a subject?" A text can be faithfully derived from its source and still be about nothing. No threshold tuning fixes that, because the signal isn't weak — it's measuring a different property.

So self-referential meta-text openers and leaked Concept A/Concept B scaffolding are now form failures, rejected unconditionally alongside truncation and markdown leakage, rather than weighed against grounding.

The opener check is anchored, deliberately

A legitimate memory may quote this phrasing mid-sentence — a finding about memory corruption necessarily cites the corrupt text. Matching anywhere in the string would stop the engine ever recording its own failure modes, which is precisely the knowledge worth keeping.

The test corpus keeps the real 2026-07-31 texts verbatim, including a legitimate row that must continue to pass:

HALF THE MEMORY GRAPH IS CORRUPTED… 53 were BOILERPLATE, where the definition was replaced with generic meta-text, for example "This memory phenomenon consistently occurs during the consolidation phase…"

That row quotes the boilerplate as evidence. It passes. A naive substring check would have rejected it.

Verification

check result
full suite 312 passed (35 files)
npm run build exit 0
regression 1 against fixed gate rejected — describes the memory rather than its subject
regression 2 against fixed gate rejected — meta-opener + placeholder leak
legitimate corruption-finding row still accepted
original good definition still accepted

Known limitation, stated rather than hidden

A rejected refinement leaves the existing row untouched. The gate stops new damage; it does not repair old. A definition that is already meta-text stays that way until repaired separately.

Also unrelated and still open: refine's edge-revalidate step emits Document not found: edges/… for edges that exist in the store. A lookup or scoping fault, not missing data — worth its own issue.

1.4.0 fixed the token budget behind the empty and truncated definitions,
and that fix holds. It did not fix the bug underneath, which was never
the budget: nothing checked whether what consolidation wrote was ABOUT
anything.

Found by running dream against a copy of a store that had just been
repaired to 0 damaged definitions. Of 25 refinements it produced two
regressions - one rewrote a specific, grounded definition about
SQLite-backed semantic indexes into "The memory concept involves rapid
embedding of observations through optimized processing", the other
stored its own prompt scaffolding, referring to "Concept A" and
"Concept B" as though they were the subject. Both were accepted by the
gate.

GROUNDING CANNOT CATCH THIS CLASS. The failure is a paraphrase of the
definition it replaces, so it retains that definition's vocabulary and
scores well - the case above scored 0.32 with zero generic-marker hits.
groundingScore measures whether a thought derives from its evidence; it
says nothing about whether the thought has a subject at all. So these
are now form failures, rejected unconditionally alongside truncation and
markdown leakage, rather than weighed against grounding.

The opener check is anchored to the START of the text on purpose. A
legitimate memory may quote this phrasing mid-sentence - a finding about
memory corruption necessarily cites the corrupt text - and matching
anywhere would stop the engine ever recording its own failure modes. The
test corpus keeps the real 2026-07-31 texts verbatim, including the
legitimate row that must continue to pass.

312 tests green, build clean.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DGqQGN6tcEQcE9bX1VAboL
Copilot AI review requested due to automatic review settings August 1, 2026 00:11

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR strengthens the dream thought-quality gate to reject refinements that are structurally “about the memory” (meta-text) rather than about the memory’s subject, preventing grounded-but-subjectless paraphrases from being stored during consolidation.

Changes:

  • Add unconditional “form failure” rejections for self-referential meta-text openers and leaked Concept A/Concept B scaffolding in assessThought.
  • Add a verbatim regression corpus test suite covering the 2026-07-31 incident, including a case that must continue to pass due to mid-sentence quoting.
  • Document the incident and rationale in CHANGELOG.md.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

File Description
src/engines/thought-quality.ts Adds new unconditional rejection checks for meta-text openers and placeholder leakage inside assessThought.
src/engines/thought-quality.test.ts Adds regression tests using verbatim incident texts to ensure the new gate rejects the failures while preserving legitimate corruption-reporting memories.
CHANGELOG.md Records the fix rationale and verification details under Unreleased.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment on lines +71 to +73
* Internal comparison scaffolding leaking into stored text. The refine and
* connect prompts label their inputs "Concept A" / "Concept B"; when those
* labels survive into the definition, the row describes the prompt rather than
@idapixl

idapixl commented Aug 1, 2026

Copy link
Copy Markdown
Collaborator Author

End-to-end verification: same dream, same store, fixed gate

Re-ran the full dream cycle against a fresh copy of the same store, with this branch's gate swapped into the service's resolved engine. Same corpus, same model, same code path — only the gate changed.

1.4.0 gate this branch
regressions (clean → damaged) 2 0
rows touched 39 30
damaged after 3 / 641 1 / 642
runtime 630s 696s

Zero regressions. Nothing that was clean got worse.

The rejections fired in the real path, not just in unit tests:

[dream:refine] Rejected refinement for fQfKvSt7FStPrCeXWUfq:
  describes the memory rather than its subject (meta-text opener);
  internal placeholder scaffolding leaked into thought (Concept A/B)
[dream:refine] Rejected refinement for y23jnLAihHBfQgx8xf4a:
  describes the memory rather than its subject (meta-text opener);
  generic phrasing (2 marker hits)

Each of those would have been a silent corruption on 1.4.0.

The single remaining "damaged" row is a false positive in the auditing harness, not in this gate — a legitimate memory about the corruption, flagged only because it quotes the boilerplate as evidence. This branch's anchored check accepts it correctly. The harness has been corrected to anchor too; a checker less precise than the gate it audits is exactly the self-reported result this work exists to distrust.

Why the placeholder rejections are so frequent

Worth recording, because it explains the rejection rate and points at a second bug this PR does not fix.

The connect phase writes edge evidence in the form "Concept A ... Concept B ...". In the live store that is 2,665 of 3,259 edges — 82%. refine consumes edge evidence, so the model echoes those labels straight into definitions. Six rows in the live store already carry it, one of them named Concept A describes the agent's capacity to retain….

So this gate is catching contamination whose source is upstream. Blocking it at the write is correct and should stay, but the better fix is for refine to neutralise the labels before they reach the model, or for connect to stop persisting them in evidence. Filed separately rather than widened into this PR.

Not fixed here, found while verifying

The abstract phase still stores markdown mid-text — new rows land with names like The unifying pattern is **"Persistence through Structure"**. The existing markdown check is anchored to the start, so leading ** is caught and mid-text ** is not. Separate issue; not in scope for this change.

@idapixl
idapixl merged commit 2f9f8b0 into master Aug 1, 2026
11 checks passed
@idapixl
idapixl deleted the fix/dream-meta-text-gate branch August 1, 2026 00:23
idapixl added a commit that referenced this pull request Aug 1, 2026
1.4.0 was named for silent failures and shipped with one of its own. It
fixed the token budget behind the empty and truncated definitions, and
that fix holds. It did not fix the failure underneath, which was never
the budget: nothing checked whether what consolidation wrote was about
anything.

Ships #52. Verified end-to-end on a real store: same dream, same model,
only the gate changed, regressions 2 -> 0.


Claude-Session: https://claude.ai/code/session_01DGqQGN6tcEQcE9bX1VAboL

Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants