Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 10 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,16 @@

## [Unreleased]

### Fixed

- **The thought-quality gate accepted refinements that describe the memory instead of its subject.** 1.4.0 fixed the token budget that let `refine` write empty and truncated text, and that fix holds. It did not fix the failure underneath it, which was never the budget: nothing checked whether what consolidation wrote was *about* anything. Verified by running `dream` against a copy of a store that had just been repaired to zero damaged definitions — of 25 refinements, it rewrote a specific, grounded definition about SQLite-backed semantic indexes and workspace memory into "The memory concept involves rapid embedding of observations through optimized processing", and another into text that referred to its own prompt inputs as "Concept A" and "Concept B". Both were accepted.

**Grounding cannot catch this class, structurally.** This failure is a *paraphrase* of the definition it replaces, so it keeps that definition's vocabulary and scores well — the case above scored 0.32 with zero generic-marker hits. `groundingScore` measures whether a thought is derived from its evidence; it has nothing to say about whether the thought has a subject. So `assessThought` now treats self-referential meta-text openers and leaked `Concept A`/`Concept B` scaffolding as form failures, rejected unconditionally alongside truncation and markdown leakage rather than weighed against grounding.

The opener check is **anchored to the start of the text, deliberately**. A legitimate memory may quote this phrasing mid-sentence — a finding about memory corruption necessarily cites the corrupt text — and matching anywhere would stop the engine ever recording its own failure modes, which is exactly the knowledge worth keeping. The regression corpus in `thought-quality.test.ts` holds the real 2026-07-31 texts verbatim, including the legitimate row that must keep passing.

Left unfixed and worth naming: a rejected refinement leaves the existing row untouched, so a definition that is *already* meta-text stays that way. The gate stops new damage; it does not repair old.

## [1.4.0] — 2026-07-30

### The silent-failure release
Expand Down
54 changes: 54 additions & 0 deletions src/engines/thought-quality.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,60 @@ describe('groundingScore', () => {
});
});

/**
* Regression corpus from the 2026-07-31 incident, verbatim.
*
* dream refined these two rows from specific, grounded definitions into
* meta-text describing "the memory concept" instead of the subject. Both were
* accepted by the gate as it stood. This is the same failure signature that
* corrupted 301 of 639 definitions before engine 1.4.0, so the texts are kept
* exactly as they were written rather than paraphrased into something tidier.
*/
const REAL_BEFORE_1 =
"Embedding observations now occurs in seconds due to migration from SQLite-backed semantic indexes derived from markdown, with structured reflection loops and entity-aware retrieval, as demonstrated by systems like OpenClaw's workspace memory v2 and cortex-engine's production implementation.";
const REAL_AFTER_1 =
'The memory concept involves rapid embedding of observations through optimized processing, leveraging structured reflection and entity-aware retrieval, as seen in systems like OpenClaw and cortex-engine, with performance improvements achieved through migration from SQLite-backed indexes.';

const REAL_AFTER_2 =
'The memory concept encompasses two distinct yet interconnected intellectual pursuits: a science series and a humor-focused glossary. Concept A emphasizes the capacity of the memory concept to support diverse endeavors, while Concept B explores the nature of inquiry.';

describe('assessThought — self-referential meta-text (2026-07-31 regression)', () => {
it('rejects a refinement that defines the memory instead of the subject', () => {
// Grounding cannot save us here: this text is a PARAPHRASE of the very
// definition it replaces, so it keeps the vocabulary and scored 0.32 with
// zero generic-marker hits. It was accepted. That is the hole.
const result = assessThought(REAL_AFTER_1, { evidence: [REAL_BEFORE_1] });
expect(result.ok).toBe(false);
expect(result.reasons.join(' ')).toMatch(/describes the memory|meta/i);
});

it('rejects internal placeholder scaffolding leaking into stored text', () => {
const result = assessThought(REAL_AFTER_2, { evidence: [REAL_AFTER_2] });
expect(result.ok).toBe(false);
expect(result.reasons.join(' ')).toMatch(/placeholder|Concept A/i);
});

it('still accepts a legitimate memory ABOUT memory corruption', () => {
// The discriminator is POSITION. This row quotes the boilerplate as
// evidence mid-sentence; it does not open with it. A naive substring check
// would reject it, and rejecting it would stop dream ever recording
// findings about its own failures.
const legit =
'HALF THE MEMORY GRAPH IS CORRUPTED - measured, not estimated. Direct SQLite audit: 629 memories, 301 damaged. 53 were BOILERPLATE, where the definition was replaced with generic meta-text, for example "This memory phenomenon consistently occurs during the consolidation phase, reflecting its reliability and importance".';
const result = assessThought(legit, { evidence: [legit] });
expect(result.ok).toBe(true);
});

it('accepts a definition that merely mentions memory as its subject', () => {
const evidence = ['Cortex stores memories in SQLite with 1024-d embeddings.'];
const result = assessThought(
'Cortex stores memories in SQLite alongside 1024-dimensional embeddings, so retrieval needs no external vector database.',
{ evidence },
);
expect(result.ok).toBe(true);
});
});

describe('assessThought', () => {
it('accepts a grounded, complete refinement', () => {
const result = assessThought(
Expand Down
51 changes: 50 additions & 1 deletion src/engines/thought-quality.ts
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,46 @@ export const GENERIC_PHRASE_MARKERS: readonly string[] = [
'structured coordination', 'transparent boundaries',
];

/**
* Openers that make the ARTEFACT the grammatical subject instead of the thing
* being described — "The memory concept involves...", "The concept refers
* to...". A definition that opens this way says nothing about its subject, and
* a name derived from it inherits the evasion.
*
* WHY THIS CANNOT BE LEFT TO `grounding`. This failure mode is a *paraphrase*
* of the definition it replaces, so it retains that definition's vocabulary and
* scores well. The 2026-07-31 regression scored 0.32 with zero generic-marker
* hits and was accepted. Grounding measures whether a thought is derived from
* its evidence; it cannot measure whether the thought is ABOUT anything. So
* these veto regardless of grounding, exactly like the other form checks.
*
* ANCHORED AT THE START, DELIBERATELY. A legitimate memory may quote this
* phrasing mid-sentence — a finding about corruption necessarily cites the
* corrupt text. Matching anywhere would stop dream ever recording its own
* failure modes, which is precisely the knowledge worth keeping.
*/
const SELF_REFERENTIAL_OPENERS: readonly RegExp[] = [
/^(?:the|this)\s+(?:(?:memory|refined|unifying|underlying)\s+)?concept\b/i,
/^(?:the|this)\s+memory\s+(?:phenomenon|entry|record)\b/i,
/^(?:this|it)\s+refers\s+to\b/i,
/^the\s+term\s+(?:refers|describes|is)\b/i,
/^the\s+idea\s+(?:that|of)\b/i,
/^the\s+phrase\b/i,
];

/**
* Internal comparison scaffolding leaking into stored text. The refine and
* connect prompts label their inputs "Concept A" / "Concept B"; when those
* labels survive into the definition, the row describes the prompt rather than
Comment on lines +71 to +73
* the world. Seen verbatim on 2026-07-31: "Concept A emphasizes the capacity of
* the memory concept to support diverse endeavors".
*
* A legitimate definition could in principle discuss an abstract "Concept A",
* and this would reject it. That trade is deliberate: a rejected refinement
* leaves the existing row untouched, which is the safe direction.
*/
const PLACEHOLDER_LEAK = /\bConcept\s+[A-Z]\b/;

export interface ThoughtQualityOptions {
/**
* Texts the thought is supposed to be derived from (current definition,
Expand Down Expand Up @@ -92,7 +132,10 @@ export function groundingScore(text: string, evidence: string[]): number {
* Assess whether model-generated text is acceptable as a stored thought.
*
* Decision rule:
* - Form failures (empty, truncated, markdown-formatted, out of bounds) reject.
* - Form failures (empty, truncated, markdown-formatted, out of bounds,
* self-referential meta-text, placeholder leakage) reject. These are
* unconditional: grounding cannot arbitrate a thought that is well-derived
* from its evidence and still about nothing.
* - Two or more generic-phrase markers reject.
* - With evidence: grounding below `minGrounding` rejects, and a single
* generic marker rejects when grounding is only marginal
Expand Down Expand Up @@ -123,6 +166,12 @@ export function assessThought(
if (/^(#{1,6}\s|\*\*)/.test(trimmed)) {
reasons.push('markdown formatting leaked into thought');
}
if (SELF_REFERENTIAL_OPENERS.some((re) => re.test(trimmed))) {
reasons.push('describes the memory rather than its subject (meta-text opener)');
}
if (PLACEHOLDER_LEAK.test(trimmed)) {
reasons.push('internal placeholder scaffolding leaked into thought (Concept A/B)');
}

const lower = trimmed.toLowerCase();
const genericHits = GENERIC_PHRASE_MARKERS.filter((m) => lower.includes(m));
Expand Down
Loading