Skip to content

fix(content): strip model preamble/meta-chatter from AI summaries (#912) - #992

Merged
laboef1900 merged 1 commit into
devfrom
feature/issue-912-strip-summary-preamble
Jul 11, 2026
Merged

fix(content): strip model preamble/meta-chatter from AI summaries (#912)#992
laboef1900 merged 1 commit into
devfrom
feature/issue-912-strip-summary-preamble

Conversation

@laboef1900

Copy link
Copy Markdown
Contributor

Summary

  • AI summaries were persisting the model's raw output verbatim, so framing preambles ("Here's a summary of the article in Markdown format:") and trailing meta/offer lines ("…please provide it so I can respond in German.") got stored and rendered.
  • Adds an exported, conservative stripSummaryPreamble() in the summary worker, applied right after the stream is collected and before the empty-check, PII scan, and Markdown→HTML conversion, so both summary_text and summary_html are stored clean.
  • Hardens the summarize system prompt to tell the model to output only the summary (defense-in-depth; the strip pass is the guarantee).
  • Backend-only: no schema, contract, or frontend change.

Closes #912.

Root cause

summarizePage() stored the collected LLM stream directly, and the summarize prompt — unlike the improve_* prompts — never told the model to omit preamble/closing remarks, so framing and meta-chatter leaked into the stored/rendered summary.

Fix

  • New pure helper stripSummaryPreamble(markdown): strips a leading framing line (requires an explicit "here's/below is/the following is … summary …" opener plus a following newline) and any bare "Summary:" heading, then peels trailing lines carrying an explicit offer/meta phrase (e.g. "please provide", "let me know", "respond in German/English").
  • Wired in before the empty-check/PII-scan/HTML step; the existing empty-summary guard now also covers a model that returned only preamble.
  • Tightened the summarize prompt.

Testing

  • TDD: extended backend/src/domains/knowledge/services/summary-worker.test.ts — pure unit tests of the helper (strip leading/trailing, no false-positive on a real "If…" sentence, no-op on clean input) plus a DB-backed wiring test asserting the stored summary_text drops the framing and keeps the real content. Fails before the fix (stripSummaryPreamble is not a function; wiring test stored the framing text), passes after.
  • cd backend && npx vitest run src/domains/knowledge/services/summary-worker.test.ts (24 pass) - eslint (pass) - tsc --noEmit (pass)

Generated with Claude Code

Summaries persisted the LLM's raw output verbatim, so framing lines ("Here's a summary…") and trailing offer lines ("…please provide it so I can respond in German.") were stored and rendered. Adds an exported, conservative stripSummaryPreamble() applied before the empty-check/PII-scan/HTML conversion, plus prompt hardening telling the model to output only the summary. Covered by pure unit tests of the strip helper and a DB-backed wiring test in summary-worker.test.ts.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@laboef1900
laboef1900 merged commit 77c4387 into dev Jul 11, 2026
5 checks passed
@laboef1900
laboef1900 deleted the feature/issue-912-strip-summary-preamble branch July 13, 2026 05:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

AI summaries render the model's preamble and meta-chatter

1 participant