Skip to content

fix: strip preamble text before heading validation in enhance validator#4125

Merged
yujonglee merged 3 commits intomainfrom
devin/1771588492-trim-before-validate
Feb 20, 2026
Merged

fix: strip preamble text before heading validation in enhance validator#4125
yujonglee merged 3 commits intomainfrom
devin/1771588492-trim-before-validate

Conversation

@devin-ai-integration
Copy link
Contributor

@devin-ai-integration devin-ai-integration bot commented Feb 20, 2026

Summary

Adds a stripPreamble step to the enhance validator pipeline that strips text before the first # before running heading validation. This addresses the bug where cloud LLMs (pro/mistral) emit preamble text (e.g. "Here is the summary:\n# ...") which caused the early validator to reject the stream before the existing trimBeforeMarker("#") transform could fix it.

The pipe function is generalized via a PipeStep type to support both text-transform steps (returning { text }) and validator steps (returning { valid: true } or { valid: false, feedback }), with stripPreamble → requireH1 → matchSectionHeading as the new pipeline order.

Review & Testing Checklist for Human

  • Edge case: # in preamble that isn't a headingstripPreamble uses text.indexOf("#") which would match # in e.g. "Issue toast #123:\n# Actual Heading". This would slice at the wrong #, then requireH1 would reject #123:... since it doesn't match # (with space). Consider whether this is a realistic LLM output pattern.
  • Long preambles still fail — The early validator checks the first 10-30 chars. If preamble is >30 chars before #, validation fires before # appears and still fails. This PR improves short-preamble cases but doesn't fully solve the original bug for verbose LLM preambles.
  • Test the fix end-to-end — Generate a summary with a cloud LLM (pro/mistral) using an auto-generated template (no user template) and verify it no longer fails with "Validation failed after 2 attempts: Output must start with the first template section heading".

Notes

Co-Authored-By: john@hyprnote.com <john@hyprnote.com>
@devin-ai-integration
Copy link
Contributor Author

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

  • Address comments on this PR that start with 'DevinAI' or '@devin'.
  • Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

  • Disable automatic comment and CI monitoring

@netlify
Copy link

netlify bot commented Feb 20, 2026

Deploy Preview for hyprnote canceled.

Name Link
🔨 Latest commit 995525d
🔍 Latest deploy log https://app.netlify.com/projects/hyprnote/deploys/69984fe20d9c130008bcf8df

@netlify
Copy link

netlify bot commented Feb 20, 2026

Deploy Preview for hyprnote-storybook canceled.

Name Link
🔨 Latest commit 995525d
🔍 Latest deploy log https://app.netlify.com/projects/hyprnote-storybook/deploys/69984fe2938012000886c490

devin-ai-integration bot and others added 2 commits February 20, 2026 12:00
Co-Authored-By: john@hyprnote.com <john@hyprnote.com>
Co-Authored-By: john@hyprnote.com <john@hyprnote.com>
@yujonglee yujonglee merged commit e105e6d into main Feb 20, 2026
14 of 15 checks passed
@yujonglee yujonglee deleted the devin/1771588492-trim-before-validate branch February 20, 2026 12:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants