fix(boringstack): read eslint from structured JSON, not scraped stylish text#121
Merged
Conversation
…ish text Root cause of a whole class of near-green stalls: the gate scraped eslint's "stylish" (human) output, which is ambiguous — multi-line messages truncated to their first line (the model never saw the fix), and interleaved output mis-attributed errors to the wrong file. #120 patched the heuristic; the reviewer panel proved (5 rounds) that stylish text can't be parsed reliably — it has no delimiters. Fix at the source: the gate now emits each app's OWN eslint as `--format json` (via `bun run --silent lint -- --format json`, reusing the app's exact config/globs — no coupling), wrapped in `::tsforge-eslint-json::` markers. extract-failures parses that into exact `failure:<file>:<line>:<rule>:<message>` signatures — unambiguous file, line, ruleId, and FULL message. When a JSON block is present the line loop ignores the stylish eslint rows (JSON is the single source of truth, no double-count); when it's ABSENT or malformed it falls back to scraping stylish, so a lint error is never silently lost (no false green). tsc / bun `(fail)` / knip / lint-meta are unchanged. This raises feedback fidelity for EVERY eslint rule at once — the model always gets the complete error + fix, not a mutilated first line. Tests: JSON → exact signatures (incl. the full multi-line module-boundaries message); stylish rows for the same run aren't double-counted; a `error TS…` string inside a JSON message isn't mis-read as tsc; tsc/`(fail)` still parse alongside JSON; malformed JSON falls back to stylish. Gate tests assert the JSON pass is emitted (fast + full). Full `bun run validate` green.
… lost lint) Panel caught a critical: the "eslint came from JSON" decision was GLOBAL, but the gate emits one block per app (api, ui). If the API block parsed (even `[]`) while the UI block was empty/malformed, the UI's stylish rows were suppressed and its real lint errors were silently lost — the exact near-green regression this change exists to fix. Also, any JSON array (`[1,2,3]`) counted as coverage. Fix: coverage is PER-APP. Each block is tagged `::tsforge-eslint-json <app>::`; a block counts only if it parses into a genuine eslint-result array (every element has `filePath` + `messages`; empty `[]` = a valid green app). `extractFailures` tracks the set of covered apps and ignores stylish ERROR rows only for an app in that set — an uncovered app (missing/malformed/wrong-shaped block) still falls back to scraping stylish, so no lint error is ever lost. Skip narrowed to error rows (warnings were never captured by the stylish path anyway). Tests: API `[]` + malformed UI block + UI stylish error → the UI error is still captured; a wrong-shaped `[1,2,3]` array does not count as coverage (stylish scraped). Plus the prior JSON tests, now app-tagged. Full `bun run validate` green. Notes (accepted): the JSON pass reruns eslint but with `--cache`, so `check`'s lint is a warm cache hit; `2>/dev/null` keeps the JSON clean — a real lint crash still surfaces via `check`'s own lint run.
…table signatures
Second panel pass on the per-app eslint-JSON change surfaced four more real ways a
lint error could be silently lost. All fixed:
1. Coverage now means "the JSON actually YIELDED ≥1 error", not merely "the block
parsed". An empty `[]`, an all-green app, a 0-file run, or a shape-valid payload
whose message entries are malformed (`[{filePath,messages:[{}]}]`) yield zero
errors → the app is NOT covered → its stylish rows are still scraped. Safe both
ways: a truly green app has no stylish error rows to lose, and a broken JSON pass
falls back to `check`'s own stylish output. Resolves the `[]`-vacuous and
incomplete-payload findings at once.
2. Truncation safety: an unterminated JSON block (missing end marker) now ends at the
next `::tsforge-app` stage marker instead of leaving `inEslintJson` true across the
app boundary and swallowing every later line (tsc errors, bun fails, stylish rows).
3. Stable signatures: a multi-line JSON message is whitespace-collapsed so it keys to
the SAME signature as the stylish path — else a block that intermittently fails to
parse would make a pre-existing failure look novel in the differential.
4. `isEslintResultArray` is now a type guard, dropping the dead `!Array.isArray` re-check.
Tests: malformed-message block → not covered (stylish captured); unterminated block →
next app's tsc error still captured; JSON multi-line message signature has no %0A and
matches the space-joined form. Full `bun run validate` green (7/7).
… per-app suppression) Third panel pass drove the design to its correct shape. The prior per-app "coverage" model could, in edge cases, suppress a real lint error. Replaced it with per-error dedup: - Dedup key is per (file,line,rule), not per app. A stylish row is dropped ONLY when the JSON already reported the SAME error at that exact location; a stylish-only error the JSON never emitted (a JSON subset) is always kept. This removes the whole class of "JSON parsed → suppress all stylish for the app" findings. - Truncation safety via STRIPPING closed blocks rather than a sticky skip flag. Only a CLOSED `::tsforge-eslint-json …::…::-end::` block is removed from the scanned text (its signatures already extracted). An UNTERMINATED block (killed/truncated capture) is left in place and parsed line-by-line — its partial JSON line isn't an error row, so it is skipped harmlessly and later tsc/stylish/knip diagnostics are never swallowed. - Deleted the `inEslintJson` + `eslintJsonApps` state fields and the sticky consumeEslintJsonBlock() entirely — the design is now stateless w.r.t. JSON blocks. Tests: JSON error at a.ts:6 + stylish-only error at a.ts:9 → BOTH captured (the subset case); unterminated block → next app's tsc error still captured; malformed/empty/wrong- shaped blocks contribute no dedup keys so stylish is preserved; gate tests tightened so a dropped UI check/validate now fails. Full `bun run validate` green (7/7).
…s-block strip)
Fourth panel pass, two more real edges closed:
- Dedup key now includes COLUMN (file:line:column:rule), so two reports of the same rule
on the same line at different columns stay distinct — a column-less key could drop a
stylish error at a different column that the JSON never reported. Column lives only in
the internal dedup key, not the signature, so no signature format changes. parsedDiagnostic
now returns { signature, dedupKey }; the loop dedups on dedupKey.
- The block regex is TEMPERED against a second opening marker, so an unterminated block
can no longer pair with a LATER closed block's end marker and strip the tsc/stylish/knip
output between them. One shared source (ESLINT_JSON_BLOCK_SOURCE) for both the strip and
the parse; a fresh RegExp per use avoids g-flag lastIndex carryover.
Tests: unterminated-then-closed block keeps the intervening tsc error AND the later UI JSON
error; a stylish error at a different column on the same line/rule is not dropped. Full
`bun run validate` green (7/7).
agjs
added a commit
that referenced
this pull request
Jul 17, 2026
…ung the gate PR #121 added a tempered-quantifier regex to locate closed eslint-JSON blocks: `::…json \S+::((?:(?!::…json \S+::)[\s\S])*?)::…json-end::`. On tiny test fixtures it was instant, but on REAL gate output (measured 1.24MB: full `bun test` + eslint --format json for ~160 files × 2 apps) the negative-lookahead-per-char non-greedy body backtracks O(n²)+ — extractFailures ran >240s without finishing, silently hanging the whole gate (no subprocess, pure CPU: the exact "running gate · turn N…" freeze observed on a live build). Fix: locate closed blocks by a LINEAR merge scan (findClosedEslintJsonBlocks) — collect all opening markers (one regex pass) and all end markers (allIndicesOf, one pass), then pair them with a single forward-only pointer so each opening claims the first end after it and before the next opening. No per-opening rescan-to-EOF (which would itself be O(n²)), no catastrophic regex. Unterminated blocks are skipped (same semantics). Strip parsed blocks by range (stripRanges) instead of a second catastrophic .replace. Result: the 1.24MB output parses in ~4ms. All #121 structured-JSON behavior is unchanged (per-error dedup, column keys, whitespace-normalized signatures, truncation safety). Tests: a ~1MB realistic output AND a pathological ~1MB of openings-with-no-ends both parse in <2s (the latter is the per-opening-indexOf O(n²) trap the merge scan defeats).
agjs
added a commit
that referenced
this pull request
Jul 17, 2026
…ung the gate PR #121 added a tempered-quantifier regex to locate closed eslint-JSON blocks: `::…json \S+::((?:(?!::…json \S+::)[\s\S])*?)::…json-end::`. On tiny test fixtures it was instant, but on REAL gate output (measured 1.24MB: full `bun test` + eslint --format json for ~160 files × 2 apps) the negative-lookahead-per-char non-greedy body backtracks O(n²)+ — extractFailures ran >240s without finishing, silently hanging the whole gate (no subprocess, pure CPU: the exact "running gate · turn N…" freeze observed on a live build). Fix: locate closed blocks by a LINEAR merge scan (findClosedEslintJsonBlocks) — collect all opening markers (one regex pass) and all end markers (allIndicesOf, one pass), then pair them with a single forward-only pointer so each opening claims the first end after it and before the next opening. No per-opening rescan-to-EOF (which would itself be O(n²)), no catastrophic regex. Unterminated blocks are skipped (same semantics). Strip parsed blocks by range (stripRanges) instead of a second catastrophic .replace. Result: the 1.24MB output parses in ~4ms. All #121 structured-JSON behavior is unchanged (per-error dedup, column keys, whitespace-normalized signatures, truncation safety). Tests: a ~1MB realistic output AND a pathological ~1MB of openings-with-no-ends both parse in <2s (the latter is the per-opening-indexOf O(n²) trap the merge scan defeats).
agjs
added a commit
that referenced
this pull request
Jul 17, 2026
…ung the gate (#122) PR #121 added a tempered-quantifier regex to locate closed eslint-JSON blocks: `::…json \S+::((?:(?!::…json \S+::)[\s\S])*?)::…json-end::`. On tiny test fixtures it was instant, but on REAL gate output (measured 1.24MB: full `bun test` + eslint --format json for ~160 files × 2 apps) the negative-lookahead-per-char non-greedy body backtracks O(n²)+ — extractFailures ran >240s without finishing, silently hanging the whole gate (no subprocess, pure CPU: the exact "running gate · turn N…" freeze observed on a live build). Fix: locate closed blocks by a LINEAR merge scan (findClosedEslintJsonBlocks) — collect all opening markers (one regex pass) and all end markers (allIndicesOf, one pass), then pair them with a single forward-only pointer so each opening claims the first end after it and before the next opening. No per-opening rescan-to-EOF (which would itself be O(n²)), no catastrophic regex. Unterminated blocks are skipped (same semantics). Strip parsed blocks by range (stripRanges) instead of a second catastrophic .replace. Result: the 1.24MB output parses in ~4ms. All #121 structured-JSON behavior is unchanged (per-error dedup, column keys, whitespace-normalized signatures, truncation safety). Tests: a ~1MB realistic output AND a pathological ~1MB of openings-with-no-ends both parse in <2s (the latter is the per-opening-indexOf O(n²) trap the merge scan defeats).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What & why
The BoringStack gate composes
bun test+tsc+eslintoutput and the harness scrapes it into failure signatures (failure:<file>:<line>:<rule>:<message>) for the differential gate. eslint's human "stylish" format is ambiguous — it truncates multi-line rule messages and interleaves with other tool output — so the model repeatedly saw unactionable half-messages (e.g.…detected in module:with no categories, no fix) and sprayed near-green on live builds.This makes each app also emit its own eslint as
--format json(wrapped in::tsforge-eslint-json <app>::markers, non-aborting —checkstill owns pass/fail), and the parser reads exact{file,line,column,ruleId,message}from that structured data instead of guessing from text.How it stays correct (the load-bearing part)
Both the JSON aid and
check's own stylish eslint appear in the output, so the parser must reconcile them without either double-counting or losing an error. The design that survived review:file:line:column:rule. A stylish-only error the JSON never emitted (different line, column, or rule) is always kept. Column is in the dedup key so two reports of one rule on the same line at different columns stay distinct.[], a wrong-shaped array ([1,2,3]), or a malformed message entry contributes no dedup keys, so it suppresses nothing — a broken/absent JSON block always falls back to stylish.Tests
New cases cover: JSON → exact signatures with stylish deduped; a stylish-only error at another line/column kept; unterminated block doesn't swallow later tsc errors; unterminated-then-closed blocks don't cross-strip; malformed/empty/wrong-shaped blocks preserve stylish;
error TS…inside a JSON message not mis-parsed; whitespace-normalized signatures stable across JSON/stylish. Gate tests tightened so a dropped UIcheck/validatenow fails. Fullbun run validategreen (7/7).Known limitation (pre-existing, out of scope)
structuredFailuredoes not carry column, so two errors with identical file/line/rule/message at different columns collapse to one signature — this has always been true of the stylish path and is unchanged here. Fixing it means putting column in the signature format itself, which ripples into the differential baseline,signatureToError, and memory; tracked as a follow-up rather than bolted onto this parsing change.Review
Passed the local
harness-reviewpanel (4 reviewers) across 4 hardening rounds — the panel caught a real critical (global vs per-app coverage) and several edge cases that drove the design to per-error dedup + strip-based truncation safety.