fix(ai): score responses on groundedness and withhold ungrounded ones

[NathanTarbert]

Stacked on #142 — base is fix/ai-response-epistemic-guardrails, merge that one first.

What this fixes

Confidence scoring measured how good the docs match was and never looked at whether the answer stayed inside it.

assessConfidence didn't read the response at all — the parameter was _response, unused — and the LLM scorer's rubric actively rewarded specificity: "cites specific features, APIs, or code patterns," with no check that the citations were real. So a confident fabrication scored exactly as high as a cited answer, and #142's prompt rules had nothing enforcing them.

New ai/groundedness.ts does the enforcing. No model call — regexes over the response plus a substring lookup against the sources it was generated from, so it's cheap, deterministic and unit-testable.

• Unverified claims — "bug confirmed", "root cause is", "known issue", "the fix is", "I reproduced/tested this". Any one of these suppresses the response.
• Unsourced identifiers — CopilotKit names the response invents. One is a penalty, two suppress. Scoped deliberately narrow to CSS classes and backticked tokens carrying our own name, so useRef, useLayoutEffect and @copilotkit/react-core never trip it.
• Hedge density — two markers free, then a small penalty each. Never suppresses on its own; hedging is a smell, not a fabrication.

Wired into four places:

• generator.assessConfidence reads the response and deducts the penalty
• pipeline.ts applies the penalty after feedback calibration, so aggregate 👍 can't buy back a fabrication, and clamps a suppressed response below the escalation gate
• confidence.ts rubric now weighs groundedness above specificity
• ai-response.ts withholds a suppressed response and escalates

The behavior change worth reviewing closely

Confidence never gated the post-back. A low score only picked the disclaimer and queued an escalation job — the response posted either way (ai-response.ts:203). So a penalty on its own would have lowered a number and changed nothing about what reached a public thread.

Now a suppressed response never posts as written — the reporter gets the safe replacement copy — and escalation fires regardless of score.

Correction (was wrong in an earlier version of this description): the draft persists as the BOT Message only. suggestedResponse holds formatted.text, which for a suppressed draft is the replacement copy, not the draft — so an agent looking at the ticket's suggestion field sees boilerplate and has to find the draft in the message thread. Message.content (raw draft) and the posted text also diverge under suppression, so anyone auditing bot output from the DB reads the draft as though it were published. Both are tracked in #148, which now also covers stashing the published text alongside it.

That means some tickets get no immediate bot reply where they used to get one. That's the intended trade — silence beats a confident fabrication on a public issue — but it is a real change in what reporters see, so flagging it rather than burying it.

One thing a test caught

The penalty cap alone wasn't sufficient. Top retrieval score plus maximum positive calibration minus the capped penalty lands on exactly 0.4, and the escalation gate is < 0.4 — so the worst-case response would have silently declined to escalate. Hence the explicit clamp to just below the gate, which also keeps the dashboard, the disclaimer and the escalation logic agreeing with each other.

Tests

• groundedness.test.ts — 23 cases: each claim pattern, sourced vs unsourced identifiers, title-as-well-as-body matching, the false-positive guards, penalty cap, empty/partial input, and the verbatim #6167 response as a fixture asserting it gets suppressed
• pipeline.test.ts — penalty deducted from the final score, suppression flagged, and calibration ordering (a boost can't rescue a fabrication)
• ai-response.test.ts — a suppressed response never posts, escalates even with a 0.95 score, and still leaves the draft on the ticket

npm test 840 passing across all 10 packages · npm run typecheck clean · npm run build clean. Lint and prettier are unchanged from #142 — both were already failing on clean main.

[jerelvelarde]

The design here is right, and the part I want to call out first is that you found the real gap: assessConfidence scoring retrieval quality while ignoring the answer meant the guardrails in #142 had nothing behind them. A deterministic regex-plus-substring check with no model call is the correct shape for enforcement — cheap, ordered predictably, and actually testable. Deducting after calibration so aggregate 👍 can't buy back a fabrication is a genuinely subtle call and the comment explaining it is the kind of thing that stops someone reordering it in six months. Using the verbatim #6167 response as a fixture is the right way to pin a regression. And flagging the "some tickets now get no immediate reply" behavior change in the description rather than burying it is exactly right.

One correctness bug that I think needs fixing before this lands, then some smaller things.

The groundedness penalty is applied twice

assessGroundedness is called at two sites and both subtract the penalty:

1. generator.ts:246 — assessConfidence now ends with return Math.max(0, retrievalScore - penalty), so the penalty is baked into generatedResponse.confidenceScore.
2. pipeline.ts:141 — finalConfidenceScore = Math.max(0, finalConfidenceScore - groundedness.penalty), subtracting the same value again.

In between, pipeline.ts:123 does combinedConfidenceScore = Math.min(generatedResponse.confidenceScore, confidenceAssessment.score). Whenever the generator's (already-penalized) score is the lower of the two — which a large penalty makes likely, since it's the thing pushing it down — the penalty is counted twice. When the LLM scorer happens to be lower, it's counted once. So the deduction is both doubled and path-dependent on which signal wins the Math.min.

Worked example on the single-identifier case, penalty 0.15, avg relevance 0.80 over 3 sources:

retrievalScore   = 0.80 + 0.15 bonus      = 0.95
generator score  = 0.95 − 0.15            = 0.80   ← penalty #1
combined         = min(0.80, 0.85)        = 0.80
calibration 0                             = 0.80
pipeline         = 0.80 − 0.15            = 0.65   ← penalty #2

Intended 0.80, actual 0.65. One invented identifier costs 0.30, not the 0.15 the constant declares.

The suppression clamp hides this for anything that trips suppress, so the visible damage is confined to the deliberately-designed middle tier — "one is a penalty, two suppress." That's the band where a single invented identifier can now get pushed under AI_CONFIDENCE.ESCALATE and fire a spurious escalation on a response that was meant to post with a disclaimer.

I'm fairly confident this is unintended rather than deliberate, because the description's own arithmetic assumes single application:

Top retrieval score plus maximum positive calibration minus the capped penalty lands on exactly 0.4, and the escalation gate is < 0.4

That only reaches 0.4 if the penalty is subtracted once (1.0 − 0.6). Applied twice it lands at 0, comfortably below the gate, and the explicit clamp you added wouldn't have been necessary at all.

Why the tests didn't catch it

pipeline.test.ts:43 mocks the whole ResponseGenerator class, and sampleGeneratedResponse.confidenceScore is hardcoded to 0.85. So assessConfidence — penalty site #1 — never executes in any pipeline test. That's what makes the 0.85 − 0.15 = 0.70 comment on the new test read as correct: it is correct against the mock, and unreachable in production. Any regression test for this has to exercise a real ResponseGenerator, or assert at the generator level directly.

Suggested fix

Keep the ordering guarantee you documented — the deduction must land after calibration, otherwise a positive boost can partially offset it — and remove the duplicate:

• assessConfidence goes back to returning the raw retrievalScore and does not deduct.
• The generator computes the assessment once and carries it on GeneratedResponse (e.g. a groundedness field), so pipeline consumes it instead of recomputing.
• pipeline remains the single place that deducts, still after calibration.

That drops the penalty to one application, keeps the post-calibration invariant, and removes the redundant second assessGroundedness call per request as a side effect.

CI has never run the test suite on this PR

.github/workflows/ci.yml triggers on pull_request: branches: [main, staging]. This PR's base is fix/ai-response-epistemic-guardrails, so Lint, Typecheck & Test never fired — only zizmor and notify ran, and GitHub still reports mergeStateStatus: CLEAN. Worth knowing that the green tick here means less than usual. Once #142 is in, retarget this to main and let the full suite run before merging.

Smaller things

Suppression regexes are negation-blind. /\bknown\s+(?:bug|issue|regression)\b/i matches "this is not a known issue"; /\broot\s+cause\s*(?:is\b|:)/i matches "I can't determine what the root cause is without reproducing it"; /\bthe\s+fix\s+is\b/i matches "I don't know what the fix is." Those are all well-behaved responses of exactly the kind #142's prompt asks for, and each now yields silence for the reporter. Since suppression is user-visible absence rather than a lowered number, a negation lookbehind or tightening to assertive forms seems worth it.

The human draft is pre-poisoned with the AI disclaimer. ai-response.ts:172 sets suggestedResponse: pipelineResult.formatted.text, which for a suppressed response carries > ⚠️ This is an AI-generated response. We've escalated this to our engineering team…. reply-editor.tsx:59 prefills the agent's reply box with that string verbatim. So the "a human can edit and send it" path hands them a draft opening by declaring itself AI-generated and announcing an escalation. pipelineResult.response (already persisted raw as Message.content on line 158) is the better source for the editor. Pre-existing, but this PR promotes it from cosmetic to the primary path for suppressed responses. Happy to split it out if you'd rather keep this PR tight.

CSS_CLASS_PATTERN is case-sensitive. /\.(copilotKit[A-Za-z0-9_-]*)/g won't match an invented .copilotkit-input or .CopilotKitInput. Adding i costs nothing given the /copilotkit/i guard already downstream.

Substring matching gives false negatives. haystack.includes(id.toLowerCase()) means an invented copilotKitTextarea counts as grounded if any source happens to mention copilotKitTextareaWrapper. Fine for a cheap heuristic and it errs toward not suppressing, which is the safe direction — just worth a comment so the next reader doesn't assume it's exact.

I checked the obvious escape hatch and it's closed: nothing auto-posts suggestedResponse — reply-editor.tsx is its only consumer and it's human-driven — so the suppression gate can't be bypassed that way.

Requesting changes on the double-penalty. Everything else is take-it-or-leave-it, and this is close.

[jerelvelarde]

#142 is merged, so I've retargeted this to main — the diff is unchanged (+644 across 10 files), which confirms it's cleanly just this PR's own work.

Worth noting why that mattered beyond housekeeping: while the base was fix/ai-response-epistemic-guardrails, ci.yml's pull_request: branches: [main, staging] trigger meant the Lint, Typecheck & Test job never ran on this branch — only zizmor and notify did, and GitHub still reported the PR as CLEAN. So the green tick was covering two jobs, neither of which runs a test. Now that it's on main the full suite should fire; if it doesn't pick up automatically from the base change alone, a no-op push will do it.

The double-penalty item from my review stands regardless — and note the suite as it exists wouldn't catch it even when it runs, since pipeline.test.ts:43 mocks the whole ResponseGenerator, so assessConfidence (the first of the two deduction sites) never executes in any pipeline test.

[NathanTarbert]

Pushed 28fb1fb addressing the review. Thanks for the double-penalty catch — that was a real bug and your read of it was exact.

Double penalty. Fixed as you described: assessConfidence goes back to scoring retrieval quality only, and the generator now passes the assessment through on GeneratedResponse.groundedness for the pipeline to apply once, after calibration. The second assessGroundedness call per request is gone with it.

Your note that the suite couldn't catch this was the more useful half — pipeline.test.ts mocking the whole generator means assessConfidence never executes in any pipeline test, so no assertion I could add there would have failed. New pipeline-groundedness.test.ts runs the real generator against aimock and pins the arithmetic. I confirmed it fails on the pre-fix code (0.675 where 0.80 is expected) and passes after, so it's a genuine regression guard rather than a test written to agree with the new behavior.

Negation-blind regexes. Taken. Claims are now checked per-occurrence against a same-sentence negation lookbehind ([^.!?]{0,80}$-anchored, so "That's confirmed. I have not tested it." still counts). Checking every occurrence rather than the first means a negated mention doesn't excuse an assertive one later in the same response — both directions have tests. Your framing that suppression is user-visible absence rather than a lowered number is what made this worth fixing properly.

Case-sensitive CSS pattern. Fixed, with tests for .copilotkit-input and .CopilotKitInput.

Substring matching. Left as-is, comment added explaining that it errs toward not suppressing and that that's the safe direction.

suggestedResponse prefill. Filed as #145 rather than folded in here, on your offer to split it. Two reasons: the line isn't in this PR's diff, and its severity depends on work that isn't here yet. Under the in-progress always-answers change, a suppressed draft's formatted.text is replaced with the no-answer copy — so the human's editor would prefill with the apology while the draft they need sits only in Message.content. That turns your cosmetic nit into the thing that breaks "a human can edit and send it," so it should land with that PR, where the reasoning is legible.

On CI: retargeting to main did it — the full suite is running on this push. Locally: 854 tests green across packages/outpost, 10/10 packages, typecheck and build clean.

[NathanTarbert]

Pushed four more commits (28fb1fb..89cb3d5) and ran the CR loop over the result — three rounds, 33 reviewer agents, a convergence audit, and five fix agents. CI is green at 89cb3d5.

What changed since your review

Your double-penalty catch was fixed as you described, and then the loop found that the negation logic underneath it was unsalvageable. Two more attempts failed in two more directions — window-scoped, then sentence-scoped — before the design changed:

Suppression is now driven by the objective signal only. Identifiers named in the response but absent from every retrieved source. That is verifiable against the sources; no English parsing. The negation machinery (isNegated, NEGATION_WINDOW, NEGATOR_PATTERN, splitSentences) is deleted outright, not patched.

Claim wording is penalty-only. "Bug confirmed" still charges 0.35, still reports in unverifiedClaims, and now always clamps below the escalation gate — but it never withholds a reply. The consequence of a misread dropped from "reporter gets silence" to "score is lower and a human looks at it." Your negation-blindness finding is what pointed at this: once suppression is user-visible absence, the cost asymmetry decides the design.

The gate moved to the boundary. generateSupportResponse swaps the safe copy into formatted, so every consumer inherits it rather than each one re-implementing the check. The queue handler's suppressed-arm is deleted — which removes the shadow-mode ordering bug by construction rather than reordering it — and apps/web/src/app/api/qa/route.ts (which was streaming result.response and bypassing the gate entirely) now streams the published text.

autoSend is gone. Zero readers repo-wide; it was a public boolean computed from the pre-deduction score, so it read true for exactly the responses the gate withholds. Removed rather than repaired. confidenceLevel now classifies from the penalised value while confidenceScore stays retrieval-only, so the single-deduction invariant holds.

Six test guards repaired, each mutation-proven. Your instinct that the suite could not catch the double-penalty generalised: deleting || suppressed from the handler left 31/31 green, and deleting generatedResponse.groundedness ?? left 37/37 green. Both now fail when their behaviour is removed. The old 0.65 guard could never have caught the bug it was named for — double application yields 0.675.

916 tests, typecheck and build clean.

What I did not fix, and why

Three rounds in, every new bucket-(a) finding was in code I had written during that same cycle — a signal's meaning changed and some reader of the old meaning went unswept. Three instances (autoSend, then streaming + generator level + QA metadata). Patching the third set invites a fourth, so I stopped the loop rather than run a round I expected to produce more of my own cleanup:

• Groundedness gate: streaming path, generator level, and QA metadata still read the pre-#143 signal #146 — streaming path, generator confidenceLevel, and QA metadata still read the pre-change signal. Includes the structural fix worth making: one parameterized test asserting the gate holds on every path and consumer.
• Groundedness identifier signal exempts every name that isn't literally CopilotKit* (useCopilotAction, CopilotChat, …) #147 — the identifier signal exempts every name that is not literally CopilotKit*, so a fabricated useCopilotAction or CopilotChat is invisible. That signal now carries the whole gate, so this is the highest-value follow-up.
• Withheld drafts are indistinguishable from delivered replies (replayed as assistant history, no DB marker) #148 — withheld drafts are indistinguishable from delivered replies: persisted as BOT, replayed as assistant history next run, no DB marker. Relates to Reply editor prefills the AI disclaimer instead of the draft answer #145.

Also confirmed and left alone as out-of-subject: temperature is sent unconditionally while the model is env-overridable, so a bump to Opus 4.7+/Sonnet 5 400s and every ticket silently degrades to the apology fallback. Worth its own issue — it is a live production hazard, not a latent one.

npm run lint cannot run at all in this repo (no eslint.config.*; ESLint 9 flat-config migration outstanding), identically on clean main. Prettier flags several touched files both before and after these commits, so formatting was left alone rather than bundled in.

Ready for another look.

[NathanTarbert]

Correction to my comment above — three of the four remaining reviewers landed after I posted it, and two of my coverage claims were incorrect.

"Discord parts included" is not true. The boundary test loops over formatted.parts asserting no part carries the draft, but the suppressed copy plus disclaimer is ~390 characters against Discord's 2000-char limit, so parts is undefined and the loop body never runs. The per-platform assertion on formatted.text does hold; the multi-part case is unasserted.

The "escalates on suppression at score 0.95" fixture describes an unreachable state. pipeline.ts clamps a suppressed response to below the escalation gate at two separate sites, so suppressed: true with a score of 0.95 cannot occur. The test still proves the handler ORs suppressed into its escalation condition — the mutation check confirms that — but the scenario it dramatises is fictional, and PipelineResult does not encode the invariant that would have made the fixture impossible to write.

And one more guard that isn't. pipeline-groundedness.test.ts — "reports the penalised level for an ungrounded but publishable response": deleting the penalty subtraction from classifyGroundedConfidence leaves all tests green. That test was added in this cycle specifically to pin the grounded-confidenceLevel fix, and it does not.

All three are folded into #146 rather than fixed here, alongside the parameterized-coverage lever. Same root cause as everything else in that issue: I verified the fixes by mutation where I remembered to, and the places I did not are exactly where the coverage turned out to be hollow.

The six commits and the CI-green result stand — what changed is my confidence in three specific assertions, not in the behaviour they were meant to cover. The behaviour is separately verified by probes recorded in the CR ledger.

[jerelvelarde]

Approving. The gate placement is the best decision in here — formatted is safe by construction, so consumers publish it unconditionally and never have to know the gate exists. The QA route switching from result.response to result.formatted.text is that contract paying off immediately. Test coverage is unusually good: the verbatim #6167 fixture, per-platform suppression, calibration ordering, the double-count guard, and the escalate-despite-high-score path are all pinned.

A few things, none of them blocking.

1. Any single claim phrase now forces an escalation — worth quantifying

pipeline.ts clamps below the escalation gate on unverifiedClaims.length > 0. Two of the patterns are ordinary English in a correct, docs-grounded answer:

• /\bthe\s+fix\s+is\b/i — "The fix is to pass the input prop."
• /\bknown\s+(?:bug|issue|regression)\b/i — "This is a known issue, fixed in 1.9.2."

Both are things the bot should be able to say when the docs say them, and each one now creates an ESCALATION job plus the escalated disclaimer regardless of how good the answer is. The design is deliberate and the comment defends it — but the PR quantifies the withholding trade and not this one, and this is the one that hits ops volume every day rather than only on fabrications. Worth running over the last N real responses (or a week of shadow mode) to get a false-positive rate. Cheaper mitigation if it turns out noisy: skip the clamp when the claim's sentence is itself grounded in a source.

2. The description is wrong about suggestedResponse

The draft still persists as a BOT message and as suggestedResponse on the ticket, so a human can edit and send it.

The second half isn't true — ai-response.ts:156 sets suggestedResponse: pipelineResult.formatted.text, which for a suppressed draft is the canned replacement. The handler's own inline comment gets this right ("suggestedResponse holds the publishable text bots pick up"); only the PR body overstates it. Practically: an agent looking at the ticket's suggestion field sees boilerplate, and the draft is only findable in the BOT message thread.

Related follow-up: Message.content is the raw draft while the posted text is formatted.text. Suppression makes those diverge hard — the DB says the bot posted "## Bug Confirmed…" when the public thread got "I couldn't find an answer." Anyone auditing bot output from the DB gets the wrong answer. Worth stashing the published text in attachments.

3. AI_CONFIDENCE.AUTO_RESPOND is now dead

Removing autoSend leaves no src/ consumer, but shared/src/constants.ts:32 still documents it as in use by the generator. Drop it or fix the comment. (The autoSend removal itself is clean — nothing outside ai/src read it.)

4. classifyGroundedConfidence guards a field nobody reads

ResponseGenerator is only ever consumed by AIPipeline, which reads confidenceScore and recomputes the level itself, so GeneratedResponse.confidenceLevel has no production reader. Defensible as public-API hygiene since the class is exported from index.ts, but it's ~25 lines defending a value that never escapes.

5. "Exactly once" is narrower than it reads

True of groundedness.penalty. But confidence.ts now scores groundedness as rubric factor 5, and that enters through min(generator, scorer) before the deterministic deduction — so an ungrounded answer can be charged by two independent mechanisms. Probably fine as defense in depth and MAX_GROUNDEDNESS_PENALTY bounds it; just not the same claim the comments make.

6. Nits

• Two consecutive if blocks in pipeline.ts both do Math.min(finalConfidenceScore, SUPPRESSED_CONFIDENCE_CAP). Collapse them.
• generateStreamingResponse now buffers the whole draft, so TTFT == total latency. No caller in src/ today and the doc comment is honest about it, but a future wiring loses streaming silently.

Checked and clean

• SUPPRESSED_CONFIDENCE_CAP = ESCALATE - 0.01 = 0.39 < 0.4 — clears the gate, no float boundary problem.
• The generator's error path supplies groundedness, so the pipeline's ?? never recomputes over the fallback copy.
• Type-only import breaks the types.ts ↔ groundedness.ts cycle properly.
• Shadow-mode ordering fix is right — no early return ahead of the SHADOW_MODE arm.
• I stress-tested BARE_HOST_PATTERN for catastrophic backtracking, since it has the classic (a(b*a)?\.)+ shape. It's quadratic, not exponential — 6.4KB of adversarial hyphenated input runs in 26ms. Not a ReDoS.

#1 is the one I'd want an answer on before this is enabled in production, even if the answer is just "watching escalation volume in shadow mode first." #2 is a description fix. The rest are follow-ups.

[NathanTarbert]

Pushed 3cf34b3. Thanks for the approval and for #1 in particular — you were right that it was the item with daily cost, and it turned out to be a design bug rather than something to quantify.

1. Claim phrases forcing escalation — fixed rather than measured.

You framed it as "worth running over the last N responses to get a false-positive rate," but both examples you gave are things the bot should be able to say when the docs say them, so I'd have been measuring how often a wrong design misfires. Split the claims by who is being quoted instead:

• Escalates — asserts we verified something: bug confirmed, this is a real bug, root cause is, I reproduced.
• Priced only — reports what the docs say: known issue, the fix is.

GroundednessAssessment gains forcesEscalation, and the pipeline clamps on suppress || forcesEscalation. Publication is untouched — claim wording still never withholds.

Your exact example didn't work on the first attempt. known was in the real-bug pattern's adjective list, so "this is a known issue" matched both categories and escalated anyway — defeating the split for its most common phrasing. Removed, with a named regression test. Verified:

"This is a known issue, fixed in 1.9.2."     escalates: false  penalty: 0.35
"The fix is to pass the `input` prop."       escalates: false  penalty: 0.35
"This is a real bug worth fixing in core."   escalates: true   penalty: 0.35
"Bug Confirmed: cursor resets."              escalates: true   penalty: 0.35
"I reproduced this locally."                 escalates: true   penalty: 0.35

Reverting the category split fails 5 tests.

2. Description was wrong — fixed in the body, not just replied to. Your read is right and the handler's inline comment was the accurate one. Your follow-up is the sharper half, though: Message.content holds the raw draft while the thread got the replacement, so the DB doesn't merely omit that a draft was withheld — it says the bot posted the fabrication. Anyone auditing bot output from the database reads the wrong answer. Added to #148 with your attachments suggestion (no migration needed).

3. AUTO_RESPOND / SUGGEST. Kept the constants, rewrote the doc to say plainly they have no reader, why the action-based scheme was never implemented, and that autoSend was the last reader. Also noted they should be deleted rather than rewired if a genuine auto-post gate is ever built — otherwise someone wires them to something that happens to want a 0.9 cutoff.

4. classifyGroundedConfidence — keeping it, and here's the argument. You're right there's no production reader; AIPipeline recomputes the level. But ResponseGenerator is exported from index.ts, so confidenceLevel is public API, and without those lines it reads HIGH for a response the gate would withhold. That is precisely the bug class autoSend was, and I'd rather spend 25 lines than ship a public field that lies. If we ever stop exporting the generator, this goes with it.

5. "Exactly once" — corrected. You're right that it's narrower than it reads. It's true of the deterministic penalty; confidence.ts scores groundedness as rubric factor 5 and enters through the same min() beforehand. Comment now names that as intentional defense in depth with MAX_GROUNDEDNESS_PENALTY bounding the deterministic half, rather than claiming one mechanism.

6. Nits. Two clamps collapsed. Streaming TTFT left as documented — tracked in #146 along with the fact that the streaming path skips the penalty, the disclaimer, and escalation entirely, which is the more serious half of that trap.

Also, thank you for stress-testing BARE_HOST_PATTERN for backtracking. I wrote that pattern and did not check it, and "quadratic, 26ms on 6.4KB of adversarial input" is exactly the kind of thing I'd have shipped on faith.

One thing I want to flag as more urgent than anything in your list. After this PR, unsourced identifiers are the only signal that withholds a response — and that check only matches names containing the literal string copilotkit. A fabricated useCopilotAction, CopilotChat, or CopilotSidebar is invisible to it, and those are the names all over our docs and therefore the most natural thing for a model to invent. Filed as #147, and it's the next thing I'm working on. Until it lands, the gate covers less than this PR's description implies.