ci: harden LLM output validation

[Benoît Cortier (CBenoit)]

A valid classifier result for PR #1564 was discarded because structured output represented an empty string as the literal text "". That left the PR at risk/unknown and hid the concrete validation failure from maintainers.

This change canonicalizes that exact empty-string artifact in the shared text normalizer used by classifier, protocol-analysis, and reviewer outputs. Required prose still fails closed, while optional empty fields normalize consistently. Classification failures now publish a neutral SHA-bound AI classification check containing the precise validator or prerequisite reason, without opening the automated review gate.

Regression coverage includes the PR #1564 payload, reviewer and protocol text behavior, and neutral diagnostic check publication.

[Copilot]

Pull request overview

Hardens LLM output normalization and improves classification-failure diagnostics.

Changes:

• Canonicalizes literal "" artifacts while rejecting empty required prose.
• Publishes neutral, SHA-bound checks for classification failures.
• Adds regression tests and documentation.

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
.github/pr-automation/write-state.js	Supports configurable check conclusions.
.github/pr-automation/validation.js	Normalizes quoted empty strings.
.github/pr-automation/validate-reviewer.js	Rejects empty finding rationales.
.github/pr-automation/resolve-state.js	Adds neutral failure diagnostics.
.github/pr-automation/automation.test.js	Covers normalization and check publication.
.github/PR_AUTOMATION.md	Documents failure behavior.

Suppressed comments (1)

.github/pr-automation/resolve-state.js:132

• When the classification gate itself is unavailable, labeler.yml:481-484 still invokes this resolver while the classifier and semver jobs are skipped (labeler.yml:247-248,345-346). classifier is therefore "", so this branch publishes invalid classifier object, incorrectly diagnosing a prerequisite outage as malformed model output. Pass the gate status/reason into this resolver and handle it before classifier validation so the advertised diagnostic is precise.

  if (!classifierResult?.ok || classifierResult.value?.head_sha !== expectedSha) {
    const reason = classifierResult?.reason || "classifier output unavailable";
    return failedClassification(expectedSha, deterministic, reason, rateLimit, semverStatus);

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.