Skip to content

fix(cli): warn instead of failing the gate when the PR comment cannot be posted - #31

Merged
royalpinto007 merged 1 commit into
mainfrom
fix/24-comment-failure-warns
Sep 5, 2026
Merged

fix(cli): warn instead of failing the gate when the PR comment cannot be posted#31
royalpinto007 merged 1 commit into
mainfrom
fix/24-comment-failure-warns

Conversation

@royalpinto007

Copy link
Copy Markdown
Collaborator

Fixes #24

Summary

On fork PRs the github.token is read-only, so the comment POST returns 403. upsertComment threw, the error escaped the top-level handler, and the gate exited 2 even when the comparison found no regression. Fork contributions are exactly where this bites.

Changes

  • cmdCompare: wrap upsertComment in try/catch, log a warning with the reason, keep the comparison verdict.
  • cli/index.ts: guard the module-level main() behind an entry-point check so tests can import cmdCompare.
  • tests/comment-failure.test.ts: stubs fetch to 403 and asserts exit code 0 on a clean comparison with comment enabled.

Test plan

  • npm test (69 passed, 10 files)
  • npm run typecheck
  • npm run lint

@github-actions

github-actions Bot commented Sep 5, 2026

Copy link
Copy Markdown

evalgate: support-agent

PASS - No regressions detected.

Overall score: 94.2% (base) -> 94.2% (head) = 0.0pp

Case Base Head Delta Change
greeting-exact 100.0% 100.0% 0.0pp flat
refund-intent-json 100.0% 100.0% 0.0pp flat
mentions-policy 100.0% 100.0% 0.0pp flat
order-id-format 100.0% 100.0% 0.0pp flat
paraphrase-quality 86.0% 86.0% 0.0pp flat
tone-rubric 100.0% 100.0% 0.0pp flat
judge-helpfulness 73.6% 73.6% 0.0pp flat

All cases unchanged.

tolerance 1.0% - worst case 0.0pp

Posted by evalgate - the build fails when your prompt gets dumber.

@royalpinto007
royalpinto007 merged commit 81b90b3 into main Sep 5, 2026
9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

A failed PR comment fails the gate: fork PRs exit 2 on a passing eval

1 participant