Add EvalOps PR lens review workflow

[haasonsaas]

Summary

• Add a scheduled/manual EvalOps PR lens review workflow for open PRs across evalops/platform, evalops/deploy, and evalops/maestro-internal.
• Fan out six parallel lens reviewers per PR with stable commit status contexts under evalops-pr-lens/<lens>.
• Add a meta-review publisher that ranks/dedupes by confidence and only posts PR comments for findings above the configured threshold.
• Wire repository secrets for EVALOPS_PR_LENS_TOKEN and ANTHROPIC_API_KEY from 1Password so the workflow can write cross-repo statuses/comments and use Opus.

Test plan

• ruby -c .github/scripts/evalops-pr-lens-review.rb
• ruby -Itest -e 'ARGV.each { |path| require "./#{path}" }' test/*_test.rb
• ruby -e 'require "yaml"; ARGV.each { |f| YAML.load_file(f); puts "ok #{f}" }' .github/workflows/*.yml .github/workflow-templates/*.yml
• git diff --check
• actionlint .github/workflows/evalops-pr-lens-review.yml
• ruby .github/scripts/evalops-pr-lens-review.rb discover --repos 'evalops/platform,evalops/deploy,evalops/maestro-internal' --matrix-output /tmp/pr-lens-matrix.json --targets-output /tmp/pr-lens-targets.json

Credential setup

• Installed EVALOPS_PR_LENS_TOKEN from op://DevTools/GitHub Automation/credential.
• Installed ANTHROPIC_API_KEY from op://EvalOps Automation/Anthropic/credential.

[cursor]

PR Summary

Medium Risk
Adds a scheduled GitHub Actions workflow that uses cross-repo tokens to post commit statuses and PR comments, plus calls external LLM APIs; failures or misconfiguration could spam PRs or leak metadata to providers.

Overview
Introduces an automated “PR lens review” pipeline that sweeps open PRs in evalops/platform, evalops/deploy, and evalops/maestro-internal on a schedule (or via manual dispatch), fans out one LLM-backed review job per lens, and publishes stable commit statuses under evalops-pr-lens/<lens>.

Adds a Ruby helper (.github/scripts/evalops-pr-lens-review.rb) to discover PRs, fetch diffs/file metadata, call Anthropic/OpenAI, normalize/score findings, and a meta-review step that dedupes/ranks high-confidence findings, upserts a single marker comment, and sets evalops-pr-lens/meta-review to success/failure based on priority.

Documents required secrets and behavior in README.md, and adds Minitest coverage for PR filtering, matrix/status context generation, finding normalization, ranking, and comment formatting.

^{Reviewed by Cursor Bugbot for commit 99f86c2. Bugbot is set up for automated code reviews on this repo. Configure here.}

[cursor]

Cursor Bugbot has reviewed your changes and found 2 potential issues.

^{Bugbot Autofix is ON, but it could not run because the branch was deleted or merged before autofix could start.}

^{Reviewed by Cursor Bugbot for commit 99f86c2. Configure here.}

[cursor]

Pending and success statuses target different SHAs

Medium Severity

The "Mark lens pending" step posts the pending status to matrix.head_sha (from the discover job), but the "Complete lens status" step posts the success status to steps.refs.outputs.head_sha (fetched live from the API). If the PR is force-pushed between discovery and lens review, these SHAs differ — leaving a stale "pending" status on the old commit that never resolves. The error step correctly uses matrix.head_sha, making the inconsistency with the success step more apparent.

Additional Locations (1)

• .github/workflows/evalops-pr-lens-review.yml#L136-L137

 

^{Reviewed by Cursor Bugbot for commit 99f86c2. Configure here.}