Skip to content

feat(workflow-sdk): add pr_url verification check#852

Merged
khaliqgant merged 1 commit into
mainfrom
feat/workflow-pr-url-verification
May 14, 2026
Merged

feat(workflow-sdk): add pr_url verification check#852
khaliqgant merged 1 commit into
mainfrom
feat/workflow-pr-url-verification

Conversation

@khaliqgant
Copy link
Copy Markdown
Member

Summary

Adds a new `'pr_url'` value to the `VerificationCheck` type union. Workflow authors can now gate step completion on the worker leaving behind a GitHub PR URL — closing the loophole that let multiple Track A–F workers of the proactive-runtime M3 swarm post `OWNER_DECISION: COMPLETE` with green tests and a clean process exit but no PR.

Motivation

`resolveOwnerCompletionDecision` already runs `step.verification` before honoring `OWNER_DECISION: COMPLETE` (a failing check throws `failed_verification`). The gap was that no built-in check expressed "this step must publish a PR." Workflow authors could compose it with `type: 'custom'` and a shell command, but the friction is high enough that nobody did — including the M3 workflow, which relied on a free-text brief instructing workers to `gh pr create`. None of the workers did, and evidence-based completion (positive-conclusion text + process-exit=0) waved them through.

API

```ts
// any GitHub PR URL in step output counts
verification: { type: 'pr_url', value: '' }

// require the PR belong to a specific repo
verification: { type: 'pr_url', value: 'AgentWorkforce/relaycast' }
```

Repo qualifier is case-insensitive. PR URLs echoed inside the injected task brief are stripped before scanning (via the existing `stripInjectedTaskEcho` path), so reference URLs in the brief don't satisfy the gate.

Test plan

  • `npx vitest run packages/sdk/src/workflows/tests/verification.test.ts` — 44 tests pass (36 existing + 8 new)
  • Pre-existing failures in `verification-traceback.test.ts` and `verification-custom.test.ts` (5 total) are unchanged on this branch — verified by stashing my diff and re-running

Follow-ups

  • Update the proactive-runtime M3 workflow (and any others whose steps ship PRs) to declare `verification: { type: 'pr_url', value: '/' }` on the implementation steps. The runtime change here is additive — no existing workflow regresses.
  • Consider tightening `judgeOwnerCompletionByEvidence` so file-modification + clean exit alone never imply completion when a workflow declares an artifact expectation. Out of scope for this PR.

🤖 Generated with Claude Code

@khaliqgant khaliqgant requested a review from willwashburn as a code owner May 14, 2026 09:14
@coderabbitai
Copy link
Copy Markdown

coderabbitai Bot commented May 14, 2026

Review Change Stack

Warning

Rate limit exceeded

@khaliqgant has exceeded the limit for the number of commits that can be reviewed per hour. Please wait 25 seconds before requesting another review.

You’ve run out of usage credits. Purchase more in the billing tab.

⌛ How to resolve this issue?

After the wait time has elapsed, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

We recommend that you space out your commits to avoid hitting the rate limit.

🚦 How do rate limits work?

CodeRabbit enforces hourly rate limits for each developer per organization.

Our paid plans have higher rate limits than the trial, open-source and free plans. In all cases, we re-allow further reviews after a brief timeout.

Please see our FAQ for further information.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 199de8fc-6b62-414f-a31c-fe68000eef3a

📥 Commits

Reviewing files that changed from the base of the PR and between f3be8c8 and 99c2356.

📒 Files selected for processing (3)
  • packages/sdk/src/workflows/__tests__/verification.test.ts
  • packages/sdk/src/workflows/verification.ts
  • packages/workflow-types/src/index.ts
📝 Walkthrough

Walkthrough

This PR adds support for a pr_url verification type to the relay workflow verification system, enabling validation that workflow steps produce GitHub PR URLs. The change spans type definitions, PR URL detection logic, comprehensive test coverage, and documentation across multiple files in the SDK and type packages.

Changes

PR URL verification feature

Layer / File(s) Summary
Type definition for PR URL verification
packages/workflow-types/src/index.ts
VerificationCheck interface expands to include 'pr_url' as a valid type, with documentation describing optional <owner>/<repo> value scoping.
PR URL detection and verification logic
packages/sdk/src/workflows/verification.ts
New findPrUrl exported helper uses regex to extract GitHub PR URLs from sanitized output with optional qualifier matching (case-insensitive). runVerification adds a 'pr_url' case that calls findPrUrl and fails with a descriptive error when no matching URL is found.
Tests for PR URL verification
packages/sdk/src/workflows/__tests__/verification.test.ts
Imports findPrUrl and adds test suites validating pr_url verification success, missing-URL failure, qualifier rejection, case-insensitive repo matching, and findPrUrl helper behavior including first-match selection, null handling, and filtering of PR URLs in injected task text.
Documentation of PR URL verification
.agents/skills/writing-agent-relay-workflows/SKILL.md, .claude/skills/writing-agent-relay-workflows/SKILL.md
Updates both agent skill documentation to document the new pr_url verification type, usage guidance for steps that publish changes, <owner>/<repo> scoping behavior, and example configurations.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~10 minutes

Suggested reviewers

  • willwashburn

Poem

🐰 A PR URL hops into view,
Verification gates now welcome this too!
From output we sniff, with qualifier care,
GitHub workflows publish their changes with flair! 🚀

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 33.33% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Title check ✅ Passed The title accurately and concisely captures the main change: adding a new 'pr_url' verification check to the workflow SDK.
Description check ✅ Passed The PR description fully addresses the template requirements: it includes a comprehensive Summary section explaining the feature and motivation, and clearly states that tests were added/updated with a Test Plan noting vitest results. All required template sections are complete.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch feat/workflow-pr-url-verification

Warning

Review ran into problems

🔥 Problems

Git: Failed to clone repository. Please run the @coderabbitai full review command to re-trigger a full review. If the issue persists, set path_filters to include or exclude specific files.


Comment @coderabbitai help to get the list of available commands and usage tips.

@khaliqgant khaliqgant force-pushed the feat/workflow-pr-url-verification branch from f3be8c8 to 042d3b2 Compare May 14, 2026 09:41
Lets workflow authors gate step completion on the worker actually opening
a pull request. Without this, evidence-based completion accepts "tests
pass + clean process exit + files modified" as sufficient — which is how
multiple Track A-F implementation steps of the proactive-runtime M3 swarm
returned OWNER_DECISION: COMPLETE while leaving every PR unopened.

- workflow-types: add 'pr_url' to the VerificationCheck type union with a
  doc-string explaining the `value` semantics
- sdk: scan worker output for https://github.com/<owner>/<repo>/pull/<N>
  URLs, optionally filtered by an `<owner>/<repo>` qualifier; reuses the
  existing stripInjectedTaskEcho path so PR URLs echoed inside the
  injected task brief don't satisfy the gate
- tests: 8 new vitest cases (4 against runVerification + 4 against the
  exported findPrUrl helper) covering happy path, missing URL, wrong
  repo, qualifier case-insensitivity, and injected-task echo guard

Workflow author guidance for using this check (and pairing it with
`createGitHubStep({ action: 'createPR' })` so the URL flows through the
github-primitive instead of raw `gh pr create`) lives in the canonical
skills repo: AgentWorkforce/skills @ feat/pr-url-verification-guidance.
@khaliqgant khaliqgant force-pushed the feat/workflow-pr-url-verification branch from 042d3b2 to 99c2356 Compare May 14, 2026 09:41
@khaliqgant khaliqgant merged commit 9de8d53 into main May 14, 2026
41 of 42 checks passed
@khaliqgant khaliqgant deleted the feat/workflow-pr-url-verification branch May 14, 2026 10:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant