Skip to content

Add sample code branch push step to PR verification agent#137

Merged
YunchuWang merged 3 commits intomainfrom
wangbill/add-pr-verification-agent
Mar 5, 2026
Merged

Add sample code branch push step to PR verification agent#137
YunchuWang merged 3 commits intomainfrom
wangbill/add-pr-verification-agent

Conversation

@YunchuWang
Copy link
Member

Summary

This PR updates the PR verification agent to push verification sample code to a dedicated branch after verification passes, making the samples available for future reference.

Changes

  • Added Step 5.5: Push Verification Sample to Branch - After verification passes, the agent now:

    • Creates a branch named verification/pr-<pr-number>
    • Places the sample in examples/verification-samples/pr-<pr-number>-verify.ts
    • Commits and pushes the sample with descriptive commit message
    • Updates the issue comment with a link to the pushed sample
  • Updated Behavioral Rules:

    • Clarified "No source code changes" rule - agent only creates verification samples, not SDK source changes
    • Added "Branch isolation" rule - verification samples are pushed to verification/pr-<N> branches, never to main or feature branches
  • Updated Success Criteria - Added "Verification samples are pushed to verification/pr-<N> branches"

  • Updated Step 7 (Clean Up) - Samples are now pushed to branches instead of being deleted

Benefits

  • Verification samples are preserved for future reference
  • Samples can be reviewed or re-run by other team members
  • Creates an audit trail of verification work
  • Samples remain accessible even after the issue comment may become hard to find

After verification passes, the agent now pushes the verification sample
to a dedicated branch (verification/pr-<N>) for future reference.

Changes:
- Added Step 5.5: Push Verification Sample to Branch
- Updated behavioral rules: clarify no source code changes, add branch isolation rule
- Updated success criteria to include branch pushing
- Updated Step 7: samples are now pushed, not deleted
Copilot AI review requested due to automatic review settings March 5, 2026 21:42
- Add Step 5: Create E2E tests in test/e2e-azuremanaged/ for regression coverage
- Add Step 6: Push verification sample + e2e tests to verification/pr-<N> branch
- Remove duplicate Step 5.5 (old push logic) and consolidate into Step 6
- Update verification report format to include E2E test details and branch link
- Update hard constraints to allow creating verification artifacts and e2e tests
- Renumber steps for clean sequential flow (Steps 5-9)
The agent now only pushes verification samples to branch, without
creating e2e tests in test/e2e-azuremanaged/.
Copy link

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Updates the pr-verification agent instructions so verification work is preserved in-repo by creating regression e2e tests and pushing verification artifacts to a dedicated verification/pr-<N> branch.

Changes:

  • Adds a new step to create Azure Managed e2e tests under test/e2e-azuremanaged/ after verification passes.
  • Adds a step to commit/push verification artifacts (sample + e2e test changes) to a verification/pr-<pr-number> branch and link it from the verification report.
  • Updates the verification report template to include e2e test/branch information.
Comments suppressed due to low confidence (1)

.github/agents/pr-verification.agent.md:404

  • Step 6 says the verification sample should be placed under examples/verification/, but other parts of this same agent doc still reference examples/verification-samples/ for sample storage. This inconsistency will cause the agent to stage/commit the wrong paths. Please pick one directory convention and update all steps (including any older push/cleanup rules) to match it.
|-------|----------|--------|--------|
| <scenario name> | <expected> | <actual> | ✅ PASS / ❌ FAIL |

### Console Output

<details>

Comment on lines 410 to 413

</details>

### Conclusion
Copy link

Copilot AI Mar 5, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Step 6 assumes the verification sample and e2e test edits are already located inside the repo paths it stages (git add examples/verification/ and git add test/e2e-azuremanaged/), but Step 3 says the sample is created in a temporary working directory. Add an explicit instruction to copy/move the generated sample into the chosen examples/... path (and mkdir -p the directory) before staging/committing, otherwise git add will be a no-op for the sample.

Copilot uses AI. Check for mistakes.
Comment on lines 416 to 421
<FAIL: "Verification failed. See details above. The fix may need additional work.">
```

**Important:** The comment must start with `<!-- pr-verification-agent -->` (HTML comment)
so the idempotency check in Step 1 can detect it.

Copy link

Copilot AI Mar 5, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The multi-line git commit -m "..." example relies on embedding newlines inside a single -m argument, which is not consistently copy/paste-friendly across shells. To make the instructions robust, prefer multiple -m flags (one per paragraph) or another well-known approach to supply multi-line commit messages.

Copilot uses AI. Check for mistakes.
Comment on lines +440 to 443
```bash
git checkout main
```

Copy link

Copilot AI Mar 5, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This new flow adds Steps 5–7, but the document still appears to contain an older, differently-numbered push/label/cleanup sequence later on (including a "Step 5.5" push step). That creates conflicting guidance (e.g., which branch to base from, which directories to use, and which steps are authoritative). Please remove or fully update the older sequence so there is a single, consistent set of steps and step numbers.

Copilot uses AI. Check for mistakes.
@YunchuWang YunchuWang merged commit 647e363 into main Mar 5, 2026
16 checks passed
@YunchuWang YunchuWang deleted the wangbill/add-pr-verification-agent branch March 5, 2026 21:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants