Add sample code branch push step to PR verification agent#137
Add sample code branch push step to PR verification agent#137YunchuWang merged 3 commits intomainfrom
Conversation
After verification passes, the agent now pushes the verification sample to a dedicated branch (verification/pr-<N>) for future reference. Changes: - Added Step 5.5: Push Verification Sample to Branch - Updated behavioral rules: clarify no source code changes, add branch isolation rule - Updated success criteria to include branch pushing - Updated Step 7: samples are now pushed, not deleted
- Add Step 5: Create E2E tests in test/e2e-azuremanaged/ for regression coverage - Add Step 6: Push verification sample + e2e tests to verification/pr-<N> branch - Remove duplicate Step 5.5 (old push logic) and consolidate into Step 6 - Update verification report format to include E2E test details and branch link - Update hard constraints to allow creating verification artifacts and e2e tests - Renumber steps for clean sequential flow (Steps 5-9)
The agent now only pushes verification samples to branch, without creating e2e tests in test/e2e-azuremanaged/.
There was a problem hiding this comment.
Pull request overview
Updates the pr-verification agent instructions so verification work is preserved in-repo by creating regression e2e tests and pushing verification artifacts to a dedicated verification/pr-<N> branch.
Changes:
- Adds a new step to create Azure Managed e2e tests under
test/e2e-azuremanaged/after verification passes. - Adds a step to commit/push verification artifacts (sample + e2e test changes) to a
verification/pr-<pr-number>branch and link it from the verification report. - Updates the verification report template to include e2e test/branch information.
Comments suppressed due to low confidence (1)
.github/agents/pr-verification.agent.md:404
- Step 6 says the verification sample should be placed under
examples/verification/, but other parts of this same agent doc still referenceexamples/verification-samples/for sample storage. This inconsistency will cause the agent to stage/commit the wrong paths. Please pick one directory convention and update all steps (including any older push/cleanup rules) to match it.
|-------|----------|--------|--------|
| <scenario name> | <expected> | <actual> | ✅ PASS / ❌ FAIL |
### Console Output
<details>
|
|
||
| </details> | ||
|
|
||
| ### Conclusion |
There was a problem hiding this comment.
Step 6 assumes the verification sample and e2e test edits are already located inside the repo paths it stages (git add examples/verification/ and git add test/e2e-azuremanaged/), but Step 3 says the sample is created in a temporary working directory. Add an explicit instruction to copy/move the generated sample into the chosen examples/... path (and mkdir -p the directory) before staging/committing, otherwise git add will be a no-op for the sample.
| <FAIL: "Verification failed. See details above. The fix may need additional work."> | ||
| ``` | ||
|
|
||
| **Important:** The comment must start with `<!-- pr-verification-agent -->` (HTML comment) | ||
| so the idempotency check in Step 1 can detect it. | ||
|
|
There was a problem hiding this comment.
The multi-line git commit -m "..." example relies on embedding newlines inside a single -m argument, which is not consistently copy/paste-friendly across shells. To make the instructions robust, prefer multiple -m flags (one per paragraph) or another well-known approach to supply multi-line commit messages.
| ```bash | ||
| git checkout main | ||
| ``` | ||
|
|
There was a problem hiding this comment.
This new flow adds Steps 5–7, but the document still appears to contain an older, differently-numbered push/label/cleanup sequence later on (including a "Step 5.5" push step). That creates conflicting guidance (e.g., which branch to base from, which directories to use, and which steps are authoritative). Please remove or fully update the older sequence so there is a single, consistent set of steps and step numbers.
Summary
This PR updates the PR verification agent to push verification sample code to a dedicated branch after verification passes, making the samples available for future reference.
Changes
Added Step 5.5: Push Verification Sample to Branch - After verification passes, the agent now:
verification/pr-<pr-number>examples/verification-samples/pr-<pr-number>-verify.tsUpdated Behavioral Rules:
verification/pr-<N>branches, never tomainor feature branchesUpdated Success Criteria - Added "Verification samples are pushed to
verification/pr-<N>branches"Updated Step 7 (Clean Up) - Samples are now pushed to branches instead of being deleted
Benefits