ci(e2e): retry final-main hosted runner loss#7392
Conversation
Signed-off-by: Prekshi Vyas <prekshiv@nvidia.com>
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
📝 WalkthroughWalkthroughAdds hosted-runner-loss classification utilities, a boundary validator for a trusted GitHub Actions retry workflow, and a controller that retries eligible E2E main runs after verifying workflow identity, ancestry, complete job evidence, and run freshness. ChangesE2E runner-loss retry
Estimated code review effort: 4 (Complex) | ~60 minutes Sequence Diagram(s)sequenceDiagram
participant E2ERetryWorkflow
participant RetryController
participant GitHubAPI
participant RunnerLossClassifier
E2ERetryWorkflow->>RetryController: provide run ID and attempt
RetryController->>GitHubAPI: fetch run and main ancestry
GitHubAPI-->>RetryController: validated run metadata
RetryController->>GitHubAPI: list complete job pages
GitHubAPI-->>RunnerLossClassifier: workflow jobs
RunnerLossClassifier-->>RetryController: verified runner-loss evidence
RetryController->>GitHubAPI: POST rerun-failed-jobs
Suggested labels: Suggested reviewers: 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Comment |
Code Coverage OverviewLanguages: TypeScript TypeScript / code-coverage/pluginThe overall coverage in commit 408352d in the TypeScript / code-coverage/cliThe overall coverage in commit 408352d in the Show a code coverage summary of the most impacted files.
Updated |
Signed-off-by: Prekshi Vyas <prekshiv@nvidia.com>
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@test/e2e/support/main-e2e-runner-loss-retry.test.ts`:
- Around line 99-137: Update the routes construction in the test setup to
eliminate the if statement checking options.secondPage. Add the optional
second-page githubFetchRoute through a conditional spread in the routes array
literal, preserving the existing route matching and response behavior while
keeping the test body linear.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 87ea051b-5dfe-4a0b-9592-fb516530c45f
📒 Files selected for processing (11)
.github/workflows/e2e-main-runner-loss-retry.yamltest/e2e/README.mdtest/e2e/support/hosted-runner-loss.test.tstest/e2e/support/main-e2e-runner-loss-retry-workflow.test.tstest/e2e/support/main-e2e-runner-loss-retry.test.tstest/helpers/vitest-watch-triggers.tstest/vitest-watch-triggers.test.tstools/e2e/hosted-runner-loss.mtstools/e2e/main-e2e-runner-loss-retry-workflow-boundary.mtstools/e2e/main-e2e-runner-loss-retry.mtstools/e2e/pr-e2e-gate.mts
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
Signed-off-by: Prekshi Vyas <prekshiv@nvidia.com>
senthilr-nv
added
chore
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@tools/e2e/main-e2e-runner-loss-retry.mts`:
- Around line 270-277: Update renderRetrySummary() and its caller
appendJobSummary() to accept the validated original and retry attempt links,
then include only those safe URLs in the generated GITHUB_STEP_SUMMARY content.
Reuse the links already produced by renderRetryLog() rather than stdout-only
text, and update the existing summary test to verify both links are present.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 8d72efda-921f-4128-8409-5a805d8731ce
📒 Files selected for processing (3)
test/e2e/README.mdtest/e2e/support/main-e2e-runner-loss-retry.test.tstools/e2e/main-e2e-runner-loss-retry.mts
🚧 Files skipped from review as they are similar to previous changes (2)
- test/e2e/README.md
- test/e2e/support/main-e2e-runner-loss-retry.test.ts
PR Review Advisor — InformationalAdvisor assessment: Informational / high confidence Model lanes
Nemotron output stays in workflow artifacts and does not change the assessment above. E2E guidanceAdvisory only. E2E / PR Gate selects and runs jobs independently. Recommended E2E: 1 warning · 0 suggestionsWarningsWarnings do not block.
|
Signed-off-by: Carlos Villela <cvillela@nvidia.com>
Signed-off-by: Carlos Villela <cvillela@nvidia.com>
Signed-off-by: Carlos Villela <cvillela@nvidia.com>
cv
left a comment
cv
left a comment
There was a problem hiding this comment.
Approved on exact head beebc05 against base 7e8945a. Product-scope gate passes through accepted v0.0.93 issue #7140, proposed direction item 5. Nine-category security review found no findings: the trusted workflow has least-privilege permissions and pinned actions, receives no repository secrets, validates repository/workflow/event/branch/run/SHA ancestry and complete job evidence, rejects cancelled or mixed ordinary failures, serializes by run, and rereads the attempt before the one allowed mutation. Adversarial, pagination, workflow-boundary, stale-confirmation, and cancelled-run regressions cover the sensitive path. CodeRabbit has no unresolved threads and the advisor second opinion reports 0 blockers/warnings/suggestions. Approval remains contingent on current GitHub checks and exact-SHA E2E authorization/evidence completing successfully.
cv
left a comment
cv
left a comment
There was a problem hiding this comment.
Re-approved on refreshed exact head 1036d73 against base 02cf73d. The merge from current main was conflict-free and adds only the already-merged Quickstart documentation/test ownership changes; the PR-specific diff and prior product-scope/security conclusions are unchanged. Fresh hosted CI and new exact-SHA E2E evidence remain required.
cv
left a comment
cv
left a comment
There was a problem hiding this comment.
Re-approved on refreshed exact head cecdb89 against base 1bb15a0. The #7343 merge was conflict-free; its documentation-receipt workflow/package/test changes are inherited from main and disappear from the three-dot PR diff. The PR-specific file set, product scope, and nine-category security assessment remain unchanged. Fresh hosted CI and new exact-SHA E2E evidence are required.
|
@prekshivyas @cv — the current-state duplication audit confirms that #7392 / Would you like to revive and salvage #7392 on current If you are no longer taking this follow-up, please say so here and I can prepare the narrow current-main successor without duplicating your work. |
5 participants
Summary
Final-main E2E runs currently remain red when GitHub loses hosted runners, even when every failed job has the confirmed stranded-step signature. This change adds a separate trusted controller that reruns failed jobs once for that exact infrastructure-loss case while leaving assertions, timeouts, mixed failures, cancelled runs, and later attempts untouched.
Related Issue
Addresses #7140, proposed direction item 5; this does not close the larger epic. It extends the classification and retry policy delivered by #7168 for #7146 from the PR gate to final-main E2E runs.
Changes
workflow_runcontroller for completed scheduled or full-manualE2E mainruns. A separate controller is required because the subject E2E run cannot rerun itself before its own terminal jobs complete..github/workflows/e2e.yamlor those features' files.Type of Change
Quality Gates
test/e2e/README.mddocuments the maintainer behavior; the required documentation review confirmed that no user-facingdocs/page or changelog entry applies.github.workflow_shais checked out, no repository secrets are exposed, subject provenance is independently queried, mutation tests protect the boundary, and the run is reread immediately before mutation. Maintainer review remains requested through this PR.Documentation Writer Review
docs-updatedtest/e2e/README.mdagainst.github/workflows/e2e-main-runner-loss-retry.yamland the trusted retry classifier after integrating ci(e2e): gate final-main fanout on base-image publication #7373 and feat: require writer receipts for docs-only PRs #7400. It accurately documents first-attempt final-main retry eligibility, exclusions, complete-evidence handling, same-run and tested-SHA behavior, and the removal condition.DGX Station Hardware Evidence
Verification
Signed-off-by:line and every commit appears asVerifiedin GitHubpre-commit,commit-msg, andpre-pushhooks passed, ornpm run check:diffpassed when hooks were skipped or unavailablenpm run test:changedpassed 23 tests.npm run checks,npm run typecheck:cli, project-membership, test-title, and source-shape validation passed.npm run docsbuilds without warnings (doc changes only)Signed-off-by: Prekshi Vyas prekshiv@nvidia.com
Summary by CodeRabbit