fix(prow-job-executor): retry EV2 gating once for pre-test infra precondition failures by raelga · Pull Request #324 · Azure/ARO-Tools

Rael Garcia (raelga) · 2026-08-28T07:01:31Z

AROSLSRE-1926

Why

AROSLSRE-1721 added an EV2 gating auto-retry for known-issue test failures, driven by prow-job-executor reading finished.json metadata that the aro-hcp-tests binary writes once it runs. That mechanism can't help when a gating job fails in a shared multi-stage pre-step (e.g. aro-hcp-lease-acquire) before the test binary ever runs, since none of the job's finished.json files ever carry the ev2-failed-tests key at all in that case.

Observed case: a stg e2e gating job failed in aro-hcp-lease-acquire with no candidate pool for environment "stg" yielded an immediate lease for 1h0m0s across 41 full pass(es) (e2e slot pool capacity exhaustion).

What

jobAllowsEV2Retry now reports a new InfraPreconditionEligible class (distinct from the existing KnownIssueEligible) when no step's finished.json ever carried ev2-failed-tests at all. ExecuteAndWait wires this to a new EV2InfraRetryableError / ev2-retryable-infra-precondition-failure marker, alongside the existing EV2RetryableError/EV2RetryableMarker. Same bounded single retry as before - EV2's pipeline-level automatedRetry.maximumRetryCount stays at 1, prow-job-executor never resubmits jobs itself.

Follow-up: a companion sdp-pipelines PR is needed to bump the prowjobexecutor dependency, add the new marker to e2e-pipeline.yaml's automatedRetry.errorContainsAny, and fix the AROSLSRE-1764 gap so automatedRetry takes effect in stg/int (not just prod) - tracked under the same Jira story.

Testing

go build ./... and go vet ./... for tools/prow-job-executor
go test ./... for tools/prow-job-executor/prowjob - all passing, including new TestExecuteAndWaitFailsWithInfraMarkerWhenNoStepRanTests and updated TestJobAllowsEV2RetryNoStepCarriesMetadata
golangci-lint run on the module - 0 issues

…ondition failures AROSLSRE-1721's finished.json-driven retry only covers per-test known-issue failures, since it depends on metadata the aro-hcp-tests binary writes once it runs. A gating job that fails in a shared multi-stage pre-step (e.g. aro-hcp-lease-acquire exhausting the e2e slot pool) never gets that far, so none of its finished.json files ever carry ev2-failed-tests at all. jobAllowsEV2Retry now reports this shape as a new InfraPreconditionEligible class, distinct from KnownIssueEligible, wired to a new EV2InfraRetryableError / ev2-retryable-infra-precondition-failure marker. Same bounded single retry as the existing mechanism (EV2's automatedRetry.maximumRetryCount stays at 1). AROSLSRE-1926

Copilot

Pull request overview

This PR extends prow-job-executor’s EV2 gating auto-retry signaling to cover “pre-test infra precondition” failures (where aro-hcp-tests never ran and therefore no finished.json ever contains ev2-failed-tests), in addition to the existing known-issue test-failure retry path.

Changes:

Introduces a RetryEligibility classification (NotEligible / KnownIssueEligible / InfraPreconditionEligible) returned by jobAllowsEV2Retry.
Adds a distinct infra retry marker + error type (EV2InfraRetryableMarker / EV2InfraRetryableError) and wires ExecuteAndWait to emit the correct marker based on eligibility.
Updates and adds unit tests to validate the new eligibility and infra-marker behavior.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.

File	Description
tools/prow-job-executor/prowjob/retrymarker.go	Adds `RetryEligibility` and updates `jobAllowsEV2Retry` to distinguish infra-precondition eligibility when no step reports test metadata.
tools/prow-job-executor/prowjob/retrymarker_test.go	Updates tests for the new return type and adds coverage for the “no step carries metadata” eligibility case.
tools/prow-job-executor/prowjob/monitor.go	Adds infra retry marker/error type and updates `ExecuteAndWait` to return the appropriate retryable error.
tools/prow-job-executor/prowjob/monitor_test.go	Adds a new test validating infra-marker behavior and updates existing tests for the new eligibility type.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

…bility - add RetryEligibility.String() so logs/test failures show readable names instead of a bare integer - clarify the NotEligible doc comment: it means a step reported results but they weren't eligible (or an error occurred), distinct from InfraPreconditionEligible where no step ever reported results at all

Rael Garcia (raelga) · 2026-08-29T10:26:39Z

Pushed a follow-up commit (d4f2353) addressing the two Copilot review comments on retrymarker.go:

added RetryEligibility.String() so log/test output shows readable names instead of a bare int
clarified the NotEligible doc comment to distinguish it from the new InfraPreconditionEligible case

go build/go vet/go test ./... and golangci-lint run all pass on tools/prow-job-executor.

Copilot

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.

…eview Describes the actual precondition (candidate finished.json carried ev2FailedTestsKey) instead of a nonexistent found parameter.

Rael Garcia (raelga) · 2026-08-29T10:30:48Z

Pushed 77eed39 addressing the latest Copilot comment: reworded the retryEligibilityFrom doc comment to describe its real precondition instead of a nonexistent found parameter. Build/vet/test/lint all pass.

Copilot

Pull request overview

Copilot reviewed 5 out of 5 changed files in this pull request and generated 2 comments.

Not related to this PR's scope - was a local helper for monitoring CI, added by mistake via a broad git add.

Rael Garcia (raelga) · 2026-08-29T10:34:30Z

Pushed dd19287: removed watch-324.sh, a local CI-watching helper that got accidentally committed via a broad git add -A on my end - not part of this PR's scope. Nothing else changed in this push.

Copilot

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated no new comments.

Suppressed comments (3)

Previously missed (1) — in code that hasn't changed since the last review.

tools/prow-job-executor/prowjob/monitor.go:72

This marker comment says the failure happened "before the aro-hcp-tests binary ever ran", but InfraPreconditionEligible also covers cases where aro-hcp-tests may have started but crashed before writing metadata. Reword to match the actual condition (no step's finished.json carried the ev2-failed-tests key).

This issue also appears in the following locations of the same file:

line 76
line 236

// EV2InfraRetryableMarker is the fixed, matchable substring prow-job-executor emits when a
// gating job failed in a shared multi-stage pre-step (e.g. aro-hcp-lease-acquire exhausting
// the e2e slot pool) before the aro-hcp-tests binary ever ran, and so is eligible for the
// same bounded, single automatic EV2 retry as a KnownIssueEligible failure - see
// InfraPreconditionEligible. Kept distinct from EV2RetryableMarker (rather than reusing it)

tools/prow-job-executor/prowjob/monitor.go:79

This comment says the failure happened "before any test could run", but the code classifies InfraPreconditionEligible based on absence of aro-hcp-tests metadata (ev2-failed-tests), which can also happen if aro-hcp-tests crashed before writing metadata. Reword to avoid overstating the cause.

// EV2InfraRetryableError wraps a job failure where no step's finished.json ever reported
// aro-hcp-tests results at all, meaning the failure happened before any test could run.
// Its Error() text always contains EV2InfraRetryableMarker, so EV2's
// automatedRetry.errorContainsAny can match on it and re-run the gating step.

tools/prow-job-executor/prowjob/monitor.go:237

ExecuteAndWait's comment says EV2InfraRetryableError is used when "no step ever ran the aro-hcp-tests suite at all", but the eligibility condition is actually that no step ever reported aro-hcp-tests results (ev2-failed-tests key absent), which can also include a crash before metadata was written. Consider rewording to avoid overstating the cause.

// EV2InfraRetryableError when no step ever ran the aro-hcp-tests suite at all (a pre-test
// infra failure, e.g. lease-acquire capacity exhaustion). The EV2 gating step's

Giulio Frasca (gmfrasca)

/lgtm

openshift-ci · 2026-09-01T00:29:56Z

Giulio Frasca (@gmfrasca): changing LGTM is restricted to collaborators

Details

In response to this:

/lgtm

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

…; fix stg/int retry gap Adds the new ev2-retryable-infra-precondition-failure marker that prow-job-executor emits (Azure/ARO-Tools#324, AROSLSRE-1926) alongside the existing known-issue marker (AROSLSRE-1721) to the regionalGating step's automatedRetry.errorContainsAny. Also updates the inline comment: sdp-pipelines' generator now honors a step's custom errorContainsAny patterns even in the stg/int fail-fast path (useExclusiveLocks), so these markers take effect there too, not just in prod (AROSLSRE-1764).

Venkatesh S (venkateshsredhat) · 2026-09-01T10:59:04Z

/lgtm

openshift-ci · 2026-09-01T10:59:07Z

Venkatesh S (@venkateshsredhat): changing LGTM is restricted to collaborators

Details

In response to this:

/lgtm

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

Ashish (ashishmax31) · 2026-09-01T11:00:02Z

/lgtm

openshift-ci · 2026-09-01T11:00:05Z

Ashish (@ashishmax31): changing LGTM is restricted to collaborators

Details

In response to this:

/lgtm

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

Jan-Hendrik Boll (janboll) · 2026-09-01T12:44:00Z

/lgtm
/approve

openshift-ci · 2026-09-01T12:44:07Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: ashishmax31, gmfrasca, janboll, raelga, venkateshsredhat

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [janboll,raelga]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

…arker (AROSLSRE-1926) Picks up Azure/ARO-Tools#324, which adds a new InfraPreconditionEligible retry class (and EV2InfraRetryableError/ev2-retryable-infra-precondition-failure marker) to prow-job-executor for gating jobs that fail in a shared pre-test step (e.g. aro-hcp-lease-acquire) before the test binary ever runs, so none of the job's finished.json files carry the ev2-failed-tests key. Without this bump, test/cmd/prow-job-executor keeps running the older logic and never emits the new marker.

…; fix stg/int retry gap Adds the new ev2-retryable-infra-precondition-failure marker that prow-job-executor emits (Azure/ARO-Tools#324, AROSLSRE-1926) alongside the existing known-issue marker (AROSLSRE-1721) to the regionalGating step's automatedRetry.errorContainsAny. Also updates the inline comment: sdp-pipelines' generator now honors a step's custom errorContainsAny patterns even in the stg/int fail-fast path (useExclusiveLocks), so these markers take effect there too, not just in prod (AROSLSRE-1764).

…arker (AROSLSRE-1926) Picks up Azure/ARO-Tools#324, which adds a new InfraPreconditionEligible retry class (and EV2InfraRetryableError/ev2-retryable-infra-precondition-failure marker) to prow-job-executor for gating jobs that fail in a shared pre-test step (e.g. aro-hcp-lease-acquire) before the test binary ever runs, so none of the job's finished.json files carry the ev2-failed-tests key. Without this bump, test/cmd/prow-job-executor keeps running the older logic and never emits the new marker.

Copilot AI lite review requested due to automatic review settings August 28, 2026 07:01

openshift-ci Bot requested review from Jan-Hendrik Boll (janboll) and Rachel Weber (rachelvweber) August 28, 2026 07:01

openshift-ci Bot added the approved label Aug 28, 2026

Copilot started reviewing on behalf of Rael Garcia (raelga) August 28, 2026 07:02 View session

Copilot AI reviewed Aug 28, 2026

View reviewed changes

Comment thread tools/prow-job-executor/prowjob/retrymarker.go

Comment thread tools/prow-job-executor/prowjob/retrymarker.go Outdated

Copilot AI review requested due to automatic review settings August 29, 2026 10:26

Copilot started reviewing on behalf of Rael Garcia (raelga) August 29, 2026 10:26 View session

Copilot AI reviewed Aug 29, 2026

View reviewed changes

Comment thread tools/prow-job-executor/prowjob/retrymarker.go Outdated

Copilot started work on behalf of Rael Garcia (raelga) August 29, 2026 10:28 View session

Copilot finished work on behalf of Rael Garcia (raelga) August 29, 2026 10:29

fix(prow-job-executor): reword retryEligibilityFrom doc comment per r…

77eed39

…eview Describes the actual precondition (candidate finished.json carried ev2FailedTestsKey) instead of a nonexistent found parameter.

Copilot AI review requested due to automatic review settings August 29, 2026 10:30

Copilot started reviewing on behalf of Rael Garcia (raelga) August 29, 2026 10:31 View session

Copilot AI reviewed Aug 29, 2026

View reviewed changes

Comment thread watch-324.sh Outdated

Comment thread watch-324.sh Outdated

chore: remove local watch script accidentally committed

dd19287

Not related to this PR's scope - was a local helper for monitoring CI, added by mistake via a broad git add.

Copilot AI review requested due to automatic review settings August 29, 2026 10:34

Copilot started reviewing on behalf of Rael Garcia (raelga) August 29, 2026 10:34 View session

Copilot AI reviewed Aug 29, 2026

View reviewed changes

Copilot started work on behalf of Rael Garcia (raelga) August 29, 2026 10:38 View session

Copilot finished work on behalf of Rael Garcia (raelga) August 29, 2026 10:38

Copilot finished work on behalf of Rael Garcia (raelga) August 29, 2026 10:41

Rael Garcia (raelga) mentioned this pull request Aug 29, 2026

fix(e2e): honor infra-precondition EV2 retry marker in regionalGating Azure/ARO-HCP#6738

Merged

Giulio Frasca (gmfrasca) reviewed Sep 1, 2026

View reviewed changes

Ashish (ashishmax31) approved these changes Sep 1, 2026

View reviewed changes

openshift-ci Bot assigned Jan-Hendrik Boll (janboll) Sep 1, 2026

openshift-ci Bot added the lgtm label Sep 1, 2026

openshift-merge-bot Bot merged commit 616b5a4 into main Sep 1, 2026
5 checks passed

openshift-merge-bot Bot deleted the raelg/aroslsre-ev2-infra-retry branch September 1, 2026 13:10

Rael Garcia (raelga) mentioned this pull request Sep 1, 2026

chore(deps): bump ARO-Tools to pick up EV2 infra-precondition retry marker Azure/ARO-HCP#6768

Merged

Rael Garcia (raelga) mentioned this pull request Sep 1, 2026

feat(prow-job-executor): abort gating E2E job on rollout cancellation (AROSLSRE-1339) #259

Open

Uh oh!

Conversation

Rael Garcia (raelga) commented Aug 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why

What

Testing

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Rael Garcia (raelga) commented Aug 29, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Rael Garcia (raelga) commented Aug 29, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Rael Garcia (raelga) commented Aug 29, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Giulio Frasca (gmfrasca) left a comment

Choose a reason for hiding this comment

Uh oh!

openshift-ci Bot commented Sep 1, 2026

Uh oh!

Venkatesh S (venkateshsredhat) commented Sep 1, 2026

Uh oh!

openshift-ci Bot commented Sep 1, 2026

Uh oh!

Ashish (ashishmax31) commented Sep 1, 2026

Uh oh!

openshift-ci Bot commented Sep 1, 2026

Uh oh!

Jan-Hendrik Boll (janboll) commented Sep 1, 2026

Uh oh!

openshift-ci Bot commented Sep 1, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Rael Garcia (raelga) commented Aug 28, 2026 •

edited

Loading