Skip to content

[FIX] Treat ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS as retryable - #1165

Merged
canihavesomecoffee merged 1 commit into
masterfrom
fix/gcp-zone-exhausted-with-details
Aug 9, 2026
Merged

[FIX] Treat ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS as retryable#1165
canihavesomecoffee merged 1 commit into
masterfrom
fix/gcp-zone-exhausted-with-details

Conversation

@cfsmp3

@cfsmp3 cfsmp3 commented Aug 9, 2026

Copy link
Copy Markdown
Contributor

Problem

GCP reports zone exhaustion under two codes: ZONE_RESOURCE_POOL_EXHAUSTED and ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS (the latter names the resource that ran out). Only the bare code is listed in GCP_RETRYABLE_ERRORS and GCP_ERROR_MESSAGES, and is_retryable_gcp_error() matches by exact set membership:

error_code = get_gcp_error_code(result)
return error_code in GCP_RETRYABLE_ERRORS

So the _WITH_DETAILS variant is classified as permanent and goes to mark_test_failed(). In practice that is the variant Compute Engine returns for instance inserts, which means the retry path we already have almost never runs.

Two visible consequences:

  1. A PR that happens to hit an exhausted zone is failed permanently instead of staying pending for the next cron run.
  2. The contributor gets the unknown-code fallback — "VM creation failed (ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS). Please contact the administrator." — rather than the message we wrote for exactly this case, which says it will be retried automatically.

Hit on ccextractor PR #2309: both CI - linux and CI - windows failed this way, and closing/reopening the PR just reproduced it. install/ci-vm/installation.md:29 already notes this error is common.

Fix

Map both codes to the same message and mark both retryable.

Nothing else changes. When the error is retryable the test is left pending with no GcpInstance row and no terminal TestProgress, so gcp_instance() re-picks it on the next cron run — exactly the behaviour the bare code already had.

This does not paper over a real failure: RESOURCE_NOT_FOUND, RESOURCE_ALREADY_EXISTS and unknown codes still fail the test as before, and there is a test asserting that.

Testing

Four tests added to TestParseGcpError:

  • _WITH_DETAILS gets the friendly message and does not fall through to "contact the administrator"
  • both zone-exhaustion codes are retryable (subTest over each)
  • a non-transient code (RESOURCE_NOT_FOUND) is still not retryable

Verified the new tests fail on master and pass with the fix:

# with mod_ci/controllers.py reverted
FAILED (failures=2)
AssertionError: False is not true   <- is_retryable_gcp_error(_WITH_DETAILS)

# with the fix
Ran 12 tests ... OK

Full module green: tests.test_ci.test_controllers — 200 tests, OK. isort and pydocstyle clean; mypy's only complaint is the pre-existing missing PyYAML stub at the import line, which CI installs via --install-types.

Out of scope

create_instance() takes a single configured ZONE with no fallback, so a sustained outage in that zone still blocks all runs. Multi-zone failover isn't practical here since the platform and the VMs need to be co-located, so this PR only fixes the classification.

GCP reports zone exhaustion under two codes: ZONE_RESOURCE_POOL_EXHAUSTED and
ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS, the latter naming the resource that
ran out. Only the bare code was listed in GCP_RETRYABLE_ERRORS and
GCP_ERROR_MESSAGES, and is_retryable_gcp_error() matches by exact set
membership, so the _WITH_DETAILS variant fell through to mark_test_failed().

In practice _WITH_DETAILS is the variant Compute Engine returns for instance
inserts, so the retry path we already have almost never ran. A PR that hit an
exhausted zone was failed permanently instead of staying pending for the next
cron run, and the contributor got the unknown-code fallback message
("Please contact the administrator") rather than the one that explains it will
be retried.

Both codes now map to the same message and are both retryable. Nothing else
changes: the test is left pending, has no GcpInstance row and no terminal
TestProgress, so gcp_instance() picks it up on the next run exactly as the bare
code already did.

Seen on ccextractor PR #2309, where CI - linux and CI - windows both failed with
"VM creation failed (ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS). Please contact
the administrator." and reopening the PR just reproduced it.
@sonarqubecloud

sonarqubecloud Bot commented Aug 9, 2026

Copy link
Copy Markdown

@canihavesomecoffee
canihavesomecoffee merged commit c78b71e into master Aug 9, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants