[FIX] Treat ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS as retryable - #1165
Merged
Merged
Conversation
GCP reports zone exhaustion under two codes: ZONE_RESOURCE_POOL_EXHAUSTED and
ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS, the latter naming the resource that
ran out. Only the bare code was listed in GCP_RETRYABLE_ERRORS and
GCP_ERROR_MESSAGES, and is_retryable_gcp_error() matches by exact set
membership, so the _WITH_DETAILS variant fell through to mark_test_failed().
In practice _WITH_DETAILS is the variant Compute Engine returns for instance
inserts, so the retry path we already have almost never ran. A PR that hit an
exhausted zone was failed permanently instead of staying pending for the next
cron run, and the contributor got the unknown-code fallback message
("Please contact the administrator") rather than the one that explains it will
be retried.
Both codes now map to the same message and are both retryable. Nothing else
changes: the test is left pending, has no GcpInstance row and no terminal
TestProgress, so gcp_instance() picks it up on the next run exactly as the bare
code already did.
Seen on ccextractor PR #2309, where CI - linux and CI - windows both failed with
"VM creation failed (ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS). Please contact
the administrator." and reopening the PR just reproduced it.
cfsmp3
requested review from
canihavesomecoffee and
thealphadollar
as code owners
August 9, 2026 04:41
|
canihavesomecoffee
approved these changes
Aug 9, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.



Problem
GCP reports zone exhaustion under two codes:
ZONE_RESOURCE_POOL_EXHAUSTEDandZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS(the latter names the resource that ran out). Only the bare code is listed inGCP_RETRYABLE_ERRORSandGCP_ERROR_MESSAGES, andis_retryable_gcp_error()matches by exact set membership:So the
_WITH_DETAILSvariant is classified as permanent and goes tomark_test_failed(). In practice that is the variant Compute Engine returns for instance inserts, which means the retry path we already have almost never runs.Two visible consequences:
Hit on ccextractor PR #2309: both
CI - linuxandCI - windowsfailed this way, and closing/reopening the PR just reproduced it.install/ci-vm/installation.md:29already notes this error is common.Fix
Map both codes to the same message and mark both retryable.
Nothing else changes. When the error is retryable the test is left pending with no
GcpInstancerow and no terminalTestProgress, sogcp_instance()re-picks it on the next cron run — exactly the behaviour the bare code already had.This does not paper over a real failure:
RESOURCE_NOT_FOUND,RESOURCE_ALREADY_EXISTSand unknown codes still fail the test as before, and there is a test asserting that.Testing
Four tests added to
TestParseGcpError:_WITH_DETAILSgets the friendly message and does not fall through to "contact the administrator"RESOURCE_NOT_FOUND) is still not retryableVerified the new tests fail on master and pass with the fix:
Full module green:
tests.test_ci.test_controllers— 200 tests, OK.isortandpydocstyleclean;mypy's only complaint is the pre-existing missing PyYAML stub at the import line, which CI installs via--install-types.Out of scope
create_instance()takes a single configuredZONEwith no fallback, so a sustained outage in that zone still blocks all runs. Multi-zone failover isn't practical here since the platform and the VMs need to be co-located, so this PR only fixes the classification.