Cut AWS integration suite runtime and stop credentials expiring mid-run - #342
Merged
Conversation
The AWS cloud integration job took ~59 minutes and started failing its last tests with RequestExpired. The OIDC session defaults to one hour and the credentials reach tox as static environment variables, so botocore cannot refresh them mid-run. Cleanup handlers need working credentials too, so the failing tests also leaked the instances and images they had created. Request a 3 hour OIDC session instead of the 1 hour default. This requires the IAM role's MaxSessionDuration to permit the longer session. Memoise the AWS VM type catalogue per availability zone. EC2 has no server-side paging for instance types, so list() materialises the whole catalogue and pages it client-side, and it previously refetched that catalogue on every call -- one DescribeInstanceTypeOfferings walk plus a DescribeInstanceTypes call per 100 types. That made walking a full listing quadratic in API calls: the 1343 types offered in us-east-1a, paged at the tests' result limit of 5, cost ~4300 calls where 14 suffice. This was the bulk of the runtime; test_vm_types_standard alone took ~55 minutes. Poll Route53 record changes every 5s rather than boto3's 30s default, while keeping the same ~30 minute ceiling. Measured against Route53, changes reached INSYNC inside the first poll interval every time, so a test making four record changes spent 120s asleep for work that finished in seconds.
nuwang
had a problem deploying
to
cloud-integration
July 31, 2026 14:24 — with
GitHub Actions
Failure
nuwang
had a problem deploying
to
cloud-integration
July 31, 2026 20:01 — with
GitHub Actions
Failure
nuwang
had a problem deploying
to
cloud-integration
August 1, 2026 05:46 — with
GitHub Actions
Failure
nuwang
temporarily deployed
to
cloud-integration
August 1, 2026 15:29 — with
GitHub Actions
Inactive
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The AWS cloud integration job took ~59 minutes and started failing its last
three tests with
RequestExpired(run). The
OIDC session defaults to one hour and the credentials reach tox as static
environment variables, so botocore cannot refresh them mid-run. The run before
it took 56 minutes and cleared the deadline by four.
Raising the session lifetime alone would have hidden the real problem, so this
also fixes what made the suite slow. Investigation notes below.
Changes
Request a 3 hour OIDC session instead of the 1 hour default.
Memoise the AWS VM type catalogue per availability zone. EC2 has no
server-side paging for instance types, so
AWSVMTypeService.list()materialises the whole catalogue and pages it client-side — and it previously
refetched that catalogue on every call: one
DescribeInstanceTypeOfferingswalk plus a
DescribeInstanceTypescall per 100 types, ~14 API calls to returnone page. Walking every page was therefore quadratic in API calls.
Measured against
us-east-1a: 1343 instance types, and at the tests'default_result_limitof 5 that is 269 pages × 16 calls ≈ 4300 API calls.End-to-end through the moto-backed provider, a full 255-page walk:
This was the single biggest contributor:
test_vm_types_standardalone took~55 minutes (3289s in the failing run, 3014s before it).
Poll Route53 record changes every 5s instead of boto3's 30s default,
keeping the same ~30 minute ceiling. Record create and delete block on the
resource_record_sets_changedwaiter. Instrumented against real Route53, thepattern was identical every time — poll,
PENDING, sleep 30s, poll,INSYNC:Changes reach INSYNC well inside 30s, so all 120s was poll granularity across
the test's 4 record changes.
Testing
tests/test_aws_vm_types.py(catalogue is fetched once per zone, contentsunchanged, keyed by zone) and
tests/test_aws_dns_waiters.py(drives thereal botocore waiter against a simulated clock, so it asserts how long we
would sleep without sleeping; includes guards that faster polling does not
return early or shrink the ceiling).
Known gaps
Two things this PR does not resolve, recorded so they are not lost:
test_create_and_list_imagetakes ~55 minutes and reproduces outside CI(a local run exceeded 75 minutes). Ruled out:
CreateImage/wait_till_ready(the AMI reaches
availablein seconds) and imagelist/iter(the accountholds one self-owned AMI). It never reaches the launch-from-image step, so
the time is inside
check_standard_behaviour. One contributor is confirmed —AWSImageService.findsearches all public AMIs becausecheck_finddoes notscope by owner, and an unscoped
tag:NameDescribeImagesmeasures 11.4sagainst 0.32s scoped — but that is ~12s, not 45 minutes. Not yet root-caused.
versus 127s solo and 157s under 5 workers locally). Parallel contention was
the hypothesis and it is disproved. The remaining variable is how long
Route53 actually takes to report INSYNC, which is server-side. The polling
fix helps in both regimes but will not by itself close a 20× gap.
Separately, the failed run leaked resources, because cleanup handlers also
need working credentials: instance
i-096aa37d487803314has been running since07:34 on 2026-07-31, plus
ami-07d70bca45ccaf5d1, a snapshot and a volume.These need clearing by hand; the session-lifetime change prevents a recurrence.