Fix flaky e2e remote-logging tests (LocalStack bucket creation)#67976
Merged
Conversation
…pback The remote-logging and XCom object-storage e2e tests intermittently failed with NoSuchBucket because the LocalStack init hook never created the buckets. init-aws.sh runs inside the localstack container, but connected to the docker compose service name (http://localstack:4566), which is not reliably connectable from within the container at the READY-hook stage. When it failed, 'aws s3 mb' exited non-zero and the buckets were never created, so worker PutObject calls returned NoSuchBucket. Connect over loopback (http://localhost:4566) instead, and add a short readiness wait with 'set -euo pipefail' so a transient gateway delay cannot silently leave the buckets uncreated.
potiuk
requested review from
amoghrajesh,
ashb,
gopidesupavan and
jscheffl
as code owners
potiuk
added
the
backport-to-v3-2-test
bugraoz93
approved these changes
Jun 3, 2026
Contributor
Backport successfully created: v3-2-testNote: As of Merging PRs targeted for Airflow 3.X In matter of doubt please ask in #release-management Slack channel.
|
github-actions Bot
pushed a commit
to aws-mwaa/upstream-to-airflow
that referenced
this pull request
Jun 3, 2026
…kets via loopback (apache#67976) The remote-logging and XCom object-storage e2e tests intermittently failed with NoSuchBucket because the LocalStack init hook never created the buckets. init-aws.sh runs inside the localstack container, but connected to the docker compose service name (http://localstack:4566), which is not reliably connectable from within the container at the READY-hook stage. When it failed, 'aws s3 mb' exited non-zero and the buckets were never created, so worker PutObject calls returned NoSuchBucket. Connect over loopback (http://localhost:4566) instead, and add a short readiness wait with 'set -euo pipefail' so a transient gateway delay cannot silently leave the buckets uncreated. (cherry picked from commit 23adbde) Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
bugraoz93
pushed a commit
that referenced
this pull request
Jun 3, 2026
…kets via loopback (#67976) (#67981) The remote-logging and XCom object-storage e2e tests intermittently failed with NoSuchBucket because the LocalStack init hook never created the buckets. init-aws.sh runs inside the localstack container, but connected to the docker compose service name (http://localstack:4566), which is not reliably connectable from within the container at the READY-hook stage. When it failed, 'aws s3 mb' exited non-zero and the buckets were never created, so worker PutObject calls returned NoSuchBucket. Connect over loopback (http://localhost:4566) instead, and add a short readiness wait with 'set -euo pipefail' so a transient gateway delay cannot silently leave the buckets uncreated. (cherry picked from commit 23adbde) Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The remote-logging and XCom object-storage e2e tests (Additional PROD image
tests) intermittently fail with
NoSuchBucketbecause the LocalStack inithook never creates the buckets.
airflow-e2e-tests/scripts/init-aws.shruns as a LocalStack READY init hookinside the localstack container, but it connected to the docker-compose
service name
http://localstack:4566. That endpoint is not reliablyconnectable from within the container at the READY stage, so both
aws s3 mbcalls failed (
Could not connect to the endpoint URL, script exit 255), thetest-airflow-logs/test-xcom-objectstorage-backendbuckets were nevercreated, and worker
PutObjectcalls returnedNoSuchBucket.Fix: connect over loopback (
http://localhost:4566) from inside the container,add a short readiness wait, and
set -euo pipefailso a transient gatewaydelay cannot silently leave the buckets uncreated.
Observed on the v3-2-test branch:
https://github.com/apache/airflow/actions/runs/26889226071/job/79325331114
Was generative AI tooling used to co-author this PR?
Generated-by: Claude Code (Opus 4.8) following the guidelines