Summary
S3StorageClientSpec intermittently fails 4 tests in the build / amber CI job (step Run amber and common module tests with coverage), scope WorkflowCore / Test / test. It is flaky: the same commit passes on re-run and on other PRs.
Observed:
[info] Tests: succeeded 688, failed 4, canceled 0, ignored 0, pending 1
[error] *** 4 TESTS FAILED ***
[error] Failed tests:
[error] org.apache.texera.service.util.S3StorageClientSpec
[error] (WorkflowCore / Test / test) sbt.TestsFailedException: Tests unsuccessful
Example failing run: https://github.com/apache/texera/actions/runs/30486266380/job/90692899618 (PR #7022, a CI-only change that touches no Scala — so the failure cannot be caused by the diff).
The per-test assertion messages are not in the console log (ScalaTest writes them only to target/test-reports/*.xml, which is uploaded to Codecov but not kept as a downloadable CI artifact), so the exact 4 failing test names are not recoverable from the run log.
When it started
The S3 test coverage was substantially expanded in #6987 (merged 2026-07-29, b291f75), which added heavier/new cases to S3StorageClientSpec — including a concurrent 1001-object upload (common/workflow-core/src/test/scala/org/apache/texera/service/util/S3StorageClientSpec.scala:433) and two ~5 MiB multipart cases (:622, :690). The flake surfaced after this.
Root cause (hypothesis)
Contention on the single shared MinIO container and the single shared S3StorageClient.s3Client (one connection pool) under concurrent load, intermittently exceeding the AWS SDK's default socket/connection timeouts on a resource-constrained runner:
- Four suites mix
S3StorageTestBase and share one MinIO container + one s3Client: S3StorageClientSpec, LargeBinaryManagerSpec, LargeBinaryInputStreamSpec, LargeBinaryOutputStreamSpec (common/workflow-core/src/test/scala/org/apache/texera/service/util/S3StorageTestBase.scala:29).
- ScalaTest runs suites in parallel via its own Distributor thread pool (CI log shows
pool-24-thread-N-ScalaTest-running-...Spec). The Global / concurrentRestrictions += Tags.limit(Tags.Test, 1) in common/workflow-core/build.sbt:35 only limits sbt task concurrency; it does not stop ScalaTest from running suites concurrently within a single unforked test task.
- On top of that,
S3StorageClientSpec.scala:439 fires 1001 uploads through a 16-thread pool at the same shared container.
The combined burst overwhelms MinIO / starves the shared client's connection pool → some operations time out → the owning Future/assert fails (and can cascade to neighbouring sequential tests in the suite while the container recovers). When the runner has enough headroom the whole suite passes, hence the flakiness.
Suggested fix directions
- Stop the MinIO-backed suites from running concurrently (they share one container + one client), e.g. serialize just these suites, and/or
- Reduce the concurrent-upload burst in the >1000-objects test (
:439) to a modest bounded pool.
- Optionally, upload
target/test-reports/*.xml as a CI artifact so future flakes surface the exact failing test + assertion.
Environment
- Job:
build / amber (ubuntu-latest, 17); JDK 17 (Temurin 17.0.19); sbt; MinIO testcontainer minio/minio:RELEASE.2025-02-28T09-55-16Z.
Summary
S3StorageClientSpecintermittently fails 4 tests in thebuild / amberCI job (step Run amber and common module tests with coverage), scopeWorkflowCore / Test / test. It is flaky: the same commit passes on re-run and on other PRs.Observed:
Example failing run: https://github.com/apache/texera/actions/runs/30486266380/job/90692899618 (PR #7022, a CI-only change that touches no Scala — so the failure cannot be caused by the diff).
The per-test assertion messages are not in the console log (ScalaTest writes them only to
target/test-reports/*.xml, which is uploaded to Codecov but not kept as a downloadable CI artifact), so the exact 4 failing test names are not recoverable from the run log.When it started
The S3 test coverage was substantially expanded in #6987 (merged 2026-07-29,
b291f75), which added heavier/new cases toS3StorageClientSpec— including a concurrent 1001-object upload (common/workflow-core/src/test/scala/org/apache/texera/service/util/S3StorageClientSpec.scala:433) and two ~5 MiB multipart cases (:622,:690). The flake surfaced after this.Root cause (hypothesis)
Contention on the single shared MinIO container and the single shared
S3StorageClient.s3Client(one connection pool) under concurrent load, intermittently exceeding the AWS SDK's default socket/connection timeouts on a resource-constrained runner:S3StorageTestBaseand share one MinIO container + ones3Client:S3StorageClientSpec,LargeBinaryManagerSpec,LargeBinaryInputStreamSpec,LargeBinaryOutputStreamSpec(common/workflow-core/src/test/scala/org/apache/texera/service/util/S3StorageTestBase.scala:29).pool-24-thread-N-ScalaTest-running-...Spec). TheGlobal / concurrentRestrictions += Tags.limit(Tags.Test, 1)incommon/workflow-core/build.sbt:35only limits sbt task concurrency; it does not stop ScalaTest from running suites concurrently within a single unforked test task.S3StorageClientSpec.scala:439fires 1001 uploads through a 16-thread pool at the same shared container.The combined burst overwhelms MinIO / starves the shared client's connection pool → some operations time out → the owning
Future/assertfails (and can cascade to neighbouring sequential tests in the suite while the container recovers). When the runner has enough headroom the whole suite passes, hence the flakiness.Suggested fix directions
:439) to a modest bounded pool.target/test-reports/*.xmlas a CI artifact so future flakes surface the exact failing test + assertion.Environment
build / amber (ubuntu-latest, 17); JDK 17 (Temurin 17.0.19); sbt; MinIO testcontainerminio/minio:RELEASE.2025-02-28T09-55-16Z.