HDDS-16077. Add a Testcontainers module for Ozone - #10966
Open
rich7420 wants to merge 4 commits into
Open
Conversation
Add ozone-testcontainers, a Testcontainers module so JVM integration tests can start a single-node Ozone with S3 Gateway the same way they use the MinIO or LocalStack modules. OzoneContainer wraps the all-in-one image (apache/ozone:2.2.0-all-in-one, from HDDS-14452 / ozone-docker#49) and exposes getS3Endpoint()/getAccessKey()/ getSecretKey()/getRegion(). Its wait strategy waits for the S3 Gateway port and then runs "ozone admin safemode wait" inside the container, so the cluster is ready for S3 operations as soon as start() returns. A smoke test drives create/put/get with the AWS SDK v2 (path-style, static credentials) and is skipped when Docker is unavailable. Notes: - Uses the all-in-one image today; an "ozone local run" based image (HDDS-15087) would be a future improvement, not a blocker. - testcontainers and aws-sdk versions are declared in the module for now. Relates to HDDS-14893.
…ainer OzoneContainer extends GenericContainer (which defines equals) and adds credential/region fields, so spotbugs flags EQ_DOESNT_OVERRIDE_EQUALS. A Testcontainers container is an identity object, not a value object, so equals() is intentionally not overridden; exclude the warning via the module findbugs filter rather than add the spotbugs-annotations dependency.
Contributor
There was a problem hiding this comment.
Pull request overview
Adds a new ozone-testcontainers Maven module that provides a Testcontainers OzoneContainer wrapper for running a single-node Ozone + S3 Gateway (via the all-in-one Docker image) and documents how to use it from JVM integration tests.
Changes:
- Adds
ozone-testcontainersto thehadoop-ozonereactor and introduces the new module with its dependencies. - Implements
OzoneContainer(extendsGenericContainer) with S3 endpoint/credential accessors and a readiness wait strategy (listening port + SCM safemode wait). - Adds a Docker-gated smoke test using AWS SDK v2, plus user-facing README and site documentation.
Reviewed changes
Copilot reviewed 8 out of 8 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| hadoop-ozone/pom.xml | Adds ozone-testcontainers to the hadoop-ozone multi-module build. |
| hadoop-ozone/ozone-testcontainers/pom.xml | New module POM with Testcontainers + AWS SDK v2 test dependencies and SpotBugs config. |
| hadoop-ozone/ozone-testcontainers/src/main/java/org/apache/ozone/testcontainers/OzoneContainer.java | Implements the Testcontainers wrapper and readiness strategy. |
| hadoop-ozone/ozone-testcontainers/src/main/java/org/apache/ozone/testcontainers/package-info.java | Package-level documentation for the new module. |
| hadoop-ozone/ozone-testcontainers/src/test/java/org/apache/ozone/testcontainers/TestOzoneContainer.java | Smoke test that starts the container and performs S3 create/put/get using AWS SDK v2. |
| hadoop-ozone/ozone-testcontainers/README.md | Module README describing purpose, status, and how to run locally. |
| hadoop-ozone/ozone-testcontainers/dev-support/findbugsExcludeFile.xml | SpotBugs suppression for EQ_DOESNT_OVERRIDE_EQUALS on OzoneContainer. |
| hadoop-hdds/docs/content/tools/Testcontainers.md | Adds website documentation for using ozone-testcontainers. |
Suppressed comments (1)
hadoop-ozone/ozone-testcontainers/pom.xml:73
<optional>true</optional>on atest-scoped dependency has no effect on the published artifact and is misleading. Remove it to keep the POM consistent with other Ozone test modules.
<dependency>
<groupId>software.amazon.awssdk</groupId>
<artifactId>apache-client</artifactId>
<scope>test</scope>
<optional>true</optional>
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+64
to
+68
| <dependency> | ||
| <groupId>org.slf4j</groupId> | ||
| <artifactId>slf4j-simple</artifactId> | ||
| <scope>test</scope> | ||
| </dependency> |
Comment on lines
+49
to
+71
| S3Client s3 = S3Client.builder() | ||
| .endpointOverride(URI.create(ozone.getS3Endpoint())) | ||
| .credentialsProvider(StaticCredentialsProvider.create( | ||
| AwsBasicCredentials.create(ozone.getAccessKey(), ozone.getSecretKey()))) | ||
| .region(Region.of(ozone.getRegion())) | ||
| .forcePathStyle(true) | ||
| .build(); | ||
|
|
||
| String bucket = "test-bucket"; | ||
| String key = "hello.txt"; | ||
| String body = "hello ozone"; | ||
|
|
||
| // The container is ready (out of safe mode) after start(), so no retry. | ||
| s3.createBucket(b -> b.bucket(bucket)); | ||
| s3.putObject(b -> b.bucket(bucket).key(key), RequestBody.fromString(body)); | ||
|
|
||
| ResponseBytes<GetObjectResponse> got = | ||
| s3.getObjectAsBytes(b -> b.bucket(bucket).key(key)); | ||
|
|
||
| assertEquals(body, got.asUtf8String()); | ||
| assertTrue(s3.listBuckets().buckets().stream() | ||
| .anyMatch(b -> b.name().equals(bucket))); | ||
| } |
Comment on lines
+47
to
+57
| S3Client s3 = S3Client.builder() | ||
| .endpointOverride(URI.create(ozone.getS3Endpoint())) | ||
| .credentialsProvider(StaticCredentialsProvider.create( | ||
| AwsBasicCredentials.create(ozone.getAccessKey(), ozone.getSecretKey()))) | ||
| .region(Region.of(ozone.getRegion())) | ||
| .forcePathStyle(true) | ||
| .build(); | ||
|
|
||
| s3.createBucket(b -> b.bucket("my-bucket")); | ||
| // ... put / get / list objects | ||
| } |
rich7420
marked this pull request as draft
August 7, 2026 14:21
…imple - Wrap the S3Client in try-with-resources in the test and the docs example so it does not leak HTTP resources/threads. - Remove slf4j-simple: the module already inherits slf4j-reload4j (test scope) from the parent, so adding it produced a second SLF4J binding.
rich7420
marked this pull request as ready for review
August 7, 2026 14:34
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
The goal of this task is to provide a Testcontainers module, so JVM projects can start a single-node Ozone with S3 Gateway in integration tests, the same way they use the MinIO or LocalStack modules. Ozone has no Testcontainers module yet (org.testcontainers is not used anywhere in the repo). Motivated by the container-based test-framework interest (e.g. Trino) raised on HDDS-14893.
Scope=>
New published module ozone-testcontainers under hadoop-ozone.
An OzoneContainer (extends GenericContainer) that runs single-node Ozone with S3 Gateway by wrapping the all-in-one image (apache/ozone:-all-in-one, from HDDS-14452 / ozone-docker#49).
Accessors mirroring the MinIO/LocalStack modules: getS3Endpoint(), getAccessKey(), getSecretKey(), getRegion().
Wait strategy on the S3 Gateway port (9878); a smoke test doing S3 create/put/get with AWS SDK v2 (path-style + static credentials), skipped when Docker is unavailable.
A Note here=>
uses the all-in-one image today; a cleaner
ozone local runbased image (HDDS-15087) would be a future improvement, not a blocker.Scope boundaries (intentional for this incubating, S3-focused first version)=>
S3-only: only the S3 Gateway port (9878) is exposed, so tests reach Ozone through the S3 API, the same shape as the MinIO/LocalStack modules. Reaching OM over RPC (ofs:// / OzoneClient) would need exposing the OM port plus an accessor, and secure/Kerberos mode is not covered yet. Both are reasonable follow-ups rather than blockers for this first version.
What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-16077
How was this patch tested?
https://github.com/rich7420/ozone/actions/runs/31161526340