[Bug]: MinioContainer wait strategy (log) doesn't work everytime

[ruiarodrigues]

Module

MinIO

Testcontainers version

1.19.3

Using the latest Testcontainers version?

Yes

Host OS

MacOS

Host Arch

M1

Docker version

Client: Docker Engine - Community
 Version:           20.10.22
 API version:       1.41
 Go version:        go1.19.4
 Git commit:        3a2c30b63a
 Built:             Thu Dec 15 15:37:38 2022
 OS/Arch:           darwin/arm64
 Context:           default
 Experimental:      true

Server:
 Engine:
  Version:          20.10.20
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.18.7
  Git commit:       03df974ae9e6c219862907efdd76ec2e77ec930b
  Built:            Wed Oct 19 02:58:31 2022
  OS/Arch:          linux/arm64
  Experimental:     false
 containerd:
  Version:          v1.6.8
  GitCommit:        9cd3357b7fd7218e4aec3eae239db1f68a5a6ec6
 runc:
  Version:          1.1.4
  GitCommit:        5fd4c4d144137e991c4acebb2146ab1483a97925
 docker-init:
  Version:          0.19.0
  GitCommit:

What happened?

I'm using the Minio Test container (image: minio/minio:RELEASE.2023-10-25T06-33-25Z) and the log strategy implemented by the MinIOContainer class doesn't work all the time.

waitingFor(
            Wait
                .forLogMessage(".*Status:         1 Online, 0 Offline..*", 1)
                .withStartupTimeout(Duration.of(60, ChronoUnit.SECONDS))
        );

The test container is declared in the tests as the following (S3Testcontainer is just a wrapper that extends MinIOContainer)

@Container
 private static final S3Testcontainer S3_TESTCONTAINER = new S3Testcontainer();

In the tests @BeforeEach I create a bucket for the tests and some times the creation fails with

com.amazonaws.SdkClientException: Unable to execute HTTP request: Connect to localhost:49625 [localhost/127.0.0.1, localhost/0:0:0:0:0:0:0:1] failed: Connection refused (Connection refused)

	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleRetryableException(AmazonHttpClient.java:1219)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1165)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:814)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:781)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:755)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:715)
	at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:697)
	at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:561)
	at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:541)
	at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5456)
	at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5403)
	at com.amazonaws.services.s3.AmazonS3Client.createBucket(AmazonS3Client.java:1131)
	at com.amazonaws.services.s3.AmazonS3Client.createBucket(AmazonS3Client.java:1059)

If I add a wait after the container is "started" until the "listBuckets" is successful (sometimes more than one request is needed until the container responds) all the tests are green.
It seems that the "wait for" strategy using the logging is not the best approach and can lead to flaky tests.

Below I attached the logs of the container after it's started and while the list buckets request is successful. As one can see, the container waited until the log is written but then the requests still fail for a while.

Relevant log output

Formatting 1st pool, 1 set(s), 1 drives per set.
WARNING: Host local has more than 0 drives of set. A host failure will result in data becoming unavailable.
WARNING: Detected default credentials 'minioadmin:minioadmin', we recommend that you change these values with 'MINIO_ROOT_USER' and 'MINIO_ROOT_PASSWORD' environment variables
MinIO Object Storage Server
Copyright: 2015-2023 MinIO, Inc.
License: GNU AGPLv3 <https://www.gnu.org/licenses/agpl-3.0.html>
Version: RELEASE.2023-10-25T06-33-25Z (go1.21.3 linux/arm64)

Status:         1 Online, 0 Offline. 
S3-API: http://172.17.0.3:9000  http://127.0.0.1:9000 
Console: http://172.17.0.3:9001 http://127.0.0.1:9001 

Documentation: https://min.io/docs/minio/linux/index.html
Warning: The standard parity is set to 0. This can lead to data loss.

Listing buckets
Listing buckets failed: Unable to execute HTTP request: Connect to localhost:49633 [localhost/127.0.0.1, localhost/0:0:0:0:0:0:0:1] failed: Connection refused (Connection refused)
Listing buckets
Listing buckets failed: Unable to execute HTTP request: Connect to localhost:49633 [localhost/127.0.0.1, localhost/0:0:0:0:0:0:0:1] failed: Connection refused (Connection refused)
Listing buckets
Listing buckets failed: Unable to execute HTTP request: Connect to localhost:49633 [localhost/127.0.0.1, localhost/0:0:0:0:0:0:0:1] failed: Connection refused (Connection refused)
Listing buckets
Listing buckets failed: Unable to execute HTTP request: Connect to localhost:49633 [localhost/127.0.0.1, localhost/0:0:0:0:0:0:0:1] failed: Connection refused (Connection refused)
Listing buckets
Listing buckets succeeded

Additional Information

No response

[eddumelendez]

Hi, thanks for the report. Can I ask you to try overriding configure in S3Testcontainer to something like this:

@Override
public void configure() {
    withExposedPorts(MINIO_S3_PORT, MINIO_UI_PORT);

    if (this.userName != null) {
        addEnv("MINIO_ROOT_USER", this.userName);
    } else {
        this.userName = DEFAULT_USER;
    }
    if (this.password != null) {
        addEnv("MINIO_ROOT_PASSWORD", this.password);
    } else {
        this.password = DEFAULT_PASSWORD;
    }

    withCommand("server", "--console-address", ":" + MINIO_UI_PORT, "/data");
    waitingFor(
        Wait
            .forHttp("/minio/health/live")
            .forPort(MINIO_S3_PORT)
            .withStartupTimeout(Duration.of(60, ChronoUnit.SECONDS))
    );
}

The key is switching to HttpWaitStrategy. I guess you are running a good amount of parallel tests and due to resources the container doesn't behaves properly. If this works for you, I can apply this patch after your confirmation.

[ruiarodrigues]

Sorry for the late answer, but I tried it and it works.