After upgrade to 1.16.1 JUnit tests fail on AWS CodeBuild

[catchin]

When upgrading to 1.16.1, tests using TestContainers start to fail when executed on AWS CodeBuild. Locally they still run successfully. With 1.16.0 they still run successfully.

Here is the stack trace:

Creating container for image: amazon/dynamodb-local:1.15.0
Starting container with ID: cd71e434635d54dc183620bc887d2413bd53184031832709a61fa4c0de4d20f3
Container amazon/dynamodb-local:1.15.0 is starting: cd71e434635d54dc183620bc887d2413bd53184031832709a61fa4c0de4d20f3
Could not start container

org.testcontainers.containers.ContainerLaunchException: Timed out waiting for container port to open (172.18.0.1 ports: [32768] should be listening)
at org.testcontainers.containers.wait.strategy.HostPortWaitStrategy.waitUntilReady(HostPortWaitStrategy.java:89)
at org.testcontainers.containers.wait.strategy.AbstractWaitStrategy.waitUntilReady(AbstractWaitStrategy.java:51)
at org.testcontainers.containers.GenericContainer.waitUntilContainerStarted(GenericContainer.java:929)
at org.testcontainers.containers.GenericContainer.tryStart(GenericContainer.java:468)
at org.testcontainers.containers.GenericContainer.lambda$doStart$0(GenericContainer.java:331)
at org.rnorth.ducttape.unreliables.Unreliables.retryUntilSuccess(Unreliables.java:81)
at org.testcontainers.containers.GenericContainer.doStart(GenericContainer.java:329)
at org.testcontainers.containers.GenericContainer.start(GenericContainer.java:317)

As a workaround, I added the environment variable TESTCONTAINERS_HOST_OVERRIDE=127.0.0.1. Then the tests pass successfully again.
So it seems that an "external" IP is used for the docker host instead of localhost.

Do you know what changed with detection of the environment that might cause this issue?

[joschi]

Same issue here (on AWS CodeBuild using the ubuntu/standard:5.0 image with Docker 19.03.13).

https://github.com/aws/aws-codebuild-docker-images/blob/21.08.20/ubuntu/standard/5.0/Dockerfile#L401-L428

[kiview]

Can you please set your Testcontainers logs to DEBUG level (see here https://www.testcontainers.org/supported_docker_environment/logging_config/) and provide all the logs of the test run?

If you would be able to create some kind of local reproducer, this would also be helpful, but I get that this might be challenging or nearly impossible.

[joschi]

Unfortunately I cannot provide a local reproducer, but here's the debug output you wanted to see for the following 3 tests. PostgresIT succeeds, KafkaIT and SeleniumIT are failing.

testcontainers-debug.txt

@Testcontainers
class KafkaIT {
    @SuppressWarnings("deprecation")
    @Container
    private static final KafkaContainer KAFKA = new KafkaContainer();

    @Test
    void smokeTest() {
        assertTrue(KAFKA.isRunning());
    }
}

@Testcontainers
class PostgresIT {
    @SuppressWarnings("deprecation")
    @Container
    private static final PostgreSQLContainer POSTGRES = new PostgreSQLContainer();

    @Test
    void smokeTest() {
        assertTrue(POSTGRES.isRunning());
    }
}

@Testcontainers
class SeleniumIT {
    @SuppressWarnings("deprecation")
    @Container
    private static final BrowserWebDriverContainer<?> chrome = new BrowserWebDriverContainer().withCapabilities(new ChromeOptions());

    @Test
    void smokeTest() {
        assertTrue(chrome.isRunning());
    }
}

[kiview]

@joschi Thanks a lot, this is already helpful and we find some log lines that might help us pinpoint the issue:

11:41:39.969 [Test worker] INFO  org.testcontainers.dockerclient.DockerClientProviderStrategy - Found Docker environment with local Unix socket (unix:///var/run/docker.sock)
11:41:43.359 [Test worker] INFO  org.testcontainers.DockerClientFactory - Docker host IP address is 172.17.0.1

Can you run the same tests using Testcontainers 1.16.0 as well and share the logs? Also, please change debug level of com.github.dockerjava to INFO (keep org.testcontainers on DEBUG). Your help is much appreciated.

[joschi]

@kiview Here's the output with Testcontainers 1.16.0:
testcontainers-debug.txt

[kiview]

@joschi Extremely helpful, thanks.

As we can see, the logs I was suspecting to be pointing at the issue, are similar in 1.16.0:

13:20:04.486 [Test worker] INFO  org.testcontainers.dockerclient.DockerClientProviderStrategy - Found Docker environment with local Unix socket (unix:///var/run/docker.sock)
13:20:04.520 [Test worker] INFO  org.testcontainers.DockerClientFactory - Docker host IP address is 172.17.0.1

[rnorth]

We've found (and we believe fixed) a fairly nasty race condition in our port waiting code. This bug affects 1.16.1 and isn't specific to Docker-in-Docker setups, but is more likely to occur for containers with multiple ports and in environments where ports take longer to become visibly open externally.

Would anybody care to try a pre-release build to see? The following page documents how to obtain branch builds using JitPack (in this case, the master branch contains the fix):

https://www.testcontainers.org/jitpack_dependencies/

I wouldn't rule out the possibility that we're chasing two issues, but 🤞

[joschi]

@rnorth I re-ran our build job with the following dependencies (via JitPack):

• com.github.testcontainers.testcontainers-java:testcontainers:e257baca07
• com.github.testcontainers.testcontainers-java:junit-jupiter:e257baca07
• com.github.testcontainers.testcontainers-java:postgresql:e257baca07
• com.github.testcontainers.testcontainers-java:kafka:e257baca07
• com.github.testcontainers.testcontainers-java:selenium:e257baca07

The tests succeeded this time. Here's the debug output for the successful run:
testcontainers-debug.txt

[rnorth]

@joschi - thanks for testing! I am relieved!

[kiview]

Fixed by #4597, thanks to everyone helping in debugging this! 🙇