testcontainers 1.12.2 and up does not replace versions from .env

[leCandas]

Hello,

In a couple of projects we were using test containers version 1.12.1 with local-compose and pull enabled.
I have tried to update to testcontainers 1.12.5 however I have start to some errors on my console.

We are using single file docker-compose.yml version 2 and all image versions inside the compose configuration are coming from .env file
In docker-compose.yml we have mysql image with version placeholder
image: eu-west-1.amazonaws.com/content-delivery/mysql:${mysqlversion}
In .env file we have the version for placeholder
mysqlversion=1.0.0-SNAPSHOT

with this configuration, I start to see the errors below.

ERROR c.g.d.c.async.ResultCallbackTemplate - Error during callback com.github.dockerjava.api.exception.BadRequestException: {"message":"invalid reference format"}
	at org.testcontainers.dockerclient.transport.okhttp.OkHttpInvocationBuilder.execute(OkHttpInvocationBuilder.java:277)
	at org.testcontainers.dockerclient.transport.okhttp.OkHttpInvocationBuilder.lambda$executeAndStream$3(OkHttpInvocationBuilder.java:303)
	at java.lang.Thread.run(Thread.java:748)

WARN  o.t.c.DockerComposeContainer - Failed to pull image 'eu-west-1.amazonaws.com/content-delivery/mysql:${mysqlversion}'. Exception message was {"message":"invalid reference format"}

If I put version instead of the placeholder all works fine.
Before the update, we were using 1.12.1 version and that version does not have any problems with this type of configuration.

So, basically this is an issue with the current version.

[bsideup]

cc @rnorth
it seems that the logic to pull the images via the API does not work when image values contain placeholders. We may need to use docker-compose config to "render" it before parsing

[krichter722]

I'm thinking about an implementation and I'm wondering how close the behaviour to docker-compose should be. docker-compose takes .env implicitly from the current directory if present. Should testcontainers do the same (might be unexepected) or should it take the one that is specified with withEnvFile (to be added)? I prefer the latter because testcontainers use the software in a different way, especially with changes to the current working directory which might cause unpleasant behaviour.

I assume that the function of withEnv ought to remain separated because it controls the container environment and not the template variable resolution in docker-compose.yml.

[leCandas]

@krichter722 from your comment, I did read the .env file and extracted properties as Map<String, String> and applied the map to DockerComposeContainer by calling withEnv but the result is the same... Testcontainers still try to pull images with their placeholder.

[krichter722]

@leCandas That's the expected behaviour of the current state of the software without the changes that we're discussing.

[leCandas]

ahh ok...
If you ask my opinion on that (.env) as a developer, I would expect testcontainres will automatically evaluate .env file just as docker-compose does without any additional tweaking.

[bdellegrazie]

I think the same issue applies to ARG values (with defaults) too - we use those to modify the image repository

[jdelucaa]

I'm using version 1.14.0 and I'm seeing the same issue.

I am investigating why sometimes my docker compose won't start my services. I am getting this generic error and I thought it could be related to not being able to pull (but the log doesn't say much):

org.testcontainers.containers.ContainerLaunchException: Local Docker Compose exited abnormally with code 1 whilst running command: up -d
        at org.testcontainers.containers.LocalDockerCompose.invoke(DockerComposeContainer.java:695)
        at org.testcontainers.containers.DockerComposeContainer.runWithCompose(DockerComposeContainer.java:263)
        at org.testcontainers.containers.DockerComposeContainer.createServices(DockerComposeContainer.java:211)
        at org.testcontainers.containers.DockerComposeContainer.start(DockerComposeContainer.java:170)
        at shared.UiTestSpec.setupSpec(UiTestSpec.groovy:65)

I found the following logs related to the error described in this issue:

Jun 23, 2020 5:22:26 PM org.testcontainers.containers.DockerComposeContainer lambda$pullImages$1
    INFO: Preemptively checking local images for '${DOCKER_REGISTRY}/${DOCKER_NAMESPACE}/frontend:${IMAGE_TAG_FRONTEND:-local}', referenced via a compose file or transitive Dockerfile. If not available, it will be pulled.
    Jun 23, 2020 5:22:26 PM com.github.dockerjava.api.async.ResultCallbackTemplate onError
    SEVERE: Error during callback
    com.github.dockerjava.api.exception.BadRequestException: {"message":"invalid reference format: repository name must be lowercase"}

But in the end, it seems to, sometimes, be able to pull anyway:

2020-06-23T15:31:23.050+0000 [DEBUG] [TestEventLogger]     INFO: Pulling frontend (my-registry/my-namespace/frontend:614253)...
2020-06-23T15:31:23.575+0000 [DEBUG] [TestEventLogger]     Jun 23, 2020 3:31:23 PM org.testcontainers.shaded.org.zeroturnaround.exec.stream.slf4j.Slf4jInfoOutputStream processLine
2020-06-23T15:31:23.575+0000 [DEBUG] [TestEventLogger]     INFO: 614253: Pulling from my-namespace/frontend
2020-06-23T15:31:31.908+0000 [DEBUG] [TestEventLogger]     Jun 23, 2020 3:31:31 PM org.testcontainers.shaded.org.zeroturnaround.exec.stream.slf4j.Slf4jInfoOutputStream processLine
2020-06-23T15:31:31.908+0000 [DEBUG] [TestEventLogger]     INFO: Digest: sha256:1ac32172f61ba8dc66e82ec524fd9a5881a8f3a2a7cbb9bd10cb63d5972f73ad
2020-06-23T15:31:31.926+0000 [DEBUG] [TestEventLogger]     Jun 23, 2020 3:31:31 PM org.testcontainers.shaded.org.zeroturnaround.exec.stream.slf4j.Slf4jInfoOutputStream processLine
2020-06-23T15:31:31.926+0000 [DEBUG] [TestEventLogger]     INFO: Status: Downloaded newer image for my-registry/my-namespace/frontend:614253

Do you think they could be related?

[leCandas]

Hey @jdelucaa,
The logging part of testcontainers is not good.
To see what is happening under the hood you need to add some more loggers and these can be found from the source code of Testcontainers. Its pitty that there is no single type here.

I am using these loggers with logback to fetch more informationa about failure

<logger name="🐳 [docker-compose]" level="${loglevel}" additivity="false"> <appender-ref ref="TESTCONTAINERS" /> </logger> <logger name="🐳 [docker-compose.exe]" level="${loglevel}" additivity="false"> <appender-ref ref="TESTCONTAINERS" /> </logger> <logger name="docker[docker-compose.exe]" level="${loglevel}" additivity="false"> <appender-ref ref="TESTCONTAINERS" /> </logger> <logger name="docker[docker-compose]" level="${loglevel}" additivity="false"> <appender-ref ref="TESTCONTAINERS" /> </logger> <logger name="org.testcontainers" level="${loglevel}" additivity="false"> <appender-ref ref="TESTCONTAINERS" /> </logger> <logger name="com.github.dockerjava" level="${loglevel}" additivity="false"> <appender-ref ref="TESTCONTAINERS" /> </logger>

So, if you add those you will be able to see more.

There is a pull setting in testcontainers DockerComposeContainer, and that is not working when you use references in docker-compose.

But even though testcontainers fails to pull (since its not able to use .env or systemParameters to fill versions) when docker-compose starts its "pull if not exists" policy kicks in and pulls that missing containers.

to prevent mismatching versions if you have an old container locally, this can cause inconsistency issues.

so the best solution I found on a build slave is basically running docker system prune and remove all containers before starting the build. And I have a disabled pull on testcontainer since it is not working, docker-compose pulls the images for me on a clean environment.

Hope this issue will be solved soon and this will become more useful.

[jdelucaa]

Thanks @leCandas, I will probably do the same thing. It will ensure I'll be using the correct versions.