Testcontainers failure with 500 error using Quarkus with Kafka

[NateLove]

I'm running into an issue with Testcontainers with Quarkus using the smallrye kafka library. The issue seems to be around a 500 error that gets return when the library attempts to start up the container, so it's outside of my control. To reproduce I'm using the quarkus quickstart that can be found here: kafka-quickstart. I've tried starting kubedock with both --port-forward and --reverse-proxy. For the java side I'm running the following:

export TESTCONTAINERS_RYUK_DISABLED=true
export TESTCONTAINERS_CHECKS_DISABLE=true
export DOCKER_HOST=tcp://127.0.0.1:2475
./mvnw test 

The specific error I'm seeing is:

I0801 15:51:32.355254    9237 copy.go:36] copy 3072 bytes to 2be7a7670a6c:/
E0801 15:51:32.514282    9237 v2.go:167] io: read/write on closed pipe
E0801 15:51:32.515727    9237 util.go:18] error during request[500]: command terminated with exit code 2
[GIN] 2023/08/01 - 15:51:32 | 500 |  165.357468ms |       127.0.0.1 | PUT      "/containers/2be7a7670a6c45a7035f976f9114d52f9c946974c579329de976d57f2322b482/archive?noOverwriteDirNonDir=false&path=%2F&copyUIDGID=false"

and on the java side:

2023-08-01 15:51:29,035 INFO  [tc.doc.io/.3.4] (build-17) Creating container for image: docker.io/vectorized/redpanda:v22.3.4
2023-08-01 15:51:29,040 INFO  [org.tes.uti.RegistryAuthLocator] (build-17) Failure when attempting to lookup auth config. Please ignore if you don't have images in an authenticated registry. Details: (dockerImageName: docker.io/vectorized/redpanda:v22.3.4, configFile: /home/user/.docker/config.json, configEnv: DOCKER_AUTH_CONFIG). Falling back to docker-java default behaviour. Exception message: Status 404: No config supplied. Checked in order: /home/user/.docker/config.json (file not found), DOCKER_AUTH_CONFIG (not set)
2023-08-01 15:51:29,133 INFO  [tc.doc.io/.3.4] (build-17) Container docker.io/vectorized/redpanda:v22.3.4 is starting: 2be7a7670a6c45a7035f976f9114d52f9c946974c579329de976d57f2322b482
2023-08-01 15:51:32,331 WARN  [tc.doc.io/.3.4] (build-17) The architecture 'null' for image 'docker.io/vectorized/redpanda:v22.3.4' (ID docker.io/vectorized/redpanda:v22.3.4) does not match the Docker server architecture 'amd64'. This will cause the container to execute much more slowly due to emulation and may lead to timeout failures.
2023-08-01 15:51:32,517 ERROR [tc.doc.io/.3.4] (build-17) Could not start container: com.github.dockerjava.api.exception.InternalServerErrorException: Status 500: {"message":"command terminated with exit code 2"}
        at org.testcontainers.shaded.com.github.dockerjava.core.DefaultInvocationBuilder.execute(DefaultInvocationBuilder.java:247)
        at org.testcontainers.shaded.com.github.dockerjava.core.DefaultInvocationBuilder.put(DefaultInvocationBuilder.java:223)
        at org.testcontainers.shaded.com.github.dockerjava.core.exec.CopyArchiveToContainerCmdExec.execute(CopyArchiveToContainerCmdExec.java:34)
        at org.testcontainers.shaded.com.github.dockerjava.core.exec.CopyArchiveToContainerCmdExec.execute(CopyArchiveToContainerCmdExec.java:13)
        at org.testcontainers.shaded.com.github.dockerjava.core.exec.AbstrSyncDockerCmdExec.exec(AbstrSyncDockerCmdExec.java:21)
        at org.testcontainers.shaded.com.github.dockerjava.core.command.AbstrDockerCmd.exec(AbstrDockerCmd.java:35)
        at org.testcontainers.shaded.com.github.dockerjava.core.command.CopyArchiveToContainerCmdImpl.exec(CopyArchiveToContainerCmdImpl.java:167)
        at org.testcontainers.containers.ContainerState.copyFileToContainer(ContainerState.java:313)
        at io.quarkus.kafka.client.deployment.RedPandaKafkaContainer.containerIsStarting(RedPandaKafkaContainer.java:67)
        at org.testcontainers.containers.GenericContainer.tryStart(GenericContainer.java:479)
        at org.testcontainers.containers.GenericContainer.lambda$doStart$0(GenericContainer.java:344)
        at org.rnorth.ducttape.unreliables.Unreliables.retryUntilSuccess(Unreliables.java:81)
        at org.testcontainers.containers.GenericContainer.doStart(GenericContainer.java:334)
        at org.testcontainers.containers.GenericContainer.start(GenericContainer.java:322)
        at io.quarkus.kafka.client.deployment.DevServicesKafkaProcessor.lambda$startKafka$5(DevServicesKafkaProcessor.java:237)
        at java.base/java.util.Optional.orElseGet(Optional.java:369)
        at io.quarkus.kafka.client.deployment.DevServicesKafkaProcessor.startKafka(DevServicesKafkaProcessor.java:285)
        at io.quarkus.kafka.client.deployment.DevServicesKafkaProcessor.startKafkaDevService(DevServicesKafkaProcessor.java:95)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at io.quarkus.deployment.ExtensionLoader$3.execute(ExtensionLoader.java:864)
        at io.quarkus.builder.BuildContext.run(BuildContext.java:282)
        at org.jboss.threads.ContextHandler$1.runWith(ContextHandler.java:18)
        at org.jboss.threads.EnhancedQueueExecutor$Task.run(EnhancedQueueExecutor.java:2513)
        at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1538)
        at java.base/java.lang.Thread.run(Thread.java:829)
        at org.jboss.threads.JBossThread.run(JBossThread.java:501)

I have tried this same setup in my local environment with docker and the tests run fine.

[joyrex2001]

The test initially failed because of a failing exec's that pushed configuration and start zookeeper/kafka. I pushed a fix for that, but at my end it still fails with not finding a leader.

The quickstart is also using an older testcontainers version (1.12.3) and is based on an old Quarkus version (1.4.2). If you are using Quarkus, it might make sense to look at the Kafka dev-service instead.

[joyrex2001]

Add the IP address in the network settings of the container, which made the kafka-quickstart test work.

[NateLove]

Add which IP address to which container?

Also, side note, I was working on an updated quarkus version project and discovered that part of my issue is related to scc, where a cluster admin in openshift is able to run execs against pods, but normal users are not.

[joyrex2001]

Add which IP address to which container?

The error was due to this not providing the ip-address of the docker daemon (which is in kubedock case always 127.0.0.1). Because of that property missing in the container details, it resolved to 'null'.

The result of that, was String.format("BROKER://%s:9092", internalIp) resolving to the hostname null. With this fix, it will correctly point to the docker host and the leader-election error was fixed as well.

Also, side note, I was working on an updated quarkus version project and discovered that part of my issue is related to scc, where a cluster admin in openshift is able to run execs against pods, but normal users are not.

The minimum required rbac the service account requires is documented here. Cluster admin is not required, just a service account that has that particular documented role.

[joyrex2001]

Closing, released kubedock-0.12.0 with the fix.

[Sgitario]

This issue still happened to me using latest version of Testcontainers Kafka and latest version of Kubedock.

Kafka failed to start with:

HttpClient/5.0.3 (Java/17.0.8)"}, "X-Tc-Sid":[]string{"90d1f33e-3a85-4b17-a270-f4cde906bac1"}}

I1009 05:26:45.177043       1 util.go:77] Request Body: ./PaxHeaders.X/testcontainers_start.sh0100644 0000000 0000000 00000000034 14510707425 017124 xustar000000000 0000000 28 mtime=1696829205.1759116

testcontainers_start.sh0000777 0000000 0000000 00000000574 14510707425 014444 0ustar000000000 0000000 #!/bin/bash
export KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://127.0.0.1:38135,BROKER://null:9092
echo '' > /etc/confluent/docker/ensure 
echo 'clientPort=2181' > zookeeper.properties
echo 'dataDir=/var/lib/zookeeper/data' >> zookeeper.properties
echo 'dataLogDir=/var/lib/zookeeper/log' >> zookeeper.properties
zookeeper-server-start zookeeper.properties &
/etc/confluent/docker/run 
I1009 05:26:45.180495       1 copy.go:30] copy archive to 849a67ec1b5a:/
I1009 05:26:45.180817       1 exec.go:59] exec kubedock-849a67ec1b5a:[tar -xf - -C /]
I1009 05:26:45.188514       1 logger.go:20] Forwarding from 127.0.0.1:34283 -> 2181
I1009 05:26:45.188673       1 logger.go:20] Forwarding from [::1]:34283 -> 2181
I1009 05:26:45.188732       1 logger.go:20] Forwarding from 127.0.0.1:38135 -> 9093
I1009 05:26:45.189106       1 logger.go:20] Forwarding from [::1]:38135 -> 9093
E1009 05:26:45.380505       1 util.go:17] error during request[500]: command terminated with exit code 2

Notice the BROKER://null:9092 text.

In the latest version of testcontainers kafka, this is done by: https://github.com/testcontainers/testcontainers-java/blob/dd1427ebd30bbaba7f32184f1376b7c21e725ab5/modules/kafka/src/main/java/org/testcontainers/containers/KafkaContainer.java#L307

[joyrex2001]

This issue still happened to me using latest version of Testcontainers Kafka and latest version of Kubedock.

Are you using 0.13.0? I think this is a dupe of #51 ; the fix for this has not been released yet, but is available on latest.

[Sgitario]

This issue still happened to me using latest version of Testcontainers Kafka and latest version of Kubedock.

Are you using 0.13.0? I think this is a dupe of #51 ; the fix for this has not been released yet, but is available on latest.

Thanks @joyrex2001 !