Intermittent failure of Kafka example project container startup

[jamesnetherton]

Bug description

This seems to pop up on a regular basis for the example projects build jobs.

Caused by: org.testcontainers.containers.ContainerLaunchException: Container startup failed for image quay.io/strimzi-test-container/test-container:latest-kafka-3.2.1
	at org.testcontainers.containers.GenericContainer.doStart(GenericContainer.java:359)
	at io.strimzi.test.container.StrimziKafkaContainer.doStart(StrimziKafkaContainer.java:109)
	at org.testcontainers.containers.GenericContainer.start(GenericContainer.java:330)
	at io.quarkus.kafka.client.deployment.DevServicesKafkaProcessor.lambda$startKafka$5(DevServicesKafkaProcessor.java:258)
	at java.base/java.util.Optional.orElseGet(Optional.java:364)
	at io.quarkus.kafka.client.deployment.DevServicesKafkaProcessor.startKafka(DevServicesKafkaProcessor.java:285)
	at io.quarkus.kafka.client.deployment.DevServicesKafkaProcessor.startKafkaDevService(DevServicesKafkaProcessor.java:95)
	... 12 more
Caused by: org.rnorth.ducttape.RetryCountExceededException: Retry limit hit with exception
	at org.rnorth.ducttape.unreliables.Unreliables.retryUntilSuccess(Unreliables.java:88)
	at org.testcontainers.containers.GenericContainer.doStart(GenericContainer.java:344)
	... 18 more
Caused by: org.testcontainers.containers.ContainerLaunchException: Could not create/start container
	at org.testcontainers.containers.GenericContainer.tryStart(GenericContainer.java:563)
	at org.testcontainers.containers.GenericContainer.lambda$doStart$0(GenericContainer.java:354)
	at org.rnorth.ducttape.unreliables.Unreliables.retryUntilSuccess(Unreliables.java:81)
	... 19 more
Caused by: java.lang.IllegalStateException: Wait strategy failed. Container exited with code 1
	at org.testcontainers.containers.GenericContainer.tryStart(GenericContainer.java:533)
	... 21 more
Caused by: org.testcontainers.containers.ContainerLaunchException: Timed out waiting for log output matching '.*Transitioning from RECOVERY to RUNNING.*'
	at org.testcontainers.containers.wait.strategy.LogMessageWaitStrategy.waitUntilReady(LogMessageWaitStrategy.java:47)
	at org.testcontainers.containers.wait.strategy.AbstractWaitStrategy.waitUntilReady(AbstractWaitStrategy.java:52)
	at org.testcontainers.containers.GenericContainer.waitUntilContainerStarted(GenericContainer.java:909)
	at org.testcontainers.containers.GenericContainer.tryStart(GenericContainer.java:500)
	... 21 more

[jamesnetherton]

I've also seen this happen for the tests in the Kafka extension deployment module.

[jamesnetherton]

Some more info from the logs:

2024-06-10 09:17:58,108 ERROR [tc.qua.io/.2.1] (build-7) Log output from the failed container:
usage: kafka-storage format [-h] --config CONFIG --cluster-id CLUSTER_ID
                     [--ignore-formatted]
kafka-storage: error: argument --cluster-id/-t: expected one argument
[2024-06-10 09:17:00,484] INFO Registered kafka:type=kafka.Log4jController MBean (kafka.utils.Log4jControllerRegistration$)
okeeper.common.X509Util)
[2024-06-10 09:17:00,859] WARN No meta.properties file under dir /tmp/kraft-combined-logs/meta.properties (kafka.server.BrokerMetadataCheckpoint)
[2024-06-10 09:17:00,861] ERROR Exiting Kafka due to fatal exception (kafka.Kafka$)
kafka.common.KafkaException: No `meta.properties` found in /tmp/kraft-combined-logs (have you run `kafka-storage.sh` to format the directory?)
	at kafka.server.BrokerMetadataCheckpoint$.$anonfun$getBrokerMetadataAndOfflineDirs$2(BrokerMetadataCheckpoint.scala:171)
	at scala.collection.immutable.List.foreach(List.scala:333)
	at kafka.server.BrokerMetadataCheckpoint$.getBrokerMetadataAndOfflineDirs(BrokerMetadataCheckpoint.scala:160)
LogDirs(KafkaRaftServer.scala:155)
	at kafka.server.KafkaRaftServer.<init>(KafkaRaftServer.scala:56)
	at kafka.Kafka$.buildServer(Kafka.scala:79)
	at kafka.Kafka$.main(Kafka.scala:87)
	at kafka.Kafka.main(Kafka.scala)