Docker compose example with confluent-local image #816
Description
Issue submitter TODO list
- I've searched for an already existing issues here
- I'm running a supported version of the application which is listed here and the feature is not present there
Is your proposal related to a problem?
Confluent has a confluent-local image optimized for use in Kraft Mode https://docs.confluent.io/platform/current/installation/docker/image-reference.html
I've had a hard time getting everything working and still cannot get my rest proxy that is built in working once I set all the variables the KafkaUI docs specify to add. It also seems like a lot of extra env vars than I should need in the cp-kafka examples? but Maybe not.
Describe the feature you're interested in
Minimal single instance kafka docker compose example with confluent-local and not breaking the built-in Rest Proxy
Describe alternatives you've considered
This is what I'm trying, but of those KAFKA_ env vars, I'm not sure what is all 100% necessary using the confluent-local image. Additionally with this setting I get HTTP read timesouts on localhost:8082 now
# Based on the following w/o schema-registry and kafka-connect
# - https://github.com/kafbat/kafka-ui/blob/main/documentation/compose/kafbat-ui.yaml
# - https://ui.docs.kafbat.io/configuration/complex-configuration-examples/kraft-mode-+-multiple-brokers
services:
# Run Kafka without zookeeper (Kraft Mode)
kafka0:
hostname: kafka0
container_name: kafka0
image: confluentinc/confluent-local:latest
ports:
- "9092:9092"
- "8082:8082"
environment:
KAFKA_BROKER_ID: 1
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: 'CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT'
KAFKA_ADVERTISED_LISTENERS: 'PLAINTEXT://kafka0:29092,PLAINTEXT_HOST://localhost:9092'
KAFKA_PROCESS_ROLES: 'broker,controller'
KAFKA_NODE_ID: 1
KAFKA_CONTROLLER_QUORUM_VOTERS: '1@kafka0:29093'
KAFKA_LISTENERS: 'PLAINTEXT://kafka0:29092,CONTROLLER://kafka0:29093,PLAINTEXT_HOST://0.0.0.0:9092'
KAFKA_INTER_BROKER_LISTENER_NAME: 'PLAINTEXT'
KAFKA_CONTROLLER_LISTENER_NAMES: 'CONTROLLER'
KAFKA_LOG_DIRS: '/tmp/kraft-combined-logs'
CLUSTER_ID: 'MkU3OEVBNTcwNTJENDM2Qk'
# Kafka UI
kafbat-ui:
container_name: kafbat-ui
image: ghcr.io/kafbat/kafka-ui:latest
ports:
- 8080:8080
depends_on:
- kafka0
environment:
KAFKA_CLUSTERS_0_NAME: local
KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS: kafka0:29092
kafka-init-topics:
# Not using confluent-local image because it doesn't have all the shell commands installed
image: confluentinc/cp-kafka:latest
depends_on:
- kafka0
volumes:
- ./scripts/create-topics.sh:/create-topics.sh
command: "/create-topics.sh"Version you're running
4cf17a0 v1.1.0
Additional context
When I make a rest call POST http://localhost:8082/topics/xxxx i see these logs and get a read timeout.
kafka0 | [2025-02-03 21:07:42,576] INFO KafkaJsonSerializerConfig values:
kafka0 | json.indent.output = false
kafka0 | json.write.dates.iso8601 = false
kafka0 | (io.confluent.kafka.serializers.KafkaJsonSerializerConfig)
kafka0 | [2025-02-03 21:07:42,595] INFO ProducerConfig values:
kafka0 | acks = -1
kafka0 | auto.include.jmx.reporter = true
kafka0 | batch.size = 16384
kafka0 | bootstrap.servers = [localhost:29092]
kafka0 | buffer.memory = 33554432
kafka0 | client.dns.lookup = use_all_dns_ips
kafka0 | client.id = producer-1
kafka0 | compression.gzip.level = -1
kafka0 | compression.lz4.level = 9
kafka0 | compression.type = none
kafka0 | compression.zstd.level = 3
kafka0 | connections.max.idle.ms = 540000
kafka0 | delivery.timeout.ms = 120000
kafka0 | enable.idempotence = false
kafka0 | enable.metrics.push = true
kafka0 | interceptor.classes = []
kafka0 | key.serializer = class org.apache.kafka.common.serialization.ByteArraySerializer
kafka0 | linger.ms = 0
kafka0 | max.block.ms = 60000
kafka0 | max.in.flight.requests.per.connection = 5
kafka0 | max.request.size = 1048576
kafka0 | metadata.max.age.ms = 300000
kafka0 | metadata.max.idle.ms = 300000
kafka0 | metadata.recovery.strategy = none
kafka0 | metric.reporters = []
kafka0 | metrics.num.samples = 2
kafka0 | metrics.recording.level = INFO
kafka0 | metrics.sample.window.ms = 30000
kafka0 | partitioner.adaptive.partitioning.enable = true
kafka0 | partitioner.availability.timeout.ms = 0
kafka0 | partitioner.class = null
kafka0 | partitioner.ignore.keys = false
kafka0 | receive.buffer.bytes = 32768
kafka0 | reconnect.backoff.max.ms = 1000
kafka0 | reconnect.backoff.ms = 50
kafka0 | request.timeout.ms = 30000
kafka0 | retries = 2147483647
kafka0 | retry.backoff.max.ms = 1000
kafka0 | retry.backoff.ms = 100
kafka0 | sasl.client.callback.handler.class = null
kafka0 | sasl.jaas.config = null
kafka0 | sasl.kerberos.kinit.cmd = /usr/bin/kinit
kafka0 | sasl.kerberos.min.time.before.relogin = 60000
kafka0 | sasl.kerberos.service.name = null
kafka0 | sasl.kerberos.ticket.renew.jitter = 0.05
kafka0 | sasl.kerberos.ticket.renew.window.factor = 0.8
kafka0 | sasl.login.callback.handler.class = null
kafka0 | sasl.login.class = null
kafka0 | sasl.login.connect.timeout.ms = null
kafka0 | sasl.login.read.timeout.ms = null
kafka0 | sasl.login.refresh.buffer.seconds = 300
kafka0 | sasl.login.refresh.min.period.seconds = 60
kafka0 | sasl.login.refresh.window.factor = 0.8
kafka0 | sasl.login.refresh.window.jitter = 0.05
kafka0 | sasl.login.retry.backoff.max.ms = 10000
kafka0 | sasl.login.retry.backoff.ms = 100
kafka0 | sasl.mechanism = GSSAPI
kafka0 | sasl.oauthbearer.clock.skew.seconds = 30
kafka0 | sasl.oauthbearer.expected.audience = null
kafka0 | sasl.oauthbearer.expected.issuer = null
kafka0 | sasl.oauthbearer.jwks.endpoint.refresh.ms = 3600000
kafka0 | sasl.oauthbearer.jwks.endpoint.retry.backoff.max.ms = 10000
kafka0 | sasl.oauthbearer.jwks.endpoint.retry.backoff.ms = 100
kafka0 | sasl.oauthbearer.jwks.endpoint.url = null
kafka0 | sasl.oauthbearer.scope.claim.name = scope
kafka0 | sasl.oauthbearer.sub.claim.name = sub
kafka0 | sasl.oauthbearer.token.endpoint.url = null
kafka0 | security.protocol = PLAINTEXT
kafka0 | security.providers = null
kafka0 | send.buffer.bytes = 131072
kafka0 | socket.connection.setup.timeout.max.ms = 30000
kafka0 | socket.connection.setup.timeout.ms = 10000
kafka0 | ssl.cipher.suites = null
kafka0 | ssl.enabled.protocols = [TLSv1.2, TLSv1.3]
kafka0 | ssl.endpoint.identification.algorithm = https
kafka0 | ssl.engine.factory.class = null
kafka0 | ssl.key.password = null
kafka0 | ssl.keymanager.algorithm = SunX509
kafka0 | ssl.keystore.certificate.chain = null
kafka0 | ssl.keystore.key = null
kafka0 | ssl.keystore.location = null
kafka0 | ssl.keystore.password = null
kafka0 | ssl.keystore.type = JKS
kafka0 | ssl.protocol = TLSv1.3
kafka0 | ssl.provider = null
kafka0 | ssl.secure.random.implementation = null
kafka0 | ssl.trustmanager.algorithm = PKIX
kafka0 | ssl.truststore.certificates = null
kafka0 | ssl.truststore.location = null
kafka0 | ssl.truststore.password = null
kafka0 | ssl.truststore.type = JKS
kafka0 | transaction.timeout.ms = 60000
kafka0 | transactional.id = null
kafka0 | value.serializer = class org.apache.kafka.common.serialization.ByteArraySerializer
kafka0 | (org.apache.kafka.clients.producer.ProducerConfig)
kafka0 | [2025-02-03 21:07:42,608] INFO initializing Kafka metrics collector (org.apache.kafka.common.telemetry.internals.KafkaMetricsCollector)
kafka0 | [2025-02-03 21:07:42,636] INFO These configurations '[metrics.context.resource.cluster.id, metrics.context._namespace, metrics.context.resource.version, metrics.context.resource.commit.id, metrics.context.resource.type]' were supplied but are not used yet. (org.apache.kafka.clients.producer.ProducerConfig)
kafka0 | [2025-02-03 21:07:42,636] INFO Kafka version: 7.8.0-ccs (org.apache.kafka.common.utils.AppInfoParser)
kafka0 | [2025-02-03 21:07:42,636] INFO Kafka commitId: cc7168da1fddfcfd (org.apache.kafka.common.utils.AppInfoParser)
kafka0 | [2025-02-03 21:07:42,636] INFO Kafka startTimeMs: 1738616862636 (org.apache.kafka.common.utils.AppInfoParser)
kafka0 | [2025-02-03 21:07:42,741] INFO [Producer clientId=producer-1] Node -1 disconnected. (org.apache.kafka.clients.NetworkClient)
kafka0 | [2025-02-03 21:07:42,742] WARN [Producer clientId=producer-1] Connection to node -1 (localhost/127.0.0.1:29092) could not be established. Node may not be available. (org.apache.kafka.clients.NetworkClient)
kafka0 | [2025-02-03 21:07:42,742] WARN [Producer clientId=producer-1] Bootstrap broker localhost:29092 (id: -1 rack: null) disconnected (org.apache.kafka.clients.NetworkClient)
kafka0 | [2025-02-03 21:07:42,800] INFO [Producer clientId=producer-1] Node -1 disconnected. (org.apache.kafka.clients.NetworkClient)
kafka0 | [2025-02-03 21:07:42,800] WARN [Producer clientId=producer-1] Connection to node -1 (localhost/127.0.0.1:29092) could not be established. Node may not be available. (org.apache.kafka.clients.NetworkClient)
kafka0 | [2025-02-03 21:07:42,800] WARN [Producer clientId=producer-1] Bootstrap broker localhost:29092 (id: -1 rack: null) disconnected (org.apache.kafka.clients.NetworkClient)
kafka0 | [2025-02-03 21:07:42,914] INFO [Producer clientId=producer-1] Node -1 disconnected. (org.apache.kafka.clients.NetworkClient)
kafka0 | [2025-02-03 21:07:42,915] WARN [Producer clientId=producer-1] Connection to node -1 (localhost/127.0.0.1:29092) could not be established. Node may not be available. (org.apache.kafka.clients.NetworkClient)
kafka0 | [2025-02-03 21:07:42,915] WARN [Producer clientId=producer-1] Bootstrap broker localhost:29092 (id: -1 rack: null) disconnected (org.apache.kafka.clients.NetworkClient)
kafka0 | [2025-02-03 21:07:43,130] INFO [Producer clientId=producer-1] Node -1 disconnected. (org.apache.kafka.clients.NetworkClient)
kafka0 | [2025-02-03 21:07:43,131] WARN [Producer clientId=producer-1] Connection to node -1 (localhost/127.0.0.1:29092) could not be established. Node may not be available. (org.apache.kafka.clients.NetworkClient)
kafka0 | [2025-02-03 21:07:43,131] WARN [Producer clientId=producer-1] Bootstrap broker localhost:29092 (id: -1 rack: null) disconnected (org.apache.kafka.clients.NetworkClient)
kafka0 | [2025-02-03 21:07:43,571] INFO [Producer clientId=producer-1] Node -1 disconnected. (org.apache.kafka.clients.NetworkClient)
kafka0 | [2025-02-03 21:07:43,572] WARN [Producer clientId=producer-1] Connection to node -1 (localhost/127.0.0.1:29092) could not be established. Node may not be available. (org.apache.kafka.clients.NetworkClient)
kafka0 | [2025-02-03 21:07:43,572] WARN [Producer clientId=producer-1] Bootstrap broker localhost:29092 (id: -1 rack: null) disconnected (org.apache.kafka.clients.NetworkClient)
kafka0 | [2025-02-03 21:07:44,510] INFO [Producer clientId=producer-1] Node -1 disconnected. (org.apache.kafka.clients.NetworkClient)
kafka0 | [2025-02-03 21:07:44,510] WARN [Producer clientId=producer-1] Connection to node -1 (localhost/127.0.0.1:29092) could not be established. Node may not be available. (org.apache.kafka.clients.NetworkClient)
kafka0 | [2025-02-03 21:07:44,510] WARN [Producer clientId=producer-1] Bootstrap broker localhost:29092 (id: -1 rack: null) disconnected (org.apache.kafka.clients.NetworkClient)
kafka0 | [2025-02-03 21:07:45,543] INFO [Producer clientId=producer-1] Node -1 disconnected. (org.apache.kafka.clients.NetworkClient)
kafka0 | [2025-02-03 21:07:45,544] WARN [Producer clientId=producer-1] Connection to node -1 (localhost/127.0.0.1:29092) could not be established. Node may not be available. (org.apache.kafka.clients.NetworkClient)
kafka0 | [2025-02-03 21:07:45,544] WARN [Producer clientId=producer-1] Bootstrap broker localhost:29092 (id: -1 rack: null) disconnected (org.apache.kafka.clients.NetworkClient)
kafka0 | [2025-02-03 21:07:46,561] INFO [Producer clientId=producer-1] Node -1 disconnected. (org.apache.kafka.clients.NetworkClient)
kafka0 | [2025-02-03 21:07:46,562] WARN [Producer clientId=producer-1] Connection to node -1 (localhost/127.0.0.1:29092) could not be established. Node may not be available. (org.apache.kafka.clients.NetworkClient)
kafka0 | [2025-02-03 21:07:46,562] WARN [Producer clientId=producer-1] Bootstrap broker localhost:29092 (id: -1 rack: null) disconnected (org.apache.kafka.clients.NetworkClient)
kafka0 | [2025-02-03 21:07:47,549] INFO [Producer clientId=producer-1] Node -1 disconnected. (org.apache.kafka.clients.NetworkClient)
kafka0 | [2025-02-03 21:07:47,550] WARN [Producer clientId=producer-1] Connection to node -1 (localhost/127.0.0.1:29092) could not be established. Node may not be available. (org.apache.kafka.clients.NetworkClient)
kafka0 | [2025-02-03 21:07:47,550] WARN [Producer clientId=producer-1] Bootstrap broker localhost:29092 (id: -1 rack: null) disconnected (org.apache.kafka.clients.NetworkClient)
kafka0 | [2025-02-03 21:07:48,455] INFO [Producer clientId=producer-1] Node -1 disconnected. (org.apache.kafka.clients.NetworkClient)
kafka0 | [2025-02-03 21:07:48,456] WARN [Producer clientId=producer-1] Connection to node -1 (localhost/127.0.0.1:29092) could not be established. Node may not be available. (org.apache.kafka.clients.NetworkClient)
kafka0 | [2025-02-03 21:07:48,456] WARN [Producer clientId=producer-1] Bootstrap broker localhost:29092 (id: -1 rack: null) disconnected (org.apache.kafka.clients.NetworkClient)
kafka0 | [2025-02-03 21:07:49,468] INFO [Producer clientId=producer-1] Node -1 disconnected. (org.apache.kafka.clients.NetworkClient)
kafka0 | [2025-02-03 21:07:49,469] WARN [Producer clientId=producer-1] Connection to node -1 (localhost/127.0.0.1:29092) could not be established. Node may not be available. (org.apache.kafka.clients.NetworkClient)
kafka0 | [2025-02-03 21:07:49,469] WARN [Producer clientId=producer-1] Bootstrap broker localhost:29092 (id: -1 rack: null) disconnected (org.apache.kafka.clients.NetworkClient)
kafka0 | [2025-02-03 21:07:50,473] INFO [Producer clientId=producer-1] Node -1 disconnected. (org.apache.kafka.clients.NetworkClient)
When I try to add more advertised listeners
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: 'CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT,PLAINTEXT_HOST2:PLAINTEXT'
KAFKA_ADVERTISED_LISTENERS: 'PLAINTEXT://kafka0:29092,PLAINTEXT_HOST://localhost:9092,PLAINTEXT_HOST2://localhost:8082'
KAFKA_LISTENERS: 'PLAINTEXT://kafka0:29092,CONTROLLER://kafka0:29093,PLAINTEXT_HOST://0.0.0.0:9092,PLAINTEXT_HOST2://0.0.0.0:8082'
I get this
reactor.netty.http.client.PrematureCloseException: Connection prematurely closed BEFORE response
and the following despite sending a tiny json payload in the REST call. This all works without Kafka UI and changing the kafka container's env vars
)
2025-02-03 16:28:04 [2025-02-03 21:28:04,018] WARN [SocketServer listenerType=BROKER, nodeId=1] Unexpected error from /172.20.0.1 (channelId=172.20.0.2:8082-172.20.0.1:57682-0); closing connection (org.apache.kafka.common.network.Selector)
2025-02-03 16:28:04 org.apache.kafka.common.network.InvalidReceiveException: Invalid receive (size = 1347375956 larger than 104857600)
2025-02-03 16:28:04 at org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:94)
2025-02-03 16:28:04 at org.apache.kafka.common.network.KafkaChannel.receive(KafkaChannel.java:462)
2025-02-03 16:28:04 at org.apache.kafka.common.network.KafkaChannel.read(KafkaChannel.java:412)
2025-02-03 16:28:04 at org.apache.kafka.common.network.Selector.attemptRead(Selector.java:678)
2025-02-03 16:28:04 at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:580)
2025-02-03 16:28:04 at org.apache.kafka.common.network.Selector.poll(Selector.java:485)
2025-02-03 16:28:04 at kafka.network.Processor.poll(SocketServer.scala:1111)
2025-02-03 16:28:04 at kafka.network.Processor.run(SocketServer.scala:1015)
2025-02-03 16:28:04 at java.base/java.lang.Thread.run(Unknown Source)
2025-02-03 16:25:34 error in executing the command: kafka-ready check failed2025-02-03T21:28:08.274546544Z Error while getting broker list.
2025-02-03 16:28:08 java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.TimeoutException: Timed out waiting for a node assignment. Call: listNodes
2025-02-03 16:28:08 at java.base/java.util.concurrent.CompletableFuture.reportGet(Unknown Source)
2025-02-03 16:28:08 at java.base/java.util.concurrent.CompletableFuture.get(Unknown Source)
2025-02-03 16:28:08 at org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:165)
2025-02-03 16:28:08 at io.confluent.admin.utils.ClusterStatus.isKafkaReady(ClusterStatus.java:147)
2025-02-03 16:28:08 at io.confluent.admin.utils.cli.KafkaReadyCommand.main(KafkaReadyCommand.java:149)
2025-02-03 16:28:08 Caused by: org.apache.kafka.common.errors.TimeoutException: Timed out waiting for a node assignment. Call: listNodes
2025-02-03 16:28:09 Expected 1 brokers but found only 0. Trying to query Kafka for metadata again ...
2025-02-03 16:28:09 Expected 1 brokers but found only 0. Brokers found [].
2025-02-03 16:28:09 Error: kafka-ready check failed
Like with
services:
# Run Kafka without zookeeper (Kraft Mode)
kafka0:
hostname: kafka0
container_name: kafka0
image: confluentinc/confluent-local:latest
ports:
- "9092:9092"
- "8082:8082"I get immediate REST response
HTTP/1.1 200 OK
Date: Mon, 03 Feb 2025 21:31:54 GMT
Content-Type: application/vnd.kafka.v2+json
Vary: Accept-Encoding, User-Agent
Content-Encoding: gzip
Transfer-Encoding: chunked
{
"offsets": [
{
"partition": 0,
"offset": 0,
"error_code": null,
"error": null
}
],
"key_schema_id": null,
"value_schema_id": null
}