[inclubator/kafka] Accessing Kafka outside kubernetes not working

[vidhy]

Is this a request for help?:
Yes

---

Is this a BUG REPORT or FEATURE REQUEST? (choose one):
Bug

Version of Helm and Kubernetes:
Helm version
Client: &version.Version{SemVer:"v2.9.1", GitCommit:"20adb27c7c5868466912eebdf6664e7390ebe710", GitTreeState:"clean"}
Server: &version.Version{SemVer:"v2.9.1", GitCommit:"20adb27c7c5868466912eebdf6664e7390ebe710", GitTreeState:"clean"}

Kubernetes version
Client Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.6", GitCommit:"9f8ebd171479bec0ada837d7ee641dec2f8c6dd1", GitTreeState:"clean", BuildDate:"2018-03-21T15:21:50Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.6", GitCommit:"9f8ebd171479bec0ada837d7ee641dec2f8c6dd1", GitTreeState:"clean", BuildDate:"2018-03-21T15:13:31Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}

Which chart:
[inclubator/kafka]

What happened:
Accessing kafka from outside note working as expected.

What you expected to happen:
Kafka broker should be accessible at kafka.k8s.cluster.vpc:31091

How to reproduce it (as minimally and precisely as possible):
Follwed the given steps for helm chart setup of kafka. with external enabled

external:
  enabled: true
  domain: 'k8s.cluster.vpc'

Anything else we need to know:
No issue of connectivity between the instances. Similarly the Route53 A records are getting created properly.

kc describe svc kafka-1-external -n test-kafka

Name:                     kafka-1-external
Namespace:                test-kafka
Labels:                   app=kafka
                          chart=kafka-0.8.3
                          heritage=Tiller
                          pod=kafka-1
                          release=kafka
Annotations:              dns.alpha.kubernetes.io/internal=kafka.k8s.groupplatform.vpc
Selector:                 app=kafka,pod=kafka-1,release=kafka
Type:                     NodePort
IP:                       100.70.219.240
Port:                     external-broker  19092/TCP
TargetPort:               31091/TCP
NodePort:                 external-broker  31091/TCP
Endpoints:                100.115.0.28:31091
Session Affinity:         None
External Traffic Policy:  Cluster
Events:                   <none>

Since the target port is 31091, the port should be exposed from the pod. But when i do telnet to the corresponding port connection refused.

When I exec inside the kafka-0 pod, and check the /etc/kafka/kafka.properties, it has the listeners as

advertised.listeners=PLAINTEXT://100.120.0.5:9092
listeners=PLAINTEXT://0.0.0.0:9092

[drekle]

Hi @vidhy

I believe you must also specify configurationOverrides.advertised.listeners. Guessing for your example it would be this...

configurationOverrides:
  advertised.listeners: '&& export KAFKA_ADVERTISED_LISTENERS=EXTERNAL://k8s.cluster.vpc:$((31090
    + ${KAFKA_BROKER_ID})),PLAINTEXT://${POD_IP}:9092'

Noticed from looking at the statefulset template.

[vidhy]

@drekle I dont think that fixes the issue. These are the properties I have inside the pod.

advertised.listeners=EXTERNAL://k8s.groupplatform.vpc:31090,PLAINTEXT://100.120.0.9:9092
listener.security.protocol.map=PLAINTEXT:PLAINTEXT,EXTERNAL:PLAINTEXT
listeners=EXTERNAL://0.0.0.0:31090,PLAINTEXT://0.0.0.0:9092

When i do
bin/kafka-console-producer.sh --broker-list test-kafka.k8s.groupplatform.vpc:31090 --topic my-topic

I am getting

[2018-07-25 11:19:44,717] ERROR Error when sending message to topic my-topic with key: null, value: 2 bytes with error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
org.apache.kafka.common.errors.TimeoutException: Expiring 1 record(s) for my-topic-0: 60020 ms has passed since batch creation plus linger time

[vidhy]

I am able to produce and consume properly inside the kubernetes cluster and access broker via "test-kafka.test-kafka.svc.cluster.local:9092". So the issue is with external access of kafka alone.

[drekle]

@vidhy I told you wrong above. The advertised listener should be the FQDN. You are calling it with test-kafka.k8s.groupplatform.vpc:31090 yet it is listening to k8s.groupplatform.vpc:31090. This was my mistake in my mention. Can you try changing it to test-kafka.k8s.groupplatform.vpc:31090?

configurationOverrides:
  advertised.listeners: '&& export KAFKA_ADVERTISED_LISTENERS=EXTERNAL://test-kafka.k8s.groupplatform.vpc:$((31090
    + ${KAFKA_BROKER_ID})),PLAINTEXT://${POD_IP}:9092'

[vidhy]

that worked like a charm. thanks @drekle Adding the listener config changes in docs will be helpful for novice users.

[Triasto]

@vidhy I told you wrong above. The advertised listener should be the FQDN. You are calling it with test-kafka.k8s.groupplatform.vpc:31090 yet it is listening to k8s.groupplatform.vpc:31090. This was my mistake in my mention. Can you try changing it to test-kafka.k8s.groupplatform.vpc:31090?

configurationOverrides:
  advertised.listeners: '&& export KAFKA_ADVERTISED_LISTENERS=EXTERNAL://test-kafka.k8s.groupplatform.vpc:$((31090
    + ${KAFKA_BROKER_ID})),PLAINTEXT://${POD_IP}:9092'

OMG .. thank you thank you so much!

I've been troubleshooting all day to find this answer, works like a charm now!

[pujik16]

Hi @drekle,

We configured Kafka using incubator Helm charts on rancher Kubernetes cluster by updating values.yml with below properties and using helm install --name test-kafka -f values.yml incubator/kafka --namespace test

1. enabled:true
2. advertised.listeners: '&& export KAFKA_ADVERTISED_LISTENERS=EXTERNAL://test-kafka.cluster.local:$((31090
   • ${KAFKA_BROKER_ID})),PLAINTEXT://${POD_IP}:9092'
3. uncommented listener.security.protocol.map

We are able to successfully create topics within cluster. But we still have issues in exposing Kafka outside cluster. Can you please help

[ceyhunn]

Hi @drekle , @vidhy

I have following config for external

replicas: 3
zookeeper:
  replicaCount: 3
external:
  enabled: true
  type: NodePort
  servicePort: 9092
  firstListenerPort: 30101
configurationOverrides:
  listeners: "PLAINTEXT://:9092"
  "advertised.listeners": '&& export KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://192.168.1.47:$((30101 + ${KAFKA_BROKER_ID}))'
  "listener.security.protocol.map": "PLAINTEXT:PLAINTEXT"

Then all containers are up, but brokers can't communicate each other:

java.io.IOException: Connection to 192.168.1.47:30101 (id: 0 rack: null) failed.
at org.apache.kafka.clients.NetworkClientUtils.awaitReady(NetworkClientUtils.java:70)
at kafka.controller.RequestSendThread.brokerReady(ControllerChannelManager.scala:279)
at kafka.controller.RequestSendThread.doWork(ControllerChannelManager.scala:233)
at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:82)
[2019-09-03 10:45:09,155] WARN [Controller id=2, targetBrokerId=1] Connection to node 1 could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
[2019-09-03 10:45:09,155] WARN [RequestSendThread controllerId=2] Controller 2's connection to broker 192.168.1.47:30102 (id: 1 rack: null) was unsuccessful (kafka.controller.RequestSendThread)
java.io.IOException: Connection to 192.168.1.47:30102 (id: 1 rack: null) failed.
at org.apache.kafka.clients.NetworkClientUtils.awaitReady(NetworkClientUtils.java:70)
at kafka.controller.RequestSendThread.brokerReady(ControllerChannelManager.scala:279)
at kafka.controller.RequestSendThread.doWork(ControllerChannelManager.scala:233)
at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:82)

Can you help me?