Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[inclubator/kafka] Accessing Kafka outside kubernetes not working #6670

Closed
vidhy opened this issue Jul 17, 2018 · 8 comments
Closed

[inclubator/kafka] Accessing Kafka outside kubernetes not working #6670

vidhy opened this issue Jul 17, 2018 · 8 comments

Comments

@vidhy
Copy link

@vidhy vidhy commented Jul 17, 2018

Is this a request for help?:
Yes


Is this a BUG REPORT or FEATURE REQUEST? (choose one):
Bug

Version of Helm and Kubernetes:
Helm version
Client: &version.Version{SemVer:"v2.9.1", GitCommit:"20adb27c7c5868466912eebdf6664e7390ebe710", GitTreeState:"clean"}
Server: &version.Version{SemVer:"v2.9.1", GitCommit:"20adb27c7c5868466912eebdf6664e7390ebe710", GitTreeState:"clean"}

Kubernetes version
Client Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.6", GitCommit:"9f8ebd171479bec0ada837d7ee641dec2f8c6dd1", GitTreeState:"clean", BuildDate:"2018-03-21T15:21:50Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.6", GitCommit:"9f8ebd171479bec0ada837d7ee641dec2f8c6dd1", GitTreeState:"clean", BuildDate:"2018-03-21T15:13:31Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}

Which chart:
[inclubator/kafka]

What happened:
Accessing kafka from outside note working as expected.

What you expected to happen:
Kafka broker should be accessible at kafka.k8s.cluster.vpc:31091

How to reproduce it (as minimally and precisely as possible):
Follwed the given steps for helm chart setup of kafka. with external enabled

external:
  enabled: true
  domain: 'k8s.cluster.vpc'

Anything else we need to know:
No issue of connectivity between the instances. Similarly the Route53 A records are getting created properly.

kc describe svc kafka-1-external -n test-kafka

Name:                     kafka-1-external
Namespace:                test-kafka
Labels:                   app=kafka
                          chart=kafka-0.8.3
                          heritage=Tiller
                          pod=kafka-1
                          release=kafka
Annotations:              dns.alpha.kubernetes.io/internal=kafka.k8s.groupplatform.vpc
Selector:                 app=kafka,pod=kafka-1,release=kafka
Type:                     NodePort
IP:                       100.70.219.240
Port:                     external-broker  19092/TCP
TargetPort:               31091/TCP
NodePort:                 external-broker  31091/TCP
Endpoints:                100.115.0.28:31091
Session Affinity:         None
External Traffic Policy:  Cluster
Events:                   <none>

Since the target port is 31091, the port should be exposed from the pod. But when i do telnet to the corresponding port connection refused.

When I exec inside the kafka-0 pod, and check the /etc/kafka/kafka.properties, it has the listeners as

advertised.listeners=PLAINTEXT://100.120.0.5:9092
listeners=PLAINTEXT://0.0.0.0:9092
@vidhy vidhy changed the title [inclubator/kafka] [inclubator/kafka] Accessing Kafka outside kubernetes not working Jul 17, 2018
@drekle
Copy link
Contributor

@drekle drekle commented Jul 20, 2018

Hi @vidhy

I believe you must also specify configurationOverrides.advertised.listeners. Guessing for your example it would be this...

configurationOverrides:
  advertised.listeners: '&& export KAFKA_ADVERTISED_LISTENERS=EXTERNAL://k8s.cluster.vpc:$((31090
    + ${KAFKA_BROKER_ID})),PLAINTEXT://${POD_IP}:9092'

Noticed from looking at the statefulset template.

@vidhy
Copy link
Author

@vidhy vidhy commented Jul 25, 2018

@drekle I dont think that fixes the issue. These are the properties I have inside the pod.

advertised.listeners=EXTERNAL://k8s.groupplatform.vpc:31090,PLAINTEXT://100.120.0.9:9092
listener.security.protocol.map=PLAINTEXT:PLAINTEXT,EXTERNAL:PLAINTEXT
listeners=EXTERNAL://0.0.0.0:31090,PLAINTEXT://0.0.0.0:9092

When i do
bin/kafka-console-producer.sh --broker-list test-kafka.k8s.groupplatform.vpc:31090 --topic my-topic

I am getting

[2018-07-25 11:19:44,717] ERROR Error when sending message to topic my-topic with key: null, value: 2 bytes with error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
org.apache.kafka.common.errors.TimeoutException: Expiring 1 record(s) for my-topic-0: 60020 ms has passed since batch creation plus linger time

@vidhy
Copy link
Author

@vidhy vidhy commented Jul 25, 2018

I am able to produce and consume properly inside the kubernetes cluster and access broker via "test-kafka.test-kafka.svc.cluster.local:9092". So the issue is with external access of kafka alone.

@drekle
Copy link
Contributor

@drekle drekle commented Jul 25, 2018

@vidhy I told you wrong above. The advertised listener should be the FQDN. You are calling it with test-kafka.k8s.groupplatform.vpc:31090 yet it is listening to k8s.groupplatform.vpc:31090. This was my mistake in my mention. Can you try changing it to test-kafka.k8s.groupplatform.vpc:31090?

configurationOverrides:
  advertised.listeners: '&& export KAFKA_ADVERTISED_LISTENERS=EXTERNAL://test-kafka.k8s.groupplatform.vpc:$((31090
    + ${KAFKA_BROKER_ID})),PLAINTEXT://${POD_IP}:9092'
@vidhy
Copy link
Author

@vidhy vidhy commented Jul 25, 2018

that worked like a charm. thanks @drekle Adding the listener config changes in docs will be helpful for novice users.

@vidhy vidhy closed this Jul 25, 2018
@Triasto
Copy link

@Triasto Triasto commented May 22, 2019

@vidhy I told you wrong above. The advertised listener should be the FQDN. You are calling it with test-kafka.k8s.groupplatform.vpc:31090 yet it is listening to k8s.groupplatform.vpc:31090. This was my mistake in my mention. Can you try changing it to test-kafka.k8s.groupplatform.vpc:31090?

configurationOverrides:
  advertised.listeners: '&& export KAFKA_ADVERTISED_LISTENERS=EXTERNAL://test-kafka.k8s.groupplatform.vpc:$((31090
    + ${KAFKA_BROKER_ID})),PLAINTEXT://${POD_IP}:9092'

OMG .. thank you thank you so much!

I've been troubleshooting all day to find this answer, works like a charm now!

@pujik16
Copy link

@pujik16 pujik16 commented Aug 21, 2019

Hi @drekle,

We configured Kafka using incubator Helm charts on rancher Kubernetes cluster by updating values.yml with below properties and using helm install --name test-kafka -f values.yml incubator/kafka --namespace test

  1. enabled:true
  2. advertised.listeners: '&& export KAFKA_ADVERTISED_LISTENERS=EXTERNAL://test-kafka.cluster.local:$((31090
    • ${KAFKA_BROKER_ID})),PLAINTEXT://${POD_IP}:9092'
  3. uncommented listener.security.protocol.map

We are able to successfully create topics within cluster. But we still have issues in exposing Kafka outside cluster. Can you please help

@ceyhunn
Copy link

@ceyhunn ceyhunn commented Sep 3, 2019

Hi @drekle , @vidhy

I have following config for external

replicas: 3
zookeeper:
  replicaCount: 3
external:
  enabled: true
  type: NodePort
  servicePort: 9092
  firstListenerPort: 30101
configurationOverrides:
  listeners: "PLAINTEXT://:9092"
  "advertised.listeners": '&& export KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://192.168.1.47:$((30101 + ${KAFKA_BROKER_ID}))'
  "listener.security.protocol.map": "PLAINTEXT:PLAINTEXT"

Then all containers are up, but brokers can't communicate each other:

java.io.IOException: Connection to 192.168.1.47:30101 (id: 0 rack: null) failed.
at org.apache.kafka.clients.NetworkClientUtils.awaitReady(NetworkClientUtils.java:70)
at kafka.controller.RequestSendThread.brokerReady(ControllerChannelManager.scala:279)
at kafka.controller.RequestSendThread.doWork(ControllerChannelManager.scala:233)
at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:82)
[2019-09-03 10:45:09,155] WARN [Controller id=2, targetBrokerId=1] Connection to node 1 could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)
[2019-09-03 10:45:09,155] WARN [RequestSendThread controllerId=2] Controller 2's connection to broker 192.168.1.47:30102 (id: 1 rack: null) was unsuccessful (kafka.controller.RequestSendThread)
java.io.IOException: Connection to 192.168.1.47:30102 (id: 1 rack: null) failed.
at org.apache.kafka.clients.NetworkClientUtils.awaitReady(NetworkClientUtils.java:70)
at kafka.controller.RequestSendThread.brokerReady(ControllerChannelManager.scala:279)
at kafka.controller.RequestSendThread.doWork(ControllerChannelManager.scala:233)
at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:82)

Can you help me?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
5 participants