Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[inclubator/kafka] Accessing Kafka outside kubernetes not working #6670

Closed
vidhy opened this Issue Jul 17, 2018 · 5 comments

Comments

Projects
None yet
2 participants
@vidhy
Copy link

vidhy commented Jul 17, 2018

Is this a request for help?:
Yes


Is this a BUG REPORT or FEATURE REQUEST? (choose one):
Bug

Version of Helm and Kubernetes:
Helm version
Client: &version.Version{SemVer:"v2.9.1", GitCommit:"20adb27c7c5868466912eebdf6664e7390ebe710", GitTreeState:"clean"}
Server: &version.Version{SemVer:"v2.9.1", GitCommit:"20adb27c7c5868466912eebdf6664e7390ebe710", GitTreeState:"clean"}

Kubernetes version
Client Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.6", GitCommit:"9f8ebd171479bec0ada837d7ee641dec2f8c6dd1", GitTreeState:"clean", BuildDate:"2018-03-21T15:21:50Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.6", GitCommit:"9f8ebd171479bec0ada837d7ee641dec2f8c6dd1", GitTreeState:"clean", BuildDate:"2018-03-21T15:13:31Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}

Which chart:
[inclubator/kafka]

What happened:
Accessing kafka from outside note working as expected.

What you expected to happen:
Kafka broker should be accessible at kafka.k8s.cluster.vpc:31091

How to reproduce it (as minimally and precisely as possible):
Follwed the given steps for helm chart setup of kafka. with external enabled

external:
  enabled: true
  domain: 'k8s.cluster.vpc'

Anything else we need to know:
No issue of connectivity between the instances. Similarly the Route53 A records are getting created properly.

kc describe svc kafka-1-external -n test-kafka

Name:                     kafka-1-external
Namespace:                test-kafka
Labels:                   app=kafka
                          chart=kafka-0.8.3
                          heritage=Tiller
                          pod=kafka-1
                          release=kafka
Annotations:              dns.alpha.kubernetes.io/internal=kafka.k8s.groupplatform.vpc
Selector:                 app=kafka,pod=kafka-1,release=kafka
Type:                     NodePort
IP:                       100.70.219.240
Port:                     external-broker  19092/TCP
TargetPort:               31091/TCP
NodePort:                 external-broker  31091/TCP
Endpoints:                100.115.0.28:31091
Session Affinity:         None
External Traffic Policy:  Cluster
Events:                   <none>

Since the target port is 31091, the port should be exposed from the pod. But when i do telnet to the corresponding port connection refused.

When I exec inside the kafka-0 pod, and check the /etc/kafka/kafka.properties, it has the listeners as

advertised.listeners=PLAINTEXT://100.120.0.5:9092
listeners=PLAINTEXT://0.0.0.0:9092

@vidhy vidhy changed the title [inclubator/kafka] [inclubator/kafka] Accessing Kafka outside kubernetes not working Jul 17, 2018

@drekle

This comment has been minimized.

Copy link
Contributor

drekle commented Jul 20, 2018

Hi @vidhy

I believe you must also specify configurationOverrides.advertised.listeners. Guessing for your example it would be this...

configurationOverrides:
  advertised.listeners: '&& export KAFKA_ADVERTISED_LISTENERS=EXTERNAL://k8s.cluster.vpc:$((31090
    + ${KAFKA_BROKER_ID})),PLAINTEXT://${POD_IP}:9092'

Noticed from looking at the statefulset template.

@vidhy

This comment has been minimized.

Copy link
Author

vidhy commented Jul 25, 2018

@drekle I dont think that fixes the issue. These are the properties I have inside the pod.

advertised.listeners=EXTERNAL://k8s.groupplatform.vpc:31090,PLAINTEXT://100.120.0.9:9092
listener.security.protocol.map=PLAINTEXT:PLAINTEXT,EXTERNAL:PLAINTEXT
listeners=EXTERNAL://0.0.0.0:31090,PLAINTEXT://0.0.0.0:9092

When i do
bin/kafka-console-producer.sh --broker-list test-kafka.k8s.groupplatform.vpc:31090 --topic my-topic

I am getting

[2018-07-25 11:19:44,717] ERROR Error when sending message to topic my-topic with key: null, value: 2 bytes with error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
org.apache.kafka.common.errors.TimeoutException: Expiring 1 record(s) for my-topic-0: 60020 ms has passed since batch creation plus linger time

@vidhy

This comment has been minimized.

Copy link
Author

vidhy commented Jul 25, 2018

I am able to produce and consume properly inside the kubernetes cluster and access broker via "test-kafka.test-kafka.svc.cluster.local:9092". So the issue is with external access of kafka alone.

@drekle

This comment has been minimized.

Copy link
Contributor

drekle commented Jul 25, 2018

@vidhy I told you wrong above. The advertised listener should be the FQDN. You are calling it with test-kafka.k8s.groupplatform.vpc:31090 yet it is listening to k8s.groupplatform.vpc:31090. This was my mistake in my mention. Can you try changing it to test-kafka.k8s.groupplatform.vpc:31090?

configurationOverrides:
  advertised.listeners: '&& export KAFKA_ADVERTISED_LISTENERS=EXTERNAL://test-kafka.k8s.groupplatform.vpc:$((31090
    + ${KAFKA_BROKER_ID})),PLAINTEXT://${POD_IP}:9092'
@vidhy

This comment has been minimized.

Copy link
Author

vidhy commented Jul 25, 2018

that worked like a charm. thanks @drekle Adding the listener config changes in docs will be helpful for novice users.

@vidhy vidhy closed this Jul 25, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.
You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session.