External service for kafka not working #13
Comments
|
The example kafka service If you modified the service, what kubernetes environment do you use? Does it require a Have you tried any of the pods in https://github.com/Yolean/kubernetes-kafka/tree/master/test ? |
|
The access within the pod is OK. Just I want to test expose the port out. It's in my local lab. Not on cloud. So I use NodePort Type. It works well for zookeeper NodePort. But it just failed when accessed outside. What's your suggestion if I want to expose the 9092 port out. |
|
[2016-11-21 15:09:38,373] WARN [console-consumer-72367_JUANLI2-M-G38Q-1479712160584-889927b7-leader-finder-thread], Failed to find leader for Set(test1-0) (kafka.consumer.ConsumerFetcherManager$LeaderFinderThread) It seems I need a LoadBlancer to those brokers |
|
A kafka client that supports broker redirection will work in this scenario, just map the three internal broker endpoints to the one used in your connection string. |
|
Having the same issue. Were you able to figure out how to expose the kafka service @juanli2 using NodePort? |
|
I'm not sure the kafka protocol allows such a thing. See an ongoing investigation in #21. I think consumers must be able to address each broker separately, though it can get the list of brokers from one or more "bootstrap" names. Kubernetes services are inherently round-robin. |
|
I was able to get around this using broker redirection in no-kafka. What happens is that upon connection to a broker, it sends a list of known brokers which then replace whatever endpoint you used to connect with. These are the kubernetes cluster endpoints, however, which will not be the same as what the consumer is connecting with. The broker redirection feature lets you map the external endpoint with the internal one. |
|
Ahh right on, let me try that, and will report back! |
|
@donstaheli Have you tried that setup with a topic that has fewer replicas than you have brokers? |
|
No, I haven't. What is the use case demanding that can't be met with partitioning? |
|
It's more of a gotcha than a use case. In #21 we suspect that when consumers can't resolve individual brokers (internal or external services alike) they will not know how to locate a topic unless it lives on the broker they happen to hit. I don't think a client lib can solve that, because k8s services are round-robin. There's a discussion about that in kubernetes/kubernetes#36415. |
|
Sorry for the late update, but I looked at the broker redirection no-kafka and this is not what I'm looking for. So, my question is similar to @juanil2. Let me explain. I used the repo to create the zookeeper and Kafka statefulsets. I was successful creating everything as advertised. I am able to access Kafka by either hitting the broker-0.broker.... and so forth or by using the Kafka.default.svc.... I then went ahead and modified the 30service.yml with type: NodePort and attempt to hit this service externally to the k8 cluster, and I get the same error @juanil2 reported. If I understand nodeport services correctly, this should work? Btw, my k8 cluster is in not in a cloud environment. Any ideas ? |
|
I think the no-kafka docs on remapping confirm my position that Kafka isn't meant to be used through a round-robin load balancer. It might work within a cluster though, where you can - depending on config - resolve the addresses to individual brokers that you get from your initial connection through the service. |
|
You could try one service + NodePort per kafka broker, together with remapping. |
|
Guys, have you figured out how to connect to kafka using AWS classic ELB? I receive I have TCP mapping on nginx ingress of port 9092 to service kafka/kafka:9092 data: Also zookeeper is not accessible from ELB. Internally it works fine: thanks for any help |
|
Has anyone ever got this working with an external service? |
|
I don't think so. With a single broker it might be straightforward. With >1 you'd probably have to expose each broker externally and set advertised.listeners individually. |
|
So I managed to get messages coming in, using a setup like this: And changing 50kafka.yml (note I am using your 0.11 branch of the project) I can connect externally and produce messages, but for some reason the Kafka cluster under replicates topics and I get a lot of this in the Kafka logs: Do you have any ideas why this might be happening? Ben |
|
I guess brokers won't be able to resolve each other at the |
|
BOSH! That was it, but instead of using advertising listeners I used listeners (listeners is for internal brokers) so it now look like this: Output from sangrenel before (note the error rate): After: So the only thing left is to make the advertised.listeners get it's value from the service (which I can work out) |
|
Nice! Why is it |
|
Yeah I have no idea why that worked, as when I killed the cluster and tried
again, two of the Kafka brokers kept dying....
…On 20 Jul 2017 8:37 pm, "solsson" ***@***.***> wrote:
Nice!
Why is it --override listeners=PLAINTEXT://${HOSTNAME##*-}:9092? I would
have guessed instead --override listeners=PLAINTEXT://kafka-${
HOSTNAME##*-}:9092.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<#13 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AAGbm3tvmBmQVcDeL6GZtb3-g7iuPnvIks5sP6yIgaJpZM4K2QeP>
.
|
Note that in the |
|
Got it working 😄 Changed the command: to: --override log.retention.hours=-1
--override log.dirs=/var/lib/kafka/data/topics
--override broker.id=${HOSTNAME##*-}
--override listener.security.protocol.map=INTERNAL_PLAINTEXT:PLAINTEXT,EXTERNAL_PLAINTEXT:PLAINTEXT
--override advertised.listeners="INTERNAL_PLAINTEXT://${HOSTNAME}.broker.kafka.svc.cluster.local:9092,EXTERNAL_PLAINTEXT://$(eval wget -t3 -T2 -qO- http://169.254.169.254/latest/meta-data/public-hostname):9093"
--override listeners=INTERNAL_PLAINTEXT://0.0.0.0:9092,EXTERNAL_PLAINTEXT://0.0.0.0:9093
--override inter.broker.listener.name=INTERNAL_PLAINTEXT
--override auto.create.topics.enable=true # Just our internal config
--override auto.leader.rebalance.enable=true # Just our internal config
--override num.partitions=3 # Just our internal config
--override default.replication.factor=3 # Just our internal configYou then need to change the ports: to this: Finally you need to open up security groups if needed. You can then connect using ec2-blah.us-blah.com:9093 without dropping any messages: So this is just for AWS, but for anything else just change this line: Thanks, Ben |
|
I'd like to make an "addon" PR with this, but let me get the 0.11 branch ready for merge first. What kind of access control do you plan on using? I'm curious if it can affect the choice of listeners. |
|
For us at the moment, PLAINTEXT, but we will be moving towards a more secure method soon(ish) |
|
Hi @solsson, as I understand you have added code of @BenjaminDavison and now not need add all --override options which he did, just need to modify 10broker-config.yml. Does my configuration correct: listeners=INTERNAL_PLAINTEXT://0.0.0.0:9092,EXTERNAL_PLAINTEXT://0.0.0.0:9093 advertised.listeners="INTERNAL_PLAINTEXT://${HOSTNAME}.broker.kafka.svc.cluster.local:9092,EXTERNAL_PLAINTEXT://$(eval wget -t3 -T2 -qO- http://169.254.169.254/latest/meta-data/public-hostname):9093" listener.security.protocol.map=INTERNAL_PLAINTEXT:PLAINTEXT,EXTERNAL_PLAINTEXT:PLAINTEXT Because I get exception: 2017-07-31T13:42:34.268980421Z java.lang.IllegalArgumentException: Error creating broker listeners from '"INTERNAL_PLAINTEXT://${HOSTNAME}.broker.kafka.svc.cluster.local:9092,EXTERNAL_PLAINTEXT://$(eval wget -t3 -T2 -qO- http://169.254.169.254/latest/meta-data/public-hostname):9093"': Unable to parse "INTERNAL_PLAINTEXT://${HOSTNAME}.broker.kafka.svc.cluster.local:9092 to a broker endpoint |
|
Bah, looks like Kafka schema registry doesn't support the listener security protocol map
|
|
Maybe it doesn't need to? It should always be on the inside, I guess. Can we simply rearrange the list? Edit: ah, just saw that that error can't be from the PR. The PR keeps PLAINTEXT for inside, hence schema registry might be unaffected. |
|
Sorry yes, that was slightly off topic as looks like just a schema registry limitation at the moment. I've worked around it for now by configuring my listeners as "PLAINTEXT" and "EXTERNAL_PLAINTEXT". |
|
... however the separate external/named listeners approach will not work for all Kafka clients that are external to the k8s cluster. E.g. Kafka Tool - its able to obtain the Kafka cluster info via ZK, but errors trying to load topics and consumers, complaining that it can't reach the broker. I suspect this is because its trying to connect specifically to the listener named "PLAINTEXT" as discovered from ZK (rather like schema registry does, but that's ok if its running in the same k8s cluster), which can't be accessed from outside. Kafka tool worked with a single node, one advertised listener address:port that was reachable both inside and outside. My own producer/consumer clients running outside the cluster do work with a broker/bootstrap config that initially talks to any broker and somehow the response is clever enough to only advertise the external addresses at that point. I'm still not confident I fully understand whats going on in these different cases though! I could perhaps rework configuration to go back to using the same FQDN and port internally and externally when I have time to experiment. |
I think that type of client is deprecated. Maybe the introduction of multiple listeners was a reason for such a deprecation. I guess the host used as bootstrap must be matched to one of the listener types internally, in order to return broker hosts of the same kind. Also I don't see a reason why Zookeeper should have an external service. |
|
The only reason I saw the need to expose zookeeper was because we're using Sarama for the broker clients and there's no way to create topics from that client. So we were going to use the kafka tools (which last I checked did need access to Zookeeper). We've gone with a different approach entirely (a REST based service) which allows us more control over that process. Exposing Zookeeper outside of Kubernetes seems to be fraught with security issues that I'd rather not entertain. |
|
If you're working on this and don't understand why your Edit: Calico actually now enables portmap by default, so it works. |
|
Fixed with the merge of #78. Alongs these I've been thinking that it would be quite easy to set up kafka access over one |
|
I solved it in the following way using port-forward and without having to expose kafka to outside world (tiny modification of the solsson/kafka setup): My init.sh ConfigMap Broker exposes 9094 and then you get OUTSIDE addresses of your nodes with ports 3240 + "ID OF BROKER" so in case of 3 node setup this would be 32400 32401 32402. Then i create 4 port forwards This way when client asks for list of nodes from a broker at localhost:9094 it will say: Hey I have 3 nodes their addresses are as follows: Which is correct because porf-forward maps them correctly from your local machines. And you're ready to debug. Producer is set to localhost:9094. Consumers are set to high ports with corresponding client id equal to last integer of the high number so client-0 to 32400, client-1 to 32401, client-2 to 32402 just so i can make sense of which one is which. I used 3 consumers to test the group consumption and it works fine. To be honest I just want to debug my code and then run everything inside as standard 9092 but it took me way too much time to solve this (thanks to this thread as well). Inside/Outside setup is very cool but I think many people are confused or don't know that broker setting is just to get a list of nodes and that list is produced from how you launch kafka nodes themselves. Inside/Outside mapped on different ports 9092 (regular) 9094 (outside) is a good way to have dual nodelist available for setup. It would be beneficial to document [outside] ability so people can write/debug code against kafka from their local workstations. |
|
@styk-tv Thanks for sharing this. I've also struggled with Kafka's listener model. I've set up mirror-maker using docker-compose with the 9092 ports by mapping brokers' kubernetes names to local containers. See https://gist.github.com/solsson/22e773ab148596ec39d54b7cc153ee67. Like your solution it is based on kubectl port-forward, which is complicated in containers until kubernetes/kubernetes#46517 gets merged. With #172 it could be easier. Using Updating Broker Configs #78 can probably be implemented (and customized) in a way that is persistent in ZooKeeper and takes precedence over server.properties. |
|
is there a branch with external broker endpoints? |
|
@solsson thanks for the answer |
|
On the same deployment I have another service configured similarly which I can access from browser and curl. so I suspect that it might be a Kafka related issue and not kubernetes |
|
curl -s http://localhost:30120/sso/secured/healthcheck/status
The services are the same pattern |
|
You can see a full description here: |
|
This port acts differently to curl curl: (52) Empty reply from server curl: (7) Failed to connect to localhost port 32406: Connection refused |
|
but a producer can't connect |
|
I deployed to GKE too on different nodes to no avail |
|
Hi, I have a problem exposing my kafka broker to external public using the classic load balancer. Here is my startup command My Statefulset manifest :
Now when I try to consme messages from the external DNS and when I try to publish a message, I get an error: Also, I do not see any errors in my kafka logs. I also tried forsy creating a Load balancer service and then using its DNS name as an If anybody could help me, it would be really great. Kindly let me know if more details are required. Thanks! |
|
@ridicule777 Does this differ from the other cases of name resolution outside cluster that have been discussed in this repo several times? Clients must be able to access all kafka brokers on individual listener addresses known to kafka. |
Hi,
I tried to use a NodePort type to expose the kafka service out.
Attached my service yml file.
And here's the service descrition:
$ kubectl describe svc kafka -n kafka
Name: kafka
Namespace: kafka
Labels:
Selector: app=kafka
Type: NodePort
IP: 100.73.225.207
Port: 9092/TCP
NodePort: 32093/TCP
Endpoints: 10.44.0.10:9092,10.44.0.11:9092,10.44.0.12:9092
Session Affinity: None
But when I tried using port 32039 external to connect to the kafka service it seems not working.
$ ./bin/kafka-console-producer.sh --broker-list ${master-ip}:32093 --topic test2
abc
[2016-11-18 15:26:58,157] ERROR Error when sending message to topic test2 with key: null, value: 3 bytes with error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
org.apache.kafka.common.errors.TimeoutException: Expiring 1 record(s) for test2-0 due to 1512 ms has passed since batch creation plus linger time
I'm pretty sure that the connection to the master-ip is working, and the port 32039 is listening in the cluster.
It works for the zookeeper, but I'm not sure why the kafka not working
The text was updated successfully, but these errors were encountered: