Unable to Consume Kafka Events using an External AWS NLB Type Load Balancer.

[vikrantsundriyal]

Name and Version

bitnami/kafka 14.0.2

What steps will reproduce the bug?

Here are the steps performed for Kafka setup and reproduce issue.

1. Launch an EKS Cluster (Default).
2. Install Kafka using bitnami commands: helm install my-release bitnami/kafka
3. I want to consume topics externally. To achieve expected connectivity, i launched a new service of type LoadBalance. Using annotations i was able to launch LB as NLB in aws and see instances as healthy (connected to kafka pod ip) at 9092.
   Service definition file:

apiVersion: v1
kind: Service
metadata:
  name: kafka-broker-external-new0205
  labels:
    app: nginx
    app.kubernetes.io/component: kafka
    app.kubernetes.io/instance: my-release
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: kafka
    helm.sh/chart: kafka-16.2.7      
  annotations: 
    service.beta.kubernetes.io/aws-load-balancer-type: "external"
    service.beta.kubernetes.io/aws-load-balancer-name: "nlb-kafka-new0205"
    service.beta.kubernetes.io/aws-load-balancer-backend-protocol: "tcp"
    service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: "true"
    service.beta.kubernetes.io/aws-load-balancer-healthcheck-port: "traffic-port"
    service.beta.kubernetes.io/aws-load-balancer-internal: "true"
    service.beta.kubernetes.io/aws-load-balancer-ip-address-type: "ipv4"
    service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: "ip"
    service.beta.kubernetes.io/load-balancer-source-ranges: "xxx.xx.0.0/16"
    service.beta.kubernetes.io/aws-load-balancer-private-ipv4-addresses: "xxx.xx.2.112, xxx.xx.3.112"
spec:
  type: LoadBalancer
  allocateLoadBalancerNodePorts: false
  selector:
    app.kubernetes.io/component: kafka
    app.kubernetes.io/instance: my-release
    app.kubernetes.io/name: kafka
  ports:
  - name: external
    protocol: TCP
    port: 9092
    targetPort: 9092

5. After the above setup, we are unable to consume events from a topic.
6. Please suggest whether above steps are correct OR I need to follow something else.
7. Going through the docs, i could see below configuration to enable external access.

externalAccess.enabled=true
externalAccess.service.type=LoadBalancer
externalAccess.service.ports.external=9094
externalAccess.autoDiscovery.enabled=true
serviceAccount.create=true
rbac.create=true

Here are Some queries-

1. how will be the setup after passing below attributes: will it create a new service of type LB & AWS resource (which i guess not)? Do i need to create NLB on top of this. My external use is within VPC so i dont want to create public LB.
2. If i have to use recommended method with below params, where should i place them in values.yaml.

externalAccess.enabled=true
externalAccess.service.type=LoadBalancer
externalAccess.service.ports.external=9094
externalAccess.autoDiscovery.enabled=true
serviceAccount.create=true
rbac.create=true

Am getting an error **"Error: INSTALLATION FAILED: template: kafka/templates/statefulset.yaml:10:59: executing "kafka/templates/statefulset.yaml" at <.Values.externalAccess. autoDiscovery.enabled>: can't evaluate field enabled in type interface {}"**
If i add in below manner:

externalAccess:
  enabled: true
  service:
    type: LoadBalancer
    ports:
external:9094
  autoDiscovery:
    enabled:true
serviceAccount:
  create: true
rbac:
  create: true

Are you using any custom parameters or values?

No

What is the expected behavior?

I should be able to consume events

What do you see instead?

Not receiving events

Additional information

In my setup i use AWS lambda trigger (which has OOTB feature to poll kafka events).
So the connectivity goes like this-

Kafka-pods---> AWS NLB --> AWS Lambda EVentSource.
NLB to EventSource connectivity is working fine.

[rafariossaa]

Hi,
I think in your case that error means there are some sintax error in the yaml file. Could you recheck the yaml file ?

[vikrantsundriyal]

Thanks for your response, yaml syntax - I will reiterate my queries.

1. What all resources it will create.
2. Am i configuring at right place. I just took a vanila values.yaml and added above four-five configurations under below section. Is this correct.

External Access to Kafka brokers configuration

externalAccess:

@param externalAccess.enabled Enable Kubernetes external cluster access to Kafka brokers

COMING to syntax error, wont i get syntax error for wrong yaml??
I think error "Error: INSTALLATION FAILED: template: kafka/templates/statefulset.yaml:10:59: executing "kafka/templates/statefulset.yaml" at <.Values.externalAccess.
autoDiscovery.enabled>: can't evaluate field enabled in type interface {}" means something is missing which is not mentioned in document.

[rafariossaa]

Thanks for your response, yaml syntax - I will reiterate my queries.

1. What all resources it will create.

The chart only creates a LB at kubernetes level, it is now aware if it is on AWS, GCE or others, so if you need to create resource in those platforms you have to do it by your own.

2. Am i configuring at right place. I just took a vanila values.yaml and added above four-five configurations under below section. Is this correct.

Yes, that is the place to configure LB in this chart.

[vikrantsundriyal]

Thanks @rafariossaa .
Basic setup worked for us. Am able to launch a LoadBalancer of Type Classic with listener configurations.
Just couple of final queries in this thread:

1. Is there a way to enable Network Load Balancer instead of Classic one?
2. Instead of using a Public LoadBalancer, is there a way to launch it as Private load balancer? In our case LB has to be invoked internally (within VPC).
3. Can we control the security group of LoadBalancer while provisioning?

Thanks once again for your prompt responses.

[rafariossaa]

Hi,
Maybe there is some very specific configuration for the k8s where you are deploying it. In this case my advice is to ask in AWS forums.
Also, maybe worth it for you to check what is possible at k8s level. I think this link could help.

[github-actions]

This Issue has been automatically marked as "stale" because it has not had recent activity (for 15 days). It will be closed if no further activity occurs. Thanks for the feedback.

[github-actions]

Due to the lack of activity in the last 5 days since it was marked as "stale", we proceed to close this Issue. Do not hesitate to reopen it later if necessary.