How to access the proxy pod running in cluster externally via a service ?

[ashishgidh]

Question?

Is it possible to access this example externally via a service (exposed via LoadBalancer) ?

https://github.com/grepplabs/kafka-proxy#connect-to-kafka-running-in-kubernetes-example-kafka-proxy-runs-in-cluster

This example works via the kubectl port-forward command but when we try to expose the same pod as a service, it does not work. Any feedback for this ?

What is the best way to do this ? The following example of the deployed proxy pod does not work.
kubectl expose pod/kafka-proxy-0 -n default --type LoadBalancer --name kafka-proxy-exposed

[gustavomcarmo]

Hi @ashishgidh,

Yes, it's possible.

It depends on how you publish your k8s cluster services. In my case, I'm using Ingress.

It's important to set the advertised endpoint in the kafka-proxy --bootstrap-server-mapping parameter, the same endpoint your load balancer or other publishing mechanism will use to publish the service, the same endpoint your Kafka client will use as bootstrap server.

Other requirement is that your load balancer or other publishing mechanism must be able to handle TCP connections, not only HTTP/HTTPS connections.

Just an example of configuration:

          args:
            - 'server'
            - '--bootstrap-server-mapping=kafka-0.internal:9092,0.0.0.0:32401,kafka.grepplabs.com:9092'
            - '--bootstrap-server-mapping=kafka-1.internal:9092,0.0.0.0:32402,kafka.grepplabs.com:9093'
            - '--bootstrap-server-mapping=kafka-2.internal:9092,0.0.0.0:32403,kafka.grepplabs.com:9094'
            - '--dynamic-listeners-disable'

In this case, the service have to expose the ports 9092, 9093 and 9094. The publishing mechanism must resolve kafka.grepplabs.com and forward the traffic to the corresponding service port. The Kafka client will use kafka.grepplabs.com:9092, kafka.grepplabs.com:9093 and kafka.grepplabs.com:9094 as bootstrap servers.

I hope I was helpful 😃

[ashishgidh]

Hi @gustavomcarmo

Thanks for your feedback. It worked for us after exposing the kafka proxy pod via a headless service first and then mapped the ingress TCP section like the following.
9092: "default/kafka-proxy:32400"
9093: "default/kafka-proxy:32401"
9094: "default/kafka-proxy:32402"

Any suggestion for securing it via TLS and Authentication ? Should it be done at the Proxy or Ingress level ?

Thanks a lot.

[gustavomcarmo]

Hi @ashishgidh,

I think the main benefit we get from kafka-proxy is centralizing security in a single component, therefore I would suggest to keep TLS termination and authentication in kafka-proxy itself.

I've been working on a side project for testing kafka-proxy and I'm about to add TLS termination test as well. LDAP authentication is already setup.

Cheers!

[everesio]

Hi @ashishgidh
please also check the #30 discussion.
Kind regards

[ashishgidh]

Hi @everesio

The proxy is working for me according to the mappings. Had a quick question.

I wanted simple client authentication (SASL Plain with TLS) at the proxy level (terminates on the proxy and does not pass on to brokers)

Was not sure what to pass. The following does not find the command argument. Is this incorrect ?

        - '--sasl-jaas-config-file=/etc/jaas/proxy-server-jaas.conf'
        - '--auth-local-enable'
        - '--auth-local-command=build/auth-user'

what will be the right parameters to pass if I'am using the proxy running the cluster example ?

Thanks
@ashishgidh

[gustavomcarmo]

Hi @ashishgidh,

The kafka-proxy Docker image doesn't come with all plugins. You have to build your own kafka-proxy Docker image also compiling the auth-user plugin. I had to do the same to use the auth-ldap plugin, as you can see here. After building your kafka-proxy Docker image with the auth-user plugin, you will be able to use it by setting - --auth-local-command=/auth-user.

[gustavomcarmo]

Hi @everesio,

Do you prefer to let the kafka-proxy users to build their own kafka-proxy Docker images with the plugins they need or would you like to provide the official kafka-proxy Docker image with all the plugins available? I can submit a PR with Dockerfile changes if you want the latter option 😃

[everesio]

Hi @gustavomcarmo,

You are the first really interested in the local auth feature and asking for packaging the plugins in docker image ;-)
Adding the libraries would increase the size of packaged bins from ca. 15 to 83 MB, which is still not a lot.

Current docker pull stats indicate ca. 160k pulls. So it seems the majority does not need it.
Because of that, for now, I would rather prefer to keep the image small.

[ashishgidh]

Hi @ashishgidh,

The kafka-proxy Docker image doesn't come with all plugins. You have to build your own kafka-proxy Docker image also compiling the auth-user plugin. I had to do the same to use the auth-ldap plugin, as you can see here. After building your kafka-proxy Docker image with the auth-user plugin, you will be able to use it by setting - --auth-local-command=/auth-user.

===========
Hi @gustavomcarmo

Just wanted to confirm if we are doing this correctly. We wanted SASL to terminate at the proxy server level so after compiling as above, we passed the following parameters.

        - '--sasl-jaas-config-file=/etc/jaas/proxy-server-jaas.conf'
        - '--auth-local-enable'
        - '--auth-local-command=/auth-user'
        - '--auth-local-param=--username=my-test-user' 
        - '--auth-local-param=--password=my-test-password'    

This seems to be working with the above username/password combination passed as arguments. Can we use the jaas config file to save these credentials ? Not sure of the implementation. If its only via these auth-local-param arguments, how do we secure this and have multiple username/passwords combination ?

Also if we just need to terminate TLS at the proxy server level only, what is the correct combination of arguments to pass ?

[gustavomcarmo]

Hi @ashishgidh,

I tested TLS termination and authentication in kafka-proxy by using the scripts embedded in Kafka 2.5.0 (see Tests).

If you are using the same Java client, you will need to export KAFKA_OPTS="-Djava.security.auth.login.config=/path/to/jaas.conf", pointing to your jaas.conf file, like this.

You will also need to set either the --producer.config or --consumer.config options of the scripts, pointing to your client.properties file, with the content like this.

Don't forget to import into your Kafka client truststore the CA certificate used to sign the kafka-proxy certificate, otherwise the SSL handshake will fail.

I tested the auth-ldap plugin, not the auth-user, so I guess you can set multiple combinations of username and password for all of your users, can't you? For sure @everesio can help you better in this issue.

Cheers!

[ashishgidh]

Hi @everesio

Thanks for your feedback about TLS. Will try those test.

I was not able to pass multiple username/password combinations. It always took the last combination passed.
Not sure what the correct way would be.

Also is it possible to use the jaas file (configured by sasl-jaas-config) on the server side so that we can securely mount it via a config map and keep credentials there ? It does complain even on the auth-user option to have this configured (sasl-jaas-config). But can we maintain credentials here ?

Any help would help in using this basic maintenance of user/pass combination on the kafka proxy side which then can be used to at least authenticate from a client.

Thanks.

[everesio]

sasl-jaas-config is only used when proxy should initiate sasl on its own towards broker (client does not configure sasl).
it is like setting --sasl-username and --sasl-password. it is not used in local authentication.
proxy will pass sasl to brokers if neither sasl by proxy nor local auth (sasl) are configured

[ashishgidh]

sasl-jaas-config is only used when proxy should initiate sasl on its own towards broker (client does not configure sasl).
it is like setting --sasl-username and --sasl-password. it is not used in local authentication.
proxy will pass sasl to brokers if neither sasl by proxy nor local auth (sasl) are configured

=============

Thanks for your feedback @everesio

can we configure multiple username/password combinations for auth-local ?

    - '--auth-local-param=--username=my-test-user' 
    - '--auth-local-param=--password=my-test-password'

[tontondematt]

Hi @gustavomcarmo

Thanks for your feedback. It worked for us after exposing the kafka proxy pod via a headless service first and then mapped the ingress TCP section like the following. 9092: "default/kafka-proxy:32400" 9093: "default/kafka-proxy:32401" 9094: "default/kafka-proxy:32402"

Any suggestion for securing it via TLS and Authentication ? Should it be done at the Proxy or Ingress level ?

Thanks a lot.

can you please share your service , ingress and kafa proxy configurations that made this setup work for you ?