Q: Connect to brokers with PEM and KEY

[jfsanchez91]

I would like to know how can I run/configure the kafka-proxy cli to connect to a cluster (brokers) asking for a certificate.

I have the PEM and KEY files, and I can connect using Java with a custom Keystore with those files.

How can I replicate this on the kafka-proxy client.

[everesio]

You can both connect to brokers using TLS and terminate TLS traffic on the proxy.
Any combination is possible. PEM files are required for configuration.
Keys can be also encrypted.

For TLS termination please check

--proxy-listener-tls-enable
--proxy-listener-ca-chain-cert-file
--proxy-listener-cert-file
--proxy-listener-key-file
--proxy-listener-key-password

To initiate TLS connection to brokers check

--tls-enable                
--tls-ca-chain-cert-file
--tls-client-cert-file
--tls-client-key-file
--tls-client-key-password
--tls-insecure-skip-verify

[pufffin]

same question here. trying to connect to kafka via client certificate. assuming that --tls-* are the right flags (proxy <-> kafka not client <-> proxy). ending up with a tls: unexpected message.
settings:
kafka-proxy server --tls-enable --tls-client-cert-file certs/client.cer --tls-client-key-file certs/private.key --tls-client-key-password password --tls-ca-chain-cert-file certs/ca.cer --tls-insecure-skip-verify

is it a problem with the certificates? little bit stuck here... error message does not help too much :D

[everesio]

Could you provide output from "openssl s_client -connect broker:port" and also
try "GODEBUG=tls13=0 kafka-proxy ....." ?

tcpdump would be useful as well.

[pufffin]

CONNECTED(00000005)
depth=2 DC = *, DC = *, DC = *, CN = *
verify error:num=19:self signed certificate in certificate chain
verify return:0
4712840812:error:1401E0F4:SSL routines:CONNECT_CR_FINISHED:unexpected message:/AppleInternal/BuildRoot/Library/Caches/com.apple.xbs/Sources/libressl/libressl-47.100.4/libressl-2.8/ssl/ssl_both.c:510:
---
SSL handshake has read 10430 bytes and written 169 bytes
---
New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-GCM-SHA384
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : ECDHE-RSA-AES256-GCM-SHA384
    Session-ID: ....
    Session-ID-ctx:
    Master-Key: ....
    Start Time: 1592298312
    Timeout   : 7200 (sec)
    Verify return code: 19 (self signed certificate in certificate chain)
---

[pufffin]

Flags [.], seq 19616:21056, ack 416, win 15552, options [nop,nop,TS val 3671781351 ecr 374961307], length 1440
Flags [R], seq 862807745, win 0, length 0
Flags [FP.], seq 21056:22144, ack 416, win 15552, options [nop,nop,TS val 3671781351 ecr 374961307], length 1088
Flags [R], seq 2631340709, win 0, length 0
Flags [R], seq 862807745, win 0, length 0
Flags [R], seq 2631340709, win 0, length 0

[everesio]

have you tried to disable tls 1.3 with "export GODEBUG=tls13=0" ?

[pufffin]

yes, no other result

#!/usr/bin/env bash

clientCert=certs/client.cer
clientKey=certs/private.key
clientKeyPwd=test

export GODEBUG=tls13=0a


tlsFlags="--tls-enable --tls-client-cert-file ${clientCert} --tls-client-key-file ${clientKey} --tls-client-key-password ${clientKeyPwd} --tls-ca-chain-cert-file  ${dummyServerCert}"
#proxyTlsFlags="--proxy-listener-tls-enable --proxy-listener-ca-chain-cert-file $serverCert --proxy-listener-cert-file $clientCert --proxy-listener-key-file $clientKey --proxy-listener-key-password $clientKeyPwd"

./kafka-proxy server --kafka-client-id 35466_twin_acceptance ${tlsFlags} --bootstrap-server-mapping $bootstrapMapping

[everesio]

tls13=0a ?
it should be "tls13=0"

[pufffin]

was just typo while copying... still not working. how can i see that it really isnt using tls13?

[pufffin]

well, I fixed one problm with openssl s_client -connect kafka2.dev.adp.cloud.audi:19093 -CAfile certs/root.cer and got rid of this message

depth=2 DC = *, DC = *, DC = *, CN = *
verify error:num=19:self signed certificate in certificate chain
verify return:0

but still got:

verify return:0
4567953004:error:1401E0F4:SSL routines:CONNECT_CR_FINISHED:unexpected message:/AppleInternal/BuildRoot/Library/Caches/com.apple.xbs/Sources/libressl/libressl-47.100.4/libressl-2.8/ssl/ssl_both.c:510:

[everesio]

Are you sure that your client certificate used by kafka-proxy is correct ?

you can check it with e.g. wireshark / tcpdump. if it is incorrect, sent cert will have length 0.

Certificate, Client Key Exchange, Change Cipher Spec, Encrypted Handshake Message	

Transport Layer Security
    TLSv1.2 Record Layer: Handshake Protocol: Certificate
        Content Type: Handshake (22)
        Version: TLS 1.2 (0x0303)
        Length: 7
        Handshake Protocol: Certificate
            Handshake Type: Certificate (11)
            Length: 3
            Certificates Length: 0
    TLSv1.2 Record Layer: Handshake Protocol: Client Key Exchange
    TLSv1.2 Record Layer: Change Cipher Spec Protocol: Change Cipher Spec
    TLSv1.2 Record Layer: Handshake Protocol: Encrypted Handshake Message

[pufffin]

the certificate should be fine, we're able to use it from a java client. but I'll try to reach the kafka maintainer for help. thanks for your in-depth answers. If I find a solution I'll post it here :)

[pufffin]

ok, last shot for today and we found it -> we had a p12 and exported the client certificate. but without intermediate / not the complete chain. we did that with keyStoreExplorer and now we re connected! thx for the help!