SASL_SSL: verify failed: broker certificate could not be verified #345

JimDunne · 2021-09-30T17:25:48Z

$ kcat -V
kcat - Apache Kafka producer and consumer tool
https://github.com/edenhill/kcat
Copyright (c) 2014-2021, Magnus Edenhill
Version 1.7.0 (JSON, Avro, Transactions, IncrementalAssign, librdkafka 1.8.0 builtin.features=gzip,snappy,ssl,sasl,regex,lz4,sasl_gssapi,sasl_plain,sasl_scram,plugins,zstd,sasl_oauthbearer)

kcat fails to connect to a kafka broker using SASL_SSL protocol:

$ kcat -b redacted:9094 -X sasl.mechanism=GSSAPI -X security.protocol=SASL_SSL -X ssl.ca.location=cacert.pem -X sasl.kerberos.keytab=keytab -X 'sasl.kerberos.principal=redacted@REDACTED' -X 'sasl.kerberos.kinit.cmd=kinit -k -t "%{sasl.kerberos.keytab}" "%{sasl.kerberos.principal}"' -L
%3|1633022215.858|FAIL|rdkafka#producer-1| [thrd:sasl_ssl://redacted:9094/bootstrap]: sasl_ssl://redacted:9094/bootstrap: SSL handshake failed: error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed: broker certificate could not be verified, verify that ssl.ca.location is correctly configured or root CA certificates are installed (brew install openssl) (after 106ms in state SSL_HANDSHAKE)

yet I've verified successful connectivity via openssl s_client with the same files running from the same directory as kcat above:

$ openssl s_client -host redacted -port 9094 -CAfile cacert.pem -showcerts -pause -crlf -verify 10 -verify_return_error
verify depth is 10
CONNECTED(00000005)
depth=1 C = US, ST = FL, L = weston, O = test, OU = test, CN = KDC-CA
verify return:1
depth=0 CN = redacted
verify return:1
---
Certificate chain
 0 s:/CN=redacted
   i:/C=redacted
-----BEGIN CERTIFICATE-----
MII...redacted...PQE=
-----END CERTIFICATE-----
 1 s:/C=redacted
   i:/C=redacted
-----BEGIN CERTIFICATE-----
MII...redacted...GEto=
-----END CERTIFICATE-----
---
Server certificate
subject=/CN=redacted
issuer=/C=redacted
---
No client certificate CA names sent
Server Temp Key: ECDH, P-256, 256 bits
---
SSL handshake has read 2960 bytes and written 326 bytes
---
New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-GCM-SHA384
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : ECDHE-RSA-AES256-GCM-SHA384
    Session-ID: 61...redacted...75
    Session-ID-ctx:
    Master-Key: C8...redacted...46
    Start Time: 1633022264
    Timeout   : 7200 (sec)
    Verify return code: 0 (ok)
---
DONE

The text was updated successfully, but these errors were encountered:

JimDunne · 2021-09-30T17:31:41Z

Added -X debug='securtity,broker' to the above kcat line and got this:

$ kcat -b redacted:9094 -X sasl.mechanism=GSSAPI -X security.protocol=SASL_SSL -X ssl.ca.location=cacert.pem -X sasl.kerberos.keytab=keytab -X 'sasl.kerberos.principal=redacted@REDACTED' -X 'sasl.kerberos.kinit.cmd=kinit -k -t "%{sasl.kerberos.keytab}" "%{sasl.kerberos.principal}"' -X debug='security,broker' -L
%7|1633022844.128|SASL|rdkafka#producer-1| [thrd:app]: Selected provider Cyrus for SASL mechanism GSSAPI
%7|1633022844.129|OPENSSL|rdkafka#producer-1| [thrd:app]: Using OpenSSL version OpenSSL 1.1.1l  24 Aug 2021 (0x101010cf, librdkafka built with 0x101010cf)
%7|1633022844.133|CACERTS|rdkafka#producer-1| [thrd:app]: Setting default CA certificate location to /etc/ssl/cert.pem, override with ssl.ca.location

It's telling me to override ssl.ca.location but I already do: -X ssl.ca.location=cacert.pem

edenhill · 2021-09-30T17:34:21Z

That is indeed weird, try -d security,broker,conf to see what it thinks ssl.ca.location is initially set to.
Do you have a kcat.conf or $KCAT_CONFIG?

JimDunne · 2021-09-30T17:37:00Z

I've tried both via -F kafkacat.conf and direct commandline args; both fail the same.

%7|1633023363.992|CONF|rdkafka#producer-1| [thrd:app]: Client configuration:
%7|1633023363.992|CONF|rdkafka#producer-1| [thrd:app]:   client.software.version = 1.8.0
%7|1633023363.992|CONF|rdkafka#producer-1| [thrd:app]:   metadata.broker.list = redacted:9094
%7|1633023363.992|CONF|rdkafka#producer-1| [thrd:app]:   debug = broker,security,conf
%7|1633023363.992|CONF|rdkafka#producer-1| [thrd:app]:   error_cb = 0x106005866
%7|1633023363.992|CONF|rdkafka#producer-1| [thrd:app]:   throttle_cb = 0x106005ad4
%7|1633023363.992|CONF|rdkafka#producer-1| [thrd:app]:   log_cb = 0x1060682b7
%7|1633023363.992|CONF|rdkafka#producer-1| [thrd:app]:   internal.termination.signal = 23
%7|1633023363.992|CONF|rdkafka#producer-1| [thrd:app]:   security.protocol = sasl_ssl
%7|1633023363.992|CONF|rdkafka#producer-1| [thrd:app]:   ssl.ca.location = probe
%7|1633023363.992|CONF|rdkafka#producer-1| [thrd:app]:   sasl.mechanisms = GSSAPI
%7|1633023363.992|CONF|rdkafka#producer-1| [thrd:app]:   sasl.kerberos.principal = redacted@REDACTED
%7|1633023363.992|CONF|rdkafka#producer-1| [thrd:app]:   sasl.kerberos.kinit.cmd = kinit -k -t "%{sasl.kerberos.keytab}" "%{sasl.kerberos.principal}"
%7|1633023363.992|CONF|rdkafka#producer-1| [thrd:app]:   sasl.kerberos.keytab = keytab

ssl.ca.location = probe

JimDunne · 2021-09-30T17:39:23Z

$ ls -l
total 192
-rw-r--r--  1 redacted  redacted   1.9K Sep 30 09:52 cacert.pem
...
$ cat cacert.pem
-----BEGIN CERTIFICATE-----
MII...redacted...GEto=
-----END CERTIFICATE-----

cacert.pem definitely exists in the current directory and is a normal file, world-readable.

edenhill · 2021-09-30T18:21:48Z

%7|1633023363.992|CONF|rdkafka#producer-1| [thrd:app]: ssl.ca.location = probe

for some reason it is not picking up your -X ssl.ca.location=..

try moving that argument to different places in the command line; first, last, middle..

JimDunne · 2021-09-30T18:25:36Z

No matter where I put it I end up with it believing it's set to "probe". I've also tried downgrading to kafkacat 1.6.0 and still get the same result. Perhaps the bug is in librdkafka 1.8.0?

JimDunne · 2021-09-30T18:53:19Z

in librdkafka/src/rdkafka_conf.c I see this around line 3706:

#if WITH_SSL
        if (conf->ssl.keystore_location && !conf->ssl.keystore_password)
                return "`ssl.keystore.password` is mandatory when "
                        "`ssl.keystore.location` is set";
        if (conf->ssl.ca && (conf->ssl.ca_location || conf->ssl.ca_pem))
                return "`ssl.ca.location` or `ssl.ca.pem`, and memory-based "
                       "set_ssl_cert(CERT_CA) are mutually exclusive.";
#ifdef __APPLE__
        else /* Default ssl.ca.location to 'probe' on OSX */
                rd_kafka_conf_set(conf, "ssl.ca.location", "probe", NULL, 0);
#endif
#endif

I'm running on a MacBook, so __APPLE__ should be set here.

JimDunne · 2021-09-30T19:39:23Z

It's definitely the #ifdef __APPLE__ section causing the failure. I've commented that section out and recompiled librdkafka locally. The ssl.ca.location can now be correctly provided and my original test case succeeds.

edenhill · 2021-09-30T19:49:41Z

Great find!
Can you file an issue on librdkafka so we can get it fixed in the upcoming 1.8.2 releasE?

JimDunne · 2021-09-30T19:50:31Z

This tiny printf patch for debugging:

diff --git a/src/rdkafka_conf.c b/src/rdkafka_conf.c
index a8aa5af5..9651b55c 100644
--- a/src/rdkafka_conf.c
+++ b/src/rdkafka_conf.c
@@ -3708,12 +3708,15 @@ const char *rd_kafka_conf_finalize (rd_kafka_type_t cltype,
         if (conf->ssl.keystore_location && !conf->ssl.keystore_password)
                 return "`ssl.keystore.password` is mandatory when "
                         "`ssl.keystore.location` is set";
+        printf("conf->ssl.ca=%p\nconf->ssl.ca_location='%s'\nconf->ssl.ca_pem='%s'\n", conf->ssl.ca, conf->ssl.ca_location, conf->ssl.ca_pem);
         if (conf->ssl.ca && (conf->ssl.ca_location || conf->ssl.ca_pem))
                 return "`ssl.ca.location` or `ssl.ca.pem`, and memory-based "
                        "set_ssl_cert(CERT_CA) are mutually exclusive.";
 #ifdef __APPLE__
-        else /* Default ssl.ca.location to 'probe' on OSX */
+        else /* Default ssl.ca.location to 'probe' on OSX */ {
+                printf("setting 'ssl.ca.location'='probe'\n");
                 rd_kafka_conf_set(conf, "ssl.ca.location", "probe", NULL, 0);
+        }
 #endif
 #endif

reveals this output:

conf->ssl.ca=0x0
conf->ssl.ca_location='cacert.pem'
conf->ssl.ca_pem='(null)'
setting 'ssl.ca.location'='probe'

JimDunne · 2021-09-30T19:50:47Z

Sure thing

JimDunne mentioned this issue Sep 30, 2021

Cannot override ssl.ca.location on OSX confluentinc/librdkafka#3566

Closed

7 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

SASL_SSL: verify failed: broker certificate could not be verified #345

SASL_SSL: verify failed: broker certificate could not be verified #345

JimDunne commented Sep 30, 2021

JimDunne commented Sep 30, 2021

edenhill commented Sep 30, 2021

JimDunne commented Sep 30, 2021 •

edited

JimDunne commented Sep 30, 2021

edenhill commented Sep 30, 2021

JimDunne commented Sep 30, 2021

JimDunne commented Sep 30, 2021

JimDunne commented Sep 30, 2021

edenhill commented Sep 30, 2021

JimDunne commented Sep 30, 2021

JimDunne commented Sep 30, 2021

SASL_SSL: verify failed: broker certificate could not be verified #345

SASL_SSL: verify failed: broker certificate could not be verified #345

Comments

JimDunne commented Sep 30, 2021

JimDunne commented Sep 30, 2021

edenhill commented Sep 30, 2021

JimDunne commented Sep 30, 2021 • edited

JimDunne commented Sep 30, 2021

edenhill commented Sep 30, 2021

JimDunne commented Sep 30, 2021

JimDunne commented Sep 30, 2021

JimDunne commented Sep 30, 2021

edenhill commented Sep 30, 2021

JimDunne commented Sep 30, 2021

JimDunne commented Sep 30, 2021

JimDunne commented Sep 30, 2021 •

edited