Cannot override ssl.ca.location on OSX

[JimDunne]

Description

On OSX, when supplying -X ssl.ca.location=cacert.pem via kcat, the ssl.ca.location is always reset to the special value probe, causing CA certificate validation failure because it's set to look in /etc/ssl/cacerts.pem instead.

https://github.com/edenhill/librdkafka/blob/master/src/rdkafka_conf.c#L3714

This newly added section of code causes ssl.ca.location to always be set to probe.

If you apply this printf debugging patch:

diff --git a/src/rdkafka_conf.c b/src/rdkafka_conf.c
index a8aa5af5..9651b55c 100644
--- a/src/rdkafka_conf.c
+++ b/src/rdkafka_conf.c
@@ -3708,12 +3708,15 @@ const char *rd_kafka_conf_finalize (rd_kafka_type_t cltype,
         if (conf->ssl.keystore_location && !conf->ssl.keystore_password)
                 return "`ssl.keystore.password` is mandatory when "
                         "`ssl.keystore.location` is set";
+        printf("conf->ssl.ca=%p\nconf->ssl.ca_location='%s'\nconf->ssl.ca_pem='%s'\n", conf->ssl.ca, conf->ssl.ca_location, conf->ssl.ca_pem);
         if (conf->ssl.ca && (conf->ssl.ca_location || conf->ssl.ca_pem))
                 return "`ssl.ca.location` or `ssl.ca.pem`, and memory-based "
                        "set_ssl_cert(CERT_CA) are mutually exclusive.";
 #ifdef __APPLE__
-        else /* Default ssl.ca.location to 'probe' on OSX */
+        else /* Default ssl.ca.location to 'probe' on OSX */ {
+                printf("setting 'ssl.ca.location'='probe'\n");
                 rd_kafka_conf_set(conf, "ssl.ca.location", "probe", NULL, 0);
+        }
 #endif
 #endif

you should see this output from kcat -X ssl.ca.location=cacert.pem confirming that the ssl.ca.location setting is overwritten to probe:

conf->ssl.ca=0x0
conf->ssl.ca_location='cacert.pem'
conf->ssl.ca_pem='(null)'
setting 'ssl.ca.location'='probe'

Related kcat issue: edenhill/kcat#345

How to reproduce

$ kcat -b redacted:9094 -X sasl.mechanism=GSSAPI -X security.protocol=SASL_SSL -X sasl.kerberos.keytab=keytab -X 'sasl.kerberos.principal=redacted@REDACTED' -X 'sasl.kerberos.kinit.cmd=kinit -k -t "%{sasl.kerberos.keytab}" "%{sasl.kerberos.principal}"' -X debug='security,broker,conf' -X ssl.ca.location=cacert.pem -L

...

%7|1633031336.057|CONF|rdkafka#producer-1| [thrd:app]: Client configuration:
%7|1633031336.057|CONF|rdkafka#producer-1| [thrd:app]:   client.software.version = 1.8.2-RC2
%7|1633031336.057|CONF|rdkafka#producer-1| [thrd:app]:   metadata.broker.list = redacted:9094
%7|1633031336.057|CONF|rdkafka#producer-1| [thrd:app]:   debug = broker,security,conf
%7|1633031336.057|CONF|rdkafka#producer-1| [thrd:app]:   error_cb = 0x103a9a270
%7|1633031336.057|CONF|rdkafka#producer-1| [thrd:app]:   throttle_cb = 0x103a9a500
%7|1633031336.057|CONF|rdkafka#producer-1| [thrd:app]:   log_cb = 0x103af7440
%7|1633031336.057|CONF|rdkafka#producer-1| [thrd:app]:   internal.termination.signal = 23
%7|1633031336.057|CONF|rdkafka#producer-1| [thrd:app]:   security.protocol = sasl_ssl
%7|1633031336.057|CONF|rdkafka#producer-1| [thrd:app]:   ssl.ca.location = probe
%7|1633031336.057|CONF|rdkafka#producer-1| [thrd:app]:   sasl.mechanisms = GSSAPI
%7|1633031336.057|CONF|rdkafka#producer-1| [thrd:app]:   sasl.kerberos.principal = redacted@REDACTED
%7|1633031336.057|CONF|rdkafka#producer-1| [thrd:app]:   sasl.kerberos.kinit.cmd = kinit -k -t "%{sasl.kerberos.keytab}" "%{sasl.kerberos.principal}"
%7|1633031336.057|CONF|rdkafka#producer-1| [thrd:app]:   sasl.kerberos.keytab = keytab

...

%7|1633031336.525|FAIL|rdkafka#producer-1| [thrd:sasl_ssl://redacted:9094/bootstrap]: sasl_ssl://redacted:9094/bootstrap: SSL handshake failed: ssl/statem/statem_clnt.c:1883: error:0A000086:SSL routines::certificate verify failed (after 109ms in state SSL_HANDSHAKE) (_SSL)

Please note the ssl.ca.location = probe suspect line in the CONF log as well as the -X ssl.ca.location=cacert.pem argument supplied to kcat.

The GSSAPI and kerberos configuration is not necessary for this example as the failure is limited to SSL scope.

Checklist

Please provide the following information:

• librdkafka version (release number or git tag): 1.8.2-RC2 / 2fd81e149d0d6fd1eadf58d0ab3756c5487d264b
• Apache Kafka version: N/A
• librdkafka client configuration: ssl.ca.location=cacert.pem
• Operating system: macOS Mojave 10.14.6
• Provide logs (with debug=.. as necessary) from librdkafka
• Provide broker log excerpts (N/A in this case)
• Critical issue (inability to connect to SASL_SSL broker endpoints with ssl.ca.location specified could be considered critical)

[ybelMekk]

I am up for the same issue, debugging all day yesterday on my MAC with OSX Big Sur 11.6 (20G165).

As described i could not set/override ssl.ca.location= in kcat.conf or in CLI params.

Had to change /etc/ssl/cert.pem to the broker ca to be able to make it work.

[JimDunne]

Workaround is to downgrade librdkafka from 1.8.0 to 1.7.0 as the bug exists only in 1.8.0 and beyond.

If you use brew on Mac, the following should do that for you:

wget https://github.com/Homebrew/homebrew-core/raw/f7fb08a650a3f4ef78a0c9770cce79d9a7773f06/Formula/librdkafka.rb
brew install librdkafka.rb
brew link --overwrite librdkafka

[edenhill]

This is fixed in librdkafka v1.8.2