Kafka: How to specify keystore + truststore files for authentication?

[pranavmarla]

Component(s)

exporter/kafka

What happened?

Description

We are attempting to use the OTEL agent's Kafka exporter to send data to our internal Kafka clusters. Our Kafka clusters require all Kafka clients (in this case, the OTEL agent) to authenticate to the cluster by specifying a keystore and truststore file. However, looking at the OTEL Kafka exporter auth settings (see below), there doesn't appear to be an obvious way to do so. In fact, the only cert-related settings come with a confusing instruction that they should only be used if TLS cert verification is disabled.

So, our question is: Does anyone know how to use the existing OTEL Kafka exporter auth settings to specify a keystore and truststore file, so we can authenticate to our internal Kafka clusters?
Or, alternatively, does anyone know for certain that this is not possible with OTEL as of now?

Note: This issue is focused on the Kafka exporter, but we have the exact same issue with the Kafka receiver as well.

Steps to Reproduce

For context, this is how we would authenticate to our internal Kafka clusters using the built-in Kafka console producer:

• For each client, create a user certificate. Producers and consumers should have separate certs.
• Download the cert as a Java Keystore (JKS) file. This file will contain both the private key and the root chain, and will serve as both the keystore and truststore.
• For the client, create a properties file containing the keystore/truststore details.
  Eg. For the built-in Kafka console producer, we created a file named producer.properties with the following contents:

  security.protocol=SSL
  
  ssl.truststore.location=kafka-producer-keystore-truststore.jks
  ssl.truststore.password=XXX
  
  ssl.keystore.location=kafka-producer-keystore-truststore.jks
  ssl.keystore.password=XXX
  
  ssl.key.password=YYY
  

• Finally, we run the Kafka console producer with the following command:

  kafka-console-producer.sh \
     --producer.config producer.properties \
     --bootstrap-server … : … \
     --topic …
  

Essentially, we want to follow this same authentication procedure but, instead of using the built-in Kafka console producer to send data to Kafka, we want to use the OTEL agent (specifically, its Kafka exporter).

Expected Result

OTEL agent's Kafka exporter would expose auth settings that allow you to easily specify keystore and truststore files and passwords, similar to the producer.properties file that the built-in Kafka console producer uses.

Actual Result

OTEL agent's Kafka exporter does not appear to let you specify keystore and truststore fields and passwords, preventing you from authenticating via that mechanism.

Collector version

N/A

Environment information

Environment

N/A

OpenTelemetry Collector configuration

No response

Log output

No response

Additional context

No response

[github-actions]

Pinging code owners:

• exporter/kafka: @pavolloffay @MovieStoreGuy

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[pranavmarla]

Update: After extensive research and trial and error, I was finally able to come up with a workaround: Essentially, instead of directly supplying the keystore to OTel's Kafka exporter, we extract the individual key and certificates from the keystore and directly supply those instead. See below for detailed steps:

• Ensure your keystore is in PKCS12 format:
  Although my original post mentioned using a keystore in JKS format, I switched to PKCS12 instead for simplicity and greater interoperabiltiy. If you're stuck with JKS though, you should still be able to convert it to PKCS12 via keytool.
  Eg. kafka-producer-keystore.pfx
  Note: As mentioned in the original post, for simplicity, I am using the same file as both keystore and truststore.

• Extract the individual key and certs:
  Since keystores are essentially just containers for keys and certificates, you can extract them from the PKCS12 keystore. Specifically:

  • Extract only the (private) key, unencrypted:

    openssl pkcs12 \
      -in kafka-producer-keystore.pfx \
      -nodes -nocerts \
      -out kafka-producer-key.key
    

    Note: Unlike with the built-in Kafka console clients, which support encrypted private keys (you specify the key's password in a separate parameter), OTel seems to require the private key to be unencrypted -- I've opened a feature request to support encryption here.

  • Extract only the CA (intermediate and root) certs:

    openssl pkcs12 \
      -in kafka-producer-keystore.pfx \
      -nokeys -cacerts \
      -out kafka-producer-ca-certs.crt
    

  • Extract all the certs (i.e. client cert and CA certs):

    openssl pkcs12 \
      -in kafka-producer-keystore.pfx \
      -nokeys \
      -out kafka-producer-client-and-ca-certs.crt
    

• Supply them to the OTel agent's Kafka exporter config as follows:

  ...
  exporters:
    kafka:
      ...
      auth:
        tls:
          insecure: true
          ca_file: kafka-producer-ca-certs.crt
          cert_file: kafka-producer-client-and-ca-certs.crt
          key_file: kafka-producer-key.key
  

  Note:

  • For simplicity, I set the cert_file param to be a file containing both the client and CA certs (i.e. the client cert, intermediate certs and root cert). However, technically speaking, the root cert is not required here so, if you want, you could manually edit the cert file to remove the last (root) cert.
  • I'm still confused as to why, to make use of the TLS settings, you have to first counterintuitively disable TLS verification (i.e. set insecure: true) but it does appear to work.
  • Although this example focused on the OTel Kafka exporter (producer), the same auth settings work for the OTel Kafka receiver (consumer) as well.

[pranavmarla]

Since this appears to be more of a missing feature than a bug (and I've discovered a workaround), I will close this bug report and open a feature request instead.