SSL SERVER AUTHENTICATION FOR KAFKA BROKER

Create Directory of SSL Files

mkdir -p /tmp/kafka-ssl-demo

cd /tmp/kafka-ssl-demo

CA side: Generate self-signed CA key-pair for brokers:

• #1 Create a key-pair for the CA and store in a PKCS12 file server.ca.p12. We use this for signing certificates.

keytool -genkeypair -keyalg RSA -keysize 2048 -keystore server.ca.p12 -storetype PKCS12 -storepass server-ca-password -keypass server-ca-password -alias ca -dname "CN=BrokerCA" -ext bc=ca:true -validity 365

• #2 Export the CA’s public certificate to server.ca.crt. This will be included in trust stores and certificate chains.

keytool -export -file server.ca.crt -keystore server.ca.p12 -storetype PKCS12 -storepass server-ca-password -alias ca -rfc

Print out the certificate contents:

openssl x509 -text -noout -in server.ca.crt

Server side: Create key stores for brokers with a certificate signed by the self-signed CA

• #1 Generate private key for a broker and store in the PKCS12 file server.ks.p12.

keytool -genkey -keyalg RSA -keysize 2048 -keystore server.ks.p12 -storepass server-ks-password -keypass server-ks-password -alias server -storetype PKCS12 -dname "CN=Kafka,O=Confluent,C=GB" -validity 365

• #2 Generate a certificate signing request.

keytool -certreq -file server.csr -keystore server.ks.p12 -storetype PKCS12 -storepass server-ks-password -keypass server-ks-password -alias server

• #3 Use the CA key store to sign the broker’s certificate. The signed certificate is stored in server.crt.

keytool -gencert -infile server.csr -outfile server.crt -keystore server.ca.p12 -storetype PKCS12 -storepass server-ca-password -alias ca -ext SAN=DNS:localhost -validity 365

cat server.crt server.ca.crt > serverchain.crt

• #4 Import broker’s certificate chain into broker’s key store.

keytool -importcert -file serverchain.crt -keystore server.ks.p12 -storepass server-ks-password -keypass server-ks-password -alias server -storetype PKCS12 -noprompt

Print out the certificates in the broker keystore:

keytool -list -v -keystore server.ks.p12 -storepass server-ks-password

Client side: Generate a trust store for clients with the broker’s CA certificate:

keytool -import -file server.ca.crt -keystore client.ts.p12 -storetype PKCS12 -storepass client-ts-password -alias ca -noprompt

Print out the content of the truststore:

keytool -list -v -keystore client.ts.p12 -storepass client-ts-password

Overview of generating SSL files:

Configure SSL on Kafka Broker (docker-compose.yml)

environment:
    - KAFKA_ZOOKEEPER_CONNECT=zookeeper:2181
    
    - KAFKA_LISTENERS=EXTERNAL://:9092,INTERNAL://:9093,INTERBROKER://:9094
    - KAFKA_ADVERTISED_LISTENERS=EXTERNAL://localhost:9092,INTERNAL://:9093,INTERBROKER://:9094
    - KAFKA_LISTENER_SECURITY_PROTOCOL_MAP=EXTERNAL:SSL,INTERNAL:PLAINTEXT,INTERBROKER:PLAINTEXT
    - KAFKA_INTER_BROKER_LISTENER_NAME=INTERBROKER
    
    - KAFKA_SSL_KEYSTORE_LOCATION=/tmp/kafka-ssl-demo/server.ks.p12
    - KAFKA_SSL_KEYSTORE_PASSWORD=server-ks-password
    - KAFKA_SSL_KEY_PASSWORD=server-ks-password
    - KAFKA_SSL_KEYSTORE_TYPE=PKCS12

volumes:
    - /tmp/kafka-ssl-demo:/tmp/kafka-ssl-demo

Run kafka:

docker-compose up -d

Verify the SSL configuration

• 1. Verify the SSL configuration of the broker. The following uses the Cryptography and SSL/TLS Toolkit (OpenSSL) and the client tool.

openssl s_client -connect localhost:9092

• 2. The tool should print out the certificate chain of the broker (a chain of the subjects and the issuers). At the end, you should find the following Verify return code:

Verify return code: 19 (self signed certificate in certificate chain)

Enter Ctrl-C to close the session.

• 3. Use the client tool with -CAfile option to trust the CA certificate.

openssl s_client -connect localhost:9092 -CAfile /tmp/kafka-ssl-demo/server.ca.crt

• 4. With the change, you should find the following Verify return code:

Verify return code: 0 (ok)

Enter Ctrl-C to close the session.

Configure SSL on Kafka Clients (application.yml)

spring:
  kafka:
    bootstrapServers: localhost:9092
    security:
      protocol: SSL
    ssl:
      #      trustStoreLocation: classpath:client.ts.p12 if put this file in ./src/main/resources/client.ts.p12
      trustStoreLocation: file://tmp/kafka-ssl-demo/client.ts.p12
      trustStorePassword: client-ts-password
      trustStoreType: PKCS12

Additional Information: SSL/TLS

Reference

• 1. Kafka: The Definitive Guide v2 - Chapter 6: Securing Kafka freely provided by Confluent.
• 2. Spring Cloud Stream Samples - kafka-ssl-demo
• 3. jaceklaskowski.gitbooks.io - Kafka Security / Communications Security - Demo
• 4. wurstmeister / kafka-docker link
• 5. Spring Kafka docs
• 6. SSL/TLS Under Lock and Key: A Guide to Understanding SSL/TLS Cryptography - Chapter 3: Public Key Infrastructure book