[bitnami/kafka] Multiple CA certificates are not mounted into Truststore when using PEM certs

[mrabey]

Name and Version

kafka:28.0.4

What architecture are you using?

amd64

What steps will reproduce the bug?

Using cert-manager, you will need to create a root and intermediate CA certificate. Something like the following will work:

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: test-root-issuer
spec:
  selfSigned: {}
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: test-root-cert
  namespace: cert-manager
spec:
  isCA: true
  commonName: test-root-issuer
  secretName: test-root-cert
  privateKey:
    algorithm: RSA
    size: 4096
  issuerRef:
    name: test-root-issuer
    kind: ClusterIssuer
    group: cert-manager.io
---
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: test-intermediate-ca
spec:
  ca:
    secretName: test-root-cert
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: test-intermediate-ca
  namespace: cert-manager
spec:
  isCA: true
  commonName: test-intermediate-ca
  secretName: test-intermediate-cert
  privateKey:
    algorithm: RSA
    size: 4096
  issuerRef:
    name: test-root-issuer
    kind: ClusterIssuer
    group: cert-manager.io
---
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: test-intermediate-ca
spec:
  ca:
    secretName: test-intermediate-cert
---

Then, you'll need to create a certificate for the Kafka Controller+Broker using cert-manager and that new test-intermediate-ca as the ClusterIssuer:

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: kafka-controller-0
  namespace: kafka-test
spec:
  secretName: kafka-controller-0
  commonName: kafka-controller-0
  dnsNames:
    - kafka-controller-0.kafka-controller-headless.kafka-test.svc.cluster.local
  privateKey:
    algorithm: RSA
    size: 4096
  duration: 8760h
  renewBefore: 360h
  issuerRef:
    name: test-intermediate-ca
    kind: ClusterIssuer

This will generate a certificate in the kafka-test namespace with a key tls.crt with the leaf certificate followed by the certificate for the test-intermediate-ca. Supply this certificate to the tls.existingSecret key:

tls:
  type: PEM
  pemChainIncluded: true
  existingSecret: "kafka-controller-0"
  autoGenerated: false
  ...
  endpointIdentificationAlgorithm: https
  sslClientAuth: https

I've also configured all listeners to use SSL and require client authentication. For example:

listeners:
  client:
    containerPort: 9092
    protocol: SSL
    name: CLIENT
    sslClientAuth: "required"

Your usage of the keytool will not import that test-intermediate-ca certificate into truststore. I've found that keytool, while it claims that keytool -import -file can import a certificate or certificate chain, will only import the first certificate it finds in the file supplied. In this case, it will import the leaf certificate into the truststore and completely ignore the intermediate certificate.

The result is a failure to launch the Kafka pod due to a malformed truststore.

Are you using any custom parameters or values?

Yes, I create a template file for the cert-manager certificate:

# kafka-controller-0-cert.yaml

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: {{ .Values.kafka.brokerCerts.secretName }}
  labels: {{- include "common.labels.standard" . | nindent 4 }}
  annotations:
    "helm.sh/hook": pre-install, pre-upgrade
    "helm.sh/hook-weight": "-5"
spec:
  secretName: {{ .Values.kafka.brokerCerts.secretName }}
  commonName: {{ .Values.kafka.brokerCerts.commonName }}
  dnsNames: {{- toYaml .Values.kafka.brokerCerts.dnsNames | nindent 2 }}
  privateKey:
    algorithm: RSA
    size: 4096
  duration: 8760h
  renewBefore: 360h
  issuerRef:
    name: {{ .Values.kafka.brokerCerts.issuerName }}
    kind: {{ .Values.kafka.brokerCerts.issuerType }}

That kafka Values section, I add from my own values file:

# values-certs.yaml
...
kafka:
  brokerCerts:
    secretName: kafka-controller-0
    commonName: kafka-controller-0
    issuerName: test-intermediate-ca
    issuerType: ClusterIssuer
    dnsNames:
      - 'kafka-controller-0.kafka-controller-headless.kafka-test.svc.cluster.local'

I deploy the Bitnami Kafka chart using the following command:

helm upgrade -i --create-namespace -n kafka-test kafka ./kafka-28.0.4 \
  -f ./kafka-28.0.4/values.yaml \
  -f ./kafka-28.0.4/values-certs.yaml

What is the expected behavior?

When I use a chain of certificates, the intermediate certificates are loaded into the Truststore properly in the named template file templates/scripts-configmap.yaml under the function configure_kafka_tls().

What do you see instead?

An error regarding a malformed truststore preventing the Kafka cluster from starting properly.

Additional information

The usage of keytool is misleading based on their description of how the keytool -import -file function works. It claims to import a certificate chain if one exists, but it doesn't. It only imports the first certificate it finds in the file provided. I've had to fix this for my usage in other parts of my infrastructure. The workaround is that you must import the certs one by one. I have settled on splitting all certificates in a single certificate file into their certificate files, iterating each of these split files, and one-by-one importing them into the truststore. This way, the entire chain of certificates is included in the truststore. I'd prefer to use PEM certificates, but I have a workaround by using JKS, creating an initContainer that injects the appropriate chain of certificates into the truststore, and then finally mounting these certificates in a local volume to your /mounted-certs directory.

[carrodher]

Thank you for bringing this issue to our attention. We appreciate your involvement! If you're interested in contributing a solution, we welcome you to create a pull request. The Bitnami team is excited to review your submission and offer feedback. You can find the contributing guidelines here.

Your contribution will greatly benefit the community. Feel free to reach out if you have any questions or need assistance.

[github-actions]

This Issue has been automatically marked as "stale" because it has not had recent activity (for 15 days). It will be closed if no further activity occurs. Thanks for the feedback.

[github-actions]

Due to the lack of activity in the last 5 days since it was marked as "stale", we proceed to close this Issue. Do not hesitate to reopen it later if necessary.