Kafka Lag Exporter container fails with PKCS12 certificates when generated with JDK17 #270

jeqo · 2021-11-04T12:20:45Z

Describe the bug
PKCS12 keystores are the default since Java 9[1].
Even though JDK 8 supports PKCS12, I found that when creating keystores with newer versions (e.g. 17) the container running JDK 8 is not able to read the keystore:

2021-11-04 12:02:19,721 ERROR c.l.k.ConsumerGroupCollector$ akka://kafka-lag-exporter/user/consumer-group-collector-dev-cluster - Supervisor RestartSupervisor saw failure: Failed to construct kafka consumer org.apache.kafka.common.KafkaException: Failed to construct kafka consumer
        at org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:825)
        at org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:666)
        at org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:646)
        at com.lightbend.kafkalagexporter.KafkaClient$.createConsumerClient(KafkaClient.scala:82)
        at com.lightbend.kafkalagexporter.KafkaClient$.apply(KafkaClient.scala:32)
        at com.lightbend.kafkalagexporter.MainApp$.$anonfun$start$1(MainApp.scala:32)
        at com.lightbend.kafkalagexporter.ConsumerGroupCollector$.$anonfun$init$1(ConsumerGroupCollector.scala:117)
        at akka.actor.typed.internal.BehaviorImpl$DeferredBehavior$$anon$1.apply(BehaviorImpl.scala:119)
        at akka.actor.typed.Behavior$.start(Behavior.scala:168)
        at akka.actor.typed.internal.RestartSupervisor.restartCompleted(Supervision.scala:372)
        at akka.actor.typed.internal.RestartSupervisor.aroundReceive(Supervision.scala:236)
        at akka.actor.typed.internal.InterceptorImpl.receive(InterceptorImpl.scala:85)
        at akka.actor.typed.Behavior$.interpret(Behavior.scala:274)
        at akka.actor.typed.Behavior$.interpretMessage(Behavior.scala:230)
        at akka.actor.typed.internal.adapter.ActorAdapter.handleMessage(ActorAdapter.scala:130)
        at akka.actor.typed.internal.adapter.ActorAdapter.aroundReceive(ActorAdapter.scala:106)
        at akka.actor.ActorCell.receiveMessage(ActorCell.scala:577)
        at akka.actor.ActorCell.invoke(ActorCell.scala:547)
        at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:270)
        at akka.dispatch.Mailbox.run(Mailbox.scala:231)
        at akka.dispatch.Mailbox.exec(Mailbox.scala:243)
        at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
        at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
        at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
        at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175)
Caused by: org.apache.kafka.common.KafkaException: org.apache.kafka.common.KafkaException: org.apache.kafka.common.KafkaException: Failed to load SSL keystore /etc/kafka/secrets/kafka.kafkaLagExporter.keystore.jks of type PKCS12
        at org.apache.kafka.common.network.SslChannelBuilder.configure(SslChannelBuilder.java:74)
        at org.apache.kafka.common.network.ChannelBuilders.create(ChannelBuilders.java:157)
        at org.apache.kafka.common.network.ChannelBuilders.clientChannelBuilder(ChannelBuilders.java:73)
        at org.apache.kafka.clients.ClientUtils.createChannelBuilder(ClientUtils.java:105)
        at org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:743)
        ... 24 common frames omitted
Caused by: org.apache.kafka.common.KafkaException: org.apache.kafka.common.KafkaException: Failed to load SSL keystore /etc/kafka/secrets/kafka.kafkaLagExporter.keystore.jks of type PKCS12
        at org.apache.kafka.common.security.ssl.SslEngineBuilder.createSSLContext(SslEngineBuilder.java:163)
        at org.apache.kafka.common.security.ssl.SslEngineBuilder.<init>(SslEngineBuilder.java:104)
        at org.apache.kafka.common.security.ssl.SslFactory.configure(SslFactory.java:95)
        at org.apache.kafka.common.network.SslChannelBuilder.configure(SslChannelBuilder.java:72)
        ... 28 common frames omitted
Caused by: org.apache.kafka.common.KafkaException: Failed to load SSL keystore /etc/kafka/secrets/kafka.kafkaLagExporter.keystore.jks of type PKCS12
        at org.apache.kafka.common.security.ssl.SslEngineBuilder$SecurityStore.load(SslEngineBuilder.java:292)
        at org.apache.kafka.common.security.ssl.SslEngineBuilder.createSSLContext(SslEngineBuilder.java:144)
        ... 31 common frames omitted
Caused by: java.io.IOException: parseAlgParameters failed: ObjectIdentifier() -- data isn't an object ID (tag = 48)
        at sun.security.pkcs12.PKCS12KeyStore.parseAlgParameters(PKCS12KeyStore.java:819)
        at sun.security.pkcs12.PKCS12KeyStore.engineLoad(PKCS12KeyStore.java:2027)
        at java.security.KeyStore.load(KeyStore.java:1445)
        at org.apache.kafka.common.security.ssl.SslEngineBuilder$SecurityStore.load(SslEngineBuilder.java:289)
        ... 32 common frames omitted
Caused by: java.io.IOException: ObjectIdentifier() -- data isn't an object ID (tag = 48)
        at sun.security.util.ObjectIdentifier.<init>(ObjectIdentifier.java:285)
        at sun.security.util.DerInputStream.getOID(DerInputStream.java:320)
        at com.sun.crypto.provider.PBES2Parameters.engineInit(PBES2Parameters.java:267)
        at java.security.AlgorithmParameters.init(AlgorithmParameters.java:293)
        at sun.security.pkcs12.PKCS12KeyStore.parseAlgParameters(PKCS12KeyStore.java:815)
        ... 35 common frames omitted

[1] https://openjdk.java.net/jeps/229

To Reproduce

Create a PKCS12 keystore with JDK17.
Mount keystore and configuration to Kafka Lag Exporter container.

Java versions tested:
Container Java version:

openjdk version "1.8.0_265"
OpenJDK Runtime Environment (build 1.8.0_265-b01)
OpenJDK 64-Bit Server VM (build 25.265-b01, mixed mode)

Laptop Java version:

openjdk version "17" 2021-09-14
OpenJDK Runtime Environment (build 17+35-2724)
OpenJDK 64-Bit Server VM (build 17+35-2724, mixed mode, sharing)

Configuration:

kafka-lag-exporter {
  port = 9999

  client-group-id = "kafkaLagExporter"
  lookup-table-size = 120

  clusters = [
    {
      name = "dev-cluster"
      bootstrap-brokers = "kafka1:11091,kafka2:11092"

      admin-client-properties = {
        client.id = "admin-client-id"
        security.protocol = "SSL"
        ssl.truststore.location = "/etc/kafka/secrets/kafka.kafkaLagExporter.truststore.jks"
        ssl.truststore.password = "confluent"
        ssl.keystore.location = "/etc/kafka/secrets/kafka.kafkaLagExporter.keystore.jks"
        ssl.keystore.password = "confluent"
        ssl.keystore.type = "PKCS12"
        ssl.key.password = "confluent"
      }

      consumer-properties = {
        client.id = "consumer-client-id"
        security.protocol = "SSL"
        ssl.truststore.location = "/etc/kafka/secrets/kafka.kafkaLagExporter.truststore.jks"
        ssl.truststore.password = "confluent"
        ssl.keystore.location = "/etc/kafka/secrets/kafka.kafkaLagExporter.keystore.jks"
        ssl.keystore.password = "confluent"
        ssl.keystore.type = "PKCS12"
        ssl.key.password = "confluent"
      }
    }
  ]
}

Environment

Version: 0.6.7
Version of Apache Kafka cluster: 2.8.1
Run with Helm or [x]Standalone

Additional context
Add any other context about the problem here.

The text was updated successfully, but these errors were encountered:

jeqo · 2021-11-05T12:52:00Z

Just tested with JDK11 generated keystores and fail on the same way:

2021-11-05 12:50:24,228 ERROR c.l.k.ConsumerGroupCollector$ akka://kafka-lag-exporter/user/consumer-group-collector-dev-cluster - Supervisor RestartSupervisor saw failure: Failed to construct kafka consumer org.apache.kafka.common.KafkaException: Failed to construct kafka consumer
        at org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:825)
        at org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:666)
        at org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:646)
        at com.lightbend.kafkalagexporter.KafkaClient$.createConsumerClient(KafkaClient.scala:82)
        at com.lightbend.kafkalagexporter.KafkaClient$.apply(KafkaClient.scala:32)
        at com.lightbend.kafkalagexporter.MainApp$.$anonfun$start$1(MainApp.scala:32)
        at com.lightbend.kafkalagexporter.ConsumerGroupCollector$.$anonfun$init$1(ConsumerGroupCollector.scala:117)
        at akka.actor.typed.internal.BehaviorImpl$DeferredBehavior$$anon$1.apply(BehaviorImpl.scala:119)
        at akka.actor.typed.Behavior$.start(Behavior.scala:168)
        at akka.actor.typed.internal.RestartSupervisor.restartCompleted(Supervision.scala:372)
        at akka.actor.typed.internal.RestartSupervisor.aroundReceive(Supervision.scala:236)
        at akka.actor.typed.internal.InterceptorImpl.receive(InterceptorImpl.scala:85)
        at akka.actor.typed.Behavior$.interpret(Behavior.scala:274)
        at akka.actor.typed.Behavior$.interpretMessage(Behavior.scala:230)
        at akka.actor.typed.internal.adapter.ActorAdapter.handleMessage(ActorAdapter.scala:130)
        at akka.actor.typed.internal.adapter.ActorAdapter.aroundReceive(ActorAdapter.scala:106)
        at akka.actor.ActorCell.receiveMessage(ActorCell.scala:577)
        at akka.actor.ActorCell.invoke(ActorCell.scala:547)
        at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:270)
        at akka.dispatch.Mailbox.run(Mailbox.scala:231)
        at akka.dispatch.Mailbox.exec(Mailbox.scala:243)
        at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
        at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
        at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
        at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175)
Caused by: org.apache.kafka.common.KafkaException: org.apache.kafka.common.KafkaException: org.apache.kafka.common.KafkaException: Failed to load SSL keystore /etc/kafka/secrets/kafka.kafkaLagExporter.keystore.jks of type JKS
        at org.apache.kafka.common.network.SslChannelBuilder.configure(SslChannelBuilder.java:74)
        at org.apache.kafka.common.network.ChannelBuilders.create(ChannelBuilders.java:157)
        at org.apache.kafka.common.network.ChannelBuilders.clientChannelBuilder(ChannelBuilders.java:73)
        at org.apache.kafka.clients.ClientUtils.createChannelBuilder(ClientUtils.java:105)
        at org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:743)
        ... 24 common frames omitted
Caused by: org.apache.kafka.common.KafkaException: org.apache.kafka.common.KafkaException: Failed to load SSL keystore /etc/kafka/secrets/kafka.kafkaLagExporter.keystore.jks of type JKS
        at org.apache.kafka.common.security.ssl.SslEngineBuilder.createSSLContext(SslEngineBuilder.java:163)
        at org.apache.kafka.common.security.ssl.SslEngineBuilder.<init>(SslEngineBuilder.java:104)
        at org.apache.kafka.common.security.ssl.SslFactory.configure(SslFactory.java:95)
        at org.apache.kafka.common.network.SslChannelBuilder.configure(SslChannelBuilder.java:72)
        ... 28 common frames omitted
Caused by: org.apache.kafka.common.KafkaException: Failed to load SSL keystore /etc/kafka/secrets/kafka.kafkaLagExporter.keystore.jks of type JKS
        at org.apache.kafka.common.security.ssl.SslEngineBuilder$SecurityStore.load(SslEngineBuilder.java:292)
        at org.apache.kafka.common.security.ssl.SslEngineBuilder.createSSLContext(SslEngineBuilder.java:144)
        ... 31 common frames omitted
Caused by: java.io.IOException: Invalid keystore format
        at sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:666)
        at sun.security.provider.JavaKeyStore$JKS.engineLoad(JavaKeyStore.java:57)
        at sun.security.provider.KeyStoreDelegator.engineLoad(KeyStoreDelegator.java:224)
        at sun.security.provider.JavaKeyStore$DualFormatJKS.engineLoad(JavaKeyStore.java:71)
        at java.security.KeyStore.load(KeyStore.java:1445)
        at org.apache.kafka.common.security.ssl.SslEngineBuilder$SecurityStore.load(SslEngineBuilder.java:289)
        ... 32 common frames omitted

gitpod /workspace/jmx-monitoring-stacks $ java -version
Picked up JAVA_TOOL_OPTIONS: -Xmx2576m
openjdk version "11.0.13" 2021-10-19 LTS
OpenJDK Runtime Environment Zulu11.52+13-CA (build 11.0.13+8-LTS)
OpenJDK 64-Bit Server VM Zulu11.52+13-CA (build 11.0.13+8-LTS, mixed mode)

timtebeek · 2021-11-08T12:15:40Z

Not sure if this 'll work in your case, but at work we had to add a RUN yum update -y to our Dockerfile.

Which is weird since that's also already in the Dockerfile generated.
https://github.com/lightbend/kafka-lag-exporter/blob/master/build.sbt#L55

Maybe a new tag would already be enough to fix the issue?

Although ideally swap Java 8 for 11, also given the support timeline for 8.

jeqo · 2021-11-09T11:18:03Z

Thanks for the workaround @timtebeek !

Maybe a new tag would already be enough to fix the issue?
Although ideally swap Java 8 for 11, also given the support timeline for 8.

Agree. I was thinking the best solution could be around those lines.

cc @seglo

slachiewicz · 2021-11-14T01:51:37Z

Yes, update to latest JDK will fix issue - root cause is https://bugs.java.com/bugdatabase/view_bug.do?bug_id=8267599

seglo · 2021-11-21T21:48:49Z

I just released 0.6.8. LMK if that addresses it.

jeqo · 2021-11-22T12:06:30Z

@seglo thanks for the heads up! Unfortunately, it's not fixing the issue:

2021-11-22 12:02:04,671 ERROR c.l.k.ConsumerGroupCollector$ akka://kafka-lag-exporter/user/consumer-group-collector-dev-cluster - Supervisor RestartSupervisor saw failure [3]: Failed to construct kafka consumer org.apache.kafka.common.KafkaException: Failed to construct kafka consumer
        at org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:825)
        at org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:666)
        at org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:646)
        at com.lightbend.kafkalagexporter.KafkaClient$.createConsumerClient(KafkaClient.scala:82)
        at com.lightbend.kafkalagexporter.KafkaClient$.apply(KafkaClient.scala:32)
        at com.lightbend.kafkalagexporter.MainApp$.$anonfun$start$1(MainApp.scala:32)
        at com.lightbend.kafkalagexporter.ConsumerGroupCollector$.$anonfun$init$1(ConsumerGroupCollector.scala:117)
        at akka.actor.typed.internal.BehaviorImpl$DeferredBehavior$$anon$1.apply(BehaviorImpl.scala:120)
        at akka.actor.typed.Behavior$.start(Behavior.scala:168)
        at akka.actor.typed.internal.RestartSupervisor.restartCompleted(Supervision.scala:383)
        at akka.actor.typed.internal.RestartSupervisor.aroundReceive(Supervision.scala:243)
        at akka.actor.typed.internal.InterceptorImpl.receive(InterceptorImpl.scala:85)
        at akka.actor.typed.Behavior$.interpret(Behavior.scala:274)
        at akka.actor.typed.Behavior$.interpretMessage(Behavior.scala:230)
        at akka.actor.typed.internal.adapter.ActorAdapter.handleMessage(ActorAdapter.scala:131)
        at akka.actor.typed.internal.adapter.ActorAdapter.aroundReceive(ActorAdapter.scala:107)
        at akka.actor.ActorCell.receiveMessage(ActorCell.scala:580)
        at akka.actor.ActorCell.invoke(ActorCell.scala:548)
        at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:270)
        at akka.dispatch.Mailbox.run(Mailbox.scala:231)
        at akka.dispatch.Mailbox.exec(Mailbox.scala:243)
        at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
        at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
        at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
        at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175)
Caused by: org.apache.kafka.common.KafkaException: org.apache.kafka.common.KafkaException: org.apache.kafka.common.KafkaException: Failed to load SSL keystore /etc/kafka/secrets/kafka.kafkaLagExporter.keystore.jks of type PKCS12
        at org.apache.kafka.common.network.SslChannelBuilder.configure(SslChannelBuilder.java:74)
        at org.apache.kafka.common.network.ChannelBuilders.create(ChannelBuilders.java:157)
        at org.apache.kafka.common.network.ChannelBuilders.clientChannelBuilder(ChannelBuilders.java:73)
        at org.apache.kafka.clients.ClientUtils.createChannelBuilder(ClientUtils.java:105)
        at org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:743)
        ... 24 common frames omitted
Caused by: org.apache.kafka.common.KafkaException: org.apache.kafka.common.KafkaException: Failed to load SSL keystore /etc/kafka/secrets/kafka.kafkaLagExporter.keystore.jks of type PKCS12
        at org.apache.kafka.common.security.ssl.SslEngineBuilder.createSSLContext(SslEngineBuilder.java:163)
        at org.apache.kafka.common.security.ssl.SslEngineBuilder.<init>(SslEngineBuilder.java:104)
        at org.apache.kafka.common.security.ssl.SslFactory.configure(SslFactory.java:95)
        at org.apache.kafka.common.network.SslChannelBuilder.configure(SslChannelBuilder.java:72)
        ... 28 common frames omitted
Caused by: org.apache.kafka.common.KafkaException: Failed to load SSL keystore /etc/kafka/secrets/kafka.kafkaLagExporter.keystore.jks of type PKCS12
        at org.apache.kafka.common.security.ssl.SslEngineBuilder$SecurityStore.load(SslEngineBuilder.java:292)
        at org.apache.kafka.common.security.ssl.SslEngineBuilder.createSSLContext(SslEngineBuilder.java:144)
        ... 31 common frames omitted
Caused by: java.io.IOException: Integrity check failed: java.security.NoSuchAlgorithmException: Algorithm HmacPBESHA256 not available
        at sun.security.pkcs12.PKCS12KeyStore.engineLoad(PKCS12KeyStore.java:2129)
        at java.security.KeyStore.load(KeyStore.java:1445)
        at org.apache.kafka.common.security.ssl.SslEngineBuilder$SecurityStore.load(SslEngineBuilder.java:289)
        ... 32 common frames omitted
Caused by: java.security.NoSuchAlgorithmException: Algorithm HmacPBESHA256 not available
        at javax.crypto.Mac.getInstance(Mac.java:181)
        at sun.security.pkcs12.PKCS12KeyStore.engineLoad(PKCS12KeyStore.java:2107)
        ... 34 common frames omitted

(same as https://bugs.java.com/bugdatabase/view_bug.do?bug_id=8267599)

docker ps
da95b1fe3a8f   lightbend/kafka-lag-exporter:0.6.8                         "/opt/docker/bin/kaf…"   16 minutes ago   Up 16 minutes                                                                                                                                            kafka-lag-exporter

docker exec kafka-lag-exporter java -version
openjdk version "1.8.0_312"
OpenJDK Runtime Environment (build 1.8.0_312-b07)
OpenJDK 64-Bit Server VM (build 25.312-b07, mixed mode)

seglo · 2021-11-22T14:52:43Z

Ok. I created a ticket for Java 11. Maybe that will help. #286

timtebeek · 2022-02-01T09:20:59Z

Not sure if this 'll work in your case, but at work we had to add a RUN yum update -y to our Dockerfile.

As of today this workaround no longer works, as CentOS Linux 8 is End Of Life.
There's a further vault workaround posted here, but ideally this is fixed at the root.

jeqo · 2022-03-23T18:51:47Z

Haven't tested it, but seems like #297 already solves this?

seglo · 2022-04-04T13:10:39Z

I'll be cutting a release shortly once I sort out the release process. Thanks for your patience.

seglo · 2022-04-07T01:46:38Z

Kafka Lag Exporter v0.7.0 has been released. Please let me know how it goes!

sverrehu · 2022-10-28T12:48:26Z

Is this still an issue, now that the container uses Java 17? I suggest closing it.

jeqo mentioned this issue Nov 4, 2021

MINOR: Updating docs with Keystore limitations (JDK8) confluentinc/jmx-monitoring-stacks#58

Merged

gshilo mentioned this issue Feb 8, 2022

There is no clear documentation on how to connect to a ssl and kerberos enabled Kafka cluster with the helm chart #321

Open

jeqo mentioned this issue Aug 23, 2022

fix: update lag exporter image confluentinc/jmx-monitoring-stacks#88

Closed

jeqo closed this as completed Oct 28, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Kafka Lag Exporter container fails with PKCS12 certificates when generated with JDK17 #270

Kafka Lag Exporter container fails with PKCS12 certificates when generated with JDK17 #270

jeqo commented Nov 4, 2021

jeqo commented Nov 5, 2021

timtebeek commented Nov 8, 2021 •

edited

Loading

jeqo commented Nov 9, 2021

slachiewicz commented Nov 14, 2021

seglo commented Nov 21, 2021

jeqo commented Nov 22, 2021

seglo commented Nov 22, 2021

timtebeek commented Feb 1, 2022

jeqo commented Mar 23, 2022

seglo commented Apr 4, 2022

seglo commented Apr 7, 2022

sverrehu commented Oct 28, 2022

Kafka Lag Exporter container fails with PKCS12 certificates when generated with JDK17 #270

Kafka Lag Exporter container fails with PKCS12 certificates when generated with JDK17 #270

Comments

jeqo commented Nov 4, 2021

jeqo commented Nov 5, 2021

timtebeek commented Nov 8, 2021 • edited Loading

jeqo commented Nov 9, 2021

slachiewicz commented Nov 14, 2021

seglo commented Nov 21, 2021

jeqo commented Nov 22, 2021

seglo commented Nov 22, 2021

timtebeek commented Feb 1, 2022

jeqo commented Mar 23, 2022

seglo commented Apr 4, 2022

seglo commented Apr 7, 2022

sverrehu commented Oct 28, 2022

timtebeek commented Nov 8, 2021 •

edited

Loading