[Question] How to configure Thingsboard Kafka external node as producer to Confluent Cloud

[saradindusengupta]

Component

• Rule Engine
• External Node

Description
I am trying to use the external node for Kafka in a rule chain to produce a Kafka broker in the Confluent cloud. Both the systems, i.e., thingsboard and Confluent cloud are hosted in the same AWS region. I am getting failure status in the event tab in Thingsboard with the following error log

Topic test not present in metadata after 60000 ms

I have tried to produce to the same topic from multiple Thingsoard systems with the same Thingsboard version, and I have received the same error log although I was able to send to a self-managed Kafka broker hosted in the same AWS region using the same external Kafka node following the tutorial here
None of the Thingsboard systems had SSL enabled or any root certificate stored. I have also attached the error log from thingsboard as well.

2022-06-24 12:59:21,975 [kafka-producer-network-thread | producer-tb-kafka-node-f0cfd6c0-f13b-11ec-a37e-bdd7f428800a-thingsboard-firmware-testing] WARN  o.apache.kafka.clients.NetworkClient - [Producer clientId=producer-tb-kafka-node-f0cfd6c0-f13b-11ec-a37e-bdd7f428800a-thingsboard-firmware-testing] Connection to node -1 (pkc-l7pr2.ap-south-1.aws.confluent.cloud/43.204.44.151:9092) terminated during authentication. This may happen due to any of the following reasons: (1) Authentication failed due to invalid credentials with brokers older than 1.0.0, (2) Firewall blocking Kafka TLS traffic (eg it may only allow HTTPS traffic), (3) Transient network issue.
2022-06-24 12:59:21,975 [kafka-producer-network-thread | producer-tb-kafka-node-f0cfd6c0-f13b-11ec-a37e-bdd7f428800a-thingsboard-firmware-testing] WARN  o.apache.kafka.clients.NetworkClient - [Producer clientId=producer-tb-kafka-node-f0cfd6c0-f13b-11ec-a37e-bdd7f428800a-thingsboard-firmware-testing] Bootstrap broker pkc-l7pr2.ap-south-1.aws.confluent.cloud:9092 (id: -1 rack: null) disconnected
2022-06-24 12:59:22,084 [kafka-producer-network-thread | producer-tb-kafka-node-f0cfd6c0-f13b-11ec-a37e-bdd7f428800a-thingsboard-firmware-testing] WARN  o.a.kafka.common.network.Selector - [Producer clientId=producer-tb-kafka-node-f0cfd6c0-f13b-11ec-a37e-bdd7f428800a-thingsboard-firmware-testing] Unexpected error from pkc-l7pr2.ap-south-1.aws.confluent.cloud/43.204.44.151; closing connection
java.lang.RuntimeException: Unexpected error: java.security.InvalidAlgorithmParameterException: the trustAnchors parameter must be non-empty
        at java.base/sun.security.validator.PKIXValidator.<init>(PKIXValidator.java:102)
        at java.base/sun.security.validator.Validator.getInstance(Validator.java:181)
        at java.base/sun.security.ssl.X509TrustManagerImpl.getValidator(X509TrustManagerImpl.java:300)
        at java.base/sun.security.ssl.X509TrustManagerImpl.checkTrustedInit(X509TrustManagerImpl.java:176)
        at java.base/sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:246)
        at java.base/sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:141)
        at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.checkServerCerts(CertificateMessage.java:632)
        at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.onCertificate(CertificateMessage.java:473)
        at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.consume(CertificateMessage.java:369)
        at java.base/sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:392)
        at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:443)
        at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1074)
        at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1061)
        at java.base/java.security.AccessController.doPrivileged(Native Method)
        at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask.run(SSLEngineImpl.java:1008)
        at org.apache.kafka.common.network.SslTransportLayer.runDelegatedTasks(SslTransportLayer.java:430)
        at org.apache.kafka.common.network.SslTransportLayer.handshakeUnwrap(SslTransportLayer.java:514)
        at org.apache.kafka.common.network.SslTransportLayer.doHandshake(SslTransportLayer.java:368)
        at org.apache.kafka.common.network.SslTransportLayer.handshake(SslTransportLayer.java:291)
        at org.apache.kafka.common.network.KafkaChannel.prepare(KafkaChannel.java:178)
        at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:543)
        at org.apache.kafka.common.network.Selector.poll(Selector.java:481)
        at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:561)
        at org.apache.kafka.clients.producer.internals.Sender.runOnce(Sender.java:327)
        at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:242)
        at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.security.InvalidAlgorithmParameterException: the trustAnchors parameter must be non-empty
        at java.base/java.security.cert.PKIXParameters.setTrustAnchors(PKIXParameters.java:200)
        at java.base/java.security.cert.PKIXParameters.<init>(PKIXParameters.java:120)
        at java.base/java.security.cert.PKIXBuilderParameters.<init>(PKIXBuilderParameters.java:104)
        at java.base/sun.security.validator.PKIXValidator.<init>(PKIXValidator.java:99)
        ... 25 common frames omitted
2022-06-24 12:59:22,084 [kafka-producer-network-thread | producer-tb-kafka-node-f0cfd6c0-f13b-11ec-a37e-bdd7f428800a-thingsboard-firmware-testing] WARN  o.apache.kafka.clients.NetworkClient - [Producer clientId=producer-tb-kafka-node-f0cfd6c0-f13b-11ec-a37e-bdd7f428800a-thingsboard-firmware-testing] Connection to node -1 (pkc-l7pr2.ap-south-1.aws.confluent.cloud/43.204.44.151:9092) terminated during authentication. This may happen due to any of the following reasons: (1) Authentication failed due to invalid credentials with brokers older than 1.0.0, (2) Firewall blocking Kafka TLS traffic (eg it may only allow HTTPS traffic), (3) Transient network issue.
2022-06-24 12:59:22,084 [kafka-producer-network-thread | producer-tb-kafka-node-f0cfd6c0-f13b-11ec-a37e-bdd7f428800a-thingsboard-firmware-testing] WARN  o.apache.kafka.clients.NetworkClient - [Producer clientId=producer-tb-kafka-node-f0cfd6c0-f13b-11ec-a37e-bdd7f428800a-thingsboard-firmware-testing] Bootstrap broker pkc-l7pr2.ap-south-1.aws.confluent.cloud:9092 (id: -1 rack: null) disconnected
2022-06-24 12:59:22,194 [kafka-producer-network-thread | producer-tb-kafka-node-f0cfd6c0-f13b-11ec-a37e-bdd7f428800a-thingsboard-firmware-testing] WARN  o.a.kafka.common.network.Selector - [Producer clientId=producer-tb-kafka

Environment

• OS: Ubuntu v20.04
• ThingsBoard: CE v3.3.3
• Browser: Chrome v102.0.5005.115
• Java: 11

[mde2017]

Hi @saradindusengupta,

you probably have to set the correct headers for authentication e.g. SASL/PLAIN with API Key:

Key	Value
ssl.endpoint.identification.algorithm	https
sasl.mechanism	PLAIN
sasl.jaas.config	org.apache.kafka.common.security.plain.PlainLoginModule required username="API_KEY" password="API_SECRET";
security.protocol	SASL_SSL

Furthermore I can recommend using the Kafka Integrations of the ThingsBoard PE Version. We use them for a couple of years in production environments with Confluent Cloud.

[saradindusengupta]

Hi @mde2017,
I have added the mentioned key attributes for Confluent Cloud under the other attributes section while testing, it produced the same error logs

[mde2017]

Are you able to produce messages using other clients? Have you checked the ACLs of the Apikey?

[saradindusengupta]

@mde2017, Yes, I am able to produce to the same cluster and to the same topic with confluent-kafka python SDK itself.
For testing purposes, I tested with API keys with admin-level privileges as well but every time the same error logs are produced. Also, the error logs generated by Thingsboard seem different as well.

[saradindusengupta]

@mde2017 We found the issue with the older Ubuntu LTS version [<=16]. Upgrading to a newer version solved the problem.