[Bug]: Fail to run Mirror Maker 2 connector task when using OAUTH to connect to kafka clusters #9786

Eric84626 · 2024-03-06T12:17:01Z

Bug Description

I applied Mirror Maker 2 cr, and the pod is running successfully. But I checked the status of this new connector via rest api,
I got "caused by: java.lang.IllegalArgumentException: Value not specified for key 'XXXX(mm2 cluster alias)' in JAAS config"

My authentication type is "oauth", and set trustedCertificates for TLS connection to the OAuth server.

Steps to reproduce

No response

Expected behavior

No response

Strimzi version

0.39.0

Kubernetes version

K8s 1.27

Installation method

No response

Infrastructure

No response

Configuration files and logs

No response

Additional context

I think I find the root cause of this issue, I will create a PR, and fix it.

Eric84626 · 2024-03-06T12:17:17Z

I think I find the root cause of this issue, I will create a PR, and fix it.

scholzj · 2024-03-06T12:39:52Z

You haven't provided any configuration or logs or anything. So maybe you should have opened a discussion instead of a Bug issue?

Eric84626 · 2024-03-06T12:55:31Z

Hello @scholzj
https://github.com/strimzi/strimzi-kafka-operator/blob/0.39.0/cluster-operator/src/main/java/io/strimzi/operator/cluster/operator/assembly/KafkaMirrorMaker2AssemblyOperator.java#L491C1-L492C1
As above code, you can see that there are unnecessary two double quotes on this trutstore file path.
I know this code already moved to another place(https://github.com/strimzi/strimzi-kafka-operator/blob/main/cluster-operator/src/main/java/io/strimzi/operator/cluster/model/KafkaMirrorMaker2Connectors.java#L285) in main branch.
I want to remove these two unnecessary two double quotes.

scholzj · 2024-03-06T12:58:48Z

I think you still need to provide all the information to make it possible to see if it really is a bug and if it works.

Eric84626 · 2024-03-06T13:03:35Z

I checked the config of my mm2 connecotor.
I can see below config is generated by operator
oauth.ssl.truststore.location="/tmp/kafka/clusters/"strimzi-poc-e2"-oauth.truststore.p12"

Error messsage:
org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:503)\n\t... 22 more\nCaused by: java.lang.IllegalArgumentException: Value not specified for key 'strimzi-poc-e2' in JAAS config\n\tat org.apache.kafka.com

Eric84626 · 2024-03-06T13:08:27Z

I think below commit induced this issue. Please review it.
64cf4a3#diff-c6d9b371d64b172f4e174d61f4a6b81ed129010c41a016f0bbf29dddc62df009L528

Eric84626 · 2024-03-06T13:28:27Z

Above commit was merged to main branch from 0.37.0-rc1, I reverted my operator to 0.36.1, and I can not see this issue anymore.

scholzj · 2024-03-07T08:59:55Z

Triaged on the community call on 7.3.2024: Please provide a custom resource for reproducing this because it is not really clear what should have the quotes and where should they be. They might be required for parsing and security reasons.

Eric84626 · 2024-03-07T09:42:00Z

Operator version: 0.39.0
MM2 CR yaml:

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaMirrorMaker2
metadata:
  name: kafka-mirror-maker-2
spec:
  clusters:
  - alias: "strimzi-poc-e1"
    authentication:
      type: tls
      certificateAndKey:
        secretName: strimzi-poc-kafka-bootstrap
        certificate: tls.crt
        key: tls.key
    bootstrapServers: strimzi-poc-kafka-bootstrap:9093
    tls:
      trustedCertificates:
      - certificate: ca.crt
        secretName: strimzi-poc-kafka-bootstrap
  - alias: "strimzi-poc-e2"
    authentication:
      clientId: xxxxxxx
      clientSecret: 
        key: client_secret
        secretName: jwt-client-secret
      tokenEndpointUri: https://xxxxxxxxx
# We can reproduce this issue when we enabled tlsTrustedCertificates for oauth
      tlsTrustedCertificates:
      - certificate: ca.crt
         secretName: strimzi-poc-kafka-bootstrap
      type: oauth
    bootstrapServers: XXXXXXXXXXXXX:443
    config:
      config.storage.replication.factor: -1
      offset.storage.replication.factor: -1
      status.storage.replication.factor: -1
    tls:
      trustedCertificates:
      - certificate: ca.crt
        secretName: strimzi-poc-kafka-bootstrap
  connectCluster: "strimzi-poc-e1"
  mirrors:
  - checkpointConnector:
      config:
        checkpoints.topic.replication.factor: 3
    groupsPattern: .*
    heartbeatConnector:
      config:
        heartbeats.topic.replication.factor: 3
    sourceCluster: "strimzi-poc-e2"
    sourceConnector:
      config:
        offset-syncs.topic.replication.factor: 3
        replication.factor: 3
    targetCluster: "strimzi-poc-e1"
    topicsPattern: .*
  replicas: 1
  resources:
    limits:
      cpu: 1000m
      memory: 2Gi
    requests:
      cpu: 1000m
      memory: 2Gi
  version: 3.6.1

MM2 connector config(generated by operator):
GET /connectors/strimzi-poc-e2->strimzi-poc-e1.MirrorSourceConnector/config

{
	"connector.class": "org.apache.kafka.connect.mirror.MirrorSourceConnector",
	"replication.factor": "3",
	"source.cluster.ssl.truststore.location": "/tmp/kafka/clusters/strimzi-poc-e2.truststore.p12",
	"source.cluster.ssl.truststore.type": "PKCS12",
	"source.cluster.alias": "strimzi-poc-e2",
	"target.cluster.ssl.truststore.location": "/tmp/kafka/clusters/strimzi-poc-e1.truststore.p12",
	"source.cluster.sasl.login.callback.handler.class": "io.strimzi.kafka.oauth.client.JaasClientOauthLoginCallbackHandler",
	"source.cluster.config.storage.replication.factor": "-1",
	"target.cluster.ssl.truststore.type": "PKCS12",
	"target.cluster.security.protocol": "SSL",
	"target.cluster.ssl.keystore.location": "/tmp/kafka/clusters/strimzi-poc-e1.keystore.p12",
	"consumer.client.rack": "${file:/tmp/strimzi-connect.properties:consumer.client.rack}",
	"offset-syncs.topic.replication.factor": "3",
	"topics": ".*",
	"target.cluster.ssl.keystore.password": "${file:/tmp/strimzi-mirrormaker2-connector.properties:ssl.keystore.password}",
	"groups": ".*",
	"target.cluster.ssl.truststore.password": "${file:/tmp/strimzi-mirrormaker2-connector.properties:ssl.truststore.password}",
	"source.cluster.status.storage.replication.factor": "-1",
	"source.cluster.sasl.jaas.config": "org.apache.kafka.common.security.oauthbearer.OAuthBearerLoginModule required oauth.token.endpoint.uri=\"https://XXXXXXXXX\" oauth.client.secret=\"${file:/tmp/strimzi-mirrormaker2-connector.properties:strimzi-poc-e2.oauth.client.secret}\" oauth.ssl.truststore.location=\"/tmp/kafka/clusters/\"strimzi-poc-e2\"-oauth.truststore.p12\" oauth.ssl.truststore.type=\"PKCS12\" oauth.ssl.truststore.password=\"\"${file:/tmp/strimzi-mirrormaker2-connector.properties:oauth.ssl.truststore.password}\"\" oauth.client.id=\"XXXXXXXX\";",
	"source.cluster.bootstrap.servers": "XXXXXXXX:443",
	"source.cluster.sasl.mechanism": "OAUTHBEARER",
	"target.cluster.alias": "strimzi-poc-e1",
	"target.cluster.ssl.keystore.type": "PKCS12",
	"name": "strimzi-poc-e2->strimzi-poc-e1.MirrorSourceConnector",
	"target.cluster.bootstrap.servers": "strimzi-poc-kafka-bootstrap:9093",
	"source.cluster.offset.storage.replication.factor": "-1",
	"source.cluster.ssl.truststore.password": "${file:/tmp/strimzi-mirrormaker2-connector.properties:ssl.truststore.password}",
	"source.cluster.security.protocol": "SASL_SSL"
}

You can see below jaas config from above json response, there are unnecessary two double quotes(/"strimzi-poc-e2"-oauth) on this trutstore file path.

oauth.ssl.truststore.location=\"/tmp/kafka/clusters/\"strimzi-poc-e2\"-oauth.truststore.p12\"

I think the file path str should not contain "".
As current operator logic, the truststore file name should be "strimzi-poc-e2"-oauth.truststore.p12

GET /connectors/strimzi-poc-e2->strimzi-poc-e1.MirrorSourceConnector/status
Error response:

{
	"name": "strimzi-poc-e2->strimzi-poc-e1.MirrorSourceConnector",
	"connector": {
		"state": "FAILED",
		"worker_id": "kafka-mirror-maker-2-mirrormaker2-0.kafka-mirror-maker-2-mirrormaker2.strimzi-kafka-dev.svc:8083",
		"trace": "org.apache.kafka.common.KafkaException: The constructor of org.apache.kafka.clients.admin.ForwardingAdmin threw an exception\n\tat org.apache.kafka.common.utils.Utils.newParameterizedInstance(Utils.java:471)\n\tat org.apache.kafka.connect.mirror.MirrorConnectorConfig.forwardingAdmin(MirrorConnectorConfig.java:231)\n\tat org.apache.kafka.connect.mirror.MirrorSourceConnector.start(MirrorSourceConnector.java:160)\n\tat org.apache.kafka.connect.runtime.WorkerConnector.doStart(WorkerConnector.java:192)\n\tat org.apache.kafka.connect.runtime.WorkerConnector.start(WorkerConnector.java:217)\n\tat org.apache.kafka.connect.runtime.WorkerConnector.doTransitionTo(WorkerConnector.java:376)\n\tat org.apache.kafka.connect.runtime.WorkerConnector.doTransitionTo(WorkerConnector.java:357)\n\tat org.apache.kafka.connect.runtime.WorkerConnector.doRun(WorkerConnector.java:144)\n\tat org.apache.kafka.connect.runtime.WorkerConnector.run(WorkerConnector.java:122)\n\tat org.apache.kafka.connect.runtime.isolation.Plugins.lambda$withClassLoader$1(Plugins.java:236)\n\tat java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)\n\tat java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)\n\tat java.base/java.lang.Thread.run(Thread.java:840)\nCaused by: org.apache.kafka.common.KafkaException: Failed to create new KafkaAdminClient\n\tat org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:519)\n\tat org.apache.kafka.clients.admin.Admin.create(Admin.java:144)\n\tat org.apache.kafka.clients.admin.ForwardingAdmin.<init>(ForwardingAdmin.java:51)\n\tat java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)\n\tat java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:77)\n\tat java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)\n\tat java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:499)\n\tat java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:480)\n\tat org.apache.kafka.common.utils.Utils.newParameterizedInstance(Utils.java:460)\n\t... 14 more\nCaused by: org.apache.kafka.common.KafkaException: Failed to create new NetworkClient\n\tat org.apache.kafka.clients.ClientUtils.createNetworkClient(ClientUtils.java:245)\n\tat org.apache.kafka.clients.ClientUtils.createNetworkClient(ClientUtils.java:185)\n\tat org.apache.kafka.clients.admin.KafkaAdminClient.createInternal(KafkaAdminClient.java:503)\n\t... 22 more\nCaused by: java.lang.IllegalArgumentException: Value not specified for key 'strimzi-poc-e2' in JAAS config\n\tat org.apache.kafka.common.security.JaasConfig.parseAppConfigurationEntry(JaasConfig.java:116)\n\tat org.apache.kafka.common.security.JaasConfig.<init>(JaasConfig.java:63)\n\tat org.apache.kafka.common.security.JaasContext.load(JaasContext.java:93)\n\tat org.apache.kafka.common.security.JaasContext.loadClientContext(JaasContext.java:87)\n\tat org.apache.kafka.common.network.ChannelBuilders.create(ChannelBuilders.java:167)\n\tat org.apache.kafka.common.network.ChannelBuilders.clientChannelBuilder(ChannelBuilders.java:81)\n\tat org.apache.kafka.clients.ClientUtils.createChannelBuilder(ClientUtils.java:117)\n\tat org.apache.kafka.clients.ClientUtils.createNetworkClient(ClientUtils.java:217)\n\t... 24 more\n"
	},
	"tasks": [],
	"type": "source"
}

You can focus on "Caused by: java.lang.IllegalArgumentException: Value not specified for key 'strimzi-poc-e2' in JAAS config\n\tat " from above json response.

…trimzi#9786 Signed-off-by: Jakub Scholz <www@scholzj.com>

scholzj · 2024-03-07T16:52:01Z

Thanks. That helps. For some reason, we seem to miss a system test for this. I opened #9799 to fix it and add the system test.

Eric84626 · 2024-03-08T01:08:39Z

OK, @scholzj , I wanted to create a PR to fix this issue.
If there is a new contribution opportunity, I will take that.

P.S. I watched your kafka session in kubcon North America, great session!!

Eric84626 added bug needs-triage labels Mar 6, 2024

scholzj removed the needs-triage label Mar 7, 2024

scholzj added the needs-triage label Mar 7, 2024

scholzj added a commit to scholzj/strimzi-kafka-operator that referenced this issue Mar 7, 2024

Fix wrong escaping of OAuth configuration in Mirror Maker 2 - Closes s…

448e7b6

…trimzi#9786 Signed-off-by: Jakub Scholz <www@scholzj.com>

scholzj mentioned this issue Mar 7, 2024

Fix wrong escaping of OAuth configuration in Mirror Maker 2 #9799

Merged

4 tasks

scholzj closed this as completed in #9799 Mar 8, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Bug]: Fail to run Mirror Maker 2 connector task when using OAUTH to connect to kafka clusters #9786

[Bug]: Fail to run Mirror Maker 2 connector task when using OAUTH to connect to kafka clusters #9786

Eric84626 commented Mar 6, 2024

Eric84626 commented Mar 6, 2024

scholzj commented Mar 6, 2024

Eric84626 commented Mar 6, 2024

scholzj commented Mar 6, 2024

Eric84626 commented Mar 6, 2024 •

edited

Eric84626 commented Mar 6, 2024 •

edited

Eric84626 commented Mar 6, 2024

scholzj commented Mar 7, 2024

Eric84626 commented Mar 7, 2024 •

edited

scholzj commented Mar 7, 2024

Eric84626 commented Mar 8, 2024

[Bug]: Fail to run Mirror Maker 2 connector task when using OAUTH to connect to kafka clusters #9786

[Bug]: Fail to run Mirror Maker 2 connector task when using OAUTH to connect to kafka clusters #9786

Comments

Eric84626 commented Mar 6, 2024

Bug Description

Steps to reproduce

Expected behavior

Strimzi version

Kubernetes version

Installation method

Infrastructure

Configuration files and logs

Additional context

Eric84626 commented Mar 6, 2024

scholzj commented Mar 6, 2024

Eric84626 commented Mar 6, 2024

scholzj commented Mar 6, 2024

Eric84626 commented Mar 6, 2024 • edited

Eric84626 commented Mar 6, 2024 • edited

Eric84626 commented Mar 6, 2024

scholzj commented Mar 7, 2024

Eric84626 commented Mar 7, 2024 • edited

scholzj commented Mar 7, 2024

Eric84626 commented Mar 8, 2024

Eric84626 commented Mar 6, 2024 •

edited

Eric84626 commented Mar 6, 2024 •

edited

Eric84626 commented Mar 7, 2024 •

edited