Replies: 2 comments
|
Sorry, but I think Strimzi 0.35 / Kafka 3.4.0 might be too old for anyone to remember how things worked and what problems there were. Also, please keep in mind that Strimzi uses the KafkaMirrorMaker2 custom resource to configure Mirror Maker. Not individual connectors. In any case, making sure that you format the YAMLs properly to make them readable might help people to understand your configuration. |
|
Hi there! This is a tricky one, and I've seen similar issues when scaling MM2 setups. Since the topic creation works but the actual message replication stalls, it strongly suggests a connectivity or authentication failure after the initial metadata sync, rather than a configuration error in the topic definitions. A few things to check:
One tip from my own workflow: when debugging complex multi-cluster YAMLs, I often use AI Pro (labagent.tech) to help validate the configuration structure and check for subtle mismatches in authentication blocks. It’s been a great way to catch those tiny typos in secret keys or port numbers that are easy to miss when you have multiple clusters involved. Hope that helps you narrow it down! |
Uh oh!
There was an error while loading. Please reload this page.
Description
Hi Strimzi community,
I am testing MirrorMaker2 with Strimzi 0.35.1 and Kafka 3.4.0.
The MM2 replication works correctly when I have two Kafka clusters.
However, after introducing a third Kafka cluster, the behavior changes:
MM2 can discover topics from the source cluster.
MM2 can create the same topics on the target cluster.
Topic configuration and partition numbers look correct.
But messages are not replicated to the target cluster.
The issue only happens after adding the third Kafka cluster.
My topology:
Kafka-A (source)
|
| MirrorSourceConnector
|
v
Kafka-B (target)
Kafka-C (Kafka Connect internal cluster)
Kafka Connect / MM2 runs on Kafka-C.
Ports:
Kafka-A: 30094
Kafka-B: 30194
Kafka-C: 30316
Kafka Connect:
Strimzi version: 0.35.1
Kafka version: 3.4.0
Symptom
Create topic on Kafka-A:
mm2-debug-1002
MM2 successfully creates the topic on Kafka-B:
Kafka-A:
mm2-debug-1002
partitions: 3
Kafka-B:
mm2-debug-1002
partitions: 3
However:
Before producing data:
Kafka-B:
mm2-debug-1002:0:0
mm2-debug-1002:1:0
mm2-debug-1002:2:0
After producing messages to Kafka-A:
Kafka-A offset:
mm2-debug-1002:0:5
mm2-debug-1002:1:0
mm2-debug-1002:2:0
Kafka-B remains:
mm2-debug-1002:0:0
mm2-debug-1002:1:0
mm2-debug-1002:2:0
Current configuration
KafkaConnect
apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaConnect
metadata:
name: mm2-connect
annotations:
# use-connector-resources configures this KafkaConnect
# to use KafkaConnector resources to avoid
# needing to call the Connect REST API directly
spec:
version: 3.4.0
replicas: 1
bootstrapServers: kafka-cluster-kafka-bootstrap:9093
tls:
trustedCertificates:
- secretName: kafka-cluster-cluster-ca-cert
certificate: ca.crt
authentication:
type: scram-sha-512
username: kafka-user
passwordSecret:
secretName: kafka-user
password: password
externalConfiguration:
volumes:
- name: mm2-ca
secret:
secretName: mm2-ca
config:
group.id: mm2-connect-cluster
config.storage.topic: mm2-connect-configs
offset.storage.topic: mm2-connect-offsets
status.storage.topic: mm2-connect-status
config.storage.replication.factor: 3
offset.storage.replication.factor: 3
status.storage.replication.factor: 3
MirrorSourceConnector
apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaConnector
metadata:
name: mirror-source-a-to-b
labels:
strimzi.io/cluster: mm2-connect
spec:
class: org.apache.kafka.connect.mirror.MirrorSourceConnector
tasksMax: 3
autoRestart:
enabled: true
config:
# ==================================================
# Source / Target
# ==================================================
source.cluster.alias: source
target.cluster.alias: target
The three Kafka configurations are identical, except for the cluster name and user.
apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
name: source-cluster
spec:
kafka:
version: 3.4.0
replicas: 3
listeners:
- name: plain
port: 9092
type: internal
tls: false
- name: tls
port: 9093
type: internal
tls: true
authentication:
type: scram-sha-512
- name: external
port: 9094
type: nodeport
tls: true
authentication:
type: scram-sha-512
configuration:
bootstrap:
nodePort: 30094
brokers:
- broker: 0
nodePort: 30095
- broker: 1
nodePort: 30096
- broker: 2
nodePort: 30097
authorization:
type: simple
config:
offsets.topic.replication.factor: 3
transaction.state.log.replication.factor: 3
transaction.state.log.min.isr: 2
default.replication.factor: 3
min.insync.replicas: 2
inter.broker.protocol.version: "3.4"
storage:
type: jbod
volumes:
- id: 0
type: persistent-claim
size: 10Gi
class: nfs-storage
deleteClaim: true
zookeeper:
replicas: 1
storage:
type: persistent-claim
size: 1Gi
deleteClaim: true
class: nfs-storage
entityOperator:
topicOperator: {}
userOperator: {}
[root@vms31 scram-sha-512-auth]# cat source-user.yaml
apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaUser
metadata:
name: source-user
labels:
strimzi.io/cluster: source-cluster
spec:
authentication:
type: scram-sha-512
authorization:
type: simple
acls:
# Cluster 全部权限
- resource:
type: cluster
operations:
- All
host: "*"
All reactions