Offset Sync Issue

[yogeshgoel001]

We have environments
Event Hub (Producer)
Kafka Main (Consumer)
Kafka DR (Conumser if Kafka Main goes down)
We are performing failover testing, Mirroring is working good between Producer to consumer.
But when we do failover to DR environment from Mirror Maker, it copies all the data from Event Hub to DR.

Requirement is to copy leftover data from Event HUB to DR to avoid any duplication.

configurations : consumer.auto.offset.reset : latest

But issue is still same

[scholzj]

You should probably explain what exactly you are doing, share your configurations, what versions you use etc. None of that is really clear from your description. Assuming you are talking about Azure Event Hub - it is not even an Apache Kafka - so is it really expected that any offset synchronization will work?

[yogeshgoel001]

We need to do Failover testing, so just need to avoid duplicate message (sent on primary kafka cluster) to be sent on DR kafka from Event HUB.
Source : Event Hub
Destination : Kafka clusters (primary and DR)
Here is the configuration file.

kind: KafkaMirrorMaker2
metadata:
creationTimestamp: "2024-02-29T22:48:43Z"
generation: 41
name: my-mm2-cluster-from-eh
namespace: xxxxxxxxxx
spec:
clusters:

• alias: kafka
  authentication:
  certificateAndKey:
  certificate: tls.crt
  key: tls.key
  secretName: secret-tls
  type: tls
  bootstrapServers: kafka-dev01-node1:9093
  config:
  config.storage.replication.factor: 1
  consumer.auto.offset.reset: latest
  consumer.connections.max.idle.ms: 18000
  consumer.metadata.max.age.ms: 18000
  enable.auto.commit: true
  offset.storage.replication.factor: 1
  ssl.cipher.suites: TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
  ssl.enabled.protocols: TLSv1.2
  ssl.protocol: TLSv1.2
  status.storage.replication.factor: 1
  tls:
  trustedCertificates: []
• alias: ehub
  authentication:
  passwordSecret:
  password: eventhubspassword
  secretName: eventhubssecret
  type: plain
  username: $ConnectionString
  bootstrapServers: ehub-eu2-dev-test-shared:9093
  config:
  acks: all
  config.storage.replication.factor: 1
  enable.idempotence: true
  max.in.flight.requests.per.connection: 5
  retries: 2147483647
  status.storage.replication.factor: 1
  tls:
  trustedCertificates: []
  connectCluster: kafka
  livenessProbe:
  initialDelaySeconds: 15
  timeoutSeconds: 5
  mirrors:
• checkpointConnector:
  config:
  checkpoints.topic.replication.factor: 1
  replication.policy.class: org.apache.kafka.connect.mirror.IdentityReplicationPolicy
  sync.group.offsets.enabled: "false"
  groupsPattern: .*
  heartbeatConnector:
  config:
  heartbeats.topic.replication.factor: 1
  sourceCluster: ehub
  sourceConnector:
  config:
  offset-syncs.topic.replication.factor: 1
  replication.factor: 1
  replication.policy.class: org.apache.kafka.connect.mirror.IdentityReplicationPolicy
  sync.topic.acls.enabled: "true"
  targetCluster: kafka
  topicsPattern: kafka-topic
  readinessProbe:
  initialDelaySeconds: 15
  timeoutSeconds: 5
  replicas: 1
  resources:
  limits:
  cpu: "1"
  memory: 2Gi
  requests:
  cpu: "1"
  memory: 1Gi
  version: 3.6.1

[scholzj]

You will have to format it to make it readable. Also, as I said, do you have any docs from Azure or anything that suggests this is supported?

[yogeshgoel001]

kafkamirromaker2.zip
Uploading zip file. trying to find the document if this feature is supported. Or we need to find the alternative.