feat(connectors): Iggy Source/Sink Connector: Native Cluster-to-Cluster Replication

[kparisa]

Description

Summary

Implement an Iggy Source/Sink Connector that consumes data from an upstream Iggy cluster and publishes it into a downstream Iggy cluster.

This enables seamless cluster-to-cluster replication and provides the foundation for multi-region deployments, disaster recovery, active-active architectures, edge-to-cloud streaming, and hierarchical streaming topologies.

Unlike traditional application-level replication, this connector allows operators to replicate selected streams, partitions, or namespaces without requiring producers or consumers to change.

Motivation

Many production deployments require data to exist in multiple locations.

Today, applications would need to publish to multiple Iggy clusters directly, increasing complexity and coupling replication logic into business applications.

A native Iggy Source/Sink Connector would make replication a first-class capability by continuously consuming from one cluster and publishing to another while preserving ordering and delivery guarantees.

         Producers
             │
             ▼
    +------------------+
    |  Iggy Cluster A  |
    |   (Primary)      |
    +------------------+
             │
  Iggy Source/Sink Connector
             │
             ▼
    +------------------+
    |  Iggy Cluster B  |
    |  (Replica)       |
    +------------------+
             │
      Local Consumers

The connector should behave like any other Iggy connector:

• Source = upstream Iggy
• Sink = downstream Iggy

Aspects to cover (ideal, not a comprehensive list, can be broken down into follow-up tasks after the basic version is in place):

• Iggy → Iggy replication
• configurable source streams
• configurable destination streams
• partition-aware replication
• offset checkpointing
• resumable replication
• configurable batching
• configurable compression
• TLS support
• authentication support
• metrics
• retry with backoff
• configurable parallelism

Affected area / component

Connectors

Proposed solution

No response

Alternatives considered

No response

Contribution

• I'm willing to submit a pull request to implement this feature

Good first issue

• I think this could be a good first issue for a new contributor

[raman20]

hey @kparisa i would like to work on this...

[kparisa]

@raman20 Thank you! Assigned to you.
This will be a great feature!

You can implement it as a source or sink connector.

[ryerraguntla]

@raman20 - When you raise PR, please raise separate PR for Sink and source. Please ping me on discord, I want to share a work in progress blog article for developing connectors for Iggy.