feature - Order of Implementation discussion of kafka to iggy bridge

[ryerraguntla]

The discussion is branching out the discussion from #3081 around the order of implementation of the kafka_iggy_bridge . This is around the rollout sequence of the kafka features bridge to iggy details will be added here for community discussion and sharing the implementation idea. All the implementation jira tickets will be added here.

[ryerraguntla]

Phased Rollout Plan

---

**Phase 1: Core Producer/Fetch **

Goal: Kafka clients can produce and consume messages without consumer groups.

API Keys:
- ApiVersions (18) — protocol handshake
-  Metadata (3) — discover brokers and topics
-  Produce (0) — send messages
-  Fetch (1) — poll messages by offset
-  ListOffsets (2) — resolve earliest/latest offsets
-  CreateTopics (19) — admin API for topic creation

Iggy Integration:
-  IggyBridge::ensure_stream_and_topic()
-  IggyBridge::produce() with balanced partitioning
-  IggyBridge::fetch() with offset-based polling
-  IggyBridge::high_watermark()

Deliverables:
1.  Standalone proxy binary (cargo build --release)
2.  Docker Compose setup (Iggy server + Kafka bridge)
3.  End-to-end test with kafka-console-producer.sh and kafka-console-consumer.sh
4.  Documentation (README.md with quick-start guide)

---

**Phase 2: Consumer Groups & Offset Management **

Goal: Full Kafka consumer group semantics (rebalancing, offset commits).

API Keys:
-  FindCoordinator (10) — locate group coordinator
-  JoinGroup (11) — join consumer group, elect leader
-  SyncGroup (14) — receive partition assignment
-  Heartbeat (12) — keep membership alive
-  OffsetCommit (8) — commit offsets for a consumer group
-  OffsetFetch (9) — fetch last committed offsets
-  LeaveGroup (13) — graceful shutdown

Persistence:

• Option A: Store in Iggy itself (create a special __consumer_offsets topic)
• Option B: SQLite database (simple, file-based)
• Option C: In-memory only (resets on bridge restart)

Recommended: Option A (Iggy-backed) for consistency with Kafka's design.
 ---

**Phase 3: Transactions & Idempotence **

Goal: Exactly-once semantics for producers.

API Keys:
-  InitProducerId (22) — get a producer ID
-  AddPartitionsToTxn (24) — mark partitions as part of transaction
-  AddOffsetsToTxn (25) — include offset commits in transaction
-  EndTxn (26) — commit or abort transaction
-  TxnOffsetCommit (28) — commit offsets transactionally

Iggy Challenges:

• Iggy does not currently support transactional writes
• Would require buffering messages in the bridge until EndTxn, then sending atomically
• Abort would discard buffered messages

Workaround: Return UNSUPPORTED_VERSION for now. Document that idempotent producers (default in Kafka 3.x+) fall back to at-least-once delivery.

---

**Phase 4: Admin APIs & Advanced Features **

Goal: Full kafka-topics.sh and kafka-configs.sh compatibility.

API Keys:
-  DeleteTopics (20) — delete Iggy stream + topic
-  DescribeConfigs (32) — return topic/broker configs
-  AlterConfigs (33) — modify configs (limited support)
-  DeleteRecords (21) — trim topic to a specific offset
-  DescribeGroups (15) — inspect consumer group state
-  ListGroups (16) — list all active consumer groups
-  SaslHandshake (17) / SaslAuthenticate (36) — map to Iggy auth

---

Deployment Architecture

Option 1: Standalone Sidecar

┌─────────────────┐       ┌─────────────────┐       ┌─────────────────┐
│ Kafka Client    │────── │ Kafka Bridge    │────── │ Iggy Server     │
│ (port 9093)     │       │ (port 9093      │       │ (port 8090 TCP) │
└─────────────────┘       │  → 8090)        │       └─────────────────┘
                          └─────────────────┘

Pros:

• Independent scaling (scale bridge separately from Iggy)
• Easy rollback (stop bridge → clients fail gracefully)
• Zero Iggy code changes

Cons:

• Extra network hop (1-2ms latency)
• Separate deployment artifact

Migration Path for Existing Kafka Users

Scenario: Team has a Kafka cluster and wants to migrate to Iggy without changing application code.

Step 1: Deploy Iggy + Bridge

Step 2: Dual-Write to Both Kafka and Iggy

Step 3: Consumer Cutover

Step 4: Decommission Kafka

Once all consumers are on Iggy, stop producing to Kafka and shut down the cluster.

---

Milestones & Timeline

| Phase | Deliverable |  Blockers |
|-------|-------------|----------|----------|
|  Phase 1 | Core producer/fetch |  None |
|  Phase 2 | Consumer groups |  Need offset persistence design decision |
|  Phase 3 | Transactions |  Iggy transactional write support |
|  Phase 4 | Admin APIs |  None |

---

Success Metrics

Phase 1 (MVP):
- Kafka producer can send messages
-  Kafka consumer can poll messages by offset
-  No code changes required on client side

Phase 2:
-  Consumer groups rebalance correctly
-  Offsets are persisted across bridge restarts
-  Heartbeat timeout triggers rebalance

Phase 3:
-  Idempotent producers don't produce duplicates (requires Iggy support)
-  Transactional writes are atomic (requires Iggy support)

Phase 4:
-  kafka-topics.sh works against Iggy
-  kafka-console-consumer.sh --group works
-  ACLs map to Iggy permissions

---

Open Questions for Discussion

1. Offset Storage: Should consumer offsets be stored in Iggy itself (a la Kafka's __consumer_offsets topic) or in an external database (SQLite, Postgres)?

2. Partition Assignment Strategy: Should the bridge support Kafka's partition assignment strategies (range, roundrobin, sticky) or always use Iggy's balanced partitioning?

3. Transactional Support: Should transactional writes be blocked (return error) or buffered in the bridge and flushed on commit (adds complexity)?

4. Authentication Mapping: How should Kafka SASL/mTLS clients map to Iggy users? One-to-one mapping, or a shared "kafka-bridge" user?

5. Deployment Model: Should the bridge be a standalone artifact, or should it be merged into the Iggy server binary as an optional feature flag?

---

Implementation will be production-ready for Phase 1 middle to end of June. Phase 2 (consumer groups) is the next critical milestone for enterprise adoption.