Exactly-once delivery with per-sink independent checkpoints

[vnvo]

Summary

• Kafka transactional producer (exactly_once: true) — begin_transaction/commit_transaction per batch, producer fencing detection (SinkError::Fatal), transaction.timeout.ms=60000
• NATS JetStream dedup — Nats-Msg-Id header on every message for server-side dedup within duplicate_window
• Redis idempotency — idempotency_key field in every XADD payload for consumer-side dedup
• Per-sink independent checkpoints — each sink advances its own checkpoint; source replays from min(sink checkpoints); fastest sink never waits for slowest
• Legacy checkpoint fallback — seamless migration from pre-per-sink checkpoint format
• Source::compare_checkpoints — required trait method for correct checkpoint ordering (MySQL file+pos, Postgres LSN→u64, Turso change_id)
• Coordinator hardening — policy gate before commits (not after), parallel per-sink commits via join_all, partial batch flush fix (recv_many timeout)
• Default max_bytes bumped 3MB → 16MB — old default silently capped batches, causing disproportionate exactly-once overhead

Benchmarks (tuned, single dev machine, Docker containers)

Source	at-least-once	exactly-once	Overhead
MySQL 10M rows	151K events/s	134K events/s	~11%
Postgres 1M rows	57K events/s	53K events/s	~7%

Chaos & Testing

• --scenario exactly-once crash recovery with read_committed consumer verification
• 4 Kafka transactional integration tests (atomic batch, single event, producer fencing, multi-batch ordering)
• Configurable drain: --drain-target, --drain-writers, --drain-timeout, --drain-max-bytes, --exactly-once
• Effective pipeline config dump before each benchmark run
• Partial batch flush regression test

Infrastructure

• Docker Compose profile isolation: base, mysql-infra, pg-infra, kafka-infra (start/stop DeltaForge without touching infra)
• Loki + Promtail log aggregation — view all container logs in Grafana
• Grafana dashboard: template variables (instance/pipeline/source/sink/processor), "Checkpoints & Exactly-Once" row, transaction commit/abort panels, repeating per-pipeline detail rows
• Prometheus + Loki retention set to 7 days
• New metrics: deltaforge_sink_txn_commits_total, deltaforge_sink_txn_aborts_total, per-sink checkpoint status/age gauges

Docs

• Performance guide updated with corrected benchmarks and max_bytes tuning guidance
• Kafka sink: exactly-once section (requirements, transactional overrides, fatal errors, performance)
• NATS sink: deduplication section (Nats-Msg-Id, duplicate_window)
• Redis sink: idempotency section (idempotency_key field)
• Sinks overview: per-sink checkpoints, delivery guarantee tiers table
• Landing page: updated feature cards, benchmark numbers, meta tags
• Changelog updated

Test plan

• cargo test --workspace --lib — 365+ tests pass
• cargo clippy --all-targets --all-features -- -D warnings — clean
• cargo test -p sinks --test kafka_sink_tests -- --include-ignored — 4 new EO tests pass (requires Docker)
• Drain benchmark: MySQL 134K events/s with EO, 151K without
• Drain benchmark: Postgres 53K events/s with EO, 57K without
• --scenario exactly-once --source mysql chaos crash recovery
• --scenario all --source mysql regression (existing scenarios still pass)

Checklist

• Tests pass (cargo test)
• Code formatted (cargo fmt)