-
Notifications
You must be signed in to change notification settings - Fork 0
ADR 0006 Redpanda Default Broker
Tiana_ edited this page May 30, 2026
·
1 revision
Status: Accepted Date: 2026-04-25 Decider: Maintainer
The event bus is critical: outbox dispatching, AML consumer, webhook delivery, audit shipping all rely on it. Choices among Kafka-API-compatible brokers (so adopters can swap as needed):
- Apache Kafka with KRaft mode (post-ZooKeeper) - industry standard, mature, complex
- Apache Kafka with ZooKeeper - legacy mode, deprecated path
- Redpanda - Kafka API-compatible, single binary, no ZooKeeper, no JVM
- WarpStream - Kafka API on S3 - interesting but newer
- Confluent Cloud / AWS MSK - managed services for production
For local development (docker compose up):
- Apache Kafka: ~30s startup, requires ZooKeeper or KRaft setup, ~1.5 GB RAM, JVM heap tuning
- Redpanda: ~3s startup, single binary, ~200 MB RAM, no JVM
For production (where adopter deploys to their cluster), all options are viable as long as they speak Kafka API.
Redpanda as the default broker in docker compose and Helm chart values:
- Bundled in
docker composefor fast local DX - Helm chart default
values.yamlships Redpanda manifests - Adopters can swap by changing
values.broker.kindor pointingbootstrap-serversat any Kafka-compatible cluster - No code changes between brokers (we use Kafka client API only)
Production guidance documented:
- Self-hosted: Strimzi operator (Apache Kafka on K8s) or Redpanda K8s operator
- Managed: Confluent Cloud, AWS MSK, Aiven for Apache Kafka, Redpanda Cloud
- All work without changes
- 30-second sandbox: critical for our killer DX. Apache Kafka + ZK was 5x slower
- Lower memory floor: dev laptops with 8 GB run smoothly
- No JVM overhead in broker layer (Redpanda is C++)
- Same Kafka client code: producers/consumers identical regardless of broker
- Standard Kafka topics: tiered storage, compaction, ACLs all work
- Adopter freedom: drop in any Kafka-API broker for production
- Newer than Apache Kafka: less production-tested at extreme scale (though Redpanda has shipped to enterprise customers since 2020)
- License: Redpanda Community Edition is BSL-licensed (limited commercial use restriction). For our docker-compose default this is fine; adopters running Redpanda commercially read the Redpanda license terms (most adopters use Apache Kafka in production anyway)
- Some Kafka feature lag: schemas, exotic configs occasionally behind Apache Kafka. Acceptable for our use cases.
- Some adopters prefer Apache Kafka for "principles" reasons. Helm chart documents the swap (5-line config change).
- Rejected for default: 30s startup, 1.5 GB RAM hurts DX
-
First-class supported as alternative: Helm
values-kafka.yamlruns Kafka via Strimzi - Production-proven; adopters in regulated environments often pick Kafka
- Rejected: requires SaaS sign-up, conflicts with offline / self-hosted DX
- Documented as production option
- Rejected: not Kafka-API-compatible (different client lib)
- Adopters can wrap our
EventPublisherinterface to use Pulsar - out of OSS scope
- Rejected: incompatible with our Kafka-API-based outbox dispatcher
- Different consumer group semantics
- Rejected: not a streaming bus; designed for event sourcing primary store
- Different mental model
- Rejected: doesn't survive process restart
- Doesn't scale beyond one JVM
- Performance test: 5000 events/sec sustained on Redpanda single node
- Compatibility test: same producer code emits same messages on Apache Kafka 3.8 and Redpanda 24.3
- Switch test:
docker-compose -f base.yml -f kafka.yml upstarts Apache Kafka instead, all integration tests pass
- Architecture-Event-Flow - full event bus design
-
deploy/docker-compose.yml- default Redpanda config -
deploy/docker-compose.kafka.yml- Apache Kafka via Bitnami chart -
deploy/helm/fincore-engine/values-kafka.yaml- production Kafka via Strimzi
- Overview
- Services
- Data Model
- Domain Model
- Event Flow
- Security
- Observability
- Resilience
- SLA / SLI / SLO