Kafka/distributed commit log service in Go.
- Implement Kafka in Go
- Protocol compatible with Kafka so Kafka clients and services work with Jocko
- Make operating simpler
- Distribute a single binary
- Use Serf for discovery, Raft for consensus (and remove the need to run ZooKeeper)
- Smarter configuration settings
- Able to use percentages of disk space for retention policies rather than only bytes and time kept
- Handling size configs when you change the number of partitions or add topics
- Learn a lot and have fun
- Producing
- Fetching
- Partition consensus and distribution
- Protocol
- Produce
- Fetch
- Metadata
- Create Topics
- Delete Topics
- Consumer group
- Discovery
- API versioning
- Replication [current task]
- Tests [current task]
- How Jocko's built-in service discovery and consensus works
- How Jocko's (and Kafka's) storage internals work
├── broker broker subsystem
├── cmd commands
│ └── jocko command to run a Jocko broker and manage topics
├── commitlog low-level commit log implementation
├── examples examples running/using Jocko
│ ├── cluster example booting up a 3-broker Jocko cluster
│ └── sarama example producing/consuming with Sarama
├── protocol golang implementation of Kafka's protocol
├── raft wrapper around Hashicorp's Raft lib to handle consensus
├── serf wrapper around Hashicorp's Serf lib to handle service discovery
├── prometheus wrapper around Prometheus' client lib to handle metrics
├── server API subsystem
└── testutil test utils
└── mock mocks of the various subsystems
MIT
- travisjeffery.com
- GitHub @travisjeffery
- Twitter @travisjeffery
- Medium @travisjeffery