A mini Kafka; written in Go.
Dislog is a distributed, stateful, observable, streaming log, that runs on a gRPC server deployed with Kubernetes, with self-managed service discovery & consensus.
Dislog is a result of follow-along learnings from Travis Jeffery's Distributed Services with Go.
The Log package consists of 5 parts:
- Record: the structure of basic data stored in the log
- Store: the system file where logs are stored
- Index: the system file that houses the index entries
- Segment: the abstraction that ties a store and an index together
- Log: the abstraction that ties all the segments together
Segmentation is made possible by the gommap package. All memory managed through this package is outside of Go's own memory management. Segments are used to keep logs rolling so that older segments can be deleted when approaching max file system memory.
- Cloudfare CFSSL
- cfssl to sign, verify, and bundle TLS certificates and output the results as JSON.
- cfssljson to take that JSON output and split them into separate key, certificate, CSR, and bundle files.
- Crypto/tls
Calling Dislog a mini-Kafka is partially apt, as we do have segmentation, offsets and a bunch of other stuff that are characteristic of Kafka. However, we do not have partitioning or topics which make Kafka the go-to event streamer for production-grade systems. There is much scope for improvement in Dislog and hopefully, one day I will have the time and patience to do the research and write the code!
- Elaborate Readme
- GKE config
- FQDN
- Deploy to cloud
- Handle ungraceful shutdowns