Stability: 1 - Experimental
AllData is a distributed master-less append-only immutable event store database implementing "All Data" part of Lambda Architecture.
... to make the irreducible basic elements as simple and as few as possible without having to surrender the adequate representation of a single datum of experience.
-- Albert Einstein, "On the Method of Theoretical Physics" The Herbert Spencer Lecture, delivered at Oxford (10 June 1933)
AllData is a distributed master-less append-only immutable event store database implementing "All Data" part of Lambda Architecture. It attempts to do nothing more than fulfill the "All Data" design considerations that are as follows:
- Lambda Architecture. AllData is meant to be part of a system designed according to Lambda Architecture.
- Event storage. AllData is an event store. It stores events and is an "append-only" store.
- Event idempotency. AllData is designed to store idempotent events. You may see the same event more than once during batch processing.
- Always available writes. AllData is structured to always write the data somewhere.
- No immediate read requirement. As part of Lambda Architecture, the availability of reads written to AllData is not required to be immediate. Instead, it is available for batch processing later. This allows one to use something simpler and non-persistent for real-time processing because AllData serves as the eventual backup.
- No data loss. Because AllData needs to store everything, it needs to handle replication in case of node failures.
- No single point of failure. To support always available writes as well as no data loss, a master-less (peer-to-peer) design is required.
- Efficient batch processing. AllData is designed to support efficient sequential reads that are characteristic of Lambda Architecture batch processing (although this is delegated to particular storage module implementations).
- alldata-client-http: AllData HTTP client module
- alldata-server-http: AllData HTTP server module
- alldata-peer-client-http: AllData Peer HTTP client module
- alldata-peer-server-http: AllData Peer HTTP server module
- alldata-storage-leveldb: AllData LevelDB-backed storage module
- alldata-coordinator: AllData request coordinator module
- alldata-keygen: AllData key generation module
- gossipmonger: Scuttlebutt gossip protocol implementation for real-time peer-to-peer state distribution