Skip to content
This repository

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

Distributed master-less append-only immutable event store database implementing "All Data" part of Lambda Architecture.

branch: master

Fetching latest commit…

Cannot retrieve the latest commit at this time

README.md

alldata

Stability: 1 - Experimental

NPM version

AllData is a distributed master-less append-only immutable event store database implementing "All Data" part of Lambda Architecture.

All Data in Lambda Architecture

Overview

... to make the irreducible basic elements as simple and as few as possible without having to surrender the adequate representation of a single datum of experience.

-- Albert Einstein, "On the Method of Theoretical Physics" The Herbert Spencer Lecture, delivered at Oxford (10 June 1933)

AllData is a distributed master-less append-only immutable event store database implementing "All Data" part of Lambda Architecture. It attempts to do nothing more than fulfill the "All Data" design considerations that are as follows:

  • Lambda Architecture. AllData is meant to be part of a system designed according to Lambda Architecture.
  • Event storage. AllData is an event store. It stores events and is an "append-only" store.
  • Event idempotency. AllData is designed to store idempotent events. You may see the same event more than once during batch processing.
  • Always available writes. AllData is structured to always write the data somewhere.
  • No immediate read requirement. As part of Lambda Architecture, the availability of reads written to AllData is not required to be immediate. Instead, it is available for batch processing later. This allows one to use something simpler and non-persistent for real-time processing because AllData serves as the eventual backup.
  • No data loss. Because AllData needs to store everything, it needs to handle replication in case of node failures.
  • No single point of failure. To support always available writes as well as no data loss, a master-less (peer-to-peer) design is required.
  • Efficient batch processing. AllData is designed to support efficient sequential reads that are characteristic of Lambda Architecture batch processing (although this is delegated to particular storage module implementations).

Available Modules

Clients

Servers (for Clients)

Peer Clients

Peer Servers (for Peer Clients)

Storage

Internal

Something went wrong with that request. Please try again.