-
Notifications
You must be signed in to change notification settings - Fork 12
Home
AtomEventStore is a .NET library that uses the Atom Syndication Format to provide a server-less implementation of an Event Store.
It was created to solve a particular set of problems:
- It should support Test-Driven Development (particularly Outside-In TDD, or GOOS-style TDD) without adding much overhead.
- It should work well with Platform-as-a-Service solutions such as Microsoft Azure.
- It should be scalable.
- It should be lock-free.
- It should be reusable.
- It should be reasonably efficient.
As always in software development, there are trade-offs involved, so while AtomEventStore is designed mainly with the above goals in mind, it's not optimized for low latency, efficient storage, or fastest possible transmission times. It's also not designed to handle concurrent writes; the applications using AtomEventStore are expected to address this problem through applications of the Single Writer pattern.
AtomEventStore comes with a built-in in-memory storage option optimal for unit testing. Since AtomEventStore is based on storing data as Atom Feed documents, the in-memory store simply keeps the documents in memory as strings.
The AtomEventsInMemory storage class is even thread-safe, which also enables you to use it in Integration Tests, or other types of (near) full stack tests.
AtomEventStore works well with Platform-as-a-Service (PaaS) offerings such as Microsoft Azure, because it doesn't rely on a server component. AtomEventStore is a client library that utilizes a well-known storage format (Atom Syndication Format). Since the client library writes directly to files or the like, it can also be used with e.g. Azure BLOB storage. This is PaaS friendly because you don't have to install and maintain a virtual machine to host a database server.
The optional AtomEventsOnAzure storage class supports Microsoft Azure BLOB storage.
AtomEventStore supports as many independent event streams as the underlying storage mechanism can handle.
- When using the file-based storage, the support is ultimately limited by how many files and folders can be stored under a single root folder.
- When using Microsoft Azure BLOB storage, the support is ultimately limited by the number and size of BLOBs that a sigle storage container can hold.
- When using the memory-based storage, the support is ultimately limitied by the available memory of the client machine.
When writing events to storage, each entry is appended to a document with a configurable page size. When the page is full, a new page is created, which guarantees that a write operation is never more costly than when the page size is reached.
During reads, events are supplied via an Iterator over the pages in the underlying Atom Feed. The Iterator itself never reads more than two Atom Feed pages at a time, and each page is limited in size by the configurable page size. This means that the Iterator can enumerate even very long event streams. If a client of such a lazily enumerated sequence can adequately deal with long event streams (e.g. by aggregating the events instead of keeping each event in memory), it can scale to very long event streams.
Note: That AtomEventStore is designed to be scalable doesn't necessarily imply that it's fast. While it's designed to be reasonably efficient, where design-tradeoff were necessary, scalability have been prioritized over speed.
Since AtomEventStore uses various document-oriented storage implementations, it can't provide any transaction guarantees. Some storage implementations (e.g. Azure BLOB storage) supports Optimistic Concurrency, but other storage mechanisms may not offer that.
Instead, AtomEventStore uses an ordered approach to writes. Onless you've configured the page size to be 1 or 2, most writes will only involve an update of a single document, so these write operations either succeed or fail as single units of operation.
However, when a page has been filled up, a new page is created first, after which a link is added from the old page to the new page. It's possible that the first write operation succeeds, but the next operation fails. In this case, the entire write operation fails with an exception, letting the client know that the write operation failed. While this leaves behind an orphaned page, it doesn't jeopardize the consistency of the system if the client retries the write operation.
Since Optimistic Concurrency can't be guaranteed be all storage implementations, the clients have the responsibility to prevent concurrency conflicts. The easiest way to do that is to use a single writer per event stream.
Event streams are append-only, so multiple concurrent readers can read the same event stream without concurrency conflicts.
Although AtomEventStore comes with a few usable storage implementations out of the box, it's designed around the SOLID principles in order to be reusable.
- If you need to use another storage mechanism, you can write a custom storage implementation.
- If you need to use a custom XML serializer, you can write one.
Additionally, AtomEventStore's most frequently accessed APIs are based on the already well-defined interfaces IEnumerable<T> and IObserver<T>.
Despite its priority is first and foremost to be scalable, AtomEventStore still strives towards being reasonably efficient where possible.
Since there's no server component, the client must perform all filtering and projection of an event stream. That involves reading the events from storage into memory, which, depending on the storage mechanism in use, may take some time. However, if the client knows that it isn't going to need to read the entire event stream, the choice of reading forwards or backwards through the event stream may impact performance. Therefore, AtomEventStore includes both a forward-reading Iterator and a backwards-reading Iterator.
Additionally, both Iterators utilize a read-ahead strategy where they pre-fetch the next Atom Feed page to read once enumeration starts in earnest. Still, in order to prevent too many unnecessary pre-fetch operations, pre-fetching first kicks in when the Iterator moves past the first element in a sequence. Thus, peeking at the head of the sequence doesn't trigger a pre-fetch.
The idea that an Event Store can be modelled as a Linked List was originally put to my attention by Yves Reynhout in his article Your EventStream is a linked list.