Join GitHub today
GitHub is home to over 40 million developers working together to host and review code, manage projects, and build software together.Sign up
WIP: I just got back from a vacation and got to thinking about the event store after getting enough rest for once
Big Existing Issues
Other ideas for improvements
@jeremydmiller Thank's for this write-up! Here are my thoughts and other ideas that I was thinking on:
Internals of it's implementation are still enigmatic for me, so to give more detailed answers. So at first I'd propose to provide good documentation and samples for that (as we still lack it, afaik it's still only your blog post about that).
I like the idea for the prebuilded apps and samples with some integration to other tools. Regarding which cloud? Dunno, I still believe that Azure is poor mans AWS, but on the other hand, .NET community is in love with the Microsoft tools like MSSQL and others, so probably it would be better to start with Azure. Maybe with some cooperation with Microsoft it would give us some grants or at least marketing?
Imho that's must have. As I gathered recently people's fears about Event Sourcing - performance is one of them. Also when I was explaining Marten to new people there is always a question (how single events table will handle the big load). Although I think that those fears are exaggerated, then I see a point of making our store more performant and also giving people hard numbers that "yes, we can do it, see, there is no point of being afraid".
I fully agree, current ViewProjection mechanism is hard to maitain. I'm currently working on #1302. I already started some small unification of projections mechanism to make it (at least from the abstractions perspective) more generic. I was thinking that maybe that would be good start for discussions around the potential refactoring? I could provide my first PoC and from that having some concrete proposal we could work to make it right?
About Projection snapshotting - it would be nice to give some flexible way of snapshoting. I'm not sure if doing snapshot one per few times would be huge benefit, but if we give eg. possibility to define that she/he would like to have it once per day, or other custom filter expression - then imho that might be huge benefit.
I think that also two other types of projections are low hanging fruits and would be good "marketing" for coexistance with/migration from the ORMs like:
I'm all up for making this pluggable for other solutions (eg. Liquid projections)
I think that it's must have. For sure it should be optional, but for the distributed systems things like
I think that it would be worth to check how NEventStore is handling that - as I know they have quite good implementation of the Metadata.
Other things that I consider
Integration with messaging systems
It's not easy for those systems to always keep the ordering of events, and it's rare for those systems to have "exactly once delivery" semantic. Normally consumers need to handle indempotency by themselves.
Currently Marten doesn't allow to put events out of order (so eg. 2nd, 1st, 3rd). We'd need to change the current versioning mechanism to allow that and projections rebuild.
Imho it shouldn't be super hard to deliver first option to give user possibility to set the version number for imported events.
We discussed some time ago that maybe mechanism simmilar to Async Daemon would be also some potential options for that.
Integration points with other Event Stores / UI
I'd like to create the integration point as I described here: #1194 (comment) and discussed with @gregoryyoung. So start with exposing our event store features as atom feed. Then maybe provide some swagger like simple UI (that might be also used for document part).
Long Version for Events
#1080 - imho this is must have for the version 4.0 if we'd like to make it high scale.
@jeremydmiller what ar your thoughts?
I probably forgot about something, so I might add something later.
More on the async daemon
Convention Based Projections Concept
The main idea here is to allow users more flexibility to do whatever it is they need to do with less code ceremony and easier to author code. Drop mandatory base classes and interfaces (they're still there, just wrapped around it). Marten itself will use some kind of dynamic code generation (ala Jasper or Lamar from Jeremy's prior work) to create an
The following shows some of the possible method signatures and the hopefully minimal set of optional attributes:
Anything that would increase the performance of rebuilding projections in the Async Daemon, would be huge for us. Snapshotting and the performance optimizations for rebuilding that @jeremydmiller was mentioning would be great.
For context, we currently have >3 million events in our event store and are storing ~25k new events per day now. Our rebuilding performance has noticeably gotten worse as the number of events increase.
I am definitely willing to help contribute to any of these event store improvements for v4.
Some observations based on our usage of Marten.
I'm glad to help with some of these improvements. First one will probably be metadata.