Special "Archived" Event

[jeremydmiller]

We need an easy way to get folks to leverage the "event archival" in Marten to maximize the effectiveness of the hot/cold partitioning. Likewise, as a companion to JasperFx/wolverine#1161 over in Wolverine and it's "aggregate handler workflow" integration with Marten, one of the things missing is an easy way to declare that a stream is now archived.

What if we had a Marten -- later JasperFx -- special event type like this:

Archived

And if this is appended to any Marten stream, either as part of the wolverine aggregate handler workflow or just IDocumentSession.Events.Append(), it marks the events in that stream as archived. The hope here is to plug into the partitioning.

What I don't know if this helps is at least one client workflow where they want an archival process that archives all events except for the last one. Might need to work around that.

[elexisvenator]

We already have a restreaming event that is similar to this, declare the old stream as done, point to the new stream. The pattern works and I definitely support the idea of an archived event. Separately we have an explicit "archived" event, the sole purpose of which is to mark the date a stream was archived.

This could also be helpful for handling async projections - ive always been concerned about the chance of an async projection not completing its work on a stream before it is archived. My view for the ideal default behaviour of a projection is that it should process every stream that was not archived when the projection was created, and process through to completion. This marker could be used to make sure projections are not left with partly-processed streams.

One thing to think about is if there is metadata we can put on this event. For example: right now we only archive events as a precursor to deleting them, as a sort of 7-day undo feature. We then have a worker go through daily and delete archived streams whose latest event is older than 7 days. It would be great if we could add matadata to the event to say something like "why it was archived", though it would also be possible to put a marker event before the archive to achieve the same outcome.