Expiration/cleanup of events

[acetinick]

Hi,

Firstly great project, I am really excited to use this library and we are currently testing it out within our SaaS platform.

I see that overtime we would be adding thousands/millions of rows added to the event_store_events and event_store_events_in_stream tables, on events that we publish.

I don't see a need for us to persistantly store these events forever, so is is common to build a rake task or something to clean these events up events older than a period of time? eg. 14 days.

I usually familiar with Kinesis which typcially stores events for up to 7 days.

[mpraglowski]

Hi

Firstly great project, I am really excited to use this library and we are currently testing it out within our SaaS platform.

💚 Thanks! We will be happy to learn more about your experiences and use cases.

I see that overtime we would be adding thousands/millions of rows added to the event_store_events and event_store_events_in_stream tables, on events that we publish.

What are your expectations? How many events will you store per time unit? (day/week/month)

I don't see a need for us to persistantly store these events forever, so is is common to build a rake task or something to clean these events up events older than a period of time? eg. 14 days.

RES does not have that API now. There is an option to delete_stream however this will not delete any events from events table, just the assignments of an events to removed stream.

The scavenging API is not something we have in our roadmap but will be happy to see your solution for that, PR maybe?
Things to consider:

• some events might be linked to several streams, removing only events could potentially break ordering of the stream
• projections that rely on stream or several streams might give different results after removal of events

[pmackay]

Hi, could I ask if the approach to event storage has evolved since this conversation was started? I am also trialling RES and wondering if storage could grow to be a problem.

Related to that, is there any way to avoid an event being stored, but just be published to listeners?

[mpraglowski]

Hi, could I ask if the approach to event storage has evolved since this conversation was started? I am also trialling RES and wondering if storage could grow to be a problem.

Hi - we are using RES in several project and we have not observed issues with number of domain events stored. AFAIK we are not the ones who have highest number of domain events stored without issues.

Related to that, is there any way to avoid an event being stored, but just be published to listeners?

Sure - you could implement your own event repository with no-op behavior - similar pattern to NullSerializer - https://github.com/RailsEventStore/rails_event_store/blob/master/ruby_event_store/lib/ruby_event_store/null.rb.
This could event be something like ComposedRepository when based on event type you will decide which repository (NullRepository or default one) could be used - it all depends on your needs.
However if you do not need to store events then maybe some other solution to pub/sub might be more suitable for you?

We have started our publicly-available Rails Event Store Discord — https://discord.gg/qjPr9ZBpX6
where You could discuss RES related topics with other RES users and with us.