A suggestion about the event topic versioning/point in time

[alexyarmoshko]

Hi John,

I am learning both your excellent "eventsourcing" library and the event sourcing in general and just came across a potential implementation issue regarding the "amended events"/"point in time database". Unless I am missing something.

As the event data in the event store (database storage) ideally should be immutable, there is seems to be no graceful way to handle the situation when an event handler (class) that for example reconstruct an entity must be amended (due to the error) or extended (due to the change in the requirements).

It is of cause possible to amend the class itself, but then the ability to look "back in time" will be lost as the state/entity will be reconstructed according to the latest code that is running, not the one that was active at the given time. And the "point in time" aspect of event sourcing is of major benefits provided by this approach.

As "topic" is effectively a fully qualified "class path" binding the data record in the data storage (repository) to the event class, it will probably be useful to introduce some type of "topic cut-off point"/"topic version"/"event revision" that will allow to bind an old (existing) data item to the most recent active class implementation by default, or to the other one in a standard/customizeble way.

This will also allow to deal in a standard way with the classes being moved across the modules (which is a lesser issue IMHO, but still).

Thanks.

Regards.

[johnbywater]

Yes, let's talk about this. Thanks for opening the issue.

[johnbywater]

Hi Alex,

Sorry it's taken me a while to find a good answer to this question. It seems to me, at the moment at least, that one way to resolve this concern is to say the following:

Firstly, it would be an error to have stored events in an event store that have no corresponding domain object class in the code. So if the domain model code is changed, then it should be changed in such a way that events generated and stored by a previous version of the code are still working.

Secondly, it follows that if the domain model code is changed so that there would be stored events that will no longer work, then the stored events should be migrated so that whatever stored events there are do have corresponding domain object classes.

Thirdly, it follows that if we want never to migrate stored events, then we must restrict the changes we make to domain model code to be strictly additive. We can add a new aggregate and have it trigger new events, and expect existing aggregates to work just fine. We can add a new command to an existing aggregate and have it trigger new events, and expect existing events from an existing commands to continue to work. Similarly, we can replace the events of an existing command with new events, and expect existing events will continue to work, so long as the old event class is not removed. Furthermore, we can add new attributes to an existing event, so long as we provide default values to supplement the state of existing stored events. If this results over time in a lot of clutter, then it would be possible to reduce the clutter by migrating stored events and removing old classes from the code.

If you feel that these considerations do not resolve the concern, please reopen this issue. :-)

All good wishes,

John

[alexyarmoshko]

Hi John,

Thanks for the detailed comment.

I am actually looking to implement "strictly additive changes" along with a "insert only data storage" and as such I need to be able to execute potentially different actions for the existing stored event (while doing reconstruction) depending on the desired point in time (system version).

So all events will have a corresponding domains, potentially even more than one. If no point in time has been specified, then everything works as does now - the most recent (default) domain is used for an event. So far so good.

If a point in time is specified, then the "dispatcher" that links domains to stored events will use a domain that was the "most recent (default)" at that time (system version).

Point in time could be the actual point in time or just a version of the system.

This gives the ability to see the whole system state as it was at a specific time (version of the system) and allows for changes/fixes to the domains (code) to be implemented by strict "addition" of a new domain (version of domain).

Ideally I would like for all code to be present in the system and changes to be made by adding new code (classes, modules). So the existing events may trigger new actions added later. This is conceptually similar to Windows "Side-by-Side assemblies" (to certain extent).

In the current "eventsourcing" implementation each event can invoke only one predefined class (class name is actually stored in the event itself) and to change the actions performed for a particular event, this class must be changed. Once this done, old class is no longer available. If the new aggregate (class) is defined, it will not consume "old" events due to the different class path.

There are ways around this, but not exactly elegant or explicitly provided by this library, which is pity for the otherwise an impressive library.

The immediate solution that comes to mind is to implement a configurable "dispatcher factory" that is solely responsible for providing "topic" (class path) for given domain and linking an event to a class (domain) which can be defined/customized (bidirectional link "event" <-> "class"). A default one will just use the existing logic, once extended it can invoke different/descendant classes.

Thanks.

Regards,
Alex.

[johnbywater]

Yes it would be useful to make a TopicStrategy that can be replaced. You could, in the meantime, provide your own implementation of get_topic and resolve_topic (eventsourcing.utils.topic). Just be careful to either patch that module before it is imported by other things in the library. Or if you do it later than that, then make sure also to set your functions in the library modules that use those functions (eventsourcing.domain.model.entity, eventsourcing.domain.model.events, eventsourcing.utils.transcoding, eventsourcing.infrastructure.sequenceditemmapper).

[johnbywater]

@alexyarmoshko I just made some changes (35288f6) which allow new topics to be substituted for old ones. When a topic is resolved, the dict eventsourcing.utils.topic.substitutions is consulted. So if you set a key as old topic with the new topic as the value, then you can move and rename domain classes and still read old events. Let me know if you think that's helpful at all?