RFC: Unified changelog stream schema

[sergiimk]

Rendered RFC

Closes #47

[Wizzy-wooz]

hey @sergiimk! I think it's a good feature to have but we need to be very careful as there can be some challenges with implementation:

1. order of events: if events are processed out of order, it can lead to inconsistencies in the data. we need a complex mechanism to avoid this.
2. schema evolution.
3. replaying and rollback.
4. if datasets are large and constantly changed we need storage, indexing, etc.
5. operational Complexity:monitoring and maintaining: handling failures, retries, and ensuring high availability.

But I encourage to try out this implementation.

[sergiimk]

1. Events in ODF datasets are strictly ordered, so an engine simply must consume events in the same order as they appear in the Parquet file to avoid any inconsistencies.

ODF of course cannot guarantee that all events are ordered by event_time (backfills can still happen), so this is why we use stream processing which can provide consistent results even in the presence of out-of-order events.

2. Schema evolution is on our radar (Schema evolution #67) - I think this RFC does not make it any more difficult

3. We can implement rollbacks on several levels:

   • dataset can be reset to a previous block (similar to git reset --hard) which erases part of the history
   • or we could issue retractions for a batch of individual events (this is history-preserving)

Replays seem to me as mostly an ingestion-time concern. If we realize that some derivative dataset is faulty - we can always just start over, because root datasets by default preserve all history.

4. For storage we are planning to introduce periodic compactions (Compactions #45).

Re. indexing - the only index we have in ODF datasets is time (apart from built-in Parquet headers). To speed up other queries I think we will need an ability to maintain state projections (#75).

5. Don't think this RFC affects any of that. Those are probably out-of-scope of the ODF spec as such - we just need to be sure that ODF protocol doesn't jeopardize any of those properties for implementors.