DataStore changelog 2: introduce StoreEvents

[teh-cmc]

Introduces StoreEvent, an event that describes the atomic unit of change in the Rerun DataStore: a row has been added to or removed from the store.

StoreEvents are fired on both the insertion and garbage collection paths, enabling listeners to build arbitrary, always up-to-date views & trigger systems.

/// The atomic unit of change in the Rerun [`DataStore`].
///
/// A [`StoreEvent`] describes the changes caused by the addition or deletion of a
/// [`re_log_types::DataRow`] in the store.
///
/// Methods that mutate the [`DataStore`], such as [`DataStore::insert_row`] and [`DataStore::gc`],
/// return [`StoreEvent`]s that describe the changes.
///
/// Refer to field-level documentation for more details and check out [`StoreDiff`] for a precise
/// definition of what an event involves.
#[derive(Debug, Clone, PartialEq)]
pub struct StoreEvent {
    /// Which [`DataStore`] sent this event?
    pub store_id: StoreId,

    /// What was the store's generation when it sent that event?
    pub store_generation: StoreGeneration,

    /// Monotonically increasing ID of the event.
    ///
    /// This is on a per-store basis.
    ///
    /// When handling a [`StoreEvent`], if this is the first time you process this [`StoreId`] and
    /// the associated `event_id` is not `1`, it means you registered late and missed some updates.
    pub event_id: u64,

    /// What actually changed?
    ///
    /// Refer to [`StoreDiff`] for more information.
    pub diff: StoreDiff,
}

/// Describes an atomic change in the Rerun [`DataStore`]: a row has been added or deleted.
///
/// From a query model standpoint, the [`DataStore`] _always_ operates one row at a time:
/// - The contents of a row (i.e. its columns) are immutable past insertion, by virtue of
///   [`RowId`]s being unique and non-reusable.
/// - Similarly, garbage collection always removes _all the data_ associated with a row in one go:
///   there cannot be orphaned columns. When a row is gone, all data associated with it is gone too.
///
/// Refer to field-level documentation for more information.
#[derive(Debug, Clone, PartialEq)]
pub struct StoreDiff {
    /// Addition or deletion?
    ///
    /// The store's internals are opaque and don't necessarily reflect the query model (e.g. there
    /// might be data in the store that cannot by reached by any query).
    ///
    /// A [`StoreDiff`] answers a logical question: "does there exist a query path which can return
    /// data from that row?".
    pub kind: StoreDiffKind,

    /// What's the row's [`RowId`]?
    ///
    /// [`RowId`]s are guaranteed to be unique within a single [`DataStore`].
    ///
    /// Put another way, the same [`RowId`] can only appear twice in a [`StoreDiff`] event:
    /// one addition and (optionally) one deletion (in that order!).
    pub row_id: RowId,

    /// The [`TimePoint`] associated with that row.
    ///
    /// Since insertions and deletions both work on a row-level basis, this is guaranteed to be the
    /// same value for both the insertion and deletion events (if any).
    pub timepoint: TimePoint,

    /// The [`EntityPath`] associated with that row.
    ///
    /// Since insertions and deletions both work on a row-level basis, this is guaranteed to be the
    /// same value for both the insertion and deletion events (if any).
    pub entity_path: EntityPath,

    /// All the [`DataCell`]s associated with that row.
    ///
    /// Since insertions and deletions both work on a row-level basis, this is guaranteed to be the
    /// same set of values for both the insertion and deletion events (if any).
    pub cells: IntMap<ComponentName, DataCell>,
}

---

DataStore changelog PR series:

• DataStore changelog 1: let DataStores know about their StoreId #4202
• DataStore changelog 2: introduce StoreEvents #4203
• DataStore changelog 3: introduce StoreViews #4205
• DataStore changelog 4: add standalone "Custom StoreView" example #4206
• DataStore changelog 5: event-driven time histograms #4208
• DataStore changelog 6: event-driven EntityTree #4209

Checklist

• I have read and agree to Contributor Guide and the Code of Conduct
• I've included a screenshot or gif (if applicable)
• I have tested demo.rerun.io (if applicable)
• The PR title and labels are set such as to maximize their usefulness for the next release's CHANGELOG

• PR Build Summary
• Docs preview
• Examples preview
• Recent benchmark results
• Wasm size tracking

[Wumpf]

nicely documented <3
a bit of a pitty how a few things in store_gc increased in code length, but not seeing any obvious shortucts

[Wumpf]

👍

[Wumpf]

makes sense, but what has the reachability of rows to do with whether this is an addition or deletion?

[teh-cmc]

That's the point I'm trying to make: an event of kind deletion only tells you that, from this point on, no query can return data from that row.

But that doesn't mean that the data is actually gone, i.e. don't make assumptions of e.g. the size in bytes of the store based on these events. They are in "query-model space" and are not an accurate representation of what happens in storage space.

[Wumpf]

love how you clarify all the logical invariants here, will make it much easier for new comers to implement StoreViews!

[Wumpf]

at the very least in this pr this doesn't seem to happen. Why is it needed now?
(also, makes me sad!)

[teh-cmc]

Probably needed at some point, dont remember... in any case it can never hurt and will probably save someone some pain in the future so I'll leave it there 🤷

[Wumpf]

shouldn't the delta be always negative? Or asked differently, isn't the right thing to do to first check whether the event is a deletion event? Seems strange not to do that

[teh-cmc]

Yes, this looks weird because this is in a temporary state. In future PRs this whole Deleted thing actually becomes CompactedStoreEvents, and works for both additions and deletions.