`DataStore` changelog 5: event-driven time histograms #4208

teh-cmc · 2023-11-13T08:27:50Z

This is mostly preliminary work for #4209, which makes this PR a bit weird. Basically just trying to offload complexity from #4209.

TimesPerTimeline as well as TimeHistogramPerTimeline are now living on their own and are maintained as StoreViews, i.e. they react to changes to the DataStore rather than constructing alternate truths.

This is the first step towards turning the EntityTree giga-structure into an event-driven view in the next PR.

DataStore changelog PR series:

Checklist

I have read and agree to Contributor Guide and the Code of Conduct
I've included a screenshot or gif (if applicable)
I have tested demo.rerun.io (if applicable)
The PR title and labels are set such as to maximize their usefulness for the next release's CHANGELOG

jleibs

Looks good. I'm still a bit confused on the value of a StoreView that doesn't implement on_events but other than that the code makes sense.

jleibs · 2023-11-14T16:55:43Z

crates/re_data_store/src/time_histogram_per_timeline.rs

+    #[allow(clippy::unimplemented)]
+    fn on_events(&mut self, _events: &[StoreEvent]) {
+        unimplemented!(
+            r"TimeHistogramPerTimeline view is maintained as a sub-view of `EntityTree`",
+        );
+    }


What's the value of it even being a StoreView in this case?

I went back and forth on this.

If I don't implement it, then people will rightfully ask "Wait, I thought this thing was a view of the datastore? But it doesn't implement StoreView?"... but then if I do implement it then they rightfully ask "Why implement StoreView then?" 😭

I'll add a comment.

…4202) The upcoming `StoreView` works in global scope: by registering a view you subscribe to changes to _all_ `DataStore`s, including those that are yet to be created. This is very powerful as it allows views & triggers implementers to build cross-recording indices as well as be notified as soon as new recordings come in and go out. But it means that `StoreEvent`s must indicate which `DataStore` they originate from, which isn't possible today since the stores themselves don't know who they are to begin with. This trivial PR plumbs the `StoreId` all the way through so `DataStore`s know about their own ID. Also made `StoreGeneration` account for the GC counter while I was at it. --- Requires: - #4215 `DataStore` changelog PR series: - #4202 - #4203 - #4205 - #4206 - #4208 - #4209

Introduces `StoreEvent`, an event that describes the atomic unit of change in the Rerun `DataStore`: a row has been added to or removed from the store. `StoreEvent`s are fired on both the insertion and garbage collection paths, enabling listeners to build arbitrary, always up-to-date views & trigger systems. ```rust /// The atomic unit of change in the Rerun [`DataStore`]. /// /// A [`StoreEvent`] describes the changes caused by the addition or deletion of a /// [`re_log_types::DataRow`] in the store. /// /// Methods that mutate the [`DataStore`], such as [`DataStore::insert_row`] and [`DataStore::gc`], /// return [`StoreEvent`]s that describe the changes. /// /// Refer to field-level documentation for more details and check out [`StoreDiff`] for a precise /// definition of what an event involves. #[derive(Debug, Clone, PartialEq)] pub struct StoreEvent { /// Which [`DataStore`] sent this event? pub store_id: StoreId, /// What was the store's generation when it sent that event? pub store_generation: StoreGeneration, /// Monotonically increasing ID of the event. /// /// This is on a per-store basis. /// /// When handling a [`StoreEvent`], if this is the first time you process this [`StoreId`] and /// the associated `event_id` is not `1`, it means you registered late and missed some updates. pub event_id: u64, /// What actually changed? /// /// Refer to [`StoreDiff`] for more information. pub diff: StoreDiff, } /// Describes an atomic change in the Rerun [`DataStore`]: a row has been added or deleted. /// /// From a query model standpoint, the [`DataStore`] _always_ operates one row at a time: /// - The contents of a row (i.e. its columns) are immutable past insertion, by virtue of /// [`RowId`]s being unique and non-reusable. /// - Similarly, garbage collection always removes _all the data_ associated with a row in one go: /// there cannot be orphaned columns. When a row is gone, all data associated with it is gone too. /// /// Refer to field-level documentation for more information. #[derive(Debug, Clone, PartialEq)] pub struct StoreDiff { /// Addition or deletion? /// /// The store's internals are opaque and don't necessarily reflect the query model (e.g. there /// might be data in the store that cannot by reached by any query). /// /// A [`StoreDiff`] answers a logical question: "does there exist a query path which can return /// data from that row?". pub kind: StoreDiffKind, /// What's the row's [`RowId`]? /// /// [`RowId`]s are guaranteed to be unique within a single [`DataStore`]. /// /// Put another way, the same [`RowId`] can only appear twice in a [`StoreDiff`] event: /// one addition and (optionally) one deletion (in that order!). pub row_id: RowId, /// The [`TimePoint`] associated with that row. /// /// Since insertions and deletions both work on a row-level basis, this is guaranteed to be the /// same value for both the insertion and deletion events (if any). pub timepoint: TimePoint, /// The [`EntityPath`] associated with that row. /// /// Since insertions and deletions both work on a row-level basis, this is guaranteed to be the /// same value for both the insertion and deletion events (if any). pub entity_path: EntityPath, /// All the [`DataCell`]s associated with that row. /// /// Since insertions and deletions both work on a row-level basis, this is guaranteed to be the /// same set of values for both the insertion and deletion events (if any). pub cells: IntMap<ComponentName, DataCell>, } ``` --- `DataStore` changelog PR series: - #4202 - #4203 - #4205 - #4206 - #4208 - #4209

Introducing the `StoreView` trait and registration system, allowing anybody to subscribe to `DataStore` changes, even from external code. `StoreView`s global scope: by registering a view you subscribe to changes to _all_ `DataStore`s, including those that are yet to be created. This is very powerful as it allows views & triggers implementers to build cross-recording indices as well as be notified as soon as new recordings come in and go out. ```rust /// A [`StoreView`] subscribes to atomic changes in one or more [`DataStore`]s through [`StoreEvent`]s. /// /// [`StoreView`]s can be used to build both secondary indices and trigger systems. pub trait StoreView: std::any::Any + Send + Sync { /// Arbitrary name for the view. /// /// Does not need to be unique. fn name(&self) -> String; /// Workaround for downcasting support, simply return `self`: /// ```ignore /// fn as_any(&self) -> &dyn std::any::Any { /// self /// } /// ``` fn as_any(&self) -> &dyn std::any::Any; /// Workaround for downcasting support, simply return `self`: /// ```ignore /// fn as_any_mut(&mut self) -> &mut dyn std::any::Any { /// self /// } /// ``` fn as_any_mut(&mut self) -> &mut dyn std::any::Any; /// The core of this trait: get notified of changes happening in one or more [`DataStore`]s. /// /// This will be called automatically by the [`DataStore`] itself if the view has been /// registered: [`DataStore::register_view`]. /// Or you might want to feed it [`StoreEvent`]s manually, depending on your use case. /// /// ## Example /// /// ```ignore /// fn on_events(&mut self, events: &[StoreEvent]) { /// use re_arrow_store::StoreDiffKind; /// for event in events { /// match event.kind { /// StoreDiffKind::Addition => println!("Row added: {}", event.row_id), /// StoreDiffKind::Deletion => println!("Row removed: {}", event.row_id), /// } /// } /// } /// ``` fn on_events(&mut self, events: &[StoreEvent]); } ``` --- `DataStore` changelog PR series: - #4202 - #4203 - #4205 - #4206 - #4208 - #4209

…4206) Standalone example of how to implement and register custom `StoreView`s, even from external code. <picture> <img src="https://static.rerun.io/custom_store_view/f7258673486f91d944180bd4a83307bce09b741e/full.png" alt=""> <source media="(max-width: 480px)" srcset="https://static.rerun.io/custom_store_view/f7258673486f91d944180bd4a83307bce09b741e/480w.png"> <source media="(max-width: 768px)" srcset="https://static.rerun.io/custom_store_view/f7258673486f91d944180bd4a83307bce09b741e/768w.png"> <source media="(max-width: 1024px)" srcset="https://static.rerun.io/custom_store_view/f7258673486f91d944180bd4a83307bce09b741e/1024w.png"> <source media="(max-width: 1200px)" srcset="https://static.rerun.io/custom_store_view/f7258673486f91d944180bd4a83307bce09b741e/1200w.png"> </picture> --- `DataStore` changelog PR series: - #4202 - #4203 - #4205 - #4206 - #4208 - #4209

Turns the `EntityTree` giga-datastructure into a `StoreView`, meaning it now reacts to `StoreEvent`s rather than creating alternate truths. This introduces the notion of cascading side-effects, and more specifically `ClearCascade`s. When the `EntityTree` reacts to changes in the store, this might cause cascading effects (e.g. pending clears), that in turn need to write back to the store, which in turn sends more events to react to! The cycle is guaranteed finite because "clears don't get cleared"! Cascading side-effects have an interesting requirement: they need to log their cascaded data using a `RowId` _similar_ to the one used in the original event that caused the cascade (so they get GC'd at roughly the same pace). "Similar" in this cases means that their `TUID` shares the same timestamp and that the new `RowId` is strictly greater than the old one. `PathOp` has finally been annihilated. According to our new "Clears" & "Time Histograms" test suites, this behaves exactly like the `main` branch. --- `DataStore` changelog PR series: - #4202 - #4203 - #4205 - #4206 - #4208 - #4209

teh-cmc changed the title ~~DataStore changelog 5: rework time-related views~~ DataStore changelog 5: event-driven time-related views Nov 13, 2023

teh-cmc force-pushed the cmc/store_changelog_5_times_per_timeline branch from 8a98ae5 to d231e2a Compare November 13, 2023 08:51

teh-cmc added ⛃ re_datastore affects the datastore itself do-not-merge Do not merge this PR exclude from changelog PRs with this won't show up in CHANGELOG.md 🔩 data model labels Nov 13, 2023

teh-cmc mentioned this pull request Nov 13, 2023

DataStore changelog 6: event-driven EntityTree #4209

Merged

4 tasks

teh-cmc force-pushed the cmc/store_changelog_4_custom_storeview branch 2 times, most recently from d171205 to 0994a17 Compare November 13, 2023 17:08

teh-cmc force-pushed the cmc/store_changelog_5_times_per_timeline branch from d231e2a to 6dcd79e Compare November 13, 2023 17:13

teh-cmc force-pushed the cmc/store_changelog_4_custom_storeview branch from 0994a17 to 0fc5593 Compare November 13, 2023 18:08

teh-cmc force-pushed the cmc/store_changelog_5_times_per_timeline branch from 6dcd79e to 1f3c790 Compare November 13, 2023 18:10

teh-cmc changed the title ~~DataStore changelog 5: event-driven time-related views~~ DataStore changelog 5: event-driven time histograms Nov 14, 2023

teh-cmc marked this pull request as ready for review November 14, 2023 08:18

Wumpf self-requested a review November 14, 2023 12:26

jleibs self-requested a review November 14, 2023 13:03

jleibs approved these changes Nov 14, 2023

View reviewed changes

teh-cmc force-pushed the cmc/store_changelog_4_custom_storeview branch from 0fc5593 to 5834e90 Compare November 15, 2023 09:57

teh-cmc removed the request for review from Wumpf November 15, 2023 10:04

Base automatically changed from cmc/store_changelog_4_custom_storeview to main November 15, 2023 10:05

teh-cmc added 2 commits November 15, 2023 11:06

DataStore::gc now return StoreEvents and nothing else

c812f0d

split TimesPerTimeline & TimeHistogramPerTimeline and clean em up

8b26bac

teh-cmc added 3 commits November 15, 2023 11:06

rework StoreDb/EntityDb/EntityTree for new time views

9a1dc22

propagate changes

39f49d7

post-rebase fixes

452875b

teh-cmc force-pushed the cmc/store_changelog_5_times_per_timeline branch from 1f3c790 to 452875b Compare November 15, 2023 10:06

teh-cmc removed the do-not-merge Do not merge this PR label Nov 15, 2023

teh-cmc merged commit edab782 into main Nov 15, 2023
31 of 32 checks passed

teh-cmc deleted the cmc/store_changelog_5_times_per_timeline branch November 15, 2023 10:18

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

`DataStore` changelog 5: event-driven time histograms #4208

`DataStore` changelog 5: event-driven time histograms #4208

teh-cmc commented Nov 13, 2023 •

edited by github-actions bot

Loading

jleibs left a comment

jleibs Nov 14, 2023

teh-cmc Nov 15, 2023

DataStore changelog 5: event-driven time histograms #4208

DataStore changelog 5: event-driven time histograms #4208

Conversation

teh-cmc commented Nov 13, 2023 • edited by github-actions bot Loading

Checklist

jleibs left a comment

Choose a reason for hiding this comment

jleibs Nov 14, 2023

Choose a reason for hiding this comment

teh-cmc Nov 15, 2023

Choose a reason for hiding this comment

`DataStore` changelog 5: event-driven time histograms #4208

`DataStore` changelog 5: event-driven time histograms #4208

teh-cmc commented Nov 13, 2023 •

edited by github-actions bot

Loading