Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Formalize behaviour of our many time histograms in a test suite #4215

Merged
merged 7 commits into from
Nov 15, 2023

Conversation

teh-cmc
Copy link
Member

@teh-cmc teh-cmc commented Nov 13, 2023

We maintain three different kinds of time histograms as part of our EntityDb:

  • times_per_timeline: instantiated once per EntityDb, this keeps track of the number of distinct times per timeline that have been logged to, across all entities and components.
  • prefix_times: instantiated once per EntityTree (and thus once per entity path), this keeps track of the number of raw components that have been logged per timestamp, across this entity and its recursive children.
  • components: instantiated once per EntityTree per ComponentName, this keeps track of the number of raw components that have been logged per timestamp per component, just for this entity.

For each of those we have to ask ourselves:

  • Are they global scope? per entity? per component?
  • Are they flat? recursive?
  • Do they count the number of log events? of rows? the number of distinct components? the number of raw components?!
  • Do they ignore some components? Do they ignore auto-generated data?
  • How do they behave in case of Clear events?
  • Are they affected by GC or not? If so, in what way?
  • Etc

Answers to these questions are now formalized in a test suite, A) allowing us to easily iterate on them in the future and B) allowing me to match the current behavior in new event-driven EntityTree that's about to ship.

This also highlighted a bug in the current implementation, which is fixed in the upcoming one.


Checklist

  • I have read and agree to Contributor Guide and the Code of Conduct
  • I've included a screenshot or gif (if applicable)
  • I have tested demo.rerun.io (if applicable)
  • The PR title and labels are set such as to maximize their usefulness for the next release's CHANGELOG

@teh-cmc teh-cmc added 🔨 testing testing and benchmarks ⛃ re_datastore affects the datastore itself 📺 re_viewer affects re_viewer itself exclude from changelog PRs with this won't show up in CHANGELOG.md labels Nov 13, 2023
@teh-cmc teh-cmc marked this pull request as ready for review November 14, 2023 07:45
@Wumpf Wumpf self-requested a review November 14, 2023 09:26
Copy link
Member

@Wumpf Wumpf left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

very clean, nice!

crates/re_data_store/tests/time_histograms.rs Outdated Show resolved Hide resolved
crates/re_data_store/tests/time_histograms.rs Outdated Show resolved Hide resolved
@teh-cmc teh-cmc merged commit 7b53ba8 into main Nov 15, 2023
19 of 25 checks passed
@teh-cmc teh-cmc deleted the cmc/time_counts_test branch November 15, 2023 09:12
teh-cmc added a commit that referenced this pull request Nov 15, 2023
…4202)

The upcoming `StoreView` works in global scope: by registering a view
you subscribe to changes to _all_ `DataStore`s, including those that are
yet to be created.
This is very powerful as it allows views & triggers implementers to
build cross-recording indices as well as be notified as soon as new
recordings come in and go out.

But it means that `StoreEvent`s must indicate which `DataStore` they
originate from, which isn't possible today since the stores themselves
don't know who they are to begin with.
This trivial PR plumbs the `StoreId` all the way through so `DataStore`s
know about their own ID.

Also made `StoreGeneration` account for the GC counter while I was at
it.

---

Requires:
- #4215 

`DataStore` changelog PR series:
- #4202
- #4203
- #4205
- #4206
- #4208
- #4209
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
exclude from changelog PRs with this won't show up in CHANGELOG.md ⛃ re_datastore affects the datastore itself 📺 re_viewer affects re_viewer itself 🔨 testing testing and benchmarks
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants