GC the blueprints before saving while preserving the current state #3148

jleibs · 2023-08-29T20:50:18Z

What

Resolves: #3098
Related to: #1803

Because blueprints used timeless data and timeless data wasn't GC'd, we previously had no great way to clean up blueprints.

This PR paves the way for better overall GC behavior in the future but doesn't change the default behavior yet.

This PR:

Introduces a new GarbageCollectionOptions instead of just providing a target. This allows you to configure whether you want to gc the timeless data, and additionally how many latest_at values you want to preserve.
Introduces a new gc target: Everything.
Calculates a set of protected rows for every component based on the last relevant row across every timeline (including timeless).
Modifies both gc_drop_at_least_num_bytes and the new gc_everything to respect the protected rows during gc.
Modifies the store_hub to gc the blueprint before saving it.

Photogrammetry with --no-frames is another "worst-case" for blueprint because every image is a space-view, so you can easily create a huge blueprint history by repeatedly resetting the blueprint.

Checklist

I have read and agree to Contributor Guide and the Code of Conduct
I've included a screenshot or gif (if applicable)
I have tested demo.rerun.io (if applicable)

…nating the profile

…ecified

emilk

It looks really good, but there was one piece of logic I failed to follow.

There is also, I believe, a chance to reduce code duplication by recognizing that "Everything" can be expressed numerically

emilk · 2023-08-30T06:28:46Z

crates/re_arrow_store/src/store_gc.rs

@@ -13,6 +16,31 @@ pub enum GarbageCollectionTarget {
    ///
    /// The fraction must be a float in the range [0.0 : 1.0].
    DropAtLeastFraction(f64),
+
+    /// GC Everything that isn't protected
+    Everything,


How is this different from DropAtLeastFraction(1.0) ?

Two differences:

DropAtLeastFraction does size book-keeping and potentially decides to stop early. I was running into edge cases where it would conclude it had met the 1.0 threshold, but in fact still had stuff it could GC.

DropAtLeastFraction needs to do an oldest-first incremental traversal. Everything has lots more room for optimizations that require less shuffling and re-sorting.

Special-casing the numerical value of 1.0 felt weirder to me than special-casing on an explicit enum but I can make that switch if you think it's clearer.

No, that makes sense 👍

crates/re_arrow_store/src/store_gc.rs

crates/re_arrow_store/tests/data_store.rs

crates/re_data_store/src/store_db.rs

crates/re_viewer/src/store_hub.rs

…3148) Resolves: #3098 Related to: #1803 Because blueprints used timeless data and timeless data wasn't GC'd, we previously had no great way to clean up blueprints. This PR paves the way for better overall GC behavior in the future but doesn't change the default behavior yet. This PR: - Introduces a new `GarbageCollectionOptions` instead of just providing a target. This allows you to configure whether you want to gc the timeless data, and additionally how many latest_at values you want to preserve. - Introduces a new gc target: Everything. - Calculates a set of protected rows for every component based on the last relevant row across every timeline (including timeless). - Modifies both `gc_drop_at_least_num_bytes` and the new `gc_everything` to respect the protected rows during gc. - Modifies the store_hub to gc the blueprint before saving it. Photogrammetry with `--no-frames` is another "worst-case" for blueprint because every image is a space-view, so you can easily create a huge blueprint history by repeatedly resetting the blueprint. ![image](https://github.com/rerun-io/rerun/assets/3312232/03df3d06-a780-47b3-b0d9-aaf564793230) * [x] I have read and agree to [Contributor Guide](https://github.com/rerun-io/rerun/blob/main/CONTRIBUTING.md) and the [Code of Conduct](https://github.com/rerun-io/rerun/blob/main/CODE_OF_CONDUCT.md) * [x] I've included a screenshot or gif (if applicable) * [x] I have tested [demo.rerun.io](https://demo.rerun.io/pr/3148) (if applicable) - [PR Build Summary](https://build.rerun.io/pr/3148) - [Docs preview](https://rerun.io/preview/60f3747383780c50886ac781bdf81b32fbff76bd/docs) - [Examples preview](https://rerun.io/preview/60f3747383780c50886ac781bdf81b32fbff76bd/examples) - [Recent benchmark results](https://ref.rerun.io/dev/bench/) - [Wasm size tracking](https://ref.rerun.io/dev/sizes/)

jleibs added 8 commits August 29, 2023 21:40

New functionality to protect rows and gc timeless

f544f3f

Add a unit-test for protected garbage-collection

55b4bc7

Get rid of accidental swap_remove

09fb360

GC the blueprint on save

4f7e16f

Get Blueprint to really GC all the timeless data

9b87fe2

Unit-test for flattening of cleared elements

4066307

Profile scope for persisting blueprint

8fafd31

Remove profiling from individual row-drops because profiling was domi…

d7b5a51

…nating the profile

jleibs added 🟦 blueprint The data that defines our UI ⛃ re_datastore affects the datastore itself labels Aug 29, 2023

Doclinks and typos

e86dab2

jleibs marked this pull request as ready for review August 29, 2023 21:00

Include timeless data in GC target/completion stats if timeless is sp…

314bc83

…ecified

jleibs force-pushed the jleibs/flatten_blueprint branch from 6eead83 to 314bc83 Compare August 29, 2023 21:15

jleibs added 4 commits August 30, 2023 02:21

Enable blueprint-gc on web and fix some edge-cases around gc

762c737

Reword TODO

feba935

typos

78a55bd

wasm build

2c4a3ec

jleibs force-pushed the jleibs/flatten_blueprint branch from 7c9ae50 to 2c4a3ec Compare August 30, 2023 00:37

emilk self-requested a review August 30, 2023 06:27

emilk approved these changes Aug 30, 2023

View reviewed changes

jleibs added 7 commits August 30, 2023 16:08

Clarify 'limited' latest-at functionality

aab6369

Cleanup stats calculation

cc9872d

TimeInt constructors

25d9a91

Purge empty tables as a separate pass

c4d01a1

Clarity comment

eb1c80a

Remove gc_everything until we need the more explicit optimizations

1f23ef9

Stop skipping blueprints on other GC pass

60f3747

jleibs added this to the 0.8.2 milestone Aug 30, 2023

jleibs merged commit 620ea8e into main Aug 30, 2023
26 checks passed

jleibs deleted the jleibs/flatten_blueprint branch August 30, 2023 19:03

jleibs mentioned this pull request Aug 31, 2023

Release 0.8.2 #3175

Closed

3 tasks

jleibs mentioned this pull request Sep 19, 2023

Save loop should include a latest-at view from the beginning of the loop #3366

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

GC the blueprints before saving while preserving the current state #3148

GC the blueprints before saving while preserving the current state #3148

jleibs commented Aug 29, 2023 •

edited by github-actions bot

emilk left a comment

emilk Aug 30, 2023

jleibs Aug 30, 2023 •

edited

emilk Aug 30, 2023

GC the blueprints before saving while preserving the current state #3148

GC the blueprints before saving while preserving the current state #3148

Conversation

jleibs commented Aug 29, 2023 • edited by github-actions bot

What

Checklist

emilk left a comment

Choose a reason for hiding this comment

emilk Aug 30, 2023

Choose a reason for hiding this comment

jleibs Aug 30, 2023 • edited

Choose a reason for hiding this comment

emilk Aug 30, 2023

Choose a reason for hiding this comment

jleibs commented Aug 29, 2023 •

edited by github-actions bot

jleibs Aug 30, 2023 •

edited