Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Primary cache: don't denormalize defaulted values #4779

Closed
teh-cmc opened this issue Jan 10, 2024 · 0 comments · Fixed by #4891
Closed

Primary cache: don't denormalize defaulted values #4779

teh-cmc opened this issue Jan 10, 2024 · 0 comments · Fixed by #4891
Assignees
@teh-cmc
Labels
📉 performance Optimization, memory use, etc 🔍 re_query affects re_query itself

Comments

@teh-cmc
Copy link
Member

teh-cmc commented Jan 10, 2024 •
edited

The ArchetypeView correctly turns missing optional components into streams of default values when they get joined, but the cache then proceeds to materialize and cache them as such, which wastes a lot of space.
This deserves to be special cased asap.

Let's not do anything before range support is in though; it's very easy to draw ourselves into a corner here.
@teh-cmc teh-cmc added 🔍 re_query affects re_query itself 📉 performance Optimization, memory use, etc labels Jan 10, 2024
@teh-cmc teh-cmc mentioned this issue Jan 10, 2024
Merged
4 tasks
teh-cmc added a commit that referenced this issue Jan 15, 2024
@teh-cmc
Primary caching 11: cache stats and integration with memory panel (#4773
1f19805 
)

The primary cache now tracks memory statistics and display them in the
memory panel.

This immediately highlights a very stupid thing that the cache does:
missing optional components that have been turned into streams of
default values by the `ArchetypeView` are materialized as such
:man_facepalming:
- #4779


https://github.com/rerun-io/rerun/assets/2910679/876b264a-3f77-4d91-934e-aa8897bb32fe



- Fixes #4730 


---

Part of the primary caching series of PR (index search, joins,
deserialization):
- #4592
- #4593
- #4659
- #4680 
- #4681
- #4698
- #4711
- #4712
- #4721 
- #4726 
- #4773
- #4784
- #4785
- #4793
- #4800
@teh-cmc teh-cmc self-assigned this Jan 23, 2024
teh-cmc added a commit that referenced this issue Jan 23, 2024
@teh-cmc
don't denormalize defaulted values (fix #4779)
d82279b
teh-cmc added a commit that referenced this issue Jan 23, 2024
@teh-cmc
don't denormalize defaulted values (fix #4779)
5bf921d
@teh-cmc teh-cmc mentioned this issue Jan 23, 2024
Merged
4 tasks
@teh-cmc teh-cmc linked a pull request Jan 23, 2024 that will close this issue
Primary cache: don't denormalize defaulted component values #4891
Merged
4 tasks
teh-cmc added a commit that referenced this issue Jan 23, 2024
@teh-cmc
don't denormalize defaulted values (fix #4779)
bf89c30
teh-cmc added a commit that referenced this issue Jan 25, 2024
@teh-cmc
Primary cache: don't denormalize defaulted component values (#4891)
ce3ab5f 
The cache will now keep track of missing optional components, and store
an empty slice instead of a bunch of `None` values.
When queried, an empty shows up as a `None` option to the end-user, who
can act appropriately.

SFM before:

![image](https://github.com/rerun-io/rerun/assets/2910679/34256f8b-3b4b-4d1a-b1ea-5f9e1fd7860b)

SFM after:

![image](https://github.com/rerun-io/rerun/assets/2910679/b676052a-b1fc-4840-bddf-67e5f490add2)


---

- Fixes #4779
- DNR: requires #4856

### What

### Checklist
* [x] I have read and agree to [Contributor
Guide](https://github.com/rerun-io/rerun/blob/main/CONTRIBUTING.md) and
the [Code of
Conduct](https://github.com/rerun-io/rerun/blob/main/CODE_OF_CONDUCT.md)
* [x] I've included a screenshot or gif (if applicable)
* [x] I have tested the web demo (if applicable):
* Using newly built examples:
[app.rerun.io](https://app.rerun.io/pr/4891/index.html)
* Using examples from latest `main` build:
[app.rerun.io](https://app.rerun.io/pr/4891/index.html?manifest_url=https://app.rerun.io/version/main/examples_manifest.json)
* Using full set of examples from `nightly` build:
[app.rerun.io](https://app.rerun.io/pr/4891/index.html?manifest_url=https://app.rerun.io/version/nightly/examples_manifest.json)
* [x] The PR title and labels are set such as to maximize their
usefulness for the next release's CHANGELOG

- [PR Build Summary](https://build.rerun.io/pr/4891)
- [Docs
preview](https://rerun.io/preview/bf89c307dac5dc8fd8016dd985f8af30a6ee73a7/docs)
<!--DOCS-PREVIEW-->
- [Examples
preview](https://rerun.io/preview/bf89c307dac5dc8fd8016dd985f8af30a6ee73a7/examples)
<!--EXAMPLES-PREVIEW-->
- [Recent benchmark results](https://build.rerun.io/graphs/crates.html)
- [Wasm size tracking](https://build.rerun.io/graphs/sizes.html)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@teh-cmc teh-cmc

Labels
📉 performance Optimization, memory use, etc 🔍 re_query affects re_query itself
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Primary cache: don't denormalize defaulted component values rerun-io/rerun
1 participant
@teh-cmc