Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

re_datastore: properly handle already sorted data during insertion #433

Closed
Tracked by #443
, #1898
...
teh-cmc opened this issue Dec 2, 2022 · 0 comments · Fixed by #3088
Closed
Tracked by #443
, #1898
...

re_datastore: properly handle already sorted data during insertion #433

teh-cmc opened this issue Dec 2, 2022 · 0 comments · Fixed by #3088
Labels
🏹 arrow concerning arrow enhancement New feature or request ⛃ re_datastore affects the datastore itself
Milestone
0.8.2

Comments

@teh-cmc
Copy link
Member

teh-cmc commented Dec 2, 2022

Do not mark an index as dirty (unsorted) when newly inserted data happens to already be
time-ordered.

  • this is the happy path and in fact most common path: data arrives in time order, and so
    inserting it at the end of the bucket doesn't actually result in the bucket becoming
    unsorted
@teh-cmc teh-cmc added enhancement New feature or request 🏹 arrow concerning arrow ⛃ re_datastore affects the datastore itself labels Dec 2, 2022
@teh-cmc teh-cmc mentioned this issue Dec 12, 2022
Closed
11 tasks
@teh-cmc teh-cmc assigned teh-cmc and unassigned teh-cmc Feb 28, 2023
@teh-cmc teh-cmc mentioned this issue Apr 18, 2023
Closed
23 tasks
@teh-cmc teh-cmc mentioned this issue Apr 26, 2023
Closed
@emilk emilk mentioned this issue Aug 23, 2023
Closed
@emilk emilk added this to the 0.8.2 milestone Aug 23, 2023
@emilk emilk mentioned this issue Aug 23, 2023
Merged
3 tasks
emilk added a commit that referenced this issue Aug 23, 2023
@emilk
Fix quadratic slowdown when ingesting data with uniform time (#3088)
4bc05e5 
### What
* Closes #3086
* Closes #433

This should also overall just speed up data insertion for the common
case of already-sorted data

### Checklist
* [x] I have read and agree to [Contributor
Guide](https://github.com/rerun-io/rerun/blob/main/CONTRIBUTING.md) and
the [Code of
Conduct](https://github.com/rerun-io/rerun/blob/main/CODE_OF_CONDUCT.md)
* [x] I've included a screenshot or gif (if applicable)
* [x] I have tested [demo.rerun.io](https://demo.rerun.io/pr/3088) (if
applicable)

- [PR Build Summary](https://build.rerun.io/pr/3088)
- [Docs
preview](https://rerun.io/preview/e5adb1aa580de2274b4eca9f6c5de38ae503b521/docs)
<!--DOCS-PREVIEW-->
- [Examples
preview](https://rerun.io/preview/e5adb1aa580de2274b4eca9f6c5de38ae503b521/examples)
<!--EXAMPLES-PREVIEW--><!--EXAMPLES-PREVIEW-->
- [Recent benchmark results](https://ref.rerun.io/dev/bench/)
- [Wasm size tracking](https://ref.rerun.io/dev/sizes/)
jleibs pushed a commit that referenced this issue Aug 31, 2023
@emilk @jleibs
Fix quadratic slowdown when ingesting data with uniform time (#3088)
62b5061 
### What
* Closes #3086
* Closes #433

This should also overall just speed up data insertion for the common
case of already-sorted data

### Checklist
* [x] I have read and agree to [Contributor
Guide](https://github.com/rerun-io/rerun/blob/main/CONTRIBUTING.md) and
the [Code of
Conduct](https://github.com/rerun-io/rerun/blob/main/CODE_OF_CONDUCT.md)
* [x] I've included a screenshot or gif (if applicable)
* [x] I have tested [demo.rerun.io](https://demo.rerun.io/pr/3088) (if
applicable)

- [PR Build Summary](https://build.rerun.io/pr/3088)
- [Docs
preview](https://rerun.io/preview/e5adb1aa580de2274b4eca9f6c5de38ae503b521/docs)
<!--DOCS-PREVIEW-->
- [Examples
preview](https://rerun.io/preview/e5adb1aa580de2274b4eca9f6c5de38ae503b521/examples)
<!--EXAMPLES-PREVIEW--><!--EXAMPLES-PREVIEW-->
- [Recent benchmark results](https://ref.rerun.io/dev/bench/)
- [Wasm size tracking](https://ref.rerun.io/dev/sizes/)
jleibs pushed a commit that referenced this issue Aug 31, 2023
@emilk @jleibs
Fix quadratic slowdown when ingesting data with uniform time (#3088)
c7ea722 
### What
* Closes #3086
* Closes #433

This should also overall just speed up data insertion for the common
case of already-sorted data

### Checklist
* [x] I have read and agree to [Contributor
Guide](https://github.com/rerun-io/rerun/blob/main/CONTRIBUTING.md) and
the [Code of
Conduct](https://github.com/rerun-io/rerun/blob/main/CODE_OF_CONDUCT.md)
* [x] I've included a screenshot or gif (if applicable)
* [x] I have tested [demo.rerun.io](https://demo.rerun.io/pr/3088) (if
applicable)

- [PR Build Summary](https://build.rerun.io/pr/3088)
- [Docs
preview](https://rerun.io/preview/e5adb1aa580de2274b4eca9f6c5de38ae503b521/docs)
<!--DOCS-PREVIEW-->
- [Examples
preview](https://rerun.io/preview/e5adb1aa580de2274b4eca9f6c5de38ae503b521/examples)
<!--EXAMPLES-PREVIEW--><!--EXAMPLES-PREVIEW-->
- [Recent benchmark results](https://ref.rerun.io/dev/bench/)
- [Wasm size tracking](https://ref.rerun.io/dev/sizes/)
jleibs pushed a commit that referenced this issue Aug 31, 2023
@emilk @jleibs
Fix quadratic slowdown when ingesting data with uniform time (#3088)
1f7537d 
### What
* Closes #3086
* Closes #433

This should also overall just speed up data insertion for the common
case of already-sorted data

### Checklist
* [x] I have read and agree to [Contributor
Guide](https://github.com/rerun-io/rerun/blob/main/CONTRIBUTING.md) and
the [Code of
Conduct](https://github.com/rerun-io/rerun/blob/main/CODE_OF_CONDUCT.md)
* [x] I've included a screenshot or gif (if applicable)
* [x] I have tested [demo.rerun.io](https://demo.rerun.io/pr/3088) (if
applicable)

- [PR Build Summary](https://build.rerun.io/pr/3088)
- [Docs
preview](https://rerun.io/preview/e5adb1aa580de2274b4eca9f6c5de38ae503b521/docs)
<!--DOCS-PREVIEW-->
- [Examples
preview](https://rerun.io/preview/e5adb1aa580de2274b4eca9f6c5de38ae503b521/examples)
<!--EXAMPLES-PREVIEW--><!--EXAMPLES-PREVIEW-->
- [Recent benchmark results](https://ref.rerun.io/dev/bench/)
- [Wasm size tracking](https://ref.rerun.io/dev/sizes/)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
🏹 arrow concerning arrow enhancement New feature or request ⛃ re_datastore affects the datastore itself
Projects
None yet
Milestone
0.8.2
Development

Successfully merging a pull request may close this issue.

Fix quadratic slowdown when ingesting data with uniform time rerun-io/rerun
2 participants
@emilk @teh-cmc