Fix garbage collection #1560
Merged
Fix garbage collection #1560
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
teh-cmc
added
🪳 bug
jleibs
approved these changes
Mar 10, 2023
emilk
reviewed
Mar 11, 2023
| @@ -37,11 +37,21 @@ impl Default for EntityDb { | |||
| data_store: re_arrow_store::DataStore::new( | |||
| InstanceKey::name(), | |||
| DataStoreConfig { | |||
| component_bucket_size_bytes: 1024 * 1024, // 1 MiB | |||
| // Garbage collection of the datastore is currently driven by the `MsgId` | |||
| // component column, as a workaround for the `MsgId` mismatch issue. | |||
There was a problem hiding this comment.
What is "the MsgId mismatch issue"? Is there an actual issue describing it?
There was a problem hiding this comment.
Essentially, the store has no easy way of returning a list of MsgIds to the main app when it garbage collects data (because there's no such thing as a message in the store: it gets split into an arbitrary number of index and component buckets during insertion).
We do some major hacks in the current GC implementation to somewhat work around the issue, but this leads to a whole bunch of other problems: one of which is addressed by this PR.
Waiting for us to discuss more during this week's session before opening issues.
1 task
Closed
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Garbage collection is currently broken (since the port to Arrow) due to a configuration that has a hard time coexisting with the workarounds in place for the
MsgIdmismatch problem.This is all very reminiscing of the issues faced in #1535 and #1558 and the reason for that is that this yet another manifestation of the same exact underlying problem: dealing
MsgIdmismatches when going across the viewer<>store boundary.The short-term fix is again a configuration change; the long-term fix will be to eliminate the root problem while we design and implement batch support.
Closes #1539
canny, capped at500MiB:23-03-10_17.32.12.patched.mp4
clockrunning indefinitely as fast as my machine will allow it, capped at500MiB:23-03-10_17.34.53.patched.mp4
Notice that the memory taken by the index doesn't shrink, again this is very much related to #1558 and the mismatch issue.