Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Update merge-tree snapshot to include segments between MSN and snapshot sequence number #84

Closed
kurtb opened this issue Feb 5, 2019 · 4 comments
Closed

Update merge-tree snapshot to include segments between MSN and snapshot sequence number #84

kurtb opened this issue Feb 5, 2019 · 4 comments
Assignees
@anthony-murphy
Labels
area: dds Issues related to distributed data structures feature-request New feature or requests
Milestone
August 2020

Comments

@kurtb
Copy link
Contributor

kurtb commented Feb 5, 2019

The current merge tree snapshot includes all zamboni'd segments. And then plays tardis'd messages between the min sequence number and the snapshot sequence number. This requires us to create, load and replay these tardis'd operations. Load can be simplified and storage decreased by including segments that haven't been zamboni'd in the snapshot.
@anthony-murphy anthony-murphy transferred this issue from another repository Sep 19, 2019
@DLehenbauer
Copy link
Contributor

@anthony-murphy - Is this as simple as changing extractSegment to include all segments and deleting the code that maintains the list of messages since MSN?

@DLehenbauer
Copy link
Contributor

...I ask because I probably need to duplicate or refactor the code in question in order to move SparseMatrix to using 2D ops instead of piggy backing on MergeTree's ops... which I think is probably necessary for efficient dependency tracking / invalidation.

@DLehenbauer DLehenbauer mentioned this issue Nov 4, 2019
Merged
DLehenbauer added a commit that referenced this issue Nov 8, 2019
@DLehenbauer
MergeTree: Include ACKed segments in summary (Fix #84)
8b599a6
DLehenbauer added a commit that referenced this issue Nov 22, 2019
@DLehenbauer
MergeTree: Include ACKed segments in snapshot (issue #84) (#590)
e62907f 
* MergeTree: Remove unused 'EmptyChunk'
* MergeTree: Add new Snapshot format which includes ACKed segments
* MergeTree: Add unit tests for new snapshot format
* MergeTree: Update comment on `walkAllSegments`
* MergeTree: CR feedback
@curtisman curtisman added the feature-request New feature or requests label Jan 21, 2020
@curtisman curtisman added the area: dds Issues related to distributed data structures label Jan 31, 2020
@curtisman curtisman added this to the Build 2020 milestone Feb 3, 2020
@curtisman curtisman modified the milestones: Build 2020, May 2020 May 8, 2020
@anthony-murphy anthony-murphy modified the milestones: May 2020, June 2020 May 28, 2020
@anthony-murphy
Copy link
Contributor

The work for this is in. Need to come up with a flighting plan to test this before rolling out broadly

@anthony-murphy anthony-murphy modified the milestones: June 2020, July 2020 Jun 30, 2020
@anthony-murphy anthony-murphy modified the milestones: July 2020, August 2020 Jul 9, 2020
@danielroney
Copy link
Contributor

Closing per conversation with Tony

@arinwt arinwt mentioned this issue Jan 6, 2021
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@anthony-murphy anthony-murphy

Labels
area: dds Issues related to distributed data structures feature-request New feature or requests
Projects
None yet
Milestone
August 2020
Development

No branches or pull requests
5 participants
@kurtb @anthony-murphy @DLehenbauer @curtisman @danielroney