Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Correctly track lowest closed instant/snapshot seqno #61

Closed
marvin-j97 opened this issue Jul 1, 2024 · 1 comment
Closed

Correctly track lowest closed instant/snapshot seqno #61

marvin-j97 opened this issue Jul 1, 2024 · 1 comment
Assignees
Labels
enhancement New feature or request test

Comments

@marvin-j97
Copy link
Collaborator

marvin-j97 commented Jul 1, 2024

Currently, MVCC versions of items may be GC'd if there is no open snapshot and the level is deep enough.
However, if there's always some snapshot hanging around, it may completely prevent GC of old items.
It would be better to keep track of open snapshots, and store the lowest seqno that can be definitely be dropped without affecting existing snapshots, and using that number inside compaction filtering.

Example:

LowestSeqno = 0 // means nothing can be cleaned up

OpenSnapshot(1)
OpenSnapshot(2)

CloseSnapshot(2)
LowestSeqno = 0 // because Snapshot=1 is still hanging around, the counter was not changed

CloseSnapshot(1)
LowestSeqno = 2 // means we can remove any value that has seqno < 2

Note: There can be multiple snapshots with the same seqno

Possible API

SnapshotTracker::default() => starts at 0
SnapshotTracker::register(seqno)
SnapshotTracker::release(seqno)
SnapshotTracker::get_lowest() -> Option => gets the highest seqno we can safely free

@marvin-j97 marvin-j97 added enhancement New feature or request help wanted Extra attention is needed test labels Jul 1, 2024
@marvin-j97 marvin-j97 changed the title Correctly track lowest closed snapshot seqno Correctly track lowest closed instant/snapshot seqno Jul 3, 2024
@marvin-j97 marvin-j97 transferred this issue from fjall-rs/lsm-tree Jul 3, 2024
marvin-j97 added a commit that referenced this issue Aug 10, 2024
@marvin-j97 marvin-j97 self-assigned this Aug 10, 2024
@marvin-j97 marvin-j97 removed the help wanted Extra attention is needed label Aug 10, 2024
@marvin-j97
Copy link
Collaborator Author

Needs a bit more testing

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request test
Projects
None yet
Development

No branches or pull requests

1 participant