Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fsa/new version management #5440

Merged
merged 74 commits into from
Sep 23, 2022
Merged

Conversation

finnschiermer
Copy link
Contributor

@finnschiermer finnschiermer commented Apr 27, 2022

Reclaim intermediate (frozen) versions.

This PR holds two major upgrades

  • The lock-file gets a new updated format which allows for separate tracking of frozen and live versions. It also drastically simplifies the machinery (now called the VersionList) by using plain locking instead of a fancy lock-free data structure.
  • The "backdating" algorithm is now part of every commit. It looks for file space which is not reachable through any live or frozen version and recycles it for new allocations, thereby eliminating most of the file growth otherwise caused by version pinning.

Testing is done by two random scenarios, one at Core Storage level, one at ObjectStore level. The tests generate scenarios with random use of both live version pinning, frozen version pinning and dropping of versions. The test at ObjectStore level uses notifications. In debug mode every use of the backdating algorithm includes verification that a) recycled blocks are not referenced and b) kept blocks are indeed referenced. This validation is expensive, so only enabled in debug mode.

The complexity is O(free-list-size * log2(max(versions,free-list-size)) so it may add CPU overhead to commit. However, recycling blocks earlier often leads to less fragmentation and shorter free-lists. The observed net result for the tests is a clear performance win.

@finnschiermer finnschiermer self-assigned this Apr 27, 2022
@cla-bot cla-bot bot added the cla: yes label Apr 27, 2022
@finnschiermer finnschiermer marked this pull request as ready for review June 13, 2022 12:14
src/realm/transaction.cpp Outdated Show resolved Hide resolved
src/realm/group_writer.hpp Outdated Show resolved Hide resolved
src/realm/group_writer.cpp Outdated Show resolved Hide resolved
src/realm/group_writer.cpp Outdated Show resolved Hide resolved
src/realm/group_writer.cpp Outdated Show resolved Hide resolved
src/realm/db.cpp Show resolved Hide resolved
src/realm/db.cpp Outdated Show resolved Hide resolved
src/realm/db.cpp Outdated Show resolved Hide resolved
src/realm/db.cpp Outdated Show resolved Hide resolved
src/realm/db.cpp Show resolved Hide resolved
finnschiermer and others added 6 commits September 15, 2022 17:24
Co-authored-by: Mathias Stearn <redbeard0531@gmail.com>
Co-authored-by: Mathias Stearn <redbeard0531@gmail.com>
Co-authored-by: Mathias Stearn <redbeard0531@gmail.com>
Copy link
Contributor

@RedBeard0531 RedBeard0531 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this will be the last roundtrip

src/realm/db.cpp Outdated Show resolved Hide resolved
src/realm/db.cpp Show resolved Hide resolved
src/realm/db.cpp Outdated Show resolved Hide resolved
src/realm/db.cpp Outdated Show resolved Hide resolved
src/realm/db.cpp Outdated Show resolved Hide resolved
src/realm/db.cpp Show resolved Hide resolved
Copy link
Contributor

@RedBeard0531 RedBeard0531 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🚢🇮🇹!

(after fixing a minor typo)

src/realm/db.cpp Outdated Show resolved Hide resolved
Co-authored-by: Mathias Stearn <redbeard0531@gmail.com>
@ironage
Copy link
Contributor

ironage commented Sep 20, 2022

In case you missed it, there are 20 new warnings reported on CI "Code will never be executed" around calls to ALLOC_DBG_COUT in debug mode when REALM_ALLOC_DEBUG is defined as 0.

@finnschiermer finnschiermer changed the base branch from master to next-major September 21, 2022 11:07
src/realm/db.cpp Outdated Show resolved Hide resolved
@finnschiermer finnschiermer merged commit 633a9aa into next-major Sep 23, 2022
@finnschiermer finnschiermer deleted the fsa/new-version-management branch September 23, 2022 08:32
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Mar 21, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants