Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

New page buffer management #54

Merged
merged 22 commits into from
Apr 28, 2023
Merged

New page buffer management #54

merged 22 commits into from
Apr 28, 2023

Conversation

Scooletz
Copy link
Contributor

@Scooletz Scooletz commented Apr 25, 2023

This PR changes the way data are stored in-page from frame based to a modified slot array. A slot array uses a slice of memory to store two growing sets of data, one from the beginning, one from the end. The one growing from the beginning is responsible for storing metadata entries called Slots. The one growing from the end, the data.

In case of Paprika a modified version is used, called FixedMap, that with some overhead, is self-sufficient in handling keys and no external indexing is required.

The FixedMap is size agnostic so can be used in any type of pages out there.

Paprika.Runner

The run of Paprika.Runner shows the reduction by ~20% of the disk space and according reduction of the pages allocated. This is a good indicator to move forward with #22.

Before (main)

Writing state of 1000 accounts per block through 2000 blocks, generated 2000000 accounts, used 5.67GB
Reading state of all of 2000000 accounts from the last block took 00:00:00.2241420
90th percentiles:
   - new pages allocated per block: 943
   - pages reused allocated per block: 1020
   - total pages written per block: 1963

After (this branch)

Writing state of 1000 accounts per block through 2000 blocks, generated 2000000 accounts, used 4.76GB
Reading state of all of 2000000 accounts from the last block took 00:00:00.3171895
90th percentiles:
   - new pages allocated per block: 868
   - pages reused allocated per block: 1019
   - total pages written per block: 1887

Resolves #23

@Scooletz Scooletz marked this pull request as ready for review April 28, 2023 08:52
@github-actions
Copy link

Code Coverage

Package Line Rate Branch Rate Health
Paprika 89% 77%
Summary 89% (1017 / 1140) 77% (245 / 318)

Minimum allowed line rate is 75%

@Scooletz Scooletz merged commit 8b065ed into main Apr 28, 2023
@Scooletz Scooletz deleted the buffering branch April 28, 2023 08:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Better page packing
1 participant