Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fixedmap optimizations #71

Merged
merged 9 commits into from
May 7, 2023
Merged

Fixedmap optimizations #71

merged 9 commits into from
May 7, 2023

Conversation

Scooletz
Copy link
Contributor

@Scooletz Scooletz commented May 6, 2023

This PR optimizes FixedMap component internal encoding of data, so that more entries can be packed into a single map. This PR lowers the disk space by total 10%.

Changes introduced:

  1. The length of size reduced to one byte. The size of length reduced from 2 bytes to 1 byte as no entry will ever be more than 255 bytes long. This gives savings ~5%.
  2. The prefix extraction and encoding. This removes 1 or 2 bytes per entry as well, but required crazy unsafe ref magic over enumerator.

Before

Using in-memory DB for greater speed.
Initializing db of size 18GB
Starting benchmark with commit level FlushDataOnly
Preparing random accounts addresses...
Accounts prepared

(P) - 90th percentile of the value

At Block        | Avg. speed      | Space used      | New pages(P)    | Pages reused(P) | Total pages(P)
           5000 |  892.0 blocks/s |          2.35GB |            1606 |             188 |            1733
          10000 |  713.3 blocks/s |          3.58GB |            1917 |             171 |            1962
          15000 |  684.2 blocks/s |          4.06GB |            2042 |             157 |            2063
          20000 |  661.6 blocks/s |          4.31GB |            2111 |             143 |            2121
          25000 |  639.4 blocks/s |          4.60GB |            2147 |             129 |            2163
          30000 |  620.2 blocks/s |          5.42GB |            2181 |             116 |            2219
          35000 |  564.8 blocks/s |          7.38GB |            2205 |             127 |            2293
          40000 |  523.8 blocks/s |         10.37GB |            2241 |             158 |            2385
          45000 |  499.8 blocks/s |         13.79GB |            2299 |             177 |            2475
          49999 |  489.7 blocks/s |         17.13GB |            2365 |             184 |            2545

Writing state of 1000 accounts per block, each with 1 storage, through 50000 blocks, generated 50000000 accounts, used 17.13GB

Reading and asserting values...
Reading state of all of 50000000 accounts from the last block took 00:00:35.0334893
90th percentiles:
   - new pages allocated per block: 0
   - pages reused allocated per block: 1239
   - total pages written per block: 1244

After one-byte length

Using in-memory DB for greater speed.
Initializing db of size 18GB
Starting benchmark with commit level FlushDataOnly
Preparing random accounts addresses...
Accounts prepared

(P) - 90th percentile of the value

At Block        | Avg. speed      | Space used      | New pages(P)    | Pages reused(P) | Total pages(P)
           5000 |  873.9 blocks/s |          2.34GB |            1605 |             187 |            1730
          10000 |  705.7 blocks/s |          3.58GB |            1915 |             170 |            1960
          15000 |  679.5 blocks/s |          4.05GB |            2041 |             156 |            2063
          20000 |  656.4 blocks/s |          4.29GB |            2107 |             142 |            2119
          25000 |  629.4 blocks/s |          4.54GB |            2143 |             129 |            2155
          30000 |  613.4 blocks/s |          5.19GB |            2177 |             116 |            2207
          35000 |  541.7 blocks/s |          6.88GB |            2199 |             118 |            2273
          40000 |  487.9 blocks/s |          9.70GB |            2231 |             151 |            2365
          45000 |  486.9 blocks/s |         13.11GB |            2287 |             174 |            2457
          49999 |  496.1 blocks/s |         16.56GB |            2353 |             184 |            2535

Writing state of 1000 accounts per block, each with 1 storage, through 50000 blocks, generated 50000000 accounts, used 16.56GB

Reading and asserting values...
Reading state of all of 50000000 accounts from the last block took 00:00:34.3674436
90th percentiles:
   - new pages allocated per block: 0
   - pages reused allocated per block: 1239
   - total pages written per block: 1244

After prefix extraction

Using in-memory DB for greater speed.
Initializing db of size 18GB
Starting benchmark with commit level FlushDataOnly
Preparing random accounts addresses...
Accounts prepared

(P) - 90th percentile of the value

At Block        | Avg. speed      | Space used      | New pages(P)    | Pages reused(P) | Total pages(P)
           5000 |  771.9 blocks/s |          2.30GB |            1593 |             185 |            1720
          10000 |  672.3 blocks/s |          3.55GB |            1909 |             169 |            1954
          15000 |  634.6 blocks/s |          4.03GB |            2036 |             154 |            2057
          20000 |  581.8 blocks/s |          4.28GB |            2103 |             141 |            2115
          25000 |  610.8 blocks/s |          4.51GB |            2143 |             128 |            2155
          30000 |  595.9 blocks/s |          5.01GB |            2175 |             115 |            2199
          35000 |  535.0 blocks/s |          6.42GB |            2195 |             110 |            2257
          40000 |  503.5 blocks/s |          8.98GB |            2223 |             140 |            2341
          45000 |  471.7 blocks/s |         12.18GB |            2273 |             165 |            2433
          49999 |  464.2 blocks/s |         15.48GB |            2337 |             175 |            2509

Writing state of 1000 accounts per block, each with 1 storage, through 50000 blocks, generated 50000000 accounts, used 15.48GB

Reading and asserting values...
Reading state of all of 50000000 accounts from the last block took 00:00:33.6083939
90th percentiles:
   - new pages allocated per block: 0
   - pages reused allocated per block: 1239
   - total pages written per block: 1244

@github-actions
Copy link

github-actions bot commented May 7, 2023

Code Coverage

Package Line Rate Branch Rate Health
Paprika 89% 77%
Summary 89% (1151 / 1299) 77% (275 / 357)

Minimum allowed line rate is 75%

@Scooletz Scooletz marked this pull request as ready for review May 7, 2023 06:03
@Scooletz Scooletz merged commit 3ed172c into main May 7, 2023
2 checks passed
@Scooletz Scooletz deleted the fixedmap-optimization branch May 7, 2023 08:03
@Scooletz Scooletz added the 💾 disk size A change makes the disk size smaller label May 7, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
💾 disk size A change makes the disk size smaller
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

1 participant