Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Importer #173

Merged
merged 39 commits into from
Oct 6, 2023
Merged

Importer #173

merged 39 commits into from
Oct 6, 2023

Conversation

Scooletz
Copy link
Contributor

@Scooletz Scooletz commented Oct 5, 2023

Introduces and Importer that does the import from RocksDb of Nethermind into Paprika's custom storage. To make it work it introduces the following changes:

  1. Moves the concept of the Ethereum aware Key away from the PagedDb
  2. Makes the SlottedArray is now based or raw ReadOnlySpan<byte> key
  3. Introduces StoreKey that allows to efficiently encode Key into a Span<byte> wrapping structure
  4. Introduces a mapping of account address to an int as a separate tree in the PagedDb.
    1. This does not decrease the size of the database per se, but as the entries will repeatedly appear for same accounts, should decrease the size of the entry and number of updated pages.
    2. It introduces one additional layer of indirection meaning that the reads and writes will need to read/write through the id tree. It can be amortized later with some caching (this is not done now as it doesn't seem to be needed).
  5. Merkle's Node.Branch that is full will have its value encoded as one byte instead of 3. This is a small improvement but should provide some space in upper layers of the PagedDb tree.
  6. IBatchContext & co. allows now pages to be RegisterForFutureReuse. Whenever a page is no longer needed it can be abandoned with this call without leaking it out.
  7. DataPage has the biggest change introduced with Leaf pages. Whenever the internal map of the page is full but it has no nibble children, it won't have the biggest nibble flushed first. It will use the buckets for addresses to create more pages with maps until they are full. Only once they are full they will be flushed down. This single change reduced the Sepolia size by 40%-50%. Should be observed with Mainnet.

Closes #156

@Scooletz Scooletz linked an issue Oct 5, 2023 that may be closed by this pull request
@Scooletz Scooletz added ethereum An Ethereum specific work item that requires a good understanding of Eth 💾 disk size A change makes the disk size smaller 🌴Merkle Merkle construct labels Oct 5, 2023
@Scooletz Scooletz marked this pull request as ready for review October 5, 2023 14:32
@github-actions
Copy link

github-actions bot commented Oct 5, 2023

Code Coverage

Package Line Rate Branch Rate Health
Paprika 85% 76%
Summary 85% (2491 / 2917) 76% (671 / 880)

Minimum allowed line rate is 75%

@Scooletz Scooletz merged commit b72b1f7 into main Oct 6, 2023
2 checks passed
@Scooletz Scooletz deleted the importer branch October 6, 2023 09:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
💾 disk size A change makes the disk size smaller ethereum An Ethereum specific work item that requires a good understanding of Eth 🌴Merkle Merkle construct
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Nethermind -> Paprika importer
1 participant