Trie: cache CheckpointTrie in memory #1035

jochem-brouwer · 2021-01-07T19:24:04Z

This PR:

Seperates the DB used by BaseTrie and the extended DB used by CheckpointTrie (and SecureTrie) into two DBs; the base DB and the CheckpointDB.
All checkpoint operations are cached into memory. This makes reverting operations very easy (just throw away the latest cache) and also does not write to disk (so less I/O operations)

trie: seperate base DB and checkpoint DB

codecov · 2021-01-07T19:25:44Z

Codecov Report

Merging #1035 (827a184) into master (83e591a) will decrease coverage by 0.10%.
The diff coverage is n/a.

Flag	Coverage Δ
block	`77.92% <ø> (ø)`
blockchain	`77.92% <ø> (ø)`
client	`88.23% <ø> (-0.17%)`	⬇️
common	`91.87% <ø> (-0.25%)`	⬇️
devp2p	`82.60% <ø> (+0.01%)`	⬆️
ethash	`82.08% <ø> (ø)`
tx	`90.00% <ø> (-0.16%)`	⬇️
vm	`83.09% <ø> (ø)`

Flags with carried forward coverage won't be shown. Click here to find out more.

packages/trie/src/checkpointDb.ts

jochem-brouwer · 2021-01-09T21:16:52Z

packages/trie/src/checkpointDb.ts

+  async put(key: Buffer, val: Buffer): Promise<void> {
+    if (this.isCheckpoint) {
+      // put value in cache
+      this.checkpoints[this.checkpoints.length - 1].keyValueMap.set(key.toString('hex'), val)


There was a weird bug where, if you would set the value to just the raw key.toString(), then if you'd use Buffer.from(key) you'd get a different buffer. So I have settled with hex encoding here, but it should be faster/better if we can just use the raw buffer so we don't have to convert it (maybe has to do with some encoding issue?). Very weird.

I think hex encoding is the right option here.

I found in the Buffer docs that it says:

If encoding is 'utf8' and a byte sequence in the input is not valid UTF-8, then each invalid byte is replaced with the replacement character U+FFFD.

so I wonder if that could be related to the bug you encountered.

Another idea I had is that base64 encoding would actually output a shorter string for the key, but I'm unsure if that would make a significant difference in the performance of the map. How large could these checkpoint maps conceivably get?

These checkpoint maps can get huge, since we checkpoint the trie before the block and then flush it when the block is done. The only thing I'm pretty scary of is when we hit an extremely bad block which hits a lot of trie items and caches these all, this could probably run into some memory issues... 🤔

The problem I have with this hex encoding is that we actually convert these raw bytes to strings. So if you'd use one byte (let's say 0xff) then internally it will be represented as "ff" which is 2 bytes, so this doubles the strain on memory. I'll take a look at this encoding, maybe if I explicitly pass utf8 as encoding it works.

jochem-brouwer · 2021-01-09T21:24:54Z

Hmm, weird that it crashes on a random blockchain test while passing thousands of others 🤔

trie: save key-values read from disk in cache

jochem-brouwer · 2021-01-10T17:18:48Z

Very weird as the Trie changes apparently found some edge case in the blockchain tests which are unrelated to the changes in Trie. Have fixed.

ryanio · 2021-01-11T21:30:08Z

packages/trie/src/checkpointDb.ts

+import { LevelUp } from 'levelup'
+import { DB, BatchDBOp } from './db'
+
+export const ENCODING_OPTS = { keyEncoding: 'binary', valueEncoding: 'binary' }


ENCODING_OPTS can also be imported from ./db

ryanio · 2021-01-11T21:44:52Z

packages/trie/src/secure.ts

+    const db = this.db.copy()
+    const secureTrie = new SecureTrie(db._leveldb, this.root)
+    if (includeCheckpoints && this.isCheckpoint) {
+      secureTrie.db.checkpoints = this.db.checkpoints.slice()


is this slice for a (shallow) copy?

I've never seen slice without any params, in ES6 you could use the spread syntax: [...this.db.checkpoints]

Yep this copies the array. I took this from the existing implementation of CheckpointTrie.copy - but I agree it looks a bit confusing. Will improve.

ryanio · 2021-01-11T21:49:06Z

packages/trie/src/checkpointDb.ts

+      keyValueMap.forEach(function (value, key) {
+        currentKeyValueMap.set(key, value)
+      })


more of a nit than anything else, but this could be written shorthand in ES6:
keyValueMap.forEach((value, key) => currentKeyValueMap.set(key, value))

ryanio · 2021-01-11T21:54:07Z

overall this looks really awesome, nice inline comments that provide explanation.

just left a few small comments in review, otherwise will approve 👍

holgerd77 · 2021-01-13T13:34:57Z

Have updated the branch here.

@jochem-brouwer have you run this PR in the client? How does this behave?

Otherwise once you find some time to address the comments from @ryanio I think we can get this merged relatively soon.

jochem-brouwer · 2021-01-13T18:46:30Z

Hi @holgerd77, thanks for the rebase, I just tested on client and it seems to have improved execution speed.

@ryanio ready for re-review 😄

ryanio · 2021-01-14T01:31:24Z

packages/vm/tests/BlockchainTestsRunner.ts

@@ -73,6 +73,10 @@ export default async function runBlockchainTest(options: any, testData: any, t:
    common,
  })

+  // Need to await the init promise: in some tests, we do not run the iterator (which awaits the initPromise)
+  // If the initPromise does not finish, the `rawHead` of `blockchain.meta()` is still `undefined`.
+  await blockchain.initPromise


Do you think there is any chance users of the blockchain library may run into this?

If so then as you previously suggested we can make an async public factory method to generate the blockchain and make the constructor private. For example libp2p uses .create(opts) as an async factory method (although in their case they don't restrict the use of the constructor, if a PeerId is supplied then it doesn't need to be async and the constructor can be used)

ryanio

lgtm!

ryanio · 2021-01-14T01:32:36Z

oh if you wanted to include an update to the changelog i will re-approve 👍

holgerd77

I will merge here since I would want to rebase further work on top, hope that's ok. Gave this just a brief look, looks great also with this new CheckpointDB subclass, makes a lot of sense.

I also tested in the client with blocks containing transactions. I also have the impression that this has gotten significantly faster again.

If one is looking at the time diffs from the logs on execution there is also some evidence that this is the case, here is a run from master:

And here one using the code base from this PR:

Seems to be more execution logs per second on the second run. Just a very rough indicator though of course.

Any, @jochem-brouwer, nice work! 😄

trie: initial work on checkpoint cache

8e4f060

trie: seperate base DB and checkpoint DB

jochem-brouwer added PR state: WIP type: performance type: enhancement package: trie labels Jan 7, 2021

ryanio reviewed Jan 7, 2021

View reviewed changes

packages/trie/src/checkpointDb.ts Outdated Show resolved Hide resolved

jochem-brouwer added 3 commits January 8, 2021 19:08

trie: partial checkpointing fix

1a26e39

trie: fix checkpointDb key encoding

895f995

trie: fix secureTrie copy

a4e2053

jochem-brouwer commented Jan 9, 2021

View reviewed changes

jochem-brouwer added the type: test all hardforks This special label enables VM state and blockchain tests for all hardforks on the respective PR. label Jan 10, 2021

vm: fix blockchain tests crashing

66aa28a

trie: save key-values read from disk in cache

jochem-brouwer added PR state: needs review and removed PR state: WIP labels Jan 10, 2021

jochem-brouwer marked this pull request as ready for review January 10, 2021 17:24

jochem-brouwer requested review from ryanio and holgerd77 January 11, 2021 18:47

ryanio reviewed Jan 11, 2021

View reviewed changes

Merge branch 'master' into checkpoint-trie-cache

a569e6d

trie: binary encode checkpointTrie keys

827a184

ryanio reviewed Jan 14, 2021

View reviewed changes

ryanio approved these changes Jan 14, 2021

View reviewed changes

holgerd77 approved these changes Jan 14, 2021

View reviewed changes

holgerd77 merged commit 5ac5bc7 into master Jan 14, 2021

holgerd77 deleted the checkpoint-trie-cache branch January 14, 2021 10:21

holgerd77 mentioned this pull request Jan 19, 2021

update checkpointing API ethereumjs/merkle-patricia-tree#58

Closed

holgerd77 mentioned this pull request Nov 2, 2021

Client/VM/MPT: Shanghai dDoS Attacks Resilience #1536

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Trie: cache CheckpointTrie in memory #1035

Trie: cache CheckpointTrie in memory #1035

jochem-brouwer commented Jan 7, 2021

codecov bot commented Jan 7, 2021 •

edited

Loading

jochem-brouwer Jan 9, 2021

ryanio Jan 11, 2021

jochem-brouwer Jan 11, 2021

jochem-brouwer commented Jan 9, 2021

jochem-brouwer commented Jan 10, 2021

ryanio Jan 11, 2021 •

edited

Loading

ryanio Jan 11, 2021

jochem-brouwer Jan 11, 2021

ryanio Jan 11, 2021

ryanio commented Jan 11, 2021

holgerd77 commented Jan 13, 2021

jochem-brouwer commented Jan 13, 2021

ryanio Jan 14, 2021 •

edited

Loading

ryanio left a comment

ryanio commented Jan 14, 2021 •

edited

Loading

holgerd77 left a comment •

edited

Loading

Trie: cache CheckpointTrie in memory #1035

Trie: cache CheckpointTrie in memory #1035

Conversation

jochem-brouwer commented Jan 7, 2021

codecov bot commented Jan 7, 2021 • edited Loading

Codecov Report

jochem-brouwer Jan 9, 2021

Choose a reason for hiding this comment

ryanio Jan 11, 2021

Choose a reason for hiding this comment

jochem-brouwer Jan 11, 2021

Choose a reason for hiding this comment

jochem-brouwer commented Jan 9, 2021

jochem-brouwer commented Jan 10, 2021

ryanio Jan 11, 2021 • edited Loading

Choose a reason for hiding this comment

ryanio Jan 11, 2021

Choose a reason for hiding this comment

jochem-brouwer Jan 11, 2021

Choose a reason for hiding this comment

ryanio Jan 11, 2021

Choose a reason for hiding this comment

ryanio commented Jan 11, 2021

holgerd77 commented Jan 13, 2021

jochem-brouwer commented Jan 13, 2021

ryanio Jan 14, 2021 • edited Loading

Choose a reason for hiding this comment

ryanio left a comment

Choose a reason for hiding this comment

ryanio commented Jan 14, 2021 • edited Loading

holgerd77 left a comment • edited Loading

Choose a reason for hiding this comment

codecov bot commented Jan 7, 2021 •

edited

Loading

ryanio Jan 11, 2021 •

edited

Loading

ryanio Jan 14, 2021 •

edited

Loading

ryanio commented Jan 14, 2021 •

edited

Loading

holgerd77 left a comment •

edited

Loading