(VDB-1249) backfill storage #71

rmulhol · 2020-03-31T14:25:46Z

TODO:

inline getStorageValue command to backfillStorage
intelligent insert backfill diff function (no dupes)

rmulhol · 2020-03-31T21:32:20Z

Dockerfile updates will be made in transformers repo since config is required

elizabethengelman · 2020-04-01T14:13:24Z

cmd/backfillStorage.go

+addresses and storage slots must be back-filled).
+Requires CLI flags are backfill-storage-start-block (-s) and backfill-storage-end-block (-e) to define range of blocks
+that need to be back-filled.
+Optional CLI flag is backfill-storage-contract-address (-a) to specify a single contract address that needs to be


If this flag is passed in, is the config file still required?

Yeah bc still need the transformer in the exporter for the keys loader

elizabethengelman · 2020-04-01T14:15:01Z

cmd/backfillStorage.go

+that need to be back-filled.
+Optional CLI flag is backfill-storage-contract-address (-a) to specify a single contract address that needs to be
+back-filled (if not necessary for all transformers).
+Before running this command, verify that you have run headerSync and execute for the desired blocks. Headers are


I wonder if we should add a note here (and/or in the README) about if this can be run while execute and headerSync are running.

elizabethengelman · 2020-04-01T15:57:07Z

db/migrations/20200331113100_create_create_back_filled_diff_function.sql

+        RETURN 0;
+    END IF;
+
+    IF last_storage_value is null and create_back_filled_diff.storage_value = empty_storage_value THEN


You mentioned in standup that the backfill process will need to be run in order (by block number) since otherwise this piece would potentially miss setting a value to zero when it should be, right? Is there anyway we can enforce this, to make sure that we're backfilling in order?

Oh, I'm seeing now that HeaderRepository. GetHeadersInRange sorts the headers it's returning, nice!

Yeah, still a good thought - will think about ways we can verify that all headers have been synced/checked. Seems like we could verify that we have all headers in the range and that they have a check count > 0, at least

elizabethengelman · 2020-04-01T15:58:39Z

libraries/shared/storage/diff_repository.go

@@ -55,6 +55,14 @@ func (repository diffRepository) CreateStorageDiff(rawDiff types.RawDiff) (int64
 	return storageDiffID, err
 }

+func (repository diffRepository) CreateBackFilledStorageValue(rawDiff types.RawDiff) (int64, error) {


Nice, i like this being it's own method, rather than reusing CreateStorageDiff. 💯

I wonder if this needs to return a the diff id, since we're not using it in the StorageValueLoader. But it may be nice to keep a similar patter as CreateStorageDiff. 🤔

Dig it, removed 👍

rmulhol · 2020-04-02T02:16:31Z

This is pretty slow running on my local machine (~1:30/block, with an incomplete set of keys). Will continue brainstorming ways to speed it up. Unfortunately only getting relevant keys (e.g. exclude entries in mappings and dynamic arrays where event was fired at a later block) imposes some additional cost since we'd need to refresh keys for every header (as opposed to once for all headers), but maybe that's worth it if we can exclude rpc calls 🤔

OTOH I'm a bit tempted to merge and fire it off for the first chunk of missing diffs to see how performance differs in a different env

rmulhol · 2020-04-02T21:31:40Z

Update: batching improves performance by 80% locally to 18s/block 🎉

elizabethengelman · 2020-04-06T15:19:55Z

pkg/eth/blockchain.go

@@ -104,6 +104,33 @@ func (blockChain *BlockChain) LastBlock() (*big.Int, error) {
 	return block.Number, err
 }

+func (blockChain *BlockChain) BatchGetStorageAt(account common.Address, keys []common.Hash, blockNumber *big.Int) (map[common.Hash][]byte, error) {


- Enable backfilling storage from multiple contracts across a range of blocks

- redundant with backfillStorage (can pass single block for starting and ending block to achieve same results)

- Avoid inserting storage values if they duplicate a previous block or are zero-valued and do not have a previous value (e.g. when looking up storage keys that haven't yet been assigned). - NOTE: excluding zero-valued storage values for storage keys that were not previously associated with a non-zero value assumes that we are backfilling by block number in ascending order. Using this function before all previous values have been populated could cause a failure to persist legitimate assignments to zero.

rmulhol force-pushed the vdb-1249-backfill branch from ed276f7 to b0ad859 Compare March 31, 2020 19:17

rmulhol requested review from elizabethengelman, Gslaughl and yaoandrew March 31, 2020 19:18

rmulhol force-pushed the vdb-1249-backfill branch from b0ad859 to 1a980d3 Compare March 31, 2020 19:20

rmulhol changed the title ~~[WIP] (VDB-1249) backfill storage~~ (VDB-1249) backfill storage Mar 31, 2020

elizabethengelman approved these changes Apr 1, 2020

View reviewed changes

rmulhol force-pushed the vdb-1249-backfill branch 10 times, most recently from 432ae7a to aea3f2e Compare April 2, 2020 21:14

rmulhol force-pushed the vdb-1249-backfill branch from 4ea17cf to d83b4ba Compare April 2, 2020 23:45

elizabethengelman reviewed Apr 6, 2020

View reviewed changes

rmulhol added 4 commits April 6, 2020 18:37

(VDB-1249) Add command for backfilling storage

5e6f7b1

- Enable backfilling storage from multiple contracts across a range of blocks

Make backfillContractDiffs a CLI option for backfillStorage

478c98e

Remove getStorageValue command

abf4bbe

- redundant with backfillStorage (can pass single block for starting and ending block to achieve same results)

Remove obsolete levelDbPath flag

96679df

rmulhol force-pushed the vdb-1249-backfill branch from d83b4ba to 1e9e66f Compare April 6, 2020 23:38

rmulhol added 3 commits April 6, 2020 18:45

Batch getStorageAt calls

74f1b5f

Remove unused code

7cf862c

rmulhol force-pushed the vdb-1249-backfill branch from 1e9e66f to 7cf862c Compare April 6, 2020 23:51

rmulhol merged commit 97afac1 into staging Apr 7, 2020

rmulhol deleted the vdb-1249-backfill branch April 7, 2020 14:28

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

(VDB-1249) backfill storage #71

(VDB-1249) backfill storage #71

rmulhol commented Mar 31, 2020 •

edited

Loading

rmulhol commented Mar 31, 2020

elizabethengelman Apr 1, 2020

rmulhol Apr 2, 2020

elizabethengelman Apr 1, 2020

elizabethengelman Apr 1, 2020

elizabethengelman Apr 1, 2020

rmulhol Apr 2, 2020

elizabethengelman Apr 1, 2020

rmulhol Apr 2, 2020

rmulhol commented Apr 2, 2020 •

edited

Loading

rmulhol commented Apr 2, 2020

elizabethengelman Apr 6, 2020

(VDB-1249) backfill storage #71

(VDB-1249) backfill storage #71

Conversation

rmulhol commented Mar 31, 2020 • edited Loading

rmulhol commented Mar 31, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rmulhol commented Apr 2, 2020 • edited Loading

rmulhol commented Apr 2, 2020

Choose a reason for hiding this comment

rmulhol commented Mar 31, 2020 •

edited

Loading

rmulhol commented Apr 2, 2020 •

edited

Loading