eth, internal: Implement getModifiedAccountsBy(Hash|Number) using trie diffs #15512

Arachnid · 2017-11-17T19:15:20Z

No description provided.

karalabe

Two questions/suggestions beside the review comments:

If I understand correctly, the differential iterator can only really find differences between two blocks, but if I lets say send some tokens from account A to B and B sends it back to A, then the token contract itself will not be present in the dump due it it being "clean", even though it changed multiple times internally. Not sure how we can highlight this best in a method doc to ensure people are aware and use it correctly. Similarly pondering whether this could be used to abuse some service relying on it (though arguably that's a bad implementation of said service).
This may be tied exactly to the above, but I tried an example diff on Rinkeby, and I may or may not be missing an account. Perhaps could you double check?

debug.getDirtyAccountsByNumber(1269158, 1269159)
["0xe096e9ad65e61a4c096c79bdbda450bf7ca755c4", "0x91ab965f15fd0945ecd57ba09e267e41b550fb7a", "0xfc18cbc391de84dbd87db83b20935d3e89f5dd91", "0x0cb510e2f16c36ce039ee3164330d5f00ecf9eac"]

This essentially "diffs" the changes in block https://rinkeby.etherscan.io/block/1269159. That particular block updates the miner account 0xfc18cbc391de84dbd87db83b20935d3e89f5dd91 and contains two txs:

0xfc18cbc391de84dbd87db83b20935d3e89f5dd91 -> 0x91ab965f15fd0945ecd57ba09e267e41b550fb7a
0x0cb510e2f16c36ce039ee3164330d5f00ecf9eac->
0x95f35f87608c3a871838f11cbefa8bd76fdf5689

However account 0x95f35f87608c3a871838f11cbefa8bd76fdf5689 isn't reported by the diff. Was it really not modified? It would seems so, but could you perhaps double check because the tx was indeed successful and it did do some SSTOREs.

karalabe · 2017-11-19T11:35:32Z

eth/api.go

+}
+
+func (api *PrivateDebugAPI) getDirtyAccounts(startBlock, endBlock *types.Block) ([]common.Address, error) {
+	oldTrie, err := trie.NewSecure(startBlock.Root(), api.eth.chainDb, 0)


startBlock and endBlock might not exist here if the user specifies a number > head, or a non existent hash. That will cause startBlock.Root() and endBlock.Root() to panic.

karalabe · 2017-11-19T11:35:50Z

eth/api.go

@@ -636,3 +636,33 @@ func storageRangeAt(st state.Trie, start []byte, maxResult int) StorageRangeResu
 	}
 	return result
 }
+
+func (api *PrivateDebugAPI) GetDirtyAccountsByNumber(startNum, endNum uint64) ([]common.Address, error) {


Please add a method doc.

karalabe · 2017-11-19T11:35:54Z

eth/api.go

+	return api.getDirtyAccounts(api.eth.blockchain.GetBlockByNumber(startNum), api.eth.blockchain.GetBlockByNumber(endNum))
+}
+
+func (api *PrivateDebugAPI) GetDirtyAccountsByHash(startHash, endHash common.Hash) ([]common.Address, error) {


Please add a method doc.

karalabe · 2017-11-19T11:38:44Z

eth/api.go

+		api.eth.blockchain.GetBlockByHash(endHash))
+}
+
+func (api *PrivateDebugAPI) getDirtyAccounts(startBlock, endBlock *types.Block) ([]common.Address, error) {


I'd suggest adding a check to ensure startBlock is before endBlock to not be surprising that the inverted parametrization also works.

Also perhaps it would be nice to check that the blocks are on the same chain, though that may be expensive for large chains, so I'm uncertain about that.

Also perhaps it would be nice to check that the blocks are on the same chain, though that may be expensive for large chains, so I'm uncertain about that.

Actually, now that you mention it, it can be another interesting usecase for this; investigating if there double-spends or generally diffs between non-canon blocks and canon-blocks. If the functionality is kept as is.

karalabe · 2017-11-19T11:42:10Z

eth/api.go

+	iter := trie.NewIterator(diff)
+	var dirty []common.Address
+	for iter.Next() {
+		dirty = append(dirty, common.BytesToAddress(newTrie.GetKey(iter.Key)))


Perhaps the GetKey part could use some checks if the preimage of something is not available?

karalabe · 2017-11-19T11:52:27Z

Similarly I'd suggest to allow these methods to have a single parameter form too (i.e. make the second parameter optional), in which case internally it would diff the specified block with the one before it. It might be a nice short-hand usecase.

holiman

I'd suggest maybe naming the methods not getDirty.., but instead getChanged.. or getModified....

That makes it more intuitive that if something changes, then changes back, it would not be included. And also, dirty has connotations about caching, which this isn't about. And other connotations.

Arachnid · 2017-11-19T13:47:27Z

PTAL

karalabe · 2017-11-20T11:16:47Z

eth/api.go

+// GetModifiedAccountsByHash returns all accounts that have changed between the
+// two blocks specified. A change is defined as a difference in nonce, balance,
+// code hash, or storage hash.
+func (api *PrivateDebugAPI) GetModifiedAccountsByHash(startHash, endHash common.Hash) ([]common.Address, error) {


Why no single hash version for this?

karalabe · 2017-11-20T11:17:30Z

eth/api.go

+// two blocks specified. A change is defined as a difference in nonce, balance,
+// code hash, or storage hash.
+// With one parameter, returns the list of accounts modified in the specified block.
+func (api *PrivateDebugAPI) GetModifiedAccountsByNumber(startNum, endNum *uint64) ([]common.Address, error) {


IMHO keep startNum as uint64 instead of pointers, otherwise not specifying any params will crash hard.

karalabe · 2017-11-20T11:17:55Z

eth/api.go

+	// With one argument, compare x-1 with x
+	if endNum == nil {
+		return api.getModifiedAccounts(
+			api.eth.blockchain.GetBlockByNumber(*startNum - 1),


If startnum is the genesis, this will get funky :) Perhaps a special handling is needed for that?

> debug.getModifiedAccountsByNumber(0) Error: Both start block and end block are required at web3.js:3143:20 at web3.js:6347:15

karalabe · 2017-11-20T11:19:13Z

eth/api.go

+func (api *PrivateDebugAPI) getModifiedAccounts(startBlock, endBlock *types.Block) ([]common.Address, error) {
+	if startBlock == nil || endBlock == nil {
+		return nil, fmt.Errorf("Both start block and end block are required")
+	}


You could probably make the two invocations above simpler if you allowed endBlock to be nil, and if so, the make

endBlock = startBlock startBlock = startBlock.Parent

holiman · 2017-11-20T11:20:32Z

Overall I still think it looks good, but some minor beauty-spots:

> debug.getModifiedAccountsByNumber(1,2)
Error: missing trie node d67e4d450343046425ae4271474353857ab860dbc0a1dde64b41b5cd3a532bf3 (path )
    at web3.js:3143:20

Correct ^


> debug.getModifiedAccountsByNumber(1,20000000000000)
Error: Both start block and end block are required
    at web3.js:3143:20
    at web3.js:6347:15

> debug.getModifiedAccountsByNumber(100000000)
Error: Both start block and end block are required
    at web3.js:3143:20

Don't know what's wrong here:

> debug.getModifiedAccountsByHash("0x9f71f1a6a49b8375a64e2626d1757381cf60165fb203b16adf1a4ea5c0e0ec7f")
Error: invalid argument 1: json: cannot unmarshal non-string into Go value of type common.Hash
    at web3.js:3143:20
    at web3.js:6347:15
    at web3.js:5081:36
    at <anonymous>:1:1

> debug.getModifiedAccountsByHash("0x9f71f1a6a49b8375a64e2626d1757381cf60165fb203b16adf1a4ea5c0e0ec7f","0x9f71f1a6a49b8375a64e2626d1757381cf60165fb203b16adf1a4ea5c0e0ec7f")
Error: Start block height (4516259) must be less than end block height (4516259)
    at web3.js:3143:20
    at web3.js:6347:15
    at web3.js:5081:36
    at <anonymous>:1:1

Ah, I see, I assumed there'd be a single-value-version of byHash aswell

karalabe

LGTM

eth, internal: Implement using trie diffs

7df396d

Arachnid requested review from fjl, karalabe and holiman November 17, 2017 19:15

karalabe requested changes Nov 19, 2017

View reviewed changes

holiman approved these changes Nov 19, 2017

View reviewed changes

eth, internal: Changes in response to review

f45aabf

holiman changed the title ~~eth, internal: Implement getDirtyAccountsBy(Hash|Number) using trie diffs~~ eth, internal: Implement getModifiedAccountsBy(Hash|Number) using trie diffs Nov 20, 2017

karalabe requested changes Nov 20, 2017

View reviewed changes

Arachnid and others added 2 commits November 20, 2017 13:01

eth: More fixes to getModifiedAccountsBy*

c0108a4

eth: minor polishes on error capitalization

98b8134

karalabe approved these changes Nov 20, 2017

View reviewed changes

karalabe added this to the 1.7.3 milestone Nov 20, 2017

karalabe merged commit 72ed186 into ethereum:master Nov 20, 2017

Arachnid deleted the getdirtyaccounts branch November 20, 2017 15:24

This was referenced Nov 20, 2017

Documentation for debug_getModifiedAccountsBy(Hash|Number) ethereum/wiki#499

Closed

Documentation for debug_getModifiedAccountsBy(Hash|Number) #15530

Closed

5chdn mentioned this pull request Nov 27, 2017

Merge go-ethereum updates into 2.5.3 Musicoin/go-musicoin#69

Merged

AristideFangbe mentioned this pull request Nov 29, 2017

Download page has a breakdown? #3712

Closed

karalabe mentioned this pull request Dec 14, 2017

RFC: API method eth_watchCall ethereum/EIPs#781

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

eth, internal: Implement getModifiedAccountsBy(Hash|Number) using trie diffs #15512

eth, internal: Implement getModifiedAccountsBy(Hash|Number) using trie diffs #15512

Arachnid commented Nov 17, 2017

karalabe left a comment

karalabe Nov 19, 2017

karalabe Nov 19, 2017

karalabe Nov 19, 2017

karalabe Nov 19, 2017

holiman Nov 19, 2017

karalabe Nov 19, 2017

karalabe commented Nov 19, 2017

holiman left a comment

Arachnid commented Nov 19, 2017

karalabe Nov 20, 2017

karalabe Nov 20, 2017

karalabe Nov 20, 2017

holiman Nov 20, 2017

karalabe Nov 20, 2017

holiman commented Nov 20, 2017 •

edited

Loading

karalabe left a comment

eth, internal: Implement getModifiedAccountsBy(Hash|Number) using trie diffs #15512

eth, internal: Implement getModifiedAccountsBy(Hash|Number) using trie diffs #15512

Conversation

Arachnid commented Nov 17, 2017

karalabe left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

karalabe commented Nov 19, 2017

holiman left a comment

Choose a reason for hiding this comment

Arachnid commented Nov 19, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

holiman commented Nov 20, 2017 • edited Loading

karalabe left a comment

Choose a reason for hiding this comment

holiman commented Nov 20, 2017 •

edited

Loading