Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

eth/tracers: improve tracing performance #23016

Merged
merged 4 commits into from Jul 1, 2021
Merged

Conversation

holiman
Copy link
Contributor

@holiman holiman commented Jun 8, 2021

This PR first adds a benchmark of the "opcode tracer", i.e, the barebones tracer which stores data in memory before sending it out over RPC. The benchmark is a very simple loop which just runs through 1M gas.

The second commit adds some optimizations which improves the performance quite a lot:

name                old time/op    new time/op    delta
TransactionTrace-6     100ms ±12%      28ms ±13%  -72.03%  (p=0.000 n=10+9)

name                old alloc/op   new alloc/op   delta
TransactionTrace-6    40.1MB ± 4%     9.0MB ±12%  -77.58%  (p=0.000 n=10+10)

name                old allocs/op  new allocs/op  delta
TransactionTrace-6      571k ± 0%      163k ± 0%  -71.42%  (p=0.000 n=10+8)

@holiman holiman changed the title Bench trace eth/tracers: improve tracing performance Jun 8, 2021
@holiman
Copy link
Contributor Author

holiman commented Jun 8, 2021

Further improvements:

name                old time/op    new time/op    delta
TransactionTrace-6    28.1ms ±13%    22.7ms ± 8%  -19.07%  (p=0.000 n=9+9)

name                old alloc/op   new alloc/op   delta
TransactionTrace-6    8.98MB ±12%    7.14MB ±14%  -20.55%  (p=0.000 n=10+10)

name                old allocs/op  new allocs/op  delta
TransactionTrace-6      163k ± 0%       82k ± 0%  -49.98%  (p=0.002 n=8+10)

Total:

name                old time/op    new time/op    delta
TransactionTrace-6     100ms ±12%      23ms ± 8%  -77.36%  (p=0.000 n=10+9)

name                old alloc/op   new alloc/op   delta
TransactionTrace-6    40.1MB ± 4%     7.1MB ±14%  -82.18%  (p=0.000 n=10+10)

name                old allocs/op  new allocs/op  delta
TransactionTrace-6      571k ± 0%       82k ± 0%  -85.71%  (p=0.000 n=10+10)

@ethereum ethereum deleted a comment Jun 8, 2021
@holiman
Copy link
Contributor Author

holiman commented Jun 11, 2021

I ran this against this transaction,

#!/bin/bash

time ./build/bin/geth attach --exec "debug.traceTransaction('0x87ba373aa69ee675ae764b49c3d179d2367dd0d970c36b9177dc67f00603b4e9', {'disableMemory':true, 'disableStorage':true})" 

It took about 52s, and 27s with stack disabled.
I also took a cpu profile when running it with stack enabled:
profile001

@MariusVanDerWijden
Copy link
Member

So I tried this on a really heavy tx we have on ropsten
time ./build/bin/geth --ropsten attach --exec "debug.traceTransaction('0xcafaf3669df640365da47527902e3e4dff48bd334ab0c5c894dae5ba1f82df27', {'disableMemory':true, 'disableStorage':true, reexec:10000})"

This PR:
(60 seconds of it is state regeneration)

Executed in  281.69 secs    fish           external
   usr time  151.47 secs  323.00 micros  151.47 secs
   sys time  103.44 secs   67.00 micros  103.44 secs

Current Master:
(59 seconds of it is state regeneration)
(I killed the node, call didn't finish)

Executed in  269.17 mins    fish           external
   usr time    6.21 secs  358.00 micros    6.21 secs
   sys time    0.50 secs   77.00 micros    0.50 secs

@holiman
Copy link
Contributor Author

holiman commented Jun 30, 2021

So I tried this on a really heavy tx we have on ropsten

Looks like it worked then? Can the PR haz an approve?

core/vm/logger.go Outdated Show resolved Hide resolved
Copy link
Member

@MariusVanDerWijden MariusVanDerWijden left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@holiman holiman added this to the 1.10.5 milestone Jul 1, 2021
@holiman holiman merged commit f5f906d into ethereum:master Jul 1, 2021
@holiman
Copy link
Contributor Author

holiman commented Jul 1, 2021

0xcafaf3669df640365da47527902e3e4dff48bd334ab0c5c894dae5ba1f82df27

18M gas on that transactions -- yeah might be a bit slow :)

sidhujag pushed a commit to sidhujag/go-ethereum that referenced this pull request Jul 1, 2021
Improves the performance of debug.traceTransaction
yehjxraymond added a commit to yehjxraymond/go-ethereum that referenced this pull request Jul 13, 2021
* eth/tracers: expose contextual infos (block hash, tx hash, tx index)

* eth/tracers: convert int/hash values from context into js object (ethereum#23108)

* Convert int/hash values from context into js object

* Use js fixed buffer

Co-authored-by: William <william.berman@coinbase.com>

* cmd/geth: fix IPC probe in les test (ethereum#23094)

Previously, the test waited a second and then failed if geth had not
started. This caused the test to fail intermittently. This change checks
whether the IPC is open 10 times over a 5 second period and then fails
if geth is still not available.

* eth/gasprice: fix typo in comment (ethereum#22998)

* core/state: remove unused methods ReturnGas, GetStorageProofByHash (ethereum#23092)

Co-authored-by: lidongwei <lidongwei@huobi.com>

* eth/downloader: increase downloader block body allowance (ethereum#23074)

This change increases the cache size from 64 to 256 Mb for block bodies.
Benchmarks have shown this to be one bottleneck when trying to achieve
higher download speeds.

The commit also includes a minor optimization for header inserts in package
core: previously, the presence of headers in the database was checked for
every header before writing it. With the change, if one header fails the
presence check, all subsequent headers are also assumed to be missing.
This is an improvement because in practice, the headers are almost always
missing during sync.

* p2p/enode: fix method doc (ethereum#23115)

This is an obvious spelling error

Co-authored-by: liuyaxiong <liuyaxiong@inspur.com>

* core, eth: fix precompile addresses for tracers (ethereum#23097)

* core,eth/tracers: make isPrecompiled dependent on HF

* eth/tracers: use keys when constructing chain config struct

* eth/tracers: dont initialize activePrecompiles with random value

* eth/gasprice: implement feeHistory API (ethereum#23033)

* eth/gasprice: implement feeHistory API

* eth/gasprice: factored out resolveBlockRange

* eth/gasprice: add sanity check for missing block

* eth/gasprice: fetch actual gas used from receipts

* miner, eth/gasprice: add PendingBlockAndReceipts

* internal/ethapi: use hexutil.Big

* eth/gasprice: return error when requesting beyond head block

* eth/gasprice: fixed tests and return errors correctly

* eth/gasprice: rename receiver name

* eth/gasprice: return directly if blockCount == 0

Co-authored-by: rjl493456442 <garyrong0905@gmail.com>

* accounts/abi/bind: fix gas price suggestion with pre EIP-1559 clients (ethereum#23102)

This fixes transaction sending in the case where an app using go-ethereum v1.10.4
is talking to a pre-EIP-1559 RPC node. In this case, the eth_maxPriorityFeePerGas
endpoint is not available and we can only rely on eth_gasPrice.

* p2p/server: fix method name in comment (ethereum#23123)

* ethclient/gethclient: RPC client wrapper for geth-specific API (ethereum#22977)

This commit adds the package gethclient which is similar to the ethclient
and implements some geth specific functionality.

Co-authored-by: Edgar Aroutiounian <edgar.factorial@gmail.com>
Co-authored-by: Felix Lange <fjl@twurst.com>

* cmd/geth: dont fail on deprecated toml config fields (ethereum#23118)

* internal/ethapi: fix panic in access list creation (ethereum#23133)

Fixes test failure in the last commit.

* all: removed blockhash from statedb (ethereum#23126)

This PR removes the blockhash from the statedb

* eth/fetcher, trie: unit test reliability fixes (ethereum#23020)

Some tests take quite some time during exit, which I think causes
some appveyor fails like this:

    https://ci.appveyor.com/project/ethereum/go-ethereum/builds/39511210/job/xhom84eg2e4uulq3

One of the things that seem to take time during exit is waiting
(up to 100ms) for the syncbloom to close. This PR changes it to use
a channel, instead of looping with a 100ms wait.

This also includes some unrelated changes improving the reliability of
eth/fetcher tests, which fail a lot because they are time-dependent.

* crypto: gofuzz build directives (ethereum#23137)

* eth/tracers: improve tracing performance (ethereum#23016)

Improves the performance of debug.traceTransaction

* go.mod: update UPNP dependency (ethereum#23116)

* les: avoid shutdown hang (ethereum#23139)

* consensus/clique: avoid a copy in clique (ethereum#23149)

* consensus/clique:optimize to avoid a copy in clique

* consensus/clique: test for sealhash

Co-authored-by: Martin Holst Swende <martin@swende.se>

* cmd/evm, eth/ethconfig: regenerate struct codecs (ethereum#23140)

* ethstats: fix full node interface post 1559

* dockerfile: get rid of make and env, see if that fixes builds

* cmd/puppeth: fix dashboard crash caused by updated base image

* core: fix bad parent hash when jumping to genesis in setHead (ethereum#23162)

* p2p: fix array out of bounds issue (ethereum#23165)

* fuzzing: fix typo in fuzzer definitions (ethereum#23169)

* eth, miner: add RPC method to modify miner gaslimit (pre london: ceiling) (ethereum#23134)

* core/types: sanity check the basefee length inside a header (ethereum#23171)

* all: remove noop vm config flags (ethereum#23111)

* all: rm external interpreter and ewasm config

* core/vm: rm Interpreter interface

* cmd/geth: deprecate interpreter config fields

* cmd/devp2p: fixes for eth and discv4 tests (ethereum#23155)

This PR fixes a false positive PONG 'to' endpoint mismatch seen in hive tests:

    got {IP:172.17.0.7 UDP:44025 TCP:44025}, want {IP:172.17.0.7 UDP:44025 TCP:0}

Co-authored-by: Felix Lange <fjl@twurst.com>

* core/types: remove LogForStorage type (ethereum#23173)

The encoding of Log and LogForStorage is exactly the same
now. After tracking it down it seems like ethereum#17106 changed the
storage schema of logs to be the same as the consensus
encoding.

Support for the legacy format was dropped in ethereum#22852 and if
I'm not wrong there's no reason anymore to have these two
equivalent types.

Since the RLP encoding simply contains the first three fields
of Log, we can also avoid creating a temporary struct for
encoding/decoding, and use the rlp:"-" tag in Log instead.

Note: this is an API change in core/types. We decided it's OK
to make this change because LogForStorage is an implementation
detail of go-ethereum and the type has zero uses outside of
package core/types.

Co-authored-by: Felix Lange <fjl@twurst.com>

* core/types: go generate (ethereum#23177)

* core, params: define london block at 12965000 (ethereum#23176)

* core, params: define london block at 12965000

* core/forkid: fix test

* eth/gasprice, internal/ethapi, miner: minor feehistory fixes

* params: update CHTs for the 1.10.5 release

* core/state: avoid unnecessary alloc in trie prefetcher (ethereum#23198)

* internal/ethapi: fix transaction APIs (ethereum#23179)

* internal/ethapi: fix transaction APIs

* internal/ethapi: fix typo

* internal/ethapi: address comments

* internal/ethapi: address comment from Peter

* internal: get pending and queued transaction by address (ethereum#22992)

* core, eth, internal, les, light: get pending and queued transaction by address

* core: tiny nitpick fixes

* light: tiny nitpick

Co-authored-by: mark <mark@amis.com>
Co-authored-by: Péter Szilágyi <peterke@gmail.com>

* consensus/clique: implement getSigner API method (ethereum#22987)

* clique: implement getSignerForBlock

* consensus/clique: use blockNrOrHash in getSignerForBlock

* consensus/clique: implement getSigner

* consensus/clique: fixed rlp decoding

* consensus/clique: use Author instead of getSigner

* consensus/clique: nit nit nit

* consensus/clique: nit nit nit

* feat: merged changes from head

Co-authored-by: Péter Szilágyi <peterke@gmail.com>
Co-authored-by: williamberman <wlbberman@gmail.com>
Co-authored-by: William <william.berman@coinbase.com>
Co-authored-by: piersy <pierspowlesland@gmail.com>
Co-authored-by: haryu703 <34744512+haryu703@users.noreply.github.com>
Co-authored-by: Li Dongwei <lidw1988@126.com>
Co-authored-by: lidongwei <lidongwei@huobi.com>
Co-authored-by: Martin Holst Swende <martin@swende.se>
Co-authored-by: ForLina <471133417@qq.com>
Co-authored-by: liuyaxiong <liuyaxiong@inspur.com>
Co-authored-by: Sina Mahmoodi <1591639+s1na@users.noreply.github.com>
Co-authored-by: Felföldi Zsolt <zsfelfoldi@gmail.com>
Co-authored-by: rjl493456442 <garyrong0905@gmail.com>
Co-authored-by: Pierre R <p.rousset@gmail.com>
Co-authored-by: Ahyun <urbanart2251@gmail.com>
Co-authored-by: Marius van der Wijden <m.vanderwijden@live.de>
Co-authored-by: Edgar Aroutiounian <edgar.factorial@gmail.com>
Co-authored-by: Felix Lange <fjl@twurst.com>
Co-authored-by: ucwong <ucwong@126.com>
Co-authored-by: Evolution404 <35091674+Evolution404@users.noreply.github.com>
Co-authored-by: Mark <markya0616@gmail.com>
Co-authored-by: mark <mark@amis.com>
Co-authored-by: Raymond Yeh <ray@geek.sg>
maoueh pushed a commit to streamingfast/go-ethereum that referenced this pull request Aug 20, 2021
* eth/tracers: improve tracing performance (ethereum#23016)

Improves the performance of debug.traceTransaction

* Update the import order

Co-authored-by: Martin Holst Swende <martin@swende.se>
Co-authored-by: John.H <john.h@binance.com>
reds pushed a commit to reds/go-ethereum that referenced this pull request Aug 28, 2021
Improves the performance of debug.traceTransaction
i-norden pushed a commit to cerc-io/go-ethereum that referenced this pull request Sep 10, 2021
Improves the performance of debug.traceTransaction
atif-konasl pushed a commit to frozeman/pandora-execution-engine that referenced this pull request Oct 15, 2021
Improves the performance of debug.traceTransaction
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants