Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

core/state: trie-prefetcher nitpick #23198

Merged
merged 1 commit into from Jul 12, 2021

Conversation

MariusVanDerWijden
Copy link
Member

Might reduce memory consumption a bit.
We don't need to declare a new variable here, casting to string twice might be a bit slower but less memory intensive

OUTINE ======================== github.com/ethereum/go-ethereum/core/state.(*subfetcher).loop in github.com/ethereum/go-ethereum/core/state/trie_prefetcher.go
  823.72MB    63.55GB (flat, cum) 13.25% of Total
         .          .    277:func (sf *subfetcher) loop() {
 [...]
         .   638.60MB    282:    trie, err := sf.db.OpenTrie(sf.root)
         .          .    283:    if err != nil {
         .          .    284:        log.Warn("Trie prefetcher failed opening trie", "root", 
[...]
         .          .    289:    // Trie opened successfully, keep prefetching items
         .          .    290:    for {
      10MB       10MB    291:        select {
         .          .    292:        case <-sf.wake:
  [...]
         .          .    304:                    sf.lock.Lock()
    3.06MB     3.06MB    305:                    sf.tasks = append(sf.tasks, tasks[i:]...)
         .          .    306:                    sf.lock.Unlock()
         .          .    307:                    return
         .          .    308:
         .          .    309:                case ch := <-sf.copy:
         .          .    310:                    // Somebody wants a copy of the current trie, grant them
         .          .    311:                    ch <- sf.db.CopyTrie(sf.trie)
         .          .    312:
         .          .    313:                default:
         .          .    314:                    // No termination request yet, prefetch the next entry
  308.01MB   308.01MB    315:                    taskid := string(task)
         .          .    316:                    if _, ok := sf.seen[taskid]; ok {
         .          .    317:                        sf.dups++
         .          .    318:                    } else {
         .    62.05GB    319:                        sf.trie.TryGet(task)
  502.65MB   502.65MB    320:                        sf.seen[taskid] = struct{}{}

@fjl fjl changed the title state: trie-prefetcher nitpick core/state: trie-prefetcher nitpick Jul 12, 2021
Copy link
Contributor

@fjl fjl left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Doubt this will fix the memory issues, but good change nonetheless.

Copy link
Contributor

@holiman holiman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@holiman holiman added this to the 1.10.5 milestone Jul 12, 2021
@holiman holiman merged commit 8fe47b0 into ethereum:master Jul 12, 2021
yehjxraymond added a commit to yehjxraymond/go-ethereum that referenced this pull request Jul 13, 2021
* eth/tracers: expose contextual infos (block hash, tx hash, tx index)

* eth/tracers: convert int/hash values from context into js object (ethereum#23108)

* Convert int/hash values from context into js object

* Use js fixed buffer

Co-authored-by: William <william.berman@coinbase.com>

* cmd/geth: fix IPC probe in les test (ethereum#23094)

Previously, the test waited a second and then failed if geth had not
started. This caused the test to fail intermittently. This change checks
whether the IPC is open 10 times over a 5 second period and then fails
if geth is still not available.

* eth/gasprice: fix typo in comment (ethereum#22998)

* core/state: remove unused methods ReturnGas, GetStorageProofByHash (ethereum#23092)

Co-authored-by: lidongwei <lidongwei@huobi.com>

* eth/downloader: increase downloader block body allowance (ethereum#23074)

This change increases the cache size from 64 to 256 Mb for block bodies.
Benchmarks have shown this to be one bottleneck when trying to achieve
higher download speeds.

The commit also includes a minor optimization for header inserts in package
core: previously, the presence of headers in the database was checked for
every header before writing it. With the change, if one header fails the
presence check, all subsequent headers are also assumed to be missing.
This is an improvement because in practice, the headers are almost always
missing during sync.

* p2p/enode: fix method doc (ethereum#23115)

This is an obvious spelling error

Co-authored-by: liuyaxiong <liuyaxiong@inspur.com>

* core, eth: fix precompile addresses for tracers (ethereum#23097)

* core,eth/tracers: make isPrecompiled dependent on HF

* eth/tracers: use keys when constructing chain config struct

* eth/tracers: dont initialize activePrecompiles with random value

* eth/gasprice: implement feeHistory API (ethereum#23033)

* eth/gasprice: implement feeHistory API

* eth/gasprice: factored out resolveBlockRange

* eth/gasprice: add sanity check for missing block

* eth/gasprice: fetch actual gas used from receipts

* miner, eth/gasprice: add PendingBlockAndReceipts

* internal/ethapi: use hexutil.Big

* eth/gasprice: return error when requesting beyond head block

* eth/gasprice: fixed tests and return errors correctly

* eth/gasprice: rename receiver name

* eth/gasprice: return directly if blockCount == 0

Co-authored-by: rjl493456442 <garyrong0905@gmail.com>

* accounts/abi/bind: fix gas price suggestion with pre EIP-1559 clients (ethereum#23102)

This fixes transaction sending in the case where an app using go-ethereum v1.10.4
is talking to a pre-EIP-1559 RPC node. In this case, the eth_maxPriorityFeePerGas
endpoint is not available and we can only rely on eth_gasPrice.

* p2p/server: fix method name in comment (ethereum#23123)

* ethclient/gethclient: RPC client wrapper for geth-specific API (ethereum#22977)

This commit adds the package gethclient which is similar to the ethclient
and implements some geth specific functionality.

Co-authored-by: Edgar Aroutiounian <edgar.factorial@gmail.com>
Co-authored-by: Felix Lange <fjl@twurst.com>

* cmd/geth: dont fail on deprecated toml config fields (ethereum#23118)

* internal/ethapi: fix panic in access list creation (ethereum#23133)

Fixes test failure in the last commit.

* all: removed blockhash from statedb (ethereum#23126)

This PR removes the blockhash from the statedb

* eth/fetcher, trie: unit test reliability fixes (ethereum#23020)

Some tests take quite some time during exit, which I think causes
some appveyor fails like this:

    https://ci.appveyor.com/project/ethereum/go-ethereum/builds/39511210/job/xhom84eg2e4uulq3

One of the things that seem to take time during exit is waiting
(up to 100ms) for the syncbloom to close. This PR changes it to use
a channel, instead of looping with a 100ms wait.

This also includes some unrelated changes improving the reliability of
eth/fetcher tests, which fail a lot because they are time-dependent.

* crypto: gofuzz build directives (ethereum#23137)

* eth/tracers: improve tracing performance (ethereum#23016)

Improves the performance of debug.traceTransaction

* go.mod: update UPNP dependency (ethereum#23116)

* les: avoid shutdown hang (ethereum#23139)

* consensus/clique: avoid a copy in clique (ethereum#23149)

* consensus/clique:optimize to avoid a copy in clique

* consensus/clique: test for sealhash

Co-authored-by: Martin Holst Swende <martin@swende.se>

* cmd/evm, eth/ethconfig: regenerate struct codecs (ethereum#23140)

* ethstats: fix full node interface post 1559

* dockerfile: get rid of make and env, see if that fixes builds

* cmd/puppeth: fix dashboard crash caused by updated base image

* core: fix bad parent hash when jumping to genesis in setHead (ethereum#23162)

* p2p: fix array out of bounds issue (ethereum#23165)

* fuzzing: fix typo in fuzzer definitions (ethereum#23169)

* eth, miner: add RPC method to modify miner gaslimit (pre london: ceiling) (ethereum#23134)

* core/types: sanity check the basefee length inside a header (ethereum#23171)

* all: remove noop vm config flags (ethereum#23111)

* all: rm external interpreter and ewasm config

* core/vm: rm Interpreter interface

* cmd/geth: deprecate interpreter config fields

* cmd/devp2p: fixes for eth and discv4 tests (ethereum#23155)

This PR fixes a false positive PONG 'to' endpoint mismatch seen in hive tests:

    got {IP:172.17.0.7 UDP:44025 TCP:44025}, want {IP:172.17.0.7 UDP:44025 TCP:0}

Co-authored-by: Felix Lange <fjl@twurst.com>

* core/types: remove LogForStorage type (ethereum#23173)

The encoding of Log and LogForStorage is exactly the same
now. After tracking it down it seems like ethereum#17106 changed the
storage schema of logs to be the same as the consensus
encoding.

Support for the legacy format was dropped in ethereum#22852 and if
I'm not wrong there's no reason anymore to have these two
equivalent types.

Since the RLP encoding simply contains the first three fields
of Log, we can also avoid creating a temporary struct for
encoding/decoding, and use the rlp:"-" tag in Log instead.

Note: this is an API change in core/types. We decided it's OK
to make this change because LogForStorage is an implementation
detail of go-ethereum and the type has zero uses outside of
package core/types.

Co-authored-by: Felix Lange <fjl@twurst.com>

* core/types: go generate (ethereum#23177)

* core, params: define london block at 12965000 (ethereum#23176)

* core, params: define london block at 12965000

* core/forkid: fix test

* eth/gasprice, internal/ethapi, miner: minor feehistory fixes

* params: update CHTs for the 1.10.5 release

* core/state: avoid unnecessary alloc in trie prefetcher (ethereum#23198)

* internal/ethapi: fix transaction APIs (ethereum#23179)

* internal/ethapi: fix transaction APIs

* internal/ethapi: fix typo

* internal/ethapi: address comments

* internal/ethapi: address comment from Peter

* internal: get pending and queued transaction by address (ethereum#22992)

* core, eth, internal, les, light: get pending and queued transaction by address

* core: tiny nitpick fixes

* light: tiny nitpick

Co-authored-by: mark <mark@amis.com>
Co-authored-by: Péter Szilágyi <peterke@gmail.com>

* consensus/clique: implement getSigner API method (ethereum#22987)

* clique: implement getSignerForBlock

* consensus/clique: use blockNrOrHash in getSignerForBlock

* consensus/clique: implement getSigner

* consensus/clique: fixed rlp decoding

* consensus/clique: use Author instead of getSigner

* consensus/clique: nit nit nit

* consensus/clique: nit nit nit

* feat: merged changes from head

Co-authored-by: Péter Szilágyi <peterke@gmail.com>
Co-authored-by: williamberman <wlbberman@gmail.com>
Co-authored-by: William <william.berman@coinbase.com>
Co-authored-by: piersy <pierspowlesland@gmail.com>
Co-authored-by: haryu703 <34744512+haryu703@users.noreply.github.com>
Co-authored-by: Li Dongwei <lidw1988@126.com>
Co-authored-by: lidongwei <lidongwei@huobi.com>
Co-authored-by: Martin Holst Swende <martin@swende.se>
Co-authored-by: ForLina <471133417@qq.com>
Co-authored-by: liuyaxiong <liuyaxiong@inspur.com>
Co-authored-by: Sina Mahmoodi <1591639+s1na@users.noreply.github.com>
Co-authored-by: Felföldi Zsolt <zsfelfoldi@gmail.com>
Co-authored-by: rjl493456442 <garyrong0905@gmail.com>
Co-authored-by: Pierre R <p.rousset@gmail.com>
Co-authored-by: Ahyun <urbanart2251@gmail.com>
Co-authored-by: Marius van der Wijden <m.vanderwijden@live.de>
Co-authored-by: Edgar Aroutiounian <edgar.factorial@gmail.com>
Co-authored-by: Felix Lange <fjl@twurst.com>
Co-authored-by: ucwong <ucwong@126.com>
Co-authored-by: Evolution404 <35091674+Evolution404@users.noreply.github.com>
Co-authored-by: Mark <markya0616@gmail.com>
Co-authored-by: mark <mark@amis.com>
Co-authored-by: Raymond Yeh <ray@geek.sg>
sidhujag pushed a commit to sidhujag/go-ethereum that referenced this pull request Jul 28, 2021
reds pushed a commit to reds/go-ethereum that referenced this pull request Aug 28, 2021
i-norden pushed a commit to cerc-io/go-ethereum that referenced this pull request Sep 10, 2021
atif-konasl pushed a commit to frozeman/pandora-execution-engine that referenced this pull request Oct 15, 2021
@MariusVanDerWijden MariusVanDerWijden deleted the trie-prefetch-nit branch November 30, 2021 15:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants