Minimal effective gas price in the queue #8934

tomusdrw · 2018-06-20T11:36:43Z

For small (and saturated) queues this optimizes the verification pipeline to avoid recovering a sender in case we are quite sure that the transaction won't get into the pool anyway.

ascjones

Looks good. Noticed an unrelated potential issue in pool.remove_worst()

ascjones · 2018-06-20T16:54:25Z

transaction-pool/src/pool.rs

+			return None
+		}
+
+		self.worst_transactions.iter().next().map(|x| x.score.clone())


Using next() here which looks correct (worst_transactions is ordered by ascending score). However I noticed: https://github.com/paritytech/parity/blob/0335e927e5d2133ceba4b54579094f1b4d810eac/transaction-pool/src/pool.rs#L277
Is the next_back() wrong then? Seems like it would check for the least worst transaction to replace.

Great catch, I'll prepare a test and fix that issue with remove_worst

Actually it's correct in remove_worst. ScoreWithRef has a custom Ord implementation that is ascending if the internal score is descending. So the error is actually here, will make sure to add a correct test to cover this.

Looks good. Should this one also change to next_back() then?

https://github.com/paritytech/parity/blob/b56200916dfb90149ab22901dac5ef951688974b/transaction-pool/src/pool.rs#L393

Indeed, preparing a test case to cover that and a fix.

5chdn · 2018-06-22T09:18:16Z

Needs a 2nd review.

tomusdrw · 2018-06-22T09:46:06Z

@ascjones thanks for the review, the issue with worst_transaction is addressed now.

ascjones

Approval 2.0 after worst_transaction fixes. (Still needs a second reviewer though)

jimpo · 2018-06-25T23:05:18Z

Code looks fine, but I think this can be better encapsulated in scoring. Instead of passing the gas price into the verifier, could the queue just check should_replace against the pool's worst_transaction directly? This is more in line with the check that happens in import anyway, and this way the score <-> gas price equivalence doesn't leak as much.

The current implementation also has the effect of potentially rejecting txs even if the queue is not full (but at 90% capacity), which I'm not sure is intended/desirable.

EDIT: I see using should_replace is not so easy because it takes VerifiedTransaction, not verifier::Transaction. Note that this change modifies behavior in the case that the worst transaction is a higher-gas-price, higher-nonce transaction by the same sender as the incoming one. The current behavior is to accept the new transaction because the nonce is lower, whereas after the change it would be rejected.

tomusdrw · 2018-06-27T10:32:27Z

@jimpo thanks for the review, I've updated the PR a bit:

Removed the threshold (it didn't have much sense)
Moved the decision making to the Scoring implementation.

Unfortunately we are not able to re-use should_replace, since the whole point is that it's just a heuristic to quickly reject some transactions, we want to do it without recovering the sender in verification, so there is no way to check if the nonce is lower/higher.

That said it's not really that much of an issue, the pool decisions are always local and are not part of the consensus itself. To have the most accurate image of the pending transactions one should run with big enough pool to store them all.

svyatonik

lgtm, cargo test --all also completes without any errors locally

5chdn · 2018-06-29T08:23:07Z

Please rebase on latest master to fix builds.

* master: Minimal effective gas price in the queue (#8934) parity: fix db path when migrating to blooms db (#8975) Preserve the current abort behavior (#8995) Improve should_replace on NonceAndGasPrice (#8980) Tentative fix for missing dependency error (#8973)

* Minimal effective gas price. * Fix naming, add test * Fix minimal entry score and add test. * Fix worst_transaction. * Remove effective gas price threshold. * Don't leak gas_price decisions out of Scoring.

* parity-version: bump beta to 1.11.6 * scripts: remove md5 checksums (#8884) * Add support for --chain tobalaba * Convert indents to tabs :) * Fixes for misbehavior reporting in AuthorityRound (#8998) * aura: only report after checking for repeated skipped primaries * aura: refactor duplicate code for getting epoch validator set * aura: verify_external: report on validator set contract instance * aura: use correct validator set epoch number when reporting * aura: use epoch set when verifying blocks * aura: report skipped primaries when generating seal * aura: handle immediate transitions * aura: don't report skipped steps from genesis to first block * aura: fix reporting test * aura: refactor duplicate code to handle immediate_transitions * aura: let reporting fail on verify_block_basic * aura: add comment about possible failure of reporting * Only return error log for rustls (#9025) * Transaction Pool improvements (#8470) * Don't use ethereum_types in transaction pool. * Hide internal insertion_id. * Fix tests. * Review grumbles. * Improve should_replace on NonceAndGasPrice (#8980) * Additional tests for NonceAndGasPrice::should_replace. * Fix should_replace in the distinct sender case. * Use natural priority ordering to simplify should_replace. * Minimal effective gas price in the queue (#8934) * Minimal effective gas price. * Fix naming, add test * Fix minimal entry score and add test. * Fix worst_transaction. * Remove effective gas price threshold. * Don't leak gas_price decisions out of Scoring. * Never drop local transactions from different senders. (#9002) * Recently rejected cache for transaction queue (#9005) * Store recently rejected transactions. * Don't cache AlreadyImported rejections. * Make the size of transaction verification queue dependent on pool size. * Add a test for recently rejected. * Fix logging for recently rejected. * Make rejection cache smaller. * obsolete test removed * obsolete test removed * Construct cache with_capacity. * Optimize pending transactions filter (#9026) * rpc: return unordered transactions in pending transactions filter * ethcore: use LruCache for nonce cache Only clear the nonce cache when a block is retracted * Revert "ethcore: use LruCache for nonce cache" This reverts commit b382c19. * Use only cached nonces when computing pending hashes. * Give filters their own locks, so that they don't block one another. * Fix pending transaction count if not sealing. * Clear cache only when block is enacted. * Fix RPC tests. * Address review comments. * A last bunch of txqueue performance optimizations (#9024) * Clear cache only when block is enacted. * Add tracing for cull. * Cull split. * Cull after creating pending block. * Add constant, remove sync::read tracing. * Reset debug. * Remove excessive tracing. * Use struct for NonceCache. * Fix build * Remove warnings. * Fix build again. * miner: add missing macro use for trace_time * ci: remove md5 merge leftovers

…rp_sync_on_light_client * 'master' of https://github.com/paritytech/parity: Attempt to graceful shutdown in case of panics (openethereum#8999) simplify kvdb error types (openethereum#8924) Add option for user to set max size limit for RPC requests (openethereum#9010) bump ntp to 0.5.0 (openethereum#9009) Removed duplicate dependency (openethereum#9021) Minimal effective gas price in the queue (openethereum#8934)

tomusdrw added 3 commits June 11, 2018 10:20

Minimal effective gas price.

7f4d825

Fix naming, add test

04bb389

Merge branch 'master' into td-min-eff-gas-price

0335e92

tomusdrw added A0-pleasereview 🤓 Pull request needs code review. M4-core ⛓ Core client code / Rust. labels Jun 20, 2018

5chdn added this to the 1.12 milestone Jun 20, 2018

ascjones approved these changes Jun 20, 2018

View reviewed changes

Fix minimal entry score and add test.

b562009

tomusdrw added A4-gotissues 💥 Pull request is reviewed and has significant issues which must be addressed. and removed A0-pleasereview 🤓 Pull request needs code review. labels Jun 22, 2018

Fix worst_transaction.

733091c

tomusdrw added A0-pleasereview 🤓 Pull request needs code review. and removed A4-gotissues 💥 Pull request is reviewed and has significant issues which must be addressed. labels Jun 22, 2018

ascjones approved these changes Jun 22, 2018

View reviewed changes

tomusdrw mentioned this pull request Jun 25, 2018

Massive increase in CPU usage with 1.11.1 #8696

Closed

jimpo mentioned this pull request Jun 25, 2018

Improve should_replace on NonceAndGasPrice #8980

Merged

tomusdrw added the B0-patchthis label Jun 27, 2018

tomusdrw added 2 commits June 27, 2018 11:22

Remove effective gas price threshold.

b3a77c0

Don't leak gas_price decisions out of Scoring.

38debef

jimpo approved these changes Jun 28, 2018

View reviewed changes

svyatonik approved these changes Jun 29, 2018

View reviewed changes

5chdn added A7-looksgoodtestsfail 🤖 Pull request is reviewed well, but cannot be merged due to tests failing. and removed A0-pleasereview 🤓 Pull request needs code review. labels Jun 29, 2018

Merge branch 'master' into td-min-eff-gas-price

9308c33

5chdn mentioned this pull request Jun 30, 2018

Beta: 1.11.6 backports #9015

Merged

12 tasks

5chdn added A8-looksgood 🦄 Pull request is reviewed well. and removed A7-looksgoodtestsfail 🤖 Pull request is reviewed well, but cannot be merged due to tests failing. labels Jun 30, 2018

5chdn merged commit 1792725 into master Jun 30, 2018

5chdn deleted the td-min-eff-gas-price branch June 30, 2018 09:11

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Minimal effective gas price in the queue #8934

Minimal effective gas price in the queue #8934

tomusdrw commented Jun 20, 2018

ascjones left a comment

ascjones Jun 20, 2018

tomusdrw Jun 21, 2018

tomusdrw Jun 21, 2018

ascjones Jun 21, 2018

tomusdrw Jun 22, 2018

5chdn commented Jun 22, 2018

tomusdrw commented Jun 22, 2018

ascjones left a comment •

edited

Loading

jimpo commented Jun 25, 2018 •

edited

Loading

tomusdrw commented Jun 27, 2018

svyatonik left a comment

5chdn commented Jun 29, 2018

Minimal effective gas price in the queue #8934

Minimal effective gas price in the queue #8934

Conversation

tomusdrw commented Jun 20, 2018

ascjones left a comment

Choose a reason for hiding this comment

ascjones Jun 20, 2018

Choose a reason for hiding this comment

tomusdrw Jun 21, 2018

Choose a reason for hiding this comment

tomusdrw Jun 21, 2018

Choose a reason for hiding this comment

ascjones Jun 21, 2018

Choose a reason for hiding this comment

tomusdrw Jun 22, 2018

Choose a reason for hiding this comment

5chdn commented Jun 22, 2018

tomusdrw commented Jun 22, 2018

ascjones left a comment • edited Loading

Choose a reason for hiding this comment

jimpo commented Jun 25, 2018 • edited Loading

tomusdrw commented Jun 27, 2018

svyatonik left a comment

Choose a reason for hiding this comment

5chdn commented Jun 29, 2018

ascjones left a comment •

edited

Loading

jimpo commented Jun 25, 2018 •

edited

Loading