Mempool should cache "final ledger state" #1301
Assignees
Labels
benchmarking
consensus
issues related to ouroboros-consensus
enhancement
New feature or request
mempool
optimisation
Performance optimisation
Milestone
Comments
edsko
added
consensus
|
Nice idea. Not critical yet. Lets wait 'til we have a good benchmark and/or profile of high throughput tx submission so we can measure the effect of the change (which should mostly be reduced CPU use for the case of a relay getting lots of txs from lots of peers at once so that it does not benefit from batching). |
This was referenced Nov 27, 2019
|
Apparently the latest benchmarks indicate that this is now a measurable effect, with the time to add a single tx being linearly related to the size of the mempool. This doesn't mean we have to prioritise this now. |
mrBliss
added a commit
that referenced
this issue
Feb 10, 2020
Fixes #1565 and #1301. * Replace `Mempool.addTxs` with `Mempool.tryAddTxs`, which doesn't block. This function will be much easier to test. We provide an implementation of `Mempool.addTxs` in terms of `Mempool.tryAddTxs`. We simplify the implementations by relying on the background thread for synchronising the Mempool with an updated ledger state. * Cache the `TickedLedgerState` after applying all transactions in the Mempool to it (#1301). Previously, we had to reapply all transactions whenever we added a new transaction. The new approach is much faster. * As we're keeping the `TickedLedgerState` in memory, make sure it is thunk-free. * Correct revalidation in (#1564) after explicitly removing transactions from the Mempool, see `revalidateTxsFor`. * Cleanup of conversion functions in `TxSeq`. * O(1) `MempoolSize` calculation for a `TxSeq` using its fingertree measure.
mrBliss
added a commit
that referenced
this issue
Feb 10, 2020
Fixes #1565 and #1301. * Replace `Mempool.addTxs` with `Mempool.tryAddTxs`, which doesn't block. This function will be much easier to test. We provide an implementation of `Mempool.addTxs` in terms of `Mempool.tryAddTxs`. We simplify the implementations by relying on the background thread for synchronising the Mempool with an updated ledger state. * Cache the `TickedLedgerState` after applying all transactions in the Mempool to it (#1301). Previously, we had to reapply all transactions whenever we added a new transaction. The new approach is much faster. * As we're keeping the `TickedLedgerState` in memory, make sure it is thunk-free. * Correct revalidation in (#1564) after explicitly removing transactions from the Mempool, see `revalidateTxsFor`. * Cleanup of conversion functions in `TxSeq`. * O(1) `MempoolSize` calculation for a `TxSeq` using its fingertree measure.
mrBliss
added a commit
that referenced
this issue
Feb 10, 2020
Fixes #1565 and #1301. * Replace `Mempool.addTxs` with `Mempool.tryAddTxs`, which doesn't block. This function will be much easier to test. We provide an implementation of `Mempool.addTxs` in terms of `Mempool.tryAddTxs`. We simplify the implementations by relying on the background thread for synchronising the Mempool with an updated ledger state. * Cache the `TickedLedgerState` after applying all transactions in the Mempool to it (#1301). Previously, we had to reapply all transactions whenever we added a new transaction. The new approach is much faster. * As we're keeping the `TickedLedgerState` in memory, make sure it is thunk-free. * Correct revalidation in (#1564) after explicitly removing transactions from the Mempool, see `revalidateTxsFor`. * Cleanup of conversion functions in `TxSeq`. * O(1) `MempoolSize` calculation for a `TxSeq` using its fingertree measure.
mrBliss
added a commit
that referenced
this issue
Feb 10, 2020
Fixes #1565 and #1301. * Replace `Mempool.addTxs` with `Mempool.tryAddTxs`, which doesn't block. This function will be much easier to test. We provide an implementation of `Mempool.addTxs` in terms of `Mempool.tryAddTxs`. We simplify the implementations by relying on the background thread for synchronising the Mempool with an updated ledger state. * Cache the `TickedLedgerState` after applying all transactions in the Mempool to it (#1301). Previously, we had to reapply all transactions whenever we added a new transaction. The new approach is much faster. * As we're keeping the `TickedLedgerState` in memory, make sure it is thunk-free. * Correct revalidation in (#1565) after explicitly removing transactions from the Mempool, see `revalidateTxsFor`. * Cleanup of conversion functions in `TxSeq`. * O(1) `MempoolSize` calculation for a `TxSeq` using its fingertree measure.
mrBliss
added a commit
that referenced
this issue
Feb 10, 2020
Fixes #1565 and #1301. * Replace `Mempool.addTxs` with `Mempool.tryAddTxs`, which doesn't block. This function will be much easier to test. We provide an implementation of `Mempool.addTxs` in terms of `Mempool.tryAddTxs`. We simplify the implementations by relying on the background thread for synchronising the Mempool with an updated ledger state. * Cache the `TickedLedgerState` after applying all transactions in the Mempool to it (#1301). Previously, we had to reapply all transactions whenever we added a new transaction. The new approach is much faster. * As we're keeping the `TickedLedgerState` in memory, make sure it is thunk-free. * Correct revalidation in (#1565) after explicitly removing transactions from the Mempool, see `revalidateTxsFor`. * Cleanup of conversion functions in `TxSeq`. * O(1) `MempoolSize` calculation for a `TxSeq` using its fingertree measure.
mrBliss
added a commit
that referenced
this issue
Feb 10, 2020
Fixes #1565 and #1301. * Replace `Mempool.addTxs` with `Mempool.tryAddTxs`, which doesn't block. This function will be much easier to test. We provide an implementation of `Mempool.addTxs` in terms of `Mempool.tryAddTxs`. We simplify the implementations by relying on the background thread for synchronising the Mempool with an updated ledger state. * Cache the `TickedLedgerState` after applying all transactions in the Mempool to it (#1301). Previously, we had to reapply all transactions whenever we added a new transaction. The new approach is much faster. * As we're keeping the `TickedLedgerState` in memory, make sure it is thunk-free. * Correct revalidation in (#1565) after explicitly removing transactions from the Mempool, see `revalidateTxsFor`. * Cleanup of conversion functions in `TxSeq`. * O(1) `MempoolSize` calculation for a `TxSeq` using its fingertree measure.
mrBliss
added a commit
that referenced
this issue
Feb 10, 2020
Fixes #1565 and #1301. * Replace `Mempool.addTxs` with `Mempool.tryAddTxs`, which doesn't block. This function will be much easier to test. We provide an implementation of `Mempool.addTxs` in terms of `Mempool.tryAddTxs`. We simplify the implementations by relying on the background thread for synchronising the Mempool with an updated ledger state. * Cache the `TickedLedgerState` after applying all transactions in the Mempool to it (#1301). Previously, we had to reapply all transactions whenever we added a new transaction. The new approach is much faster. * As we're keeping the `TickedLedgerState` in memory, make sure it is thunk-free. * Correct revalidation in (#1565) after explicitly removing transactions from the Mempool, see `revalidateTxsFor`. * Cleanup of conversion functions in `TxSeq`. * O(1) `MempoolSize` calculation for a `TxSeq` using its fingertree measure.
mrBliss
added a commit
that referenced
this issue
Feb 10, 2020
Fixes #1565 and #1301. * Replace `Mempool.addTxs` with `Mempool.tryAddTxs`, which doesn't block. This function will be much easier to test. We provide an implementation of `Mempool.addTxs` in terms of `Mempool.tryAddTxs`. We simplify the implementations by relying on the background thread for synchronising the Mempool with an updated ledger state. * Cache the `TickedLedgerState` after applying all transactions in the Mempool to it (#1301). Previously, we had to reapply all transactions whenever we added a new transaction. The new approach is much faster. * As we're keeping the `TickedLedgerState` in memory, make sure it is thunk-free. * Correct revalidation in (#1565) after explicitly removing transactions from the Mempool, see `revalidateTxsFor`. * Cleanup of conversion functions in `TxSeq`. * O(1) `MempoolSize` calculation for a `TxSeq` using its fingertree measure.
mrBliss
added a commit
that referenced
this issue
Feb 10, 2020
Fixes #1565 and #1301. * Replace `Mempool.addTxs` with `Mempool.tryAddTxs`, which doesn't block. This function will be much easier to test. We provide an implementation of `Mempool.addTxs` in terms of `Mempool.tryAddTxs`. We simplify the implementations by relying on the background thread for synchronising the Mempool with an updated ledger state. * Cache the `TickedLedgerState` after applying all transactions in the Mempool to it (#1301). Previously, we had to reapply all transactions whenever we added a new transaction. The new approach is much faster. * As we're keeping the `TickedLedgerState` in memory, make sure it is thunk-free. * Correct revalidation in (#1565) after explicitly removing transactions from the Mempool, see `revalidateTxsFor`. * Cleanup of conversion functions in `TxSeq`. * O(1) `MempoolSize` calculation for a `TxSeq` using its fingertree measure.
mrBliss
added a commit
that referenced
this issue
Feb 10, 2020
Fixes #1565 and #1301. * Replace `Mempool.addTxs` with `Mempool.tryAddTxs`, which doesn't block. This function will be much easier to test. We provide an implementation of `Mempool.addTxs` in terms of `Mempool.tryAddTxs`. We simplify the implementations by relying on the background thread for synchronising the Mempool with an updated ledger state. * Cache the `TickedLedgerState` after applying all transactions in the Mempool to it (#1301). Previously, we had to reapply all transactions whenever we added a new transaction. The new approach is much faster. * As we're keeping the `TickedLedgerState` in memory, make sure it is thunk-free. * Correct revalidation in (#1565) after explicitly removing transactions from the Mempool, see `revalidateTxsFor`. * Cleanup of conversion functions in `TxSeq`. * O(1) `MempoolSize` calculation for a `TxSeq` using its fingertree measure.
iohk-bors bot
added a commit
that referenced
this issue
Feb 10, 2020
1599: Mempool: better revalidation and caching of the ledger state r=mrBliss a=mrBliss Fixes #1565 and #1301. * Replace `Mempool.addTxs` with `Mempool.tryAddTxs`, which doesn't block. This function will be much easier to test. We provide an implementation of `Mempool.addTxs` in terms of `Mempool.tryAddTxs`. We simplify the implementations by relying on the background thread for synchronising the Mempool with an updated ledger state. * Cache the `TickedLedgerState` after applying all transactions in the Mempool to it (#1301). Previously, we had to reapply all transactions whenever we added a new transaction. The new approach is much faster. * As we're keeping the `TickedLedgerState` in memory, make sure it is thunk-free. * Correct revalidation in (#1564) after explicitly removing transactions from the Mempool, see `revalidateTxsFor`. * Cleanup of conversion functions in `TxSeq`. * O(1) `MempoolSize` calculation for a `TxSeq` using its fingertree measure. Also includes a rewrite of the Mempool tests using the `Mock.Utxo`. Using the `Mock.Utxo` instead of the overly simplistic `TestTx` we were using allows for more complicated dependencies between transactions. This better resembles the real transactions, at the cost of some complexity. With the former transaction type, we weren't even able to trigger #1565. Co-authored-by: Thomas Winant <thomas@well-typed.com>
mrBliss
added a commit
that referenced
this issue
Feb 10, 2020
Fixes #1565 and #1301. * Replace `Mempool.addTxs` with `Mempool.tryAddTxs`, which doesn't block. This function will be much easier to test. We provide an implementation of `Mempool.addTxs` in terms of `Mempool.tryAddTxs`. We simplify the implementations by relying on the background thread for synchronising the Mempool with an updated ledger state. * Cache the `TickedLedgerState` after applying all transactions in the Mempool to it (#1301). Previously, we had to reapply all transactions whenever we added a new transaction. The new approach is much faster. * As we're keeping the `TickedLedgerState` in memory, make sure it is thunk-free. * Correct revalidation in (#1565) after explicitly removing transactions from the Mempool, see `revalidateTxsFor`. * Cleanup of conversion functions in `TxSeq`. * O(1) `MempoolSize` calculation for a `TxSeq` using its fingertree measure.
iohk-bors bot
added a commit
that referenced
this issue
Feb 10, 2020
1599: Mempool: better revalidation and caching of the ledger state r=mrBliss a=mrBliss Fixes #1565 and #1301. * Replace `Mempool.addTxs` with `Mempool.tryAddTxs`, which doesn't block. This function will be much easier to test. We provide an implementation of `Mempool.addTxs` in terms of `Mempool.tryAddTxs`. We simplify the implementations by relying on the background thread for synchronising the Mempool with an updated ledger state. * Cache the `TickedLedgerState` after applying all transactions in the Mempool to it (#1301). Previously, we had to reapply all transactions whenever we added a new transaction. The new approach is much faster. * As we're keeping the `TickedLedgerState` in memory, make sure it is thunk-free. * Correct revalidation in (#1564) after explicitly removing transactions from the Mempool, see `revalidateTxsFor`. * Cleanup of conversion functions in `TxSeq`. * O(1) `MempoolSize` calculation for a `TxSeq` using its fingertree measure. Also includes a rewrite of the Mempool tests using the `Mock.Utxo`. Using the `Mock.Utxo` instead of the overly simplistic `TestTx` we were using allows for more complicated dependencies between transactions. This better resembles the real transactions, at the cost of some complexity. With the former transaction type, we weren't even able to trigger #1565. Co-authored-by: Thomas Winant <thomas@well-typed.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
The mempool should store a reference to the ledger state after all transactions have been validated. Then when new transactions get added, and the ledger state at the tip of the chain has not changed, we can skip transaction evaluation altogether (
initVR,afterKnownValid) and just reuse the previously stored ledger state. This could greatly improve the performance of the mempool, with relatively low memory overhead (I think).The text was updated successfully, but these errors were encountered: