DoS protection: Weighted random drop of txs if mempool full #4145

Eirik0 · 2019-09-26T15:23:17Z

No description provided.

daira

I didn't do a full review, but this looks to be along the right lines.

qa/rpc-tests/mempool_limit.py

src/gtest/test_mempoollimit.cpp

src/init.cpp

Eirik0 · 2019-09-27T20:12:37Z

This is the implementation for zcash/zips#275

src/init.cpp

src/Makefile.am

src/txmempool.cpp

src/init.cpp

src/main.cpp

src/txmempool.cpp

Co-Authored-By: Daira Hopwood <daira@jacaranda.org>

Eirik0 · 2019-10-07T20:48:45Z

I rewrote the representation of the collection of transactions/weights because I was concerned about performance when removing transactions from the mempool. In the first iteration we rebuild the complete list of weighted transactions each time a transaction was removed from the mempool. Now, we store the transactions in a tree and are able to add and remove transactions from the collection in logarithmic time.

I also did some more cleanup and rebased on to master now that #4060 has been merged.

zebambam

I don't understand some of the locking and also the considerations that go into the RecentlyEvictedList constants. Would be great if we can talk about those so that I can give this a better review.

src/mempoollimit.cpp

zebambam · 2019-10-07T23:08:44Z

src/mempoollimit.cpp

+    pruneList();
+    if (txIdsAndTimes[txIdsAndTimesIndex].is_initialized()) {
+        auto txIdAndTime = txIdsAndTimes[txIdsAndTimesIndex];
+        txIdSet.erase(txIdAndTime.get().first);


So.. we just overwrite anything existing in there? This seems like someone could fill up the RecentlyEvictedList with permutations of (malleable) transactions and we still end up in DoS. What have I missed?

Note that the RecentlyEvictedList is not intended to mitigate intentional DoS.

It's a ring buffer, so by design it eventually overwrites itself if it is being filled more quickly than the entries expire. RecentlyEvictedList clearly can't be an unbounded buffer, or it would itself be a DoS vector. The question is how large the buffer's bound should be.

We could make the size of the recently evicted list configurable, but I am not sure about how beneficial that would be.

zebambam · 2019-10-07T23:16:08Z

src/mempoollimit.h

+#include "uint256.h"
+
+const size_t DEFAULT_MEMPOOL_TOTAL_WEIGHT_LIMIT = 80000000;
+const int64_t DEFAULT_MEMPOOL_EVICTION_MEMORY_MINUTES = 60;


I don't understand the considerations that went into this figure.

They're described in the draft ZIP.

src/mempoollimit.h

ebfull

Just did a brief first pass...

I think there need to be more comments explaining the purpose of each piece of the API and the algorithms. As it stands I have to reverse engineer the intentions behind much of the code and understand each edge case you are addressing.

ebfull · 2019-10-09T01:31:06Z

src/mempoollimit.cpp

+void WeightedTxTree::remove(const uint256& txId)
+{
+    if (txIdToIndexMap.find(txId) == txIdToIndexMap.end()) {
+        return;


This should never happen anyway, right?

In testing I found that this does actually happen - somewhere remove is being called on transactions that have already been removed. I did not research where or why, but as it stands this is necessary.

Please file a ticket to investigate this.

I created #4159, but I question how much time it is worth spending on this.

src/mempoollimit.cpp

ebfull · 2019-10-09T01:41:29Z

src/mempoollimit.cpp

+WeightedTxInfo WeightedTxInfo::from(const CTransaction& tx, const CAmount& fee)
+{
+    size_t memUsage = RecursiveDynamicUsage(tx);
+    memUsage += tx.vJoinSplit.size() * JOINSPLIT_SIZE;


Should these instead be folded into RecursiveDynamicUsage?

Yes, they should be (this is another example of inherited logic we've not fully integrated Zcash changes into).

Also, JOINSPLIT_SIZE now depends on the network upgrade level (which means that we computing it, we should amend the computation to trigger the post-Sapling serialization logic in the same way we amend the logic in the transaction serialization itself).

Should this be fixed as part of this issue, or can it be done as follow up?

If we did this as follow up, we could also incorporate this in to z_mergetoaddress which does its own tx size estimation.

src/mempoollimit.cpp

qa/rpc-tests/mempool_limit.py

src/main.cpp

src/gtest/test_mempoollimit.cpp

src/txmempool.h

src/init.cpp

src/txmempool.h

src/mempoollimit.h

src/mempoollimit.cpp

src/mempoollimit.h

str4d · 2019-10-09T04:38:03Z

src/mempoollimit.cpp

+WeightedTxInfo WeightedTxInfo::from(const CTransaction& tx, const CAmount& fee)
+{
+    size_t memUsage = RecursiveDynamicUsage(tx);
+    memUsage += tx.vJoinSplit.size() * JOINSPLIT_SIZE;


Yes, they should be (this is another example of inherited logic we've not fully integrated Zcash changes into).

Also, JOINSPLIT_SIZE now depends on the network upgrade level (which means that we computing it, we should amend the computation to trigger the post-Sapling serialization logic in the same way we amend the logic in the transaction serialization itself).

src/mempoollimit.cpp

str4d · 2019-10-09T04:40:59Z

src/mempoollimit.cpp

+    pruneList();
+    if (txIdsAndTimes[txIdsAndTimesIndex].is_initialized()) {
+        auto txIdAndTime = txIdsAndTimes[txIdsAndTimesIndex];
+        txIdSet.erase(txIdAndTime.get().first);


It's a ring buffer, so by design it eventually overwrites itself if it is being filled more quickly than the entries expire. RecentlyEvictedList clearly can't be an unbounded buffer, or it would itself be a DoS vector. The question is how large the buffer's bound should be.

src/mempool_limit.h

daira

reACK modulo some minor comments, and the issue about parameter names relative to the ZIP. Nice doc improvements.

Eirik0 · 2019-10-18T19:37:28Z

@zkbot try

zkbot · 2019-10-18T19:37:34Z

⌛ Trying commit e744cec with merge 5c0b3a6...

DoS protection: Weighted random drop of txs if mempool full

zkbot · 2019-10-18T20:53:19Z

💔 Test failed - pr-try

Eirik0 · 2019-10-18T21:26:25Z

Intermittent failures.

LarryRuane · 2019-10-18T23:10:55Z

src/init.cpp

-    strUsage += HelpMessageOpt("-mempoolevictionmemoryminutes=<n>", strprintf(_("The number of minutes before allowing rejected transactions to re-enter the mempool. (default: %u)"), DEFAULT_MEMPOOL_EVICTION_MEMORY_MINUTES));
-    strUsage += HelpMessageOpt("-mempooltotalcostlimit=<n>",strprintf(_("An upper bound on the maximum size in bytes of all transactions in the mempool. (default: %s)"), DEFAULT_MEMPOOL_TOTAL_COST_LIMIT));
+    strUsage += HelpMessageOpt("-mempool.eviction_memory_minutes=<n>", strprintf(_("The number of minutes before allowing rejected transactions to re-enter the mempool. (default: %u)"), DEFAULT_MEMPOOL_EVICTION_MEMORY_MINUTES));
+    strUsage += HelpMessageOpt("-mempool.tx_cost_limit=<n>",strprintf(_("An upper bound on the maximum size in bytes of all transactions in the mempool. (default: %s)"), DEFAULT_MEMPOOL_TOTAL_COST_LIMIT));


Sorry, but I think the old way is better, because, as Eirik suspected, . does seem to have a special meaning at least in Bitcoin:
https://github.com/bitcoin/bitcoin/blob/b6e34afe9735faf97d6be7a90fafd33ec18c0cbb/src/util/system.cpp#L291
https://github.com/bitcoin/bitcoin/blob/b6e34afe9735faf97d6be7a90fafd33ec18c0cbb/src/util/system.cpp#L439
https://github.com/bitcoin/bitcoin/blob/b6e34afe9735faf97d6be7a90fafd33ec18c0cbb/src/util/system.cpp#L827

I think this relates to "sections" of the configuration file. We haven't merged that yet, but we could someday.

I removed the . and _.

Eirik0 · 2019-10-21T16:53:06Z

@zkbot r+

zkbot · 2019-10-21T16:53:06Z

📌 Commit 40a7156 has been approved by Eirik0

zkbot · 2019-10-21T16:53:16Z

⌛ Testing commit 40a7156 with merge 0b0c724cde98061fe98ab5acf543f9dcef0d45cb...

zkbot · 2019-10-21T18:19:10Z

💔 Test failed - pr-merge

Eirik0 · 2019-10-21T19:40:37Z

Passed on some workers but not others, looks like transient failures.

Running 310 test cases...
0%   10   20   30   40   50   60   70   80   90   100%
|----|----|----|----|----|----|----|----|----|----|
***************************�[0;39;49
m************
********test/rpc_wallet_tests.cpp(294): error: in "rpc_wallet_tests/rpc_wallet": unexpected exception thrown by CallRPC("getblock 0")
test/rpc_wallet_tests.cpp(295): error: in "rpc_wallet_tests/rpc_wallet": unexpected exception thrown by CallRPC("getblock 0 0")
test/rpc_wallet_tests.cpp(296): error: in "rpc_wallet_tests/rpc_wallet": unexpected exception thrown by CallRPC("getblock 0 1")
test/rpc_wallet_tests.cpp(297): error: in "rpc_wallet_tests/rpc_wallet": unexpected exception thrown by CallRPC("getblock 0 2")
****
*** 4 failures are detected in the test module "Bitcoin Test Suite"

I am less familiar with the above transient failure, but those tests pass for me locally.

Eirik0 · 2019-10-21T19:40:48Z

@zkbot retry

zkbot · 2019-10-21T19:40:54Z

⌛ Testing commit 40a7156 with merge fffd5da...

DoS protection: Weighted random drop of txs if mempool full

zkbot · 2019-10-21T20:56:33Z

☀️ Test successful - pr-merge
Approved by: Eirik0
Pushing fffd5da to master...

rex4539 · 2019-10-22T00:57:30Z

Merging this broke the compilation on macOS #4164

Eirik0 requested review from str4d, daira and LarryRuane September 26, 2019 15:23

Eirik0 added this to Needs Prioritization in Arborist Team via automation Sep 26, 2019

Eirik0 added this to the v2.1.0 milestone Sep 26, 2019

Eirik0 requested a review from ebfull September 26, 2019 16:27

Eirik0 moved this from Needs Prioritization to Security Issue Backlog in Arborist Team Sep 26, 2019

Eirik0 moved this from Security Issue Backlog to In Progress in Arborist Team Sep 26, 2019

Eirik0 added the I-SECURITY Problems and improvements related to security. label Sep 26, 2019

daira reviewed Sep 26, 2019

View reviewed changes

Eirik0 changed the title ~~[WIP] DoS protection: Weighted random drop of txs if mempool full~~ DoS protection: Weighted random drop of txs if mempool full Sep 27, 2019

Eirik0 moved this from In Progress to In Review in Arborist Team Sep 27, 2019

str4d requested changes Sep 30, 2019

View reviewed changes

Eirik Ogilvie-Wigley and others added 8 commits October 7, 2019 14:23

DoS protection: Weighted random drop of txs if mempool full

ae9ecf0

Rebuild weighted list on removal and fix size calculation

a48b1ba

Grammatical fixes

0b2c5cf

Co-Authored-By: Daira Hopwood <daira@jacaranda.org>

Remove transactions when ensuring size limit

1407fd1

Help message cleanup and add lock

6fdd8f5

Performance: Store weighted transactions in a tree

1a06727

Fix naming conventions

9e97833

No need to activate Overwinter/Sapling in rpc test

a515b00

Eirik0 force-pushed the limit-mempool branch from 4968b73 to a515b00 Compare October 7, 2019 20:43

Eirik0 requested review from str4d and zebambam October 7, 2019 20:49

zebambam reviewed Oct 7, 2019

View reviewed changes

ebfull requested changes Oct 9, 2019

View reviewed changes

LarryRuane reviewed Oct 9, 2019

View reviewed changes

str4d requested changes Oct 9, 2019

View reviewed changes

daira reviewed Oct 17, 2019

View reviewed changes

src/mempool_limit.h Outdated Show resolved Hide resolved

daira reviewed Oct 17, 2019

View reviewed changes

src/mempool_limit.h Outdated Show resolved Hide resolved

daira approved these changes Oct 17, 2019

View reviewed changes

Eirik Ogilvie-Wigley added 4 commits October 17, 2019 15:16

Wording and grammatical fixes

e7aed48

Update parameter names to match ZIP

64fc25e

Clarify the difference between cost and evictionWeight

8988102

Fix test cases: default mempool limiters

e744cec

zkbot added a commit that referenced this pull request Oct 18, 2019

Auto merge of #4145 - Eirik0:limit-mempool, r=<try>

5c0b3a6

DoS protection: Weighted random drop of txs if mempool full

LarryRuane reviewed Oct 18, 2019

View reviewed changes

Remove dots and underscores from parameter names

40a7156

zkbot added a commit that referenced this pull request Oct 21, 2019

Auto merge of #4145 - Eirik0:limit-mempool, r=Eirik0

fffd5da

DoS protection: Weighted random drop of txs if mempool full

zkbot merged commit 40a7156 into zcash:master Oct 21, 2019

Arborist Team automation moved this from In Review to Released (Merged in Master) Oct 21, 2019

zkbot mentioned this pull request Oct 21, 2019

remove misleading "ERROR" from AcceptToMemoryPool debug.log lines #3477

Closed

Eirik0 mentioned this pull request Oct 21, 2019

limit the mempool size #2344

Closed

daira mentioned this pull request Aug 6, 2020

Rejected Transactions Cache #2414

Closed

DoS protection: Weighted random drop of txs if mempool full #4145

DoS protection: Weighted random drop of txs if mempool full #4145

Conversation

Eirik0 commented Sep 26, 2019

daira left a comment

Choose a reason for hiding this comment

Eirik0 commented Sep 27, 2019

Eirik0 commented Oct 7, 2019

zebambam left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ebfull left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Eirik0 Oct 9, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

daira left a comment

Choose a reason for hiding this comment

Eirik0 commented Oct 18, 2019

zkbot commented Oct 18, 2019

zkbot commented Oct 18, 2019

Eirik0 commented Oct 18, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Eirik0 commented Oct 21, 2019

zkbot commented Oct 21, 2019

zkbot commented Oct 21, 2019

zkbot commented Oct 21, 2019

Eirik0 commented Oct 21, 2019

Eirik0 commented Oct 21, 2019

zkbot commented Oct 21, 2019

zkbot commented Oct 21, 2019

rex4539 commented Oct 22, 2019

Eirik0 Oct 9, 2019 •

edited

Loading