Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Mempool package tracking #6654

Merged
merged 2 commits into from Sep 21, 2015
Merged

Mempool package tracking #6654

merged 2 commits into from Sep 21, 2015

Conversation

@sdaftuar
Copy link
Member

sdaftuar commented Sep 9, 2015

This is a preparatory pull to try to make reviewing #6557 easier.

In #6557, I added tracking packages of transactions to the mempool (for each tx, we track all the "descendant" transactions that depend on that tx), in order to make the mempool limiting code more effective. This PR is a standalone implementation of mempool descendant tracking, including the policy limits on transaction chains (limiting ancestors and descendants) proposed in #6557.

I'll rebase that pull off these commits assuming we can agree on this approach.

@sdaftuar
Copy link
Member Author

sdaftuar commented Sep 9, 2015

@sipa I did end up reworking the tracking of in-mempool parents/children to use iterators rather than hashes as you had suggested (which I never pushed up to #6557).

@btcdrak
Copy link
Contributor

btcdrak commented Sep 9, 2015

@laanwj looks like travis crapped out on one of the jobs and needs restarting.

@sdaftuar
Copy link
Member Author

sdaftuar commented Sep 9, 2015

@btcdrak Actually I think this is some kind of problem with the unit test code -- not sure why it fails to compile only in that one job but I was just able to reproduce locally. Working on a fix...

@sdaftuar sdaftuar force-pushed the sdaftuar:mempool-packages-only branch to d6401c1 Sep 9, 2015
@sdaftuar
Copy link
Member Author

sdaftuar commented Sep 9, 2015

@btcdrak Fixed and pushed back up, travis is happy now...

src/main.h Outdated
@@ -466,5 +474,7 @@ static const unsigned int REJECT_HIGHFEE = 0x100;
static const unsigned int REJECT_ALREADY_KNOWN = 0x101;
/** Transaction conflicts with a transaction already known */
static const unsigned int REJECT_CONFLICT = 0x102;
/** Transaction would result in too long in-mempool chain */
static const unsigned int REJECT_LONGCHAIN = 0x103;

This comment has been minimized.

Copy link
@sdaftuar

sdaftuar Sep 14, 2015

Author Member

Will move this to not be an internal code (so we send a reject message back).

// Calculate in-mempool ancestors, up to a limit.
CTxMemPool::setEntries setAncestors;
size_t nLimitAncestors = GetArg("-limitancestorcount", DEFAULT_ANCESTOR_LIMIT);
size_t nLimitAncestorSize = GetArg("-limitancestorsize", DEFAULT_ANCESTOR_SIZE_LIMIT)*1000;

This comment has been minimized.

Copy link
@petertodd

petertodd Sep 14, 2015

Contributor

As the idea is the ancestor size is set such that the whole string of transactions could be included in a single block, we really want this to be a less than the max block size, as there's always overhead to consider. (e.g. the coinbase, soft-limit, etc)

I'd suggest we set this to 900KB for a 100KB buffer - plenty even in the case of p2pool's large max 50KB coinbase. Another option might be to set it based on the soft-limit - default 750KB - with another 100KB of overhad buffer.

@petertodd
Copy link
Contributor

petertodd commented Sep 14, 2015

In general, it'd be good to think about separating the reorg case entirely from the main mempool codebase. For instance, keep a simple linear list of reorged transactions, and after a reorg attempt to add them back the mempool one-by-one. This separate code could also handle cases where we might want to remine transactions that we otherwise wouldn't, as a "goodwill" gesture to reduce the impact of large reorgs.

{
// Track the number of entries (outside setExclude) that we'd need to visit
// (will bail out if it exceeds maxDescendantsToVisit)
int nChildrenToVisit = 0;

This comment has been minimized.

Copy link
@petertodd

petertodd Sep 14, 2015

Contributor

trailing whitespace

@pstratem
Copy link
Contributor

pstratem commented Sep 14, 2015

Needs a comment defining ancestor/descendant.


while (!parentHashes.empty()) {
setAncestors.insert(parentHashes.begin(), parentHashes.end());
setEntries stageParentSet;

This comment has been minimized.

Copy link
@petertodd

petertodd Sep 14, 2015

Contributor

trailing whitespace

@morcos morcos mentioned this pull request Sep 14, 2015
errString = strprintf("too many unconfirmed ancestors [limit: %u]", limitAncestorCount);
return false;
}
}

This comment has been minimized.

Copy link
@petertodd

petertodd Sep 14, 2015

Contributor

trailing whitespace

minerPolicyEstimator->removeTx(hash);
}

// Calculates descendants of hash that are not already in setDescendants, and adds to

This comment has been minimized.

Copy link
@petertodd

petertodd Sep 14, 2015

Contributor

trailing whitespace

innerUsage += it->second.DynamicMemoryUsage();
const CTransaction& tx = it->second.GetTx();
checkTotal += it->GetTxSize();
innerUsage += it->DynamicMemoryUsage();

This comment has been minimized.

Copy link
@petertodd

petertodd Sep 14, 2015

Contributor

trailing whitespace

@@ -138,6 +363,29 @@ class CTxMemPool
void ApplyDeltas(const uint256 hash, double &dPriorityDelta, CAmount &nFeeDelta);
void ClearPrioritisation(const uint256 hash);

public:
void RemoveStaged(std::set<uint256>& stage);

This comment has been minimized.

Copy link
@petertodd

petertodd Sep 14, 2015

Contributor

Need to document what RemoveStaged() is.

/** When adding transactions from a disconnected block back to the mempool,
* new mempool entries may have children in the mempool (which is generally
* not the case when otherwise adding transactions).
* UpdateTransactionsFromBlock will find child transactions and update the

This comment has been minimized.

Copy link
@petertodd

petertodd Sep 14, 2015

Contributor

Would prefer if comments like this are written as "Name()" rather than just "Name" to make it clear what's a variable and what's a function.

/** UpdateForDescendants is used by UpdateTransactionsFromBlock to update
* the descendants for a single transaction that has been added to the
* mempool but may have child transactions in the mempool, eg during a
* chain reorg. setExclude is the set of descendant transactions in the

This comment has been minimized.

Copy link
@petertodd

petertodd Sep 14, 2015

Contributor

Double-space after periods? Obvs you're actually Satoshi.

* descendant state for each transaction in hashesToUpdate (excluding any
* child transactions present in hashesToUpdate, which are already accounted
* for).
*/

This comment has been minimized.

Copy link
@petertodd

petertodd Sep 14, 2015

Contributor

This comment doesn't make it clear whether or not transactions in hashesToUpdate are or are not already in the mempool. :)

@sipa
Copy link
Member

sipa commented Sep 14, 2015

Concept ACK.

@petertodd
Copy link
Contributor

petertodd commented Sep 14, 2015

utACK, modulo nits.

@jonasschnelli
Copy link
Member

jonasschnelli commented Sep 14, 2015

Concept ACK / code review ACK.
Passes gitian/osx/debian build/unit-tests/rpc-tests.

@petertodd
Copy link
Contributor

petertodd commented Sep 14, 2015

Tested with my FSS and Full RBF stress tests and -checkmempool, no unexpected failures. (this pull-req of course doesn't enable any RBF behavior, so replacements failed!)

@dcousens
Copy link
Contributor

dcousens commented Sep 14, 2015

concept ACK

@@ -921,6 +921,17 @@ bool AcceptToMemoryPool(CTxMemPool& pool, CValidationState &state, const CTransa
REJECT_HIGHFEE, "absurdly-high-fee",
strprintf("%d > %d", nFees, ::minRelayTxFee.GetFee(nSize) * 10000));

// Calculate in-mempool ancestors, up to a limit.
CTxMemPool::setEntries setAncestors;
size_t nLimitAncestors = GetArg("-limitancestorcount", DEFAULT_ANCESTOR_LIMIT);

This comment has been minimized.

Copy link
@dcousens

dcousens Sep 14, 2015

Contributor

Using GetArg (an application level function) in mempool code feels a bit odd.

This comment has been minimized.

Copy link
@sdaftuar

sdaftuar Sep 16, 2015

Author Member

I agree but is there a better way to do this? I think I'm following the common practice in the code base, though I look forward to a future refactoring when we have better encapsulation.


setEntries entriesToAdd;
BOOST_FOREACH(const txiter cit, stageEntries) {
if (cit->IsDirty()) {

This comment has been minimized.

Copy link
@dcousens

dcousens Sep 14, 2015

Contributor

Dirty implies something changed typically, perhaps instead change this to ShouldSkip?

// include them, and update their setMemPoolParents to include this tx.
for (; iter != mapNextTx.end() && iter->first.hash == hash; ++iter) {
const uint256 &childHash = iter->second.ptx->GetHash();
txiter childIter = mapTx.find(childHash);

This comment has been minimized.

Copy link
@dcousens

dcousens Sep 14, 2015

Contributor

Do we need to check childIter == mapTx.end()?

This comment has been minimized.

Copy link
@sdaftuar

sdaftuar Sep 16, 2015

Author Member

I'll add an assert to make clear that is not possible.

{
// For each entry, walk back all ancestors and decrement size associated with this
// transaction
uint64_t nNoLimit = std::numeric_limits<uint64_t>::max();

This comment has been minimized.

Copy link
@dcousens

dcousens Sep 14, 2015

Contributor

const?

{
nCountWithDescendants=0;
nSizeWithDescendants=nTxSize;
nFeesWithDescendants=nFee;

This comment has been minimized.

Copy link
@dcousens

dcousens Sep 14, 2015

Contributor

nit: Spacing between operators

}
int64_t updateCount = (add ? 1 : -1);
int64_t updateSize = updateCount * it->GetTxSize();
CAmount updateFee = updateCount * it->GetFee();

This comment has been minimized.

Copy link
@dcousens

dcousens Sep 14, 2015

Contributor

nit: const on these?

}

bool CTxMemPool::addUnchecked(const uint256&hash, const CTxMemPoolEntry &entry, bool fCurrentEstimate)
{

This comment has been minimized.

Copy link
@morcos

morcos Sep 16, 2015

Member

needs LOCK(cs)?

return memusage::MallocUsage(sizeof(CTxMemPoolEntry) + 9 * sizeof(void*)) * mapTx.size() + memusage::DynamicUsage(mapNextTx) + memusage::DynamicUsage(mapDeltas) + memusage::DynamicUsage(mapLinks) + cachedInnerUsage;
}

void CTxMemPool::RemoveStaged(setEntries &stage) {

This comment has been minimized.

Copy link
@morcos

morcos Sep 16, 2015

Member

Needs LOCK(cs) if ever called from somewhere else.

This comment has been minimized.

Copy link
@TheBlueMatt

TheBlueMatt Sep 19, 2015

Contributor

You can also AssertLockHeld(cs), otherwise, to make things more readable.

On September 16, 2015 11:05:34 AM CDT, Alex Morcos notifications@github.com wrote:

@@ -429,5 +758,58 @@ bool CCoinsViewMemPool::HaveCoins(const uint256
&txid) const {

size_t CTxMemPool::DynamicMemoryUsage() const {
LOCK(cs);

  • return memusage::DynamicUsage(mapTx) +
    memusage::DynamicUsage(mapNextTx) + memusage::DynamicUsage(mapDeltas) +
    cachedInnerUsage;
  • // Estimate the overhead of mapTx to be 9 pointers + an
    allocation, as no exact formula for boost::multi_index_contained is
    implemented.
  • return memusage::MallocUsage(sizeof(CTxMemPoolEntry) + 9 *
    sizeof(void*)) * mapTx.size() + memusage::DynamicUsage(mapNextTx) +
    memusage::DynamicUsage(mapDeltas) + memusage::DynamicUsage(mapLinks) +
    cachedInnerUsage;
    +}

+void CTxMemPool::RemoveStaged(setEntries &stage) {

Needs LOCK(cs) if ever called from somewhere else.


Reply to this email directly or view it on GitHub:
https://github.com/bitcoin/bitcoin/pull/6654/files#r39649424

// for each entry, look for descendants that are outside hashesToUpdate, and
// add fee/size information for such descendants to the parent.
void CTxMemPool::UpdateTransactionsFromBlock(const std::vector<uint256> &vHashesToUpdate)
{

This comment has been minimized.

Copy link
@morcos

morcos Sep 16, 2015

Member

Needs LOCK(cs)

return a->GetTx().GetHash() < b->GetTx().GetHash();
}
};
typedef std::set<txiter, CompareIteratorByHash> setEntries;

This comment has been minimized.

Copy link
@morcos

morcos Sep 16, 2015

Member

oops, setEntries needs to be public, the rest can be private though.

@TheBlueMatt
Copy link
Contributor

TheBlueMatt commented Sep 19, 2015

My computer's battery died, but in txmempool.h, the definition of txiter as "boost::multi_index_container::iterator" is "implementation defined" according to boost's docs. In the latest versions it is defined to an equivalent of "boost::multi_index_container::nth_index<0>::type::iterator", so you should use that.

On September 16, 2015 8:44:26 AM CDT, Suhas Daftuar notifications@github.com wrote:

Thanks everyone for reviewing! I've pushed up a series of commits to
address everyone's feedback; I believe all comments should have been
addressed.

These cleanups need to be squashed, and this pull now needs to be
rebased (a merge conflict crept in now that I'm outputting additional
information in getrawmempool). Reviewers -- please let me know if you
prefer I leave these commits unsquashed/unrebased so you can review the
changes. (In the absence of any expressed preferences, I'll plan to
squash/rebase in a day or two so that this can become mergeable.)

@petertodd I added the extra information to the getrawmempool RPC call
(good idea!). Now that the RPC call is there, I realized that adding
an RPC test that exercises the new limits is a good idea, so I've
started work on that, but it's not yet complete.


Reply to this email directly or view it on GitHub:
#6654 (comment)

@dcousens
Copy link
Contributor

dcousens commented Sep 19, 2015

@sdaftuar needs rebase.

ashleyholman and others added 2 commits Jun 24, 2015
Indexes on:
- Tx Hash
- Fee Rate (fee-per-kb)
Associate with each CTxMemPoolEntry all the size/fees of descendant
mempool transactions.  Sort mempool by max(feerate of entry, feerate
of descendants).  Update statistics on-the-fly as transactions enter
or leave the mempool.

Also add ancestor and descendant limiting, so that transactions can
be rejected if the number or size of unconfirmed ancestors exceeds
a target, or if adding a transaction would cause some other mempool
entry to have too many (or too large) a set of unconfirmed in-
mempool descendants.
@sdaftuar sdaftuar force-pushed the sdaftuar:mempool-packages-only branch to 5add7a7 Sep 19, 2015
@sdaftuar
Copy link
Member Author

sdaftuar commented Sep 19, 2015

I addressed the latest comments from @morcos and @TheBlueMatt, added an rpc test (it only tests the ancestor/descendant length limits, not the size limits, nor the handling of reorgs -- so that still can be improved), and then squashed everything down and rebased on master to get rid of the merge conflict in rpcblockchain.cpp.

@Mirobit
Copy link
Contributor

Mirobit commented Sep 19, 2015

Does anyone else think the ancestor limit is way to generous? In what other case than spamming do txs have 100 ancestors in mempool? Shouldn't 5 be enough?

@petertodd
Copy link
Contributor

petertodd commented Sep 20, 2015

@Mirobit The limits are there to limit computational costs in determining things like dependent fees/size etc. There's no need to set them low unless the algorithms take a long time to compute those sums; the data structures in the mempool are mainly pointer following so it's fairly fast.

@laanwj
Copy link
Member

laanwj commented Sep 21, 2015

utACK

@laanwj laanwj merged commit 5add7a7 into bitcoin:master Sep 21, 2015
1 check passed
1 check passed
continuous-integration/travis-ci/pr The Travis CI build passed
Details
laanwj added a commit that referenced this pull request Sep 21, 2015
5add7a7 Track transaction packages in CTxMemPoolEntry (Suhas Daftuar)
34628a1 TxMemPool: Change mapTx to a boost::multi_index_container (Ashley Holman)
@sipa
Copy link
Member

sipa commented Sep 22, 2015

My node running 5add7a7 with -checkmempool crashed. Last debug.log message was:

2015-09-22 13:06:19 - Disconnect block: 90.04ms
@morcos
Copy link
Member

morcos commented Sep 22, 2015

@sipa can you give us some more details? were you running will all -debug options? for instance I assume you were not running with estimatefee debugging? it helps to narrow down where it crashed based on what didn't get printed to the debug log.

@sdaftuar sdaftuar mentioned this pull request Sep 23, 2015
zkbot added a commit to zcash/zcash that referenced this pull request Feb 20, 2018
Mempool improvements, branch ID awareness

Whenever the local chain tip is updated, transactions in the mempool which commit to an
unmineable branch ID (for example, just before a network upgrade activates, where the
next block will have a different branch ID) will be removed.

Includes commits cherry-picked from the following upstream PRs:

- bitcoin/bitcoin#6654
  - Only the mempool index change.
- bitcoin/bitcoin#6776
- bitcoin/bitcoin#7020
- bitcoin/bitcoin#6915

Part of #2074.
zkbot added a commit to zcash/zcash that referenced this pull request Feb 20, 2018
Mempool improvements, branch ID awareness

Whenever the local chain tip is updated, transactions in the mempool which commit to an
unmineable branch ID (for example, just before a network upgrade activates, where the
next block will have a different branch ID) will be removed.

Includes commits cherry-picked from the following upstream PRs:

- bitcoin/bitcoin#6654
  - Only the mempool index change.
- bitcoin/bitcoin#6776
- bitcoin/bitcoin#7020
- bitcoin/bitcoin#6915

Part of #2074.
furszy added a commit to PIVX-Project/PIVX that referenced this pull request Jun 14, 2020
2105947 Implement helper class for CTxMemPoolEntry constructor (Alex Morcos)
1cef905 Make -checkmempool=1 not fail through int32 overflow (Pieter Wuille)
0f72ff2 Support -checkmempool=N, which runs checks on average once every N transactions (Pieter Wuille)
89483d0 [Bug] Make operator() a const function in CompareTxMemPoolEntryByX (random-zebra)
a50ad77 Lower default policy limits (random-zebra)
03f7152 fix locking issue with new mempool limiting (random-zebra)
1598961 Fix stale comment in CTxMemPool::TrimToSize. (random-zebra)
98d0d68 Undo GetMinFee-requires-extra-call-to-hit-0 (random-zebra)
6ad6ee6 Add reasonable test case for mempool trimming (random-zebra)
8dcbb7e Only call TrimToSize once per reorg/blocks disconnect (random-zebra)
c20cd38 Implement on-the-fly mempool size limitation. (random-zebra)
aee2e17 Print mempool size in KB when adding txn (random-zebra)
f7c85fd Add CFeeRate += operator (random-zebra)
5bd2a00 Track (and define) ::minRelayTxFee in CTxMemPool (random-zebra)
0b50f6c Add Mempool Expire function to remove old transactions (random-zebra)
d26f5e0 Fix calling mempool directly, instead of pool, in ATMP (random-zebra)
fc5eddb Reverse the sort on the mempool's feerate index (random-zebra)
0ce1df0 [BUG] Fix CTxMemPool::check excluding zerocoins from children checks (random-zebra)
1f7bd52 Track transaction packages in CTxMemPoolEntry (random-zebra)
1fd406b TxMemPool: Change mapTx to a boost::multi_index_container (random-zebra)

Pull request description:

  built on top of
  - [x] #1645

  This PR pulls some updates from upstream in the mempool area, adding the required adjustments for legacy zerocoin txes and updating the functional test suite.

  Specifically, here we:
  - track mempool descendants (in-mempool transactions that depend on other mempool transactions)
  - turn `mapTx` into a `boost::multi_index_container` that sorts the mempool on 3 criteria:
    - transaction hash
    - fee rate
    - time in the mempool
  - Add a max size for the mempool (throwing away the cheapest txs and bumping the min relay fee, when full)
  - Implement on-the-fly mempool size limit with the flag `-maxmempool`
  - Implement `-checkmempool=N` to customize the frequency of the mempool check
  - Implement helper for `CTxMemPoolEntry` for the unit tests.

  Backports:

  - bitcoin#6654
  - bitcoin#6722 [`*`]
  - bitcoin#6889
  - bitcoin#6771
  - bitcoin#6776
  - bitcoin#6896
  - bitcoin#7020

  [`*`] excluding bitcoin@9e93640 as our default minimum tx fee rate of 10k satoshis is only 0,00003 USD at the time of writing.

ACKs for top commit:
  Fuzzbawls:
    utACK 2105947
  furszy:
    Re utACK 2105947 and merging this nice upgrade :) .

Tree-SHA512: 51a7d75bd52f7646d461252c78f0dd9d7e8b5c1c66c22944120bfe293b28f5d48135de339ebf3d8a5b4c61ca5452383ed1b10c417be06dc4a335ac645842ea14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

You can’t perform that action at this time.