Mempool package tracking #6654

Merged
merged 2 commits into from Sep 21, 2015

Conversation

Projects
None yet
@sdaftuar
Member

sdaftuar commented Sep 9, 2015

This is a preparatory pull to try to make reviewing #6557 easier.

In #6557, I added tracking packages of transactions to the mempool (for each tx, we track all the "descendant" transactions that depend on that tx), in order to make the mempool limiting code more effective. This PR is a standalone implementation of mempool descendant tracking, including the policy limits on transaction chains (limiting ancestors and descendants) proposed in #6557.

I'll rebase that pull off these commits assuming we can agree on this approach.

@sdaftuar

This comment has been minimized.

Show comment
Hide comment
@sdaftuar

sdaftuar Sep 9, 2015

Member

@sipa I did end up reworking the tracking of in-mempool parents/children to use iterators rather than hashes as you had suggested (which I never pushed up to #6557).

Member

sdaftuar commented Sep 9, 2015

@sipa I did end up reworking the tracking of in-mempool parents/children to use iterators rather than hashes as you had suggested (which I never pushed up to #6557).

@btcdrak

This comment has been minimized.

Show comment
Hide comment
@btcdrak

btcdrak Sep 9, 2015

Member

@laanwj looks like travis crapped out on one of the jobs and needs restarting.

Member

btcdrak commented Sep 9, 2015

@laanwj looks like travis crapped out on one of the jobs and needs restarting.

@sdaftuar

This comment has been minimized.

Show comment
Hide comment
@sdaftuar

sdaftuar Sep 9, 2015

Member

@btcdrak Actually I think this is some kind of problem with the unit test code -- not sure why it fails to compile only in that one job but I was just able to reproduce locally. Working on a fix...

Member

sdaftuar commented Sep 9, 2015

@btcdrak Actually I think this is some kind of problem with the unit test code -- not sure why it fails to compile only in that one job but I was just able to reproduce locally. Working on a fix...

@sdaftuar

This comment has been minimized.

Show comment
Hide comment
@sdaftuar

sdaftuar Sep 9, 2015

Member

@btcdrak Fixed and pushed back up, travis is happy now...

Member

sdaftuar commented Sep 9, 2015

@btcdrak Fixed and pushed back up, travis is happy now...

src/main.h
@@ -466,5 +474,7 @@ static const unsigned int REJECT_HIGHFEE = 0x100;
static const unsigned int REJECT_ALREADY_KNOWN = 0x101;
/** Transaction conflicts with a transaction already known */
static const unsigned int REJECT_CONFLICT = 0x102;
+/** Transaction would result in too long in-mempool chain */
+static const unsigned int REJECT_LONGCHAIN = 0x103;

This comment has been minimized.

@sdaftuar

sdaftuar Sep 14, 2015

Member

Will move this to not be an internal code (so we send a reject message back).

@sdaftuar

sdaftuar Sep 14, 2015

Member

Will move this to not be an internal code (so we send a reject message back).

+ // Calculate in-mempool ancestors, up to a limit.
+ CTxMemPool::setEntries setAncestors;
+ size_t nLimitAncestors = GetArg("-limitancestorcount", DEFAULT_ANCESTOR_LIMIT);
+ size_t nLimitAncestorSize = GetArg("-limitancestorsize", DEFAULT_ANCESTOR_SIZE_LIMIT)*1000;

This comment has been minimized.

@petertodd

petertodd Sep 14, 2015

Contributor

As the idea is the ancestor size is set such that the whole string of transactions could be included in a single block, we really want this to be a less than the max block size, as there's always overhead to consider. (e.g. the coinbase, soft-limit, etc)

I'd suggest we set this to 900KB for a 100KB buffer - plenty even in the case of p2pool's large max 50KB coinbase. Another option might be to set it based on the soft-limit - default 750KB - with another 100KB of overhad buffer.

@petertodd

petertodd Sep 14, 2015

Contributor

As the idea is the ancestor size is set such that the whole string of transactions could be included in a single block, we really want this to be a less than the max block size, as there's always overhead to consider. (e.g. the coinbase, soft-limit, etc)

I'd suggest we set this to 900KB for a 100KB buffer - plenty even in the case of p2pool's large max 50KB coinbase. Another option might be to set it based on the soft-limit - default 750KB - with another 100KB of overhad buffer.

@petertodd

This comment has been minimized.

Show comment
Hide comment
@petertodd

petertodd Sep 14, 2015

Contributor

In general, it'd be good to think about separating the reorg case entirely from the main mempool codebase. For instance, keep a simple linear list of reorged transactions, and after a reorg attempt to add them back the mempool one-by-one. This separate code could also handle cases where we might want to remine transactions that we otherwise wouldn't, as a "goodwill" gesture to reduce the impact of large reorgs.

Contributor

petertodd commented Sep 14, 2015

In general, it'd be good to think about separating the reorg case entirely from the main mempool codebase. For instance, keep a simple linear list of reorged transactions, and after a reorg attempt to add them back the mempool one-by-one. This separate code could also handle cases where we might want to remine transactions that we otherwise wouldn't, as a "goodwill" gesture to reduce the impact of large reorgs.

src/txmempool.cpp
+{
+ // Track the number of entries (outside setExclude) that we'd need to visit
+ // (will bail out if it exceeds maxDescendantsToVisit)
+ int nChildrenToVisit = 0;

This comment has been minimized.

@petertodd

petertodd Sep 14, 2015

Contributor

trailing whitespace

@petertodd

petertodd Sep 14, 2015

Contributor

trailing whitespace

@pstratem

This comment has been minimized.

Show comment
Hide comment
@pstratem

pstratem Sep 14, 2015

Contributor

Needs a comment defining ancestor/descendant.

Contributor

pstratem commented Sep 14, 2015

Needs a comment defining ancestor/descendant.

src/txmempool.cpp
+
+ while (!parentHashes.empty()) {
+ setAncestors.insert(parentHashes.begin(), parentHashes.end());
+ setEntries stageParentSet;

This comment has been minimized.

@petertodd

petertodd Sep 14, 2015

Contributor

trailing whitespace

@petertodd

petertodd Sep 14, 2015

Contributor

trailing whitespace

@morcos morcos referenced this pull request Sep 14, 2015

Closed

Simple mempool protect #6673

src/txmempool.cpp
+ errString = strprintf("too many unconfirmed ancestors [limit: %u]", limitAncestorCount);
+ return false;
+ }
+ }

This comment has been minimized.

@petertodd

petertodd Sep 14, 2015

Contributor

trailing whitespace

@petertodd

petertodd Sep 14, 2015

Contributor

trailing whitespace

src/txmempool.cpp
+ minerPolicyEstimator->removeTx(hash);
+}
+
+// Calculates descendants of hash that are not already in setDescendants, and adds to

This comment has been minimized.

@petertodd

petertodd Sep 14, 2015

Contributor

trailing whitespace

@petertodd

petertodd Sep 14, 2015

Contributor

trailing whitespace

src/txmempool.cpp
- innerUsage += it->second.DynamicMemoryUsage();
- const CTransaction& tx = it->second.GetTx();
+ checkTotal += it->GetTxSize();
+ innerUsage += it->DynamicMemoryUsage();

This comment has been minimized.

@petertodd

petertodd Sep 14, 2015

Contributor

trailing whitespace

@petertodd

petertodd Sep 14, 2015

Contributor

trailing whitespace

src/txmempool.h
@@ -138,6 +363,29 @@ class CTxMemPool
void ApplyDeltas(const uint256 hash, double &dPriorityDelta, CAmount &nFeeDelta);
void ClearPrioritisation(const uint256 hash);
+public:
+ void RemoveStaged(std::set<uint256>& stage);

This comment has been minimized.

@petertodd

petertodd Sep 14, 2015

Contributor

Need to document what RemoveStaged() is.

@petertodd

petertodd Sep 14, 2015

Contributor

Need to document what RemoveStaged() is.

src/txmempool.h
+ /** When adding transactions from a disconnected block back to the mempool,
+ * new mempool entries may have children in the mempool (which is generally
+ * not the case when otherwise adding transactions).
+ * UpdateTransactionsFromBlock will find child transactions and update the

This comment has been minimized.

@petertodd

petertodd Sep 14, 2015

Contributor

Would prefer if comments like this are written as "Name()" rather than just "Name" to make it clear what's a variable and what's a function.

@petertodd

petertodd Sep 14, 2015

Contributor

Would prefer if comments like this are written as "Name()" rather than just "Name" to make it clear what's a variable and what's a function.

+ /** UpdateForDescendants is used by UpdateTransactionsFromBlock to update
+ * the descendants for a single transaction that has been added to the
+ * mempool but may have child transactions in the mempool, eg during a
+ * chain reorg. setExclude is the set of descendant transactions in the

This comment has been minimized.

@petertodd

petertodd Sep 14, 2015

Contributor

Double-space after periods? Obvs you're actually Satoshi.

@petertodd

petertodd Sep 14, 2015

Contributor

Double-space after periods? Obvs you're actually Satoshi.

src/txmempool.h
+ * descendant state for each transaction in hashesToUpdate (excluding any
+ * child transactions present in hashesToUpdate, which are already accounted
+ * for).
+ */

This comment has been minimized.

@petertodd

petertodd Sep 14, 2015

Contributor

This comment doesn't make it clear whether or not transactions in hashesToUpdate are or are not already in the mempool. :)

@petertodd

petertodd Sep 14, 2015

Contributor

This comment doesn't make it clear whether or not transactions in hashesToUpdate are or are not already in the mempool. :)

@sipa

This comment has been minimized.

Show comment
Hide comment
@sipa

sipa Sep 14, 2015

Member

Concept ACK.

Member

sipa commented Sep 14, 2015

Concept ACK.

@petertodd

This comment has been minimized.

Show comment
Hide comment
@petertodd

petertodd Sep 14, 2015

Contributor

utACK, modulo nits.

Contributor

petertodd commented Sep 14, 2015

utACK, modulo nits.

@jonasschnelli

This comment has been minimized.

Show comment
Hide comment
@jonasschnelli

jonasschnelli Sep 14, 2015

Member

Concept ACK / code review ACK.
Passes gitian/osx/debian build/unit-tests/rpc-tests.

Member

jonasschnelli commented Sep 14, 2015

Concept ACK / code review ACK.
Passes gitian/osx/debian build/unit-tests/rpc-tests.

@petertodd

This comment has been minimized.

Show comment
Hide comment
@petertodd

petertodd Sep 14, 2015

Contributor

Tested with my FSS and Full RBF stress tests and -checkmempool, no unexpected failures. (this pull-req of course doesn't enable any RBF behavior, so replacements failed!)

Contributor

petertodd commented Sep 14, 2015

Tested with my FSS and Full RBF stress tests and -checkmempool, no unexpected failures. (this pull-req of course doesn't enable any RBF behavior, so replacements failed!)

@dcousens

This comment has been minimized.

Show comment
Hide comment
@dcousens

dcousens Sep 14, 2015

Contributor

concept ACK

Contributor

dcousens commented Sep 14, 2015

concept ACK

@@ -921,6 +921,17 @@ bool AcceptToMemoryPool(CTxMemPool& pool, CValidationState &state, const CTransa
REJECT_HIGHFEE, "absurdly-high-fee",
strprintf("%d > %d", nFees, ::minRelayTxFee.GetFee(nSize) * 10000));
+ // Calculate in-mempool ancestors, up to a limit.
+ CTxMemPool::setEntries setAncestors;
+ size_t nLimitAncestors = GetArg("-limitancestorcount", DEFAULT_ANCESTOR_LIMIT);

This comment has been minimized.

@dcousens

dcousens Sep 14, 2015

Contributor

Using GetArg (an application level function) in mempool code feels a bit odd.

@dcousens

dcousens Sep 14, 2015

Contributor

Using GetArg (an application level function) in mempool code feels a bit odd.

This comment has been minimized.

@sdaftuar

sdaftuar Sep 16, 2015

Member

I agree but is there a better way to do this? I think I'm following the common practice in the code base, though I look forward to a future refactoring when we have better encapsulation.

@sdaftuar

sdaftuar Sep 16, 2015

Member

I agree but is there a better way to do this? I think I'm following the common practice in the code base, though I look forward to a future refactoring when we have better encapsulation.

src/txmempool.cpp
+
+ setEntries entriesToAdd;
+ BOOST_FOREACH(const txiter cit, stageEntries) {
+ if (cit->IsDirty()) {

This comment has been minimized.

@dcousens

dcousens Sep 14, 2015

Contributor

Dirty implies something changed typically, perhaps instead change this to ShouldSkip?

@dcousens

dcousens Sep 14, 2015

Contributor

Dirty implies something changed typically, perhaps instead change this to ShouldSkip?

+ // include them, and update their setMemPoolParents to include this tx.
+ for (; iter != mapNextTx.end() && iter->first.hash == hash; ++iter) {
+ const uint256 &childHash = iter->second.ptx->GetHash();
+ txiter childIter = mapTx.find(childHash);

This comment has been minimized.

@dcousens

dcousens Sep 14, 2015

Contributor

Do we need to check childIter == mapTx.end()?

@dcousens

dcousens Sep 14, 2015

Contributor

Do we need to check childIter == mapTx.end()?

This comment has been minimized.

@sdaftuar

sdaftuar Sep 16, 2015

Member

I'll add an assert to make clear that is not possible.

@sdaftuar

sdaftuar Sep 16, 2015

Member

I'll add an assert to make clear that is not possible.

src/txmempool.cpp
+{
+ // For each entry, walk back all ancestors and decrement size associated with this
+ // transaction
+ uint64_t nNoLimit = std::numeric_limits<uint64_t>::max();

This comment has been minimized.

@dcousens

dcousens Sep 14, 2015

Contributor

const?

@dcousens

dcousens Sep 14, 2015

Contributor

const?

src/txmempool.cpp
+{
+ nCountWithDescendants=0;
+ nSizeWithDescendants=nTxSize;
+ nFeesWithDescendants=nFee;

This comment has been minimized.

@dcousens

dcousens Sep 14, 2015

Contributor

nit: Spacing between operators

@dcousens

dcousens Sep 14, 2015

Contributor

nit: Spacing between operators

src/txmempool.cpp
+ }
+ int64_t updateCount = (add ? 1 : -1);
+ int64_t updateSize = updateCount * it->GetTxSize();
+ CAmount updateFee = updateCount * it->GetFee();

This comment has been minimized.

@dcousens

dcousens Sep 14, 2015

Contributor

nit: const on these?

@dcousens

dcousens Sep 14, 2015

Contributor

nit: const on these?

src/txmempool.cpp
+// TODO: pass a txiter instead?
+void CTxMemPool::UpdateChildrenForRemoval(const uint256 &hash)
+{
+ txiter it = mapTx.find(hash);

This comment has been minimized.

@dcousens

dcousens Sep 14, 2015

Contributor

Check/assert it == mapTx.end()?

@dcousens

dcousens Sep 14, 2015

Contributor

Check/assert it == mapTx.end()?

src/txmempool.cpp
+ uint64_t nNoLimit = std::numeric_limits<uint64_t>::max();
+ BOOST_FOREACH(const uint256& removeHash, hashesToRemove) {
+ setEntries setAncestors;
+ const CTxMemPoolEntry &entry = *mapTx.find(removeHash);

This comment has been minimized.

@dcousens

dcousens Sep 14, 2015

Contributor

Check/assert it == mapTx.end(), especially before de-reference

@dcousens

dcousens Sep 14, 2015

Contributor

Check/assert it == mapTx.end(), especially before de-reference

src/txmempool.cpp
+ setDescendants.insert(stage.begin(), stage.end());
+ std::set<uint256> setNext;
+ BOOST_FOREACH(const uint256 &stagehash, stage) {
+ indexed_transaction_set::iterator it = mapTx.find(stagehash);

This comment has been minimized.

@dcousens

dcousens Sep 14, 2015

Contributor

Check/assert it == mapTx.end()?
Would only be possible on the first stagehash AFAIK

Also, const/const_iterator?

@dcousens

dcousens Sep 14, 2015

Contributor

Check/assert it == mapTx.end()?
Would only be possible on the first stagehash AFAIK

Also, const/const_iterator?

src/txmempool.cpp
+ setAllRemoves = txToRemove;
+ }
+ BOOST_FOREACH(const uint256& hash, setAllRemoves) {
+ removed.push_back(mapTx.find(hash)->GetTx());

This comment has been minimized.

@dcousens

dcousens Sep 14, 2015

Contributor

Again, assumption of hash in mapTx, worth asserting?

@dcousens

dcousens Sep 14, 2015

Contributor

Again, assumption of hash in mapTx, worth asserting?

+ int64_t childSizes = 0;
+ CAmount childFees = 0;
+ for (; iter != mapNextTx.end() && iter->first.hash == it->GetTx().GetHash(); ++iter) {
+ txiter childit = mapTx.find(iter->second.ptx->GetHash());

This comment has been minimized.

@dcousens

dcousens Sep 14, 2015

Contributor

Again, find not checked, if this assumption can hold true, can it be documented?

@dcousens

dcousens Sep 14, 2015

Contributor

Again, find not checked, if this assumption can hold true, can it be documented?

This comment has been minimized.

@sdaftuar

sdaftuar Sep 15, 2015

Member

mapNextTx contains exactly the in-mempool children of a given transaction, so this code should be safe. I'll add an assert to make the assumption clearer (the semantics of mapNextTx weren't changed by this pull).

@sdaftuar

sdaftuar Sep 15, 2015

Member

mapNextTx contains exactly the in-mempool children of a given transaction, so this code should be safe. I'll add an assert to make the assumption clearer (the semantics of mapNextTx weren't changed by this pull).

src/txmempool.cpp
+ checkTotal += it->GetTxSize();
+ innerUsage += it->DynamicMemoryUsage();
+ const CTransaction& tx = it->GetTx();
+ const TxLinks &links = mapLinks.find(it)->second;

This comment has been minimized.

@dcousens

dcousens Sep 14, 2015

Contributor

Is mapLinks in perfect sync with mapTx?

@dcousens

dcousens Sep 14, 2015

Contributor

Is mapLinks in perfect sync with mapTx?

This comment has been minimized.

@sdaftuar

sdaftuar Sep 15, 2015

Member

Yes -- I'll be adding some additional asserts to make this more clear.

@sdaftuar

sdaftuar Sep 15, 2015

Member

Yes -- I'll be adding some additional asserts to make this more clear.

@sdaftuar

This comment has been minimized.

Show comment
Hide comment
@sdaftuar

sdaftuar Sep 16, 2015

Member

Thanks everyone for reviewing! I've pushed up a series of commits to address everyone's feedback; I believe all comments should have been addressed.

These cleanups need to be squashed, and this pull now needs to be rebased (a merge conflict crept in now that I'm outputting additional information in getrawmempool). Reviewers -- please let me know if you prefer I leave these commits unsquashed/unrebased so you can review the changes. (In the absence of any expressed preferences, I'll plan to squash/rebase in a day or two so that this can become mergeable.)

@petertodd I added the extra information to the getrawmempool RPC call (good idea!). Now that the RPC call is there, I realized that adding an RPC test that exercises the new limits is a good idea, so I've started work on that, but it's not yet complete.

Member

sdaftuar commented Sep 16, 2015

Thanks everyone for reviewing! I've pushed up a series of commits to address everyone's feedback; I believe all comments should have been addressed.

These cleanups need to be squashed, and this pull now needs to be rebased (a merge conflict crept in now that I'm outputting additional information in getrawmempool). Reviewers -- please let me know if you prefer I leave these commits unsquashed/unrebased so you can review the changes. (In the absence of any expressed preferences, I'll plan to squash/rebase in a day or two so that this can become mergeable.)

@petertodd I added the extra information to the getrawmempool RPC call (good idea!). Now that the RPC call is there, I realized that adding an RPC test that exercises the new limits is a good idea, so I've started work on that, but it's not yet complete.

@dcousens

This comment has been minimized.

Show comment
Hide comment
@dcousens

dcousens Sep 16, 2015

Contributor

utACK, thanks @sdaftuar!

edit: wait up, might need to re-evaluate the policy options in regards to comments in #6403

Contributor

dcousens commented Sep 16, 2015

utACK, thanks @sdaftuar!

edit: wait up, might need to re-evaluate the policy options in regards to comments in #6403

@morcos

View changes

src/txmempool.h
- std::map<uint256, CTxMemPoolEntry> mapTx;
+ indexed_transaction_set mapTx;
+ typedef indexed_transaction_set::iterator txiter;
+ struct CompareIteratorByHash {

This comment has been minimized.

@morcos

morcos Sep 16, 2015

Member

Can all these new members from here through to UpdateChild be made private?

@morcos

morcos Sep 16, 2015

Member

Can all these new members from here through to UpdateChild be made private?

+}
+
+bool CTxMemPool::addUnchecked(const uint256&hash, const CTxMemPoolEntry &entry, bool fCurrentEstimate)
+{

This comment has been minimized.

@morcos

morcos Sep 16, 2015

Member

needs LOCK(cs)?

@morcos

morcos Sep 16, 2015

Member

needs LOCK(cs)?

+ return memusage::MallocUsage(sizeof(CTxMemPoolEntry) + 9 * sizeof(void*)) * mapTx.size() + memusage::DynamicUsage(mapNextTx) + memusage::DynamicUsage(mapDeltas) + memusage::DynamicUsage(mapLinks) + cachedInnerUsage;
+}
+
+void CTxMemPool::RemoveStaged(setEntries &stage) {

This comment has been minimized.

@morcos

morcos Sep 16, 2015

Member

Needs LOCK(cs) if ever called from somewhere else.

@morcos

morcos Sep 16, 2015

Member

Needs LOCK(cs) if ever called from somewhere else.

This comment has been minimized.

@TheBlueMatt

TheBlueMatt Sep 19, 2015

Contributor

You can also AssertLockHeld(cs), otherwise, to make things more readable.

On September 16, 2015 11:05:34 AM CDT, Alex Morcos notifications@github.com wrote:

@@ -429,5 +758,58 @@ bool CCoinsViewMemPool::HaveCoins(const uint256
&txid) const {

size_t CTxMemPool::DynamicMemoryUsage() const {
LOCK(cs);

  • return memusage::DynamicUsage(mapTx) +
    memusage::DynamicUsage(mapNextTx) + memusage::DynamicUsage(mapDeltas) +
    cachedInnerUsage;
  • // Estimate the overhead of mapTx to be 9 pointers + an
    allocation, as no exact formula for boost::multi_index_contained is
    implemented.
  • return memusage::MallocUsage(sizeof(CTxMemPoolEntry) + 9 *
    sizeof(void*)) * mapTx.size() + memusage::DynamicUsage(mapNextTx) +
    memusage::DynamicUsage(mapDeltas) + memusage::DynamicUsage(mapLinks) +
    cachedInnerUsage;
    +}

+void CTxMemPool::RemoveStaged(setEntries &stage) {

Needs LOCK(cs) if ever called from somewhere else.


Reply to this email directly or view it on GitHub:
https://github.com/bitcoin/bitcoin/pull/6654/files#r39649424

@TheBlueMatt

TheBlueMatt Sep 19, 2015

Contributor

You can also AssertLockHeld(cs), otherwise, to make things more readable.

On September 16, 2015 11:05:34 AM CDT, Alex Morcos notifications@github.com wrote:

@@ -429,5 +758,58 @@ bool CCoinsViewMemPool::HaveCoins(const uint256
&txid) const {

size_t CTxMemPool::DynamicMemoryUsage() const {
LOCK(cs);

  • return memusage::DynamicUsage(mapTx) +
    memusage::DynamicUsage(mapNextTx) + memusage::DynamicUsage(mapDeltas) +
    cachedInnerUsage;
  • // Estimate the overhead of mapTx to be 9 pointers + an
    allocation, as no exact formula for boost::multi_index_contained is
    implemented.
  • return memusage::MallocUsage(sizeof(CTxMemPoolEntry) + 9 *
    sizeof(void*)) * mapTx.size() + memusage::DynamicUsage(mapNextTx) +
    memusage::DynamicUsage(mapDeltas) + memusage::DynamicUsage(mapLinks) +
    cachedInnerUsage;
    +}

+void CTxMemPool::RemoveStaged(setEntries &stage) {

Needs LOCK(cs) if ever called from somewhere else.


Reply to this email directly or view it on GitHub:
https://github.com/bitcoin/bitcoin/pull/6654/files#r39649424

+// for each entry, look for descendants that are outside hashesToUpdate, and
+// add fee/size information for such descendants to the parent.
+void CTxMemPool::UpdateTransactionsFromBlock(const std::vector<uint256> &vHashesToUpdate)
+{

This comment has been minimized.

@morcos

morcos Sep 16, 2015

Member

Needs LOCK(cs)

@morcos

morcos Sep 16, 2015

Member

Needs LOCK(cs)

+ return a->GetTx().GetHash() < b->GetTx().GetHash();
+ }
+ };
+ typedef std::set<txiter, CompareIteratorByHash> setEntries;

This comment has been minimized.

@morcos

morcos Sep 16, 2015

Member

oops, setEntries needs to be public, the rest can be private though.

@morcos

morcos Sep 16, 2015

Member

oops, setEntries needs to be public, the rest can be private though.

@TheBlueMatt

This comment has been minimized.

Show comment
Hide comment
@TheBlueMatt

TheBlueMatt Sep 19, 2015

Contributor

My computer's battery died, but in txmempool.h, the definition of txiter as "boost::multi_index_container::iterator" is "implementation defined" according to boost's docs. In the latest versions it is defined to an equivalent of "boost::multi_index_container::nth_index<0>::type::iterator", so you should use that.

On September 16, 2015 8:44:26 AM CDT, Suhas Daftuar notifications@github.com wrote:

Thanks everyone for reviewing! I've pushed up a series of commits to
address everyone's feedback; I believe all comments should have been
addressed.

These cleanups need to be squashed, and this pull now needs to be
rebased (a merge conflict crept in now that I'm outputting additional
information in getrawmempool). Reviewers -- please let me know if you
prefer I leave these commits unsquashed/unrebased so you can review the
changes. (In the absence of any expressed preferences, I'll plan to
squash/rebase in a day or two so that this can become mergeable.)

@petertodd I added the extra information to the getrawmempool RPC call
(good idea!). Now that the RPC call is there, I realized that adding
an RPC test that exercises the new limits is a good idea, so I've
started work on that, but it's not yet complete.


Reply to this email directly or view it on GitHub:
#6654 (comment)

Contributor

TheBlueMatt commented Sep 19, 2015

My computer's battery died, but in txmempool.h, the definition of txiter as "boost::multi_index_container::iterator" is "implementation defined" according to boost's docs. In the latest versions it is defined to an equivalent of "boost::multi_index_container::nth_index<0>::type::iterator", so you should use that.

On September 16, 2015 8:44:26 AM CDT, Suhas Daftuar notifications@github.com wrote:

Thanks everyone for reviewing! I've pushed up a series of commits to
address everyone's feedback; I believe all comments should have been
addressed.

These cleanups need to be squashed, and this pull now needs to be
rebased (a merge conflict crept in now that I'm outputting additional
information in getrawmempool). Reviewers -- please let me know if you
prefer I leave these commits unsquashed/unrebased so you can review the
changes. (In the absence of any expressed preferences, I'll plan to
squash/rebase in a day or two so that this can become mergeable.)

@petertodd I added the extra information to the getrawmempool RPC call
(good idea!). Now that the RPC call is there, I realized that adding
an RPC test that exercises the new limits is a good idea, so I've
started work on that, but it's not yet complete.


Reply to this email directly or view it on GitHub:
#6654 (comment)

@dcousens

This comment has been minimized.

Show comment
Hide comment
@dcousens

dcousens Sep 19, 2015

Contributor

@sdaftuar needs rebase.

Contributor

dcousens commented Sep 19, 2015

@sdaftuar needs rebase.

ashleyholman and others added some commits Jun 24, 2015

TxMemPool: Change mapTx to a boost::multi_index_container
Indexes on:
- Tx Hash
- Fee Rate (fee-per-kb)
Track transaction packages in CTxMemPoolEntry
Associate with each CTxMemPoolEntry all the size/fees of descendant
mempool transactions.  Sort mempool by max(feerate of entry, feerate
of descendants).  Update statistics on-the-fly as transactions enter
or leave the mempool.

Also add ancestor and descendant limiting, so that transactions can
be rejected if the number or size of unconfirmed ancestors exceeds
a target, or if adding a transaction would cause some other mempool
entry to have too many (or too large) a set of unconfirmed in-
mempool descendants.
@sdaftuar

This comment has been minimized.

Show comment
Hide comment
@sdaftuar

sdaftuar Sep 19, 2015

Member

I addressed the latest comments from @morcos and @TheBlueMatt, added an rpc test (it only tests the ancestor/descendant length limits, not the size limits, nor the handling of reorgs -- so that still can be improved), and then squashed everything down and rebased on master to get rid of the merge conflict in rpcblockchain.cpp.

Member

sdaftuar commented Sep 19, 2015

I addressed the latest comments from @morcos and @TheBlueMatt, added an rpc test (it only tests the ancestor/descendant length limits, not the size limits, nor the handling of reorgs -- so that still can be improved), and then squashed everything down and rebased on master to get rid of the merge conflict in rpcblockchain.cpp.

@Mirobit

This comment has been minimized.

Show comment
Hide comment
@Mirobit

Mirobit Sep 19, 2015

Contributor

Does anyone else think the ancestor limit is way to generous? In what other case than spamming do txs have 100 ancestors in mempool? Shouldn't 5 be enough?

Contributor

Mirobit commented Sep 19, 2015

Does anyone else think the ancestor limit is way to generous? In what other case than spamming do txs have 100 ancestors in mempool? Shouldn't 5 be enough?

@petertodd

This comment has been minimized.

Show comment
Hide comment
@petertodd

petertodd Sep 20, 2015

Contributor

@Mirobit The limits are there to limit computational costs in determining things like dependent fees/size etc. There's no need to set them low unless the algorithms take a long time to compute those sums; the data structures in the mempool are mainly pointer following so it's fairly fast.

Contributor

petertodd commented Sep 20, 2015

@Mirobit The limits are there to limit computational costs in determining things like dependent fees/size etc. There's no need to set them low unless the algorithms take a long time to compute those sums; the data structures in the mempool are mainly pointer following so it's fairly fast.

@laanwj

This comment has been minimized.

Show comment
Hide comment
@laanwj

laanwj Sep 21, 2015

Member

utACK

Member

laanwj commented Sep 21, 2015

utACK

@laanwj laanwj merged commit 5add7a7 into bitcoin:master Sep 21, 2015

1 check passed

continuous-integration/travis-ci/pr The Travis CI build passed
Details

laanwj added a commit that referenced this pull request Sep 21, 2015

Merge pull request #6654
5add7a7 Track transaction packages in CTxMemPoolEntry (Suhas Daftuar)
34628a1 TxMemPool: Change mapTx to a boost::multi_index_container (Ashley Holman)
@sipa

This comment has been minimized.

Show comment
Hide comment
@sipa

sipa Sep 22, 2015

Member

My node running 5add7a7 with -checkmempool crashed. Last debug.log message was:

2015-09-22 13:06:19 - Disconnect block: 90.04ms
Member

sipa commented Sep 22, 2015

My node running 5add7a7 with -checkmempool crashed. Last debug.log message was:

2015-09-22 13:06:19 - Disconnect block: 90.04ms
@morcos

This comment has been minimized.

Show comment
Hide comment
@morcos

morcos Sep 22, 2015

Member

@sipa can you give us some more details? were you running will all -debug options? for instance I assume you were not running with estimatefee debugging? it helps to narrow down where it crashed based on what didn't get printed to the debug log.

Member

morcos commented Sep 22, 2015

@sipa can you give us some more details? were you running will all -debug options? for instance I assume you were not running with estimatefee debugging? it helps to narrow down where it crashed based on what didn't get printed to the debug log.

@sdaftuar sdaftuar referenced this pull request Sep 23, 2015

Merged

Fix mempool packages #6715

@ruimarinho ruimarinho referenced this pull request in ruimarinho/bitcoin-core Feb 12, 2017

Closed

Fix camelcase consistency for mempool calls #20

zkbot added a commit to zcash/zcash that referenced this pull request Feb 20, 2018

Auto merge of #2940 - str4d:nu-activation-mempool-expiry, r=str4d
Mempool improvements, branch ID awareness

Whenever the local chain tip is updated, transactions in the mempool which commit to an
unmineable branch ID (for example, just before a network upgrade activates, where the
next block will have a different branch ID) will be removed.

Includes commits cherry-picked from the following upstream PRs:

- bitcoin/bitcoin#6654
  - Only the mempool index change.
- bitcoin/bitcoin#6776
- bitcoin/bitcoin#7020
- bitcoin/bitcoin#6915

Part of #2074.

zkbot added a commit to zcash/zcash that referenced this pull request Feb 20, 2018

Auto merge of #2940 - str4d:nu-activation-mempool-expiry, r=str4d
Mempool improvements, branch ID awareness

Whenever the local chain tip is updated, transactions in the mempool which commit to an
unmineable branch ID (for example, just before a network upgrade activates, where the
next block will have a different branch ID) will be removed.

Includes commits cherry-picked from the following upstream PRs:

- bitcoin/bitcoin#6654
  - Only the mempool index change.
- bitcoin/bitcoin#6776
- bitcoin/bitcoin#7020
- bitcoin/bitcoin#6915

Part of #2074.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment