Prefer coins that have fewer ancestors, sanity check txn before ATMP #9262

instagibbs · 2016-12-02T04:09:35Z

This does a few extra possible runs of SelectCoinsMinConf, each with larger allowable mempool ancestor numbers, up to the acceptable mempool limit. As long as each input is not above this value, it passes. This means the sum of the history could actually be larger.

This is why I then catch the transaction at the end of CreateTransaction, and directly check. It simply fails, and does not retry. This logic is gated by a new command line argument -rejectlongwalletchains to regain previous behavior. We can most likely remove this check once we have better rebroadcasting systems.

sipa · 2016-12-02T04:58:40Z

There should probably be a function in the place where mempool acceptance is decided (main.cpp now, validation.cpp after The Main Split) to do preliminary checking like thid.

gmaxwell · 2016-12-02T14:52:54Z

Probably not for now but when I read this code I wondered if we need an "available balance"-- e.g. the sum of the coins that selectcoins would consider using... so that transaction failures due to this (and spendunconfirmed set off) are more explicable.

gmaxwell · 2016-12-02T14:53:27Z

FWIW, I think this should be backported.

instagibbs · 2016-12-02T16:45:04Z

While writing tests I found that this logic somehow is falling through on non-default ancestorcount limit. Debugging.

sdaftuar · 2016-12-02T17:00:47Z

src/wallet/wallet.cpp

@@ -1957,6 +1957,20 @@ void CWallet::AvailableCoins(vector<COutput>& vCoins, bool fOnlyConfirmed, const
            if (nDepth == 0 && !pcoin->InMempool())
                continue;

+            LockPoints lp;
+            CTxMemPoolEntry entry(*pcoin, 0, 0, 0, 0, false, 0, false, 0, lp);


Since we've already checked that pcoin is in the mempool, we don't need to do any of this -- we can just use the cached values. (I think the only way the calculation you do below can fail is if the limits are violated during a reorg.)

instagibbs · 2016-12-02T21:02:11Z

Updated the pull request to new design discussed on IRC.

sdaftuar

Code review ack, just a couple nits. Will test.

sdaftuar · 2016-12-02T21:28:52Z

src/wallet/wallet.h

@@ -400,6 +400,7 @@ class CWalletTx : public CMerkleTx
    bool IsEquivalentTo(const CWalletTx& tx) const;

    bool InMempool() const;
+    uint64_t GetMempoolAncestorCount() const;


nit: add documentation to this function that the tx must be in the mempool in order to call?

sdaftuar · 2016-12-02T21:33:15Z

src/wallet/wallet.cpp

@@ -2042,6 +2048,9 @@ bool CWallet::SelectCoinsMinConf(const CAmount& nTargetValue, int nConfMine, int
        if (output.nDepth < (pcoin->IsFromMe(ISMINE_ALL) ? nConfMine : nConfTheirs))
            continue;

+        if (pcoin->InMempool() && pcoin->GetMempoolAncestorCount() > nMaxAncestors)


Perhaps worth a comment here that we're assuming AvailableCoins() has already filtered out any unconfirmed and not-in-mempool coins?

Inequality here is wrong, should be >= I believe. This will never fire as-is. I should be able to easily test this fix.

sdaftuar · 2016-12-02T21:36:51Z

src/wallet/wallet.cpp

@@ -2558,6 +2558,21 @@ bool CWallet::CreateTransaction(const vector<CRecipient>& vecSend, CWalletTx& wt
        }
    }

+    // Lastly, ensure it doesn't have too many ancestors


nit: perhaps this comment should be "Lastly, ensure this tx will pass the mempool's chain limits"

sdaftuar · 2016-12-02T21:38:55Z

src/wallet/wallet.cpp

+    size_t nLimitDescendantSize = GetArg("-limitdescendantsize", DEFAULT_DESCENDANT_SIZE_LIMIT)*1000;
+    std::string errString;
+    if (!mempool.CalculateMemPoolAncestors(entry, setAncestors, nLimitAncestors, nLimitAncestorSize, nLimitDescendants, nLimitDescendantSize, errString)) {
+        strFailReason = _("Transaction has too long of a mempool chain");


The errString filled in by CMPA might be useful for debugging issues (eg a LogPrintf() so that someone debugging could inspect), though i see we don't log any info about the transaction we tried to generate, so maybe not worth it...

instagibbs · 2016-12-02T22:18:15Z

I also think this is good for backport.

sdaftuar · 2016-12-02T22:24:32Z

src/wallet/wallet.cpp

@@ -2042,6 +2048,9 @@ bool CWallet::SelectCoinsMinConf(const CAmount& nTargetValue, int nConfMine, int
        if (output.nDepth < (pcoin->IsFromMe(ISMINE_ALL) ? nConfMine : nConfTheirs))
            continue;

+        if (pcoin->GetMempoolAncestorCount() > nMaxAncestors)


I think you need the pcoin->InMempool() check you had before, in case you're looking at in-chain coins?

gmaxwell · 2016-12-04T19:11:21Z

Looks good! -- so, I think the additional test in "Don't return success" should be controlled by an option, since the failure is only temporary (at least once we fix the rebroadcast bug) and many callers handle failure to create a transaction poorly. I might even go as far as to suggest that this option be set in 0.13.2 to continue to create the transaction because that would be less of an API change. Perhaps @sdaftuar or @laanwj has an opinion on that?

Other than that, utACK.

instagibbs · 2016-12-04T19:45:53Z

Couple of fixes/changes:

if (pcoin->InMempool() && pcoin->GetMempoolAncestorCount() > nMaxAncestors)

is now

if (pcoin->InMempool() && pcoin->GetMempoolAncestorCount() >= nMaxAncestors)

so it actually filters correctly on the very last call of SelectCoinsMinConf.

I also revamped the tests to catch the various cases.

@gmaxwell we'll have to catch that logic in two places, SelectCoinsMinConf(at the last instance of it) and at the end of CreateTransaction. I suppose once rebroadcasting is fixed, we can set the default to "false" and let it take care of it. I'll wait to see what others are thinking. I think current behavior is horrendous :)

gmaxwell · 2016-12-05T02:42:49Z

Current behavior is horrendous indeed. Thank you for working on this.

sdaftuar · 2016-12-05T19:59:22Z

@gmaxwell No strong opinion on the issue of default behavior for 0.13.2, but given that you currently have to restart your node in order to rebroadcast, I'd lean towards making the default behavior be to enforce this new restriction, rather than create a tx that doesn't make it to the mempool.

sdaftuar · 2016-12-05T20:35:05Z

src/wallet/wallet.cpp

@@ -2042,6 +2048,9 @@ bool CWallet::SelectCoinsMinConf(const CAmount& nTargetValue, int nConfMine, int
        if (output.nDepth < (pcoin->IsFromMe(ISMINE_ALL) ? nConfMine : nConfTheirs))
            continue;

+        if (pcoin->InMempool() && pcoin->GetMempoolAncestorCount() >= nMaxAncestors)


@morcos and I were discussing, I think this would make more sense to limit based on both ancestor and descendant count, something like:

if (pcoin->InMempool() && max(pcoin->GetMempoolAncestorCount(), pcoin->GetMempoolDescendantCount()) >= nMaxChain) continue;

Otherwise, if you have an unspent output with 1 ancestor but 25 descendants, you (needlessly) could fail on one of the early calls.

sdaftuar · 2016-12-05T20:39:53Z

src/wallet/wallet.cpp

+        SelectCoinsMinConf(nTargetValue - nValueFromPresetInputs, 1, 1, 0, vCoins, setCoinsRet, nValueRet) ||
+        (bSpendZeroConfChange && SelectCoinsMinConf(nTargetValue - nValueFromPresetInputs, 0, 1, nLimitAncestors/4, vCoins, setCoinsRet, nValueRet)) ||
+        (bSpendZeroConfChange && SelectCoinsMinConf(nTargetValue - nValueFromPresetInputs, 0, 1, nLimitAncestors/2, vCoins, setCoinsRet, nValueRet)) ||
+        (bSpendZeroConfChange && SelectCoinsMinConf(nTargetValue - nValueFromPresetInputs, 0, 1, nLimitAncestors, vCoins, setCoinsRet, nValueRet));


Instead of N/4, N/2, and N for the ancestor/chain limit passed in to SCMC, how about: 2, 4, N/2, N? (And set N = min(limitancestorcount, limitdescendantcount), assuming we go with ancestor and descendant limiting as I mentioned in another comment.) This way we first try utxo's from transactions that have no ancestors or descendants, which should provide a strong preference for not creating long chains.

instagibbs · 2016-12-05T21:59:20Z

@sdaftuar made sense to me, done

laanwj · 2016-12-06T06:09:05Z

but given that you currently have to restart your node in order to rebroadcast

Why is that so? You can't take the tx hex and send it through sendrawtransaction in this case?

sipa · 2016-12-19T08:02:15Z

utACK 5882c09, with one nit.

sipa · 2016-12-19T07:56:53Z

src/txmempool.cpp

+
+bool CTxMemPool::TransactionWithinChainLimit(const uint256& txid, size_t chainLimit) const {
+    LOCK(cs);
+    if (exists(txid) && std::max(mapTx.find(txid)->GetCountWithAncestors(), mapTx.find(txid)->GetCountWithDescendants()) >= chainLimit)


You're doing 3 map lookups here. What about

auto it = mapTx.find(txid); return it != mapTx.end() && it->GetCountWithAncestors() < chainLimit && it->GetCountWithDescendants() < chainLimit;

instagibbs · 2016-12-19T14:51:36Z

fixed @sipa nit

laanwj · 2016-12-19T15:47:30Z

Travis doesn't agree with your change, it seems.

instagibbs · 2016-12-19T15:55:23Z

@laanwj yes the logic made no sense, it will reject anything not in mempool rather than anything in mempool with too long a chain. Fixed.

TheBlueMatt

utACK ff79329ccfe3ee4387b9de72d82431c7b8132272 some comments but none worth delaying 0.13.2 for.

TheBlueMatt · 2016-12-19T21:28:19Z

src/wallet/wallet.h

@@ -685,7 +687,7 @@ class CWallet : public CCryptoKeyStore, public CValidationInterface
     * completion the coin set and corresponding actual target value is
     * assembled
     */
-    bool SelectCoinsMinConf(const CAmount& nTargetValue, int nConfMine, int nConfTheirs, std::vector<COutput> vCoins, std::set<std::pair<const CWalletTx*,unsigned int> >& setCoinsRet, CAmount& nValueRet) const;
+    bool SelectCoinsMinConf(const CAmount& nTargetValue, int nConfMine, int nConfTheirs, uint64_t nMaxAncestors, std::vector<COutput> vCoins, std::set<std::pair<const CWalletTx*,unsigned int> >& setCoinsRet, CAmount& nValueRet) const;


nit: you forgot to const-ify the first two ints which you did in the .cpp

TheBlueMatt · 2016-12-19T21:46:07Z

src/wallet/wallet.cpp

@@ -2545,6 +2567,21 @@ bool CWallet::CreateTransaction(const vector<CRecipient>& vecSend, CWalletTx& wt
        }
    }

+    if (GetBoolArg("-walletrejectlongchains", DEFAULT_WALLET_REJECT_LONG_CHAINS)) {
+        // Lastly, ensure this tx will pass the mempool's chain limits


Agreed, though in the interest of backporting and getting 0.13.2 out cleaing up the interface here probably isnt the priority. Lets do it in another pr.

TheBlueMatt · 2016-12-19T22:07:31Z

src/wallet/wallet.cpp

+    if (GetBoolArg("-walletrejectlongchains", DEFAULT_WALLET_REJECT_LONG_CHAINS)) {
+        // Lastly, ensure this tx will pass the mempool's chain limits
+        LockPoints lp;
+        CTxMemPoolEntry entry(txNew, 0, 0, 0, 0, false, 0, false, 0, lp);


Note that by not filling in the sigOpsCost here we dont have that information for the virtual transaction size, so you could still run over the limit here. I don't think its a big deal, but we should fix when we clean things up in 0.14 as suggested two lines up.

gmaxwell · 2016-12-20T01:42:38Z

src/txmempool.cpp

-    return true;
+    auto it = mapTx.find(txid);
+    return it == mapTx.end() || (it->GetCountWithAncestors() < chainLimit &&
+    it->GetCountWithDescendants() < chainLimit);


This really should be indented.

… before ATMP cee1612 reduce number of lookups in TransactionWithinChainLimit (Gregory Sanders) af9bedb Test for fix of txn chaining in wallet (Gregory Sanders) 5882c09 CreateTransaction: Don't return success with too-many-ancestor txn (Gregory Sanders) 0b2294a SelectCoinsMinConf: Prefer coins with fewer ancestors (Gregory Sanders)

laanwj · 2016-12-20T12:41:31Z

Backported in #9382, removing label

…eck txn before ATMP cee1612 reduce number of lookups in TransactionWithinChainLimit (Gregory Sanders) af9bedb Test for fix of txn chaining in wallet (Gregory Sanders) 5882c09 CreateTransaction: Don't return success with too-many-ancestor txn (Gregory Sanders) 0b2294a SelectCoinsMinConf: Prefer coins with fewer ancestors (Gregory Sanders)

fanquake added Wallet Mempool labels Dec 2, 2016

sdaftuar reviewed Dec 2, 2016

View reviewed changes

instagibbs force-pushed the toolong branch from a31ffdd to 542b899 Compare December 2, 2016 20:52

instagibbs changed the title ~~Don't consider coins available if too many ancestors in mempool~~ Prefer coins that have fewer ancestors, sanity check txn before ATMP Dec 2, 2016

instagibbs force-pushed the toolong branch from 542b899 to 551012e Compare December 2, 2016 20:58

sdaftuar reviewed Dec 2, 2016

View reviewed changes

instagibbs force-pushed the toolong branch 2 times, most recently from 4695526 to ede1939 Compare December 2, 2016 22:13

sdaftuar reviewed Dec 2, 2016

View reviewed changes

maflcko added this to the 0.13.2 milestone Dec 2, 2016

maflcko added the Needs backport label Dec 2, 2016

instagibbs force-pushed the toolong branch from ede1939 to 5516f93 Compare December 2, 2016 22:44

instagibbs force-pushed the toolong branch 2 times, most recently from 05ae82b to 1fc42a1 Compare December 4, 2016 19:39

sdaftuar reviewed Dec 5, 2016

View reviewed changes

instagibbs force-pushed the toolong branch from 1fc42a1 to 2da7734 Compare December 5, 2016 21:08

instagibbs added 2 commits December 13, 2016 09:41

CreateTransaction: Don't return success with too-many-ancestor txn

5882c09

Test for fix of txn chaining in wallet

af9bedb

instagibbs force-pushed the toolong branch from 383c085 to af9bedb Compare December 13, 2016 14:41

sipa reviewed Dec 19, 2016

View reviewed changes

instagibbs force-pushed the toolong branch from af5e312 to ff79329 Compare December 19, 2016 15:54

TheBlueMatt approved these changes Dec 19, 2016

View reviewed changes

btcdrak mentioned this pull request Dec 19, 2016

[backport 0.13] Prefer coins that have fewer ancestors, sanity check txn before ATMP #9382

Merged

gmaxwell reviewed Dec 20, 2016

View reviewed changes

reduce number of lookups in TransactionWithinChainLimit

cee1612

instagibbs force-pushed the toolong branch from ff79329 to cee1612 Compare December 20, 2016 01:51

btcdrak approved these changes Dec 20, 2016

View reviewed changes

laanwj merged commit cee1612 into bitcoin:master Dec 20, 2016

laanwj removed the Needs backport label Dec 20, 2016

TheBlueMatt mentioned this pull request Dec 20, 2016

Wallet ancestor sanity-check ignores sigops #9392

Closed

maflcko mentioned this pull request Dec 27, 2016

Avoid making chains of txn >25 deep. #9019

Closed

sipa mentioned this pull request Jan 10, 2017

TODO for release notes 0.14.0 #8455

Closed

18 tasks

jonasschnelli mentioned this pull request Feb 13, 2017

Max unconfirmed chainlength #9752

Closed

jonasschnelli mentioned this pull request Feb 28, 2017

FundRawTransaction select inputs which can be rejected by the mempool #9874

Closed

sdaftuar mentioned this pull request Mar 16, 2017

After max chain of unconfirmed change transactions, last tx is missing from memory until rescan #10004

Closed

jnewbery mentioned this pull request Mar 16, 2017

Wallet should reject long chains by default #10015

Closed

bitcoin locked as resolved and limited conversation to collaborators Sep 8, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Prefer coins that have fewer ancestors, sanity check txn before ATMP #9262

Prefer coins that have fewer ancestors, sanity check txn before ATMP #9262

instagibbs commented Dec 2, 2016 •

edited

Loading

sipa commented Dec 2, 2016

gmaxwell commented Dec 2, 2016

gmaxwell commented Dec 2, 2016

instagibbs commented Dec 2, 2016 •

edited

Loading

sdaftuar Dec 2, 2016

instagibbs commented Dec 2, 2016

sdaftuar left a comment

sdaftuar Dec 2, 2016

sdaftuar Dec 2, 2016

instagibbs Dec 2, 2016

sdaftuar Dec 2, 2016

sdaftuar Dec 2, 2016

instagibbs commented Dec 2, 2016

sdaftuar Dec 2, 2016

gmaxwell commented Dec 4, 2016

instagibbs commented Dec 4, 2016 •

edited

Loading

gmaxwell commented Dec 5, 2016

sdaftuar commented Dec 5, 2016

sdaftuar Dec 5, 2016

sdaftuar Dec 5, 2016

instagibbs commented Dec 5, 2016

laanwj commented Dec 6, 2016

sipa commented Dec 19, 2016

sipa Dec 19, 2016

instagibbs commented Dec 19, 2016

laanwj commented Dec 19, 2016

instagibbs commented Dec 19, 2016

TheBlueMatt left a comment

TheBlueMatt Dec 19, 2016

TheBlueMatt Dec 19, 2016

TheBlueMatt Dec 19, 2016

gmaxwell Dec 20, 2016

laanwj commented Dec 20, 2016

Prefer coins that have fewer ancestors, sanity check txn before ATMP #9262

Prefer coins that have fewer ancestors, sanity check txn before ATMP #9262

Conversation

instagibbs commented Dec 2, 2016 • edited Loading

sipa commented Dec 2, 2016

gmaxwell commented Dec 2, 2016

gmaxwell commented Dec 2, 2016

instagibbs commented Dec 2, 2016 • edited Loading

Choose a reason for hiding this comment

instagibbs commented Dec 2, 2016

sdaftuar left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

instagibbs commented Dec 2, 2016

Choose a reason for hiding this comment

gmaxwell commented Dec 4, 2016

instagibbs commented Dec 4, 2016 • edited Loading

gmaxwell commented Dec 5, 2016

sdaftuar commented Dec 5, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

instagibbs commented Dec 5, 2016

laanwj commented Dec 6, 2016

sipa commented Dec 19, 2016

Choose a reason for hiding this comment

instagibbs commented Dec 19, 2016

laanwj commented Dec 19, 2016

instagibbs commented Dec 19, 2016

TheBlueMatt left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

laanwj commented Dec 20, 2016

instagibbs commented Dec 2, 2016 •

edited

Loading

instagibbs commented Dec 2, 2016 •

edited

Loading

instagibbs commented Dec 4, 2016 •

edited

Loading