[RPC] Expose ancestor/descendant information over RPC #7292

sdaftuar · Jan 4, 2016

This adds 3 new RPC calls (~~getancestors~~ getmempoolancestors, ~~getdescendants~~ getmempooldescendants, getmempoolentry) to expose more mempool information over RPC.

Now that we have policy rules that are tied to transaction chains (including limits on the number of in-mempool ancestors and in-mempool descendants, and mempool eviction and RBF policies that depend on fees of descendants), it seems helpful to be able to expose more information about the chains a given tx is part of over RPC.

This is motivated by the discussion in #7222. @petertodd You mentioned wanting to see specific use-cases for this information; I think there are clear improvements to the fee-bumping tools in your replace-by-fee repo that could be made if we had these functions (the first two tools I looked at just now don't seem to consider transaction chains at all in fee calculations?). I'll try to post a link in this PR to specific proposed improvements when I have something to share that will demonstrate this more clearly.

btcdrak · Jan 5, 2016

Concept ACK

jonasschnelli · Jan 5, 2016

Concept ACK.
Some concrete use-cases would be nice.
Nice to see the new RPC commands cs_main lock free.

sdaftuar · Jan 5, 2016

Updated to remove dead code I inadvertently left in rpcblockchain.cpp.

sdaftuar · Jan 6, 2016

Here's an example of a tool that would benefit from the getdescendants function:
https://github.com/sdaftuar/replace-by-fee-tools/blob/d0121b03d02c1a7e5e16425b55da1302ddf081aa/combine-transactions.py

This is a script that takes two txid's (that are signaling opt-in RBF) and combines them down to one -- consolidating duplicate output addresses (if any) and removing pay-to-self outputs, assumed for demo purposes to be change outputs -- and paying for all fees including those of descendants. The main idea is that this avoids over-paying for descendants -- it calls getdescendants on each transaction and takes the union of the two sets so that each descendant is paid for once. I don't know any simple or straightforward way to do this without this new RPC call.

There are similar benefits to having getancestors, such as if you wanted to understand why a stuck transaction wasn't confirming, then getancestors would give you the set of unconfirmed parents to look at (and you could write a replace-by-fee-tool that was smart enough to drop an input coming from a stuck unconfirmed parent, when possible, to replace a transaction with one likely to confirm sooner; or better still, replace such a stuck parent instead if possible).

sipa · Jan 6, 2016

Ultra nit before reading the code: it's not obvious from the RPC names that it's about the mempool.

sdaftuar · Jan 7, 2016

@sipa True. Perhaps getmempoolancestors/getmempooldescendants? I'm open to suggestions.

@laanwj if this code is ok I think it would make sense to include this in 0.12

sdaftuar · Jan 12, 2016

Updated with new RPC names (getmempoolancestors and getmempooldescendants).

luke-jr · Jan 15, 2016

Any reason not to just add the ancestors/descendants info to getrawmempool (and also the getmempoolentry)?

sdaftuar · Jan 15, 2016

The goal was to avoid having to dump the entire mempool if you were only interested in a very small subset of it. But I suppose there's no reason we couldn't also show the txid's of ancestors and descendants when you call getrawmempool (with verbose=true). If that seems useful, I'm happy to add that.

(edited)
Oh, right, you are suggesting that, and then yes I agree it would also make sense to update getmempoolentry to match. Sure, I'll go ahead and do that.

sdaftuar · Jan 18, 2016

@luke-jr On further thought, the downside to that is that doing the calculation could cause getrawmempool to get up to ~25x slower (default policy allows length 25 ancestor and descendant chains, and the calculation of ancestors and descendants requires walking those chains), which on a full mempool could be a meaningful -- and perhaps unacceptable? -- slowdown.

The other possibility would be to add a new option to getrawmempool to determine whether to do the calculation or not, but I think we try to avoid adding new arguments to existing RPC calls? So now I think it's probably better to leave as-is. Let me know if you disagree...

laanwj · Jan 22, 2016

Concept ACK

laanwj · Feb 3, 2016

Needs rebase.

sdaftuar · Feb 11, 2016

Rebased. There weren't any actual conflicts though, so not sure why github thought it wouldn't merge cleanly?

laanwj · Feb 12, 2016

Thanks!
I've noticed that before. Github is somewhat less good (or "more careful") at merging than git.

sipa · May 17, 2016

Rebase needed.

sdaftuar · May 17, 2016

Rebased. Also added a commit that extends the mempool entry output to include the ancestor state (added by #7594), as well as a brief update to the release notes.

nit: missing four whitespace before (

jonasschnelli · May 17, 2016

I don't know why Travis WIN32 fails on the unit tests (sadly, It seems that the unit-test log is missing in the travis dump).

mempool_packages.py needs to be added to the rpc-tests.py script.

Tested ACK 69c9553586befdbc9ca6a474ae294bebe998163b modulo nits.

sdaftuar · May 17, 2016

Fixed @jonasschnelli's nits. No idea how these changes could affect the unit tests on any platform; the travis failure seems disturbing...

json array of string_s_

paveljanik · May 17, 2016

ACK 170bfea

sdaftuar · May 18, 2016

Pushed a fix to address @paveljanik's nit

sdaftuar · May 18, 2016

Unit tests are now failing on linux. I'll see if I can reproduce locally.

sdaftuar · May 18, 2016

Oh, I suppose it could be this: #8069 (comment)

sdaftuar · May 20, 2016

Looks like the travis issue was fixed by #8070, so I think this is ready now.

paveljanik · May 20, 2016

reACK 5379307

MarcoFalke · May 20, 2016

utACK 5379307

arowser · May 25, 2016

Can one of the admins verify this patch?

sipa · Jun 3, 2016

utACK 5379307a23177c33b5a869513d1d1d09790b0269

sdaftuar · Jun 9, 2016

Is there anything holding this up?

sipa · Jun 9, 2016

Can you squash the typo?

sdaftuar · Jun 9, 2016

@sipa done

sipa · Jun 9, 2016

utACK 176e19b (same tree as before)

sdaftuar referenced this pull request Jan 4, 2016
Merged
RPC: Indicate which transactions are signaling opt-in RBF #7222

jonasschnelli added the RPC/REST/ZMQ label Jan 5, 2016

luke-jr added a commit to luke-jr/bitcoin that referenced this pull request Feb 13, 2016

Refactor logic for converting mempool entries to JSON
Github-Pull: #7292 Rebased-From: bf4fbe814df537e69d5b1092565efb6f8bb618b9

85b0e45

luke-jr added a commit to luke-jr/bitcoin that referenced this pull request Feb 13, 2016

Add getmempoolancestors RPC call
Github-Pull: #7292 Rebased-From: 2409317c8ba1ea11b0bb519121f8c441439a71db

7ce4ee6

luke-jr added a commit to luke-jr/bitcoin that referenced this pull request Feb 13, 2016

Add getmempooldescendants RPC call
Github-Pull: #7292 Rebased-From: 9644106526eea536cf056b4058a40b0dbbdbe32f

dc7641d

luke-jr added a commit to luke-jr/bitcoin that referenced this pull request Feb 13, 2016

Add getmempoolentry RPC call
Github-Pull: #7292 Rebased-From: 31ef9d5c649beea8812d2ba92f779d639e1c8b2c

9d557ea

luke-jr added a commit to luke-jr/bitcoin that referenced this pull request Feb 13, 2016

Add test coverage for new RPC calls
Github-Pull: #7292 Rebased-From: bc9020c1ab30ea9527ef63d4df4df10513f8756f

24c954c

sdaftuar referenced this pull request Feb 24, 2016
Merged
Mempool: Add tracking of ancestor packages #7594

Refactor logic for converting mempool entries to JSON

5ec0cde

Add getmempoolancestors RPC call

8f7b5dc

sdaftuar added some commits Jan 4, 2016

Add getmempooldescendants RPC call

0dfd869

Add getmempoolentry RPC call

b09b813

Add test coverage for new RPC calls

a9b8390

Add ancestor statistics to mempool entry RPC output

7f6eda8

Mention new RPC's in release notes

176e19b

sipa merged commit 176e19b into bitcoin:master Jun 9, 2016
1 check was pending

1 check was pending

continuous-integration/travis-ci/pr The Travis CI build is in progress
Details

luke-jr added a commit to bitcoinknots/bitcoin that referenced this pull request Jun 28, 2016

Merge #7292 rpc_getmempoolentry

6a325a1

sdaftuar referenced this pull request May 12, 2017
Open
Add "effective fee rate" in rpc's mempool entries #10389

+    +        return o;
+    +    }
+    +}
+    +
+    +UniValue getmempooldescendants(const UniValue& params, bool fHelp)
+    +{
+    +    if (fHelp || params.size() < 1 || params.size() > 2) {
+    +        throw runtime_error(
+    +            "getmempooldescendants txid (verbose)\n"
+    +            "\nIf txid is in the mempool, returns all in-mempool descendants.\n"
+    +            "\nArguments:\n"
+    +            "1. \"txid\"                   (string, required) The transaction id (must be in mempool)\n"
+    +            "2. verbose                  (boolean, optional, default=false) true for a json object, false for array of transaction ids\n"
+    +            "\nResult (for verbose=false):\n"
+    +            "[                       (json array of string)\n"
+    +            "  \"transactionid\"       (string) The transaction id of an in-mempool descendant transaction\n"

@@ -292,6 +292,70 @@ UniValue getrawmempool(const UniValue& params, bool fHelp)
          return mempoolToJSON(fVerbose);
      }
+    +UniValue getmempoolancestors(const UniValue& params, bool fHelp)
+    +{
+    +    if (fHelp || params.size() < 1 || params.size() > 2) {
+    +        throw runtime_error(
+    +            "getmempoolancestors txid (verbose)\n"
+    +            "\nIf txid is in the mempool, returns all in-mempool ancestors.\n"
+    +            "\nArguments:\n"
+    +            "1. \"txid\"                   (string, required) The transaction id (must be in mempool)\n"
+    +            "2. verbose                  (boolean, optional, default=false) true for a json object, false for array of transaction ids\n"
+    +            "\nResult (for verbose=false):\n"
+    +            "[                       (json array of string)\n"

bitcoin/bitcoin

Join GitHub today

[RPC] Expose ancestor/descendant information over RPC #7292

Conversation

sdaftuar commented Jan 4, 2016

sdaftuar referenced this pull request Jan 4, 2016

RPC: Indicate which transactions are signaling opt-in RBF #7222

btcdrak commented Jan 5, 2016

jonasschnelli commented Jan 5, 2016

jonasschnelli added the RPC/REST/ZMQ label Jan 5, 2016

sdaftuar commented Jan 5, 2016

sdaftuar commented Jan 6, 2016

sipa commented Jan 6, 2016

sdaftuar commented Jan 7, 2016

sdaftuar commented Jan 12, 2016

luke-jr commented Jan 15, 2016

sdaftuar commented Jan 15, 2016

sdaftuar commented Jan 18, 2016

laanwj commented Jan 22, 2016

laanwj commented Feb 3, 2016

sdaftuar commented Feb 11, 2016

laanwj commented Feb 12, 2016

luke-jr added a commit to luke-jr/bitcoin that referenced this pull request Feb 13, 2016

luke-jr added a commit to luke-jr/bitcoin that referenced this pull request Feb 13, 2016

luke-jr added a commit to luke-jr/bitcoin that referenced this pull request Feb 13, 2016

luke-jr added a commit to luke-jr/bitcoin that referenced this pull request Feb 13, 2016

luke-jr added a commit to luke-jr/bitcoin that referenced this pull request Feb 13, 2016

sdaftuar referenced this pull request Feb 24, 2016

Mempool: Add tracking of ancestor packages #7594

sipa commented May 17, 2016

sdaftuar commented May 17, 2016

Show 1 comment Hide comment jonasschnelli commented on an outdated diff May 17, 2016

jonasschnelli May 17, 2016

jonasschnelli commented May 17, 2016

sdaftuar commented May 17, 2016

Show 1 comment Hide comment paveljanik commented on an outdated diff May 17, 2016

paveljanik May 17, 2016

paveljanik commented May 17, 2016

sdaftuar commented May 18, 2016

sdaftuar commented May 18, 2016

sdaftuar commented May 18, 2016

sdaftuar commented May 20, 2016

paveljanik commented May 20, 2016

MarcoFalke commented May 20, 2016

arowser commented May 25, 2016

sipa commented Jun 3, 2016

sdaftuar commented Jun 9, 2016

sipa commented Jun 9, 2016

sdaftuar added some commits Jan 4, 2016

sdaftuar commented Jun 9, 2016

sipa commented Jun 9, 2016

Hide details View details sipa merged commit 176e19b into bitcoin:master Jun 9, 2016 1 check was pending

1 check was pending

sipa added a commit that referenced this pull request Jun 9, 2016

luke-jr added a commit to bitcoinknots/bitcoin that referenced this pull request Jun 28, 2016

sdaftuar referenced this pull request May 12, 2017

Add "effective fee rate" in rpc's mempool entries #10389

codablock added a commit to codablock/dash that referenced this pull request Sep 16, 2017

codablock added a commit to codablock/dash that referenced this pull request Sep 19, 2017

jonasschnelli commented on an outdated diff May 17, 2016

paveljanik commented on an outdated diff May 17, 2016

sipa merged commit `176e19b` into bitcoin:master Jun 9, 2016
1 check was pending