rpc: deprecate top-level fee fields in getmempool RPCs #22689

josibake · 2021-08-12T14:12:28Z

per #22682 , top level fee fields for mempool entries have been deprecated since 0.17 but are still returned. this PR properly deprecates them so that they are no longer returned unless -deprecatedrpc=fees is passed.

the first commit takes care of deprecation and also updates test/functional/mempool_packages.py to only use the fees object. the second commit adds a new functional test for -deprecatedrpc=fees

closes #22682

questions for the reviewer

-deprecatedrpc=fees made the most sense to me, but happy to change if there is a name that makes more sense
properly deprecate mempool entry RPC fee fields #22682 seems to indicate that after some period of time, the fields will be removed all together. if we have a rough idea of when this will be, i can add a TODO: fully remove in vXX comment to entryToJSON

testing

to get started on testing, compile, run the tests, and start your node with the deprecated rpcs flag:

./src/bitcoind -daemon -deprecatedrpc=fees

you should see entries with the deprecated fields like so:

{
  "<txid>": {
    "fees": {
      "base": 0.00000671,
      "modified": 0.00000671,
      "ancestor": 0.00000671,
      "descendant": 0.00000671
    },
    "fee": 0.00000671,
    "modifiedfee": 0.00000671,
    "descendantfees": 671,
    "ancestorfees": 671,
    "vsize": 144,
    "weight": 573,
   ...
  },

you can also check getmempoolentry using any of the txid's from the output above.

next start the node without the deprecated flag, repeat the commands from above and verify that the deprecated fields are no longer present at the top level, but present in the "fees" object

josibake · 2021-08-12T15:16:38Z

force pushed from 084d443 to 251d636 to fix linting and ci errors

DrahtBot · 2021-08-12T19:14:53Z

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Conflicts

Reviewers, this pull request conflicts with the following ones:

#23319 (rpc: Return fee and prevout (utxos) to getrawtransaction by dougEfresh)
#22698 (Implement RBF inherited signaling and fix getmempoolentry returned bip125-replaceable status by mjdietzx)
#22341 (rpc: add getxpub by Sjors)

If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

glozow

Concept and approach ACK

src/rpc/blockchain.cpp

test/functional/rpc_mempool_entry_fee_fields_deprecation.py

glozow · 2021-08-13T09:44:10Z

test/functional/rpc_mempool_entry_fee_fields_deprecation.py

+        # note: we are only testing getmempoolentry even though
+        # getrawmempool(verbose=True) also returns mempool entries
+        # this is because getmempool entry and getrawmempool(verbose=True)
+        # both call the same underlying entryToJSON function so testing
+        # getmempoolentry is sufficient


You could also add the assertions to a helper function, and call that helper function with results you get from each RPC

test/functional/rpc_mempool_entry_fee_fields_deprecation.py

test/functional/rpc_fundrawtransaction.py

theStack · 2021-08-13T10:26:11Z

Concept ACK

Zero-1729 · 2021-08-13T14:02:40Z

Concept ACK

josibake · 2021-08-13T14:50:57Z

thanks for the review, @glozow! I added your suggestions and left a comment on why I think it makes sense to keep rpc_mempool_entry_fields_deprecation.py separate from rpc_deprecated.py; ultimately, I'm happy to go with whichever option you recommend

mjdietzx

Code review ACK

You'll probably need to add release notes, something like 90ae3d8

mjdietzx · 2021-08-16T06:07:07Z

src/rpc/blockchain.cpp

@@ -491,19 +492,20 @@ static void entryToJSON(const CTxMemPool& pool, UniValue& info, const CTxMemPool
    fees.pushKV("ancestor", ValueFromAmount(e.GetModFeesWithAncestors()));
    fees.pushKV("descendant", ValueFromAmount(e.GetModFeesWithDescendants()));
    info.pushKV("fees", fees);
-
+    if (include_deprecated_fee_fields) {


Is there a reason you don't do:

if (IsDeprecatedRPCEnabled("fees")) {

here? And not have to change the function header?

i was following ScriptPubKeyToUniv as an example:

bitcoin/src/rpc/blockchain.cpp

Lines 1936 to 1939 in 42b00a3

void ScriptPubKeyToUniv(const CScript& scriptPubKey, UniValue& out, bool fIncludeHex)

{

ScriptPubKeyToUniv(scriptPubKey, out, fIncludeHex, IsDeprecatedRPCEnabled("addresses"));

}

personally, i agree with this approach more than using the IsDeprecatedRPCEnabled function directly as it it's more clear, imo, what's happening and allows for more flexibility.

for example, lets say we wanted to only allow getmempoolentry to return the fields when -deprecated=fees is passed but wanted getrawmempool to never return the fields. with this setup, we can have one of the functions use the original function signature with IsDeprecated passed as the default, whereas we could call entryToJSON with false passed as the default whenever it's used in getrawmempool.

tldr; i think functional overloading is super cool and this seemed a good use case for it 😄

As this entryToJSON isn't called anywhere, except for entryToJSON, I think it is fine to use the suggestion by @mjdietzx . No strong opinion, though.

I agree as well, but no big deal so didn't mention it.

seems like its the more preferred approach, happy to switch

josibake · 2021-08-16T15:47:03Z

Code review ACK

You'll probably need to add release notes, something like 90ae3d8

thanks for the review, @mjdietzx ! ive updated the release notes in 4df6ffd

fanquake · 2021-08-17T11:45:42Z

thanks for the review, @mjdietzx ! ive updated the release notes in 4df6ffd

Please write more descriptive commit messages than update release notes. i.e This could be something like doc: add release note for fee field deprecation in RPC.

josibake · 2021-08-17T12:06:17Z

Please write more descriptive commit messages than update release notes. i.e This could be something like doc: add release note for fee field deprecation in RPC.

great callout, @fanquake . ive update all three commit messages with relevant tags (rpc, test, doc) and wrote more descriptive commit messages

jonatack

Code review ACK modulo the comment below and updating the first commit message, as getmempoolancestors and getmempooldescendants are also changed unless I'm misreading the code

Edit: perhaps rename the PR to "rpc: deprecate top-level fee fields in getmempool RPCs"

jonatack · 2021-08-18T13:37:56Z

doc/release-notes.md

@@ -63,6 +63,8 @@ P2P and network changes

 Updated RPCs
 ------------
+- `getmempoolentry` and `getrawmempool true` no longer return top level fee fields `fee`, `modifiedfee`, `ancestorfees`, `descendantfees`. These fields were deprecated in `0.17` and moved to the `fees` sub-object in the response.
+The `-deprecated=fees` flag must be passed for these fields to be included in the top level response. Note: this flag will only be available until `v24`, after which the fields will be fully removed. It is recommended you migrate to using the `fees` sub-object in the response


suggestions

add the PR number

add getmempoolancestors and getmempooldescendants that are also changed

s/true/verbose=true/ for clarity

wrap the lines

s/sub-object/JSON object/

s/top level/top-level/

no need to wrap version numbers in code markup

-- `getmempoolentry` and `getrawmempool true` no longer return top level fee fields `fee`, `modifiedfee`, `ancestorfees`, `descendantfees`. These fields were deprecated in `0.17` and moved to the `fees` sub-object in the response. -The `-deprecated=fees` flag must be passed for these fields to be included in the top level response. Note: this flag will only be available until `v24`, after which the fields will be fully removed. It is recommended you migrate to using the `fees` sub-object in the response +- RPCs `getmempoolentry`, `getrawmempool verbose=true`, `getmempoolancestors + verbose=true` and `getmempooldescendants verbose=true` no longer return the + top-level fee fields `fee`, `modifiedfee`, `ancestorfees` and + `descendantfees`. These fields were deprecated in v0.17 and moved to the + `fees` JSON object in the response. The `-deprecated=fees` configuration + option must be passed for these fields to be included in the top-level + response. This option will only be available until v24.0, after which the + fields will be fully removed. Use the `fees` JSON object for these RPCS + instead. (#22689)

josibake · 2021-08-18T19:26:27Z

getmempoolancestors and getmempooldescendants are also changed unless I'm misreading the code

good catch! i verified by grepping for entryToJSON and it is used in those RPC's as well. updated the commit message and release docs. thanks for the suggestions on the release doc; it reads much better

josibake · 2021-08-18T19:33:26Z

force pushed e5b22ec to fix a failure in the linter

josibake · 2021-10-28T15:43:47Z

rebased and force pushed. the rebase changes can be checked with git range-diff 4792072...61af49c

also removed the extra newlines from the newly added functional test as suggested by @glozow and included the two wording changes from @jonatack in doc/release-notes.md as part of the rebase

josibake · 2021-10-28T15:55:05Z

looking at ci failures, not immediately clear to me what's happening - will take a look tomorrow

test/functional/rpc_mempool_entry_fee_fields_deprecation.py

josibake · 2021-10-29T14:22:20Z

one-line change to fix the newly added test (git range-diff 61af49c...285cb3d)

jnewbery

The set of changes look fine, but there a couple of issues with commits:

The mempool_packages.py fails after the first commit since you remove the COIN import (and then add it back in the second commit). I think you just need to squash the changes to mempool_packages.py in the second commit into the first commit.
Can you fixup the commit logs with standard punctuation/capitalization, backticks to indicate code, full filenames, etc.

src/rpc/blockchain.cpp

josibake · 2021-11-01T12:26:50Z

The set of changes look fine, but there a couple of issues with commits:

* The mempool_packages.py fails after the first commit since you remove the `COIN` import (and then add it back in the second commit). I think you just need to squash the changes to mempool_packages.py in the second commit into the first commit.

* Can you fixup the commit logs with standard punctuation/capitalization, backticks to indicate code, full filenames, etc.

thanks for taking a look at this again, @jnewbery ! looks like i added back the COIN import as part of the rebase but added it to the wrong commit. fixed now. also cleaned up the commit messages and added the newline per your suggestion.

git range-diff 285cb3d...7330beb

jnewbery

Code review ACK 7330beb

One suggestion inline to speed up the new function test. Thanks for the quick turnaround @josibake!

test/functional/rpc_mempool_entry_fee_fields_deprecation.py

Unless `-deprecatedrpc=fees` is passed, top level fee fields are no longer returned for mempool entries. Add instructions to field help on how to access deprecated fields, update help text for readability, and include units. This is important to help avoid any confusion as users move from deprecated fields to the fee fields object (credit: jonatack). This affects `getmempoolentry`, `getrawmempool`, `getmempoolancestors`, and `getmempooldescendants` Modify `test/functional/mempool_packages.py` and `test/functional/rpc_fundrawtransaction.py` tests to no longer use deprecated fields. Co-authored-by: jonatack <jon@atack.com>

Test for old fields when `-deprecatedrpc=fees` flag is passed and verify values in the deprecated fields match values in the fees sub-object.

jnewbery · 2021-11-02T10:45:51Z

reACK 2f9515f

One minor comment above: #22689 (comment). No need to resolve that unless you need to push again for some other reason.

glozow

utACK 2f9515f

test/functional/rpc_mempool_entry_fee_fields_deprecation.py

maflcko · 2021-12-07T14:48:28Z

src/rpc/blockchain.cpp

-    RPCResult{RPCResult::Type::STR_AMOUNT, "fee", "transaction fee in " + CURRENCY_UNIT + " (DEPRECATED)"},
-    RPCResult{RPCResult::Type::STR_AMOUNT, "modifiedfee", "transaction fee with fee deltas used for mining priority (DEPRECATED)"},
+    RPCResult{RPCResult::Type::STR_AMOUNT, "fee", "transaction fee, denominated in " + CURRENCY_UNIT + " (DEPRECATED, returned only if config option -deprecatedrpc=fees is passed)"},
+    RPCResult{RPCResult::Type::STR_AMOUNT, "modifiedfee", "transaction fee with fee deltas used for mining priority, denominated in " + CURRENCY_UNIT + " (DEPRECATED, returned only if config option -deprecatedrpc=fees is passed)"},


missing optional. see #23694

…l RPCs

…etmempool RPCs 0ab0f52 rpc: move fees object to match help (josibake) 23889a0 doc: add release note for fee field deprecation (josibake) 12993d4 test: add functional test for deprecatedrpc=fees (josibake) 79f6fca rpc: deprecate fee fields from mempool entries (josibake) Pull request description: per #22682 , top level fee fields for mempool entries have been deprecated since 0.17 but are still returned. this PR properly deprecates them so that they are no longer returned unless `-deprecatedrpc=fees` is passed. the first commit takes care of deprecation and also updates `test/functional/mempool_packages.py` to only use the `fees` object. the second commit adds a new functional test for `-deprecatedrpc=fees` closes #22682 ## questions for the reviewer * `-deprecatedrpc=fees` made the most sense to me, but happy to change if there is a name that makes more sense * #22682 seems to indicate that after some period of time, the fields will be removed all together. if we have a rough idea of when this will be, i can add a `TODO: fully remove in vXX` comment to `entryToJSON` ## testing to get started on testing, compile, run the tests, and start your node with the deprecated rpcs flag: ```bash ./src/bitcoind -daemon -deprecatedrpc=fees ``` you should see entries with the deprecated fields like so: ```json { "<txid>": { "fees": { "base": 0.00000671, "modified": 0.00000671, "ancestor": 0.00000671, "descendant": 0.00000671 }, "fee": 0.00000671, "modifiedfee": 0.00000671, "descendantfees": 671, "ancestorfees": 671, "vsize": 144, "weight": 573, ... }, ``` you can also check `getmempoolentry` using any of the txid's from the output above. next start the node without the deprecated flag, repeat the commands from above and verify that the deprecated fields are no longer present at the top level, but present in the "fees" object ACKs for top commit: jnewbery: reACK 0ab0f52 glozow: utACK 0ab0f52 Tree-SHA512: b175f4d39d26d96dc5bae26717d3ccfa5842d98ab402065880bfdcf4921b14ca692a8919fe4e9969acbb5c4d6e6d07dd6462a7e0a0a7342556279b381e1a004e

…l entries 885694d doc: add release note about removal of `deprecatedrpc=fees` flag (Sebastian Falbesoner) 387ae8b rpc: remove deprecated fee fields from mempool entries (Sebastian Falbesoner) Pull request description: Deprecating the top-level fee fields (`fee`, `modifiedfee`, `ancestorfees` and `descendantfees`) from the mempool entries and introducing `-deprecatedrpc=fees` was done in PR #22689 (released in v23.0). For the next release v24.0, this configuration option can be removed. ACKs for top commit: fanquake: ACK 885694d Tree-SHA512: fec6b5be5c3f0cd55738a888b390ef9271e70b2dba913a14ce82427dac002e999f93df298bb3b494f3d1b850a23d2b5b3e010e901543b0d18db9be133579e1ec

DrahtBot added the RPC/REST/ZMQ label Aug 12, 2021

josibake force-pushed the josibake-deprecate-mempool-entry-fee-fields branch from 084d443 to 251d636 Compare August 12, 2021 15:14

josibake force-pushed the josibake-deprecate-mempool-entry-fee-fields branch from 251d636 to 8cae682 Compare August 12, 2021 15:43

DrahtBot mentioned this pull request Aug 12, 2021

Remove -deprecatedrpc=addresses flag and corresponding code/logic #22650

Merged

fanquake requested a review from glozow August 13, 2021 00:48

DrahtBot mentioned this pull request Aug 13, 2021

RPC: Add ancestor{count,size,fees} to listunspent output #12677

Merged

josibake force-pushed the josibake-deprecate-mempool-entry-fee-fields branch from 8cae682 to c4646f8 Compare August 13, 2021 09:01

glozow reviewed Aug 13, 2021

View reviewed changes

josibake force-pushed the josibake-deprecate-mempool-entry-fee-fields branch 2 times, most recently from f470aaf to 6940fe0 Compare August 13, 2021 14:35

mjdietzx reviewed Aug 16, 2021

View reviewed changes

josibake force-pushed the josibake-deprecate-mempool-entry-fee-fields branch from 8fb8dbf to 4df6ffd Compare August 16, 2021 15:45

DrahtBot mentioned this pull request Aug 16, 2021

test: refactor use of getrawmempool in functional tests for efficiency #22707

Merged

josibake requested a review from glozow August 17, 2021 08:18

josibake force-pushed the josibake-deprecate-mempool-entry-fee-fields branch from 4df6ffd to c3a2842 Compare August 17, 2021 12:04

jonatack reviewed Aug 18, 2021

View reviewed changes

josibake changed the title ~~rpc: properly deprecate mempool entry fee fields~~ rpc: deprecate top-level fee fields in getmempool RPCs Aug 18, 2021

josibake force-pushed the josibake-deprecate-mempool-entry-fee-fields branch from c3a2842 to 2ba3ead Compare August 18, 2021 19:26

josibake requested a review from jonatack August 18, 2021 19:29

josibake force-pushed the josibake-deprecate-mempool-entry-fee-fields branch from 2ba3ead to e5b22ec Compare August 18, 2021 19:32

DrahtBot mentioned this pull request Oct 28, 2021

rpc: add path to gethdkey #22341

Closed

glozow reviewed Oct 29, 2021

View reviewed changes

test/functional/rpc_mempool_entry_fee_fields_deprecation.py Outdated Show resolved Hide resolved

josibake force-pushed the josibake-deprecate-mempool-entry-fee-fields branch from 61af49c to 285cb3d Compare October 29, 2021 14:20

DrahtBot mentioned this pull request Oct 30, 2021

rpc: Return fee and prevout (utxos) to getrawtransaction #23319

Merged

jnewbery reviewed Nov 1, 2021

View reviewed changes

src/rpc/blockchain.cpp Show resolved Hide resolved

josibake force-pushed the josibake-deprecate-mempool-entry-fee-fields branch from 285cb3d to 7330beb Compare November 1, 2021 12:07

jnewbery reviewed Nov 1, 2021

View reviewed changes

test/functional/rpc_mempool_entry_fee_fields_deprecation.py Outdated Show resolved Hide resolved

josibake and others added 4 commits November 2, 2021 10:05

test: add functional test for deprecatedrpc=fees

2ee406c

Test for old fields when `-deprecatedrpc=fees` flag is passed and verify values in the deprecated fields match values in the fees sub-object.

doc: add release note for fee field deprecation

07ade7d

rpc: move fees object to match help

2f9515f

josibake force-pushed the josibake-deprecate-mempool-entry-fee-fields branch from 7330beb to 2f9515f Compare November 2, 2021 09:06

This was referenced Nov 14, 2021

Refactor: Improve API design of ScriptToUniv, TxToUniv etc to return the UniValue instead of mutating a parameter/reference #23507

Closed

rpc, wallet: addhdseed, infer seed when importing descriptor with xpub #23544

Closed

DrahtBot mentioned this pull request Nov 29, 2021

Implement RBF inherited signaling and fix getmempoolentry returned bip125-replaceable status #22698

Closed

glozow reviewed Dec 7, 2021

View reviewed changes

test/functional/rpc_mempool_entry_fee_fields_deprecation.py Show resolved Hide resolved

glozow approved these changes Dec 7, 2021

View reviewed changes

maflcko merged commit 4fd0ce7 into bitcoin:master Dec 7, 2021

maflcko reviewed Dec 7, 2021

View reviewed changes

sidhujag pushed a commit to syscoin/syscoin that referenced this pull request Dec 7, 2021

Merge bitcoin#22689: rpc: deprecate top-level fee fields in getmempoo…

10749fc

…l RPCs

theStack mentioned this pull request May 25, 2022

rpc: remove deprecated top-level fee fields from mempool entries #25204

Merged

bitcoin locked and limited conversation to collaborators Dec 7, 2022

josibake deleted the josibake-deprecate-mempool-entry-fee-fields branch January 26, 2024 10:51

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

rpc: deprecate top-level fee fields in getmempool RPCs #22689

rpc: deprecate top-level fee fields in getmempool RPCs #22689

josibake commented Aug 12, 2021

josibake commented Aug 12, 2021

DrahtBot commented Aug 12, 2021 •

edited

Loading

glozow left a comment

glozow Aug 13, 2021

theStack commented Aug 13, 2021

Zero-1729 commented Aug 13, 2021

josibake commented Aug 13, 2021 •

edited

Loading

mjdietzx left a comment

mjdietzx Aug 16, 2021

josibake Aug 16, 2021

maflcko Aug 18, 2021

jonatack Aug 18, 2021

josibake Aug 19, 2021

josibake commented Aug 16, 2021

fanquake commented Aug 17, 2021

josibake commented Aug 17, 2021

jonatack left a comment •

edited

Loading

jonatack Aug 18, 2021 •

edited

Loading

josibake commented Aug 18, 2021

josibake commented Aug 18, 2021

josibake commented Oct 28, 2021

josibake commented Oct 28, 2021

josibake commented Oct 29, 2021

jnewbery left a comment

josibake commented Nov 1, 2021

jnewbery left a comment

jnewbery commented Nov 2, 2021

glozow left a comment

maflcko Dec 7, 2021

	void ScriptPubKeyToUniv(const CScript& scriptPubKey, UniValue& out, bool fIncludeHex)
	{
	ScriptPubKeyToUniv(scriptPubKey, out, fIncludeHex, IsDeprecatedRPCEnabled("addresses"));
	}

rpc: deprecate top-level fee fields in getmempool RPCs #22689

rpc: deprecate top-level fee fields in getmempool RPCs #22689

Conversation

josibake commented Aug 12, 2021

questions for the reviewer

testing

josibake commented Aug 12, 2021

DrahtBot commented Aug 12, 2021 • edited Loading

Conflicts

glozow left a comment

Choose a reason for hiding this comment

glozow Aug 13, 2021

Choose a reason for hiding this comment

theStack commented Aug 13, 2021

Zero-1729 commented Aug 13, 2021

josibake commented Aug 13, 2021 • edited Loading

mjdietzx left a comment

Choose a reason for hiding this comment

mjdietzx Aug 16, 2021

Choose a reason for hiding this comment

josibake Aug 16, 2021

Choose a reason for hiding this comment

maflcko Aug 18, 2021

Choose a reason for hiding this comment

jonatack Aug 18, 2021

Choose a reason for hiding this comment

josibake Aug 19, 2021

Choose a reason for hiding this comment

josibake commented Aug 16, 2021

fanquake commented Aug 17, 2021

josibake commented Aug 17, 2021

jonatack left a comment • edited Loading

Choose a reason for hiding this comment

jonatack Aug 18, 2021 • edited Loading

Choose a reason for hiding this comment

josibake commented Aug 18, 2021

josibake commented Aug 18, 2021

josibake commented Oct 28, 2021

josibake commented Oct 28, 2021

josibake commented Oct 29, 2021

jnewbery left a comment

Choose a reason for hiding this comment

josibake commented Nov 1, 2021

jnewbery left a comment

Choose a reason for hiding this comment

jnewbery commented Nov 2, 2021

glozow left a comment

Choose a reason for hiding this comment

maflcko Dec 7, 2021

Choose a reason for hiding this comment

DrahtBot commented Aug 12, 2021 •

edited

Loading

josibake commented Aug 13, 2021 •

edited

Loading

jonatack left a comment •

edited

Loading

jonatack Aug 18, 2021 •

edited

Loading