[RPC] Simplified bumpfee command. #8456

Open
wants to merge 6 commits into
from
@mrbandrews
Contributor

This purports to be a simplified "bumpfee" RPC command.

This pull was motivated by the discussion in #7865, where at the end, the idea was floated of submitting a simplified version of the code. I put a little thought into how a simple "bumpfee" command should work and here is what I came up with:

  1. The user should specify which output to decrement in order to pay increased fee. This saves us from trying to figure out which address(es) are "change" addresses. This is preferable not only because it simplifies the code, but because as far as I can tell, the code that identifies change outputs is not particularly robust and may be modified in the future. If it's desirable to have user-friendly code that figures out which output to decrement, perhaps that logic could be placed in the GUI?

  2. The command should not add new inputs. In other words, if the user can't identify an output that can be decremented in an amount sufficient to bump the fee then the command should fail. It seems likely that the vast majority of the time, identifying such an output would not be a problem. Adding new inputs complicates the code because the size of the transaction increases, perhaps significantly, so then the user would have to pay more fee for that reason as well, as opposed to just bumping the fee on the tx as it currently exists.

  3. If the tx has descendant transactions, the bumped tx has to pay the fees for all of the descendants since they will get evicted from the mempool, and the rule as I understand it is that replacing a tx cannot cause the total fees in the mempool to go down.

So, the required inputs are the txid and the output index.

The optional inputs are:

confirmTarget - perhaps the reason the user feels the need to bump the fee is because the original confirmTarget was too slow
totalFee - for full control, the user can simply specify how many satoshis to pay as a fee on the bumped transaction (this may be useful when the parent tx has children tx'es)

This pull includes a python test.

@laanwj laanwj added the RPC/REST/ZMQ label Aug 4, 2016
@jonasschnelli
Member

Thanks! Looks good.
Concept ACK will review and test soon.

@NicolasDorier NicolasDorier and 1 other commented on an outdated diff Aug 5, 2016
src/wallet/rpcwallet.cpp
+ newConfirmTarget = options["confTarget"].get_int();
+ if (newConfirmTarget <= 0) // upper-bound will be checked by estimatefee/smartfee
+ throw JSONRPCError(RPC_INVALID_PARAMETER, "Invalid confTarget (cannot be <= 0)");
+ }
+ if (options.exists("totalFee")) {
+ totalFee = options["totalFee"].get_int();
+ if (totalFee <= 0)
+ throw JSONRPCError(RPC_INVALID_PARAMETER, "Invalid totalFee (cannot be <= 0)");
+ else if (totalFee > maxTxFee)
+ throw JSONRPCError(RPC_INVALID_PARAMETER, "Invalid totalFee (higher than maxTxFee)");
+ }
+ }
+
+ // calculate the old fee and fee-rate
+ CAmount nDebit = wtx.GetDebit(ISMINE_SPENDABLE);
+ CAmount nOldFee = -(wtx.IsFromMe(ISMINE_SPENDABLE) ? wtx.GetValueOut() - nDebit : 0);
@NicolasDorier
NicolasDorier Aug 5, 2016 Member

nit: Double negation makes it hards to read. I suggest to replace by

 CAmount nOldFee = wtx.IsFromMe(ISMINE_SPENDABLE) ? nDebit - wtx.GetValueOut() : 0;
@mrbandrews
mrbandrews Oct 18, 2016 Contributor

Fixed.

@NicolasDorier NicolasDorier and 1 other commented on an outdated diff Aug 5, 2016
src/wallet/rpcwallet.cpp
+ if (totalFee > 0) {
+ CAmount minTotalFee = nOldFeeRate.GetFee(txSize) + minRelayTxFee.GetFee(txSize);
+ if (totalFee < minTotalFee) {
+ strError = strprintf("Invalid totalFee, must be at least oldFee + relayFee: %s", FormatMoney(minTotalFee));
+ throw JSONRPCError(RPC_INVALID_PARAMETER, strError);
+ }
+ nNewFee = totalFee;
+ nNewFeeRate = CFeeRate(totalFee, txSize);
+ }
+
+ // check that fee rate is higher than mempool's mininum fee
+ // (no point in bumping fee if we know that the new tx won't be accepted to the mempool)
+ CFeeRate minMempoolFeeRate = mempool.GetMinFee(GetArg("-maxmempool", DEFAULT_MAX_MEMPOOL_SIZE) * 1000000);
+ if (nNewFeeRate.GetFeePerK() < minMempoolFeeRate.GetFeePerK()) {
+ strError = strprintf("New fee rate (%s) is too low to get into the mempool (min rate: %s)", FormatMoney(nNewFeeRate.GetFeePerK()), FormatMoney(minMempoolFeeRate.GetFeePerK()));
+ throw JSONRPCError(RPC_MISC_ERROR, strError);
@NicolasDorier
NicolasDorier Aug 5, 2016 Member

I'm not sure about this. The problem is that such case will rarely happen so applications risk to break during sudden peak usage, amplifying bad news.

I suggest to silently bump fees to the minMempoolFeeRate.

@mrbandrews
mrbandrews Oct 18, 2016 Contributor

I edited the comment to explain why I'm reporting an error. Especially if the user set totalFee (or had in the recent past used paytxfee to set an explicit fee rate) I think silently bumping could surprise the user.

@NicolasDorier NicolasDorier and 1 other commented on an outdated diff Aug 5, 2016
src/wallet/rpcwallet.cpp
+ if (it != mempool.mapTx.end()) {
+
+ // Tx with descendants
+ // --------------------
+ // Idea: When a tx is bumped, its descendants (if any) are evicted from the mempool.
+ // Policy is that when you replace a tx, the total fees in the mempool cannot go down.
+ // So when you bump a tx that has children, you have to bump your fee by the sum of all of the
+ // descendants, plus the new bumped tx's relay fee.
+ //
+ // If the bumped fee is less than what's required because of children, we fail. This is a different
+ // situation than bumping the fee to pay the minimum relay fee. Here, we can't be sure whether the user
+ // really wants to pay full price for all of the child transactions. If so, the user can set payTxFee
+ // and run the command again.
+ //
+ CAmount nFeesWithDescendants = it->GetModFeesWithDescendants();
+ if (nNewFee < nFeesWithDescendants + ::minRelayTxFee.GetFee(txSize)) {
@NicolasDorier
NicolasDorier Aug 5, 2016 edited Member

nit:

 nFeesWithDescendants + ::minRelayTxFee.GetFee(txSize)

Is repeated. Suggest using variable.

@mrbandrews
mrbandrews Oct 18, 2016 Contributor

Fixed.

@NicolasDorier NicolasDorier and 2 others commented on an outdated diff Aug 5, 2016
src/wallet/rpcwallet.cpp
+ // then it doesn't need to be replaced), but it may still be in the mempol of peers
+ // (perhaps a peer has allocated more space for the mempool).
+ //
+ // Our approach is to go ahead and bump/commit/relay the transaction. In the event that
+ // the tx does have children and the fee is insufficient to cover, the peer(s) will reject
+ // the tx on that basis, so we warn the user of this possibility.
+ //
+ LogPrint("rpc", "Warning: bumping fee on tx that is not in the mempool; if it has child transactions, it may be rejected by peers\n");
+ }
+ }
+
+ // Now modify the output to increase the fee.
+ // Output must be able to pay the increased fee, without being reduced to dust
+ CMutableTransaction tx(wtx);
+ CTxOut* poutput = &(tx.vout[nOutput]);
+ if (poutput->nValue >= nDelta + poutput->GetDustThreshold(::minRelayTxFee)) {
@NicolasDorier
NicolasDorier Aug 5, 2016 Member

additional check should be done:

nDelta >= poutput->GetDustThreshold(::minRelayTxFee);

Or it won't get propagated.

@mrbandrews
mrbandrews Oct 18, 2016 Contributor

I checked earlier that the bumped fee pays for the new relay fee; I'm not following as to the relationship between nDelta and dust.

@luke-jr
luke-jr Nov 23, 2016 Member

@NicolasDorier I also don't see a need for that...

@NicolasDorier NicolasDorier and 1 other commented on an outdated diff Aug 5, 2016
src/wallet/rpcwallet.cpp
+
+ // Now modify the output to increase the fee.
+ // Output must be able to pay the increased fee, without being reduced to dust
+ CMutableTransaction tx(wtx);
+ CTxOut* poutput = &(tx.vout[nOutput]);
+ if (poutput->nValue >= nDelta + poutput->GetDustThreshold(::minRelayTxFee)) {
+ poutput->nValue = poutput->nValue - nDelta;
+ }
+ else {
+ throw JSONRPCError(RPC_MISC_ERROR, "Output does not have enough money to bump the fee");
+ }
+
+ // sign the new tx
+ CTransaction txNewConst(tx);
+ int nIn = 0;
+ for (auto it(tx.vin.begin()); it != tx.vin.end(); ++it, nIn++) {
@NicolasDorier
NicolasDorier Aug 5, 2016 edited Member

can you use C++11 loop style ?

@mrbandrews
mrbandrews Oct 18, 2016 Contributor

Good idea, fixed.

@NicolasDorier NicolasDorier and 1 other commented on an outdated diff Aug 5, 2016
src/wallet/rpcwallet.cpp
+ if (poutput->nValue >= nDelta + poutput->GetDustThreshold(::minRelayTxFee)) {
+ poutput->nValue = poutput->nValue - nDelta;
+ }
+ else {
+ throw JSONRPCError(RPC_MISC_ERROR, "Output does not have enough money to bump the fee");
+ }
+
+ // sign the new tx
+ CTransaction txNewConst(tx);
+ int nIn = 0;
+ for (auto it(tx.vin.begin()); it != tx.vin.end(); ++it, nIn++) {
+ std::map<uint256, CWalletTx>::const_iterator mi = pwalletMain->mapWallet.find((*it).prevout.hash);
+ if (mi != pwalletMain->mapWallet.end() && (nIn < (int)(*mi).second.vout.size())) {
+ const CScript& scriptPubKey = (*mi).second.vout[(*it).prevout.n].scriptPubKey;
+ SignatureData sigdata;
+ if (!ProduceSignature(TransactionSignatureCreator(pwalletMain, &txNewConst, nIn, SIGHASH_ALL), scriptPubKey, sigdata))
@NicolasDorier
NicolasDorier Aug 5, 2016 edited Member

You should use the same SIGHASH than the previous one. (might be done separate PR ?)

@mrbandrews
mrbandrews Oct 18, 2016 Contributor

Let's punt to separate PR.

@NicolasDorier NicolasDorier and 1 other commented on an outdated diff Aug 5, 2016
src/wallet/rpcwallet.cpp
+ // Output must be able to pay the increased fee, without being reduced to dust
+ CMutableTransaction tx(wtx);
+ CTxOut* poutput = &(tx.vout[nOutput]);
+ if (poutput->nValue >= nDelta + poutput->GetDustThreshold(::minRelayTxFee)) {
+ poutput->nValue = poutput->nValue - nDelta;
+ }
+ else {
+ throw JSONRPCError(RPC_MISC_ERROR, "Output does not have enough money to bump the fee");
+ }
+
+ // sign the new tx
+ CTransaction txNewConst(tx);
+ int nIn = 0;
+ for (auto it(tx.vin.begin()); it != tx.vin.end(); ++it, nIn++) {
+ std::map<uint256, CWalletTx>::const_iterator mi = pwalletMain->mapWallet.find((*it).prevout.hash);
+ if (mi != pwalletMain->mapWallet.end() && (nIn < (int)(*mi).second.vout.size())) {
@NicolasDorier
NicolasDorier Aug 5, 2016 Member

(nIn < (int)(*mi).second.vout.size())

I think this is a bug, why does the index of the input to sign should be below the number of vout of the parent ? Does not make sense.

@mrbandrews
mrbandrews Oct 18, 2016 Contributor

Good catch, edited to bounds-check the prevout.

@MarcoFalke MarcoFalke added the Wallet label Aug 7, 2016
@jonasschnelli
Member

Needs rebase and reviewers... setting 0.14 milestone.

@jonasschnelli jonasschnelli added this to the 0.14.0 milestone Aug 19, 2016
@mrbandrews
Contributor

Rebased and addressed feedback.

@luke-jr luke-jr added a commit to bitcoinknots/bitcoin that referenced this pull request Oct 20, 2016
@mrbandrews @luke-jr mrbandrews + luke-jr [RPC] Simplified bumpfee command.
Allows a user to increase the fee on a wallet transaction in the mempool.  Requires user to specify which output to decrement.

Includes code by @jonasschnelli

Github-Pull: #8456
Rebased-From: 41572b1
75ed0b3
@luke-jr luke-jr added a commit to bitcoinknots/bitcoin that referenced this pull request Oct 20, 2016
@mrbandrews @luke-jr mrbandrews + luke-jr Fixed nits
Github-Pull: #8456
Rebased-From: a8f28c2
f012520
@luke-jr luke-jr added a commit to bitcoinknots/bitcoin that referenced this pull request Oct 20, 2016
@mrbandrews @luke-jr mrbandrews + luke-jr fixed comments
Github-Pull: #8456
Rebased-From: a5ab642
e73a5f1
@jonasschnelli
Member

Needs rebase again...

@mrbandrews
Contributor

Rebased and edited to use JSONRPCRequest, consistent with #8788.

src/wallet/rpcwallet.cpp
+
+ if (request.fHelp || request.params.size() < 2 || request.params.size() > 3)
+ throw runtime_error(
+ "bumpfee \"txid\"\n"
@luke-jr
luke-jr Oct 22, 2016 Member

This line is outdated

@RHavar
RHavar commented Oct 26, 2016 edited

I think it's very important for this command to actual figure out the change output on its own. I understand that it's a bit messier and more fragile doing it here -- but it actually has enough information to do this, and by avoiding it, it just pushes that mess into the caller which comes at a very significant usability issue.

Compare:
Transaction stuck? Use bumpfee $txid

vs

Transaction stuck? First use gettransaction $txid, and now get the data from the "hex" field to feed into decoderawtransaction $hex now go through the "vout" fields until you find "addresses" nested in the "scriptPubKey". Now try figure out which is a change address. (I can't even see this exposed over rpc at all?). So maybe go validateaddress $address and check the ismine field. If one of them is yours, but the other one isn't -- then the one that is yours is a change address, so now go back through the results of decoderawtransaction to figure out which index that is. And finally bumpfee $txid $index. Oh yeah, becareful to handle all the edge cases, like the transaction only having 1 output, some of the outputs not having address, both having the same address etc.

I think you'll find the usability problem of needing to know the index will prevent the majority of use by users and services for this =)

@RHavar
RHavar commented Oct 26, 2016 edited

Also ideally, I think the txid argument should be an array of transaction ids to bump. And then it creates a single transaction that bumps the fees on all of those transactions (stripping out extraneous change outputs as it goes). However, I imagine this can be done separately and later as the txid argument could be overloaded to either accept a string or array of strings?

@JeremyRubin

Overall seems like the right behavior for bumpfee, just a few minor suggestions.

It seems that the descendants fee adds additional complexity that probably can't be addressed in the scope of this PR but at least could merit some documentation so that a stuck user can find out what's wrong.

src/wallet/rpcwallet.cpp
+ // optional parameters
+ int newConfirmTarget = nTxConfirmTarget;
+ CAmount totalFee = 0;
+ if (request.params.size() > 2) {
@JeremyRubin
JeremyRubin Oct 26, 2016 Contributor

Perhaps make this fail if there are any options other than confTarget or totalFee passed in to guard against the case where a user has a typo or something.

src/wallet/rpcwallet.cpp
+ totalFee = options["totalFee"].get_int();
+ if (totalFee <= 0)
+ throw JSONRPCError(RPC_INVALID_PARAMETER, "Invalid totalFee (cannot be <= 0)");
+ else if (totalFee > maxTxFee)
@JeremyRubin
JeremyRubin Oct 26, 2016 Contributor

Maybe add a tighter limit to your bumpfee code, bumpfeeMaxTxFee. Set this to be an optional default parameter in options. maxTxFee is huge, so might be safer to have something smaller and not much added code. This parameter can be overridden if need be, but not to more the maxTxFee.

src/wallet/rpcwallet.cpp
+ CFeeRate nOldFeeRate(nOldFee, txSize);
+
+ // use the user-defined payTxFee if possible, otherwise use smartfee / fallbackfee
+ CAmount nNewFee = 0;
@JeremyRubin
JeremyRubin Oct 26, 2016 Contributor

Using a nNewFee = 0 is fine here, it just slightly bothers me to use this as a null value when indeed 0 is a valid fee amount.

-1 would be an invalid fee amount (creates coins) so I have a slight preference to use this.

src/wallet/rpcwallet.cpp
+ throw JSONRPCError(RPC_MISC_ERROR, strError);
+ }
+
+ CAmount nDelta = nNewFee - nOldFee;
@JeremyRubin
JeremyRubin Oct 26, 2016 Contributor

There is kind of a weird deal here where nDelta can be <= 0 if totalFee is set, and will be accepted on a race condition with mempool, I would explicitly guard against this case by restricting nDelta > 0.

@mrbandrews
mrbandrews Oct 27, 2016 Contributor

Done.

@morcos
morcos Nov 21, 2016 Contributor

@JeremyRubin how could nDelta be negative here?

src/wallet/rpcwallet.cpp
+ // and run the command again.
+ //
+ CAmount nFeesWithDescendantsPlusRelay = it->GetModFeesWithDescendants() + ::minRelayTxFee.GetFee(txSize);
+ if (nNewFee < nFeesWithDescendantsPlusRelay) {
@JeremyRubin
JeremyRubin Oct 26, 2016 Contributor

I would suggest that this should also return in the error message the txid of the furthest child, and suggest bumping the fee on that one if possible to take advantage of ancestor fee based mining and keep txs valid?

@mrbandrews
mrbandrews Oct 27, 2016 Contributor

As far as I can tell it's a little tricky to track down the furthest child (and there could be numerous furthest children at the same level) but I edited the error message to give the user more info (number of children and size of those transactions) and explaining the situation a bit better.

src/wallet/rpcwallet.cpp
+ // and run the command again.
+ //
+ CAmount nFeesWithDescendantsPlusRelay = it->GetModFeesWithDescendants() + ::minRelayTxFee.GetFee(txSize);
+ if (nNewFee < nFeesWithDescendantsPlusRelay) {
@JeremyRubin
JeremyRubin Oct 26, 2016 Contributor

There is something kinda funky when a user is running in say blocksonlymode and doesn't know about any child transactions that may exist and therefore has trouble setting the fee correctly for those that they will invalidate.

This is probably a hard problem to solve; so I'm just pointing it out.

@mrbandrews
Contributor

Addressed JeremyRubin feedback, edited the python test, and made a few other small changes I noticed with further testing.

RHavar: I understand your point but I still think it's better for this command to be low-level and not fragile. A more user-friendly RPC (e.g., "bumpfeeauto" or something) could be layered on top, identifying the change output and then using this code. Then if the change-output-identifying code breaks, it might break the user-friendly version but it wouldn't dismantle RBF entirely.

@Victorsueca
Victorsueca commented Nov 11, 2016 edited

ACK f3833f4
Tested on Windows x64

bumpfee 8c56b13830405a55ec4bc58b26b531f1b187d2349ee19bd0dd01aa835972929a 1

{
  "txid": "67d6af1a3e29e246eaef0b7ce272f745e2ae6178050ccb78fca515b13c0f9e92",
  "oldfee": 0.00000260,
  "fee": 0.00000520
}
src/wallet/rpcwallet.cpp
+ "This command will fail if fee is not high enough or output is not large enough.\n"
+ "User can specify totalFee, or use RPC setpaytxfee to set a higher fee rate.\n"
+ "\nArguments:\n"
+ "1. \"transactionid\" (string, required) The txid to be bumped\n"
@jonasschnelli
jonasschnelli Nov 11, 2016 Member

nit: Maybe use txid to correspond with the call header at L2587 or use transactionid there?

src/wallet/rpcwallet.cpp
+
+ if (request.fHelp || request.params.size() < 2 || request.params.size() > 3)
+ throw runtime_error(
+ "bumpfee \"txid\" \"output\" \n"
@jonasschnelli
jonasschnelli Nov 11, 2016 Member

add third parameter ("options")?

src/wallet/rpcwallet.cpp
+ "3. options (object, optional)\n"
+ " {\n"
+ " \"confTarget\" (int, optional) Confirmation target (in blocks)\n"
+ " \"totalFee\" (numeric, optional) Total fee to pay, in satoshis (not btc)\n"
@jonasschnelli
jonasschnelli Nov 11, 2016 Member

I'm not sure if we want absolute fee values here.
The user can't be sure how many inputs are getting added when setting this value, probably resulting in an uncontrollable feerate.

What about switching this to a feerate?

@Victorsueca
Victorsueca Nov 11, 2016 edited

Another possibility is to leave the RPC with an absolute value and use Fee/KB on the GUI.
Some software may want to use it's own relative fee rate.

src/wallet/rpcwallet.cpp
+ vector<COutput> vecOutputs;
+ pwalletMain->AvailableCoins(vecOutputs, false, NULL, true);
+ BOOST_FOREACH(const COutput& out, vecOutputs) {
+ if (out.tx->GetHash().GetHex() == hash.GetHex() && out.nDepth > 0)
@jonasschnelli
jonasschnelli Nov 11, 2016 Member

This check looks really expensive for large wallets. Why not calling mapWallet.find(hash) and check nDepth (and maybe call GetConflicts() to ensure its not conflicted with a already mined tx)?

@jonasschnelli
Member

Oh. I just realized that this PR does not add new inputs (it requires an output index to identify the change-output which then can be reduced).

IMO we should...

1.) not let the user identify which output is change
2.) allow bumping fees including adding new inputs (some transactions do not have a change output, some will not allow a reasonable bumping without adding a new input).

But 1.) & 2.) can also be solved later.

@mrbandrews
Contributor

Feedback addressed. Good catch on checking whether the tx had already been mined - that code was able to be shortened to a single line.
Yes, the approach of this PR is to solve 1) and 2) later.
If this looks good I'll squash the commits.

src/wallet/rpcwallet.cpp
+ // This prevents us from trying to bump fee on a tx that has already been bumped. So if the
+ // user created txid1, bumped the fee which resulted in txid2, and now wishes to bump the fee
+ // again, the user must call bumpfee on txid2.
+ const CWalletTx *conflictTx;
@morcos
morcos Nov 14, 2016 Contributor

I don't think this section makes sense. Transactions which are in newer in the wallet are not a good indication of the most "up to date" spend of the outputs. If anything, conflicts in the mempool would be a good proxy, because at least for those you'd be able to calculate any descendants fees. But it may make the most sense to make this more of a utility function and just assume the user is trying to bump the right txid. I would just eliminate this set of checks entirely.

@mrbandrews
mrbandrews Nov 17, 2016 Contributor

Fixed (removed)

src/wallet/rpcwallet.cpp
+ "User can specify totalFee, or use RPC setpaytxfee to set a higher fee rate.\n"
+ "\nArguments:\n"
+ "1. \"txid\" (string, required) The txid to be bumped\n"
+ "2. output (int, required) The output to be decremented\n"
@morcos
morcos Nov 15, 2016 Contributor

Would be helpful to refer to this as the output index throughout... (comments, help, and error messages)

@mrbandrews
mrbandrews Nov 17, 2016 Contributor

Done.

src/wallet/rpcwallet.cpp
+ }
+
+ // calculate the old fee and fee-rate
+ CAmount nDebit = wtx.GetDebit(ISMINE_SPENDABLE);
@morcos
morcos Nov 15, 2016 Contributor

We need to be sure we are calculating the correct fee here. Either using new IsAllFromMe or getting the fee from the mempool.

@mrbandrews
mrbandrews Nov 17, 2016 Contributor

This should be correct since now we're checking that the user owns all the inputs.

src/wallet/rpcwallet.cpp
+ nNewFeeRate = CFeeRate(payTxFee.GetFeePerK());
+ }
+ else {
+ int estimateFoundTarget = newConfirmTarget;
@morcos
morcos Nov 15, 2016 Contributor

you don't need this, you can just leave it out as an argument

@mrbandrews
mrbandrews Nov 17, 2016 Contributor

Fixed.

src/wallet/rpcwallet.cpp
+ // and run the command again.
+ //
+ CAmount nFeesWithDescendantsPlusRelay = it->GetModFeesWithDescendants() + ::minRelayTxFee.GetFee(txSize);
+ if (nNewFee < nFeesWithDescendantsPlusRelay) {
@morcos
morcos Nov 15, 2016 edited Contributor

I think this reduces bumpfee to a two step process in the common case (whenever there are children) and makes it so the second step is going to require setting totalfee. It seems more user friendly to just pay what it takes?

On a separate note, I think it might be nice to check all of the children transactions and make sure their ancestor fee rate isn't higher than the new feerate you are bumping to. If one of them has a higher ancestor fee rate you are actually make your situation worse.

@morcos
morcos Nov 17, 2016 Contributor

I spoke with @sdaftuar and @mrbandrews about this and withdraw both of the above points. If the transaction has children, it becomes quite complicated to decide whether bumping fee makes sense and by how much and its probably better to just report to the user the minimum fee that would be required to bump and let the user decide whether that makes sense.

@mrbandrews
Contributor

morcos feedback addressed, including adding the first commit from #9167.

src/wallet/rpcwallet.cpp
+ const CWalletTx& wtx = pwalletMain->mapWallet[hash];
+
+ // check that we are not trying to bump a tx that has already been mined
+ if (wtx.GetDepthInMainChain() > 0)
@morcos
morcos Nov 21, 2016 Contributor

You should probably check that the depth is exactly 0, because otherwise the tx is already mined or conflicted with a mined tx and no point in bumping.

@luke-jr

IMO the main logic should probably be split up into non-RPC files so it can be reused in the GUI...

src/wallet/rpcwallet.cpp
+
+ if (request.fHelp || request.params.size() < 2 || request.params.size() > 3)
+ throw runtime_error(
+ "bumpfee \"txid\" output ( options ) \n"
@luke-jr
luke-jr Nov 23, 2016 Member

Maybe move output into options? Even if it's required in this version, it should ideally become optional in the future...

src/wallet/rpcwallet.cpp
+
+ // check that original tx signals opt-in-RBF
+ if (!SignalsOptInRBF(wtx))
+ throw JSONRPCError(RPC_INVALID_ADDRESS_OR_KEY, "Transaction is not bip-125 replaceable");
@luke-jr
luke-jr Nov 23, 2016 Member

"BIP 125"

src/wallet/rpcwallet.cpp
+
+ // calculate the old fee and fee-rate
+ CAmount nOldFee = wtx.GetDebit(ISMINE_SPENDABLE) - wtx.GetValueOut();
+ int txSize = (int)GetVirtualTransactionSize((CTransaction)wtx);
@luke-jr
luke-jr Nov 23, 2016 Member

Just use int64_t? int is only guaranteed to be 15-bit, so it's too small anyway.

src/wallet/rpcwallet.cpp
+ CAmount nNewFee = 0;
+ CFeeRate nNewFeeRate;
+ if (payTxFee.GetFeePerK() > 0) {
+ nNewFeeRate = CFeeRate(payTxFee.GetFeePerK());
@luke-jr
luke-jr Nov 23, 2016 Member

nNewFeeRate = payTxFee;?

src/wallet/rpcwallet.cpp
+
+ // if user set totalFee, use that instead
+ if (totalFee > 0) {
+ CAmount minTotalFee = nOldFeeRate.GetFee(txSize) + minRelayTxFee.GetFee(txSize);
@luke-jr
luke-jr Nov 23, 2016 Member

+ wtx.vin.size() seems appropriate here too?

src/wallet/rpcwallet.cpp
+ std::string strError = strprintf("Insufficent fee due to the child transactions.\n");
+ strError += strprintf("The bumped fee must be at least: %s.\n", FormatMoney(nFeesWithDescendantsPlusRelay));
+ strError += strprintf("Number of child transactions: %u, total size of child transactions: %u\n", numDescendants, sizeDescendants);
+ strError += strprintf("Note that the child transactions would be evicted from the mempool and would not be mined.\n");
@luke-jr
luke-jr Nov 23, 2016 Member

s/would/may probably/

In the future, it may make sense to combine some children transactions (ie, our own) into this one while bumping the fee.

src/wallet/rpcwallet.cpp
+ // Output must be able to pay the increased fee, without being reduced to dust
+ CMutableTransaction tx(wtx);
+ CTxOut* poutput = &(tx.vout[nOutput]);
+ if (poutput->nValue >= nDelta + poutput->GetDustThreshold(::minRelayTxFee)) {
@luke-jr
luke-jr Nov 23, 2016 Member

@NicolasDorier I also don't see a need for that...

src/wallet/rpcwallet.cpp
+ poutput->nValue = poutput->nValue - nDelta;
+ }
+ else {
+ throw JSONRPCError(RPC_MISC_ERROR, "Output does not have enough money to bump the fee");
@luke-jr
luke-jr Nov 23, 2016 Member

Why not just discard the change?

src/wallet/rpcwallet.cpp
+ // commit/broadcast the tx
+ CReserveKey reservekey(pwalletMain);
+ CWalletTx wtxBumped(pwalletMain, tx);
+ if (!pwalletMain->CommitTransaction(wtxBumped, reservekey, g_connman.get()))
@luke-jr
luke-jr Nov 23, 2016 Member

This needs to be rebased...

src/wallet/rpcwallet.cpp
+ if (nNewFee < nFeesWithDescendantsPlusRelay) {
+ uint64_t numDescendants = it->GetCountWithDescendants()-1;
+ uint64_t sizeDescendants = it->GetSizeWithDescendants() - it->GetTxSize();
+ std::string strError = strprintf("Insufficent fee due to the child transactions.\n");
@luke-jr
luke-jr Nov 23, 2016 Member

No need for strprintf here

src/wallet/rpcwallet.cpp
+ std::string strError = strprintf("Insufficent fee due to the child transactions.\n");
+ strError += strprintf("The bumped fee must be at least: %s.\n", FormatMoney(nFeesWithDescendantsPlusRelay));
+ strError += strprintf("Number of child transactions: %u, total size of child transactions: %u\n", numDescendants, sizeDescendants);
+ strError += strprintf("Note that the child transactions would be evicted from the mempool and would not be mined.\n");
@luke-jr
luke-jr Nov 23, 2016 Member

Also no strprintf here

src/wallet/rpcwallet.cpp
+ "3. options (object, optional)\n"
+ " {\n"
+ " \"confTarget\": \"n\", (numeric, optional) Confirmation target (in blocks)\n"
+ " \"totalFee\": \"n\", (numeric, optional) Total fee to pay, in satoshis (not btc)\n"
@jonasschnelli
jonasschnelli Nov 23, 2016 Member

I think, because every else use fee-ratres per KB, we should make clear at this point that totalFee and maxFee are absolute fee-values (and not per KB).

src/wallet/rpcwallet.cpp
+ strError += strprintf("The bumped fee must be at least: %s.\n", FormatMoney(nFeesWithDescendantsPlusRelay));
+ strError += strprintf("Number of child transactions: %u, total size of child transactions: %u\n", numDescendants, sizeDescendants);
+ strError += strprintf("Note that the child transactions would be evicted from the mempool and would not be mined.\n");
+ strError += strprintf("To avoid mempool eviction, consider bumping fee on the child transactions (with fee to pay for the ancestors).\n");
@luke-jr
luke-jr Nov 23, 2016 Member

del strprintf

@mrbandrews
Contributor

OK, rebased and squashed, with edits addressing the most recent feedback in a separate commit.

I didn't separate code into non-rpc files because I'm not 100% sure which logic should be moved, and thought this decision could be made when using it from the GUI, as moving the code then should be easy enough.

src/wallet/rpcwallet.cpp
+
+ // if user set totalFee, use that instead
+ if (totalFee > 0) {
+ CAmount minTotalFee = nOldFeeRate.GetFee(txSize+wtx.vin.size()) + minRelayTxFee.GetFee(txSize+wtx.vin.size());
@luke-jr
luke-jr Dec 4, 2016 Member

You should calculate txSize + wtx.vin.size() once and keep it in a const variable for these...

const int64_t maxNewTxSize = txSize + wtx.vin.size();
@gmaxwell
Member
gmaxwell commented Dec 4, 2016

Asking the user to identify change is unreasonable and dangerous. Within our own wallet we should know which outputs are our own. This also has a problem of creating a mess when the original version of the transaction it spent, but later sends may have spent the replacements output.

I would suggest that a minimal bump-fee would do this:

(1) Only be available on transactions where none of their outputs have been spent (even in mempool).
(2) Preserve all the user provided outputs, potentially changing the change amount, potentially adding more inputs if needed to get past dust or just sufficient fees. Obey the opt-in RBF minimum increment
(3) Mark its local output(s) as unspendable until confirmed.

1/3 are required so that we don't end up with a mess when the 'wrong' version of the transaction confirms and invalidates the others.

A somewhat more advanced -- and perhaps better to do instead-- would be a "bumpunconfirmed" which would not act on a single transaction but on all valid unconfirmed sends of the local wallet at once-- generating a single replacement transaction which conflicts each of the originals, pays a higher fee, and marks its own output as unspendable until confirmed.

I suggest this might be better instead because it would not need to have the requirement that its local outputs not be unspent, since it would replace all those spends at once. The reduced number of change outputs plus the ability to drop some of the inputs (so long as it still conflicts, and still marks them as unspendable until this transaction confirms) means that it can reduce the total transaction sizes.

@mrbandrews
Contributor

Addressed the gmaxwell feedback as follows: now it identifies the change output and guards against spending outputs until the bumped transaction (or perhaps the original transaction) is mined.

I reorganized this into two commits, the latter being the python test, which is updated.

src/wallet/rpcwallet.cpp
+ throw JSONRPCError(RPC_INVALID_ADDRESS_OR_KEY, "Transaction is not BIP 125 replaceable");
+
+ if (wtx.mapValue.count("replaced_by_txid"))
+ throw JSONRPCError(RPC_INVALID_REQUEST, strprintf("Cannot bump transaction %s which was already bumped by transaction %s", hash.ToString(), wtx.mapValue.at("replaced_by_txid")));
@luke-jr
luke-jr Dec 10, 2016 Member

Any reason not to automatically just switch to the new txid here?

@mrbandrews
mrbandrews Dec 13, 2016 Contributor

For now, I'd rather leave it as is, and let the user resubmit with the correct txid.

src/wallet/rpcwallet.cpp
+ // figure out which output was change
+ // if there was no change output or if there were multiple change outputs, fail
+ int nOutput = -1;
+ for (int i=0; i < (int) wtx.tx->vout.size(); i++) {
@luke-jr
luke-jr Dec 10, 2016 Member

Just use a size_t here, not cast to int... And use ++i, not i++

But more importantly, better to use C++11 iterators in this case:

for (const CTxOut& txout : wtx.tv->vout) {
    if (pwalletMain->IsChange(txout)) {
        if (nOutput != -1) {
            throw JSONRPCError(RPC_MISC_ERROR, "Transaction has multiple change outputs");
        }
        nOutput = i;
    }
}
@sdaftuar
sdaftuar Dec 12, 2016 Contributor

@luke-jr That won't compile sincei is undefined... Is there a better way to loop here where the index is needed? Also I'd suggest that this kind of style nit can be cleaned up in a later pull anyway; this one has been open for long enough.

@mrbandrews
mrbandrews Dec 13, 2016 edited Contributor

Changed to a size_t (and ++i)

src/wallet/rpcwallet.cpp
+
+ // mark the original tx as bumped
+ {
+ CWalletDB walletdb(pwalletMain->strWalletFile, "r+");
@luke-jr
luke-jr Dec 10, 2016 Member

I think we're trying to keep CWalletDB only used by CWallet. @pstratem ?

@luke-jr
luke-jr Dec 10, 2016 Member

Probably this needs to go through CWallet::AddToWallet

@luke-jr
luke-jr Dec 10, 2016 Member

May need to be atomic with adding the new one as well...

@mrbandrews
mrbandrews Dec 13, 2016 Contributor

I moved this code into a new method on the wallet so the CWalletDB is only used from CWallet. I looked at AddToWallet but I would need to have edited it a bit, plus it does some stuff I don't need so I thought a new but concise method was a better approach.

@luke-jr
Member
luke-jr commented Dec 10, 2016

I wonder if this ought to interact with abandontransaction in some way?

@sdaftuar

Looks pretty good to me overall, though there's one bug here (after #8850) that needs to be fixed, plus some doc fixes to reflect the changes that have been made in this PR from when it was first opened.

Also, I think bumpfee.py should be added to rpc-tests.py.

src/wallet/rpcwallet.cpp
+ "\nBumps the fee of a opt-in-RBF transaction.\n"
+ "This command requires that the transaction with the given txid is in the wallet.\n"
+ "This command will NOT add new inputs.\n"
+ "Fee must be high enough to pay a new relay fee.\n"
@sdaftuar
sdaftuar Dec 12, 2016 Contributor

I found this sentence, and the ones below ("This command will fail if fee is not high enough...") to be confusing. Perhaps some explanation of what automatic fee calculation will occur, unless overriden/modified by the options, and that if the resulting fee isn't high enough then the command will fail?

src/wallet/rpcwallet.cpp
+ "This command requires that the transaction with the given txid is in the wallet.\n"
+ "This command will NOT add new inputs.\n"
+ "Fee must be high enough to pay a new relay fee.\n"
+ "If tx has child transactions in mempool, the new fee must pay for them as well.\n"
@sdaftuar
sdaftuar Dec 12, 2016 Contributor

This line in the help text needs to be updated for the latest semantics (no descendants in mempool or wallet).

src/wallet/rpcwallet.cpp
+ "}\n"
+ "\nExamples:\n"
+ "\nBump the fee, get the new transaction\'s txid\n"
+ + HelpExampleCli("bumpfee", "<txid> <output>")
@sdaftuar
sdaftuar Dec 12, 2016 Contributor

<output> should not be here anymore.

src/wallet/rpcwallet.cpp
+
+ // calculate the old fee and fee-rate
+ CAmount nOldFee = wtx.GetDebit(ISMINE_SPENDABLE) - wtx.tx->GetValueOut();
+ int64_t txSize = GetVirtualTransactionSize((CTransaction)wtx);
@sdaftuar
sdaftuar Dec 12, 2016 Contributor

I think this cast isn't correct after #8580 was merged; looks like the correct way to do this is GetVirtualTransactionSize(*wtx->tx).

src/wallet/rpcwallet.cpp
+
+ // Now modify the output to increase the fee.
+ // If the output is not large enough to pay the fee, fail.
+ CMutableTransaction tx(wtx);
@sdaftuar
sdaftuar Dec 12, 2016 edited Contributor

I don't think we should rely on the CMerkleTx function

operator const CTransaction&() const { return *tx; }

that makes this work, as that function has a comment saying it should be removed. *wtx->tx here again I think?

src/rpc/client.cpp
@@ -112,6 +112,8 @@ static const CRPCConvertParam vRPCConvertParams[] =
{ "setnetworkactive", 0 },
{ "getmempoolancestors", 1 },
{ "getmempooldescendants", 1 },
+ { "bumpfee", 1 },
+ { "bumpfee", 2 },
@sdaftuar
sdaftuar Dec 12, 2016 Contributor

I think this line should be removed now that the output is no longer specified?

@mrbandrews
Contributor

Pushed a new commit addressing the recent feedback and rebased due to a conflict in rpc-tests.py.

@ryanofsky

Looks good. Lots of nitpicky comments, nothing major.

src/wallet/rpcwallet.cpp
+ "bumpfee \"txid\" ( options ) \n"
+ "\nBumps the fee of an opt-in-RBF transaction T, replacing it with a new transaction B.\n"
+ "An opt-in RBF transaction with the given txid must be in the wallet.\n"
+ "The command will not add new inputs.\n"
@ryanofsky
ryanofsky Dec 14, 2016 Contributor

Maybe say this will decrease and possibly remove the change output, but leave other outputs alone. Also could say explicitly that this will not change existing inputs.

src/wallet/rpcwallet.cpp
+ RPCTypeCheck(request.params, boost::assign::list_of(UniValue::VSTR));
+ uint256 hash;
+ hash.SetHex(request.params[0].get_str());
+ std::string strError;
@ryanofsky
ryanofsky Dec 14, 2016 Contributor

Unclear what good having this variable does. Seems like it's always just set in one line then thrown in the next line, making the code more verbose and causing unnecessary string copies.

src/wallet/rpcwallet.cpp
+ // optional parameters
+ int newConfirmTarget = nTxConfirmTarget;
+ CAmount totalFee = 0;
+ CAmount bumpfeeMaxTxFee = 0.01 * COIN;
@ryanofsky
ryanofsky Dec 14, 2016 Contributor

Why use 0.01 * COIN instead of the global maxTxFee? I think it would be worth explaining in a comment. Also, if 0.01 * COIN is a significant value, maybe it would be good to declare it in validation.h alongside similar sounding constants like HIGH_MAX_TX_FEE.

@MarcoFalke
MarcoFalke Dec 14, 2016 Member

Regardless, we don't want to use flops for CAmount.

(I know that in this case the compiler can do static analysis and figure out the exact result, but we should not promote bad practice; It is proved to break on 32 bit platforms when static code analysis does not catch the flop.)

src/wallet/rpcwallet.cpp
+ CFeeRate nOldFeeRate(nOldFee, txSize);
+
+ // use the user-defined payTxFee if possible, otherwise use smartfee / fallbackfee
+ CAmount nNewFee = 0;
@ryanofsky
ryanofsky Dec 14, 2016 Contributor

I would move this declaration down closer to where the variable is first used (line 2719 nNewFee = nNewFeeRate.GetFee(maxNewTxSize);

src/wallet/rpcwallet.cpp
+ // use the user-defined payTxFee if possible, otherwise use smartfee / fallbackfee
+ CAmount nNewFee = 0;
+ CFeeRate nNewFeeRate = payTxFee;
+ if (nNewFeeRate.GetFeePerK() == 0)
@ryanofsky
ryanofsky Dec 14, 2016 Contributor

This block of code seems very similar to existing code in CWallet::GetMinimumFee. If you tweaked that function to return the fee rate in addition to the fee, could that function be reused here?

src/wallet/rpcwallet.cpp
+ nNewFee = nNewFeeRate.GetFee(maxNewTxSize);
+
+ // if user set totalFee, use that instead
+ if (totalFee > 0) {
@ryanofsky
ryanofsky Dec 14, 2016 Contributor

Seems like if this condition is true, the work done above gets thrown away. Maybe that code could be moved into an else block so the sematics of the totalFee > 0 case are more straightforward.

src/wallet/rpcwallet.cpp
+ }
+
+ CAmount nDelta = nNewFee - nOldFee;
+ if (nDelta <= 0) // it should not be possible to have a negative delta at this point (an attempt to reduce the fee)
@ryanofsky
ryanofsky Dec 14, 2016 Contributor

Would suggest changing error code below to RPC_INTERNAL_ERROR if it's really not possible for this condition to happen at this point.

src/wallet/rpcwallet.cpp
+
+ CAmount nDelta = nNewFee - nOldFee;
+ if (nDelta <= 0) // it should not be possible to have a negative delta at this point (an attempt to reduce the fee)
+ throw JSONRPCError(RPC_INVALID_PARAMETER, "New fee must be higher than old fee");
@ryanofsky
ryanofsky Dec 14, 2016 Contributor

What should the user do when this case happens? Should they report a bug because it should not be possible for this condition to happen? Or should they just try passing in a high(er) totalFee value? I think it'd be good to clarify in the error message.

src/wallet/rpcwallet.cpp
+ throw JSONRPCError(RPC_INVALID_PARAMETER, "New fee must be higher than old fee");
+
+ // Fail if the tx has any descendants - check both the wallet and the mempool
+ if (pwalletMain->HasWalletSpend(hash))
@ryanofsky
ryanofsky Dec 14, 2016 Contributor

I think it would make sense to move this check up towards the top of this function near the pwalletMain / wtx checks, instead of leaving it down here after fee estimation & option parsing. As a user I wouldn't want to first resolve a bunch of errors about fees and options before finding out that the transaction I am trying to bump can't be bumped in the first place.

src/wallet/rpcwallet.cpp
+ throw JSONRPCError(RPC_MISC_ERROR, "Change output is too small to bump the fee");
+
+ // If the output would become dust, discard it (converting the dust to fee)
+ poutput->nValue = poutput->nValue - nDelta;
@ryanofsky
ryanofsky Dec 14, 2016 Contributor

Maybe use poutput->nValue -= nDelta syntax here (and nNewFee += poutput->nValue below). += and -= operators I think would make the code easier to read and less prone to typo errors.

src/wallet/rpcwallet.cpp
+
+ // If the output would become dust, discard it (converting the dust to fee)
+ poutput->nValue = poutput->nValue - nDelta;
+ if (poutput->nValue < poutput->GetDustThreshold(::minRelayTxFee)) {
@ryanofsky
ryanofsky Dec 14, 2016 Contributor

Think <= would be better than < here. Thinking in a theoretical situation where nValue was 0 and dust threshould was somehow also 0, it would be more consistent to remove the output than keep it.

src/wallet/rpcwallet.cpp
+ // sign the new tx
+ CTransaction txNewConst(tx);
+ int nIn = 0;
+ for (auto &it : tx.vin) {
@ryanofsky
ryanofsky Dec 14, 2016 Contributor

it isn't really a good name for this variable, since it's a direct reference to the input, and not an iterator. Maybe in or input would be a better name.

src/wallet/rpcwallet.cpp
+ CTransaction txNewConst(tx);
+ int nIn = 0;
+ for (auto &it : tx.vin) {
+ std::map<uint256, CWalletTx>::const_iterator mi = pwalletMain->mapWallet.find(it.prevout.hash);
@ryanofsky
ryanofsky Dec 14, 2016 Contributor

Maybe use auto here since you're already using it one line up.

src/wallet/rpcwallet.cpp
+ int nIn = 0;
+ for (auto &it : tx.vin) {
+ std::map<uint256, CWalletTx>::const_iterator mi = pwalletMain->mapWallet.find(it.prevout.hash);
+ if (mi != pwalletMain->mapWallet.end() && it.prevout.n < (*mi).second.tx->vout.size()) {
@ryanofsky
ryanofsky Dec 14, 2016 Contributor

Can write mi->second instead of (*mi).second.

src/wallet/rpcwallet.cpp
+ SignatureData sigdata;
+ if (!ProduceSignature(TransactionSignatureCreator(pwalletMain, &txNewConst, nIn, SIGHASH_ALL), scriptPubKey, sigdata))
+ throw JSONRPCError(RPC_WALLET_ERROR, "Can't sign transaction.");
+ tx.vin[nIn].scriptSig = sigdata.scriptSig;
@ryanofsky
ryanofsky Dec 14, 2016 Contributor

Would suggest replacing tx.vin[nIn] here with it or replacing the opposite way above to be consistent about how the input is referred to inside the loop.

src/wallet/rpcwallet.cpp
+ CWalletTx wtxBumped(pwalletMain, MakeTransactionRef(std::move(tx)));
+ wtxBumped.mapValue["replaces_txid"] = hash.ToString();
+ CValidationState state;
+ if (!pwalletMain->CommitTransaction(wtxBumped, reservekey, g_connman.get(), state)) {
@ryanofsky
ryanofsky Dec 14, 2016 Contributor

Maybe add || !state.IsValid() to the condition here or assert(state.IsValid()) as a sanity check below.

qa/rpc-tests/bumpfee.py
+ self.is_network_split=False
+ self.sync_all()
+
+ def create_fund_sign_send(self, node, outputs, feerate=0):
@ryanofsky
ryanofsky Dec 14, 2016 Contributor

self isn't being used here, would suggest making this a standalone function like submit_block_with_tx below.

qa/rpc-tests/bumpfee.py
+ self.sync_all()
+ assert_equal(self.nodes[0].getbalance(), Decimal('0.01'))
+
+ # create and bump an RBF transaction
@ryanofsky
ryanofsky Dec 14, 2016 Contributor

This test is long and it's not clear right away whether the different parts of it depend on each other. If it could be broken up in some way, I think it would be more readable and easier to work with in the future. E.g.

self.test_simple_bumpfee_succeeds(a1)
self.test_nonrbf_bumpfee_fails(a1)
self.test_notmine_bumpfee_fails(a0)
...
@mrbandrews
Contributor

Pushed a new commit addressing ryanofsky's feedback.

I removed the maxFee option. The rationale for this option was to have a more conservative tx fee limit for a bumpfee user, as the default maxTxFee (0.1 btc) is quite high and in particular, if bumping fee on a transaction with lots of descendants a user might end up paying a pretty high fee for the parent transaction alone (since the descendants would be removed). Since the new approach is to not allow bumpfee to be used on a tx with descendants at all, this rationale mostly goes away and I think the maxFee option is not worth it.

I moved the python test above 'nulldummy.py' in the pull-tester list for the purpose of avoiding a conflict.

@mrbandrews
Contributor

I got a travis failure I think due to the default transaction version=2, which caused a failure in the python test. I corrected the test, rebased, and force-pushed.

src/wallet/rpcwallet.cpp
+ "bumpfee \"txid\" ( options ) \n"
+ "\nBumps the fee of an opt-in-RBF transaction T, replacing it with a new transaction B.\n"
+ "An opt-in RBF transaction with the given txid must be in the wallet.\n"
+ "The command will not add new inputs or alter existing inputs.\n"
@luke-jr
luke-jr Dec 20, 2016 Member

I don't think we should guarantee this. (sorry for reposting)

@ryanofsky
ryanofsky Jan 4, 2017 Contributor

Thanks, I changed the wording to describe the current behavior but suggest it could be improved in the future.

@luke-jr luke-jr added a commit to bitcoinknots/bitcoin that referenced this pull request Dec 21, 2016
@sdaftuar @luke-jr sdaftuar + luke-jr [wallet] Add IsAllFromMe: true if all inputs are from wallet
Github-Pull: #8456
Rebased-From: 250137f
844f0e1
@luke-jr luke-jr added a commit to bitcoinknots/bitcoin that referenced this pull request Dec 21, 2016
@mrbandrews @luke-jr mrbandrews + luke-jr [RPC] bumpfee
This command allows a user to increase the fee on a wallet transaction T, creating a "bumper" transaction B.
T must signal that it is BIP-125 replaceable.
T's change output is decremented to pay the additional fee.  (B will not add inputs to T.)
T cannot have any descendant transactions.
Once B bumps T, neither T nor B's outputs can be spent until either T or (more likely) B is mined.

Includes code by @jonasschnelli and @ryanofsky

Github-Pull: #8456
Rebased-From: 15a7e3f
60d8e8c
@luke-jr luke-jr added a commit to bitcoinknots/bitcoin that referenced this pull request Dec 21, 2016
@mrbandrews @luke-jr mrbandrews + luke-jr [QA] python test for bumpfee
Github-Pull: #8456
Rebased-From: 722be4f
3645224
@luke-jr luke-jr added a commit to bitcoinknots/bitcoin that referenced this pull request Dec 21, 2016
@mrbandrews @luke-jr mrbandrews + luke-jr squashme: more edits
Github-Pull: #8456
Rebased-From: 5ec4f33
9111809
@luke-jr luke-jr added a commit to bitcoinknots/bitcoin that referenced this pull request Dec 21, 2016
@mrbandrews @luke-jr mrbandrews + luke-jr squashme: more edits 2
Github-Pull: #8456
Rebased-From: 2443193
8718402
@sdaftuar

A couple nits that you can take or leave, and one minor bug that should be fixed. Did some light testing (including review of the bumpfee.py test included here) and this otherwise looks good to me.

src/wallet/rpcwallet.cpp
#include "timedata.h"
+#include "ui_interface.h"
@sdaftuar
sdaftuar Jan 5, 2017 Contributor

I think this include is not needed.

@ryanofsky
ryanofsky Jan 6, 2017 Contributor

Thanks, removed.

src/wallet/rpcwallet.cpp
+ + HelpExampleCli("bumpfee", "<txid>")
+ );
+
+ RPCTypeCheck(request.params, boost::assign::list_of(UniValue::VSTR));
@sdaftuar
sdaftuar Jan 5, 2017 Contributor

nit: We're calling RPCTypeCheck multiple times, which I don't think should be needed. I believe we can just change this to RPCTypeCheck(request.params, boost::assign::list_of(UniValue::VSTR)(UniValue::VOBJ)) and remove the one at line 2675 below.

@ryanofsky
ryanofsky Jan 6, 2017 Contributor

Fixed.

src/wallet/rpcwallet.cpp
+ if (totalFee < minTotalFee)
+ throw JSONRPCError(RPC_INVALID_PARAMETER, strprintf("Invalid totalFee, must be at least oldFee + relayFee: %s", FormatMoney(minTotalFee)));
+ nNewFee = totalFee;
+ nNewFeeRate = CFeeRate(totalFee, txSize);
@sdaftuar
sdaftuar Jan 5, 2017 Contributor

I think this should be CFeeRate(totalFee, maxNewTxSize).

@ryanofsky
ryanofsky Jan 6, 2017 Contributor

Added a comment. I think this was probably intentional since it makes the fee rate check below more conservative.

@ryanofsky
ryanofsky Jan 6, 2017 Contributor

Now fixed (got this backward).

@sdaftuar
Contributor
sdaftuar commented Jan 6, 2017 edited

ACK b1a0620 (feel free to squash). EDIT: ACK pending resolution of #8456 (comment)

I believe all of @gmaxwell's comments in #8456 (comment) have been addressed, with the exception of being able to add new inputs to a transaction to bump the fee. I think that would be a fine improvement for the future, but I think this is still a useful feature even without it, so I don't think that should be a merge blocker. @gmaxwell -- what do you think?

@ryanofsky
Contributor

Squashed b1a0620 -> f4fa93b.

src/wallet/rpcwallet.cpp
+ SignatureData sigdata;
+ if (!ProduceSignature(TransactionSignatureCreator(pwalletMain, &txNewConst, nIn, SIGHASH_ALL), scriptPubKey, sigdata))
+ throw JSONRPCError(RPC_WALLET_ERROR, "Can't sign transaction.");
+ input.scriptSig = sigdata.scriptSig;
@morcos
morcos Jan 9, 2017 Contributor

@sdaftuar points out that this won't correctly update scriptWitness for witness transactions
Probably better to use the UpdateTransaction function

@ryanofsky
ryanofsky Jan 10, 2017 Contributor

Added UpdateTransaction in e4a7662, and after a lot of struggle, eventually got a test to work which creates a segwit transaction and makes sure scriptWitness is set.

The test also uncovered another bug in the TransactionSignatureCreator call above where the SIGHASH_ALL value was incorrectly being interpretted as a CAmount. This is also fixed in e4a7662.

src/wallet/rpcwallet.cpp
+ "If the change output is not big enough to cover the increased fee, the command will currently fail\n"
+ "instead of adding new inputs to compensate. (A future implementation could improve this.)\n"
+ "The command will fail if the wallet or mempool contains a transaction that spends one of T's outputs.\n"
+ "By default, the new fee will be calculated automatically using estimatefee/fallbackfee.\n"
@morcos
morcos Jan 9, 2017 Contributor

nit: mentioning fallbackfee is too much implementation detail here

@ryanofsky
ryanofsky Jan 10, 2017 Contributor

Removed in e4a7662.

src/wallet/rpcwallet.cpp
+ if (newConfirmTarget <= 0) // upper-bound will be checked by estimatefee/smartfee
+ throw JSONRPCError(RPC_INVALID_PARAMETER, "Invalid confTarget (cannot be <= 0)");
+ }
+ if (options.exists("totalFee")) {
@morcos
morcos Jan 9, 2017 Contributor

Should there be a check against providing both options

@ryanofsky
ryanofsky Jan 10, 2017 Contributor

Added in e4a7662.

qa/rpc-tests/bumpfee.py
+ # of the mempool, then invalidate the block so the rbf transaction will be
+ # put back in the mempool. this makes it possible to check whether the rbf
+ # transaction outputs are spendable before the rbf tx is confirmed.
+ block = submit_block_with_tx(rbf_node, rbftx)
@morcos
morcos Jan 9, 2017 Contributor

i'd abandon bumpid here so it doesn't accidentally rereplace rbfid if its reaccepted from the wallet

if later abandoning conflicts/replacements frees up the original to spend again then thats a change in behavior that can result in changing the test

@ryanofsky
ryanofsky Jan 10, 2017 Contributor

Added abandontransaction call in b0d4f93.

@morcos
Contributor
morcos commented Jan 9, 2017

ACK modulo comments above.

@jtimon
Contributor
jtimon commented Jan 10, 2017

Concept ACK, needs rebase.

@ryanofsky
Contributor

Rebased to handle named arguments.

qa/rpc-tests/bumpfee.py
@@ -213,6 +213,10 @@ def test_unconfirmed_not_spendable(rbf_node, rbf_node_address):
assert bumpid not in rbf_node.getrawmempool()
assert rbfid in rbf_node.getrawmempool()
+ # call abandontransaction to make sure the wallet does not add the bump
+ # transaction back into the mempool.
+ rbf_node.abandontransaction(bumpid)
@morcos
morcos Jan 11, 2017 Contributor

This should take place after submit_block_with_tx and before invalidateblock
That is the only time it is guaranteed that bumpid is not in the mempool.

@ryanofsky
ryanofsky Jan 11, 2017 Contributor

Oops, yes, I got confused about what the test was doing with the two transactions. Fixed and updated the comment in a1046e0.

@gmaxwell
Member

I believe this should make the new transaction non-RBF replaceable. (1) we currently won't replace it, (2) if someone made an original payment and the receiver is squaking because it's replaceable, a mechanism is needed to issue a non-replacable version. Since we won't replace it, we can do this in one step. Does this make sense?

@gmaxwell
Member

Hm. I expected this to change the walletrbf default.

src/wallet/rpcwallet.cpp
+ // retrieve the original tx from the wallet
+ assert(pwalletMain != NULL);
+ LOCK2(cs_main, pwalletMain->cs_wallet);
+ if (!pwalletMain->mapWallet.count(hash))
@gmaxwell
gmaxwell Jan 12, 2017 Member

I can't see where it checks if the wallet is unlocked. Am I missing it?

@morcos
morcos Jan 12, 2017 Contributor

yikes that seems like a pretty big oversight

@ryanofsky
ryanofsky Jan 12, 2017 Contributor

Added EnsureWalletIsUnlocked call in 59e425a

@laanwj
Member
laanwj commented Jan 12, 2017 edited

Hm. I expected this to change the walletrbf default.

I don't think we should change that last-minute for 0.14 (only mentioning that because this is tagged 0.14).

Edit: Concept ACK otherwise.

@ryanofsky ryanofsky added a commit to ryanofsky/bitcoin that referenced this pull request Jan 12, 2017
@ryanofsky ryanofsky Enable RBF transactions in wallet by default
Suggested by Gregory Maxwell <greg@xiph.org> in
bitcoin#8456 (comment)
700af94
@ryanofsky
Contributor

Created #9527 for changing the walletrbf default.

@ryanofsky
Contributor

Added replaceable option in 8d3cd28, to allow users to create replacement transactions that are not replaceable.

I ran into a lot of spurious errors related to fundrawtransaction in the bumpfee tests while trying to add a simple test for this. fundrawtransaction would sometimes chose different sized inputs than the small_output_fails and dust_to_fee tests were expecting, so I rewrote these tests in 13960f8 to manually create RBF transactions with the right inputs.

I also ran into problems with fundrawtransaction using up all the right-sized inputs before these tests could use them, so I changed the order of the tests so they would run before any fundrawtransaction calls.

I also ran into problems with fundrawtransaction sometimes chosing a higher fee for the RBF transaction than the test_rebumping test expected, causing the bumpfee command in the test to fail with an "Invalid totalFee, must be at least ..." error. I did not figure out what was causing this, but it seemed to somehow be related to the number of peer_node.sendtoaddress(rbf_node_address, 0.001) calls made during the test setup.

@sdaftuar
Contributor

re-ACK 8d3cd28

@morcos
Contributor
morcos commented Jan 12, 2017

tested ACK 8d3cd28

@mrbandrews @ryanofsky mrbandrews [RPC] bumpfee
This command allows a user to increase the fee on a wallet transaction T, creating a "bumper" transaction B.
T must signal that it is BIP-125 replaceable.
T's change output is decremented to pay the additional fee.  (B will not add inputs to T.)
T cannot have any descendant transactions.
Once B bumps T, neither T nor B's outputs can be spent until either T or (more likely) B is mined.

Includes code by @jonasschnelli and @ryanofsky
1d71fd0
@ryanofsky
Contributor

Squashed 8d3cd28 -> 1d71fd0

@instagibbs

tACK with random nits. If you were to make one change I think printing out the necessary flat fee on rejection(for whatever reason) would be a solid improvement.

src/wallet/rpcwallet.cpp
+ "By default, the new fee will be calculated automatically using estimatefee.\n"
+ "The user can specify a confirmation target for estimatefee.\n"
+ "Alternatively, the user can specify totalFee, or use RPC setpaytxfee to set a higher fee rate.\n"
+ "At a minimum, the new fee rate must be high enough to pay a new relay fee and to enter the node's mempool.\n"
@instagibbs
instagibbs Jan 13, 2017 edited Contributor

nit: maybe name the relay fee constant in case user wants to look it up.

@ryanofsky
ryanofsky Jan 13, 2017 Contributor

Added in 7c9ad69

+ "1. txid (string, required) The txid to be bumped\n"
+ "2. options (object, optional)\n"
+ " {\n"
+ " \"confTarget\" (numeric, optional) Confirmation target (in blocks)\n"
@instagibbs
instagibbs Jan 13, 2017 Contributor

nit: add n here as well as below

@ryanofsky
ryanofsky Jan 13, 2017 Contributor

Options style here is the same as the style used in fundrawtransaction (and it seems clearer without all the n's and extra punctuation).

src/wallet/rpcwallet.cpp
+ "2. options (object, optional)\n"
+ " {\n"
+ " \"confTarget\" (numeric, optional) Confirmation target (in blocks)\n"
+ " \"totalFee\" (numeric, optional) Total fee (NOT feerate) to pay, in satoshis\n"
@instagibbs
instagibbs Jan 13, 2017 Contributor

can you explain why user-chosen feerate isn't an option, but absolute fee?

@ryanofsky
ryanofsky Jan 13, 2017 Contributor

User chosen feerate is possible with settxfee, and could be added as another option here in the future. Absolute fee is a reasonable thing a user might want to set, probably determining it from the fee of the previous transaction.

src/wallet/rpcwallet.cpp
+ " be left unchanged from the original. If false, any input sequence numbers in the\n"
+ " original transaction that were less than 0xfffffffe will be increased to 0xfffffffe\n"
+ " so the new transaction will not be explicitly bip-125 replaceable (though it may\n"
+ " still be replacable in practice if it has unconfirmed ancestors which are\n"
@instagibbs
instagibbs Jan 13, 2017 edited Contributor

nit: or full RBF nodes relay it

@ryanofsky
ryanofsky Jan 13, 2017 Contributor

Referring to nodes that don't follow BIP-125? Maybe a more general note about these nodes would be helpful, but it seems like an odd thing to mention only in this context.

@luke-jr
luke-jr Jan 13, 2017 Member

"(though it may still be replacable in practice)" seems sufficient.

src/wallet/rpcwallet.cpp
+ hash.SetHex(request.params[0].get_str());
+
+ // retrieve the original tx from the wallet
+ assert(pwalletMain != NULL);
@instagibbs
instagibbs Jan 13, 2017 edited Contributor

nit: you already check that the wallet is available above, I believe unneeded.

@ryanofsky
ryanofsky Jan 13, 2017 Contributor

Removed in 7c9ad69

+ // (no point in bumping fee if we know that the new tx won't be accepted to the mempool)
+ // This may occur if the user set TotalFee or paytxfee too low, if fallbackfee is too low, or, perhaps,
+ // in a rare situation where the mempool minimum fee increased significantly since the fee estimation just a
+ // moment earlier. In this case, we report an error to the user, who may use totalFee to make an adjustment.
@instagibbs
instagibbs Jan 13, 2017 Contributor

if this is the intended user flow, we might want to inform them via the following error message by including a totalfee guess alongside the minimum required feerate with a suggestion to use that option.

@ryanofsky
ryanofsky Jan 13, 2017 Contributor

Added in 7c9ad69

+ // Mark new tx not replaceable, if requested.
+ if (!replaceable) {
+ for (auto& input : tx.vin) {
+ if (input.nSequence < 0xfffffffe) input.nSequence = 0xfffffffe;
@instagibbs
instagibbs Jan 13, 2017 Contributor

just set it

@morcos
morcos Jan 13, 2017 Contributor

I guess I don't see how it makes a difference, but it seems to me that the whole point of this PR is to change the transaction to the least degree possible. I don't know why someone would have chosen to set their sequence to 0xffffffff, but if they did, why should we change it for them here?

@luke-jr
luke-jr Jan 13, 2017 Member

Agreed with @morcos, leave this as-is.

src/wallet/rpcwallet.cpp
+ throw runtime_error(
+ "bumpfee \"txid\" ( options ) \n"
+ "\nBumps the fee of an opt-in-RBF transaction T, replacing it with a new transaction B.\n"
+ "An opt-in RBF transaction with the given txid must be in the wallet.\n"
@morcos
morcos Jan 13, 2017 Contributor

why does it need to be in the mempool?

@instagibbs
instagibbs Jan 13, 2017 Contributor

(weird, comments are in wrong order) it doesn't you're right.

@luke-jr
luke-jr Jan 13, 2017 Member

It says in the wallet, not the mempool...?

@TheBlueMatt

Only reviewed parts of the wallet.cpp stuff just yet, but figured I'd post it now.

+ // Ensure for now that we're not overwriting data
+ assert(wtx.mapValue.count("replaced_by_txid") == 0);
+
+ wtx.mapValue["replaced_by_txid"] = newHash.ToString();
@TheBlueMatt
TheBlueMatt Jan 13, 2017 Contributor

I'm super not convinced that this is sufficient. See all the places where isAbandoned is checked - we probably need to check for replacement in most, if not all, of the same places.

eg probably want to check in RelayWalletTransaction and possibly isSpent?

@ryanofsky
ryanofsky Jan 13, 2017 Contributor

This is being set specifically to change the behavior of AvailableCoins to avoid creating new transactions which spend outputs of rbf & bumpfee transactions, for reasons described in comments in AvailableCoins.

I'm actually not sure why we don't mark the transaction that is being replaced to be abandoned. But I don't think it should make any practical difference right now.

RelayWalletTransaction in the normal case should already skip this transaction because it conflicts with the bumpfee transaction and won't be in mempool.

IsSpent could perhaps be written to not consider inputs of a transaction that's been replaced to be spent, though in practice it doesn't make a difference because the bumpfee replacing transaction always spends the same inputs as the replaced transaction.

@sdaftuar
sdaftuar Jan 13, 2017 Contributor

Fee bumped should not imply abandoned -- it's entirely possible the original version of the transaction will confirm.

Changing IsSpent as you suggest ought to have no visible effect (since bumpfee spends all the same inputs, currently), yet I think it would be incorrect to make your change -- if we add smarter fee bumping behavior in the future, and it becomes possible to drop an input from the original tx in a replacement transaction, we would still want to treat the original input as spent until the transaction that spends it becomes conflicted (or the user explicitly abandons it).

In RelayWalletTransaction, I think it's more debatable whether to attempt relay of the original version, but since it would only succeed if somehow the feebumped version dropped out of your mempool, I think it's reasonable for the wallet to attempt to relay unless the user explicitly calls abandontransaction on it. We don't care which version gets mined, and if something is subtly wrong with the bumped transaction then we might as well attempt the original. The effect on tx relay bandwidth should be negligible.

Also -- I believe this is no different behavior than if we have conflicting transactions in our wallet that originate via some means other than bumpfee. For instance if someone sends funds to us and then fee bumps it themselves, we'll wind up with both transactions in our wallet, and we'll attempt to relay both, which I think is correct behavior.

@morcos
morcos Jan 13, 2017 Contributor

100% agree with sdaftuar. it would be incorrect to auto abandon the bumpee.

@TheBlueMatt
TheBlueMatt Jan 13, 2017 Contributor

Abandoned doesn't imply the original version of the transaction won't confirm, it just means "pretend this isnt here, unless it somehow gets confirmed"?

But, ok, fair point regarding IsSpent - I'm not convinced thats is definitely The Right Behavior to not allow a user to spend something they removed from their transaction, but this is the more conservative option, so have no problem with it for now.

re: RelayWalletTransaction: I think this absolutely should have a replacement check...if a transaction has been replaced it shouldn't make it back into our mempool without the other transaction also replacing it, sure, but belt-and-suspenders. Plus a user could, theoretically, restart with a higher min relay fee (or upgrade to a new version), and this could break.

@TheBlueMatt
TheBlueMatt Jan 13, 2017 Contributor

I'll withdraw my request after IRC discussion. I've been convinced that in some use-cases a user might prefer us to accept the old version of a transaction which has been bumped, so no need to change that.

@ryanofsky
ryanofsky Jan 13, 2017 Contributor

Long discussion in IRC about this beginning here: https://botbot.me/freenode/bitcoin-core-dev/msg/79332875/.

I think the conclusion is that it is preferable to keep the current code which attempts to add all replaced and replacing transactions to the mempool , because there are cases when there is a problem with the replacing tx (e.g. output too close to dust limit) that would cause it to get rejected from mempool, and where it would be preferable to relay the original replaced transaction.

Also, the risk of the wallet relaying both replaced and replaced transactions to peers is very low. It will not happen unless a CWallet::ResendWalletTransactions call somehow is made before the initial CWallet::ReacceptWalletTransactions call in CWallet::postInitProcess finishes.

src/wallet/wallet.cpp
+ wtx.mapValue["replaced_by_txid"] = newHash.ToString();
+
+ CWalletDB walletdb(strWalletFile, "r+");
+ walletdb.WriteTx(wtx);
@TheBlueMatt
TheBlueMatt Jan 13, 2017 Contributor

Hum, should we be ignoring the return value here? (We check it in some places, ignore it in others in wallet.cpp, but it seems wrong to ignore it).

@ryanofsky
ryanofsky Jan 13, 2017 Contributor

Will add a check.

@sdaftuar
sdaftuar Jan 13, 2017 Contributor

I think we can just clean this up in a subsequent PR? I agree that the differing behaviors are not good, but perhaps this is a small enough issue to not hold this up...

@ryanofsky
ryanofsky Jan 13, 2017 Contributor

Either way seems fine. Added a check and error print for now in f9c4007.

@TheBlueMatt
TheBlueMatt Jan 13, 2017 Contributor

I only brought it up because I'd prefer new code to check it...it seems bad to not do so.

@ryanofsky
ryanofsky Jan 13, 2017 Contributor

Agreed, check is added in f9c4007.

src/wallet/wallet.cpp
+ CWalletDB walletdb(strWalletFile, "r+");
+ walletdb.WriteTx(wtx);
+
+ NotifyTransactionChanged(this, originalHash, CT_UPDATED);
@TheBlueMatt
TheBlueMatt Jan 13, 2017 Contributor

Do we need/want a MarkDirty here, too? I think we might not need it, but it seems strange to not have it here when we have it in all the other similar things (conflicted, abandon, etc).

@ryanofsky
ryanofsky Jan 13, 2017 Contributor

Will look more, but I would want to find a specific case it is needed before adding MarkDirty here. I have another PR #9381 which eliminates MarkDirty from AddToWallet() and will get rid of this entire MarkReplaced() method, replacing it with an AddToWallet call.

@sdaftuar
sdaftuar Jan 13, 2017 Contributor

I don't think we should treat this differently from any other unconfirmed, conflicting transaction entering the wallet, so I don't believe this is necessary (and it seems confusing to me to add calls like that in places where we think that balances shouldn't be changing).

src/wallet/wallet.cpp
+ // intending to replace A', but potentially resulting in a scenario
+ // where A, A', and D could all be accepted (instead of just B and
+ // D, or just A and A' like the user would want).
+ if (nDepth == 0 && pcoin->mapValue.count("replaced_by_txid"))
@TheBlueMatt
TheBlueMatt Jan 13, 2017 Contributor

Obviously prefer to leave this in because belt-and-suspenders, but shouldn't this check be redundant with the !InMempool() check, two checks above?

@ryanofsky
ryanofsky Jan 13, 2017 Contributor

It's not completely redundant (see https://github.com/mrbandrews/bitcoin/blob/1d71fd0dab96430643b818ea993acea2aaa09256/qa/rpc-tests/bumpfee.py#L251 for the test which triggers this condition), but this is checking for a really esoteric case. See description above for all the gory details, but in this can happen when the replaced transaction A temporarily becomes part of a block instead of the replacing transaction B, and then gets added back to the mempool when there's a reorg.

@ryanofsky ryanofsky fixup! [RPC] bumpfee
Add check for WriteTx failure as suggested by @TheBlueMatt
in bitcoin#8456 (comment)
f9c4007
@luke-jr

Almost there... (if you could use braces more, that'd be nice, but not critical)

src/wallet/rpcwallet.cpp
+ bool replaceable = true;
+ if (request.params.size() > 1) {
+ UniValue options = request.params[1];
+ if (options.size() > 2)
@luke-jr
luke-jr Jan 13, 2017 Member

>3 now... but I don't know that we need an explicit check here. Seems like asking for bugs.

@ryanofsky
ryanofsky Jan 15, 2017 Contributor

Makes sense, removed in 23c389a.

src/wallet/rpcwallet.cpp
+ throw JSONRPCError(RPC_INVALID_PARAMETER, "confTarget and totalFee options should not both be set. Please provide either a confirmation target for fee estimation or an explicit total fee for the transaction.");
+ } else if (options.exists("confTarget")) {
+ newConfirmTarget = options["confTarget"].get_int();
+ if (newConfirmTarget <= 0) // upper-bound will be checked by estimatefee/smartfee
@luke-jr
luke-jr Jan 13, 2017 Member

Should be 1 now (as of #9239). Maybe make a const int in some header for this number...

@morcos
morcos Jan 14, 2017 Contributor

nothing bad happens if you call it with 1, i think it makes most sense to allow users to select 1 meaning as fast as possible, and then we just return the fastest estimate we're comfortable with

src/wallet/rpcwallet.cpp
+ if (newConfirmTarget <= 0) // upper-bound will be checked by estimatefee/smartfee
+ throw JSONRPCError(RPC_INVALID_PARAMETER, "Invalid confTarget (cannot be <= 0)");
+ } else if (options.exists("totalFee")) {
+ totalFee = options["totalFee"].get_int();
@luke-jr
luke-jr Jan 13, 2017 Member

get_int64

@ryanofsky
ryanofsky Jan 15, 2017 Contributor

Thanks, fixed in 23c389a.

src/wallet/rpcwallet.cpp
+ }
+
+ if (options.exists("replaceable") && !options["replaceable"].get_bool()) {
+ replaceable = false;
@luke-jr
luke-jr Jan 13, 2017 Member

Prefer to just do:

if (options.exists("replaceable")) {
    replaceable = options["replaceable"].get_bool();
}

So that if the default changes, this still works right.

@ryanofsky
ryanofsky Jan 15, 2017 Contributor

Good point, fixed in 23c389a.

+ if (poutput->nValue <= poutput->GetDustThreshold(::minRelayTxFee)) {
+ LogPrint("rpc", "Bumping fee and discarding dust output\n");
+ nNewFee += poutput->nValue;
+ tx.vout.erase(tx.vout.begin() + nOutput);
@luke-jr
luke-jr Jan 13, 2017 Member

So long as the user is explicitly setting a total fee, we should either fail when discarding dust, or at least document this behaviour in the RPC help.

@ryanofsky
ryanofsky Jan 15, 2017 edited Contributor

Added note to totalFee documentation in 4e1a040.

+ // Mark new tx not replaceable, if requested.
+ if (!replaceable) {
+ for (auto& input : tx.vin) {
+ if (input.nSequence < 0xfffffffe) input.nSequence = 0xfffffffe;
@luke-jr
luke-jr Jan 13, 2017 Member

Agreed with @morcos, leave this as-is.

src/wallet/rpcwallet.cpp
+ // along with an exception. It would be good to return information about
+ // wtxBumped to the caller even if marking the original transaction
+ // replaced does not succeed for some reason.
+ throw JSONRPCError(RPC_WALLET_ERROR, "Unable to mark the original transaction as replaced.");
@luke-jr
luke-jr Jan 13, 2017 Member

Maybe just add "errors" to the response below?

@ryanofsky
ryanofsky Jan 15, 2017 Contributor

Clarified the error message in 23c389a. If there are other examples of returning an "errors" key in a RPC response, I could change this to conform. But my preference would be to keep the exception (since this would be a sign of a serious problem that should not be ignored) and to just return more complete status information with the exception.

@ryanofsky

Implemented suggestions from @luke-jr in 23c389a, adding braces to all if statements, and making most of the other requested changes.

src/wallet/rpcwallet.cpp
+ throw runtime_error(
+ "bumpfee \"txid\" ( options ) \n"
+ "\nBumps the fee of an opt-in-RBF transaction T, replacing it with a new transaction B.\n"
+ "An opt-in RBF transaction with the given txid must be in the wallet.\n"
@ryanofsky
ryanofsky Jan 15, 2017 Contributor

morcos was responding to a review comment "painfully obvious, but needs to be in mempool as well :)" that is gone now.

src/wallet/rpcwallet.cpp
+ " be left unchanged from the original. If false, any input sequence numbers in the\n"
+ " original transaction that were less than 0xfffffffe will be increased to 0xfffffffe\n"
+ " so the new transaction will not be explicitly bip-125 replaceable (though it may\n"
+ " still be replacable in practice if it has unconfirmed ancestors which are\n"
@ryanofsky
ryanofsky Jan 15, 2017 Contributor

Reworded, adding "for example" in 23c389a. Just saying non-replaceable transactions are replaceable sounds mysterious. Maybe I'm less knowledgeable than the target user, but at least I would find it confusing.

src/wallet/rpcwallet.cpp
+ bool replaceable = true;
+ if (request.params.size() > 1) {
+ UniValue options = request.params[1];
+ if (options.size() > 2)
@ryanofsky
ryanofsky Jan 15, 2017 Contributor

Makes sense, removed in 23c389a.

src/wallet/rpcwallet.cpp
+ if (newConfirmTarget <= 0) // upper-bound will be checked by estimatefee/smartfee
+ throw JSONRPCError(RPC_INVALID_PARAMETER, "Invalid confTarget (cannot be <= 0)");
+ } else if (options.exists("totalFee")) {
+ totalFee = options["totalFee"].get_int();
@ryanofsky
ryanofsky Jan 15, 2017 Contributor

Thanks, fixed in 23c389a.

src/wallet/rpcwallet.cpp
+ }
+
+ if (options.exists("replaceable") && !options["replaceable"].get_bool()) {
+ replaceable = false;
@ryanofsky
ryanofsky Jan 15, 2017 Contributor

Good point, fixed in 23c389a.

+ if (poutput->nValue <= poutput->GetDustThreshold(::minRelayTxFee)) {
+ LogPrint("rpc", "Bumping fee and discarding dust output\n");
+ nNewFee += poutput->nValue;
+ tx.vout.erase(tx.vout.begin() + nOutput);
@ryanofsky
ryanofsky Jan 15, 2017 edited Contributor

Added note to totalFee documentation in 4e1a040.

+ // Mark new tx not replaceable, if requested.
+ if (!replaceable) {
+ for (auto& input : tx.vin) {
+ if (input.nSequence < 0xfffffffe) input.nSequence = 0xfffffffe;
@ryanofsky
ryanofsky Jan 15, 2017 Contributor

Ok, keeping as is to change the transaction as little as possible.

src/wallet/rpcwallet.cpp
+ // along with an exception. It would be good to return information about
+ // wtxBumped to the caller even if marking the original transaction
+ // replaced does not succeed for some reason.
+ throw JSONRPCError(RPC_WALLET_ERROR, "Unable to mark the original transaction as replaced.");
@ryanofsky
ryanofsky Jan 15, 2017 Contributor

Clarified the error message in 23c389a. If there are other examples of returning an "errors" key in a RPC response, I could change this to conform. But my preference would be to keep the exception (since this would be a sign of a serious problem that should not be ignored) and to just return more complete status information with the exception.

@TheBlueMatt
Contributor

I know other ways of calling getbalance are pretty broken (#8183), but I think we need to fix a bit of how bumpfee interacts with getbalance "". In testing I got getbalance "" to give me a negative number, which seems super shitty.

Additionally, we need documentation on how bumpfee interacts with listunspent. While I dont mind the change outputs from bumpfee not appearing in the results, this is likely to be surprising to users (I believe this only otherwise occurs if the transaction was abandoned), so needs documented.

@@ -2609,7 +2609,10 @@ UniValue bumpfee(const JSONRPCRequest& request)
"2. options (object, optional)\n"
" {\n"
" \"confTarget\" (numeric, optional) Confirmation target (in blocks)\n"
- " \"totalFee\" (numeric, optional) Total fee (NOT feerate) to pay, in satoshis\n"
+ " \"totalFee\" (numeric, optional) Total fee (NOT feerate) to pay, in satoshis.\n"
+ " In rare cases, the actual fee paid might be slightly higher than the specified\n"
@morcos
morcos Jan 16, 2017 Contributor

I'd vote against making this statement.
I think it should be a more general principle of our wallet that you might always lose unexpectedly something on the order of the dust threshold, rather than explicitly mention it every time we think it could happen. But if I'm outvoted, so be it..

@morcos
Contributor
morcos commented Jan 16, 2017

@TheBlueMatt I don't see how this PR has any affect on any of the getbalance calls. I think what you are seeing is pre-existing misbehavior for multiple spends of the same outputs being present in the wallet.

I went through all the calls to AvailableCoins (which is the only thing this PR changes) and I think the only open decision is deciding what we want the interaction with listunspent to be and documenting it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment