Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

nSequence-based Full-RBF opt-in #6871

Merged
merged 9 commits into from Nov 27, 2015

Conversation

petertodd
Copy link
Contributor

@petertodd petertodd commented Oct 22, 2015

Replaces transactions already in the mempool if a new transaction seen with a higher fee, specifically both a higher fee per KB and a higher absolute fee. Children are evaluated for replacement as well, using the mempool package tracking to calculate replaced fees/size efficiently. Transactions opt-in to transaction replacement by setting nSequence < maxint-1 on at least one input. (which no wallet currently does)

No first-seen-safe functionality, but that can easily be added as a separate pull if there's demand from wallet vendors.

If anyone feels like converting the tests to the internal python library used by the qa/rpc-tests, pull-reqs accepted.

@gmaxwell
Copy link
Contributor

gmaxwell commented Oct 22, 2015

Feedback I heard from wallet vendors previously was that FSS was burdensome (needed extra txins, and so less useful).

@petertodd
Copy link
Contributor Author

petertodd commented Oct 22, 2015

@gmaxwell Same feedback I've heard too.

@TheBlueMatt
Copy link
Contributor

TheBlueMatt commented Oct 22, 2015

Concept ACK.

1 similar comment
@btcdrak
Copy link
Contributor

btcdrak commented Oct 22, 2015

Concept ACK.

@dcousens
Copy link
Contributor

dcousens commented Oct 22, 2015

concept ACK

@dcousens
Copy link
Contributor

dcousens commented Oct 22, 2015

How does this work with CPFP, and descendent transactions (e.g a chained transaction that could have its based replaced?)

@petertodd
Copy link
Contributor Author

petertodd commented Oct 22, 2015

@dcousens What do you mean "how does this work" with CPFP? CPFP isn't implemented in Bitcoin Core, so there's nothing to affect.

Now, when replacing a tx with children, the fees and size of all children are taken into account before deciding to replace or not.

@dcousens
Copy link
Contributor

dcousens commented Oct 22, 2015

@dcousens What do you mean "how does this work" with CPFP? CPFP isn't implemented in Bitcoin Core, so there's nothing to affect.

I meant conceptually, sorry. CPFP is something that was on the roadmap AFAIK?

all children are taken into account before deciding to replace or not.

By taken into account, do you mean that you have to have a higher fee than all subsequent children to replace?

@petertodd
Copy link
Contributor Author

petertodd commented Oct 22, 2015

By taken into account, do you mean that you have to have a higher fee than all subsequent children?

Yes.

Only extremely sophisticated CPFP that does relaying of whole packages of transactions based on fees paid by children is affected by RBF, and no-one has any plans to actually implement that yet.

@petertodd petertodd force-pushed the 2015-10-rbf-with-opt-out branch from 17c0f4f to b233451 Compare Oct 22, 2015
@petertodd petertodd changed the title Full-RBF with opt-out nSequence-based Full-RBF opt-in Oct 23, 2015
@greenaddress
Copy link
Contributor

greenaddress commented Oct 23, 2015

Concept ACK

@rubensayshi
Copy link
Contributor

rubensayshi commented Oct 23, 2015

concept ACK

@laanwj laanwj added the Mempool label Oct 23, 2015
@jtimon
Copy link
Member

jtimon commented Oct 23, 2015

Concept ACK

// the replacements.
CFeeRate oldFeeRate(nConflictingFees, nConflictingSize);
CFeeRate newFeeRate(nFees, nSize);
if (newFeeRate <= oldFeeRate)
Copy link
Member

@sdaftuar sdaftuar Oct 23, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure this comparison makes sense. The existence of a low-fee-rate descendant doesn't make a transaction worse for miners, but it would cause the feerate to look worse in this comparison.

Copy link
Contributor Author

@petertodd petertodd Oct 23, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So, you mean the scenario where we have a high fee-rate transaction that is spent by one or more low fee-rate transactions? For instance suppose we have two transactions in our mempool: tx1a, 1KB w/ 1mBTC fee, which is spent by tx2, 10KB w/ 1mBTC fee. We get tx1b, 10KB w/ 2.1mBTC fee. Since the overall feerate of tx1b is higher than tx1a+tx2, it'll be accepted, even though a miner might have rather mined tx1a instead, ignoring tx2 (for now).

I agree that's less than optimal. If we make the assumption that there's always more demand for blockchain space than supply, it might be reasonable for the replacement logic criteria to be whether or not we're increasing the fee-rate of any subset of the mempool. (while still paying more fees than the replaced transactions)

Without taking CPFP into account, you could simply use the same kind of priority heap logic as in CreateNewBlock() on the list of transactions that would be replaced by the new transaction. You'd iterate through the heap from highest priority to lowest, stopping once you had found as many bytes worth of transactions as the candidate replacement. If the fee-rate of the replacement is higher than the fee-rate of those transactions, accept the replacement into the mempool.

To take CPFP into account... Thinking about it more the mempool package tracking is probably backwards from what we want. Right now it tracks descendants, when really what we want to know is "what's the total fee-rate if I mine this transaction, and all ancestors?" If we kept track of "packages" that way, we'd be able to do the comparison by determining if the total fee-rate of the new package created by the replacement is higher than the fee-rates of all the packages invalidated by it. I actually did some work on something along these lines a few years ago, though didn't finish it - the implementation is a lot simpler now that we have strict ancestor limits. (when limiting, just throw away the tx associated with the lowest fee-rate package, which is guaranteed to have no descendants)

For now though I'd be inclined to merge this PR as-is, as all the above options are pretty complex. I also don't see any way this code could be used in an DoS attack.

Copy link
Member

@sdaftuar sdaftuar Oct 26, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right, the distinction between ancestor and descendant package is what I was getting at.

Descendant packages are I think the right thing to use for mempool limiting. I don't follow what you're saying about a limiting algorithm that uses fee with ancestors -- it's entirely possible that the worst thing under that sort would have descendants in the mempool.

(FYI I have a branch that implements ancestor packages. I might propose merging it at some point if it looks like we can take advantage of it in the mining code, but I'm not ready to advocate that now.)

Anyway in this code, we are comparing the feerate of the replacing transaction (with uncalculated/unknown fee rate including its ancestors) to an estimate of the feerate of the descendants of all the conflicting transactions. This strikes me as incorrect on both fronts; by not putting any bounds on the ancestor fee rate, we might be accepting a replacement transaction that is unlikely to confirm anytime soon. On the other hand, by looking at feerate with children (and overcounting those children, at least in the current implementation), we might be making it so that miners would prefer the original transactions to the replacing one.

I don't know that either of these issues constitutes an attack, but I do think it's useful to try to help users avoid shooting themselves in the foot, say by accidentally adding an input that is part of a long unconfirmed chain (causing the replacing transaction to be worse), and to give miners code that doesn't require further optimization to do the economically rational thing.

So with that in mind, how about this:

  • Require that any new inputs that show up in the replacing transaction be already confirmed. In the future, if we do merge something like ancestor package tracking and better mining code, we could update this test appropriately.
  • Require that for each entry E in setConflicts,
    feerate(replacing transaction) > max(feerate of E, feerate of E with descendants). That doesn't completely eliminate the possibility that it could be somehow worse for a miner to accept the new transaction, but it eliminates some degenerate cases (where a high fee rate transaction is dragged down by lower fee rate transactions) and is easy to calculate.

Copy link
Contributor Author

@petertodd petertodd Oct 30, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed both these issues.

I decided not to do the max() version of this, as I think the requirement that the new fees > total-replaced-fees is sufficient; might help to get txs unstuck in some cases, and non-CPFP miners aren't evaluating that anyway.

@instagibbs
Copy link
Member

instagibbs commented Oct 26, 2015

Concept ACK. FSS is a pain.

# Make sure mining works
mempool_size = 1
while mempool_size:
cls.proxy.generate(1)
Copy link
Member

@sdaftuar sdaftuar Oct 26, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When I check out the commit referenced in the README and run the test, I get an error:
AttributeError: 'Proxy' object has no attribute 'generate'

I think this function is not defined in the Proxy class? Adding it in the appropriate place in python-bitcoinlib seems to fix it.

Copy link
Member

@instagibbs instagibbs Oct 26, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's not included, no. You can just do a "call" instead.

proxy.call("generate", 1)

Copy link
Contributor Author

@petertodd petertodd Oct 30, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed.

I forgot to add generate() when python-bitcoinlib dropped support for calling RPC commands implicitly; replaced with .call() so as to continue to use the official v0.5.0 release rather than git master.

@petertodd
Copy link
Contributor Author

petertodd commented Nov 4, 2015

Added @sdaftuar's changes from petertodd#4 (comment)

@dcousens
Copy link
Contributor

dcousens commented Nov 5, 2015

Great work @sdaftuar on the added tests, once-over ACK

@btcdrak
Copy link
Contributor

btcdrak commented Nov 5, 2015

needs rebase

@gmaxwell gmaxwell added this to the 0.12.0 milestone Nov 5, 2015
@laanwj
Copy link
Member

laanwj commented Nov 10, 2015

Concept ACK

@jgarzik
Copy link
Contributor

jgarzik commented Nov 10, 2015

concept ACK

Previously was using the system-wide python-bitcoinlib, if it existed,
rather than the local copy that you check out in the README.
REJECT_INSUFFICIENTFEE, "insufficient fee");
}
}

// Check against previous transactions
// This is done last to help prevent CPU exhaustion denial-of-service attacks.
if (!CheckInputs(tx, state, view, true, STANDARD_SCRIPT_VERIFY_FLAGS, true))
Copy link
Member

@jtimon jtimon Nov 23, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wouldn't it be better to put the new code (from L987) after this check instead of before?
It would avoid doing further mempool validations for transactions that are going to be rejected by this check anyway.
I know there can be transactions that would pass this check but be rejected as a replacement, but this check already has the inputs of the transaction cached, so it seems cheaper than the RBF logic and it makes sense to me to do it first.
Note that this minor nit wouldn't change the total diff (although it can be done later with other changes too [for example, a rebased #6445 ]).

Copy link
Contributor Author

@petertodd petertodd Nov 25, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You mean after the CheckInputs() line? The RBF code is just a bunch of pointer following of in-memory data, with limited depth and breadth, so it shouldn't be expensive code to run - remember that the mempool has tx fee and size data in the CTxMemPool structs.

Copy link
Member

@jtimon jtimon Nov 25, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Never mind, I misread this line for AreInputsStandard().

@dcousens
Copy link
Contributor

dcousens commented Nov 26, 2015

Is a higher sequence number still preferred?
In the case of a fee tie?

specifically both a higher fee per KB and a higher absolute fee

Or only the higher fee?

What is the behaviour if a transaction is broadcast with the sequence number 0xffffffff after a transaction was already found that was 'opt-in'?

@petertodd
Copy link
Contributor Author

petertodd commented Nov 26, 2015

@dcousens Just higher fee. Ties get rejected to avoid them being used as a way to waste bandwidth.

If a nSequence > maxint-2 transaction is broadcast it is subject to the same rules as any other replacement; it won't get accepted without paying a (sufficiently) higher fee. That said, if it is accepted further replacements will be rejected. The replacement behavior is stateless, acting only on what is in the mempool right now.

@dcousens
Copy link
Contributor

dcousens commented Nov 26, 2015

Thanks for clarifying that @petertodd 👍

@kristovatlas I wonder if the 'default' sequence number when opting into RBF should just be 0 then?
Just thinking of the privacy implications.

@petertodd
Copy link
Contributor Author

petertodd commented Nov 26, 2015

@dcousens nSequence=0 makes sense from the perspective of https://github.com/bitcoin/bips/blob/master/bip-0068.mediawiki too.

@laanwj laanwj merged commit 63b5840 into bitcoin:master Nov 27, 2015
laanwj added a commit that referenced this pull request Nov 27, 2015
63b5840 Fix usage of local python-bitcoinlib (Peter Todd)
16a2f93 Fix incorrect locking of mempool during RBF replacement (Peter Todd)
97203f5 Port test to rpc-test framework (Suhas Daftuar)
20367d8 Add test for max replacement limit (Suhas Daftuar)
73d9040 Improve RBF replacement criteria (Suhas Daftuar)
b272ecf Reject replacements that add new unconfirmed inputs (Peter Todd)
fc8c19a Prevent low feerate txs from (directly) replacing high feerate txs (Peter Todd)
0137e6f Add tests for transaction replacement (Peter Todd)
5891f87 Add opt-in full-RBF to mempool (Peter Todd)
@kristovatlas
Copy link

kristovatlas commented Nov 27, 2015

@dcousens defaulting to nSequence=0 for the purpose of reducing wallet client fingerprintability makes sense to me.

braydonf pushed a commit to braydonf/btcjs that referenced this pull request Dec 9, 2015
- Useful for bidding transactions as described in: https://bitpay.com/chaindb.pdf
- Reference: nSequence-based opt-in: bitcoin/bitcoin#6871
laanwj added a commit to laanwj/bitcoin that referenced this pull request Feb 24, 2016
Continues "Make logging for validation optional" from bitcoin#6519.

The idea there was to remove all ERROR logging of rejected transaction,
and move it to one message in the class 'mempoolrej' which logs the
state message (and debug info). The superfluous ERRORs in the log
"terrify" users, see for example issue bitcoin#5794.

Unfortunately a lot of new logging was introduced in bitcoin#6871 (RBF) and
 bitcoin#7287 (misc refactoring). This pull updates that new code.

// Save these to avoid repeated lookups
setIterConflicting.insert(mi);

Copy link
Contributor

@rebroad rebroad Aug 25, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The following was later removed by #7594

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet