Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add chapter about payment batching #18

Merged
merged 1 commit into from Mar 1, 2019

Conversation

@harding
Copy link
Contributor

harding commented Feb 5, 2019

Closes #10

@jnewbery

This comment has been minimized.

Copy link
Contributor

jnewbery commented Feb 22, 2019

Dave, I'm sorry I haven't reviewed this yet. I plan to do so next week!

@moneyball

This comment has been minimized.

Copy link

moneyball commented Feb 22, 2019

Copy link

moneyball left a comment

Excellent chapter!!

consolidated inputs using payment batching and achieves the best-case
efficiency described above.

If we assume that consolidation transactions will pay only 20% of the

This comment has been minimized.

Copy link
@moneyball

moneyball Feb 27, 2019

Is there a reference, model, or dashboard that we can link to that supports the assumption of 20%. Sounds reasonable to me but would be further strengthened with data supporting it. Perhaps https://statoshi.info/dashboard/db/fee-estimates ?

This comment has been minimized.

Copy link
@harding

harding Feb 27, 2019

Author Contributor

I could probably generate one, but Statoshi's record of fee estimates is a good one and supports the conclusion (estimatesmartfee on my node says waiting 144 blocks is 1/10th the cost of waiting 6 blocks). Instead of linking it here, however, I think perhaps we should just put the relevant information in the chapter about consolidations and then link to that.

a different fee provided for each option. For example:

[X] Free withdrawal (payment sent within 6 hours)
[ ] Immediate withdrawal (withdrawal fee 0.123 mBTC)

This comment has been minimized.

Copy link
@moneyball

moneyball Feb 27, 2019

We haven't discussed personal preferences for units yet, but I'm becoming a fan of using sats - do you have strong feelings? An alternative here is 12.3k sats. For reference: https://twitter.com/moneyball/status/1099081824452530178

This comment has been minimized.

Copy link
@harding

harding Feb 27, 2019

Author Contributor

Ideally we could use a made-up unit like bogocoin so that our documentation doesn't become outdated each time the price swings 10x either way. But if we're going to use an actual denomination, I leave it to you and John (and Mike if he wants) to decide what to use. As long as it isn't something ridiculous like tonal, I'll conform.

problems that will need to be addressed by any service using the
technique.

### Delayed gratification

This comment has been minimized.

Copy link
@moneyball

moneyball Feb 27, 2019

Do we want to speak to different triggering methods for when to determine a transaction is ready to be broadcast? Factors include:

  1. number of payments
  2. longest allowable delay for a payment
  3. if there are 2 tiers of payments (urgent and low fee) then a flag indicating there is an urgent payment in the queue
  4. matching input allowing for changeless output [just added this as i read it later in this chapter...]

The logic could look something like:
while (
payments.count() < N &&
current_time - payments[0].queued_time < longest_allowable_delay &&
!urgent_payment)
{
// keep waiting for more payments to batch
}

This comment has been minimized.

Copy link
@harding

harding Feb 27, 2019

Author Contributor

Really, the algorithm I'd recommend is an hourly/daily cronjob that just sends all pending payments, with urgent payments being sent immediately in an independent transaction.

This comment has been minimized.

Copy link
@moneyball

moneyball Feb 27, 2019

Ok, do you think we should at minimum outline that simple algorithm as an example simple algorithm?

I'm still keen on writing something like what I have, even if not describing an algorithm, at least stating the factors that one might consider when designing an algorithm (ie 1-4 plus any others that I may have missed).

This comment has been minimized.

Copy link
@harding

harding Feb 27, 2019

Author Contributor

Adding a line about the cronjob. If the periodicity of the cron job is less than or equal to the maximum allowable delay, it takes care of concern 1 and 2. If urgent payments are sent independently, that takes care of 3. Concern 4 isn't directly addressed by the cronjob; on reconsideration, maybe I should remove the parts about changless transactions and just push consolidations.

An additional nice benefit of a cronjob is that you can provide your customers with predicatability. E.g. "Withdrawals every hour on the hour."

out the company payroll for every employee at the same time), many
services primarily send money to users when those users make a
withdrawal request. In order to batch payments, the service must get
the user to accept that their payment will not be sent immediately---it

This comment has been minimized.

Copy link
@moneyball

moneyball Feb 27, 2019

For high frequency spenders such as large exchanges, this is not generally true, as they'd naturally be able to batch 10+ payments per block, right? Should we call that out explicitly - something like if a service is doing on the order of more than 1K payments per day, they'll likely be able to gain efficiencies with batching while rarely delaying any customer payment?

This comment has been minimized.

Copy link
@harding

harding Feb 27, 2019

Author Contributor

Although you're correct that batching will not create a delay in receiving 1-confirmation payments, it'll still be a delay for users who expect notification of unconfirmed transactions (unless they using rolling RBF updates). See also this old tweet of mine for some color: https://twitter.com/hrdng/status/739825949466845185

This comment has been minimized.

Copy link
@jnewbery

jnewbery Feb 27, 2019

Contributor

This does actually introduce a delay, even for frequent spending. We always expect the first block to be mined 10 minutes after a transaction is transmitted. That means that if an exchange holds on to a withdraw orderfor five minutes to batch with other payments before sending, then the first expected block is 15 minutes after the order is received. There's no way to avoid this - every second you hold on to a transaction before transmitting adds a second to the expected time that it'll be mined.

The difference is that if an exchange is sending out payments frequently, it can benefit a bit more by batching:

  1. if it's sending a payment every minute, it could send a batch every ten minutes. No payment would have more than 10 minutes additional delay added, but on average, the exchange would save 75% in the best case.
  2. if it's sending a payment every thirty minutes and it 'batched' every ten minutes, then the payment would have 10 minutes of delay added, but on average each transaction would only contain one payment, so no savings would be made.

We can take this further. If Bitcoin blockspace does become competitive, then frequent spenders could use batching not to save fees, but to increase priority (by attaching the same total fee as would have been used without batching, the feerate increases). In that model, there would be an introduced delay before the transaction is broadcast, but we would expect it to confirm in fewer blocks.

This comment has been minimized.

Copy link
@moneyball

moneyball Feb 27, 2019

Good stuff! I think it'd be valuable to add to the chapter.

currently-available coinjoin implementation is fully compatible with the
needs of payment batching.

## Possible inability to fee bump

This comment has been minimized.

Copy link
@moneyball

moneyball Feb 27, 2019

This is a good description of this problem, but I am left wondering what the probability is of this happening. If the probability is that this would occur daily for a service, I suspect they wouldn't be keen on implementing this optimization. However, it is likely low probability, like probably never or only when a disgruntled customer wants to target a service to attack them.

Can we at least share the Core limits that have been set, and perhaps other guidance as to the probability of this naturally occurring or the difficulty of an intentional attack?

This comment has been minimized.

Copy link
@harding

harding Feb 27, 2019

Author Contributor

I am left wondering what the probability is of this happening.

If you're Coinbase and you send batches once an hour, the probability is pretty close to 1.0. :-)

Can we at least share the Core limits that have been set, and perhaps other guidance as to the probability of this naturally occurring or the difficulty of an intentional attack?

I will mention that here, although the main limit (100,000 vbytes per tx) is mentioned elsewhere in the chapter. You're likely to hit the limit a bit before you get to 3,000 payments in one tx, which is easily obtainable by large exchanges sending payments once an hour or mid-sized exchanges sending once a day.

The other limits are 25 unconfirmed ancestor or descendent transactions. If you pay 1,000 users in a batch, you only need 2.5% of them to create a single unconfirmed respend to hit that limit. Alternatively, if any one of those 1,000 users pays one of your competitors (e.g. a trader moving funds from exchange A to exchange B), your competitor can spend that unconfirmed tx 24 times to screw you up (and they could rightfully claim that they did nothing wrong, as spending unconfirmed txes is explicitly allowed by Bitcoin Core and has many real use cases).

This comment has been minimized.

Copy link
@moneyball

moneyball Feb 27, 2019

This is great - I think it'd be valuable to readers to see this in the chapter. We discovered in meetings with Optech members that they were unaware of the 25 unconfirmed limit, and even @jnewbery had to go look it up to confirm his understanding (ie I don't believe it is broadly known).

RBF is typically promoted as a strategy for increasing the fees paid in
a transaction so that it confirms sooner, but it also allows for adding
outputs to a transaction, making it suitable for adding even more
payments to a previously-sent transaction.

This comment has been minimized.

Copy link
@moneyball

moneyball Feb 27, 2019

How much risk is there that when fee bumping TX1 (which includes payment P1), with TX2 (w/ P1 and P2), that TX1 gets mined instead of TX2 (either due to a timing issue or bizarre miner behavior choosing the lower fee TX). The service would need to monitor this and realize P2 needs to be resent. Are there other reasons the service would already be doing such monitoring or would this be an additional burden added if using RBF combined with batching?

This comment has been minimized.

Copy link
@harding

harding Feb 27, 2019

Author Contributor

How much risk is there that when fee bumping TX1 (which includes payment P1), with TX2 (w/ P1 and P2), that TX1 gets mined instead of TX2 (either due to a timing issue or bizarre miner behavior choosing the lower fee TX).

If txes take 15 seconds to propagate to miners, the risk is about 2.5%:

In [1]: from math import exp

In [2]: 1-exp(-15/600)
Out[2]: 0.024690087971667385

I'll add a note here about the need to specifically monitor for which payments were confirmed. Good catch, thanks!

This comment has been minimized.

Copy link
@harding

harding Feb 27, 2019

Author Contributor

Oh, goodness. While writing this, I just realized that it complicates things significantly. If you send tx0 with {p1,p2}, replace it with tx0' with {p1,p2,p3}, have tx0 confirmed, and finally send tx1 with {p3}---you have to make sure tx1 spends the change of tx0 or a reorg could allow confirmation of both tx0' and tx1.

Maybe this should be removed too. I'm not sure the gain is worth the complication. Also, ISTR that Electrum has this behavior implemented; I wonder whether they handle it safely.

This comment has been minimized.

Copy link
@moneyball

moneyball Feb 28, 2019

Very good discovery! FYI I've reached out to Electrum on #electrum asking how they implemented RBF batching in 3.3.

This comment has been minimized.

Copy link
@moneyball

moneyball Feb 28, 2019

ThomasV confirmed on #electrum that Electrum does not automatically create and send tx1 so would not fall into the trap identified.

## Recommendations summary

1. Try to create systems where your users and customers don't expect
their payments immediately but are willing to waiting for some time

This comment has been minimized.

Copy link
@moneyball

moneyball Feb 27, 2019

"willing to wait" ?

x.payment_batching/payment_batching.md Show resolved Hide resolved
transaction to contain only a single input.

4. Don't depend on being able to fee bump the batched payments. This
means using a high-enough feerate on the initial transaction to

This comment has been minimized.

Copy link
@moneyball

moneyball Feb 27, 2019

This is a good recommendation - should it also explicitly be called out in the original description above?

Core's `estimatesmartfee` RPC.

5. Optionally, look for opportunities to send the batched payments
without a change input.

This comment has been minimized.

Copy link
@moneyball

moneyball Feb 27, 2019

change output?

Copy link
Contributor

jnewbery left a comment

This is great Dave. Thanks (and sorry again for taking so long to review!)


It's often possible to add more outputs to a transaction without
increasing how many inputs it contains, meaning a single transaction can
increase the number of people it pays faster than it proportionally

This comment has been minimized.

Copy link
@jnewbery

jnewbery Feb 27, 2019

Contributor

I think the words 'pays faster' here might trip people up on their first reading. The sentence is completely correct in meaning (tx size grows sublinearly against number of payments), but I think it could be phrased in a way that's a bit simpler.

I'm not sure if this is any better:

It's often possible to add more outputs to a transaction without increasing how many inputs it contains, meaning that combining many payments into a single transaction is smaller than making many transactions with a single payment each.

This comment has been minimized.

Copy link
@harding

harding Feb 27, 2019

Author Contributor

I spent a few minutes trying to rework it, but it's an unnecessary embellishment and so shouldn't be there unless it's really good (which it's not). I ended up deleting it, skipping straight to "This chapter explains..." and merging the two intro paragraphs together. It's much more direct now.

This comment has been minimized.

Copy link
@jnewbery

jnewbery Feb 28, 2019

Contributor

Oh that's a shame. I liked the introductory sentence. I just thought the words "pays faster" weren't quite right.

As of February 2019, payment batching is used by multiple popular
Bitcoin services (mainly exchanges), is available as a built-in feature
of many wallets (including Bitcoin Core), and should be easy to
implement in custom wallets and payment-sending solutions. On the

This comment has been minimized.

Copy link
@jnewbery

jnewbery Feb 27, 2019

Contributor

I think I'd prefer these to be described as 'things to consider' rather than 'downsides'. Similarly, I think the section below (## Problems should be renamed ## Considerations or similar). Similar to your good feedback on my PR (#5 (comment)), I wouldn't want people to read this chapter and think that batching wasn't a good idea because of the downsides.

This comment has been minimized.

Copy link
@harding

harding Feb 28, 2019

Author Contributor

I made a few changes based on this feedback, but not as much as you asked. I think these are clearly downsides, but elsewhere in the document I've labeled them in the most prominent position as "concerns" rather than "problems" (while still using "problem" in more casual use). I'm happy to discuss further.


Realistically, the more a transaction spends, the more likely it is to
need additional inputs. This doesn't prevent payment batching from
being useful, although it does reduce its effectiveness. For example,

This comment has been minimized.

Copy link
@jnewbery

jnewbery Feb 27, 2019

Contributor

I like the analysis of the 'tough case', but I think you should consider just having the best case and typical case here, and then adding the tough case in a subsection later. The reason that batching is gives less relative savings for a user that receives a lot of small payments is that txins will dominate their transactions. That's true whether they batch or not. Note however that they still receive the same absolute savings from batching. Any payment included in a batch saves them one change output.

I also think moving the tough case makes sense since it will only apply to a small minority of readers. As you point out, for most people, the average size of txins will roughly equal the average size of txouts.

This comment has been minimized.

Copy link
@harding

harding Feb 28, 2019

Author Contributor

Separating related information goes against my writing instincts. I'll just delete the tough case altogether.

P2WSH), the number of vbytes used to spend them are even larger, so the
savings rate will be higher.

## Problems

This comment has been minimized.

Copy link
@jnewbery

jnewbery Feb 27, 2019

Contributor

See note above. I prefer "implementation considerations" or similar.

problems that will need to be addressed by any service using the
technique.

### Delayed gratification

This comment has been minimized.

Copy link
@jnewbery

jnewbery Feb 27, 2019

Contributor

"Delayed gratification" is a fun subsection title. I think "Delayed payments" is a more precise but perhaps drier alternative. Depends what tone you're going for.

templates) by closely matching the total value of the inputs and outputs
in the transaction, plus fees, so that there's no need to return any
change back to the spender's wallet. This is also a perfectly efficient
way to consolidate inputs as you decrease the total number of your

This comment has been minimized.

Copy link
@jnewbery

jnewbery Feb 27, 2019

Contributor

By 'decrease the total number of your inputs', do you mean decrease the number of UTXOs in your wallet?

Changeless transactions always decrease overall transaction size, but
because the efficiency of payment batching comes from spreading the
fixed costs of the transaction across multiple receivers, the benefit
of changeless transactions is weakened (in percentage terms) the more

This comment has been minimized.

Copy link
@jnewbery

jnewbery Feb 27, 2019

Contributor

It took me a while to understand what your point was here. I think you're saying: If you're batching, then implementing change avoidance saves one change output per n payments. If you're not batching, then you save one change output for every payment. The (relative) benefit from implementing change avoidance is therefore lower if you've already implemented payment batching.

I suppose that's true, but I don't know how important it is and whether it needs to be emphasized so strongly.


Although the benefit may be small in percentage terms, the cost of
implementing changeless transactions may also be small. Your goal is to
find a single input you own whose value is equal to or slightly larger

This comment has been minimized.

Copy link
@jnewbery

jnewbery Feb 27, 2019

Contributor

Why just a single input? Presumably you could use multiple inputs to make up the value of all of the outputs and avoid change.

find a single input you own whose value is equal to or slightly larger
than the total value of the outputs you want to send. A simple
algorithm would be to wait until you have a minimum number of payments
to batch and check to see if you have an input that is equal to or

This comment has been minimized.

Copy link
@jnewbery

jnewbery Feb 27, 2019

Contributor

I like this. It means that payment batching should make change avoidance more achievable in many cases.

Note that if you're prepared to use multiple input, then this becomes a subset sum problem. There are many more solutions available than if you're just considering one input.


3. Within each time window, send all payments together in the same
transaction. Ideally, your prior consolidations should allow the
transaction to contain only a single input.

This comment has been minimized.

Copy link
@jnewbery

jnewbery Feb 27, 2019

Contributor

Again, I don't understand the point about 'single input' here. If you're paying during a period of relatively high fees, then you want to reduce the number of inputs, but in general I don't think it matters when you use up your UTXOs.

This comment has been minimized.

Copy link
@harding

harding Feb 28, 2019

Author Contributor

I don't understand your criticism here. Or, rather, I think I understand it, but it seems to be based on not understanding how consolidations reduce fees---something I know you understand and something that this article illustrates in conjunction with batching. So now I'm sure I'm missing something. Help please!

This comment has been minimized.

Copy link
@moneyball

moneyball Feb 28, 2019

I think @jnewbery is just saying that batching is still generally a big win even if multiple inputs are necessary to create the transaction, and even if UTXO consolidation is generally being performed.

This comment has been minimized.

Copy link
@jnewbery

jnewbery Feb 28, 2019

Contributor

ah. Sorry, I was confusing this comment about a single input with the comment in the change avoidance section about a single inputs (#18 (comment)).

I still think this comment about consolidation isn't necessary. I think UTXO consolidation is orthogonal to the benefits of batching, or am I missing something?

This comment has been minimized.

Copy link
@harding

harding Feb 28, 2019

Author Contributor

Had a good out-of-band discussion with @jnewbery about this and we decided to (1) leave this unchanged for now, (2) I'll write a chapter about consolidation next, and (3) we'll re-evaluate the consolidation-related parts of this chapter based on what we put in the consolidation chapter.

@harding

This comment has been minimized.

Copy link
Contributor Author

harding commented Feb 28, 2019

Pushed separate commits addressing @moneyball and @jnewbery feedback. Another separate commit removes the sections about RBF and changeless transactions per my comments #18 (comment) and #18 (comment)


For the typical case,
consolidation actually loses money when only making a single payment,
but when actually batching, it performs almost as well as the best case

This comment has been minimized.

Copy link
@jnewbery

jnewbery Feb 28, 2019

Contributor

micronit: avoid repeated 'actually' in this sentence.

![Screenshot of a possible transaction batch in a block explorer](img/batch-screenshot.png)

Note that transactions belonging to particular Bitcoin services are
often identifiable by experts even if they don't use payment

This comment has been minimized.

Copy link
@jnewbery

jnewbery Feb 28, 2019

Contributor

Thanks for calling me out on this. No, I haven't done any testing of OXT myself. I've just assumed that the Laurent's analysis is good.

Incidentally, he's got a good blog here: https://medium.com/@laurentmt . It'd be interesting to see if he's written anything about his methodology in exchange tx correlation.

Copy link
Contributor

jnewbery left a comment

A few small nits inline.

I liked the Combining batching with other opt-in scaling techniques section, and would like to re-add it, but perhaps we should merge this without it and then have a future PR to add that section.

the transactions they relay to prevent attackers from wasting bandwidth,
CPU, and other node resources. By yourself, you can easily avoid
reaching these limits, but the receivers of the payments you send can
respend that money in child transactions that become part of the

This comment has been minimized.

Copy link
@jnewbery

jnewbery Feb 28, 2019

Contributor

consider s/that money/the unspent outputs/ to emphasize that this is only a problem while the tx in unconfirmed.

payment spends their output to the point where fee bumping becomes
impossible, all the other receivers of that transaction are also affected.

As of Bitcoin Core 0.18 (April 2019), the limits are[^limits] that a

This comment has been minimized.

Copy link
@jnewbery

jnewbery Feb 28, 2019

Contributor

This notation isn't part of github flavored markdown and so doesn't work if I view the file in github (eg https://github.com/bitcoinops/scaling-book/blob/115d56423916b7e633237999fe7e4f495f4adaca/x.payment_batching/payment_batching.md).

This brings up a larger question of how we're planning to publish this book. Should it be part of the jekyll bitcoinops site, or somewhere different?

This comment has been minimized.

Copy link
@harding

harding Feb 28, 2019

Author Contributor

I'm changing this to an unlinked superscript ¹ for the sake of this PR.

I think making the draft book part of the site would be ideal as we already have tools there for linting and having people linking to it on our site improves pagerank (for what that's worth). When we've got all the content in place, we can add some build dependencies that will allow compiling the "final" book to PDF, ePub, and other formats if we want.


3. Within each time window, send all payments together in the same
transaction. Ideally, your prior consolidations should allow the
transaction to contain only a single input.

This comment has been minimized.

Copy link
@jnewbery

jnewbery Feb 28, 2019

Contributor

ah. Sorry, I was confusing this comment about a single input with the comment in the change avoidance section about a single inputs (#18 (comment)).

I still think this comment about consolidation isn't necessary. I think UTXO consolidation is orthogonal to the benefits of batching, or am I missing something?

@harding harding force-pushed the harding:2019-02-chapter-batching branch from 115d564 to 48905c5 Feb 28, 2019
@harding

This comment has been minimized.

Copy link
Contributor Author

harding commented Feb 28, 2019

Forced pushed a squash of previous fixup commits plus changes in response to @jnewbery's latest feedback. A branch with the old commits may be found here (I'm keeping it around in case we reconsider including the RBF and changeless sections later): https://github.com/harding/scaling-book/tree/save-chapter-batching-with-fixups

@jnewbery

This comment has been minimized.

Copy link
Contributor

jnewbery commented Mar 1, 2019

Merging. Looks great, Dave! Let's revisit Combining batching with other opt-in scaling techniques once you've written the consolidation chapter.

@jnewbery jnewbery merged commit dfa8d91 into bitcoinops:master Mar 1, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
3 participants
You can’t perform that action at this time.