Add newsletter #6 (2018-07-31)

_includes/articles/towns-xapo-consolidation.md

@@ -0,0 +1,83 @@
+{% comment %}<!--
+Copyright 2018 Anthony Towns
+-->{% endcomment %}
+
+As mentioned in [newsletter #3][newsletter 3], the past few months of low
+transaction fees makes it a great time to do UTXO consolidation!
+Consolidation has been one of a variety of activities [Xapo][Xapo] has been
+undertaking to be prepared for the next time fees spike like they did in the
+last few months of 2017.
+
+{% assign img1_label = "Plot of total Bitcoin UXTOs, January - July 2018" %}
+
+{:.center}
+![{{img1_label}}](/img/posts/utxo-consolidation-2018.png)<br>
+*{{img1_label}},
+source: [Statoshi](https://statoshi.info/dashboard/db/unspent-transaction-output-set?panelId=6&fullscreen&from=1514759562608&to=1532039707168)*
+
+[newsletter 3]: https://bitcoinops.org/en/newsletters/2018/07/10/#dashboard-items
+[Xapo]: https://www.xapo.com/
+
+The idea behind UTXO consolidation is essentially this: when your average
+outgoing payment is larger than your average incoming payment (or when they’re
+the same, but you’re batching outgoing payments), you’ll often have to combine
+many UTXOs in order to fund an outgoing transaction, which increases the size
+of your transactions and hence the fees you pay. By consolidating UTXOs in
+advance, you can combine inputs ahead of time, giving you more
+control over when most of those costs are incurred. If you can do it when
+fees are low, that lets you reduce those costs pretty substantially.
+
+For example, if you would have spent a dozen 2-of-3 multisig inputs at 100 s/B
+(satoshis per byte), that would cost around 360,000 satoshis; while if you
+consolidated those inputs beforehand at 2 s/B, and then spent the single
+consolidated input later at 100 s/B, your total cost for the two transactions
+is only about 41,000 satoshis: i.e. 87% less paid in fees. And if fees don’t
+rise the risk isn’t huge: if fees just sat at 2 s/B, you’d be spending 7,900
+satoshis across two transactions if you consolidated, rather than spending
+7,200 satoshis in a single transaction if you did nothing.
+
+Consolidation also gives an opportunity to update the addresses you use for
+your UTXOs, for example to roll keys over, switch to multisig, or switch to
+segwit or bech32 addresses. And reducing the number of UTXOs makes it easier to
+run a full node too, marginally improving Bitcoin’s decentralisation and
+overall security, which is always nice.
+
+Of course, one thing you really don’t want to have happen is for your
+consolidation transactions to somehow fill up the blockchain and cause fees to
+immediately start rising! There are two metrics to watch to avoid this risk:
+one is whether the [mempool][mempool] is full (which causes the minimum
+acceptable fee to rise), and the other is how much empty space there has been
+in recent blocks (which gives an indication of whether miners will accept more
+transactions at the minimum fee). Both these metrics have been very promising
+most of the time over the past few months: the mempool has regularly been close
+to empty, meaning the transactions paying as little as 1 s/B have been
+propagated to miners; and many blocks have not been full, meaning cheap
+consolidation transactions will get mined reasonably quickly rather than
+creating a backlog that will cause fees to rise.
+
+[mempool]: https://statoshi.info/dashboard/db/memory-pool?panelId=1&fullscreen&from=1509458400000&to=1531813659334&theme=dark
+
+The approach we took to actually doing the consolidation was to have a script
+that would select groups of small UTXOs and create a consolidation transaction
+spending them to a single pool address at a fee rate of 1.01 satoshis per byte.
+The script gradually feeds consolidation transactions into the network, so it
+doesn’t cause too large a spike in the mempool, and perhaps more importantly so
+we don’t risk having our transactions get dropped because they have low fees
+and the mempool has filled up. We triggered this manually when we were
+comfortable it wouldn't interfere with our operations, and when there didn’t seem
+to be much load on the Bitcoin network in general.
+
+All in all, this has worked out pretty well; we’ve reduced our UTXO count by
+something like [4 million UTXOs][4 million UTXOs] this year, and aside from
+some [concerned][redditors1] [redditors][redditors2], the cost to the network
+as a whole has been minimal, as has the cost to us.
+
+[4 million UTXOs]: https://www.oxt.me/entity/Xapo
+[redditors1]: https://www.reddit.com/r/BitcoinDiscussion/comments/8ocyc9/massive_consolidation_currently_underway/
+[redditors2]: https://www.reddit.com/r/Bitcoin/comments/8p3y5b/does_xapo_spamming_the_blockchain/
+
+{{include.hlevel | default: '##'}} Additional resources
+
+- [Techniques to reduce transaction fees: consolidation](https://en.bitcoin.it/wiki/Techniques_to_reduce_transaction_fees#Consolidation) - Bitcoin Wiki
+- [How to cheaply consolidate coins to reduce miner fees](https://en.bitcoin.it/wiki/How_to_cheaply_consolidate_coins_to_reduce_miner_fees) - Bitcoin Wiki
+- [What are some best practices regarding the usage of consolidations and fanouts?](https://bitgo.freshdesk.com/support/solutions/articles/27000044185-what-are-some-best-practices-regarding-the-usage-of-consolidations-and-fanouts-) - BitGo

_includes/references.md

@@ -1,9 +1,24 @@
+[Bitcoin Core 0.16.2]: https://bitcoincore.org/en/releases/0.16.2/
+[bitcoin.se]: https://bitcoin.stackexchange.com/
+[c-lightning]: https://github.com/ElementsProject/lightning
+
+{% comment %}<!-- BIPs in order lowest to highest -->{% endcomment %}
+[BIP157]: https://github.com/bitcoin/bips/blob/master/bip-0157.mediawiki
+
+{% comment %}<!-- old newsletters in date order earliest to latest -->{% endcomment %}
+[newsletter #0]: /en/newsletters/2018/06/08/
+[newsletter #1]: /en/newsletters/2018/06/26/
+[newsletter #2]: /en/newsletters/2018/07/03/
+[newsletter #3]: /en/newsletters/2018/07/10/
+[newsletter #4]: /en/newsletters/2018/07/17/
+[newsletter #5]: /en/newsletters/2018/07/24/
+[newsletter #6]: /en/newsletters/2018/07/31/
+
 {% comment %}
 <!--REQUIRES PERIODIC UPDATE: update rpc_version below to latest
 version of BitcoinCore.org's RPC docs-->
 {% endcomment %}
-{% assign rpc_version = "0.16.1" %}
-[rpc fundrawtransaction]: https://bitcoincore.org/en/doc/{{rpc_version}}/rpc/rawtransactions/fundrawtransaction/
-[rpc abandontransaction]: https://bitcoincore.org/en/doc/{{rpc_version}}/rpc/rawtransactions/fundrawtransaction/
-
-
+{% assign rpc_prefix = "https://bitcoincore.org/en/doc/0.16.1" %}
+[rpc fundrawtransaction]: {{rpc_prefix}}/rpc/rawtransactions/fundrawtransaction/
+[rpc abandontransaction]: {{rpc_prefix}}/rpc/wallet/abandontransaction/
+[rpc verifytxoutproof]:   {{rpc_prefix}}/rpc/blockchain/verifytxoutproof/

_posts/en/2018-07-30-xapo-consolidation.md

@@ -0,0 +1,15 @@
+---
+title: 'Field Report: Consolidation of 4 Million UTXOs at Xapo'
+permalink: /en/xapo-utxo-consolidation/
+name: 2018-07-30-xapo-utxo-consolidation
+type: posts
+layout: post
+lang: en
+version: 1
+---
+
+{:.post-meta}
+*by [Anthony Towns](https://twitter.com/ajtowns)<br>Developer on Bitcoin Core
+at [Xapo][]*
+
+{% include articles/towns-xapo-consolidation.md hlevel='##' %}

_posts/en/newsletters/2018-07-31-newsletter.md

@@ -0,0 +1,177 @@
+---
+title: 'Bitcoin Optech Newsletter #6'
+permalink: /en/newsletters/2018/07/31/
+name: 2018-07-31-newsletter
+type: newsletter
+layout: newsletter
+lang: en
+---
+This week's newsletter includes the usual dashboard and action items, a
+feature article by developer Anthony Towns about the consolidation of
+four million UTXOs at Xapo, news about possible upgrades to Bitcoin's
+script system, links to a few highly-voted questions and answers on
+the Bitcoin StackExchange, and some notable commits in the development
+branches of the Bitcoin Core, Lightning Network Daemon (LND), and
+C-lightning projects.
+
+## Action items
+
+- **Bitcoin Core 0.16.2 released:** a minor release that brings bug
+  fixes and minor improvements.  If you use the [abandontransaction][rpc
+  abandontransaction] or [verifytxoutproof][rpc verifytxoutproof] RPCs,
+  you should particularly consider upgrading.  Otherwise, we recommend
+  you review the [release notes][bitcoin core 0.16.2] for other changes
+  that may affect your operation and upgrade when convenient.
+
+## Dashboard items
+
+- **Fees still low:** hash rate increased difficulty by more than 14%
+  in the 2,016-block retarget period ending Sunday, giving an average
+  time between blocks of 8 minutes and 41 seconds. This helped to absorb the
+  increased transaction load from the past week's price
+  speculation. Immediately following a difficulty retarget, the average
+  time between blocks is restored to 10 minutes.
+
+  As we transition away from the normal weekend transaction lull into the
+  new week, there's the possibility for a rapid increase in estimated
+  transaction fees.  We recommend being careful sending large low-fee
+  transactions such as consolidations until closer to the weekend when
+  transaction volume begins to taper off again.
+
+## Field Report: Consolidation of 4 Million UTXOs at Xapo
+
+*by [Anthony Towns](https://twitter.com/ajtowns), Developer on Bitcoin Core at [Xapo][]*
+
+{% include articles/towns-xapo-consolidation.md hlevel='###' %}
+
+## News
+
+- **"Improvements in the Bitcoin Scripting Language" by Pieter
+  Wuille:** a talk last week giving a high-level overview of several
+  possible near-term improvements to Bitcoin.  We highly recommend
+  watching the [video][sfdev video], viewing the [slides][sipa slides],
+  or reading the [transcript][kanzure transcript] (with references) by
+  Bryan Bishop---but if you're too busy, here's Wuille's conclusion: "my
+  initial focus here is Schnorr signatures and taproot. The reason for
+  this focus is that the ability to make any input and output in the
+  cooperative case look identical is an enormous win for how script
+  execution works.
+
+    "Schnorr is necessary for this because without it we cannot encode
+    multiple parties into a single key. Having multiple branches in
+    there is a relatively simple change.
+
+    "If you look at the consensus changes necessary for these things,
+    it's really remarkably small, dozens of lines of code. It looks like
+    a lot of the complexity is in explaining why these things are useful
+    and how to use them and not so much in the impact on the consensus
+    rules.
+
+    "Things like aggregation, I think, are something that can be
+    done after we have explored various options for structural
+    improvements to the scripting language, once it's clear around what
+    the structuring should be, because we will probably learn from the
+    deployments how these things get used in practice. That's what I'm
+    working on with a number of collaborators and we'll hopefully be
+    proposing something soon, and that's the end of my talk."
+
+[sfdev video]: https://www.youtube.com/watch?v=YSUVRj8iznU
+[sipa slides]: https://prezi.com/view/YkJwE7LYJzAzJw9g1bWV/
+[kanzure transcript]: http://diyhpl.us/wiki/transcripts/sf-bitcoin-meetup/2018-07-09-taproot-schnorr-signatures-and-sighash-noinput-oh-my/
+
+## Selected Q&A from Bitcoin StackExchange
+
+{% comment %}<!--
+https://bitcoin.stackexchange.com/search?tab=votes&q=created%3a1m..%20is%3aanswer
+-->{% endcomment %}
+
+*[Bitcoin StackExachange][bitcoin.se] is one of the first places Optech
+contributors look for answers to their questions---or when we have a
+few spare moments of time to help answer other people's questions.  In
+this new monthly feature, we highlight some of the top voted questions
+and answers made there in the past month.*
+
+- [Schnorr versus ECDSA][bse 77235]: in this answer, Bitcoin protocol
+  developer Pieter Wuille explains some of the chief advantages of the
+  Schnorr signature scheme over Bitcoin's current ECDSA signature
+  scheme.
+
+- [Why does HD key derivation stop working after a certain index in
+  BIP44 wallets?][bse 76998]: a developer testing his wallet finds that 
+  a payment sent to low-numbered key indexes works as expected, but
+  payments sent to high-numbered indexes never appear in his wallets.
+  An answer from Viktor T. reveals why.
+
+- [The maximum size of a Bitcoin DER-encoded signature is...][bse
+  77191]: in this answer, Pieter Wuille provides the math for calculating
+  the size of a Bitcoin signature.  As mentioned in [Newsletter #3][],
+  the maximum size using a regular wallet is 72 bytes---but Wuille
+  explains how you can create a non-standard transaction with a 73 byte
+  signature and why you might think you saw a signature that was 74 or
+  even 75 bytes.
+
+- [If you can use almost any opcode in P2SH, why can't you use them in
+  scriptPubKeys?][bse 76541]: in this answer, Bitcoin technical writer
+  David A. Harding explains why early versions of Bitcoin restricted the
+  types of transactions that could be sent to "standard transactions"
+  and why most of those restrictions are still in place even though
+  almost all opcodes are available for standard use now via P2SH and
+  segwit P2WSH.
+
+## Notable commits
+
+*A quick look at recent merges and commits made in various open source
+Bitcoin projects.*
+
+{% comment %}
+bitcoin: git log --merges 07ce278455757fb46dab95fb9b97a3f6b1b84faf..ef4fac0ea5b4891f4529e4b59dfd1f7aeb3009b5
+lnd: git log --topo-order -p 271db7d06da3edf696e22109ce0277eaff11cc5e..92b0b10dc75de87be3a9f895c8dfc5a84a2aec7a
+c-lightning: git log --topo-order -p d84d358562a3bcdf48856fdea24511907ff53fd9..0b597f671aa31c1c56d32a554fcdf089646fc7c1
+{% endcomment %}
+
+- [Bitcoin Core #12257][#12257]: if you start Bitcoin Core with the
+  optional flag `-avoidpartialspends`, the wallet will by default spend
+  all outputs received to the same address whenever any one of them
+  would be spent.  This prevents two outputs to the same address from being spent
+  in separate transactions, which is a common way address reuse reduces
+  privacy.  The downside is that it may make transactions larger than
+  the smallest they need to be.  Bitcoin businesses using Bitcoin Core's
+  built-in wallet who don't need the extra privacy may still want to
+  toggle this flag on when fees are low for automatic consolidation of
+  related inputs.
+
+- [LND #1617][]: updates estimates for the size of on-chain
+  transactions to prevent transactions from accidentally paying too low
+  of a fee and getting stuck.  [This commit][lnd
+  ee2f2573c1b1b33288d05ba59a1e8ef9e8fb621c] may be interesting for
+  anyone wondering about the size of the transactions (and parts of
+  transactions) produced in the current protocol.
+
+- [LND #1531][]: the daemon no longer looks for spends in the memory
+  pool---it waits for them to be a confirmed part of a block first.
+  This allows the same code to work on full nodes like Bitcoin Core and
+  btcd as well as on [BIP157][]-based lightweight clients that don't have
+  access to unconfirmed transactions.  This is part of the ongoing effort
+  to help people without full nodes use LN.
+
+- In several commits, [C-lightning][] developers have mostly completed
+  the transition from handling peer-related functions in `gossipd` to
+  handling them in `channeld` or `connectd` as appropriate.
+
+- C-lightning has improved its secret handling so that secrets and
+  signatures are always generated and stored by a separate daemon than the
+  parts of the system directly connected to the network.
+
+{% include references.md %}
+{% include link-to-issues.md issues="12257" %}
+
+{% assign bse = "https://bitcoin.stackexchange.com/a/" %}
+[bse 77235]: {{bse}}77235
+[bse 76998]: {{bse}}76998
+[bse 77191]: {{bse}}77191
+[bse 76541]: {{bse}}76541
+
+{% assign lnd = "https://github.com/lightningnetwork/lnd/pull/" %}
+[lnd #1617]: {{lnd}}1617
+[lnd #1531]: {{lnd}}1531
+[lnd ee2f2573c1b1b33288d05ba59a1e8ef9e8fb621c]: https://github.com/lightningnetwork/lnd/commit/ee2f2573c1b1b33288d05ba59a1e8ef9e8fb621c