lnd v0.8.0-beta
Roasbeef
Olaoluwa Osuntokun
Database Migrations
This release includes two migrations. The first migration upgrades existing payment attempts and results to support the new TLV onion format. The migration should look like this:
2019-09-26 15:37:31.794 [INF] CHDB: Checking for schema update: latest_version=11, min_upgrade_version=9, db_version=9
2019-09-26 15:37:31.794 [INF] CHDB: Performing database schema migration
2019-09-26 15:37:31.794 [INF] CHDB: Applying migration #10
2019-09-26 15:37:31.795 [INF] CHDB: Migration of route/hop serialization complete!
2019-09-26 15:37:31.795 [INF] CHDB: Migrating to new mission control store by clearing existing data
2019-09-26 15:37:31.795 [INF] CHDB: Migration to new mission control completed!
The second migration upgrades invoices to support tracking multiple HTLCs, rather than a single one. This increases the accuracy of invoice accounting within lnd across restarts. Note that you cannot update if you have any pending hodl invoice. The migration should look like this:
2019-09-26 15:37:31.795 [INF] CHDB: Applying migration #11
2019-09-26 15:37:31.795 [INF] CHDB: Migrating invoices to new invoice format
2019-09-26 15:37:31.795 [INF] CHDB: Migration of invoices completed!
Once these migrations succeed, it will not be possible to return to a prior version of lnd.
Verifying the Release
In order to verify the release, you'll need to have gpg or gpg2 installed on your system. Once you've obtained a copy (and hopefully verified that as well), you'll first need to import the keys that have signed this release if you haven't done so already:
curl https://keybase.io/roasbeef/pgp_keys.asc | gpg --import
Once you have the required PGP keys, you can verify the release (assuming manifest-v0.8.0-beta.txt and manifest-v0.8.0-beta.txt.sig are in the current directory) with:
gpg --verify manifest-v0.8.0-beta.txt.sig
You should see the following if the verification was successful:
gpg: assuming signed data in 'manifest-v0.8.0-beta.txt'
gpg: Signature made Tue Oct 22 15:45:18 2019 PDT
gpg: using RSA key 4AB7F8DA6FAEBB3B70B1F903BC13F65E2DC84465
gpg: Good signature from "Olaoluwa Osuntokun <laolu32@gmail.com>" [ultimate]
That will verify the signature of the manifest file, which ensures integrity and authenticity of the archive you've downloaded locally containing the binaries. Next, depending on your operating system, you should then re-compute the sha256 hash of the archive with shasum -a 256 <filename>, compare it with the corresponding one in the manifest file, and ensure they match exactly.
Verifying the Release Binaries
As of this release, our release binaries are fully reproducible thanks to go1.13! Third parties are now able to verify that the release binaries were produced properly without having to trust the release manager(s). See our reproducible builds guide for how this can be achieved. The release binaries are compiled with go1.13.3, which is required by verifiers to arrive at the same ones. They include the following build tags: autopilotrpc, signrpc, walletrpc, chainrpc, invoicesrpc, routerrpc, and watchtowerrpc. Note that these are already included in the release script, so they do not need to be provided.
Finally, you can also verify the tag itself with the following command:
git verify-tag v0.8.0-beta
Building the Contained Release
Users are able to rebuild the target release themselves without having to fetch any of the dependencies. In order to do so, assuming that vendor.tar.gz and lnd-source-v0.8.0-beta.tar.gz are in the current directory, follow these steps:
tar -xvzf vendor.tar.gz
tar -xvzf lnd-source-v0.8.0-beta.tar.gz
GO111MODULE=on go install -v -mod=vendor -ldflags "-X github.com/lightningnetwork/lnd/build.Commit=v0.8.0-beta" ./cmd/lnd
GO111MODULE=on go install -v -mod=vendor -ldflags "-X github.com/lightningnetwork/lnd/build.Commit=v0.8.0-beta" ./cmd/lncli
The -mod=vendor flag tells the go build command that it doesn't need to fetch the dependencies, and instead, they're all enclosed in the local vendor directory.
Additionally, it's now possible to use the enclosed release.sh script to bundle a release for a specific system like so:
LNDBUILDSYS="linux-arm64 darwin-amd64" ./build/release/release.sh
⚡️⚡️⚡️ OK, now to the rest of the release notes! ⚡️⚡️⚡️
Release Notes
Protocol Upgrades
Multi-Frame TLV Based Onion
In this release of lnd, we’ll now by default signal adherence to the new multi-frame TLV based onion format used when routing HTLCs end to end within the network. Compared to the legacy format, this format is much more flexible as rather than using a handpacked encoding for each field, we’ll now transmit TLV (type-length-value) streams, which is essentially a key-value map. As a result, the onion format is now far more extensible, as we can easily add new types without triggering a network wide update. In many instances (such as AMP), only the sender and receiver need to know of the new information in the onion, streamlining upgrade paths.
This change isn’t exposed to developers yet, but will be in future versions, allowing them to send a small amount of bytes to the receiver alongside a payment. The largest impact of this new feature is to lay the groundwork for the AMP/MPP work that has begun. In the case of AMP, this new namespace in the onion will be used for things like sending a partial payment shard, the total amount of the unified set of payments, and so on. We’ll also leverage this space to roll out our spontaneous payment feature (keysend) in a future version of lnd.
Safu Commitments (option_static_remote_key)
Another major protocol update that has shipped in this new version of lnd is something we call “safu commitments”. This new commitment format represents a large step in making Lightning safer to use for end users. Before this change, if the remote party forced closed the channel on-chain, in order to sweep your funds (in the case of partial data loss, but SCB existence), then the party that lost data would need to obtain a random nonce from the other party in order to sweep their funds. This dependency at times made recovery very difficult, or even indeterminate, if the party that lost data was unable to obtain this per-state nonce from the remote party.
With this new commitment format, we’ve instead removed this nonce derivation from the commitment output. Instead, when the remote party force closes, the funds will go directly into one’s wallet. As a result, the only thing required to recover funds in the case of partial data loss now are one’s SCB and the on-chain event of the channel being confirmed on chain. A new SCB version (v1) has been added to signal if the channel being backed up uses the old or new commitment format, allowing us to keep our recovery flow identical to the current one.
It’s important to note that in order to use this new commitment format, new channels must be opened. The SCBs stored on disk, as well as those that are sent over the various RPC calls, will be automatically updated to reflect if this new commitment is being used in the channels being backed up. A new field has been added to the ListChannels output (static_remote_key) that indicates if the channel is using the new safu commitment format or not.
TLV Serialization Library
This release also includes a new TLV serialization library, which implements the TLV format described in BOLT 01. The tlv package produces encodings that are both forwards and backwards compatible, and will form the basis of making wire messages and onion payloads easily extensible fields for new features like MPP, AMP, or extended gossip queries.
BOLT 11 Feature Bits
In 0.8.0, lnd now supports the ability to generate and parse invoices with payment-specific feature bits. At the moment there are no such feature bits that are advertised, however this is a preliminary step in order to widely deploy the parsing logic for when this happens. The first candidate features for being advertised are likely for MPP or AMP, which can inform the sender as to whether they may pay a particular invoice using those payment methods. In theory, the sender can also require that an invoice be fulfilled with a particular payment type.
Indefinite Channel Reestablishment
Prior to 0.8.0, lnd would give the remote peer 30 seconds to send a their channel_reestablish message. Typically this duration was sufficient, however if the exchange was not completed in a timely manner lnd would incorrectly escalate this into a protocol error, and cause strict implementations to force close the channel.
In one instance with a c-lightning node, this resulted in 40 channels being force closed, since the contention at startup prevented the remote node from promptly sending channel_reestablish. In 0.7.2, c-lightning also added mitigations to avoid this situation, by spacing out reconnections to peers on startup and ignoring the escalated errors.
In 0.8.0, lnd will wait indefinitely over the span of a connection for the remote peer to send channel_reestablish. As a result, these sporadic failures should not occur between updated nodes. Further, the active flag will in ListChannels is stricter and more accurate, which now only displays true if the channel is sufficiently confirmed, has ever received funding_locked, and has received channel_reestablish on the current connection. Previously the active boolean would display true whenever the first two conditions had been met.
Chain Notifier Subserver
Subscriptions for confirmation and spend notifications now require a height hint to be provided, also known as the earliest height in the chain at which the event could have happened. This prevents lnd from scanning blocks that are irrelevant to the subscription. Subscriptions for confirmation notifications now also require at least one confirmation. Previously, subscriptions without providing a number of confirmations would lead lnd to deadlock.
Watchtower Client Subserver
Continuing our subserver saga, this release includes a new addition: the Watchtower Client subserver. This is the first subserver that is included in lnd by default and does not require a build tag.
The subserver includes the following RPCs, along with a lncli command for each:
- AddTower (
lncli wtclient add) - RemoveTower (
lncli wtclient remove) - ListTowers (
lncli wtclient towers) - GetTowerInfo (
lncli wtclient tower) - Stats (
lncli wtclient stats) - Policy (
lncli wtclient policy)
These RPCs allow users to interact with the watchtower client of their lnd node, allowing for modifications and information retrieval of any registered watchtowers and the policies used with them.
Note that due to the introduction of this interface, the --wtclient.private-tower-uris flag has now been deprecated and will be removed in the next major release, v0.9.0-beta. All that is required for the client to be active is --wtclient.active, any setup that had been configured from before will carry over. If you're setting up a new watchtower or want to change the configuration, users will now need to so as stated in the watchtower documentation.
Gossip Enhancements
A rough graph sync progress is now exposed via the GetInfo RPC as a boolean named is_graph_synced.
Gossip received as part of an initial sync is no longer forwarded to other peers as it’s assumed that they have been previously broadcast to the network. This results in a small decrease in outbound bandwidth usage.
lnd now rebroadcasts its node announcement to the network every 24 hours to ensure nodes have up-to-date information about them.
A new --ignore-historical-gossip-filters option has been added in 0.8.0 to reduce outbound bandwidth usage in syncing older lightning nodes. Some older nodes send gossip timestamp filters that request large portions of the graph on each connection; this flag will still allow new gossip messages to be forwarded that satisfy the timestamp, but forgo the initial dump which can rack up outgoing bandwidth.
Routing
Mission control, the subsystem in lnd that drives the payment process and tracks past node performance, has been made persistent. It now retains historical payment results across restarts.
Further improvements have been made to mission control that make the payment process faster and more reliable. Routing nodes forward in a non-strict matter, meaning that if the requested channel isn’t available, the node may forward an HTLC over an alternative channel to the same next hop. Therefore, it is enough to try a single channel for each node pair. To speed up payments, we now track node performance based on node pairs instead of channels.
When a payment attempt fails, the sender receives a failure message from which information can be distilled that helps identifying the cause of the failure. This information can be used to steer around the problem for the next attempt. The interpretation of the failure message has been improved.
Previously, mission control only recorded payment failures. This prevented hitting the same failure twice, but ignored previously successful routes. It didn’t distinguish between a route that worked before and an untried route. We now favor successful routes over untried routes and thereby reduce the number of required payment attempts. The explore-exploit trade-off that this presents is controlled by the existing routerrpc.attemptcost config parameter.
To enhance the transparency of the path finding process, the total route success probability is exposed on QueryRoutes calls.
A new RPC BuildRoute has been added. This call targets advanced users who want to build their own routes without needing to keep a shadow copy of the graph. The call takes a list of hop pubkeys and transforms this into a fully specified route by looking up channel policies from the graph. It also allows the construction of circular routes.
Routing Node Enhancements
Nodes will now enforce a maximum channel commitment fee for channels in which they are initiators when attempting to update their commitment fee. The maximum enforced depends on a percentage (default of 50%) of a channel initiator’s balance. The percentage can be modified through the --max-channel-fee-allocation CLI flag.
The max_htlc channel policy parameter is now exposed on the RPC interface. This allows callers of UpdateChannelPolicy to reduce the maximum htlc amount that is forwarded over a channel.
RPC Bug Fixes
The RemoteBalance, LocalChanReserveSat, and RemoteChanReserveSat fields for waiting close channels are now properly set. These weren’t set previously, leading to the fields being blank until the channel moved to pending close.
Previously, using the UpdateChannelPolicy RPC with a zero base fee and/or fee rate would lead to a desynchronization between the routing policy propagated throughout the network and the one within the channel state machine, leading to routing failures throughout the network. To work around this, users were required to restart their nodes, but this is no longer required as of #3439.
The REST endpoint for the DescribeGraph RPC call (v1/graph) has been updated to allow a larger response size due to the growing channel graph.
A bug in the ordering of channel updates for getnodeinfo has been fixed that could cause the updates not to be sorted in order of increasing pubkey. Additionally, the last_update time displayed will now properly compute the maximum of the each update's last_update instead of inconsistently selecting one or the other.
Users who set the perm field (or --perm for lncli) prior to 0.8.0 may have noticed that lnd would not automatically reconnect to peers if the connection was severed. This release fixes the bug by tracking whether or not the current connection was requested as permanent via the RPC, and restoring the original functionality of the ConnectPeer RPC.
lncli Bug Fixes
Invoice Validation
Invoice HTLC tracking
Calls to LookupInvoice and ListInvoices now return a list of all HTLCs that pay to an invoice. This provides insight into the channel through which the invoice was paid, as well as acceptance time, resolution time, and the exact HTLC amount. The handling of HTLCs internally has also been improved, which fixed several existing consistency and accounting issues.
Privacy
A potential probe vector that allowed attackers to find out the final destination of a payment via the final_expiry_too_soon response has been eliminated. To not deprive the payer of information that is required for continuation of the payment process, the height at which the receiver accepted the HTLC is added to the failure message.
New OS/Arch Release Targets
0.8.0 will provide compiled binaries for 11 new architectures: illumos-amd64, linux-ppc64le, linux-mips, linux-mipsle, linux-s390x, netbsd-arm, netbsd-arm64, openbsd-arm, openbsd-arm64, solaris-amd64, and windows-arm. The illumos-amd64 and solaris-amd64 builds were newly enabled by go1.13 while the others had been previously available but not included in the release targets. If you're interested in a target that is not listed, please file an issue!
bitcoind Compatibility
The lnd integration test suite is now also continuously being run with bitcoind as the chain backend. All supported backends (btcd, neutrino, and bitcoind) now enjoy the same test coverage, which ensures lnd will behave the same regardless of which backend users choose. As of this release, lnd is fully compatible with bitcoind versions up to v0.18.1.
Mobile Support
The mobile APIs and build tools have been included in this release. This allows developers to start building mobile apps which integrate lnd, either by checking out the latest release, or the master branch.
RBF Aware Transaction Broadcast
The wallet has been updated to understand RBF specific errors. For now, this handles a few edge cases with non-critical broadcast errors. In the future, this will pave the way for direct fee bumping through RBF in the wallet.
Channel Close Transaction Rebroadcast
To ensure proper confirmation of a channel close transaction, lnd has been updated to rebroadcast it after restarts. It also now also aids remote nodes that have lost state by resending channel reestablishment messages for already closed channels.
Mainnet Neutrino
Neutrino has steadily been improved, and this release contains several fixes and optimizations that make it stable enough for mainnet support. Keep in mind that it is still early, and stay craeful.
Changelog
The full list of changes since v0.7.1-beta can be found here:
Contributors (Alphabetical Order)
Carla Kirk-Cohen
chokoboko
Christopher Coverdale
Conner Fromknecht
Dario Sneidermanis
Eugene
Fernando Guisso
fiatjaf
Hampus Sjöberg
Johan T. Halseth
Jonathan Cross
Joost Jager
Juan Pablo Civile
Lars Lehtonen
lieteau2
Lightning Koala
Matheus Degiovani
Olaoluwa Osuntokun
Oliver Gugger
openoms
Oskar F
Spencer Dupre
Vadym Popov
Valentine Wallace
Will Roscoe
Wilmer Paulino