Skip to content

@Roasbeef Roasbeef released this Jul 2, 2019 · 125 commits to master since this release

This release marks a new major release of lnd that includes several important bug fixes, an improved payment API, pathfinding enhancements, faster initial sync times, support for fee bumping on sweeps, and an initial rollout of altruistic watchtowers. As always, users are highly encouraged to upgrade to this new version.

Database Migrations

This version includes two migrations, the first is in channel.db which modifies the structure of the payment tracking data to support the refactored router and its ability to reliably display payments via the RPC. The migration should look like this:

2019-06-14 21:54:53.576 [INF] LTND: Version: 0.7.0-beta commit=v0.7.0-beta, build=production, logging=default
2019-06-14 21:54:53.579 [INF] LTND: Active chain: Bitcoin (network=mainnet)
2019-06-14 21:54:53.583 [INF] CHDB: Checking for schema update: latest_version=9, db_version=8
2019-06-14 21:54:53.586 [INF] CHDB: Performing database schema migration
2019-06-14 21:54:53.586 [INF] CHDB: Applying migration #9
2019-06-14 21:54:53.586 [INF] CHDB: Migrating outgoing payments to new bucket structure
2019-06-14 21:54:53.586 [INF] CHDB: Migration of outgoing payment bucket structure completed!

The second migration is in wallet.db, which prunes redundant block data already being stored by the underlying backend. The migration should look like this:

2019-06-14 21:54:53.744 [INF] LNWL: Applying wallet address manager migration #8
2019-06-14 21:54:53.746 [INF] LNWL: Removing block hash entries beyond maximum reorg depth of 10000 from current tip 580748

Once updated, it will not be possible to return to an older version of lnd.

Verifying the Release

In order to verify the release, you'll need to have gpg or gpg2 installed on your system. Once you've obtained a copy (and hopefully verified that as well), you'll first need to import the keys that have signed this release if you haven't done so already:

curl https://keybase.io/roasbeef/pgp_keys.asc | gpg --import

Once you have the required PGP keys, you can verify the release (assuming manifest-v0.7.0-beta.txt and manifest-v0.7.0-beta.txt.sig are in the current directory) with:

gpg --verify manifest-v0.7.0-beta.txt.sig

You should see the following if the verification was successful:

gpg: assuming signed data in 'manifest-v0.7.0-beta.txt'
gpg: Signature made Tue Jul  2 09:46:47 2019 PDT
gpg:                using RSA key F8037E70C12C7A263C032508CE58F7F8E20FD9A2
gpg: Good signature from "Olaoluwa Osuntokun <laolu32@gmail.com>" [ultimate]

That will verify the signature on the main manifest page which ensures integrity and authenticity of the binaries you've downloaded locally. Next, depending on your operating system you should then re-calculate the sha256 sum of the binary, and compare that with the following hashes (which are included in the manifest file):

f8452608ff2e3ca6a0327c830f2fee5e837b5a39e0bf7cedd190857a5b8894f8  lnd-darwin-386-v0.7.0-beta.tar.gz
57a2eef7c337dbad6bbdc2bdffe459292d34ce354b58d74e74b329753dc134c8  lnd-darwin-amd64-v0.7.0-beta.tar.gz
c7cd6a1f4980fbefaf4acc49940332d3159b7b6af989afb9d0c211abe0208d53  lnd-dragonfly-amd64-v0.7.0-beta.tar.gz
8380e5944053f5a8255deb0f69b3dc0a2bac30402b82abec9f348cad53ec166f  lnd-freebsd-386-v0.7.0-beta.tar.gz
c818c3a983167312f3bf2c84cb285212c5052131319caaef287a97541d2ff479  lnd-freebsd-amd64-v0.7.0-beta.tar.gz
a1e861f4c9a4cf056030f40debd882c6f34502821d0edca27f415dcfbb9f7d8c  lnd-freebsd-arm-v0.7.0-beta.tar.gz
47be6c3391fadbc5a169fa1dd6dd13031d759b3d42c71a2d556751746b705c48  lnd-linux-386-v0.7.0-beta.tar.gz
2e7ed105b9e57103645bda30501cbf3386909cfed19a2fabcc3dc9117ce99a8f  lnd-linux-amd64-v0.7.0-beta.tar.gz
c995fa67d6b23e547723801de49817dda34188fba78d0fe8ae506774e54c0afd  lnd-linux-arm64-v0.7.0-beta.tar.gz
a653b66e28b30131c9b766989cb490013128022e047273f287bf0f42a19693b9  lnd-linux-armv6-v0.7.0-beta.tar.gz
ac51d96ee9b57bfcab0b05dbcfcd9ce3bd42a216354c0972e97c1a1c86c2479a  lnd-linux-armv7-v0.7.0-beta.tar.gz
a4a119855e3759e49472d7d0f1f8529e984e7e7fbcedb78463daf4d7f6aceb6e  lnd-linux-mips64-v0.7.0-beta.tar.gz
2e06c33c0c8c4f6ef680071095e652ea0b32ff164545a60aa372de3b12644db1  lnd-linux-mips64le-v0.7.0-beta.tar.gz
6d4bc470ae424bf46f1057149880f60a83057f26693f0098f3e9dd774355cdcd  lnd-linux-ppc64-v0.7.0-beta.tar.gz
d931981d6a742b07abc965e779fd8ed93a7dfdbbdeefea0a47b0cdf90a94b645  lnd-netbsd-386-v0.7.0-beta.tar.gz
db6d6be0cf5e7e791be097de944261db74ef1400b711dc6b825b21b7d3a2958f  lnd-netbsd-amd64-v0.7.0-beta.tar.gz
0ef5470c6ba928e740bf83ba86b7af05eb0df0d3077e92c347fb93cdcf7fb276  lnd-openbsd-386-v0.7.0-beta.tar.gz
ec8dae2c01d818a6cbc622b07f5be8a4355a25cc7a887216edc232c528257c20  lnd-openbsd-amd64-v0.7.0-beta.tar.gz
d5d9178178dca9a3e770dc74d655f579e6aafaec9e7b32a726c44dc093c52aa0  lnd-source-v0.7.0-beta.tar.gz
254ccdf63c2dbd95381663be0e132d60f3423c9568d304a4384823c198d12f8a  lnd-windows-386-v0.7.0-beta.zip
51badb5f690e8bc15e90331a42ea823399d1eb60708c4d682683f070ece13c23  lnd-windows-amd64-v0.7.0-beta.zip
4ee8e4b7d8372c8e750125dcdd93cd1b1b55687460a2f7fe1c8a23e60bb17e7b  vendor.tar.gz

One can use the shasum -a 256 <file name here> tool in order to re-compute the sha256 hash of the target binary for your operating system. The produced hash should be compared with the hashes listed above and they should match exactly.

Finally, you can also verify the tag itself with the following command:

git verify-tag v0.7.0-beta

The binaries are compiled with go1.12.6 and include the following build tags: autopilotrpc, signrpc, walletrpc, chainrpc, invoicesrpc, and routerrpc.

Building the Contained Release

With this new version of lnd, we've modified our release process to ensure the bundled release is now fully self contained. As a result, with only the attached payload with this release, users will be able to rebuild the target release themselves without having to fetch any of the dependencies. Note that at this stage, binaries aren't yet fully reproducible (even with go modules). This is due to the fact that by default, Go will include the full directory path where the binary was built in the binary itself. As a result, unless your file system exactly mirrors the machine used to build the binary, you'll get a different binary, as it includes artifacts from your local file system. This will be fixed in go1.13, and before then we may modify our release system to do this automatically.

In order to re-build from scratch, assuming that vendor.tar.gz and lnd-source-v0.7.0-beta.tar.gz are in the current directory:

tar -xvzf vendor.tar.gz
tar -xvzf lnd-source-v0.7.0-beta.tar.gz
GO111MODULE=on go install -v -mod=vendor -ldflags "-X github.com/lightningnetwork/lnd/build.Commit=v0.7.0-beta" ./cmd/lnd
GO111MODULE=on go install -v -mod=vendor -ldflags "-X github.com/lightningnetwork/lnd/build.Commit=v0.7.0-beta" ./cmd/lncli

The -mod=vendor flag tells the go build command that it doesn't need to fetch the dependencies, and instead, they're all enclosed in the local vendor directory.

Additionally, it's now possible to use the enclosed release.sh script to bundle a release for a specific system like so:

LNDBUILDSYS="linux-arm64 darwin-amd64" ./release.sh

⚡️⚡️⚡️ OK, now to the rest of the release notes! ⚡️⚡️⚡️

Release Notes (in progress)

bitcoind 0.18 compatibility

This new version of lnd has updated the way we check for errors over the RPC interface after we broadcast a transaction to be compatible with bitcoind v0.18. Note that the “master” branch of bitcoind isn’t yet supported yet as it includes distinct RPC changes to sendrawtransaction in particular that will require further updates in lnd. We’re targeting lnd v0.7.1. to fix these remaining RPC compatibility issues.

Private Altruist Watchtowers

In this release, we’re rolling out the ability to run a private, altruist watchtower as a fully-integrated subsystem of lnd. Watchtowers act as a second line of defense in responding to malicious or accidental breach scenarios in the event that client’s node is offline or unable to respond at the time of a breach, offering greater degree of safety to channel funds.

In contrast to a reward watchtower which demand a portion of the channel funds as a reward for fulfilling its duty, an altruist watchtower returns all of the victim’s funds (minus on-chain fees) without taking a cut. Reward watchtowers will be enabled in a subsequent release, though are still undergoing further testing and refinement.

In addition, lnd can now be configured to operate as a watchtower client, backing up encrypted breach-remedy transactions (aka. justice transactions) to other altruist watchtowers. The watchtower stores the fixed-size, encrypted blobs and is only able to decrypt and publish the justice transaction after the offending party has broadcast a revoked commitment state. Client communications with a watchtower are encrypted and authenticated using ephemeral keypairs, mitigating the amount of tracking the watchtower can perform on its clients using long-term identifiers.

Note that we have chosen to deploy a restricted set of features in this release that can begin to provide meaningful security to lnd users. Many more watchtower-related features are nearly complete or have meaningful progress, and we will continue to ship them as they receive further testing and become safe to release.

Configuring a Watchtower

To set up a watchtower, command line users should compile in the optional watchtowerrpc subserver, which will offer the ability to interface with the tower via gRPC or lncli. The minimal configuration needed to activate the tower is watchtower.active=1.

By default, the watchtower will listen on :9911 which specifies port 9911 listening on all available interfaces. Users may configure their own listeners via the --watchtower.listen= option.

Additionally, users can specify their tower’s external IP address(es) using watchtower.externalips, which will expose the full tower URI over RPC. These can be given to clients in order to connect and use the tower.

Note: The watchtower’s public key is distinct from lnd’s node public key. For now this acts as a soft whitelist as it requires clients to know the tower’s public key in order to use it for backups before more advanced whitelisting features are implemented. We recommend NOT disclosing this public key openly, unless you are prepared to open your tower up to the entire Internet.

Retrieving information about your tower’s configurations can be done using lncli tower info:

🏔 lncli tower info
{
        "pubkey": "03281d603b2c5e19b8893a484eb938d7377179a9ef1a6bca4c0bcbbfc291657b63",
        "listeners": [
                "[::]:9911"
        ],
        "uris": [
                "03281d603b2c5e19b8893a484eb938d7377179a9ef1a6bca4c0bcbbfc291657b63@1.2.3.4:9911"
        
}

The entire set of watchtower configuration options can be found using lncli -h:

watchtower:
      --watchtower.active                                     If the watchtower should be active or not
      --watchtower.towerdir=                                  Directory of the watchtower.db (default: $HOME/.lnd/data/watchtower)
      --watchtower.listen=                                    Add interfaces/ports to listen for peer connections
      --watchtower.externalip=                                Add interfaces/ports where the watchtower can accept peer connections
      --watchtower.readtimeout=                               Duration the watchtower server will wait for messages to be received before hanging up on client connections
      --watchtower.writetimeout=                              Duration the watchtower server will wait for messages to be written before hanging up on client connections

Configuring a Watchtower Client

In order to set up a watchtower client, you’ll need the watchtower URI of an active watchtower. lnd can the ben configured to start using the tower by setting:

wtclient.private-tower-uris=03281d603b2c5e19b8893a484eb938d7377179a9ef1a6bca4c0bcbbfc291657b63@1.2.3.4:9911

Users may optionally configure the fee rate of justice transactions by setting the wtclient.sweep-fee-rate option, which accepts values in sat/byte.

The entire set of watchtower client configuration options can be found using lncli -h:

wtclient:
      --wtclient.private-tower-uris=                          Specifies the URIs of private watchtowers to use in backing up revoked states. URIs must be of the form <pubkey>@<addr>. Only 1 URI is supported at this time, if none are provided the tower will not be enabled.
      --wtclient.sweep-fee-rate=                              Specifies the fee rate in sat/byte to be used when constructing justice transactions sent to the watchtower.

For now, no information regarding the operation of the watchtower client is exposed over the RPC interface. We are working to expose this information in a later release, progress on this can be tracked in this PR. Users will be reliant on logs for observing the behavior of the client. We also plan to expand on the initial feature set by permitting multiple active towers for redundancy, as well as modifying the chosen set of towers dynamically without restarting the daemon.

gRPC Prometheus Exports

A build tag has been added: monitoring. If built with this tag, lnd has the ability to export Prometheus metrics about its gRPC performance. These metrics include keeping track of how many times each gRPC call has been invoked, how many bytes have been transmitted and received for each gRPC call, and more.

To enable Prometheus exporting after building lnd with the monitoring tag, run lnd with this flag:

--prometheus.enable

To change the address that lnd listens on for Prometheus, add this flag:

--prometheus.listen=<address>

By default, lnd's Prometheus exporter listens on localhost:8989.

Wallet Bug Fixes

A bug has been fixed that would cause the EstimateFee and SendOutputs RPC commands to fail if conditions of higher fees. The root cause was using the transactions effective fee rate instead of the minrelayfee when computing dust in certain scenarios.

Wallet Chain Sync Improvements

There was a period of time where the wallet would only store the latest 25,000 block headers of the chain in order to properly recover from chain reorgs, but this was modified to the existing behavior of storing all of them due to deep reorgs on the testnet chain during lnds initial testing and development. In this release, we revert back to the prior behavior, but will instead only store the latest 10,000 as a generous safety margin.

As a result of doing so, the initial sync process is sped up. Neutrino backends saw the least improvements in this department, while bitcoind saw the most.

The wallet will also now detect that the chain backend (i.e., btcd, bitcoind, etc.) is currently syncing if its latest block does not have a timestamp within the past 2 hours, allowing us to start lnd while the chain backend syncs.

UtxoSweeper Improvements

Recent versions of lnd have seen development of a new subsystem: the UtxoSweeper. The release notes of our v0.6.0-beta release provide a brief overview of its duties. In this release, we continue to expand its functionality by allowing users to obtain information about the list of inputs being batched by the UtxoSweeper through a new PendingSweeps RPC and its corresponding lncli wallet pendingseeps command.

We also now allow users to bump the fee rate of an input with the Replace-By-Fee (RBF) policy or a transaction with the Child-Pays-For-Parent (CPFP) policy through the BumpFee RPC and its corresponding lncli wallet bumpfee command. In order to provide this functionality, the UtxoSweeper first had to be extended to support sweeping distinct batches of inputs with similar fee preferences.

Fee bumping within lnd will take a different approach than the popular bitcoin-cli bumpfee command. Since the UtxoSweeper batches transactions together, lnd cannot rely on bumping a transaction’s fee by providing its hash since it can change at any point with the addition/removal of new inputs. Instead, fee bumping relies on an outpoint being provided along with a fee preference.

A fee preference can be defined as one of any of the following options, which will likely expand in future releases.

  • Static fee rate: fixed fee rate to spent an input on-chain with.
  • Confirmation target: a delta of blocks from the current height of the chain in which an input should be spend on-chain within.

When bumping the fee of an input that currently exists within the UtxoSweeper, a transaction is created with a fee rate backed by the fee preference provided that replaces the lower fee transaction through the RBF policy. When bumping the fee of an input that points to an unconfirmed transaction output which belongs to the user’s wallet, a transaction that spends this input is created with a fee rate backed by the fee preference provided. This spending transaction can be interpreted as the child in a CPFP scenario.

The current design presents some shortcomings which will be addressed in future releases as we develop things further. These include:

  • Fee preference Validation checks are not in place to ensure they provide a sufficiently higher fee rate in order for a fee bump to be successful. This responsibility is delegated to the user.
  • The UtxoSweeper does not persist any inputs to disk, so restarting lnd will require incomplete fee bump requests to be requested again.

lncli Bug Fixes

We'll now properly parse an optional single or multi channel backup being provided as part of lncli create.

We'll now properly spend unconfirmed UTXOs as part of lncli openchannel when providing a min_confs parameter of 0.

Config Changes

Users can now use the --tlsextraip and --tlsextradomain fields to set multiple IPs or domains when generating certs.

A new neutrino.assertfilterheader filter header flag has been added to lnd. In the past the neutrino protocol underwent a number of changes, some of which affected exactly what is included in the filter. Older clients may have been on persistent incorrect filter chains as a result of this. This new argument allows a client to ensure it has the right filter chain using an externally provided filter header hash. If this hash doesn’t match, then the neutrino client will delete their state and re-sync the filter header chain.

Litecoin support has been extended to also include the simnet and regtest chains.

Interoperability Improvements

CLTV Tie-Breaking in Commitment Output Sorting

A nuanced, spec-level bug has been fixed for properly sorting HTLCs on the commitment transaction. When HTLCs of identical payment hash and amount appear on the same commitment transaction, the original BIP69 sorting could lead to a channel failure if the HTLCs had different CLTV values since the second-layer transactions would have different sighashes. This happens because the CLTV is enforced via nLockTime field of the descendant transaction, and weren’t being considered in the sorting of the commitment outputs. The new spec sorting resolves this by using the CLTV values of HTLC outputs as tiebreakers when the payment hash and amount are identical.

Since 0.5, lnd has forbidden sending out more than one HTLC for the same payment hash at a given time, so users on pre-0.7 versions are unlikely to experience this bug in purely lnd to lnd interactions, or have triggered it on other nodes in the network. However, it’s possible that forwarding payments from implementations that permit duplicate payments can trigger the bug and result in a channel closure. This bug should be completely eliminated going forward.

Increased Non-initiator Funding Timeout

When another node initiates a channel open with lnd, the non-initiator forgets about the channel if it doesn’t confirm in timely manner. In 0.7, this timeout has been raised to two weeks from the prior value of two days. This allows lnd to tolerate channel initiators with a low time preference (set a low fee rate) and better tolerate fee spikes or on-chain congestion during the funding process.

P2P and Gossip Improvements

Prior versions of lnd would send out all updates (or new channels) to connected active syncing peers in a single payload. This could at times cause slight memory spikes if many channels were mined in a single block for example. This latest version of lnd will instead now split announcements into multi sub-batches when it’s time to flush the set of pending messages. This change reduces GC pressure, and also naturally implements a variant of flow control to ensure minimal load during announcement batch flushes. It also make sure to redo a historical sync the first time it gets a peer

RPC Server and RPC Sub-Server Changes

lnd’s RPC certificates will now be automatically re-generated by the daemon if upon restart, we detect that they’ve expired. Many nodes that were created greater than a year ago may have run into this issue, and had to delete and then regenerate themselves.

The NodeUpdate and GetNodeInfo commands now include the color of the target node.

The closeallchannels command within lncli now exposes a conf_target or sat_per_byte arguments in order to allow the user to control the fee rate used to close all transactions.

Transaction Hex in GetTransactions

The GetTransactions RPC call now also includes the serialized hex version of the transaction. This exposes information the wallet already has and can be helpful for external accounting systems connected to lnd, or for usage with the new BumpFee command added in this version.

Display Paid Payment Requests

When making payments via the SendPayment RPC, the paid invoice will now be displayed in the ListPayments results. Similarly, payments via lncli payinvoice will show the invoice in lncli listpayments. This allows applications to recover the details, e.g. the invoice memo, without needing to track them separately.

Zero Fees With UpdateChanPolicy

It’s now possible to set fees to rock bottom (zero!) using lncli updatechanpolicy or the UpdateChanPolicy RPC command.

Channel Display in GetNodeInfo

The GetNodeInfo command will now optionally also return the set of channels belonging to a particular node. This is useful as it saves additional look ups, and a client-side mapping between channel ID and node using the DescribeGraph command. Arguments for the new lncli command are as follows:

⛰   lncli getnodeinfo -h
NAME:
   lncli getnodeinfo - Get information on a specific node.

USAGE:
   lncli getnodeinfo [command options] [arguments...]

CATEGORY:
   Peers

DESCRIPTION:
   Prints out the latest authenticated node state for an advertised node

OPTIONS:
   --pub_key value     the 33-byte hex-encoded compressed public of the target node
   --include_channels  if true, will return all known channels associated with the node

Structured SendToRoute Errors

A new SendToRoute rpc has been added to the router sub-server. This call improves upon the main rpc SendToRoute rpc by returning errors as proper proto messages. No more (regex) string parsing. As part of this change, the (deprecated) option to pass in multiple routes to the main SendToRoute was removed.

Payment RPCs

The interface of SendPayment in the router sub-server has been modified. It now takes a custom payment timeout value. This defines the duration after which no new attempts to complete the payment are initiated. The call also requires the fee limit to be set explicitly. The fee limit is an important parameter and not having a default value for this forces callers to make a conscious decision about its value. Another change is that SendPayment only returns payment-level failures. This set of failures is currently limited, but will be extended in the future. The previously returned last payment attempt error is no longer available, because its informative value is considered limited.

Furthermore a new call TrackPayment was added. TrackPayment can be used to pick up and track the status of pending payments, for example after losing the connection or across restarts.

Payment Persistence

A series of PRs (#3064, #3063, #2761, #2762) has been merged, making lnd better handle in-flights payments across restarts. lnd will now write any attempted payment to the database, such that should the daemon restart before the result comes back, it will correctly record it to the DB. This means that payments can now be in states “in flight”, “failed” or “succeeded”, which can all be shown by calling lncli listpayments --include_incomplete (#3190).

Probability Based Mission Control

In prior versions of lnd, when we encountered a path finding failure, we would simply ignore the edge or node that caused for error for 5 or 15 seconds, continuing one with that path finding session. This naive approach has a number of issues. As an example, this means that shortly after a node makes a payment, it forgets all the prior context it used to successfully route the payment. Additionally, the prior version didn’t share this information amongst concurrent in-flight payment sessions, meaning each session couldn’t learn from attempts of other on going sessions. Users may have noticed this “cold start” effect when they go to send a new payment a few minutes the last successful payment attempt. In older versions of lnd, the path finding would try a series of channels that it likely knew wouldn’t work, before finally finding the golden set of channels that reliably work.

In this new version of lnd, the previously used edge and node black lists in path finding are replaced by a probability based system. It modifies path finding so that it not only compares routes on fee and time lock, but also takes route success probability into account. Each channel starts out with a default “apriori” probability. As path finding attempts fail, we reduce the probability of channels that have failed to zero, exponentially decaying back to the default apiori probability. This means that lnd will now properly “learn” from its past path finding attempts. Additionally, since this algorithm is less greedy than the prior algorithm, it may return a distinct set of paths compared to the prior algorithm, exploring avenues that maybe didn’t seem “optimal”, but still have a high probability of success.

The new system is similar to “active learning” based approaches in machine learning. lnd will now continually update its view of the network based on empirical observations of payment failures. In addition to that, historical payment attempt outcomes are kept in mission control memory longer and improve path finding for future payments. A new diagnostic rpc has been added to query the current mission control state. Several configuration options are exposed to give more control of mission control behavior:

      --routerrpc.minrtprob=                                  Minimum required route success probability to attempt the payment (default: 0.01)
      --routerrpc.apriorihopprob=                             Assumed success probability of a hop in a route when no other information is available. (default: 0.95)
      --routerrpc.penaltyhalflife=                            Defines the duration after which a penalized node or channel is back at 50% probability (default: 1h0m0s)
      --routerrpc.attemptcost=                                The (virtual) cost in sats of a failed payment attempt (default: 100)

A new querymc sub-server RPC command has been added to allow users to peek into this diagnostic information. Eventually users may be able to share their mission control state (the effective “memory” of the router), amongst their other nodes in order to ensure they don’t need to re-discover the set of reliable channels.

These changes prepare for further enhancements that will increase payment reliability.

The deprecated num_routes parameter of QueryRoutes has been removed.

Invoice Settling

Several changes with respect to invoice settling have been made. This makes the logic of off-chain and on-chain invoice settling more consistent and is a preparatory step for atomic multiple path payments.

Neutrino Improvements

All integration tests are now run continuously with the neutrino chain backend , in addition to the existing tests using btcd!

When running lnd with neutrino active, the daemon will now properly shutdown even if it’s in the middle of checking for the spentness of a particular UTXO.

Static Channel Backups

A bug fix has been made to the current on-disk SCB format in the Windows operating system. Before this release, lnd wouldn’t properly close the temporary SCB file before swapping it over to the permanent file name (atomic file rename). On Windows in particular, this would cause the swap to fail, meaning that the file wouldn’t be updated on disk. We’ve fixed this issue in this latest release, by always closing the temporary file before we attempt to modify it.

Changelog

The full list of changes since 0.6.1-beta can be found here:

Contributors (Alphabetical Order)

2xic
AdamISZ
chokoboko
Conner Fromknecht
Daniel McNally
Eugene Zeigel
Federico Bond
Francisco Calderón
Geoff Taylor
Johan T. Halseth
John Griffith
Joost Jager
Matt Drollette
michael1011
Neevai Esinly
Olaoluwa Osuntokun
OrbitalTurtle
Valentine Wallace
Wilmer Paulino
Xavi Soler
Yaacov Akiba Slama

Assets 26
Pre-release
Pre-release

@cfromknecht cfromknecht released this Jun 29, 2019 · 133 commits to master since this release

This release marks a new major release of lnd that includes several important bug fixes, an improved payment API, pathfinding enhancements, faster initial sync times, support for fee bumping on sweeps, and an initial rollout of altruistic watchtowers. As always, users are highly encouraged to upgrade to this new version.

Database Migrations

This version includes two migrations, the first is in channel.db which modifies the structure of the payment tracking data to support the refactored router and its ability to reliably display payments via the RPC. The migration should look like this:

2019-06-14 21:54:53.576 [INF] LTND: Version: 0.7.0-beta commit=v0.7.0-beta-rc3, build=production, logging=default
2019-06-14 21:54:53.579 [INF] LTND: Active chain: Bitcoin (network=mainnet)
2019-06-14 21:54:53.583 [INF] CHDB: Checking for schema update: latest_version=9, db_version=8
2019-06-14 21:54:53.586 [INF] CHDB: Performing database schema migration
2019-06-14 21:54:53.586 [INF] CHDB: Applying migration #9
2019-06-14 21:54:53.586 [INF] CHDB: Migrating outgoing payments to new bucket structure
2019-06-14 21:54:53.586 [INF] CHDB: Migration of outgoing payment bucket structure completed!

The second migration is in wallet.db, which prunes redundant block data already being stored by the underlying backend. The migration should look like this:

2019-06-14 21:54:53.744 [INF] LNWL: Applying wallet address manager migration #8
2019-06-14 21:54:53.746 [INF] LNWL: Removing block hash entries beyond maximum reorg depth of 10000 from current tip 580748

Once updated, it will not be possible to return to an older version of lnd.

Verifying the Release

In order to verify the release, you'll need to have gpg or gpg2 installed on your system. Once you've obtained a copy (and hopefully verified that as well), you'll first need to import the keys that have signed this release if you haven't done so already:

curl https://keybase.io/bitconner/pgp_keys.asc | gpg --import

Once you have the required PGP keys, you can verify the release (assuming manifest-v0.7.0-beta-rc3.txt and manifest-v0.7.0-beta-rc3.txt.sig are in the current directory) with:

gpg --verify manifest-v0.7.0-beta-rc3.txt.sig

You should see the following if the verification was successful:

gpg: assuming signed data in 'manifest-v0.7.0-beta-rc3.txt'
gpg: Signature made Sat Jun 29 15:00:20 2019 PDT
gpg:                using RSA key 9C8D61868A7C492003B2744EE7D737B67FA592C7
gpg: Good signature from "Conner Fromknecht <conner@lightning.engineering>" [ultimate]

That will verify the signature on the main manifest page which ensures integrity and authenticity of the binaries you've downloaded locally. Next, depending on your operating system you should then re-calculate the sha256 sum of the binary, and compare that with the following hashes (which are included in the manifest file):

bb7ef26ed0a8dfa31f6bd2016d87a5f28db0bba1b5c1cf203c2fda23bbdbe215  lnd-darwin-386-v0.7.0-beta-rc3.tar.gz
41d48f7f0cedcf7b212ace6a34ad4679eb9130c2abc8bf21e9367e08f852d22e  lnd-darwin-amd64-v0.7.0-beta-rc3.tar.gz
a754be7b37456f998c898bd9db31247f35897a5fed9ddaaba2e9a9db6e07c869  lnd-dragonfly-amd64-v0.7.0-beta-rc3.tar.gz
96c756ac172415eb852b65f5fce58ea03adde85aa55ba53708b6dda079e1285f  lnd-freebsd-386-v0.7.0-beta-rc3.tar.gz
4c6f8534d09ebf5a9227a4d272aba88e860f81543a8c4035cccf45e3742ea998  lnd-freebsd-amd64-v0.7.0-beta-rc3.tar.gz
ae6c6f7068b1b05a9a2536f97da497b8c8de27703d482644d4779b88bf97cc0f  lnd-freebsd-arm-v0.7.0-beta-rc3.tar.gz
2723ce9dff50a2b063ba01b2b2cf4159db5aed5ade76a20978dfac361152fa06  lnd-linux-386-v0.7.0-beta-rc3.tar.gz
d90bf078edc57f12cfebfae96aaa6d686a8036a3cb1b8684855f773edd9f2ec7  lnd-linux-amd64-v0.7.0-beta-rc3.tar.gz
a0f40ec55ac9a9898657ede6084b32ae150d2d0483975eb1a6aab3c5fa691f2d  lnd-linux-arm64-v0.7.0-beta-rc3.tar.gz
d6f993aa68d02bc2ee10445e28ba6133fa903a1dae45121581f48585fa60aa40  lnd-linux-armv6-v0.7.0-beta-rc3.tar.gz
92d2cf564714057ebf63f952454e4255e3e16e590178d096f75efc40931ace9a  lnd-linux-armv7-v0.7.0-beta-rc3.tar.gz
3e11af4a161ffc0174718ab0f6e9bf6ade90acd92c0d6fbeaae10bf3471b8a6b  lnd-linux-mips64-v0.7.0-beta-rc3.tar.gz
1932492a36d6ceacb1e073ebf68ee7e4b2a4bdccd26efbccec7d607d5a21cf3f  lnd-linux-mips64le-v0.7.0-beta-rc3.tar.gz
0355917888200b4df575fe8d1600d0c3da591c7771cf0a240402a49e345a37b5  lnd-linux-ppc64-v0.7.0-beta-rc3.tar.gz
e9d7954589b6b7747d83f01b654828ae46cbff537e6d96e4732c43af61031ee6  lnd-netbsd-386-v0.7.0-beta-rc3.tar.gz
8e337e27269f50e3ed87de6649ae09d813d1e7dc6d0177555df138b07da664a6  lnd-netbsd-amd64-v0.7.0-beta-rc3.tar.gz
d1ab52fa7e454414476c6d6291802d44ef43bc0cb50e4c9169f01d1eee8d4547  lnd-openbsd-386-v0.7.0-beta-rc3.tar.gz
169bbfd8141fb41f29b4e68fd81463b7be460a411f11c8de91c9d6675617165d  lnd-openbsd-amd64-v0.7.0-beta-rc3.tar.gz
97831392024dc17f35334d423d7905f400516e4a0f3623388b6a5c870d03914b  lnd-source-v0.7.0-beta-rc3.tar.gz
1e05a93de75269f63c1019d2f0c36a9f85692253ee146e367cffb2cad1cae194  lnd-windows-386-v0.7.0-beta-rc3.zip
232bd9fad897d6a180ade63dc16a128751606ad87653fe0834cd2109c8b5754e  lnd-windows-amd64-v0.7.0-beta-rc3.zip
bea687b821e4647f5a0228c25c4b5f25609a8fe47166f4cd0578ad09a849dbfb  vendor.tar.gz

One can use the shasum -a 256 <file name here> tool in order to re-compute the sha256 hash of the target binary for your operating system. The produced hash should be compared with the hashes listed above and they should match exactly.

Finally, you can also verify the tag itself with the following command:

git verify-tag v0.7.0-beta-rc3

The binaries are compiled with go1.12.6 and include the following build tags: autopilotrpc, signrpc, walletrpc, chainrpc, invoicesrpc, and routerrpc.

Building the Contained Release

With this new version of lnd, we've modified our release process to ensure the bundled release is now fully self contained. As a result, with only the attached payload with this release, users will be able to rebuild the target release themselves without having to fetch any of the dependencies. Note that at this stage, binaries aren't yet fully reproducible (even with go modules). This is due to the fact that by default, Go will include the full directory path where the binary was built in the binary itself. As a result, unless your file system exactly mirrors the machine used to build the binary, you'll get a different binary, as it includes artifacts from your local file system. This will be fixed in go1.13, and before then we may modify our release system to do this automatically.

In order to re-build from scratch, assuming that vendor.tar.gz and lnd-source-v0.7.0-beta-rc3.tar.gz are in the current directory:

tar -xvzf vendor.tar.gz
tar -xvzf lnd-source-v0.7.0-beta-rc3.tar.gz
GO111MODULE=on go install -v -mod=vendor -ldflags "-X github.com/lightningnetwork/lnd/build.Commit=v0.7.0-beta-rc3" ./cmd/lnd
GO111MODULE=on go install -v -mod=vendor -ldflags "-X github.com/lightningnetwork/lnd/build.Commit=v0.7.0-beta-rc3" ./cmd/lncli

The -mod=vendor flag tells the go build command that it doesn't need to fetch the dependencies, and instead, they're all enclosed in the local vendor directory.

Additionally, it's now possible to use the enclosed release.sh script to bundle a release for a specific system like so:

LNDBUILDSYS="linux-arm64 darwin-amd64" ./release.sh

⚡️⚡️⚡️ OK, now to the rest of the release notes! ⚡️⚡️⚡️

Release Notes (in progress)

bitcoind 0.18 compatibility

This new version of lnd has updated the way we check for errors over the RPC interface after we broadcast a transaction to be compatible with bitcoind v0.18. Note that the “master” branch of bitcoind isn’t yet supported yet as it includes distinct RPC changes to sendrawtransaction in particular that will require further updates in lnd. We’re targeting lnd v0.7.1. to fix these remaining RPC compatibility issues.

Private Altruist Watchtowers

In this release, we’re rolling out the ability to run a private, altruist watchtower as a fully-integrated subsystem of lnd. Watchtowers act as a second line of defense in responding to malicious or accidental breach scenarios in the event that client’s node is offline or unable to respond at the time of a breach, offering greater degree of safety to channel funds.

In contrast to a reward watchtower which demand a portion of the channel funds as a reward for fulfilling its duty, an altruist watchtower returns all of the victim’s funds (minus on-chain fees) without taking a cut. Reward watchtowers will be enabled in a subsequent release, though are still undergoing further testing and refinement.

In addition, lnd can now be configured to operate as a watchtower client, backing up encrypted breach-remedy transactions (aka. justice transactions) to other altruist watchtowers. The watchtower stores the fixed-size, encrypted blobs and is only able to decrypt and publish the justice transaction after the offending party has broadcast a revoked commitment state. Client communications with a watchtower are encrypted and authenticated using ephemeral keypairs, mitigating the amount of tracking the watchtower can perform on its clients using long-term identifiers.

Note that we have chosen to deploy a restricted set of features in this release that can begin to provide meaningful security to lnd users. Many more watchtower-related features are nearly complete or have meaningful progress, and we will continue to ship them as they receive further testing and become safe to release.

Configuring a Watchtower

To set up a watchtower, command line users should compile in the optional watchtowerrpc subserver, which will offer the ability to interface with the tower via gRPC or lncli. The minimal configuration needed to activate the tower is watchtower.active=1.

By default, the watchtower will listen on :9911 which specifies port 9911 listening on all available interfaces. Users may configure their own listeners via the --watchtower.listen= option.

Additionally, users can specify their tower’s external IP address(es) using watchtower.externalips, which will expose the full tower URI over RPC. These can be given to clients in order to connect and use the tower.

Note: The watchtower’s public key is distinct from lnd’s node public key. For now this acts as a soft whitelist as it requires clients to know the tower’s public key in order to use it for backups before more advanced whitelisting features are implemented. We recommend NOT disclosing this public key openly, unless you are prepared to open your tower up to the entire Internet.

Retrieving information about your tower’s configurations can be done using lncli tower info:

🏔 lncli tower info
{
        "pubkey": "03281d603b2c5e19b8893a484eb938d7377179a9ef1a6bca4c0bcbbfc291657b63",
        "listeners": [
                "[::]:9911"
        ],
        "uris": [
                "03281d603b2c5e19b8893a484eb938d7377179a9ef1a6bca4c0bcbbfc291657b63@1.2.3.4:9911"
        
}

The entire set of watchtower configuration options can be found using lncli -h:

watchtower:
      --watchtower.active                                     If the watchtower should be active or not
      --watchtower.towerdir=                                  Directory of the watchtower.db (default: $HOME/.lnd/data/watchtower)
      --watchtower.listen=                                    Add interfaces/ports to listen for peer connections
      --watchtower.externalip=                                Add interfaces/ports where the watchtower can accept peer connections
      --watchtower.readtimeout=                               Duration the watchtower server will wait for messages to be received before hanging up on client connections
      --watchtower.writetimeout=                              Duration the watchtower server will wait for messages to be written before hanging up on client connections

Configuring a Watchtower Client

In order to set up a watchtower client, you’ll need the watchtower URI of an active watchtower. lnd can the ben configured to start using the tower by setting:

wtclient.private-tower-uris=03281d603b2c5e19b8893a484eb938d7377179a9ef1a6bca4c0bcbbfc291657b63@1.2.3.4:9911

Users may optionally configure the fee rate of justice transactions by setting the wtclient.sweep-fee-rate option, which accepts values in sat/byte.

The entire set of watchtower client configuration options can be found using lncli -h:

wtclient:
      --wtclient.private-tower-uris=                          Specifies the URIs of private watchtowers to use in backing up revoked states. URIs must be of the form <pubkey>@<addr>. Only 1 URI is supported at this time, if none are provided the tower will not be enabled.
      --wtclient.sweep-fee-rate=                              Specifies the fee rate in sat/byte to be used when constructing justice transactions sent to the watchtower.

For now, no information regarding the operation of the watchtower client is exposed over the RPC interface. We are working to expose this information in a later release, progress on this can be tracked in this PR. Users will be reliant on logs for observing the behavior of the client. We also plan to expand on the initial feature set by permitting multiple active towers for redundancy, as well as modifying the chosen set of towers dynamically without restarting the daemon.

gRPC Prometheus Exports

A build tag has been added: monitoring. If built with this tag, lnd has the ability to export Prometheus metrics about its gRPC performance. These metrics include keeping track of how many times each gRPC call has been invoked, how many bytes have been transmitted and received for each gRPC call, and more.

To enable Prometheus exporting after building lnd with the monitoring tag, run lnd with this flag:

--prometheus.enable

To change the address that lnd listens on for Prometheus, add this flag:

--prometheus.listen=<address>

By default, lnd's Prometheus exporter listens on localhost:8989.

Wallet Bug Fixes

A bug has been fixed that would cause the EstimateFee and SendOutputs RPC commands to fail if conditions of higher fees. The root cause was using the transactions effective fee rate instead of the minrelayfee when computing dust in certain scenarios.

Wallet Chain Sync Improvements

There was a period of time where the wallet would only store the latest 25,000 block headers of the chain in order to properly recover from chain reorgs, but this was modified to the existing behavior of storing all of them due to deep reorgs on the testnet chain during lnds initial testing and development. In this release, we revert back to the prior behavior, but will instead only store the latest 10,000 as a generous safety margin.

As a result of doing so, the initial sync process is sped up. Neutrino backends saw the least improvements in this department, while bitcoind saw the most.

The wallet will also now detect that the chain backend (i.e., btcd, bitcoind, etc.) is currently syncing if its latest block does not have a timestamp within the past 2 hours, allowing us to start lnd while the chain backend syncs.

UtxoSweeper Improvements

Recent versions of lnd have seen development of a new subsystem: the UtxoSweeper. The release notes of our v0.6.0-beta release provide a brief overview of its duties. In this release, we continue to expand its functionality by allowing users to obtain information about the list of inputs being batched by the UtxoSweeper through a new PendingSweeps RPC and its corresponding lncli wallet pendingseeps command.

We also now allow users to bump the fee rate of an input with the Replace-By-Fee (RBF) policy or a transaction with the Child-Pays-For-Parent (CPFP) policy through the BumpFee RPC and its corresponding lncli wallet bumpfee command. In order to provide this functionality, the UtxoSweeper first had to be extended to support sweeping distinct batches of inputs with similar fee preferences.

Fee bumping within lnd will take a different approach than the popular bitcoin-cli bumpfee command. Since the UtxoSweeper batches transactions together, lnd cannot rely on bumping a transaction’s fee by providing its hash since it can change at any point with the addition/removal of new inputs. Instead, fee bumping relies on an outpoint being provided along with a fee preference.

A fee preference can be defined as one of any of the following options, which will likely expand in future releases.

  • Static fee rate: fixed fee rate to spent an input on-chain with.
  • Confirmation target: a delta of blocks from the current height of the chain in which an input should be spend on-chain within.

When bumping the fee of an input that currently exists within the UtxoSweeper, a transaction is created with a fee rate backed by the fee preference provided that replaces the lower fee transaction through the RBF policy. When bumping the fee of an input that points to an unconfirmed transaction output which belongs to the user’s wallet, a transaction that spends this input is created with a fee rate backed by the fee preference provided. This spending transaction can be interpreted as the child in a CPFP scenario.

The current design presents some shortcomings which will be addressed in future releases as we develop things further. These include:

  • Fee preference Validation checks are not in place to ensure they provide a sufficiently higher fee rate in order for a fee bump to be successful. This responsibility is delegated to the user.
  • The UtxoSweeper does not persist any inputs to disk, so restarting lnd will require incomplete fee bump requests to be requested again.

lncli Bug Fixes

We'll now properly parse an optional single or multi channel backup being provided as part of lncli create.

We'll now properly spend unconfirmed UTXOs as part of lncli openchannel when providing a min_confs parameter of 0.

Config Changes

Users can now use the --tlsextraip and --tlsextradomain fields to set multiple IPs or domains when generating certs.

A new neutrino.assertfilterheader filter header flag has been added to lnd. In the past the neutrino protocol underwent a number of changes, some of which affected exactly what is included in the filter. Older clients may have been on persistent incorrect filter chains as a result of this. This new argument allows a client to ensure it has the right filter chain using an externally provided filter header hash. If this hash doesn’t match, then the neutrino client will delete their state and re-sync the filter header chain.

Litecoin support has been extended to also include the simnet and regtest chains.

Interoperability Improvements

CLTV Tie-Breaking in Commitment Output Sorting

A nuanced, spec-level bug has been fixed for properly sorting HTLCs on the commitment transaction. When HTLCs of identical payment hash and amount appear on the same commitment transaction, the original BIP69 sorting could lead to a channel failure if the HTLCs had different CLTV values since the second-layer transactions would have different sighashes. This happens because the CLTV is enforced via nLockTime field of the descendant transaction, and weren’t being considered in the sorting of the commitment outputs. The new spec sorting resolves this by using the CLTV values of HTLC outputs as tiebreakers when the payment hash and amount are identical.

Since 0.5, lnd has forbidden sending out more than one HTLC for the same payment hash at a given time, so users on pre-0.7 versions are unlikely to experience this bug in purely lnd to lnd interactions, or have triggered it on other nodes in the network. However, it’s possible that forwarding payments from implementations that permit duplicate payments can trigger the bug and result in a channel closure. This bug should be completely eliminated going forward.

Increased Non-initiator Funding Timeout

When another node initiates a channel open with lnd, the non-initiator forgets about the channel if it doesn’t confirm in timely manner. In 0.7, this timeout has been raised to two weeks from the prior value of two days. This allows lnd to tolerate channel initiators with a low time preference (set a low fee rate) and better tolerate fee spikes or on-chain congestion during the funding process.

P2P and Gossip Improvements

Prior versions of lnd would send out all updates (or new channels) to connected active syncing peers in a single payload. This could at times cause slight memory spikes if many channels were mined in a single block for example. This latest version of lnd will instead now split announcements into multi sub-batches when it’s time to flush the set of pending messages. This change reduces GC pressure, and also naturally implements a variant of flow control to ensure minimal load during announcement batch flushes. It also make sure to redo a historical sync the first time it gets a peer

RPC Server and RPC Sub-Server Changes

lnd’s RPC certificates will now be automatically re-generated by the daemon if upon restart, we detect that they’ve expired. Many nodes that were created greater than a year ago may have run into this issue, and had to delete and then regenerate themselves.

The NodeUpdate and GetNodeInfo commands now include the color of the target node.

The closeallchannels command within lncli now exposes a conf_target or sat_per_byte arguments in order to allow the user to control the fee rate used to close all transactions.

Transaction Hex in GetTransactions

The GetTransactions RPC call now also includes the serialized hex version of the transaction. This exposes information the wallet already has and can be helpful for external accounting systems connected to lnd, or for usage with the new BumpFee command added in this version.

Display Paid Payment Requests

When making payments via the SendPayment RPC, the paid invoice will now be displayed in the ListPayments results. Similarly, payments via lncli payinvoice will show the invoice in lncli listpayments. This allows applications to recover the details, e.g. the invoice memo, without needing to track them separately.

Zero Fees With UpdateChanPolicy

[It’s now possible to set fees to rock bottom (zero!) using lncli updatechanpolicy or the UpdateChanPolicy RPC command]#3139).

Channel Display in GetNodeInfo

The GetNodeInfo command will now optionally also return the set of channels belonging to a particular node. This is useful as it saves additional look ups, and a client-side mapping between channel ID and node using the DescribeGraph command. Arguments for the new lncli command are as follows:

⛰   lncli getnodeinfo -h
NAME:
   lncli getnodeinfo - Get information on a specific node.

USAGE:
   lncli getnodeinfo [command options] [arguments...]

CATEGORY:
   Peers

DESCRIPTION:
   Prints out the latest authenticated node state for an advertised node

OPTIONS:
   --pub_key value     the 33-byte hex-encoded compressed public of the target node
   --include_channels  if true, will return all known channels associated with the node

Structured SendToRoute Errors

A new SendToRoute rpc has been added to the router sub-server. This call improves upon the main rpc SendToRoute rpc by returning errors as proper proto messages. No more (regex) string parsing. As part of this change, the (deprecated) option to pass in multiple routes to the main SendToRoute was removed.

Payment RPCs

The interface of SendPayment in the router sub-server has been modified. It now takes a custom payment timeout value. This defines the duration after which no new attempts to complete the payment are initiated. The call also requires the fee limit to be set explicitly. The fee limit is an important parameter and not having a default value for this forces callers to make a conscious decision about its value. Another change is that SendPayment only returns payment-level failures. This set of failures is currently limited, but will be extended in the future. The previously returned last payment attempt error is no longer available, because its informative value is considered limited.

Furthermore a new call TrackPayment was added. TrackPayment can be used to pick up and track the status of pending payments, for example after losing the connection or across restarts.

Payment Persistence

A series of PRs (#3064, #3063, #2761, #2762) has been merged, making lnd better handle in-flights payments across restarts. lnd will now write any attempted payment to the database, such that should the daemon restart before the result comes back, it will correctly record it to the DB. This means that payments can now be in states “in flight”, “failed” or “succeeded”, which can all be shown by calling lncli listpayments --include_incomplete (#3190).

Probability Based Mission Control

In prior versions of lnd, when we encountered a path finding failure, we would simply ignore the edge or node that caused for error for 5 or 15 seconds, continuing one with that path finding session. This naive approach has a number of issues. As an example, this means that shortly after a node makes a payment, it forgets all the prior context it used to successfully route the payment. Additionally, the prior version didn’t share this information amongst concurrent in-flight payment sessions, meaning each session couldn’t learn from attempts of other on going sessions. Users may have noticed this “cold start” effect when they go to send a new payment a few minutes the last successful payment attempt. In older versions of lnd, the path finding would try a series of channels that it likely knew wouldn’t work, before finally finding the golden set of channels that reliably work.

In this new version of lnd, the previously used edge and node black lists in path finding are replaced by a probability based system. It modifies path finding so that it not only compares routes on fee and time lock, but also takes route success probability into account. Each channel starts out with a default “apriori” probability. As path finding attempts fail, we reduce the probability of channels that have failed to zero, exponentially decaying back to the default apiori probability. This means that lnd will now properly “learn” from its past path finding attempts. Additionally, since this algorithm is less greedy than the prior algorithm, it may return a distinct set of paths compared to the prior algorithm, exploring avenues that maybe didn’t seem “optimal”, but still have a high probability of success.

The new system is similar to “active learning” based approaches in machine learning. lnd will now continually update its view of the network based on empirical observations of payment failures. In addition to that, historical payment attempt outcomes are kept in mission control memory longer and improve path finding for future payments. A new diagnostic rpc has been added to query the current mission control state. Several configuration options are exposed to give more control of mission control behavior:

      --routerrpc.minrtprob=                                  Minimum required route success probability to attempt the payment (default: 0.01)
      --routerrpc.apriorihopprob=                             Assumed success probability of a hop in a route when no other information is available. (default: 0.95)
      --routerrpc.penaltyhalflife=                            Defines the duration after which a penalized node or channel is back at 50% probability (default: 1h0m0s)
      --routerrpc.attemptcost=                                The (virtual) cost in sats of a failed payment attempt (default: 100)

A new querymc sub-server RPC command has been added to allow users to peek into this diagnostic information. Eventually users may be able to share their mission control state (the effective “memory” of the router), amongst their other nodes in order to ensure they don’t need to re-discover the set of reliable channels.

These changes prepare for further enhancements that will increase payment reliability.

The deprecated num_routes parameter of QueryRoutes has been removed.

Invoice Settling

Several changes with respect to invoice settling have been made. This makes the logic of off-chain and on-chain invoice settling more consistent and is a preparatory step for atomic multiple path payments.

Neutrino Improvements

All integration tests are now run continuously with the neutrino chain backend , in addition to the existing tests using btcd!

When running lnd with neutrino active, the daemon will now properly shutdown even if it’s in the middle of checking for the spentness of a particular UTXO.

Static Channel Backups

A bug fix has been made to the current on-disk SCB format in the Windows operating system. Before this release, lnd wouldn’t properly close the temporary SCB file before swapping it over to the permanent file name (atomic file rename). On Windows in particular, this would cause the swap to fail, meaning that the file wouldn’t be updated on disk. We’ve fixed this issue in this latest release, by always closing the temporary file before we attempt to modify it.

Changelog

The full list of changes since 0.6.1-beta can be found here:

Contributors (Alphabetical Order)

2xic
AdamISZ
chokoboko
Conner Fromknecht
Daniel McNally
Federico Bond
Francisco Calderón
Geoff Taylor
Johan T. Halseth
John Griffith
Joost Jager
Matt Drollette
michael1011
Neevai Esinly
Olaoluwa Osuntokun
OrbitalTurtle
Valentine Wallace
Wilmer Paulino
Xavi Soler
Yaacov Akiba Slama

Assets 26
Jun 21, 2019
lnd v0.7.0-beta-rc2
Pre-release
Pre-release

@cfromknecht cfromknecht released this Jun 18, 2019 · 211 commits to master since this release

This release marks a new major release of lnd that includes several important bug fixes, an improved payment API, pathfinding enhancements, faster initial sync times, support for fee bumping on sweeps, and an initial rollout of altruistic watchtowers. As always, users are highly encouraged to upgrade to this new version.

Database migrations

This version includes two migrations, the first is in channel.db which modifies the structure of the payment tracking data to support the refactored router and its ability to reliably display payments via the RPC. The migration should look like this:

2019-06-14 21:54:53.576 [INF] LTND: Version: 0.7.0-beta commit=v0.7.0-beta-rc1, build=production, logging=default
2019-06-14 21:54:53.579 [INF] LTND: Active chain: Bitcoin (network=mainnet)
2019-06-14 21:54:53.583 [INF] CHDB: Checking for schema update: latest_version=9, db_version=8
2019-06-14 21:54:53.586 [INF] CHDB: Performing database schema migration
2019-06-14 21:54:53.586 [INF] CHDB: Applying migration #9
2019-06-14 21:54:53.586 [INF] CHDB: Migrating outgoing payments to new bucket structure
2019-06-14 21:54:53.586 [INF] CHDB: Migration of outgoing payment bucket structure completed!

The second migration is in wallet.db, which prunes redundant block data already being stored by the underlying backend. The migration should look like this:

2019-06-14 21:54:53.744 [INF] LNWL: Applying wallet address manager migration #8
2019-06-14 21:54:53.746 [INF] LNWL: Removing block hash entries beyond maximum reorg depth of 10000 from current tip 580748

Once updated, it will not be possible to return to an older version of lnd.

Verifying the Release

In order to verify the release, you'll need to have gpg or gpg2 installed on your system. Once you've obtained a copy (and hopefully verified that as well), you'll first need to import the keys that have signed this release if you haven't done so already:

curl https://keybase.io/bitconner/pgp_keys.asc | gpg --import

Once you have the required PGP keys, you can verify the release (assuming manifest-v0.7.0-beta.txt and manifest-v0.7.0-beta.txt.sig are in the current directory) with:

gpg --verify manifest-v0.7.0-beta.txt.sig

You should see the following if the verification was successful:

gpg: assuming signed data in 'manifest-v0.7.0-beta-rc1.txt'       
gpg: Signature made Fri Jun 14 14:24:37 2019 PDT                  
gpg:                using RSA key 9C8D61868A7C492003B2744EE7D737B67FA592C7                                                           
gpg: Good signature from "Conner Fromknecht <conner@lightning.engineering>" [ultimate]  

That will verify the signature on the main manifest page which ensures integrity and authenticity of the binaries you've downloaded locally. Next, depending on your operating system you should then re-calculate the sha256 sum of the binary, and compare that with the following hashes (which are included in the manifest file):

a0af101b730d3f30cb85d46ce9b70f639ed845318deb72f08196714974a9a4f7  lnd-darwin-386-v0.7.0-beta-rc1.tar.gz
2385d2dcdcf2df7623d51611074968ecc575943e7fd06e0c31821d4454ff1cc4  lnd-darwin-amd64-v0.7.0-beta-rc1.tar.gz
ba584ff4528b7a8e91785c2b53258fcae795d8faa2ab5f71b74a0111ac1ea885  lnd-dragonfly-amd64-v0.7.0-beta-rc1.tar.gz
e19ee77f5c680b3dd9cdd9286273ae076f653e31a930ec2e268ac38a2102c11a  lnd-freebsd-386-v0.7.0-beta-rc1.tar.gz
210f1977bd9e6c1904a9ef215575e7ec43bb52775d04f7e431d9919c1df3c31c  lnd-freebsd-amd64-v0.7.0-beta-rc1.tar.gz
4a1b7783e928ac56887b874851595983e62414a95e67bbd422052a7a75b4a760  lnd-freebsd-arm-v0.7.0-beta-rc1.tar.gz
034570245b113074d9b1ccaf6f74b5fe16d2cd06ba97fe357cd8d77dd3d2f744  lnd-linux-386-v0.7.0-beta-rc1.tar.gz
6cb52c42c6b837b8dda35124cb74c591b3f9167f92e73d0e3ea46b359cb5bdf4  lnd-linux-amd64-v0.7.0-beta-rc1.tar.gz
5f4666a9c12e578c41faea4868cd5ea447dfa0e80ce7a282eced36e1205df968  lnd-linux-arm64-v0.7.0-beta-rc1.tar.gz
0573093e4d84213941adab88cc5cd604f9b3e1344a28bb7b8894d2302fa05022  lnd-linux-armv6-v0.7.0-beta-rc1.tar.gz
24f4860d44d726a0e2fb14c5a368d521637b9ecc009b552a8ce987b0b28f3544  lnd-linux-armv7-v0.7.0-beta-rc1.tar.gz
4ea4e15e4b15c8c2ad86590c30315200090dcb74173448e394e3e4e4dae78415  lnd-linux-mips64-v0.7.0-beta-rc1.tar.gz
4bc989aececec53c42f8d8338e3f25358b0e05de94223259f6222b6230a1c498  lnd-linux-mips64le-v0.7.0-beta-rc1.tar.gz
876996e045ad241b594048223069be2f96dc8c4d46aaf81d7e3c47327c3b6ecc  lnd-linux-ppc64-v0.7.0-beta-rc1.tar.gz
d402847b1efeb039aa72fbf20320df30161a7aa3dca54b09de4b29c8f499f03a  lnd-netbsd-386-v0.7.0-beta-rc1.tar.gz
4cb2996dfb5fa42a9a771f0631631f102080512d406d942d5816ecc9c1d5705e  lnd-netbsd-amd64-v0.7.0-beta-rc1.tar.gz
4e7dd42d39e81cdbfb2de14f67415e0c3c986eeaa7d9ed9ec029b7d6bf963f73  lnd-openbsd-386-v0.7.0-beta-rc1.tar.gz
8bf0514c74d80584d87544f07a2e520ca9311ef7a1a6e42e9029ac8fcb3b533b  lnd-openbsd-amd64-v0.7.0-beta-rc1.tar.gz
f13d1628ccba4bc963e7ed2a81be0ed0cd6c9c11ea71a4a7a68a4ff01d2a4d0b  lnd-source-v0.7.0-beta-rc1.tar.gz
04b0768147197846603a0465b962c6fec3dfb315e3b446b0d148620a81e8266c  lnd-windows-386-v0.7.0-beta-rc1.zip
1a9ee1d3e30e69e14cb00ef982e0a63d05ed39f74e3fc7d3a16b31efe8685a89  lnd-windows-amd64-v0.7.0-beta-rc1.zip
21098c814a77dfdff0bf53329f8337ee69b49fa27829507402934a60147a9dcc  vendor.tar.gz

One can use the shasum -a 256 <file name here> tool in order to re-compute the sha256 hash of the target binary for your operating system. The produced hash should be compared with the hashes listed above and they should match exactly.

Finally, you can also verify the tag itself with the following command:

git verify-tag v0.7.0-beta

The binaries are compiled with go1.12.6 and include the following build tags: autopilotrpc, signrpc, walletrpc, chainrpc, invoicesrpc, and routerrpc.

Building the Contained Release

With this new version of lnd, we've modified our release process to ensure the bundled release is now fully self contained. As a result, with only the attached payload with this release, users will be able to rebuild the target release themselves without having to fetch any of the dependencies. Note that at this stage, binaries aren't yet fully reproducible (even with go modules). This is due to the fact that by default, Go will include the full directory path where the binary was built in the binary itself. As a result, unless your file system exactly mirrors the machine used to build the binary, you'll get a different binary, as it includes artifacts from your local file system. This will be fixed in go1.13, and before then we may modify our release system to do this automatically.

In order to re-build from scratch, assuming that vendor.tar.gz and lnd-source-v0.7.0-beta.tar.gz are in the current directory:

tar -xvzf vendor.tar.gz
tar -xvzf lnd-source-v0.7.0-beta.tar.gz
GO111MODULE=on go install -v -mod=vendor -ldflags "-X github.com/lightningnetwork/lnd/build.Commit=v0.7.0-beta" ./cmd/lnd
GO111MODULE=on go install -v -mod=vendor -ldflags "-X github.com/lightningnetwork/lnd/build.Commit=v0.7.0-beta" ./cmd/lncli

The -mod=vendor flag tells the go build command that it doesn't need to fetch the dependencies, and instead, they're all enclosed in the local vendor directory.

Additionally, it's now possible to use the enclosed release.sh script to bundle a release for a specific system like so:

LNDBUILDSYS="linux-arm64 darwin-amd64" ./release.sh

⚡️⚡️⚡️ OK, now to the rest of the release notes! ⚡️⚡️⚡️

Release Notes

Coming soon...

Changelog

The full list of changes since 0.6.1-beta can be found here:

Contributors (Alphabetical Order)

AdamISZ
chokoboko
Conner Fromknecht
Daniel McNally
Federico Bond
Francisco Calderón
Geoff Taylor
Johan T. Halseth
John Griffith
Joost Jager
Matt Drollette
michael1011
Neevai Esinly
Olaoluwa Osuntokun
Turtle
Valentine Wallace
Wilmer Paulino
Xavi Soler
Yaacov Akiba Slama

Assets 26

@Roasbeef Roasbeef released this May 9, 2019 · 543 commits to master since this release

This release marks a minor release in the 0.6-beta series. This release contains no major new features, and instead contains a series of bug-fixes, optimizations, and stability improvements.

Verifying the Release

In order to verify the release, you'll need to have gpg or gpg2 installed on your system. Once you've obtained a copy (and hopefully verified that as well), you'll first need to import the keys that have signed this release if you haven't done so already:

curl https://keybase.io/roasbeef/pgp_keys.asc | gpg --import

Once you have his PGP key you can verify the release (assuming manifest-v0.6.1-beta.txt and manifest-v0.6.1-beta.txt.sig are in the current directory) with:

gpg --verify manifest-v0.6.1-beta.txt.sig

You should see the following if the verification was successful:

gpg: assuming signed data in 'manifest-v0.6.1-beta.txt'
gpg: Signature made Thu May  9 16:31:08 2019 PDT
gpg:                using RSA key F8037E70C12C7A263C032508CE58F7F8E20FD9A2
gpg: Good signature from "Olaoluwa Osuntokun <laolu32@gmail.com>" [ultimate]

That will verify the signature on the main manifest page which ensures integrity and authenticity of the binaries you've downloaded locally. Next, depending on your operating system you should then re-calculate the sha256 sum of the binary, and compare that with the following hashes (which are included in the manifest file):

8fd4bfb26d402a4882f4c89ab700186431c178af0a9f0bf61513bbb48cd497c3  lnd-darwin-386-v0.6.1-beta.tar.gz
02330ede4e7508a37e92bcc4c0dd0ac977f51a8fd42be9f114b1e42e58ff07c9  lnd-darwin-amd64-v0.6.1-beta.tar.gz
60c7aef87789271cc1c55286068ce2e6e94ccae48d85465df0606fe96bbe01cd  lnd-dragonfly-amd64-v0.6.1-beta.tar.gz
8f328ef03e2f6fb47aab28af7a8949d3fadaac5769382d45ebbdf2fca7921593  lnd-freebsd-386-v0.6.1-beta.tar.gz
f50b91b6c0e95b2a5c0e246fb17e07eddefb77533b10c034bb5454ad4001edcd  lnd-freebsd-amd64-v0.6.1-beta.tar.gz
9e2c4b926140aacfc680e53191b1b56a18b0def422b1f8f4cc66536298721115  lnd-freebsd-arm-v0.6.1-beta.tar.gz
00a7cd0ca657bb242b0f3acb5f4e26a13fd789946fab73c252118e3f89c1cf57  lnd-linux-386-v0.6.1-beta.tar.gz
c55367edb82955dc942baf9f48f79fadde1eee0e86c1d59d2fe1993140ec1b3f  lnd-linux-amd64-v0.6.1-beta.tar.gz
d5f7280c324ebc1d322435a0eac4c42dca73ebc6a613878d9e0d33a68276da5c  lnd-linux-arm64-v0.6.1-beta.tar.gz
00ff9c61fbd272863aef677db240b04eecdaae0cfb479cf25c713b89ff81d41c  lnd-linux-armv6-v0.6.1-beta.tar.gz
5541959c7fde98d76d88cc8070ca626c681ba38c44afcb85bf417a9a677e23c2  lnd-linux-armv7-v0.6.1-beta.tar.gz
ed8aa2ef8f4e42651a16c90d5ab2d4485396b4c795bc183fd01ae63b39d4201e  lnd-linux-mips64-v0.6.1-beta.tar.gz
a97dfc432747f4dcd6deb8f263b18453c68b4882b1cc1ff84d90f25ba2ed5151  lnd-linux-mips64le-v0.6.1-beta.tar.gz
95ff869a1bbcba20b5c6387f09175eaf3a16f2026e5113eff9abb7b595039423  lnd-linux-ppc64-v0.6.1-beta.tar.gz
5efb454850f41f69a397d4a9cd4778f6a251ce21cf796960b60d6799ff128bf1  lnd-netbsd-386-v0.6.1-beta.tar.gz
6059ab2d6d03604d48c69eeaf1f59f2131b1ce76f714629a6e1c2063f4da4788  lnd-netbsd-amd64-v0.6.1-beta.tar.gz
5db4391a3d5646c3adcdaa1f57fbd9a8b9cfb585e38cccc37729bd5fe2bca8e6  lnd-openbsd-386-v0.6.1-beta.tar.gz
c5b84fe5982ae4b7745d0f083d123d304f97c5da4e6fe515f84b3272495fb1a2  lnd-openbsd-amd64-v0.6.1-beta.tar.gz
68fcede095d6e4f038ba49c46f24a07ff2d07ea11eab95d2d02d5c1467d4d2fc  lnd-source-v0.6.1-beta.tar.gz
e2cb484fda567eb03a534f36613771b1bc10c7220b255b3972784f38abdc2342  lnd-windows-386-v0.6.1-beta.zip
c370735f280ab2b5f6421dafccbdc0e09ff5cfe5f3a65f87155d5cf1e20b1249  lnd-windows-amd64-v0.6.1-beta.zip
afb75c146f03bc4d2af77c49ac4ced88f945a5d3cb9ba32181bc9145149e6af6  vendor.tar.gz

One can use the shasum -a 256 <file name here> tool in order to re-compute the sha256 hash of the target binary for your operating system. The produced hash should be compared with the hashes listed above and they should match exactly.

Finally, you can also verify the tag itself with the following command:

git verify-tag v0.6.1-beta

Building the Contained Release

With this new version of lnd, we've modified our release process to ensure the bundled release is now fully self contained. As a result, with only the attached payload with this release, users will be able to rebuild the target release themselves without having to fetch any of the dependencies. Note that at this stage, binaries aren't yet fully reproducible (even with go modules). This is due to the fact that by default, Go will include the full directory path where the binary was built in the binary itself. As a result, unless your file system exactly mirrors the machine used to build the binary, you'll get a different binary, as it includes artifacts from your local file system. This will be fixed in go1.13, and before then we may modify our release system to do this automatically.

In order to re-build from scratch, assuming that vendor.tar.gz and lnd-source-v0.6-beta.tar.gz are in the current directory:

tar -xvzf vendor.tar.gz
tar -xvzf lnd-source-v0.6.1-beta.tar.gz
GO111MODULE=on go install -v -mod=vendor -ldflags "-X github.com/lightningnetwork/lnd/build.Commit=v0.6.1-beta" ./cmd/lnd
GO111MODULE=on go install -v -mod=vendor -ldflags "-X github.com/lightningnetwork/lnd/build.Commit=v0.6.1-beta" ./cmd/lncli

The -mod=vendor flag tells the go build command that it doesn't need to fetch the dependencies, and instead, they're all enclosed in the local vendor directory.

Additionally, it's now possible to use the enclosed release.sh script to bundle a release for a specific system like so:

LNDBUILDSYS="linux-arm64 darwin-amd64" ./release.sh

The release.sh script will now also properly include the commit hash once again, as a regression caused by a change to the internal build system has been fixed.

⚡️⚡️⚡️ OK, now to the rest of the release notes! ⚡️⚡️⚡️

Release Notes

Tor

A bug has been fixed in the minimum version verification for the Tor daemon that lnd enforces. Before this fix, lnd would at times fail to connect to a Tor daemon with a version beyond the minimum version we require.

Protocol and Cross Implementation Compatibility Improvements

lnd will now properly handle the conversion from an UpdateFailMalformedHTLC error to a regular UpdateFailHTLC error. Before this fix, lnd would incorrectly send an error that wasn't able to be decrypted by the sender of an the original HTLC.

On-Chain Commitment/HTLC Handling

A series of parameters related to when lnd will go to chain for an HTLC, and also when it will reject an incoming/outgoing HTLC for being "too close for comfort" have been adjusted. As a result, it's no longer possible for lnd to accept an HTLC then immediately go to chain for it. Additionally, it's now possible for lnd to forward an HTLC as the last hop to a destination with a very low final CLTV delta.

lnd will now detect a local force close (self initiated) based on the outputs rather then the old txid based comparison. This patches an edge case related to SCB restoration where if a user broadcasted a force close then restored their SCB, lnd would miss this local force close transaction and attempt to redeem using an incorrect path. This bug has never been observed in the wild, but nevertheless warranted patching.

Gossip and P2P Handling

Non Write Pool Blocking Writes

Flushing a message to the socket is no longer blocks the write pool. One result of this change is that we are now we are able to gracefully handle timeout errors and resume partially sent messages. As of #2819, if a message is partially written to the wire due to a timeout error, we will try to write the full message again. In turn, this will produce an authentication error on the remote side (after it is able to read the number of bytes specified in the header) since the middle of the latter ciphertext will not pass as the MAC check.

We resolve this by splitting the Write call into two subroutines, WriteMessage and Flush. WriteMessage encrypts the plaintext and buffers the ciphertext on the underlying connection. A subsequent call to Flush will then attempt to write the bytes out on the wire. If a timeout error is encountered during flush, we can safely resume the byte stream by calling Flush again. In addition to being able to resume partial writes, this also has a yuuuge benefit in not blocking the write pool with network operations. This fully decouples the number of write pool workers from the number of peers, since the blocking operations now take place in each peer.

Synchronous Gossip Response Writes

All replies to gossip query messages are now fully synchronous. This means that if a peer requests a set of channels, then we won't send the next message until after the current message has been flushed to the socket. Node operators should find that start up is now much snappier, and the post start up memory burst to also be much lower.

SyncManager Simplification and Improvements

We've made some improvements to the recently introduced SyncManager. These improvements include:

  • Active syncers will no longer attempt to synchronize our graph with remote peers. Instead, we'll now rely on synchronizing our graph with remote peers through the routine historical syncs performed by the SyncManager. Since active syncers will no longer attempt this synchronization, the SyncManager's round-robin is no longer needed.
  • Handling initial historical sync disconnections: Every time lnd starts up, it attempts an initial historical graph sync with the first peer that connects. If the peer ends up disconnecting, then we wouldn't handle finding a replacement to continue performing the initial historical sync.
  • Queueing active syncers until the initial historical sync completes: We do this to ensure we can properly handle any new channel updates at tip. This is required for fresh nodes that are syncing the channel graph for the first time. If we begin accepting updates at tip while the initial historical sync is still ongoing, then we risk not processing certain updates since we've yet to learn of the channels themselves.

RPC Bug Fixes

Sends to regular p2pk(Pay to PubKey) addresses are now disallowed.

The route cache has been removed. After updates to the SendToRoute RPC command, the cache could at times cause an incorrect result from QueryRoutes.

Networking Bug Fixes

We've fixed an issue that would cause our automatic NAT traversal to fail on certain Linux distributions.

It's now again possible for the gRPC server to listen on both IPv4 and IPv6 simultaneously.

lnd will now automatically re-create port port forwarding if we detect that the external public IP has changed.

Litecoin

lnd is now able to connect to Litecoin coins running on the regest network as well as the simnet network.

No More main Package!

There is no longer a main package, instead the top-level package within the project is now lnd. This is a prep for changes to make lnd easier to embed in mobile applications for iOS and Android.

Changelog

The full list of changes since 0.6.0-beta can be found here:

Contributors (Alphabetical Order)

Chris Coverdale
Conner Fromknecht
Damian Mee
Danny Paz
frennkie
Johan T. Halseth
Joost Jager
Matt Drollette
Offer Markovich
Olaoluwa Osuntokun
Robert Habermann
Valentine Wallace
Wilmer Paulino

Assets 26
May 6, 2019
lnd v0.6.1-beta-rc2
May 4, 2019
lnd v0.6.1-beta-rc1
Apr 23, 2019
lnd v0.6.0-beta

@Roasbeef Roasbeef released this Apr 17, 2019 · 687 commits to master since this release

This release marks a new major release of lnd that includes several important bug fixes, numerous performance optimizations, static channel backups (SCB), reduced bandwidth usage for larger nodes, an overhaul of the internals of the autopilot system, and a new batch sweeping sub-system. Due to the nature of some of the bug fixes which were made during the implementation of the new SCB feature, users are highly encouraged to upgrade to this new version.

Database migrations

This version includes a single migration to modify the message store format, used to send messages to remote peers reliably when attempting to construct channel proofs. The migration should appear as below:

2019-04-03 22:35:44.596 [INF] LTND: Version: 0.6.0-beta commit=v0.6-beta, build=production, logging=default
2019-04-03 22:35:44.596 [INF] LTND: Active chain: Bitcoin (network=mainnet)
2019-04-03 22:35:44.597 [INF] CHDB: Checking for schema update: latest_version=8, db_version=7
2019-04-03 22:35:44.597 [INF] CHDB: Performing database schema migration
2019-04-03 22:35:44.597 [INF] CHDB: Applying migration #8
2019-04-03 22:35:44.597 [INF] CHDB: Migrating to the gossip message store new key format
2019-04-03 22:35:44.597 [INF] CHDB: Migration to the gossip message store new key format complete!

Verifying the Release

In order to verify the release, you'll need to have gpg or gpg2 installed on your system. Once you've obtained a copy (and hopefully verified that as well), you'll first need to import the keys that have signed this release if you haven't done so already:

curl https://keybase.io/roasbeef/pgp_keys.asc | gpg --import

Once you have his PGP key you can verify the release (assuming manifest-v0.6-beta.txt and manifest-v0.6-beta.txt.sig are in the current directory) with:

gpg --verify manifest-v0.6-beta.txt.sig

You should see the following if the verification was successful:

gpg: assuming signed data in 'manifest-v0.6-beta.txt'
gpg: Signature made Tue Apr 16 14:35:13 2019 PDT
gpg:                using RSA key F8037E70C12C7A263C032508CE58F7F8E20FD9A2
gpg: Good signature from "Olaoluwa Osuntokun <laolu32@gmail.com>" [ultimate]

That will verify the signature on the main manifest page which ensures integrity and authenticity of the binaries you've downloaded locally. Next, depending on your operating system you should then re-calculate the sha256 sum of the binary, and compare that with the following hashes (which are included in the manifest file):

e9f3ff0f551f3ce4ad113ad10f8009a3aaca82fb6cd0244a994c299602b29334  lnd-darwin-386-v0.6-beta.tar.gz
76816b2d0d0e3f4c0e7d41c0ecb1457afe2ba95b8f37f4aa1adebbd9bc19aa4b  lnd-darwin-amd64-v0.6-beta.tar.gz
f7650749dc50c3f8c1957680333d95562b159ed13fea26737dc29bff76212925  lnd-dragonfly-amd64-v0.6-beta.tar.gz
7b77ecbfcffb3e2151ff54c27133aebe0d9b6324c80594bce4df265b5f990f61  lnd-freebsd-386-v0.6-beta.tar.gz
d48ce7ed7cc71e988af65e4175e949a5e52f2b8109f5239ae595edc3b8442f05  lnd-freebsd-amd64-v0.6-beta.tar.gz
c783ce9987577d2b7a5e95b6a16767158ef98f48d0eeedf58f4c3a1ce7500e6d  lnd-freebsd-arm-v0.6-beta.tar.gz
cde995167b428696cd6e78733fd934ebda9e03c0b63938af4654c42bd2d86e88  lnd-linux-386-v0.6-beta.tar.gz
ef37b3658fd864dfb3af6af29404d92337229378c24bfb78aa2010ede4cd06af  lnd-linux-amd64-v0.6-beta.tar.gz
2f31b13a4da6217ed7e27a44e1705103d7ed846aa2f599b7e5de0e6033a66c19  lnd-linux-arm64-v0.6-beta.tar.gz
ae8571de0e033a05279469348102982fcfbd3f88c83d549a3d43165ab8ab5aa0  lnd-linux-armv6-v0.6-beta.tar.gz
effea372c207293fd42b0cc27800da3a70c22f8c9a0e7b5eb8dbe56b5b98e1a3  lnd-linux-armv7-v0.6-beta.tar.gz
61038e6cd67562ba3d832de38917d6d37b0cb74fe5e32d4d41fb6d9193f8109d  lnd-linux-mips64-v0.6-beta.tar.gz
28e5be6510fbae4f893253b34db0fcc92d720016f46abe00684a00d2d11a1be3  lnd-linux-mips64le-v0.6-beta.tar.gz
5c13f83344d2634763cf4e178a2d2ca69031a985030713d585d3b37f7a261c06  lnd-linux-ppc64-v0.6-beta.tar.gz
07e91fc56cb0cfcfe52dcaa2bdec008c401b04fe466e970449bcdb4ebb6bb077  lnd-netbsd-386-v0.6-beta.tar.gz
465f4649bdb1393543de52b0dc60aa6121fad0bcf5ad8e7ff62a72c2484dd264  lnd-netbsd-amd64-v0.6-beta.tar.gz
f75b70cf657bffef6cbf2147f69e4296fb98adb48bd18e26950aedb7802748e9  lnd-openbsd-386-v0.6-beta.tar.gz
e5ce0e16a815d5ad98a60c9f7a148efcdb083c0af73965962156f2e3fc03e0df  lnd-openbsd-amd64-v0.6-beta.tar.gz
2c46e9e1f519fe7b0177f30c77611591023442df63c0a1e186154baf7dd9a284  lnd-source-v0.6-beta.tar.gz
77069e9971cd3240891698c02d821ae28254f765c77b5f943b6b88b4943434e7  lnd-windows-386-v0.6-beta.zip
b84de3702074f7e6ecab5f60a1489fb4ee9cd83bcf7c7e9a44604c600ff1d37e  lnd-windows-amd64-v0.6-beta.zip
31205f95fcf7bab7eeff043807ef0485ca14f4506e2e3864db81411ef637aebc  vendor.tar.gz

One can use the shasum -a 256 <file name here> tool in order to re-compute the sha256 hash of the target binary for your operating system. The produced hash should be compared with the hashes listed above and they should match exactly.

Finally, you can also verify the tag itself with the following command:

git verify-tag v0.6-beta

Building the Contained Release

With this new version of lnd, we've modified our release process to ensure the bundled release is now fully self contained. As a result, with only the attached payload with this release, users will be able to rebuild the target release themselves without having to fetch any of the dependencies. Note that at this stage, binaries aren't yet fully reproducible (even with go modules). This is due to the fact that by default, Go will include the full directory path where the binary was built in the binary itself. As a result, unless your file system exactly mirrors the machine used to build the binary, you'll get a different binary, as it includes artifacts from your local file system. This will be fixed in go1.13, and before then we may modify our release system to do this automatically.

In order to re-build from scratch, assuming that vendor.tar.gz and lnd-source-v0.6-beta.tar.gz are in the current directory:

tar -xvzf vendor.tar.gz
tar -xvzf lnd-source-v0.6-beta.tar.gz
GO111MODULE=on go install -v -mod=vendor -ldflags "-X github.com/lightningnetwork/lnd/build.Commit=v0.6-beta"
GO111MODULE=on go install -v -mod=vendor -ldflags "-X github.com/lightningnetwork/lnd/build.Commit=v0.6-beta" ./cmd/lncli

The -mod=vendor flag tells the go build command that it doesn't need to fetch the dependencies, and instead, they're all enclosed in the local vendor directory.

Additionally, it's now possible to use the enclosed release.sh script to bundle a release for a specific system like so:

LNDBUILDSYS="linux-arm64 darwin-amd64" ./release.sh

The release.sh script will now also properly include the commit hash once again, as a regression caused by a change to the internal build system has been fixed.

⚡️⚡️⚡️ OK, now to the rest of the release notes! ⚡️⚡️⚡️

Release Notes

Protocol and Cross-Implementation Compatibility Fixes

We’ll now properly validate our own announcement signatures for NodeAnnouncements before writing them to disk and propagating them to other peers.

A bug has been fixed causing us to send an FinalFailExpiryTooSoon error rather than a FinalFailIncorrectCltvExpiry when the last HTLC of a route has an expiration height that is deemed too soon by the final destination of the HTLC.

Aliases received on the wire are now properly validated. Additionally, we’ll no longer disconnect peers that send us invalid aliases.

A bug has been fixed that would at times cause commitments to desynchronize in the face of multiple concurrent updates that included an UpdateFee message. The fix generalizes the existing commitment state machine logic to treat an UpdateFee message as we would any other Update* messages.

We’ll now reject funding requests that require an unreasonable confirmation depth before the channel can be used.

We’ll now space out our broadcast batches more in order to save bandwidth and consolidate more updates behind a single batch.

We’ll now require all peers we connect to, to have the DLP (Data Loss Protection) bit set. This is required for the new SCB (Static Channel Backups) to function properly.

For private channels, we’ll now always resend the latest ChannelUpdate to the remote peer on reconnecting. This update is required to properly make invoices with hop hints which are required for receiving over a non-advertised channel.

Reject and Channel Caches

A number of internal caches have been added to reduce memory idle memory usage with a large number of peers, and also reduce idle CPU usage due to stale channel updates.

In this release, lnd now maintains a small reject cache for detecting stale ChannelAnnouncment and ChannelUpdate messages from its peers. Prior versions of lnd would perform a database lookup for each incoming messages, which produced a huge amount of contention under load and as the channel graph exploded.

The reject cache maintains just 25 bytes per edge, and easily holds today's graph in memory. Users on low power devices or with a large number of peers will benefit immensely from lnd's improved ability to filter gossip traffic for the latest information and clear large backlogs received from their peers.

The number of items in the cache is configurable using the --caches.reject-cache-size flag. The default value of 50,000 comfortably fits all known channels in the reject cache, requiring 1.2MB.

Additionally, we now maintain a separate channel cache, which contains in-memory copies of ChannelAnnouncements, ChannelUpdates, and NodeAnnouncements for a given channel. This cache is used to satisfy queries in hot paths of our peers’ gossip queries, allow us to serve more responses from memory and perform fewer database reads and allocations in deserialization.

The size of the channel cache is also configurable via the --caches.chan-cache-size flag. The default value of 20,000 stores about half of all known channels in memory and constitutes about 40MB.

Graceful Shutdown via SIGTERM

It was discovered that prior versions of lnd didn’t attempt to catch the SIGTERM signal to execute a graceful shutdown. When possible, users should prefer to shutdown lnd gracefully via either SIGTERM or SIGINT to ensure the database is closed and any outstanding transactions committed in order to avoid database corruption. Commonly used process management systems such as Docker or systemd typically send SIGTERM, then wait for a period of time to allow the process to respond before forcefully killing the process. Before this release, lnd would always be forcefully killed by these platforms, rendering it unable to properly execute a graceful shutdown.

This new release of lnd will now properly catch these signals to ensure that we’re more likely to be able to execute a graceful shutdown. We believe that many reports of partial database corruption typically reported by those running on Raspberry Pi’s should be addressed by this change.

Static Channel Backups

In this release, we’ve implemented a new safe scheme for static channel backups (SCB's) for lnd. We say safe, as care has been taken to ensure that there are no foot guns in this method of backing up channels, vs doing things like rsyncing or copying the channel.db file periodically. Those methods can be dangerous as one never knows if they have the latest state of a channel or not. Instead, we aim to provide a simple safe instead to allow users to recover the settled funds in their channels in the case of partial or complete data loss. The backups themselves are encrypted using a key derived from the user's seed, this way we protect the privacy of the users channels in the back up state, and ensure that a random node can't attempt to import another user's channels. WIth this backup file, given their seed and the latest back up file, the user will be able to recover both their on-chain funds, and also funds that are fully settled within their channels. By "fully settled" we mean funds that are in the base commitment outputs, and not HTLCs. We can only restore these funds as right after the channel is created, we have all the data required to make a backup.

We call these “static” backups, as they only need to be obtained once for a given channel and are valid until the channel has been closed. One can view this backup as a final method of recovery in the case of total data loss. It’s important to note that during recovery the channels must be closed in order to recover the funds fully. This set up ensures that there’s no way to incorrectly uses an SCB that would result in broadcast of a revoked commitment state. Recovery documentation for both on-chain and off-chain coins can be found here.

Backup + Recovery Methods

The SCB feature exposes multiple safe ways to backup and recover a channel. We expect only one of them to be used primarily by unsophisticated end users, but have provided other mechanisms for more advanced users and business that already script lnd via the gRPC system.

First, the easiest method for backup+recovery. lnd now will maintain a channels.backup file in the same location that we store all the other files. Users will at any time be able to safely copy and backup this file. Each time a channel is opened or closed, lnd will update this file with the latest channel state. Users can use scripts to detect changes to the file, and upload them to their backup location. Something like fsnotify can notify a script each time the file changes to be backed up once again. The file is encrypted using an AEAD scheme, so it can safely be stored plainly in cloud storage, your SD card, etc. The file uses a special format and can be used to import via any of the recovery methods described below.

The second mechanism is via the new SubscribeChanBackups steaming gRPC method. Each time an channel is opened or closed, you'll get a new notification with all the chanbackup.Single files (described below), and a single chanbackup.Multi that contains all the information for all channels.

Finally, users are able to request a backup of a single channel, or all the channels via the cli and RPC methods. Here's an example, of a few ways users can obtain backups:

⛰ lncli --network=simnet exportchanbackup --chan_point=29be6d259dc71ebdf0a3a0e83b240eda78f9023d8aeaae13c89250c7e59467d5:0
{
    "chan_point": "29be6d259dc71ebdf0a3a0e83b240eda78f9023d8aeaae13c89250c7e59467d5:0",
    "chan_backup": "02e7b423c8cf11038354732e9696caff9d5ac9720440f70a50ca2b9fcef5d873c8e64d53bdadfe208a86c96c7f31dc4eb370a02631bb02dce6611c435753a0c1f86c9f5b99006457f0dc7ee4a1c19e0d31a1036941d65717a50136c877d66ec80bb8f3e67cee8d9a5cb3f4081c3817cd830a8d0cf851c1f1e03fee35d790e42d98df5b24e07e6d9d9a46a16352e9b44ad412571c903a532017a5bc1ffe1369c123e1e17e1e4d52cc32329aa205d73d57f846389a6e446f612eeb2dcc346e4590f59a4c533f216ee44f09c1d2298b7d6c"
}

⛰ lncli --network=simnet exportchanbackup --all
{
    "chan_points": [
        "29be6d259dc71ebdf0a3a0e83b240eda78f9023d8aeaae13c89250c7e59467d5:0"
    ],
    "multi_chan_backup": "fd73e992e5133aa085c8e45548e0189c411c8cfe42e902b0ee2dec528a18fb472c3375447868ffced0d4812125e4361d667b7e6a18b2357643e09bbe7e9110c6b28d74f4f55e7c29e92419b52509e5c367cf2d977b670a2ff7560f5fe24021d246abe30542e6c6e3aa52f903453c3a2389af918249dbdb5f1199aaecf4931c0366592165b10bdd58eaf706d6df02a39d9323a0c65260ffcc84776f2705e4942d89e4dbefa11c693027002c35582d56e295dcf74d27e90873699657337696b32c05c8014911a7ec8eb03bdbe526fe658be8abdf50ab12c4fec9ddeefc489cf817721c8e541d28fbe71e32137b5ea066a9f4e19814deedeb360def90eff2965570aab5fedd0ebfcd783ce3289360953680ac084b2e988c9cbd0912da400861467d7bb5ad4b42a95c2d541653e805cbfc84da401baf096fba43300358421ae1b43fd25f3289c8c73489977592f75bc9f73781f41718a752ab325b70c8eb2011c5d979f6efc7a76e16492566e43d94dbd42698eb06ff8ad4fd3f2baabafded"
}

⛰ lncli --network=simnet exportchanbackup --all --output_file=channels.backup

⛰ ll channels.backup
-rw-r--r--  1 roasbeef  staff   381B Dec  9 18:16 channels.backup

SCBs can be viewed as a last ditch method for recovering funds from channels due to total data loss. In future releases, we plan to implement methods that require more sophistication with respect to operational architecture, yet allow for dynamic backups. Even with these dynamic backups in place, SCBs will still serve as a fallback method if a dynamic back up may be known to be out of date, or in a partial state of consistency.

Future protocol changes will make the SCB recovery method more robust, as it will no longer rely on the remote peer to send the normal channel reestablishment handshake upon reconnection. Instead, given the SCB, lnd will be able to find the closing output directly on the chain after a force close by the remote party.

For further details w.r.t the lower level implementation of SCBs as well as the new RPC calls, users can check out the new recovery.md file which goes over methods to recover both on-chain and off-chain funds from lnd.

New Channel Status Manager

Within the protocol, nodes can mark a channel as enabled or disabled. A dsiable channel signals to other nodes that the channel isn’t to be used for routing for whatever reason. This allows clients to void these channels during path finding, and also lets routing nodes signal any faults in a channel to other nodes allowing them to ignore them and possibly remove them from their graph view. lnd has a system to automatically detect when a channel has been inactive for too long, and disable it, signalling to other peers that they can ignore it when routing. The system will also eventually re-enable a channel if it has been stable for long enough.

The prior version of sub-system had a number of flaws which would cause channels to be excessively enabled/disabled, causing ChannelUpdate spam in the network. In this release, this system has been revamped, resulting in a much more conservative, stable channel status manager. We’ll now only disable channels programmatically, and channels will only be re-enabled once the peer is stable for a long enough period of time. This period of time is now configurable.

Server and P2P Improvements

The max reconnection back off interval is now configurable. We cap this value by default to ensure we don’t wait an eternity before attempting to reconnect to a peer. However, on laptops and mobile platforms, users may want to value to be much lower to ensure they maintain connectivity in the face of roaming, or wi-ifi drops. The new field is: --maxbackoff=. A new complementary --minbackoff field has also been added.

We’ll now attempt to retry when faced with a write timeout rather than disconnect the peer immediately. This serves to generally make peer connections more stable to/from lnd.

Users operating larger lnd nodes may find that at times restarts can be rather load heavy due to the rapid burst of potentially hundreds of new p2p connections. In this new version of lnd, we’ve added a new flag (--stagger-initial-reconnect) to space out these connection attempts by several seconds, rather than trying to establish all the connections at once on start up.

Performance Enhancements

Outgoing Message Queue Prioritization

A new distinct queue of gossip messages has been added to the outgoing write queue system within lnd. We’ll now maintain two distinct queues: messages for gossip message, and everything else. Upon reconnection, certain messages are time sensitive such as sending the Channel Reestablishment message which causes a channel to shift from active to inactive. This queue optimization also means that making new channels, or updating existing channels will no longer be blocked by any outgoing gossip traffic, improving the quality of service.

Batched Pre-Image Writing in the HTLCSwitch

This new release will now batch writes for witnesses discovered in HTLC forwarding. At the same time, we correct a nuanced consistency issue related to a lack of synchronization with the channel state machine. Naively, forcing the individual preimage writes to be synchronized with the link incurs a heavy performance penalty (about 80% in profiling). Batching these allows us to minimize the number of db transactions required to write the preimages, allowing us to reinsert the batched write into the link's critical path and resolve the possible inconsistency. In fact, the benchmarks actually showed a slight performance improvement, even with the extra write in the critical path.

Unified Global SigPool

lnd uses a pool of goroutines that are tasked with signing and validating commitment and HTLC signatures for new channel updates. This pool allows us to process these commitment updates in parallel, rather than in a serial manner which would reduce payment throughput. [Rather than using a single SigPool per channel, we now use a single global SigPool](#2329_. With this change, we ensure that as the number of channels grows, the number of goroutines idling in the sigPool stays constant. It's the case that currently in the daemon, most channels are likely inactive, with only a handful actually consistently carrying out channel updates. As a result, this change should reduce the amount of idle CPU usage, as we have less active goroutines in select loops.

Read and Write Buffer Pools

In this release, we implement a write buffer pool for LN peers. Previously, each peer object would embed a 65KB byte array, which is used to serialize messages before writing them to the wire. As a result, every new peer causes a large memory allocation, which places unnecessary burden on the garbage collector when faced with short-lived or flapping peers. We’ll now use a buffer pool, that dynamically grows and shrinks based on the demand for write buffers corresponding to active peers. This greatly helps when there is a high level of churn in peer activity, or even if there is a single one flapping peer.

Similarly, whenever a new peer would connect, we would allocate a 65KB+16 byte array to use as a read buffer for each connection object. The read buffer stores the ciphertext and MAC read from the wire, and used to decrypt and then decode messages from the peer. Because the read buffer is implemented at the connection-level, as opposed to the peer-level like write buffers, simply opening a TCP connection would cause this allocation. Therefore peers that send no messages, or do not complete the handshake, will add to this memory overhead even if they are released promptly. To avoid this, we now use a similar read buffer pool to tend towards a steady working set of read buffers which drastically reduces memory usage.

Finally, we introduce a set of read/write worker pools, which are responsible for scheduling access to the read/write buffers in the underlying buffer pools. With the read and write pools, we modify the memory requirements to be at most linear in the number of specified workers. More importantly, these changes completely decouple read and write buffer allocations from the peer/connection lifecycle, allowing lnd to tolerate flapping peers with minimal overhead.

Nodes that have a large number of peers will see the most drastic benefit. In testing, we were able to create stable connections (w/o gossip queries) to over 900 unique nodes, all while keeping lnd's total memory allocations due to read/write buffers under 15 MB. This configuration could have easily connected to more nodes, though that was all that reachable via the bootstrapper.
This same test would have used between 90-100MB on master, and continues to grow as more connections are established or peers flap if the garbage collector could not keep up. In contrast, the memory used with read/write pools remains constant even as more peers are established.

Sweeper

A new sweeper subsystem has been introduced. The sweeper is responsible for sweeping mature on-chain outputs back to the wallet. It does so by combining sets of outputs in a single transaction per block. It takes care not to sweep outputs that have a negative yield at the current fee estimate. Those will be left until the fee estimate has decreased enough. Some outputs may still be contested and possibly swept by the remote party. The sweeper is aware of this and properly reports the outcome of the sweep for an output to other subsystems. sweep: create sweeper.

The new Sweeper sub-system is the start of a generalized transaction batching engine within lnd. As is today, it will batch all sweeps (HTLC timeouts, commitment sweeps, CSV sweeps) across lnd into a single transaction per block. In the future, the sweeper will be generalized in order to implement fee bumping techniques like RBF and CPFP in a single logical unit. Additionally, the existence of such a batching engine will allow us to batch all transaction daemon wide into a single transaction, which will allow us to implement block saving features such as: opening multiple channels in a single transaction, combining cross channel splice in/outs, closing out one channel in order to open a new channel or fulfill a request payment. payment.

Overtime the sweeper will also grow to obsolete the existing UtxoNursery as sweep requests will become more distributed (an HTLC asks to be swept rather than the nursery sweeping when the time is right).

Graph Sync Improvements

With the recent rapid growth of the network, it almost became unbearable for nodes to sync their routing table with their peers due to the huge number of updates/channels being announced. We’ve made significant improvements towards addressing this issue with the introduction of the SyncManager. Nodes will now only receive new graph updates from 3 peers by default. This number has been exposed as a CLI flag, —numgraphsyncpeers, and can be tuned for light clients and routing nodes for bandwidth savings. In testing, we’ve seen over a 95% bandwidth reduction as a result of these changes.

This version also reduces the batch size of channels requested via QueryShortChanIDs from 8000 to 500, leading to more stability in large or initial syncs. The previous version was found to invite disconnections from the remote peer once the receiver had received the first few thousand messages. The reduced batch size prevents us from overflowing our own internal queues for gossip messages, and ensuring the remote peer doesn’t interpret this as jammed connection.

Goodbye Zombie Channels

Within the last couple months, we started to experience a large number of zombie channels in the network being gossiped between nodes. A zombie channel is a channel that is still open, but hasn’t been updated for 2 weeks. This issue was also present on testnet a few years back, so we’ve finally addressed the issue for
good. Nodes will now maintain an index of zombie channels which they can query to determine whether they should process/forward announcements for an arbitrary channel.

Using this index, we will also refrain from requesting channels we know to be zombies from peers that think otherwise. At the time of writing, there are roughly 3.3k zombie channels on mainnet. This optimization saves us from requesting 10k individual messages, amounting to roughly 3MB when attempting historical syncs with peers presenting lnd with zombie channels.

On-Chain Commitment and HTLC Handling

A bug has been fixed that would previously cause an HTLC which was settled on chain to not properly be marked as settled..

An off-by-one-error has been fixed in the contract court when handling a remote commitment close due to a DLP execution instance. This ensures that funds will now properly be swept in the case of a force close due to DLP wherein the remote party is one state ahead of ours. Users that ran into this issue in the wild should find that the dispatch logic is re-executed, resulting in on-chain funds properly being swept back into the wallet.

Bitcoind Spend Hint Bug Fix

Fixes a bug that would cause bitcoind backends to perform historical rescans on successive restarts, even if the first rescan completed and did not find a spending transaction. Affected nodes will have to complete one more rescan after upgrading before symptoms will disappear. In more severe cases, this will save tens of thousands of getblocks calls to the backend on each restart.

Autopilot Architecture Revamp

In this release, as a prep for more advanced autopilot heuristics in a future release, we’ve completely revamped the way the system works. Before this release, the autopilot “agent” was directive based, meaning that it when queried, it would simply say “connect to these nodes”. This detective based suggestion was simple, yet limiting in that: it didn’t easily lend to combining multiple heuristics and using only the dertive model, there isn’t a clear way of comparing to distinct heuristics.

The new system instead implements a scoring based agent. Rather than simply suggesting a set of node to connect to, the agent will now return a set of scores for a target, or all peers within the network. This score is then incorporated into the main channel selection loop, adding a bit of jitter to ensure diversity. The scoring based system really shines when you start to consider adding multiple heuristics that work in tandem (connectivity optimized, uptime optimized, reliability optimized, redundancy optimized, etc). With the new scoring system, it’s now possible to create a new heuristics which is actually a combination of several sub heuristics. As an example, we’ve created a new WeightedCombAttachment heuristics that outputs a linear combination of the scores of a set of registered hueirticis.

This new scoring based system will pave the road for more advanced autopilot heuristics which may make certain trade offs in order to target specific use cases like: mobile/laptop oriented, net receiver (merchant, etc) optimized, routing network robustness (min-cut and the like). As a bonus, the new system also makes it much easier to add a new heuristic as the new interface has a single method: NodeScores: given the graph, target channel size, and existing set of node channel it should return a score for all non-filtered out nodes.

Penalize small channels for autopilot scoring

With rearchitechting autopilot to be scoring based, the default heuristic (prefattach) will now decrease the score of nodes having a large number of small channels.

New Sweep All Coins Command

A new argument has been added to the lncli sendcoins interface to allow users to sweep all coins from lnd's wallet to a target on-chain address. An example run of the new command looks something like:

⛰   lncli --network=simnet sendcoins --sweepall --addr=sb1qsy8772pkfucsvmuyw82gexyd4u69pvve9w98v3
{
    "txid": "1931f6653ecb2add24e00ce03d6a66ce705ebf633bfffb2b67674000d5f3d5d4"
}

For those using the RPC interface, the new field to set is send_all, which is a boolean that indicates the amount is left blank, and all coins should be sent to that target address.

RPC Interface Enhancements and Fixes

The default number of routes returned from QueryRoutes is now 10 (prior the default was unspecified).

QueryRoutes with used with more than one route target has been deprecated and will be phased out in future versions of lnd. In order to make up for the lost functionality, we’ve added a series of new arguments to the RPC call that allow users to ignore an arbitrary set of edges or vertexes. This new feature makes it easier to implement things like rebalancing externally, as you can know modify the source node for path finding, which can be used to find a path from node A to B, then back from B to A that must travel in/out of a specific edge set.

VerifyMessage and SignMessage now properly expose a REST endpoint.

The response to SendPayment now also includes the payment hash in order to make it easy to associate a success or failure amongst several payments when using the streaming RPC.

A number of fixes to the request validation within the RPC server have been made. These changes make the server more defensive w.r.t what it accepts from clients.

Invoices created with the --private option (include hop hints to private channels) are [now marked as such on the RPC interface}(https://github.com//pull/2222).

A new RPC call ListUnspent has been added to allow users to examine the current UTXO state of lnd. Combined with the new Signer Sub-Server, users can use this to craft arbitrary transactions using lnd’s backing keystore.

A bug has been fixed that would cause channels that are unconfirmed, but waiting to be closed from being returned via the PendingChannels RPC.

A bug has been fixed that wouldn’t allow users to expose the REST interface on all network interfaces.

The existing settled field from the Invoice proto is now deprecated. Instead, the new state field is to be used as it allows us to reflect additional states of invoices (open, settled, cancelled, etc).

A number of fields in the Invoice proto that have never been populated /used have been removed.

The name of the network is now exposed in the response of the GetInfo RPC.

The UnsettledBalance field in the PendingChannels RPC response is now properly set.

The ListChannels response now includes a field which denotes if the node is the iniatitor of a channel or not.

HTLCs which haven’t yet timed out, are now properly shown in the output of PendingChannels.

A new SubscribeChannels RPC has been added to allow clients to be notified whenever a channel becomes inactive, active, or closed. This is useful for any type of application that would otherwise need to poll the channel state to keep up to date on what channels are active, inactive, or closed.

Two new address types have been added to the NewAddress RPC call. These address types will return the same address until they have been used, then rotate to a new address. These new address types are useful for displaying a new address in UIs without running into “address inflation”.

The getnetworkinfo RPC now also returns the median channel size of the graph. The average degree output in GetNetworkInfo has also been corrected.

A bug that would cause the autopilot agent to over-allocate funds if multiple channels were opened in parallel has been fixed.

We’ll now retrieve the chan_id of an open channel from the channel database when using the ListChannels RPC/listchannels lncli command, rather than the graph. If a channel doesn’t have a new update within the last 2 weeks, then it’ll be pruned from the graph, which caused the chan_id lookup to fail and result in a 0 value being displayed.

Sub-Servers

In this new release, we’ve begun the process of slowly evolving the RPC interface via the new Sub-Server system. The gRPC system allows multiple independent services to be registered to the same endpoint. Before this release, lnd had one primary service: Lightning. All current RPC calls are directed to this unified service. In the early days of lnd, this structure emerged organically as many RPCs were added based on speculative future uses, or primarily for the purposes of testing new features added to the codebase. The result today is one mega interface, without any clear specialization or feature delineation.

Since the initial release of lnd, we’ve received a considerable amount of valuable feedback w..r the RPC interface from developers, businesses, and node operators that use the interface daily. Some of this feedback may require us to extensively re-work core RPC calls like SendPayment. Doing so directly in the main service would be disruptive as the calls may change over night, or have their behavior be drastically modified. We consider Sub-Servers to be a solution to this issue as they allow us to recreate a small subset of the existing RPC interface in a concentrated, methodical manner. By being able to start from scratch, we gain more freedom w.r.t crafting the new interface. Additionally, by being forced to examine a smaller subset of the total functionality in a new Sub-Server, we’re able to consolidate existing code, decouple the RPC interface from the rest of lnd, and also expose new functionality to the RPC interface that may only be tangentially related to Lightning.

As of this release, all sub-servers are guarded behind special build flags (make install -tags=<buildtag>). The rationale here is that the sub-servers only expose new functionality, so existing users of lnd that don’t yet have a need for these new features shouldn’t be burdened with them at runtime. Over time as the interfaces crystalize more, we’ll begin the process of depreciating certain older RPCs in order to promote the newer more design sound Sub-Server RPCs. As a result, the current Sub-Server interfaces should be considered non final and subject to change at anytime. Due to their volatile nature, we don’t yet have documentation up at api.lightning.community. On their application development side of things, using a new sub-server is as simple as creating a new gRPC client service with the existing gRPC client connection.

Sub-Servers also make lnd generally more useful as a one-stop shop for any sort of Bitcoin related programming or application as they expose some core interfaces that lnd uses across the codebase to accomplish routine tasks. When compiled in, certain sub-servers will also augment lncli with a set of new commands. The current set of Sub-Servers (and their respective build tags) include:

  • ChainNotifier (chainrpc)
  • WalletKit (walletrpc)
  • Signer (signrpc)
  • Invoices (invoicesrpc)
  • Autopilot (autopilotrpc)
  • Router (routerrpc)

The release binaries for lnd 0.6 are compiled using the signrpc, walletrpc, chainrpc, and invoicesrpc build tags, making them compatible with Lightning Loop services.

ChainNotifier Sub-Server

The ChainNotifier Sub-Server is a utility toolkit responsible for requesting the chain backend for notifications about the tip of the chain, transaction confirmations, and output spends. It also includes support for requesting these notifications for arbitrary output scripts as well.

WalletKit Sub-Server

The WalletKit Sub-Server is a utility toolkit that contains method which allow clients to perform common interactions with a wallet such as getting a new address, or sending a transaction. It also includes some supplementary actions such as fee estimation. Combined with the Signer Sub-Server, this lets users create arbitrary transactions (like CoinJoins!) using the existing set of private keys under the control of lnd.

Signer Sub-Server

The Signer Sub-Server exposes the existing input.Signer interface within lnd as an accessible sub-server. The existence of this sub-servers also opens up the possibility of having the actual signer and signing code existing outside of lnd taking the form of either a distinct process, or remote server with additional access control mechanisms.

Invoices Sub-Server

The Invoices Sub-Sever exposes a number of new ways to interact with invoices that don’t exist in the existing invoice related calls for the main service. A new type of invoice called ‘hodl invoice’ has been added. Instead of immediately locking in and settling the htlc when the payment arrives, the htlc for a hodl invoice is only locked in and not yet settled. At that point, it is not possible anymore for the sender to revoke the payment, but the receiver still can choose whether to settle or cancel the htlc and invoice htlcswitch: hodl invoice.

The new invoice function CancelInvoice has been implemented. CancelInvoice can be called on a hodl invoice, but also on a regular invoice. It makes the invoice unpayable invoices: CancelInvoice.

A last improvement to the invoices subsystem is the ability to subscribe to updates of a single invoice instead of receiving all invoice updates invoices: add subscribesingleinvoice.

Autopilot Sub-Server

The Autopilot Sub-Server allows users to programatically drive certain aspects of the autopilot system. Before this new Sub-Server, the only way to modify the settings of autopilot were to modify command line parameters and restart the system. This new Sub-Server allows users to turn autopilot on/off without restarting, and also query the score of a prospective node with the new query interface.

Router Sub-Server

The Router Sub-server presents a simplified interface for sending payments off-chain, and also getting a fee estimate for potential off-chain payments. In future releases, we’ll begin to revamp the off-chain sending interface in order to give users more control w.r.t when we start/stop attempting to fulfill a payment attempt, and also more transparency w.r.t the state of an initiated off-chain payment.

Tor

lnd will now allow a user to access the Tor daemon with NULL authentication. Additionally, it’s now possible to listen on a distinct interface that isn’t localhost when running in auto hidden service mode. This allows users that are aware of the implications to run in a hybrid mode that accepts both inbound clearnet and hidden service connections. Additionally, users can now listen on an arbitrary interface if they have outbound Tor configured.

Neutrino Enhancements

We’ll now start syncing headers and filter headers as soon as lnd’s wallet is created/unlocked. This greatly improves the user experience by reducing the amount of time to reach a fully synced light client lnd instance.

We’ll reliably broadcast transactions to our bitcoin peers to ensure they propagate throughout the network.

Library Enhancements and Multi-Module Support

This release begins some efforts towards transitioning into a multi-module repository. This allows specific packages within lnd to be used externally without the need of duplicating code or running into import cycles.

The ReadElements and WriteElements methods from lnwire are now exposed publicly. This allows any Go program to easily be able to serialize structs/data using the codec described in the BOLT documents.

Wallet Bug Fixes and Improvements

The wallet will no longer rescan from its birthday if it has no UTXOs.

The wallet will now properly remove transactions from its persistent store that the chain backend deems as
invalid.

The wallet will now properly remove transaction conflicts from its persistent store.

The default recovery window has been increased from 250 to 2500. This increase is meant to ensure that the typical wallet is able to complete the regular seed rescan/import without needing to increase the existing default look ahead value.

Routing

Two new optional restrictions have been added that influence route optimization for sending payments:

Furthermore, several new parameters have been added to the QueryRoutes rpc call to allow more control over the returned route lnrpc: deprecate QueryRoutes with more than one route. Requesting multiple routes from QueryRoutes based on the k-shortest algorithm has been deprecated. This behaviour can be re-implemented client side using the new QueryRoutes parameters.

In the SendToRoute rpc call, the ability to specify multiple routes has been deprecated lnrpc: deprecate SendToRoute with more than one route.

The default CLTV delta for channels created by lnd has been lowered from 144 blocks to 40 blocks. Future versions of lnd will begin to automatically modify this parameter based on the sampled fee levels in the chain.

Breacharbiter Preparatory Work

A number of enhancements to the Breacharbiter have been made which are required for the ultimate watch tower implementation. These changes ensure that lnd is able to continue to function if it isn’t the one that ends up sweeping all the outputs in the case of a breach (the tower might sweep the commitment outputs for example, and lnd sweeps the HTLCs itself).

Accounting

Improvements have been made to the PendingChannels report. Several categories of funds in limbo that were previously unreported have been added to the report rpc+contractcourt: merge the contractResolver state into the pendingchannels RPC response.

Changelog

The full list of changes since 0.5.2-beta can be found here:

Contributors (Alphabetical Order)

Adam Gibson
Alex Akselrod
Alex Bosworth
Brian Sipple
Carla Kirk-Cohen
Chris Coverdale
Conner Fromknecht
chokoboko
Cryptcoin Junkey
cryptagoras
Erik Ek
Eugene Siegel
Federico Bond
Francisco Calderón
Gfloresechaiz
Igor Cota
Jim Posen
Johan T. Halseth
John Ng
John Griffith
Jonathan Cross
Joost Jager
Matt Drollette
Max Kaplan
Moshe Shababo
Offer Markovich
Olaoluwa Osuntokun
Ondrej Calda
orbitalturtle
Otto Suess
Philipp Gillé
Roei Erez
Ron Cohen
Sanket Kanjalkar
Sevastos
solid-pay
Thomas Braunberger
Tom Kirkpatrick
Valentine Wallace
Vincent Woo
Wilmer Paulino
Xavi Soler
Yancy Ribbens

Assets 26
Pre-release
Pre-release

@Roasbeef Roasbeef released this Apr 12, 2019 · 702 commits to master since this release

This release marks a new major release of lnd that includes several important bug fixes, numerous performance optimizations, static channel backups (SCB), reduced bandwidth usage for larger nodes, an overhaul of the internals of the autopilot system, and a new batch sweeping sub-system. Due to the nature of some of the bug fixes which were made during the implementation of the new SCB feature, users are highly encouraged to upgrade to this new version.

Database migrations

This version includes a single migration to modify the message store format, used to send messages to remote peers reliably when attempting to construct channel proofs. The migration should appear as below:

2019-04-03 22:35:44.596 [INF] LTND: Version: 0.6.0-beta commit=v0.6-beta-rc4, build=production, logging=default
2019-04-03 22:35:44.596 [INF] LTND: Active chain: Bitcoin (network=mainnet)
2019-04-03 22:35:44.597 [INF] CHDB: Checking for schema update: latest_version=8, db_version=7
2019-04-03 22:35:44.597 [INF] CHDB: Performing database schema migration
2019-04-03 22:35:44.597 [INF] CHDB: Applying migration #8
2019-04-03 22:35:44.597 [INF] CHDB: Migrating to the gossip message store new key format
2019-04-03 22:35:44.597 [INF] CHDB: Migration to the gossip message store new key format complete!

Verifying the Release

In order to verify the release, you'll need to have gpg or gpg2 installed on your system. Once you've obtained a copy (and hopefully verified that as well), you'll first need to import the keys that have signed this release if you haven't done so already:

curl https://keybase.io/roasbeef/pgp_keys.asc | gpg --import

Once you have his PGP key you can verify the release (assuming manifest-v0.6-beta-rc4.txt and manifest-v0.6-beta-rc4.txt.sig are in the current directory) with:

gpg --verify manifest-v0.6-beta-rc4.txt.sig

You should see the following if the verification was successful:

gpg: assuming signed data in 'manifest-v0.6-beta-rc4.txt'
gpg: Signature made Thu Apr 11 16:36:48 2019 PDT
gpg:                using RSA key F8037E70C12C7A263C032508CE58F7F8E20FD9A2
gpg: Good signature from "Olaoluwa Osuntokun <laolu32@gmail.com>" [ultimate]

That will verify the signature on the main manifest page which ensures integrity and authenticity of the binaries you've downloaded locally. Next, depending on your operating system you should then re-calculate the sha256 sum of the binary, and compare that with the following hashes (which are included in the manifest file):

b5fd7f1f0fb2d15ea7733093891fec0b0da2a8559b3f8af35d9b90e7e9f1f88d  lnd-darwin-386-v0.6-beta-rc4.tar.gz
21bf54da68ff0274c9533c42960ede68bfbdd1c0d6731f98a4840581c373ef82  lnd-darwin-amd64-v0.6-beta-rc4.tar.gz
a91aba806048be5cb0edc832592628770c4a9551b1b0fa5929379e0e84697f13  lnd-dragonfly-amd64-v0.6-beta-rc4.tar.gz
b7e6061a80280fe410b0b89936c4aae1bb710ad5d2c12e753458d15e03315650  lnd-freebsd-386-v0.6-beta-rc4.tar.gz
0257b8a370966e8b4a811025bf8f7ef4db0bd41b3b5b261b7db32c8374d50bcd  lnd-freebsd-amd64-v0.6-beta-rc4.tar.gz
7db9bb741d6ff2f7eeb8971157145a4fcee448fb32410eff62bbc7a74eb5ab27  lnd-freebsd-arm-v0.6-beta-rc4.tar.gz
3af9247ac71da98db8690dcf1f2b7c3e041a05fe557c00dd25116e114851224b  lnd-linux-386-v0.6-beta-rc4.tar.gz
e03680698dc980885f2aa794953b8abffb0b672e2baf042d6deed62ecd8bfd26  lnd-linux-amd64-v0.6-beta-rc4.tar.gz
43c86a8fd50dc54d942bc85883202396da063780c249769f7b97b2159e8d5630  lnd-linux-arm64-v0.6-beta-rc4.tar.gz
92b61e47c6fa741b1f3ff7507e1fe8c540d87ece272b5b3b1d7104228ce15c7a  lnd-linux-armv6-v0.6-beta-rc4.tar.gz
3ed4b5e54afb6bf083a9693058dbf7d490e15837da5e5dc49ba06bfb942a1312  lnd-linux-armv7-v0.6-beta-rc4.tar.gz
41c96d503eb7f166b95963a3d0c25e39ad43e4719ce1b61f07481b96e4adf6c5  lnd-linux-mips64-v0.6-beta-rc4.tar.gz
fe3a4fd2ad36c1cb5aa2bbc6dcaca67cbe22fa3bc40d2494c55a6713dd1ccc31  lnd-linux-mips64le-v0.6-beta-rc4.tar.gz
f910d790956dd6db56b0fb4f08e88eff3453cc4eccac974f9cbb5dccdde794ec  lnd-linux-ppc64-v0.6-beta-rc4.tar.gz
b9591ee4d92811d3e5a6a1c7162788acb53000087db535ec53b945c2204cc42c  lnd-netbsd-386-v0.6-beta-rc4.tar.gz
4f66044356d144face2687c80a09b63860b949f874a1e80d6abdab1106bdbbf3  lnd-netbsd-amd64-v0.6-beta-rc4.tar.gz
2b9055ddd3686ebe3eb266a95fa6231319b9c29cf130b8ab8513c3af48e5158f  lnd-openbsd-386-v0.6-beta-rc4.tar.gz
8f900a1fab45aaabbee9ff50ce2140ae8f36f14e84ad4a2d56d5c50ebfcaba84  lnd-openbsd-amd64-v0.6-beta-rc4.tar.gz
0befbe91617e837287f391f08f374a8ac0b88b0770c5b1a79eaa4987fe334fe0  lnd-source-v0.6-beta-rc4.tar.gz
4ec02a37c7fd923f7f0114f889a1b6d042617e9dc870d1f42a31ef0caff1ae84  lnd-windows-386-v0.6-beta-rc4.zip
22545a7c5db140e4e42ea658befd38ea8d422d909e7811196e18308870ec5aae  lnd-windows-amd64-v0.6-beta-rc4.zip
9880b8643c3fc1cfce755cec8329e6a3372664e91bce5ae2cf2dfe7aba325d2d  vendor.tar.gz

One can use the shasum -a 256 <file name here> tool in order to re-compute the sha256 hash of the target binary for your operating system. The produced hash should be compared with the hashes listed above and they should match exactly.

Finally, you can also verify the tag itself with the following command:

git verify-tag v0.6-beta-rc4

Building the Contained Release

With this new version of lnd, we've modified our release process to ensure the bundled release is now fully self contained. As a result, with only the attached payload with this release, users will be able to rebuild the target release themselves without having to fetch any of the dependencies. Note that at this stage, binaries aren't yet fully reproducible (even with go modules). This is due to the fact that by default, Go will include the full directory path where the binary was built in the binary itself. As a result, unless your file system exactly mirrors the machine used to build the binary, you'll get a different binary, as it includes artifacts from your local file system. This will be fixed in go1.13, and before then we may modify our release system to do this automatically.

In order to re-build from scratch, assuming that vendor.tar.gz and lnd-source-v0.6-beta-rc4.tar.gz are in the current directory:

tar -xvzf vendor.tar.gz
tar -xvzf lnd-source-v0.6-beta-rc4.tar.gz
GO111MODULE=on go install -v -mod=vendor -ldflags "-X github.com/lightningnetwork/lnd/build.Commit=v0.6-beta"
GO111MODULE=on go install -v -mod=vendor -ldflags "-X github.com/lightningnetwork/lnd/build.Commit=v0.6-beta" ./cmd/lncli

The -mod=vendor flag tells the go build command that it doesn't need to fetch the dependencies, and instead, they're all enclosed in the local vendor directory.

Additionally, it's now possible to use the enclosed release.sh script to bundle a release for a specific system like so:

LNDBUILDSYS="linux-arm64 darwin-amd64" ./release.sh

The release.sh script will now also properly include the commit hash once again, as a regression caused by a change to the internal build system has been fixed.

⚡️⚡️⚡️ OK, now to the rest of the release notes! ⚡️⚡️⚡️

Release Notes

Protocol and Cross-Implementation Compatibility Fixes

We’ll now properly validate our own announcement signatures for NodeAnnouncements before writing them to disk and propagating them to other peers.

A bug has been fixed causing us to send an FinalFailExpiryTooSoon error rather than a FinalFailIncorrectCltvExpiry when the last HTLC of a route has an expiration height that is deemed too soon by the final destination of the HTLC.

Aliases received on the wire are now properly validated. Additionally, we’ll no longer disconnect peers that send us invalid aliases.

A bug has been fixed that would at times cause commitments to desynchronize in the face of multiple concurrent updates that included an UpdateFee message. The fix generalizes the existing commitment state machine logic to treat an UpdateFee message as we would any other Update* messages.

We’ll now reject funding requests that require an unreasonable confirmation depth before the channel can be used.

We’ll now space out our broadcast batches more in order to save bandwidth and consolidate more updates behind a single batch.

We’ll now require all peers we connect to, to have the DLP (Data Loss Protection) bit set. This is required for the new SCB (Static Channel Backups) to function properly.

For private channels, we’ll now always resend the latest ChannelUpdate to the remote peer on reconnecting. This update is required to properly make invoices with hop hints which are required for receiving over a non-advertised channel.

Reject and Channel Caches

A number of internal caches have been added to reduce memory idle memory usage with a large number of peers, and also reduce idle CPU usage due to stale channel updates.

In this release, lnd now maintains a small reject cache for detecting stale ChannelAnnouncment and ChannelUpdate messages from its peers. Prior versions of lnd would perform a database lookup for each incoming messages, which produced a huge amount of contention under load and as the channel graph exploded.

The reject cache maintains just 17 bytes per edge, and easily holds today's graph in memory. Users on low power devices or with a large number of peers will benefit immensely from lnd's improved ability to filter gossip traffic for the latest information and clear large backlogs received from their peers.

The number of items in the cache is configurable using the --caches.reject-cache-size flag. The default value of 50,000 comfortably fits all known channels in the reject cache, requiring 1.2MB.

Additionally, we now maintain a separate channel cache, which contains in-memory copies of ChannelAnnouncements, ChannelUpdates, and NodeAnnouncements for a given channel. This cache is used to satisfy queries in hot paths of our peers’ gossip queries, allow us to serve more responses from memory and perform fewer database reads and allocations in deserialization.

The size of the channel cache is also configurable via the --caches.chan-cache-size flag. The default value of 20,000 stores about half of all known channels in memory and constitutes about 40MB.

Graceful Shutdown via SIGTERM

It was discovered that prior versions of lnd didn’t attempt to catch the SIGTERM signal to execute a graceful shutdown. When possible, users should prefer to shutdown lnd gracefully via either SIGTERM or SIGINT to ensure the database is closed and any outstanding transactions committed in order to avoid database corruption. Commonly used process management systems such as Docker or systemd typically send SIGTERM, then wait for a period of time to allow the process to respond before forcefully killing the process. Before this release, lnd would always be forcefully killed by these platforms, rendering it unable to properly execute a graceful shutdown.

This new release of lnd will now properly catch these signals to ensure that we’re more likely to be able to execute a graceful shutdown. We believe that many reports of partial database corruption typically reported by those running on Raspberry Pi’s should be addressed by this change.

Static Channel Backups

In this release, we’ve implemented a new safe scheme for static channel backups (SCB's) for lnd. We say safe, as care has been taken to ensure that there are no foot guns in this method of backing up channels, vs doing things like rsyncing or copying the channel.db file periodically. Those methods can be dangerous as one never knows if they have the latest state of a channel or not. Instead, we aim to provide a simple safe instead to allow users to recover the settled funds in their channels in the case of partial or complete data loss. The backups themselves are encrypted using a key derived from the user's seed, this way we protect the privacy of the users channels in the back up state, and ensure that a random node can't attempt to import another user's channels. WIth this backup file, given their seed and the latest back up file, the user will be able to recover both their on-chain funds, and also funds that are fully settled within their channels. By "fully settled" we mean funds that are in the base commitment outputs, and not HTLCs. We can only restore these funds as right after the channel is created, we have all the data required to make a backup.

We call these “static” backups, as they only need to be obtained once for a given channel and are valid until the channel has been closed. One can view this backup as a final method of recovery in the case of total data loss. It’s important to note that during recovery the channels must be closed in order to recover the funds fully. This set up ensures that there’s no way to incorrectly uses an SCB that would result in broadcast of a revoked commitment state. Recovery documentation for both on-chain and off-chain coins can be found here.

Backup + Recovery Methods

The SCB feature exposes multiple safe ways to backup and recover a channel. We expect only one of them to be used primarily by unsophisticated end users, but have provided other mechanisms for more advanced users and business that already script lnd via the gRPC system.

First, the easiest method for backup+recovery. lnd now will maintain a channels.backup file in the same location that we store all the other files. Users will at any time be able to safely copy and backup this file. Each time a channel is opened or closed, lnd will update this file with the latest channel state. Users can use scripts to detect changes to the file, and upload them to their backup location. Something like fsnotify can notify a script each time the file changes to be backed up once again. The file is encrypted using an AEAD scheme, so it can safely be stored plainly in cloud storage, your SD card, etc. The file uses a special format and can be used to import via any of the recovery methods described below.

The second mechanism is via the new SubscribeChanBackups steaming gRPC method. Each time an channel is opened or closed, you'll get a new notification with all the chanbackup.Single files (described below), and a single chanbackup.Multi that contains all the information for all channels.

Finally, users are able to request a backup of a single channel, or all the channels via the cli and RPC methods. Here's an example, of a few ways users can obtain backups:

⛰ lncli --network=simnet exportchanbackup --chan_point=29be6d259dc71ebdf0a3a0e83b240eda78f9023d8aeaae13c89250c7e59467d5:0
{
    "chan_point": "29be6d259dc71ebdf0a3a0e83b240eda78f9023d8aeaae13c89250c7e59467d5:0",
    "chan_backup": "02e7b423c8cf11038354732e9696caff9d5ac9720440f70a50ca2b9fcef5d873c8e64d53bdadfe208a86c96c7f31dc4eb370a02631bb02dce6611c435753a0c1f86c9f5b99006457f0dc7ee4a1c19e0d31a1036941d65717a50136c877d66ec80bb8f3e67cee8d9a5cb3f4081c3817cd830a8d0cf851c1f1e03fee35d790e42d98df5b24e07e6d9d9a46a16352e9b44ad412571c903a532017a5bc1ffe1369c123e1e17e1e4d52cc32329aa205d73d57f846389a6e446f612eeb2dcc346e4590f59a4c533f216ee44f09c1d2298b7d6c"
}

⛰ lncli --network=simnet exportchanbackup --all
{
    "chan_points": [
        "29be6d259dc71ebdf0a3a0e83b240eda78f9023d8aeaae13c89250c7e59467d5:0"
    ],
    "multi_chan_backup": "fd73e992e5133aa085c8e45548e0189c411c8cfe42e902b0ee2dec528a18fb472c3375447868ffced0d4812125e4361d667b7e6a18b2357643e09bbe7e9110c6b28d74f4f55e7c29e92419b52509e5c367cf2d977b670a2ff7560f5fe24021d246abe30542e6c6e3aa52f903453c3a2389af918249dbdb5f1199aaecf4931c0366592165b10bdd58eaf706d6df02a39d9323a0c65260ffcc84776f2705e4942d89e4dbefa11c693027002c35582d56e295dcf74d27e90873699657337696b32c05c8014911a7ec8eb03bdbe526fe658be8abdf50ab12c4fec9ddeefc489cf817721c8e541d28fbe71e32137b5ea066a9f4e19814deedeb360def90eff2965570aab5fedd0ebfcd783ce3289360953680ac084b2e988c9cbd0912da400861467d7bb5ad4b42a95c2d541653e805cbfc84da401baf096fba43300358421ae1b43fd25f3289c8c73489977592f75bc9f73781f41718a752ab325b70c8eb2011c5d979f6efc7a76e16492566e43d94dbd42698eb06ff8ad4fd3f2baabafded"
}

⛰ lncli --network=simnet exportchanbackup --all --output_file=channels.backup

⛰ ll channels.backup
-rw-r--r--  1 roasbeef  staff   381B Dec  9 18:16 channels.backup

SCBs can be viewed as a last ditch method for recovering funds from channels due to total data loss. In future releases, we plan to implement methods that require more sophistication with respect to operational architecture, yet allow for dynamic backups. Even with these dynamic backups in place, SCBs will still serve as a fallback method if a dynamic back up may be known to be out of date, or in a partial state of consistency.

Future protocol changes will make the SCB recovery method more robust, as it will no longer rely on the remote peer to send the normal channel reestablishment handshake upon reconnection. Instead, given the SCB, lnd will be able to find the closing output directly on the chain after a force close by the remote party.

For further details w.r.t the lower level implementation of SCBs as well as the new RPC calls, users can check out the new recovery.md file which goes over methods to recover both on-chain and off-chain funds from lnd.

New Channel Status Manager

Within the protocol, nodes can mark a channel as enabled or disabled. A dsiable channel signals to other nodes that the channel isn’t to be used for routing for whatever reason. This allows clients to void these channels during path finding, and also lets routing nodes signal any faults in a channel to other nodes allowing them to ignore them and possibly remove them from their graph view. lnd has a system to automatically detect when a channel has been inactive for too long, and disable it, signalling to other peers that they can ignore it when routing. The system will also eventually re-enable a channel if it has been stable for long enough.

The prior version of sub-system had a number of flaws which would cause channels to be excessively enabled/disabled, causing ChannelUpdate spam in the network. In this release, this system has been revamped, resulting in a much more conservative, stable channel status manager. We’ll now only disable channels programmatically, and channels will only be re-enabled once the peer is stable for a long enough period of time. This period of time is now configurable.

Server and P2P Improvements

The max reconnection back off interval is now configurable. We cap this value by default to ensure we don’t wait an eternity before attempting to reconnect to a peer. However, on laptops and mobile platforms, users may want to value to be much lower to ensure they maintain connectivity in the face of roaming, or wi-ifi drops. The new field is: --maxbackoff=. A new complementary --minbackoff field has also been added.

We’ll now attempt to retry when faced with a write timeout rather than disconnect the peer immediately. This serves to generally make peer connections more stable to/from lnd.

Users operating larger lnd nodes may find that at times restarts can be rather load heavy due to the rapid burst of potentially hundreds of new p2p connections. In this new version of lnd, we’ve added a new flag (--stagger-initial-reconnect) to space out these connection attempts by several seconds, rather than trying to establish all the connections at once on start up.

Performance Enhancements

Outgoing Message Queue Prioritization

[A new distinct queue of gossip messages has been added to the outgoing write queue system within lnd](#2690. We’ll now maintain two distinct queues: messages for gossip message, and everything else. Upon reconnection, certain messages are time sensitive such as sending the Channel Reestablishment message which causes a channel to shift from active to inactive. This queue optimization also means that making new channels, or updating existing channels will no longer be blocked by any outgoing gossip traffic, improving the quality of service.

Batched Pre-Image Writing in the HTLCSwitch

This new release will now batch writes for witnesses discovered in HTLC forwarding. At the same time, we correct a nuanced consistency issue related to a lack of synchronization with the channel state machine. Naively, forcing the individual preimage writes to be synchronized with the link incurs a heavy performance penalty (about 80% in profiling). Batching these allows us to minimize the number of db transactions required to write the preimages, allowing us to reinsert the batched write into the link's critical path and resolve the possible inconsistency. In fact, the benchmarks actually showed a slight performance improvement, even with the extra write in the critical path.

Unified Global SigPool

lnd uses a pool of goroutines that are tasked with signing and validating commitment and HTLC signatures for new channel updates. This pool allows us to process these commitment updates in parallel, rather than in a serial manner which would reduce payment throughput. [Rather than using a single SigPool per channel, we now use a single global SigPool](#2329_. With this change, we ensure that as the number of channels grows, the number of goroutines idling in the sigPool stays constant. It's the case that currently in the daemon, most channels are likely inactive, with only a handful actually consistently carrying out channel updates. As a result, this change should reduce the amount of idle CPU usage, as we have less active goroutines in select loops.

Read and Write Buffer Pools

In this release, we implement a write buffer pool for LN peers. Previously, each peer object would embed a 65KB byte array, which is used to serialize messages before writing them to the wire. As a result, every new peer causes a large memory allocation, which places unnecessary burden on the garbage collector when faced with short-lived or flapping peers. We’ll now use a buffer pool, that dynamically grows and shrinks based on the demand for write buffers corresponding to active peers. This greatly helps when there is a high level of churn in peer activity, or even if there is a single one flapping peer.

Similarly, whenever a new peer would connect, we would allocate a 65KB+16 byte array to use as a read buffer for each connection object. The read buffer stores the ciphertext and MAC read from the wire, and used to decrypt and then decode messages from the peer. Because the read buffer is implemented at the connection-level, as opposed to the peer-level like write buffers, simply opening a TCP connection would cause this allocation. Therefore peers that send no messages, or do not complete the handshake, will add to this memory overhead even if they are released promptly. To avoid this, we now use a similar read buffer pool to tend towards a steady working set of read buffers which drastically reduces memory usage.

Finally, we introduce a set of read/write worker pools, which are responsible for scheduling access to the read/write buffers in the underlying buffer pools. With the read and write pools, we modify the memory requirements to be at most linear in the number of specified workers. More importantly, these changes completely decouple read and write buffer allocations from the peer/connection lifecycle, allowing lnd to tolerate flapping peers with minimal overhead.

Nodes that have a large number of peers will see the most drastic benefit. In testing, we were able to create stable connections (w/o gossip queries) to over 900 unique nodes, all while keeping lnd's total memory allocations due to read/write buffers under 15 MB. This configuration could have easily connected to more nodes, though that was all that reachable via the bootstrapper.
This same test would have used between 90-100MB on master, and continues to grow as more connections are established or peers flap if the garbage collector could not keep up. In contrast, the memory used with read/write pools remains constant even as more peers are established.

Sweeper

A new sweeper subsystem has been introduced. The sweeper is responsible for sweeping mature on-chain outputs back to the wallet. It does so by combining sets of outputs in a single transaction per block. It takes care not to sweep outputs that have a negative yield at the current fee estimate. Those will be left until the fee estimate has decreased enough. Some outputs may still be contested and possibly swept by the remote party. The sweeper is aware of this and properly reports the outcome of the sweep for an output to other subsystems. sweep: create sweeper.

The new Sweeper sub-system is the start of a generalized transaction batching engine within lnd. As is today, it will batch all sweeps (HTLC timeouts, commitment sweeps, CSV sweeps) across lnd into a single transaction per block. In the future, the sweeper will be generalized in order to implement fee bumping techniques like RBF and CPFP in a single logical unit. Additionally, the existence of such a batching engine will allow us to batch all transaction daemon wide into a single transaction, which will allow us to implement block saving features such as: opening multiple channels in a single transaction, combining cross channel splice in/outs, closing out one channel in order to open a new channel or fulfill a request payment. payment.

Overtime the sweeper will also grow to obsolete the existing UtxoNursery as sweep requests will become more distributed (an HTLC asks to be swept rather than the nursery sweeping when the time is right).

Graph Sync Improvements

With the recent rapid growth of the network, it almost became unbearable for nodes to sync their routing table with their peers due to the huge number of updates/channels being announced. We’ve made significant improvements towards addressing this issue with the introduction of the SyncManager. Nodes will now only receive new graph updates from 3 peers by default. This number has been exposed as a CLI flag, —numgraphsyncpeers, and can be tuned for light clients and routing nodes for bandwidth savings. In testing, we’ve seen over a 95% bandwidth reduction as a result of these changes.

This version also reduces the batch size of channels requested via QueryShortChanIDs from 8000 to 500, leading to more stability in large or initial syncs. The previous version was found to invite disconnections from the remote peer once the receiver had received the first few thousand messages. The reduced batch size prevents us from overflowing our own internal queues for gossip messages, and ensuring the remote peer doesn’t interpret this as jammed connection.

Goodbye Zombie Channels

Within the last couple months, we started to experience a large number of zombie channels in the network being gossiped between nodes. A zombie channel is a channel that is still open, but hasn’t been updated for 2 weeks. This issue was also present on testnet a few years back, so we’ve finally addressed the issue for
good. Nodes will now maintain an index of zombie channels which they can query to determine whether they should process/forward announcements for an arbitrary channel.

Using this index, we will also refrain from requesting channels we know to be zombies from peers that think otherwise. At the time of writing, there are roughly 3.3k zombie channels on mainnet. This optimization saves us from requesting 10k individual messages, amounting to roughly 3MB when attempting historical syncs with peers presenting lnd with zombie channels.

On-Chain Commitment and HTLC Handling

A bug has been fixed that would previously cause an HTLC which was settled on chain to not properly be marked as settled..

An off-by-one-error has been fixed in the contract court when handling a remote commitment close due to a DLP execution instance. This ensures that funds will now properly be swept in the case of a force close due to DLP wherein the remote party is one state ahead of ours. Users that ran into this issue in the wild should find that the dispatch logic is re-executed, resulting in on-chain funds properly being swept back into the wallet.

Bitcoind Spend Hint Bug Fix

Fixes a bug that would cause bitcoind backends to perform historical rescans on successive restarts, even if the first rescan completed and did not find a spending transaction. Affected nodes will have to complete one more rescan after upgrading before symptoms will disappear. In more severe cases, this will save tens of thousands of getblocks calls to the backend on each restart.

Autopilot Architecture Revamp

In this release, as a prep for more advanced autopilot heuristics in a future release, we’ve completely revamped the way the system works. Before this release, the autopilot “agent” was directive based, meaning that it when queried, it would simply say “connect to these nodes”. This detective based suggestion was simple, yet limiting in that: it didn’t easily lend to combining multiple heuristics and using only the dertive model, there isn’t a clear way of comparing to distinct heuristics.

The new system instead implements a scoring based agent. Rather than simply suggesting a set of node to connect to, the agent will now return a set of scores for a target, or all peers within the network. This score is then incorporated into the main channel selection loop, adding a bit of jitter to ensure diversity. The scoring based system really shines when you start to consider adding multiple heuristics that work in tandem (connectivity optimized, uptime optimized, reliability optimized, redundancy optimized, etc). With the new scoring system, it’s now possible to create a new heuristics which is actually a combination of several sub heuristics. As an example, we’ve created a new WeightedCombAttachment heuristics that outputs a linear combination of the scores of a set of registered hueirticis.

This new scoring based system will pave the road for more advanced autopilot heuristics which may make certain trade offs in order to target specific use cases like: mobile/laptop oriented, net receiver (merchant, etc) optimized, routing network robustness (min-cut and the like). As a bonus, the new system also makes it much easier to add a new heuristic as the new interface has a single method: NodeScores: given the graph, target channel size, and existing set of node channel it should return a score for all non-filtered out nodes.

Penalize small channels for autopilot scoring

WIth rearchitechting autopilot to be scoring based, the default heuristic (prefattach) will now ]decrease the score of nodes having a large number of small channels.](#2797)

New Sweep All Coins Command

A new argument has been added to the lncli sendcoins interface to allow users to sweep all coins from lnd's wallet to a target on-chain address. An example run of the new command looks something like:

⛰   lncli --network=simnet sendcoins --sweepall --addr=sb1qsy8772pkfucsvmuyw82gexyd4u69pvve9w98v3
{
    "txid": "1931f6653ecb2add24e00ce03d6a66ce705ebf633bfffb2b67674000d5f3d5d4"
}

For those using the RPC interface, the new field to set is send_all, which is a boolean that indicates the amount is left blank, and all coins should be sent to that target address.

RPC Interface Enhancements and Fixes

The default number of routes returned from QueryRoutes is now 10 (prior the default was unspecified).

QueryRoutes with used with more than one route target has been deprecated and will be phased out in future versions of lnd. In order to make up for the lost functionality, we’ve added a series of new arguments to the RPC call that allow users to ignore an arbitrary set of edges or vertexes. This new feature makes it easier to implement things like rebalancing externally, as you can know modify the source node for path finding, which can be used to find a path from node A to B, then back from B to A that must travel in/out of a specific edge set.

VerifyMessage and SignMessage now properly expose a REST endpoint.

The response to SendPayment now also includes the payment hash in order to make it easy to associate a success or failure amongst several payments when using the streaming RPC.

A number of fixes to the request validation within the RPC server have been made. These changes make the server more defensive w.r.t what it accepts from clients.

Invoices created with the --private option (include hop hints to private channels) are [now marked as such on the RPC interface}(https://github.com//pull/2222).

A new RPC call ListUnspent has been added to allow users to examine the current UTXO state of lnd. Combined with the new Signer Sub-Server, users can use this to craft arbitrary transactions using lnd’s backing keystore.

A bug has been fixed that would cause channels that are unconfirmed, but waiting to be closed from being returned via the PendingChannels RPC.

A bug has been fixed that wouldn’t allow users to expose the REST interface on all network interfaces.

The existing settled field from the Invoice proto is now deprecated. Instead, the new state field is to be used as it allows us to reflect additional states of invoices (open, settled, cancelled, etc).

A number of fields in the Invoice proto that have never been populated /used have been removed.

The name of the network is now exposed in the response of the GetInfo RPC.

The UnsettledBalance field in the PendingChannels RPC response is now properly set.

The ListChannels response now includes a field which denotes if the node is the iniatitor of a channel or not.

HTLCs which haven’t yet timed out, are now properly shown in the output of PendingChannels.

A new SubscribeChannels RPC has been added to allow clients to be notified whenever a channel becomes inactive, active, or closed. This is useful for any type of application that would otherwise need to poll the channel state to keep up to date on what channels are active, inactive, or closed.

Two new address types have been added to the NewAddress RPC call. These address types will return the same address until they have been used, then rotate to a new address. These new address types are useful for displaying a new address in UIs without running into “address inflation”.

The getnetworkinfo RPC now also returns the median channel size of the graph. The average degree output in GetNetworkInfo has also been corrected.

A bug that would cause the autopilot agent to over-allocate funds if multiple channels were opened in parallel has been fixed.

We’ll now retrieve the chan_id of an open channel from the channel database when using the ListChannels RPC/listchannels lncli command, rather than the graph. If a channel doesn’t have a new update within the last 2 weeks, then it’ll be pruned from the graph, which caused the chan_id lookup to fail and result in a 0 value being displayed.

Sub-Servers

In this new release, we’ve begun the process of slowly evolving the RPC interface via the new Sub-Server system. The gRPC system allows multiple independent services to be registered to the same endpoint. Before this release, lnd had one primary service: Lightning. All current RPC calls are directed to this unified service. In the early days of lnd, this structure emerged organically as many RPCs were added based on speculative future uses, or primarily for the purposes of testing new features added to the codebase. The result today is one mega interface, without any clear specialization or feature delineation.

Since the initial release of lnd, we’ve received a considerable amount of valuable feedback w..r the RPC interface from developers, businesses, and node operators that use the interface daily. Some of this feedback may require us to extensively re-work core RPC calls like SendPayment. Doing so directly in the main service would be disruptive as the calls may change over night, or have their behavior be drastically modified. We consider Sub-Servers to be a solution to this issue as they allow us to recreate a small subset of the existing RPC interface in a concentrated, methodical manner. By being able to start from scratch, we gain more freedom w.r.t crafting the new interface. Additionally, by being forced to examine a smaller subset of the total functionality in a new Sub-Server, we’re able to consolidate existing code, decouple the RPC interface from the rest of lnd, and also expose new functionality to the RPC interface that may only be tangentially related to Lightning.

As of this release, all sub-servers are guarded behind special build flags (make install -tags=<buildtag>). The rationale here is that the sub-servers only expose new functionality, so existing users of lnd that don’t yet have a need for these new features shouldn’t be burdened with them at runtime. Over time as the interfaces crystalize more, we’ll begin the process of depreciating certain older RPCs in order to promote the newer more design sound Sub-Server RPCs. As a result, the current Sub-Server interfaces should be considered non final and subject to change at anytime. Due to their volatile nature, we don’t yet have documentation up at api.lightning.community. On their application development side of things, using a new sub-server is as simple as creating a new gRPC client service with the existing gRPC client connection.

Sub-Servers also make lnd generally more useful as a one-stop shop for any sort of Bitcoin related programming or application as they expose some core interfaces that lnd uses across the codebase to accomplish routine tasks. When compiled in, certain sub-servers will also augment lncli with a set of new commands. The current set of Sub-Servers (and their respective build tags) include:

  • ChainNotifier (chainrpc)
  • WalletKit (walletrpc)
  • Signer (signrpc)
  • Invoices (invoicesrpc)
  • Autopilot (autopilotrpc)
  • Router (routerrpc)

ChainNotifier Sub-Server

The ChainNotifier Sub-Server is a utility toolkit responsible for requesting the chain backend for notifications about the tip of the chain, transaction confirmations, and output spends. It also includes support for requesting these notifications for arbitrary output scripts as well.

WalletKit Sub-Server

The WalletKit Sub-Server is a utility toolkit that contains method which allow clients to perform common interactions with a wallet such as getting a new address, or sending a transaction. It also includes some supplementary actions such as fee estimation. Combined with the Signer Sub-Server, this lets users create arbitrary transactions (like CoinJoins!) using the existing set of private keys under the control of lnd.

Signer Sub-Server

The Signer Sub-Server exposes the existing input.Signer interface within lnd as an accessible sub-server. The existence of this sub-servers also opens up the possibility of having the actual signer and signing code existing outside of lnd taking the form of either a distinct process, or remote server with additional access control mechanisms.

Invoices Sub-Server

The Invoices Sub-Sever exposes a number of new ways to interact with invoices that don’t exist in the existing invoice related calls for the main service. A new type of invoice called ‘hodl invoice’ has been added. Instead of immediately locking in and settling the htlc when the payment arrives, the htlc for a hodl invoice is only locked in and not yet settled. At that point, it is not possible anymore for the sender to revoke the payment, but the receiver still can choose whether to settle or cancel the htlc and invoice htlcswitch: hodl invoice.

The new invoice function CancelInvoice has been implemented. CancelInvoice can be called on a hodl invoice, but also on a regular invoice. It makes the invoice unpayable invoices: CancelInvoice.

A last improvement to the invoices subsystem is the ability to subscribe to updates of a single invoice instead of receiving all invoice updates invoices: add subscribesingleinvoice.

Autopilot Sub-Server

The Autopilot Sub-Server allows users to programatically drive certain aspects of the autopilot system. Before this new Sub-Server, the only way to modify the settings of autopilot were to modify command line parameters and restart the system. This new Sub-Server allows users to turn autopilot on/off without restarting, and also query the score of a prospective node with the new query interface.

Router Sub-Server

The Router Sub-server presents a simplified interface for sending payments off-chain, and also getting a fee estimate for potential off-chain payments. In future releases, we’ll begin to revamp the off-chain sending interface in order to give users more control w.r.t when we start/stop attempting to fulfill a payment attempt, and also more transparency w.r.t the state of an initiated off-chain payment.

Tor

lnd will now allow a user to access the Tor daemon with NULL authentication. Additionally, it’s now possible to listen on a distinct interface that isn’t localhost when running in auto hidden service mode. This allows users that are aware of the implications to run in a hybrid mode that accepts both inbound clearnet and hidden service connections. Additionally, users can now listen on an arbitrary interface if they have outbound Tor configured.

Neutrino Enhancements

We’ll now start syncing headers and filter headers as soon as lnd’s wallet is created/unlocked. This greatly improves the user experience by reducing the amount of time to reach a fully synced light client lnd instance.

We’ll reliably broadcast transactions to our bitcoin peers to ensure they propagate throughout the network.

Library Enhancements and Multi-Module Support

This release begins some efforts towards transitioning into a multi-module repository. This allows specific packages within lnd to be used externally without the need of duplicating code or running into import cycles.

The ReadElements and WriteElements methods from lnwire are now exposed publicly. This allows any Go program to easily be able to serialize structs/data using the codec described in the BOLT documents.

Wallet Bug Fixes and Improvements

The wallet will no longer rescan from its birthday if it has no UTXOs.

The wallet will now properly remove transactions from its persistent store that the chain backend deems as
invalid.

The wallet will now properly remove transaction conflicts from its persistent store.

The default recovery window has been increased from 250 to 2500. This increase is meant to ensure that the typical wallet is able to complete the regular seed rescan/import without needing to increase the existing default look ahead value.

Routing

Two new optional restrictions have been added that influence route optimization for sending payments:

Furthermore, several new parameters have been added to the QueryRoutes rpc call to allow more control over the returned route lnrpc: deprecate QueryRoutes with more than one route. Requesting multiple routes from QueryRoutes based on the k-shortest algorithm has been deprecated. This behaviour can be re-implemented client side using the new QueryRoutes parameters.

In the SendToRoute rpc call, the ability to specify multiple routes has been deprecated lnrpc: deprecate SendToRoute with more than one route.

The default CLTV delta for channels created by lnd has been lowered from 144 blocks to 40 blocks. Future versions of lnd will begin to automatically modify this parameter based on the sampled fee levels in the chain.

Breacharbiter Preparatory Work

A number of enhancements to the Breacharbiter have been made which are required for the ultimate watch tower implementation. These changes ensure that lnd is able to continue to function if it isn’t the one that ends up sweeping all the outputs in the case of a breach (the tower might sweep the commitment outputs for example, and lnd sweeps the HTLCs itself).

Accounting

Improvements have been made to the PendingChannels report. Several categories of funds in limbo that were previously unreported have been added to the report rpc+contractcourt: merge the contractResolver state into the pendingchannels RPC response.

Changelog

The full list of changes since 0.5.2-beta can be found here:

Contributors (Alphabetical Order)

Adam Gibson
Alex Akselrod
Alex Bosworth
Brian Sipple
Carla Kirk-Cohen
Chris Coverdale
Conner Fromknecht
chokoboko
Cryptcoin Junkey
cryptagoras
Erik Ek
Eugene Siegel
Federico Bond
Francisco Calderón
Gfloresechaiz
Igor Cota
Jim Posen
Johan T. Halseth
John Ng
John Griffith
Jonathan Cross
Joost Jager
Matt Drollette
Max Kaplan
Moshe Shababo
Offer Markovich
Olaoluwa Osuntokun
Ondrej Calda
orbitalturtle
Otto Suess
Philipp Gillé
Roei Erez
Ron Cohen
Sanket Kanjalkar
Sevastos
solid-pay
Thomas Braunberger
Tom Kirkpatrick
Valentine Wallace
Vincent Woo
Wilmer Paulino
Xavi Soler
Yancy Ribbens

Assets 26
You can’t perform that action at this time.