Skip to content
Pre-release
Pre-release

@joostjager joostjager released this Jan 15, 2020 · 2 commits to master since this release

This marks the first major release in the v0.9.x series, as well as the first lnd release of the new decade! This release contains several new features, optimizations, new protocol-level features surfaced on the RPC layer, cross-implementation protocol compatibility fixes, and as usual a ton of bug fixes. Shouts out to all the contributors (over 30+ for this release!) as well as all those that reported bugs and tested, that helped make this new release happen.

Database Migrations

This release contains a single migration which modifies the on-disk representation of the invoices to be more flexible by utilizing the TLV serialization defined in the BOLT specifications. This change allows us to easily add more information to the invoices, which will be especially useful as we finalize our support for AMP within the daemon. The migration should look something like this upon initial start up:

2020-01-09 16:35:39.586 [INF] LTND: Version: 0.9.0-beta commit=v0.9.0-beta, build=development, logging=default
2020-01-09 16:35:39.587 [INF] LTND: Active chain: Bitcoin (network=simnet)
2020-01-09 16:35:39.589 [INF] CHDB: Checking for schema update: latest_version=12, db_version=11
2020-01-09 16:35:39.589 [INF] CHDB: Performing database schema migration
2020-01-09 16:35:39.589 [INF] CHDB: Applying migration #12
2020-01-09 16:35:39.589 [INF] CHDB: Migrating invoice bodies to TLV, adding payment addresses and feature vectors.
2020-01-09 16:35:39.589 [INF] CHDB: Migration to TLV invoice bodies, payment address, and features complete!

Verifying the Release

In order to verify the release, you'll need to have gpg or gpg2 installed on your system. Once you've obtained a copy (and hopefully verified that as well), you'll first need to import the keys that have signed this release if you haven't done so already:

curl https://keybase.io/joostjager/pgp_keys.asc | gpg --import

Once you have the required PGP keys, you can verify the release (assuming manifest-v0.9.0-beta-rc2.txt and manifest-v0.9.0-beta-rc2.txt.sig are in the current directory) with:

gpg --verify manifest-v0.9.0-beta-rc2.txt.sig

You should see the following if the verification was successful:

gpg: assuming signed data in 'manifest-v0.9.0-beta-rc2.txt'
gpg: Signature made Wed 15 Jan 2020 11:43:17 PM CET
gpg:                using RSA key C49CD91BF681FDAD9AE79BBEA61B9D4C393C59C7
gpg: Good signature from "Joost Jager <joost.jager@gmail.com>" [ultimate]

That will verify the signature of the manifest file, which ensures integrity and authenticity of the archive you've downloaded locally containing the binaries. Next, depending on your operating system, you should then re-compute the sha256 hash of the archive with shasum -a 256 <filename>, compare it with the corresponding one in the manifest file, and ensure they match exactly.

Verifying the Release Binaries

Our release binaries are fully reproducible. Third parties are able to verify that the release binaries were produced properly without having to trust the release manager(s). See our reproducible builds guide for how this can be achieved. The release binaries are compiled with go1.13.4, which is required by verifiers to arrive at the same ones. They include the following build tags: autopilotrpc, signrpc, walletrpc, chainrpc, invoicesrpc, routerrpc, and watchtowerrpc. Note that these are already included in the release script, so they do not need to be provided.

Finally, you can also verify the tag itself with the following command:

git verify-tag v0.9.0-beta-rc2

Building the Contained Release

Users are able to rebuild the target release themselves without having to fetch any of the dependencies. In order to do so, assuming that vendor.tar.gz and lnd-source-v0.9.0-beta-rc2.tar.gz are in the current directory, follow these steps:

tar -xvzf vendor.tar.gz
tar -xvzf lnd-source-v0.9.0-beta-rc2.tar.gz
GO111MODULE=on go install -v -mod=vendor -ldflags "-X github.com/lightningnetwork/lnd/build.Commit=v0.9.0-beta-rc2" ./cmd/lnd
GO111MODULE=on go install -v -mod=vendor -ldflags "-X github.com/lightningnetwork/lnd/build.Commit=v0.9.0-beta-rc2" ./cmd/lncli

The -mod=vendor flag tells the go build command that it doesn't need to fetch the dependencies, and instead, they're all enclosed in the local vendor directory.

Additionally, it's now possible to use the enclosed release.sh script to bundle a release for a specific system like so:

LNDBUILDSYS="linux-arm64 darwin-amd64" ./build/release/release.sh

⚡️⚡️⚡️ OK, now to the rest of the release notes! ⚡️⚡️⚡️

Release Notes

Macaroon Bakery 🍪

A new custom macaroon bakery is now available through gRPC (BakeMacaroon) or the command line (lncli bakemacaroon)! The bakery allows users to mint macaroons with their own custom set of read/write permissions for use cases where the three pre-defined default macaroons (admin, readonly and invoice) are not enough. The documentation has also been updated to describe the process.

Note: The bakery requires a new permission that the existing admin.macaroon does not contain. Follow the steps described in the documentation to re-generate the admin macaroon.

New Payment Features

Multi-Path Payments

This release includes full support for receiving Multi-Path Payments as defined in BOLT 04, which is the first flavor of AMP rolling out on the network. MPP allows an invoice to be settled by multiple concurrent HTLCs, each of which carries a portion of the total payment.

With receiving support, MPP-aware senders can make better use of the payee's total inbound capacity in case there is no single channel that can support the payment amount. From a UX perspective, this allows wallet UIs to begin exploring a true, unified receiving capacity since the sender is able to utilize all of the receiver's channels for a single payment. As more MPP-aware senders come on to the network, 0.9 nodes will be able to accept payments without upgrading their software. We are currently targeting support for sending MPP in 0.10.

At a high-level, MPP is implemented via an additional TLV record in the final hop's onion blob, containing a random payment_secret and total_amt_msat being sent. Each HTLC carries this additional record in its onion blob, allowing the receiver to identify and settle an invoice with multiple HTLCs. Invoices will track all HTLCs in the set that are used to settle an invoices, as can be seen in this stripped down invoice settled by two HTLCs:

{                                                                  
    "memo": "multi path payment",                                             
    "r_preimage": "73d8d926c13579b0b1decf6b808a2bbf7db59073999c5d0a14d4dcc9c4225a49",
    "r_hash": "a3309015130da7cc28f529191a3d0aeb06edfedfd58dac1dfde8b1fbd4eadc91",
    "value_msat": "1000000",                                                                               
    "amt_paid_msat": "1000000",                                    
    "state": "SETTLED",                                            
    "htlcs": [                                                     
        {                                                          
            "chan_id": "571746046509056",                          
            "htlc_index": "4",                                     
            "amt_msat": "500000",                                 
            "accept_height": 525,                                  
            "accept_time": "1579112858",                           
            "resolve_time": "1579112860",                          
            "expiry_height": 565,                                  
            "state": "SETTLED",                                    
            "custom_records": {                                    
            },                                                     
            "mpp_total_amt_msat": "1000000"                        
        },
        {                                                          
            "chan_id": "571746046509056",                          
            "htlc_index": "5",                                     
            "amt_msat": "500000",                                 
            "accept_height": 525,                                  
            "accept_time": "1579112654",                           
            "resolve_time": "1579112654",                          
            "expiry_height": 565,                                  
            "state": "SETTLED",                                    
            "custom_records": {                                    
            },                                                     
            "mpp_total_amt_msat": "1000000"                        
        }                                                               
    ],                                                                                    
}

Note that amt_msat paid by each HTLC sums both to the mpp_total_amt_msat value included by the sender, as well as amt_paid_msat displaying the total amount received.

The rpc responses for ListPayments, routerrpc.SendPayment, and routerrpc.TrackPayment have also been extended with an htlcs field that exposes additional data about individual HTLC attempts. For now these will only show the last attempted HTLC, but will be populated with a full log of all HTLC attempts sent once MPP sending is fully integrated. This will allow us to display multiple successful HTLCs, as well as failed attempts. The latter will useful in allowing senders to inspect failed routes, and glean insights about why certain attempts failed.

A stripped down example of the new payment responses:

        {
            "payment_hash": "8eedb4bcd024f7f46fdf60b57e54dba60e19cb2c8934ec5b6fbb7f5efb58b29b",
            "payment_preimage": "cf6b808a2bbf7db59073973d8d926c13579b0b1de99c5d0a14d4dcc9c4225a49",
            "creation_time_ns": "1578620428000000000",
            "path": [
                "03186ce6a4a186895ee44a9864ef025fde8fb587d98926bfc370e2c366597a3f8f",
                "0206e82064f547768dcef9776b80cabdf82638aac86948e4c77b9ea931ea75bab5"
            ],
            "value_msat": "1000",
            "fee_msat": "0",
            "status": "SUCCEEDED",
            "htlcs": [
                {
                    "status": "SUCCEEDED",
                    "route": {
                        "total_time_lock": 612315,
                        "total_fees_msat": "0",
                        "total_amt_msat": "1000",
                        "hops": [
                            {
                                "chan_id": "612270746636810592",
                                "chan_capacity": "5000000",
                                "fee_msat": "0",
                                "amt_to_forward_msat": "1000",
                                "expiry": 612171,
                                "pub_key": "03186ce6a4a186895ee44a9864ef025fde8fb587d98926bfc370e2c366597a3f8f",
                                "tlv_payload": true,
                                "mpp_record": null,
                                "custom_records": {
                                }
                            },
                            {
                                "chan_id": "612270746636810592",
                                "chan_capacity": "5000000",
                                "expiry": 612171,
                                "fee_msat": "0",
                                "amt_to_forward_msat": "1000",
                                "pub_key": "0206e82064f547768dcef9776b80cabdf82638aac86948e4c77b9ea931ea75bab5",
                                "tlv_payload": true,
                                "mpp_record": {
                                    "payment_addr": "f91731b1e2227cc26137d2a2c0029ceef41ac8c481c8d3cc61510bf76ff27cbe",
                                    "total_amt_msat": "1000"
                                },
                                "custom_records": {
                                }
                            }
                        ]
                    },
                }
            ]
        }

Here we can see that each HTLC attempt displays it's full route, including the fees paid and timelocks incurred at each hop. MPP payments can be identified by the presences of an mpp_record attached to the final hop, which containing the new fields described above. If this field is not present, it indicates that payment was made using the legacy payment type.

Note: The path field has been deprecated in favor of extracting the pubkeys from the route directly. creation_date has also been deprecated and will replaced by creation_time_ns. As seen above, legacy creation_dates are converted into nanoseconds for display, but still only have one second resolution.

Weakness Addressed by MPP

MPP is more robust against active probing of the receiver, since the sender must know the invoice's payment_secret. This value serves as a one-time authentication token, further restricting the set of individuals who can settle a particular invoice. In particular, invoices can now only be settled by those who have seen it, rather than only needing to know the correct payment hash which is transmitted in clear over the network.

MPP also offers increased safety to situations involving over payment of invoices, most notably zero-value invoices used for tipping or donations. With the legacy protocol, it is possible to snipe the difference between the invoice amount and total amount carried by the HTLC, allowing intermediaries to take the difference as profit in addition to fees earned. MPP payments resolve this by privately communicating the sender's intended payment amount in the onion blob (via total_amt_msat) to the receiver. Since intermediaries will not know the payment_secret, sniping is prevented even if the intermediary can construct a valid HTLC with a smaller total_amt_msat.

Single-Shot MPP Payments by Default

Although 0.9 does not support sending true MPP payments, it does use the new MPP protocol as the default mechanism for single-shot payments (payments with only one HTLC). Doing so addresses the known weaknesses in the legacy payment mechanism discussed above, and seamlessly upgrades the overall privacy and security of the basic payment style used today.

No action needs to be taken in order to benefit from this upgrade. Senders will automatically detect the receiver's support via feature bits in BOLT 11 invoices, falling back to the legacy protocol for older nodes. As such, the new mechanism will be used when paying from 0.9 to another 0.9 node, or a newer release of C-Lightning or Eclair. Further, these improved payments are secure and can be used even if intermediaries are outdated, as it only requires support from the endpoints.

Custom Onion-Tunneled TLV Payment Metadata Records

It is now possible to attach additional data to a payment using custom records. The send calls on the RPC interface have a field dest_custom_records that takes the custom records to send. This field is also present on QueryRoutes. Custom records take up space in the fixed size onion packet, leaving fewer bytes to describe the route. The pathfinding algorithm needs to know how many bytes are left and limit the route length to that. Checks for length and size have been added.

On the receiving end, the custom records are stored in the invoice database and exposed through the existing invoice query calls (LookupInvoice, ListInvoices, etc). This new feature allows any payment on the network to attach additional meta-data along with the payment. As an example, a user could send a payment to an exchange to deposit funds, and attach their account ID along-side the payment, which the exchange then checks+verifies upon receipt before crediting their account with the deposit. This is a highly anticipated feature as it allows far a large class of new applications to be built on top of Lightning. For further ideas on how to leverage this new feature, check out Laolu's talk on Advanced Lightning Applications from last last year.

Sending a payment with custom records attached will look something like the following on the command line:

{
    "payment_error": "",
    "payment_preimage": "5c8fb9c043d00e4c1780b2e0992a979284b941700219726c71e6093c387679de",
    "payment_route": {
        "total_time_lock": 1657728,
        "total_fees": "0",
        "total_amt": "1000",
        "hops": [
            {
                "chan_id": "1589156041461923840",
                "chan_capacity": "16777215",
                "amt_to_forward": "1000",
                "fee": "0",
                "expiry": 1657728,
                "amt_to_forward_msat": "1000000",
                "fee_msat": "0",
                "pub_key": "0270685ca81a8e4d4d01beec5781f4cc924684072ae52c507f8ebe9daf0caaab7b",
                "tlv_payload": true,
                "mpp_record": null,
                "custom_records": {
                    "5482373484": "5c8fb9c043d00e4c1780b2e0992a979284b941700219726c71e6093c387679de"
                }
            }
        ],
        "total_fees_msat": "0",
        "total_amt_msat": "1000000"
    },
    "payment_hash": "8cf790cc128a0bb0552b3223d542adfba6a93c948f84e49dcd532309f5b85634"
}

Notice the new custom_records field, as well as the tlv_payload boolean which specifies that this route used the new modern onion paylaoad format.

On the receiver, end, the new output of ListInvoices looks something like:

        {
            "memo": "",
            "r_preimage": "5c8fb9c043d00e4c1780b2e0992a979284b941700219726c71e6093c387679de",
            "r_hash": "8cf790cc128a0bb0552b3223d542adfba6a93c948f84e49dcd532309f5b85634",
            <snip>
            "htlcs": [
                {
                    "chan_id": "1589156041461923840",
                    "htlc_index": "8",
                    "amt_msat": "1000000",
                    "accept_height": 1657685,
                    "accept_time": "1578617436",
                    "resolve_time": "1578617436",
                    "expiry_height": 1657728,
                    "state": "SETTLED",
                    "custom_records": {
                        "5482373484": "5c8fb9c043d00e4c1780b2e0992a979284b941700219726c71e6093c387679de"
                    },
                    "mpp_total_amt_msat": "0"
                }
            ],
          <snip>
        }

Once again the custom_records will will expose a map from the integer type of the record to the raw bytes of the value. Note that custom records are only permitted with a type greater than 65536, any records below this value are reversed for official protocol usage. For further details with respect to the new RPC-level changes to the payment sending/invoicing calls, see the lnd API documentation which is now up-to-date as of lnd v0.9.0.

With the introduction of custom record sending and receiving, it became possible to attach arbitrary data to a payment. One obvious use case is attaching a human-readable message to a payment. Especially in the case of spontaneous keysend payments, this can give the receiver some context on the payment.

For example: tipping. Usually people sending a tip would want to include some information on who they are or what the tip is for. For Lightning this may be even more desired than for other payment methods, because payments are anonymous by default.

Custom records can also be set using the lncli command like so:

lncli sendpayment -d 0374e7fb33eafd74fe1acb6db7680bb4aa78e1c839a6f954e38abfad680f645ef7 -a 100 --key_send --data 323442=00,3234556=ffff080812

To specify a string value, the standard command line tool xxd can be used (the example record id here is the 3-byte ascii string 'tip' converted to an integer):

--data 7629168=$(echo -n "Thank you!" | xxd -pu -c 10000)

(The -c parameter is to prevent xxd from inserting line breaks)

Note: The available onion blob space of 1300 bytes is used for routing info and custom records. The bigger the size of the custom records, the fewer bytes remain for routing info and the shorter the maximum route length will be.

New Payment Type: keysend

One application of custom records is a spontaneous payment, also known as keysend. In key send, a custom record is used to encode the payment preimage in the onion payload for the recipient of the payment. This allows them to pull the payment without prior knowledge of the preimage.

Note that spontaneous payment is not yet defined in the Lightning spec. Therefore the current implementation should be considered experimental and is subjected to change.

In order to send a keysend payment on the command line, a new flag --key_send needs to be specified, along with the destination (--dest) and the amount to send (--amt). Notice that we don't specify a payment hash anywhere! Instead, the sender will actually encrypt the payment pre-image to the receiver, who will then decrypt the payload along with the rest of the normal per-hop routing information.

A sample keysend executed at the command line resembles something liek the following:

🏔 tlncli sendpayment --key_send --dest=0270685ca81a8e4d4d01beec5781f4cc924684072ae52c507f8ebe9daf0caaab7b --amt=1000 --final_cltv_delta=40
{
    "payment_error": "",
    "payment_preimage": "5c8fb9c043d00e4c1780b2e0992a979284b941700219726c71e6093c387679de",
    "payment_route": {
        "total_time_lock": 1657728,
        "total_fees": "0",
        "total_amt": "1000",
        "hops": [
            {
                "chan_id": "1589156041461923840",
                "chan_capacity": "16777215",
                "amt_to_forward": "1000",
                "fee": "0",
                "expiry": 1657728,
                "amt_to_forward_msat": "1000000",
                "fee_msat": "0",
                "pub_key": "0270685ca81a8e4d4d01beec5781f4cc924684072ae52c507f8ebe9daf0caaab7b",
                "tlv_payload": true,
                "mpp_record": null,
                "custom_records": {
                    "5482373484": "5c8fb9c043d00e4c1780b2e0992a979284b941700219726c71e6093c387679de"
                }
            }
        ],
        "total_fees_msat": "0",
        "total_amt_msat": "1000000"
    },
    "payment_hash": "8cf790cc128a0bb0552b3223d542adfba6a93c948f84e49dcd532309f5b85634"
}

The key area to examine is the following:

                "custom_records": {
                    "5482373484": "5c8fb9c043d00e4c1780b2e0992a979284b941700219726c71e6093c387679de"
                }

Here we use lnd's keysend record record to package the pre-image to the receiver. You can verify that the payment_preimage field and the payment_hash field correspond. If one wishes to perform a keysend programmatically via the RPC API, then the DestCustomRecords field will need to be set accordingly.

On the receiver side, by default all nodes will reject any incoming keysend payments. In order to enable receiving such payments, nodes will need to opt-in by starting lnd with a new flag: --accept-key-send. When examining an invoice, it's easy to determine if it was payed using a keysend or not. When we receive a new incoming keysend payment that's to be accepted, we'll insert a new invoice into the database which looks something like the following:

        {
            "memo": "",
            "r_preimage": "5c8fb9c043d00e4c1780b2e0992a979284b941700219726c71e6093c387679de",
            "r_hash": "8cf790cc128a0bb0552b3223d542adfba6a93c948f84e49dcd532309f5b85634",
            "value": "1000",
            "value_msat": "1000000",
            "settled": true,
            "creation_date": "1578617436",
            "settle_date": "1578617436",
            "payment_request": "",
            "description_hash": null,
            <snip>
                {
                    "chan_id": "1589156041461923840",
                    "htlc_index": "8",
                    "amt_msat": "1000000",
                    "accept_height": 1657685,
                    "accept_time": "1578617436",
                    "resolve_time": "1578617436",
                    "expiry_height": 1657728,
                    "state": "SETTLED",
                    "custom_records": {
                        "5482373484": "5c8fb9c043d00e4c1780b2e0992a979284b941700219726c71e6093c387679de"
                    },
                    "mpp_total_amt_msat": "0"
                }
            ],
            "features": {
                "9": {
                    "name": "tlv-onion",
                    "is_required": false,
                    "is_known": true
                }
            },
            "is_key_send": true
        }

Notice that there's no payment_request value, and the set of new fields such as custom_records and information detailing the set of feature bits used to complete the payment (described in further detail below).

First-Class Rebalancing via Circular Payments to Self

We have added support to send a payment to yourself. This allows rebalancing of channels without the need for external scripts. To control the incoming channel, a new last hop routing restriction was added.

This new feature is a valuable tool in one's toolkit for node management as it allows you control the incoming and outgoing channels used to send a payment. Let's say I have a channel with ID 12345 and I sent to send funds from that channel (rebalancing to be able to receive more into the channel) to another channel that I have with peer 02e1dfa (rebalancing to be able to send more out of the channel). First, I'll create a new invoice on the command line for the amount I want to use for rebalancing purposes:

🏔 lncli --network=testnet addinvoice --amt=1000
{
    "r_hash": "ac7c8d8f72b34d68b1ed53b1401908b4fa3a2b0c41a21206686289f437bd02fa",
    "payment_request": "lntb10u1p0p0nm3pp5437gmrmjkdxk3v0d2wc5qxggknar52cvgx3pypngv2ylgdaaqtaqdqqcqzpgrzjq027z73uyyl7fy8pkrpcn7x0el82pz3fw974p2052de4uz4j5lqqx943xgqqq9qqqqqqqqqqqqqqqqqqpusp5ljp309kulcyghcfs7usd9ed0fxm85tquuyxf36z5k6wj6cddyssq9qy9qsqqwsldznwmg3prdgq7ddul7jspgk3n6fvkqgjc94phjc78jhkzc9y9q8vsmknp02mqg8tglhlvuseqq2nnar338tnqsspu79qvtmae0gpujtztv",
    "add_index": "167"
}

Now lnd is ready to execute the rebalancing attempt. To do this, we'll execute a new modified call to our trusty payinvoice command. Note that it's possible to do this programmatically+ as well using the RPC interface API. In any case, here's our command to complete this rebalancing attempt:

lncli --network=testnet payinvoice --allow_self_payment --outgoing_chan_id=12345 --last_hop=02e1dfa --pay_req=lntb10u1p0p0nm3pp5437gmrmjkdxk3v0d2wc5qxggknar52cvgx3pypngv2ylgdaaqtaqdqqcqzpgrzjq027z73uyyl7fy8pkrpcn7x0el82pz3fw974p2052de4uz4j5lqqx943xgqqq9qqqqqqqqqqqqqqqqqqpusp5ljp309kulcyghcfs7usd9ed0fxm85tquuyxf36z5k6wj6cddyssq9qy9qsqqwsldznwmg3prdgq7ddul7jspgk3n6fvkqgjc94phjc78jhkzc9y9q8vsmknp02mqg8tglhlvuseqq2nnar338tnqsspu79qvtmae0gpujtztv

Local balance check

Failure reporting in case none of the local channels has enough balance to execute the payment has been improved. We now return a specific insufficient balance payment result instead of only indicating that no route could be found.

mSAT support

Support for mSAT payments has been added on the RPC level for payments and invoices.

Privacy Enhancement

This new release of lnd will properly initialize the starting mix-header Sphinx packets with random bytes in order to patch a recently discovered privacy leak that could potentially allow the receiver of a payment to ascertain the number of hops the payment traveled. See this email by roasbeef for further details.

BOLT Compatibility

Invalid Onion Payload Failure

0.9 now returns the new invalid_onion_payload failure upon receiving a malformed onion TLV payload.

Flat Features

The legacy local and global feature bit namespaces have been merged into a single namespace, simplifying the way feature bits are assigned and used within the protocol. Features previously advertised as global in init messages will remain for backwards compatibility, while new features will be appended to the existing local features field. The global features field will be deprecated entirely once this is widespread within the network.

Validate Sorted Uncompressed Short Channel IDs

0.9 now verifies that short channel IDs in QueryShortChanIDs and ChannelRangeReply are sent in ascending order for the uncompressed format. This had already implemented for the zlib encoding, but is now checked across both encodings.

Add payment_secret to BOLT 11 Payment Requests

Payment requests containing a payment_secret are properly encoded and decoded in 0.9. In addition, every newly created payment requests will encode a random payment_secret so that senders may use the improved MPP payment mechanism.

#3788

Cross-Implementation Protocol Compatibility Fixes

It was discovered that slight deviations existed between the various implementations w.r.t the way they implemented the gossip queries protocol feature. This version of lnd has been patched to ensure that we'll properly send responses as dictated by the spec to ensure better compatibility with c-lightning and various other implementations.

Channel Funding+Management Extensions

Funding Flow Abstraction

This new release of lnd contains a dramatically refactored funding workflow internally. This new refactoring allowed us to abstract away aspects such as how the funding transaction is constructed. We have many new features planned around this cut out such as generic external channel funding driven by PSBT, internal merging of several funding transaction into a single, or specifying a cold commitment key (where all your funds are sent to on force close, only need for sweeping and not channel updates).

With this release, we're shipping a low-level call that allow external crafting of a funding transaction, given extra-protocol negotiation. In other words, a new RPC call that allows two parties to agree on the structure of a funding transaction outside of the protocol, obtain the outpoint, then use that as part of the normal funding workflow. Such a flow is useful for a number of advanced features such as the creation of channel factories.

The new FundingShim parameter allows a caller to initiate a new funding flow, with apriori knowledge of information typically generated dynamically during the funding workflow such as the outpoint of the final funding transaction and the keys used within the multi-sig output. A responder will then use the new FundingStateStep method to register a matching funding intent to dispatch automatically based on an expected pending channel ID.

As mentioned above, we have a number of interesting features planned for this new API call, so stay tuned!

Decoupled Min HTLC Settings

The default minimum htlc amount channel open parameter is lowered from 1000 msat to 1 msat. This parameter is immutable during the life of the channel. By changing it to the lowest possible value, maximum flexibility is achieved. The minimum htlc amount can still be constrained to a higher value through the forwarding policy. Updating the minimum forward amount is now possible via the RPC and lncli.

Option Upfront Shutdown Support

This release adds support for option_upfront_shutdown which enforces payout on cooperative close to an address that is set on channel open. lnd will disconnect from peers that attempt to close out to a script other than the one that they specified on startup. Further information about feature bit can be found in the BOLT2 specification. Use of this feature provides a partial protection against node compromise because channels can be set to close out to addresses which are not controlled by lnd.

An upfront shutdown address can be set with the close_address field in the lnrpc/lncli OpenChannel call. Currently, only seed controlled addresses can be set as the upfront shutdown address when we do not initiate a channel open because this step is non-interactive. lnd can be started with --enable-upfront-shutdown to set seed addresses for all channel opens and closes by default.

Sweep Small Outputs

Sweeper (responsible for sweeping close transaction outputs back into the wallet) has been enhanced so that it also sweeps small outputs that on their own do not reach the sweep transaction dust limit. It does so by attaching additional utxos from the wallet, if available. This change is a preparation for the upcoming anchor commitment format.

This change also makes it easier to use the BumpFee RPC (or lncli wallet bumpfee) call for generic CPFP usage.

Close Address

Channels can be closed out to a custom address using the delivery_address field in lncli/lnrpc's CloseChannel call. Note that this feature cannot be used if option_upfront_shutdown, described above, was pre-set for the channel on opening.

This feature allows users to close out a channel and make a payment within the same transaction. This can also be used with Lightning Loop to initiate a Loop In from a channel close, or even to create a new channel by sending to a P2WSH multi-sig script using the new funding APIs described above.

Autopilot External Score Trigger

Autopilot will now check whether it should open more channels after the external scores list has been updates.

Channel Fitness Tracking

A channelfitness subsytem for tracking the health of the node's existing set of channels has been added. This subsystem tracks the online status of the remote peers that we have channels open with. These results are not currently persisted, so uptime is tracked from node restart rather than from channel open time. A lifetime value which indicates how long a peer has been monitored for is provided to allow calculation of uptime percentages. Uptime and lifetime are exposed as experimental fields on the lnrpc/lncli ListChannels call; both are expressed in seconds.
Note: these fields are experimental and may be moved or changed in future versions on LND.

Routing

Mission control

Several updates have been applied to mission control, the sub-system in lnd that is responsible for tracking the performance of nodes involved in past payments.

The estimation of success probabilities for channels has been improved. Channels are no longer evaluated in isolation. A new factor was added that represents the overall performance of a node. This prevents pathfinding from getting stuck in trying every channel of a badly performing node.

Probability estimation for local channels is more accurate now. This prevents unnecessary detours to reach a direct peer.

Mission control now remembers more about what happened in the past. Previously it only kept the last payment attempt for a channel. With this release, this is extended to tracking both the last failure and the last success. The goal is to improve payment success rates especially when small amount probes are performed frequently.

Finally, mission control state reporting has changed slightly. Probabilities are now amount-dependent and are therefore no longer reported through QueryMissionControl. A new call QueryProbability takes over this function.

Pathfinding Improvements

The routing algorithm acknowledges that a pair of nodes may have multiple channels between them with distinct forwarding policies. This is not a recommended practice, but needs to be dealt with nevertheless. lnd will craft a route that satisfies the most demanding (fee and time lock) channel of the channel set.

Routing decisions are based on cost and reliability. We applied a small touch up in this area to tie break on reliability when costs are equal.

RPC Enhancements

Deeper Feature Bit Inspection

The tried and true lncli getinfo call has been upgraded to show additional detail including the set of feature bits that the node is configured to advertise on the network. A sample of the new format of the call looks something like:

{
    "version": "0.9.0-beta commit=queue/v1.0.2-222-g6c8c99dae99e741d2817d444f8b11945ddd15e2e",
    "identity_pubkey": "0270685ca81a8e4d4d01beec5781f4cc924684072ae52c507f8ebe9daf0caaab7b",
     <snip>
    "features": {
        "0": {
            "name": "data-loss-protect",
            "is_required": true,
            "is_known": true
        },
        "5": {
            "name": "upfront-shutdown-script",
            "is_required": false,
            "is_known": true
        },
        "7": {
            "name": "gossip-queries",
            "is_required": false,
            "is_known": true
        },
        "9": {
            "name": "tlv-onion",
            "is_required": false,
            "is_known": true
        },
        "13": {
            "name": "static-remote-key",
            "is_required": false,
            "is_known": true
        },
        "15": {
            "name": "payment-addr",
            "is_required": false,
            "is_known": true
        },
        "17": {
            "name": "multi-path-payments",
            "is_required": false,
            "is_known": true
        }
    }
}

The same feature vectors format is also now exposed on a variety of other rpc responses, each with slightly difference semantics:

  • GetNodeInfo and DescribeGraph: displays the current feature vector advertised on a node_announcement within our routing table.
  • DecodePayReq: displays the feature vector decoded from an arbitrary payment request.
  • ListInvoices: displays the feature vectors advertised on our own invoices.
  • ListPeers: displays the set of features advertised by the remote party while exchanging init messages.

Updates to Default gRPC Settings

The receive buffer of the gRPC client in lncli and the REST proxy has been increased from 50MiB to 200MiB to avoid errors on very large response data (for example, thousands of transactions in listchaintxns).

Uniform lncli Hex-Encoding

The gRPC gateway library that is used to expose lnd's interface as a REST/JSON API on top of the normal gRPC API did not support request parameters of the type []byte (or bytes in the gRPC proto language). The update to version 1.8.6 of this library fixes this and should now allow REST users to use all API calls that lnd offers. Fields declared as bytes in the proto file must be encoded as base64 when used over REST. Some string fields in rpc.proto that were only added as a workaround in earlier versions have been deprecated and will be removed in the next version.

With this change, all raw byte fields exposed on the lncli command responses will now all be hex-encoded, with no more usage of base64. This change was made to clear up confusion that users ran into when information was encoded in multiple ways across distinct calls.

Updates to QueryRoutes

The QueryRoutes RPC call is now able to pin which outgoing channel is used when path finding, pin a pubkey to target for the final hop, and also accept custom unrolled hop hints. All of these changes make the RPC even more generally useable as it can be used for things like rebalancing pre-processing, advanced more advanced payment types, and fine-grained channel selection.

New RPC Calls

With the experimental signer RPC subserver you can now sign and verify messages with custom keys of the wallet (limited to the m/1017'/ special purpose derivation branch).

Note: This RPC requires a new permission, re-generate your admin.macaron and/or signer.macaroon to get the new permission.

A new RPC to derive shared Diffie-Hellman (ECDH) keys against a node's identity public key has been added to the experimental signer subserver.

A new RPC endpoint has been added to provide a subscription for peer online and offline events.

Mobile

Building

The mobile build is now created using the github.com/lightninglabs/falafel tool.

API changes

The Start library method has gotten an additional callback. Now lnd will notify both when the wallet is ready to be unlocked, and when lnd is ready to receive commands after the wallet has been unlocked.

Macaroons and TLS support

With falafel v0.6 the proper macaroon and TLS certificate is needed also when using the in-memory gRPC connections (which is done under the hood in the mobile build). This has no external facing changes, however you might have to delete the existing TLS certificate so it can be re-created. This paves the way for using custom macaroons also on mobile, and moves the mobile API closer to the regular gRPC API.

Docker

The docker quick start files located in the docker directory have been made more user friendly by adding a persistent volume to lnd. This allows the lnd container to be restarted without losing its user data. The build instructions in docker/lnd/Dockerfile now also allow for quicker rebuilds by making use of the docker build cache.

The main Dockerfile in the root directory takes a branch, label or commit as a build argument to allow for custom builds.

The use of the docker-compose file is now easier and you can spin a btcd and lnd pair with just one command.
Note: If you've used the docker-compose.yml file before, make sure you run docker-compose build --no-cache the first time you use to new version to make sure the changes are taken into account.

Config Changes

Default unsafe-disconnect Setting and Deprecation

This option is deprecated and now defaults to true, allowing users to disconnect peers with active channels via the disconnectpeer rpc. This option was originally added long ago for purposes of testing and off by default to prevent unintended consistency bugs. This extra caution has been unneeded for several releases and no longer required. Once unsafe-disconnect is fully removed, this will remain the default behavior.

Staged Travis Builds

The travis build matrix now does preliminary sanity checks before entering a second phase that executes long-running tests. This results in less wasteful usage of travis instances, and tightens the feedback loop for developers contributing to the project.

Bug fixes

  • Accidental modification of database migration code has caused issues in the past. In this release, all migration code was isolated to prevent this from happening again.

  • Several fixes have been applied to the exchange of commitment signatures. Empty commit signatures are no longer sent.

  • Forwarding in lnd is "non-strict", meaning that any channel that is capable of carrying the payment may be utilized. The channel specified in the payload merely serves as a short-hand notation for the next node pub key. A bug was fixed where an incorrect failure message was returned if none of the channels qualified.

  • Expired invoices are now canceled correctly. This fixes the "Invoices can be paid after expiration" bug.

  • A bug in the chain notifier has been fixed that led to a panic when paying a subscribed script hash multiple times in different blocks.

Changelog

The full list of changes since v0.8.2-beta can be found here:

Contributors (Alphabetical Order)

Alex Bosworth
Andras Banki-Horvath
Andreas M. Antonopoulos
Anton Kovalenko
Arik Sosman
bitromortac
Bjarne Magnussen
bluetegu
Carla Kirk-Cohen
Carson Mullins
Conner Fromknecht
Daniel McNally
Dennis Reimann
Elle Mouton
Johan T. Halseth
Joost Jager
Juan Pablo Civile
kiwiidb
Lars Lehtonen
Matheus Degiovani
Eugene Siegel
Olaoluwa Osuntokun
Oliver Gugger
Otto Suess
Philipp Gillé
Roei Erez
Steven Roose
Tomas Carnecky
Vignesh Karthikeyan
Wilmer Paulino
Yancy Tibbens
Yan Pritzker

Assets 37
Pre-release
Pre-release

@joostjager joostjager released this Jan 15, 2020 · 43 commits to master since this release

v0.9.0-beta-rc1
Assets 37

@Roasbeef Roasbeef released this Dec 12, 2019 · 696 commits to master since this release

This release marks the second minor release in the v0.8.x series and contains no major new features. This release includes a number of fixes for SCB (Static Channel Backup) recovery, additional logging in the ChainNotifier, and a consistency fix to the SendToRoute RPC call.

Verifying the Release

In order to verify the release, you'll need to have gpg or gpg2 installed on your system. Once you've obtained a copy (and hopefully verified that as well), you'll first need to import the keys that have signed this release if you haven't done so already:

curl https://keybase.io/roasbeef/pgp_keys.asc | gpg --import

Once you have the required PGP keys, you can verify the release (assuming manifest-v0.8.2-beta.txt and manifest-v0.8.2-beta.txt.sig are in the current directory) with:

gpg --verify manifest-v0.8.2-beta.txt.sig

You should see the following if the verification was successful:

gpg: assuming signed data in 'manifest-v0.8.2-beta.txt'
gpg: Signature made Wed Dec 11 17:18:41 2019 PST
gpg:                using RSA key 4AB7F8DA6FAEBB3B70B1F903BC13F65E2DC84465
gpg: Good signature from "Olaoluwa Osuntokun <laolu32@gmail.com>" [ultimate]

That will verify the signature of the manifest file, which ensures integrity and authenticity of the archive you've downloaded locally containing the binaries. Next, depending on your operating system, you should then re-compute the sha256 hash of the archive with shasum -a 256 <filename>, compare it with the corresponding one in the manifest file, and ensure they match exactly.

Verifying the Release Binaries

As of this release, our release binaries are fully reproducible thanks to go1.13! Third parties are now able to verify that the release binaries were produced properly without having to trust the release manager(s). See our reproducible builds guide for how this can be achieved. The release binaries are compiled with go1.13.4, which is required by verifiers to arrive at the same ones. They include the following build tags: autopilotrpc, signrpc, walletrpc, chainrpc, invoicesrpc, routerrpc, and watchtowerrpc. Note that these are already included in the release script, so they do not need to be provided.

Finally, you can also verify the tag itself with the following command:

git verify-tag v0.8.2-beta

Building the Contained Release

Users are able to rebuild the target release themselves without having to fetch any of the dependencies. In order to do so, assuming that vendor.tar.gz and lnd-source-v0.8.2-beta.tar.gz are in the current directory, follow these steps:

tar -xvzf vendor.tar.gz
tar -xvzf lnd-source-v0.8.2-beta.tar.gz
GO111MODULE=on go install -v -mod=vendor -ldflags "-X github.com/lightningnetwork/lnd/build.Commit=v0.8.2-beta" ./cmd/lnd
GO111MODULE=on go install -v -mod=vendor -ldflags "-X github.com/lightningnetwork/lnd/build.Commit=v0.8.2-beta" ./cmd/lncli

The -mod=vendor flag tells the go build command that it doesn't need to fetch the dependencies, and instead, they're all enclosed in the local vendor directory.

Additionally, it's now possible to use the enclosed release.sh script to bundle a release for a specific system like so:

LNDBUILDSYS="linux-arm64 darwin-amd64" ./build/release/release.sh

⚡️⚡️⚡️ OK, now to the rest of the release notes! ⚡️⚡️⚡️

Release Notes

It's now possible to use an SCB to recover the funds of an unconfirmed channel. Before this fix, if one attempted to recover an SCB for a channel which was unconfirmed at . the time of SCB creation, then the process would fail. We'll node encode a special short channel ID to detect this case to ensure the channel funds can be recovered as normal.

We'll now warn a user that is using the lncli unlock or lncli create calls to recover an SCB. The new warning message now makes it more clear that in order to recover the funds of the channel, we need to close the channel on chain, resulting in additional chain fees.

If some or all of the channels in a Multi SCB backup have already been received, then we'll no longer exit early. Instead, this case will be detected allowing a user to re-issue an SCB recovery CLI or RPC command.

The SendToRoute RPC call has been updated to properly check the current max payment size parameter of the registered router backend, just as all the other payment RPC calls do.

Changelog

The full list of changes since v0.8.1-beta can be found here:

Contributors (Alphabetical Order)

Andreas M. Antonopoulos
Johan T. Halseth
Olaoluwa Osuntokun
Oliver Gugger

Assets 37
Pre-release
Pre-release

@Roasbeef Roasbeef released this Dec 7, 2019 · 696 commits to master since this release

lnd v0.8.2-beta-rc2
Assets 2

@Roasbeef Roasbeef released this Dec 5, 2019 · 696 commits to master since this release

lnd v0.8.2-beta-rc1
Assets 2
Nov 18, 2019
cert: create independent go modules

@Roasbeef Roasbeef released this Nov 15, 2019 · 696 commits to master since this release

This release marks the first minor release in the v0.8.x series and contains no major new features. This release fixes an issue with building v0.8.0 for some users under a particular build configuration, and also updates lnd to be compatible with the upcoming bitcoind v0.19 release as well. Additionally, several bugs in projects that lnd depends on (btcd., neutrino, btcwallet) have been fixed. There are no database migrations in this release, so users are able to downgrade back to 0.8.0 if they choose to do so.

Verifying the Release

In order to verify the release, you'll need to have gpg or gpg2 installed on your system. Once you've obtained a copy (and hopefully verified that as well), you'll first need to import the keys that have signed this release if you haven't done so already:

curl https://keybase.io/roasbeef/pgp_keys.asc | gpg --import

Once you have the required PGP keys, you can verify the release (assuming manifest-v0.8.1-beta.txt and manifest-v0.8.1-beta.txt.sig are in the current directory) with:

gpg --verify manifest-v0.8.1-beta.txt.sig

You should see the following if the verification was successful:

gpg: assuming signed data in 'manifest-v0.8.1-beta.txt'
gpg: Signature made Thu Nov 14 17:08:07 2019 PST
gpg:                using RSA key 4AB7F8DA6FAEBB3B70B1F903BC13F65E2DC84465
gpg: Good signature from "Olaoluwa Osuntokun <laolu32@gmail.com>" [ultimate]

That will verify the signature of the manifest file, which ensures integrity and authenticity of the archive you've downloaded locally containing the binaries. Next, depending on your operating system, you should then re-compute the sha256 hash of the archive with shasum -a 256 <filename>, compare it with the corresponding one in the manifest file, and ensure they match exactly.

Verifying the Release Binaries

As of this release, our release binaries are fully reproducible thanks to go1.13! Third parties are now able to verify that the release binaries were produced properly without having to trust the release manager(s). See our reproducible builds guide for how this can be achieved. The release binaries are compiled with go1.13.4, which is required by verifiers to arrive at the same ones. They include the following build tags: autopilotrpc, signrpc, walletrpc, chainrpc, invoicesrpc, routerrpc, and watchtowerrpc. Note that these are already included in the release script, so they do not need to be provided.

Finally, you can also verify the tag itself with the following command:

git verify-tag v0.8.1-beta

Building the Contained Release

Users are able to rebuild the target release themselves without having to fetch any of the dependencies. In order to do so, assuming that vendor.tar.gz and lnd-source-v0.8.1-beta.tar.gz are in the current directory, follow these steps:

tar -xvzf vendor.tar.gz
tar -xvzf lnd-source-v0.8.1-beta.tar.gz
GO111MODULE=on go install -v -mod=vendor -ldflags "-X github.com/lightningnetwork/lnd/build.Commit=v0.8.1-beta" ./cmd/lnd
GO111MODULE=on go install -v -mod=vendor -ldflags "-X github.com/lightningnetwork/lnd/build.Commit=v0.8.1-beta" ./cmd/lncli

The -mod=vendor flag tells the go build command that it doesn't need to fetch the dependencies, and instead, they're all enclosed in the local vendor directory.

Additionally, it's now possible to use the enclosed release.sh script to bundle a release for a specific system like so:

LNDBUILDSYS="linux-arm64 darwin-amd64" ./build/release/release.sh

⚡️⚡️⚡️ OK, now to the rest of the release notes! ⚡️⚡️⚡️

Release Notes

This release fixes an issue with certain build configuration that could cause lnd v0.8.0 to not build using the latest version of go.

With this release, lnd is now compatible with bitcoin v0.19.

Changelog

The full list of changes since v0.8.1-beta can be found here:

Contributors (Alphabetical Order)

Wilmer Paulino

Assets 37

@Roasbeef Roasbeef released this Oct 15, 2019 · 696 commits to master since this release

Database Migrations

This release includes two migrations. The first migration upgrades existing payment attempts and results to support the new TLV onion format. The migration should look like this:

2019-09-26 15:37:31.794 [INF] CHDB: Checking for schema update: latest_version=11, min_upgrade_version=9, db_version=9
2019-09-26 15:37:31.794 [INF] CHDB: Performing database schema migration
2019-09-26 15:37:31.794 [INF] CHDB: Applying migration #10
2019-09-26 15:37:31.795 [INF] CHDB: Migration of route/hop serialization complete!
2019-09-26 15:37:31.795 [INF] CHDB: Migrating to new mission control store by clearing existing data
2019-09-26 15:37:31.795 [INF] CHDB: Migration to new mission control completed!

The second migration upgrades invoices to support tracking multiple HTLCs, rather than a single one. This increases the accuracy of invoice accounting within lnd across restarts. Note that you cannot update if you have any pending hodl invoice. The migration should look like this:

2019-09-26 15:37:31.795 [INF] CHDB: Applying migration #11
2019-09-26 15:37:31.795 [INF] CHDB: Migrating invoices to new invoice format
2019-09-26 15:37:31.795 [INF] CHDB: Migration of invoices completed!

Once these migrations succeed, it will not be possible to return to a prior version of lnd.

Verifying the Release

In order to verify the release, you'll need to have gpg or gpg2 installed on your system. Once you've obtained a copy (and hopefully verified that as well), you'll first need to import the keys that have signed this release if you haven't done so already:

curl https://keybase.io/roasbeef/pgp_keys.asc | gpg --import

Once you have the required PGP keys, you can verify the release (assuming manifest-v0.8.0-beta.txt and manifest-v0.8.0-beta.txt.sig are in the current directory) with:

gpg --verify manifest-v0.8.0-beta.txt.sig

You should see the following if the verification was successful:

gpg: assuming signed data in 'manifest-v0.8.0-beta.txt'
gpg: Signature made Tue Oct 22 15:45:18 2019 PDT
gpg:                using RSA key 4AB7F8DA6FAEBB3B70B1F903BC13F65E2DC84465
gpg: Good signature from "Olaoluwa Osuntokun <laolu32@gmail.com>" [ultimate]

That will verify the signature of the manifest file, which ensures integrity and authenticity of the archive you've downloaded locally containing the binaries. Next, depending on your operating system, you should then re-compute the sha256 hash of the archive with shasum -a 256 <filename>, compare it with the corresponding one in the manifest file, and ensure they match exactly.

Verifying the Release Binaries

As of this release, our release binaries are fully reproducible thanks to go1.13! Third parties are now able to verify that the release binaries were produced properly without having to trust the release manager(s). See our reproducible builds guide for how this can be achieved. The release binaries are compiled with go1.13.3, which is required by verifiers to arrive at the same ones. They include the following build tags: autopilotrpc, signrpc, walletrpc, chainrpc, invoicesrpc, routerrpc, and watchtowerrpc. Note that these are already included in the release script, so they do not need to be provided.

Finally, you can also verify the tag itself with the following command:

git verify-tag v0.8.0-beta

Building the Contained Release

Users are able to rebuild the target release themselves without having to fetch any of the dependencies. In order to do so, assuming that vendor.tar.gz and lnd-source-v0.8.0-beta.tar.gz are in the current directory, follow these steps:

tar -xvzf vendor.tar.gz
tar -xvzf lnd-source-v0.8.0-beta.tar.gz
GO111MODULE=on go install -v -mod=vendor -ldflags "-X github.com/lightningnetwork/lnd/build.Commit=v0.8.0-beta" ./cmd/lnd
GO111MODULE=on go install -v -mod=vendor -ldflags "-X github.com/lightningnetwork/lnd/build.Commit=v0.8.0-beta" ./cmd/lncli

The -mod=vendor flag tells the go build command that it doesn't need to fetch the dependencies, and instead, they're all enclosed in the local vendor directory.

Additionally, it's now possible to use the enclosed release.sh script to bundle a release for a specific system like so:

LNDBUILDSYS="linux-arm64 darwin-amd64" ./build/release/release.sh

⚡️⚡️⚡️ OK, now to the rest of the release notes! ⚡️⚡️⚡️

Release Notes

Protocol Upgrades

Multi-Frame TLV Based Onion

In this release of lnd, we’ll now by default signal adherence to the new multi-frame TLV based onion format used when routing HTLCs end to end within the network. Compared to the legacy format, this format is much more flexible as rather than using a handpacked encoding for each field, we’ll now transmit TLV (type-length-value) streams, which is essentially a key-value map. As a result, the onion format is now far more extensible, as we can easily add new types without triggering a network wide update. In many instances (such as AMP), only the sender and receiver need to know of the new information in the onion, streamlining upgrade paths.

This change isn’t exposed to developers yet, but will be in future versions, allowing them to send a small amount of bytes to the receiver alongside a payment. The largest impact of this new feature is to lay the groundwork for the AMP/MPP work that has begun. In the case of AMP, this new namespace in the onion will be used for things like sending a partial payment shard, the total amount of the unified set of payments, and so on. We’ll also leverage this space to roll out our spontaneous payment feature (keysend) in a future version of lnd.

Safu Commitments (option_static_remote_key)

Another major protocol update that has shipped in this new version of lnd is something we call “safu commitments”. This new commitment format represents a large step in making Lightning safer to use for end users. Before this change, if the remote party forced closed the channel on-chain, in order to sweep your funds (in the case of partial data loss, but SCB existence), then the party that lost data would need to obtain a random nonce from the other party in order to sweep their funds. This dependency at times made recovery very difficult, or even indeterminate, if the party that lost data was unable to obtain this per-state nonce from the remote party.

With this new commitment format, we’ve instead removed this nonce derivation from the commitment output. Instead, when the remote party force closes, the funds will go directly into one’s wallet. As a result, the only thing required to recover funds in the case of partial data loss now are one’s SCB and the on-chain event of the channel being confirmed on chain. A new SCB version (v1) has been added to signal if the channel being backed up uses the old or new commitment format, allowing us to keep our recovery flow identical to the current one.

It’s important to note that in order to use this new commitment format, new channels must be opened. The SCBs stored on disk, as well as those that are sent over the various RPC calls, will be automatically updated to reflect if this new commitment is being used in the channels being backed up. A new field has been added to the ListChannels output (static_remote_key) that indicates if the channel is using the new safu commitment format or not.

TLV Serialization Library

This release also includes a new TLV serialization library, which implements the TLV format described in BOLT 01. The tlv package produces encodings that are both forwards and backwards compatible, and will form the basis of making wire messages and onion payloads easily extensible fields for new features like MPP, AMP, or extended gossip queries.

BOLT 11 Feature Bits

In 0.8.0, lnd now supports the ability to generate and parse invoices with payment-specific feature bits. At the moment there are no such feature bits that are advertised, however this is a preliminary step in order to widely deploy the parsing logic for when this happens. The first candidate features for being advertised are likely for MPP or AMP, which can inform the sender as to whether they may pay a particular invoice using those payment methods. In theory, the sender can also require that an invoice be fulfilled with a particular payment type.

Indefinite Channel Reestablishment

Prior to 0.8.0, lnd would give the remote peer 30 seconds to send a their channel_reestablish message. Typically this duration was sufficient, however if the exchange was not completed in a timely manner lnd would incorrectly escalate this into a protocol error, and cause strict implementations to force close the channel.

In one instance with a c-lightning node, this resulted in 40 channels being force closed, since the contention at startup prevented the remote node from promptly sending channel_reestablish. In 0.7.2, c-lightning also added mitigations to avoid this situation, by spacing out reconnections to peers on startup and ignoring the escalated errors.

In 0.8.0, lnd will wait indefinitely over the span of a connection for the remote peer to send channel_reestablish. As a result, these sporadic failures should not occur between updated nodes. Further, the active flag will in ListChannels is stricter and more accurate, which now only displays true if the channel is sufficiently confirmed, has ever received funding_locked, and has received channel_reestablish on the current connection. Previously the active boolean would display true whenever the first two conditions had been met.

Chain Notifier Subserver

Subscriptions for confirmation and spend notifications now require a height hint to be provided, also known as the earliest height in the chain at which the event could have happened. This prevents lnd from scanning blocks that are irrelevant to the subscription. Subscriptions for confirmation notifications now also require at least one confirmation. Previously, subscriptions without providing a number of confirmations would lead lnd to deadlock.

Watchtower Client Subserver

Continuing our subserver saga, this release includes a new addition: the Watchtower Client subserver. This is the first subserver that is included in lnd by default and does not require a build tag.

The subserver includes the following RPCs, along with a lncli command for each:

  • AddTower (lncli wtclient add)
  • RemoveTower (lncli wtclient remove)
  • ListTowers (lncli wtclient towers)
  • GetTowerInfo (lncli wtclient tower)
  • Stats (lncli wtclient stats)
  • Policy (lncli wtclient policy)

These RPCs allow users to interact with the watchtower client of their lnd node, allowing for modifications and information retrieval of any registered watchtowers and the policies used with them.

Note that due to the introduction of this interface, the --wtclient.private-tower-uris flag has now been deprecated and will be removed in the next major release, v0.9.0-beta. All that is required for the client to be active is --wtclient.active, any setup that had been configured from before will carry over. If you're setting up a new watchtower or want to change the configuration, users will now need to so as stated in the watchtower documentation.

Gossip Enhancements

A rough graph sync progress is now exposed via the GetInfo RPC as a boolean named is_graph_synced.

Gossip received as part of an initial sync is no longer forwarded to other peers as it’s assumed that they have been previously broadcast to the network. This results in a small decrease in outbound bandwidth usage.

lnd now rebroadcasts its node announcement to the network every 24 hours to ensure nodes have up-to-date information about them.

A new --ignore-historical-gossip-filters option has been added in 0.8.0 to reduce outbound bandwidth usage in syncing older lightning nodes. Some older nodes send gossip timestamp filters that request large portions of the graph on each connection; this flag will still allow new gossip messages to be forwarded that satisfy the timestamp, but forgo the initial dump which can rack up outgoing bandwidth.

Routing

Mission control, the subsystem in lnd that drives the payment process and tracks past node performance, has been made persistent. It now retains historical payment results across restarts.

Further improvements have been made to mission control that make the payment process faster and more reliable. Routing nodes forward in a non-strict matter, meaning that if the requested channel isn’t available, the node may forward an HTLC over an alternative channel to the same next hop. Therefore, it is enough to try a single channel for each node pair. To speed up payments, we now track node performance based on node pairs instead of channels.

When a payment attempt fails, the sender receives a failure message from which information can be distilled that helps identifying the cause of the failure. This information can be used to steer around the problem for the next attempt. The interpretation of the failure message has been improved.

Previously, mission control only recorded payment failures. This prevented hitting the same failure twice, but ignored previously successful routes. It didn’t distinguish between a route that worked before and an untried route. We now favor successful routes over untried routes and thereby reduce the number of required payment attempts. The explore-exploit trade-off that this presents is controlled by the existing routerrpc.attemptcost config parameter.

To enhance the transparency of the path finding process, the total route success probability is exposed on QueryRoutes calls.

A new RPC BuildRoute has been added. This call targets advanced users who want to build their own routes without needing to keep a shadow copy of the graph. The call takes a list of hop pubkeys and transforms this into a fully specified route by looking up channel policies from the graph. It also allows the construction of circular routes.

Routing Node Enhancements

Nodes can now specify the maximum outgoing CLTV delta (in number of blocks) they will accept for an outgoing HTLC through the --max-cltv-expiry CLI flag.

Nodes will now enforce a maximum channel commitment fee for channels in which they are initiators when attempting to update their commitment fee. The maximum enforced depends on a percentage (default of 50%) of a channel initiator’s balance. The percentage can be modified through the --max-channel-fee-allocation CLI flag.

The max_htlc channel policy parameter is now exposed on the RPC interface. This allows callers of UpdateChannelPolicy to reduce the maximum htlc amount that is forwarded over a channel.

RPC Bug Fixes

The RemoteBalance, LocalChanReserveSat, and RemoteChanReserveSat fields for waiting close channels are now properly set. These weren’t set previously, leading to the fields being blank until the channel moved to pending close.

Previously, using the UpdateChannelPolicy RPC with a zero base fee and/or fee rate would lead to a desynchronization between the routing policy propagated throughout the network and the one within the channel state machine, leading to routing failures throughout the network. To work around this, users were required to restart their nodes, but this is no longer required as of #3439.

The REST endpoint for the DescribeGraph RPC call (v1/graph) has been updated to allow a larger response size due to the growing channel graph.

A bug in the ordering of channel updates for getnodeinfo has been fixed that could cause the updates not to be sorted in order of increasing pubkey. Additionally, the last_update time displayed will now properly compute the maximum of the each update's last_update instead of inconsistently selecting one or the other.

Users who set the perm field (or --perm for lncli) prior to 0.8.0 may have noticed that lnd would not automatically reconnect to peers if the connection was severed. This release fixes the bug by tracking whether or not the current connection was requested as permanent via the RPC, and restoring the original functionality of the ConnectPeer RPC.

lncli Bug Fixes

lncli now returns a proper error rather than panic when providing a channel backup to the restorechanbackup command without its corresponding flag.

Invoice Validation

To mitigate potential DoS vectors when parsing invoice payment requests, any larger than the maximum size a QR code is allowed to store are now considered invalid.

Invoice HTLC tracking

Calls to LookupInvoice and ListInvoices now return a list of all HTLCs that pay to an invoice. This provides insight into the channel through which the invoice was paid, as well as acceptance time, resolution time, and the exact HTLC amount. The handling of HTLCs internally has also been improved, which fixed several existing consistency and accounting issues.

Privacy

A potential probe vector that allowed attackers to find out the final destination of a payment via the final_expiry_too_soon response has been eliminated. To not deprive the payer of information that is required for continuation of the payment process, the height at which the receiver accepted the HTLC is added to the failure message.

New OS/Arch Release Targets

0.8.0 will provide compiled binaries for 11 new architectures: illumos-amd64, linux-ppc64le, linux-mips, linux-mipsle, linux-s390x, netbsd-arm, netbsd-arm64, openbsd-arm, openbsd-arm64, solaris-amd64, and windows-arm. The illumos-amd64 and solaris-amd64 builds were newly enabled by go1.13 while the others had been previously available but not included in the release targets. If you're interested in a target that is not listed, please file an issue!

bitcoind Compatibility

The lnd integration test suite is now also continuously being run with bitcoind as the chain backend. All supported backends (btcd, neutrino, and bitcoind) now enjoy the same test coverage, which ensures lnd will behave the same regardless of which backend users choose. As of this release, lnd is fully compatible with bitcoind versions up to v0.18.1.

Mobile Support

The mobile APIs and build tools have been included in this release. This allows developers to start building mobile apps which integrate lnd, either by checking out the latest release, or the master branch.

RBF Aware Transaction Broadcast

The wallet has been updated to understand RBF specific errors. For now, this handles a few edge cases with non-critical broadcast errors. In the future, this will pave the way for direct fee bumping through RBF in the wallet.

Channel Close Transaction Rebroadcast

To ensure proper confirmation of a channel close transaction, lnd has been updated to rebroadcast it after restarts. It also now also aids remote nodes that have lost state by resending channel reestablishment messages for already closed channels.

Mainnet Neutrino

Neutrino has steadily been improved, and this release contains several fixes and optimizations that make it stable enough for mainnet support. Keep in mind that it is still early, and stay craeful.

Changelog

The full list of changes since v0.7.1-beta can be found here:

Contributors (Alphabetical Order)

Carla Kirk-Cohen
chokoboko
Christopher Coverdale
Conner Fromknecht
Dario Sneidermanis
Eugene
Fernando Guisso
fiatjaf
Hampus Sjöberg
Johan T. Halseth
Jonathan Cross
Joost Jager
Juan Pablo Civile
Lars Lehtonen
lieteau2
Lightning Koala
Matheus Degiovani
Olaoluwa Osuntokun
Oliver Gugger
openoms
Oskar F
Spencer Dupre
Vadym Popov
Valentine Wallace
Will Roscoe
Wilmer Paulino

Assets 37

@Roasbeef Roasbeef released this Oct 15, 2019 · 698 commits to master since this release

lnd v0.8.0-beta-rc3
Assets 2
You can’t perform that action at this time.