Skip to content

Commit

Permalink
docs: add new recovery.md explaning methods of fund recovery
Browse files Browse the repository at this point in the history
  • Loading branch information
@Roasbeef
Roasbeef committed Apr 11, 2019
1 parent f9a0804 commit 68da205
Showing 1 changed file with 361 additions and 0 deletions.
361 changes: 361 additions & 0 deletions docs/recovery.md
View file
@@ -0,0 +1,361 @@
# Table of Contents

* [Recovering Funds From `lnd` (funds are safu!)](#recovering-funds-from-lnd-funds-are-safu)
* [On-Chain Recovery](#on-chain-recovery)
* [24-word Cipher Seeds](#24-word-cipher-seeds)
* [Wallet+Seed Passphrases](#walletseed-passphrases)
* [Starting On-Chain Recovery](#starting-on-chain-recovery)
* [Forced In-Place Rescan](#force-in-place-rescan)
* [Off-Chain Recovery](#off-chain-recovery)
* [Obtaining SCBs](#obtaining-scbs)
* [On-Disk `channel.backup`](#on-disk-channelbackup)
* [Using the `ExportChanBackup` RPC](#using-the-exportchanbackup-rpc)
* [Streaming Updates via `SubscribeChannelBackups`.](#streaming-updates-via-subscribechannelbackups)
* [Recovering Using SCBs](#recovering-using-scbs)

# Recovering Funds From `lnd` (funds are safu!)

In this document, we'll go over the various built-in mechanisms for recovering
funds from `lnd` due to any sort of data loss, or malfunction. Coins in `lnd`
can exist in one of two pools: on-chain or off-chain. On-chain funds are
outputs under the control of `lnd` that can be spent immediately, and without
any auxiliary data. Off-chain funds on the other hand exist within a 2-of-2
multi-sig output typically referred to as a payment channel. Depending on the
exact nature of operation of a given `lnd` node, one of these pools of funds
may be empty.

Fund recovery for `lnd` will require two pieces of data:
1. Your 24-word cipher seed
2. Your encrypted Static Channel Backup file (or the raw data)

If one is only attempting to recover _on chain_ funds, then only the first item
is required.

The SCB file is encrypted using a key _derived_ from the user's seed. As a
result, it cannot be used in isolation.

## On-Chain Recovery

### 24-word Cipher Seeds

When a new `lnd` node is created, it's given a 24-word seed phrase, called an
[`cipher seed`](https://github.com/lightningnetwork/lnd/tree/master/aezeed).
The two seed formats look similar, but the only commonality they share are
using the same default English dictionary. A valid seed phrase obtained over
the CLI `lncli create` command looks something like:
```
!!!YOU MUST WRITE DOWN THIS SEED TO BE ABLE TO RESTORE THE WALLET!!!
---------------BEGIN LND CIPHER SEED---------------
1. ability 2. noise 3. lift 4. document
5. certain 6. month 7. shoot 8. perfect
9. matrix 10. mango 11. excess 12. turkey
13. river 14. pitch 15. fluid 16. rack
17. drill 18. text 19. buddy 20. pool
21. soul 22. fatal 23. ship 24. jelly
---------------END LND CIPHER SEED-----------------
!!!YOU MUST WRITE DOWN THIS SEED TO BE ABLE TO RESTORE THE WALLET!!!
```

### Wallet Seed Passphrases

During the creation process, users are first prompted to enter a **wallet
password**:
```
Input wallet password:
Confirm wallet password:
```

This password is used to _encrypt_ the wallet on disk, which includes any
derived master private keys or public key data.

Users can also _optionally_ enter a second passphrase which we call the _cipher
seed passphrase_:
```
Your cipher seed can optionally be encrypted.
Input your passphrase if you wish to encrypt it (or press enter to proceed without a cipher seed passphrase):
```

If specified, then this will be used to encrypt the cipher seed itself. The
cipher seed format is unique in that the 24-word phrase is actually a
_ciphertext_. As a result, there's no standard word list as any arbitrary
encoding can be used. If a passphrase is specified, then the cipher seed you
write down is actually an _encryption_ of the entropy used to generate the BIP
32 root key for the wallet. Unlike a BIP 39 24-word phrase, the cipher seed is
able to _detect_ incorrect passphrase. BIP 39 on the other hand, will instead
silently decrypt to a new (likely empty) wallet.

### Starting On-Chain Recovery

The initial entry point to trigger recovery of on-chain funds in the command
line is the `lncli create` command.
```
⛰ lncli create
```

Next, one can enter a _new_ wallet password to encrypt any newly derived keys
as a result of the recovery process.
```
Input wallet password:
Confirm wallet password:
```

Once a new wallet password has been obtained, the user will be prompted for
their _existing_ cipher seed:
```
Input your 24-word mnemonic separated by spaces: ability noise lift document certain month shoot perfect matrix mango excess turkey river pitch fluid rack drill text buddy pool soul fatal ship jelly
```

If a _cipher seed passphrase_ was used when the seed was created, it MUST be entered now:
```
Input your cipher seed passphrase (press enter if your seed doesn't have a passphrase):
```

Finally, the user has an option to choose a _recovery window_:
```
Input an optional address look-ahead used to scan for used keys (default 2500):
```

The recovery window is a metric that the on-chain rescanner will use to
determine when all the "used" addresses have been found. If the recovery window
is two, lnd will fail to find funds in any addresses generated after the point
in which two consecutive addresses were generated but never used." If an `lnd`
on-chain wallet was extensively used, then users may want to _increase_ the
default value.

If all the information provided was valid, then you'll be presented with the
seed again:
```
!!!YOU MUST WRITE DOWN THIS SEED TO BE ABLE TO RESTORE THE WALLET!!!
---------------BEGIN LND CIPHER SEED---------------
1. ability 2. noise 3. lift 4. document
5. certain 6. month 7. shoot 8. perfect
9. matrix 10. mango 11. excess 12. turkey
13. river 14. pitch 15. fluid 16. rack
17. drill 18. text 19. buddy 20. pool
21. soul 22. fatal 23. ship 24. jelly
---------------END LND CIPHER SEED-----------------
!!!YOU MUST WRITE DOWN THIS SEED TO BE ABLE TO RESTORE THE WALLET!!!
lnd successfully initialized!
```

In `lnd`'s logs, you should see something along the lines of (irrelevant lines skipped):
```
[INF] LNWL: Opened wallet
[INF] LTND: Wallet recovery mode enabled with address lookahead of 2500 addresses
[INF] LNWL: RECOVERY MODE ENABLED -- rescanning for used addresses with recovery_window=2500
[INF] CHBU: Updating backup file at test_lnd3/data/chain/bitcoin/simnet/channel.backup
[INF] CHBU: Swapping old multi backup file from test_lnd3/data/chain/bitcoin/simnet/temp-dont-use.backup to test_lnd3/data/chain/bitcoin/simnet/channel.backup
[INF] LNWL: Seed birthday surpassed, starting recovery of wallet from height=748 hash=3032830c812a4a6ea305d8ead13b52e9e69d6400ff3c997970b6f76fbc770920 with recovery-window=2500
[INF] LNWL: Scanning 1 blocks for recoverable addresses
[INF] LNWL: Recovered addresses from blocks 748-748
[INF] LNWL: Started rescan from block 3032830c812a4a6ea305d8ead13b52e9e69d6400ff3c997970b6f76fbc770920 (height 748) for 0 addresses
[INF] LNWL: Catching up block hashes to height 748, this might take a while
[INF] LNWL: Done catching up block hashes
[INF] LNWL: Finished rescan for 800 addresses (synced to block 3032830c812a4a6ea305d8ead13b52e9e69d6400ff3c997970b6f76fbc770920, height 748)
```

That final line indicates the rescan is complete! If not all funds have
appeared, then the user may need to _repeat_ the process with a higher recovery
window. Depending on how old the wallet is (the cipher seed stores the wallet's
birthday!) and how many addresses were used, the rescan may take anywhere from
a few minutes to a few hours.

If the rescan wasn't able to complete fully (`lnd` was shutdown for example),
then from `lncli unlock`, it's possible to _restart_ the rescan from where it
left off with the `--recovery-window` argument:
```
⛰ lncli unlock --recovery_window=2500
```

Note that if this argument is specified, then the wallet will not _re-enter_
the recovery mode and may miss funds during the portion of the rescan.

### Forced In-Place Rescan

The recovery methods described above assume a clean slate for a node, so
there's no existing UTXO or key data in the node's database. However, there're
times when an _existing_ node may want to _manually_ rescan the chain. We have
a tool for that! The tool is called
[`dropwtxmgr`](https://github.com/btcsuite/btcwallet/tree/master/cmd/dropwtxmgr).
It can be installed with the following command:
```
go get -v -u github.com/btcsuite/btcwallet/cmd/dropwtxmgr
```

The `dropwtxmgr` tool will _reset_ the best synced height of the wallet back to
its birthday, or genesis if the birthday isn't known (for some older wallets).
In order to run the tool, you must first **shutdown `lnd`**. Once `lnd` is
shutdown, the rescan can be initiated with the following commands:
```
cp $HOME/.lnd/data/chain/bitcoin/mainnet/wallet.db $HOME/wallet.db # Copy the existing databse just in case!
dropwtxmgr --db=$HOME/.lnd/data/chain/bitcoin/mainnet/wallet.db
```

Once the above command returns (if it hangs for a while, then `lnd` may not
actually be shutdown, so double check!), `lnd` can be restarted. After it's
restarted, then the wallet should being rescanning. An entry resembling the
following will show up in the logs once it's complete:
```
[INF] LNWL: Finished rescan for 800 addresses (synced to block 3032830c812a4a6ea305d8ead13b52e9e69d6400ff3c997970b6f76fbc770920, height 748)
```

## Off-Chain Recovery

After version `v0.6-beta` of `lnd`, the daemon now ships with a new feature
called Static Channel Backups (SCBs). We call these _static_ as they only need
to be obtained _once_: when the channel is created. From there on, a backup is
good until the channel is closed. The backup contains all the information we
need to initiate the Data Loss Protection (DLP) feature in the protocol, which
ultimately leads to us recovering the funds from the channel _on-chain_. This
is a foolproof _safe_ backup mechanism.

We say _safe_, as care has been taken to ensure that there are no foot guns in
this method of backing up channels, vs doing things like `rsync`ing or copying
the `channel.db` file periodically. Those methods can be dangerous as one never
knows if they have the latest state of a channel or not. Instead, we aim to
provide a simple, safe method to allow users to recover the settled funds in
their channels in the case of partial or complete data loss. The backups
themselves are encrypted using a key derived from the user's seed, this way we
protect privacy of the users channels in the back up state, and ensure that a
random node can't attempt to import another user's channels.

Given a valid SCB, the user will be able to recover _both_ funds that are fully
settled within their channels. By "fully settled" we mean funds that are in the
base commitment outputs, and not HTLCs. We can only restore these funds as
right after the channel is created, as we have all the data required to make a
backup, but lack information about the future HTLCs that the channel will
process.

### Obtaining SCBs

#### On-Disk `channel.backup`

There are multiple ways of obtaining SCBs from `lnd`. The most commonly used
method will likely be via the `channels.backup` file that's stored on-disk
alongside the rest of the chain data. This is a special file that contains SCB
entries for _all_ currently open channels. Each time a channel is opened or
closed, this file is updated on disk in a safe manner (atomic file rename). As
a result, unlike the `channel.db` file, it's _always_ safe to copy this file
for backup at ones desired location. The default location on Linux is:
```
~/.lnd/data/chain/bitcoin/mainnet/channel.backup
```

An example of using file system level notification to [copy the backup to a
distinct volume/partition/drive can be found
here](https://gist.github.com/alexbosworth/2c5e185aedbdac45a03655b709e255a3).

#### Using the `ExportChanBackup` RPC

Another way to obtain SCBS for all or a target channel is via the new
`exportchanbackup` `lncli` command:
```
⛰ lncli --network=simnet exportchanbackup --chan_point=29be6d259dc71ebdf0a3a0e83b240eda78f9023d8aeaae13c89250c7e59467d5:0
{
"chan_point": "29be6d259dc71ebdf0a3a0e83b240eda78f9023d8aeaae13c89250c7e59467d5:0",
"chan_backup": "02e7b423c8cf11038354732e9696caff9d5ac9720440f70a50ca2b9fcef5d873c8e64d53bdadfe208a86c96c7f31dc4eb370a02631bb02dce6611c435753a0c1f86c9f5b99006457f0dc7ee4a1c19e0d31a1036941d65717a50136c877d66ec80bb8f3e67cee8d9a5cb3f4081c3817cd830a8d0cf851c1f1e03fee35d790e42d98df5b24e07e6d9d9a46a16352e9b44ad412571c903a532017a5bc1ffe1369c123e1e17e1e4d52cc32329aa205d73d57f846389a6e446f612eeb2dcc346e4590f59a4c533f216ee44f09c1d2298b7d6c"
}
⛰ lncli --network=simnet exportchanbackup --all
{
"chan_points": [
"29be6d259dc71ebdf0a3a0e83b240eda78f9023d8aeaae13c89250c7e59467d5:0"
],
"multi_chan_backup": "fd73e992e5133aa085c8e45548e0189c411c8cfe42e902b0ee2dec528a18fb472c3375447868ffced0d4812125e4361d667b7e6a18b2357643e09bbe7e9110c6b28d74f4f55e7c29e92419b52509e5c367cf2d977b670a2ff7560f5fe24021d246abe30542e6c6e3aa52f903453c3a2389af918249dbdb5f1199aaecf4931c0366592165b10bdd58eaf706d6df02a39d9323a0c65260ffcc84776f2705e4942d89e4dbefa11c693027002c35582d56e295dcf74d27e90873699657337696b32c05c8014911a7ec8eb03bdbe526fe658be8abdf50ab12c4fec9ddeefc489cf817721c8e541d28fbe71e32137b5ea066a9f4e19814deedeb360def90eff2965570aab5fedd0ebfcd783ce3289360953680ac084b2e988c9cbd0912da400861467d7bb5ad4b42a95c2d541653e805cbfc84da401baf096fba43300358421ae1b43fd25f3289c8c73489977592f75bc9f73781f41718a752ab325b70c8eb2011c5d979f6efc7a76e16492566e43d94dbd42698eb06ff8ad4fd3f2baabafded"
}
⛰ lncli --network=simnet exportchanbackup --all --output_file=channels.backup
```

As shown above, a user can either: specify a specific channel to backup, backup
all existing channels, or backup directly to an on-disk file. All backups use
the same format.

#### Streaming Updates via `SubscribeChannelBackups`

Using the gRPC interace directly, [a new call:
`SubscribeChannelBackups`](https://api.lightning.community/#subscribechannelbackups).
This call allows users to receive a new notification each time the underlying
SCB state changes. This can be used to implement implement more complex backup
schemes, compared to the file system notification based approach.

### Recovering Using SCBs

If a node is being created from scratch, then it's possible to pass in an
existing SCB using the `lncli create` or `lncli unlock` commands:
```
⛰ lncli create -multi_file=channels.backup
```

Alternatively, the `restorechanbackup` command can be used if `lnd` has already
been created at the time of SCB restoration:
```
⛰ lncli restorechanbackup -h
NAME:
lncli restorechanbackup - Restore an existing single or multi-channel static channel backup
USAGE:
lncli restorechanbackup [command options] [--single_backup] [--multi_backup] [--multi_file=]
CATEGORY:
Channels
DESCRIPTION:
Allows a user to restore a Static Channel Backup (SCB) that was
obtained either via the exportchanbackup command, or from lnd's
automatically manged channels.backup file. This command should be used
if a user is attempting to restore a channel due to data loss on a
running node restored with the same seed as the node that created the
channel. If successful, this command will allows the user to recover
the settled funds stored in the recovered channels.
The command will accept backups in one of three forms:
* A single channel packed SCB, which can be obtained from
exportchanbackup. This should be passed in hex encoded format.
* A packed multi-channel SCB, which couples several individual
static channel backups in single blob.
* A file path which points to a packed multi-channel backup within a
file, using the same format that lnd does in its channels.backup
file.
OPTIONS:
--single_backup value a hex encoded single channel backup obtained from exportchanbackup
--multi_backup value a hex encoded multi-channel backup obtained from exportchanbackup
--multi_file value the path to a multi-channel back up file
```

Once the process has been initiated, `lnd` will proceed to:

1. Given the set of channels to recover, the server will then will insert a
series of "channel shells" into the database. These contain only the
information required to initiate the DLP (data loss protection) protocol
and nothing more. As a result, they're marked as "recovered" channels in
the database, and we'll disallow trying to use them for any other process.
2. Once the channel shell is recovered, the
[chanbackup](https://github.com/lightningnetwork/lnd/tree/master/chanbackup)
package will attempt to insert a LinkNode that contains all prior
addresses that we were able to reach the peer at. During the process,
we'll also insert the edge for that channel (only in the outgoing
direction) into the database as well.
3. lnd will then start up, and as usual attempt to establish connections to
all peers that we have channels open with. If `lnd` is already running,
then a new persistent connection attempt will be initiated.
4. Once we connect with a peer, we'll then initiate the DLP protocol. The
remote peer will discover that we've lost data, and then immediately force
close their channel. Before they do though, they'll send over the channel
reestablishment handshake message which contains the unrevoked commitment
point which we need to derive keys (will be fixed in
BOLT 1.1 by making the key static) to sweep our funds.
5. Once the commitment transaction confirms, given information within the SCB
we'll re-derive all keys we need, and then sweep the funds.

0 comments on commit 68da205

Please sign in to comment.