New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chanbackup: add new package implementing static channel backups #2370

Merged
merged 9 commits into from Jan 25, 2019

Conversation

Projects
None yet
2 participants
@Roasbeef
Copy link
Member

Roasbeef commented Dec 25, 2018

In this PR, we introduce a new package that implements our current scheme for static channel backups (SCB's). Note that the serialization format will likely change to use a TLV schema rather than a raw packed format. However the serialization is isolated to a single instance and can easily be modified when we decide on the final TLV format. As a result, this PR c an be reviewed as is.

Depends on #2369.

Static Channel Backup Scheme

Crypto

For encryption, we utilize chacha20poly1305 with a random 24 byte nonce. We use a larger nonce size as this can be safely generated via a CSPRNG without fear of frequency collisions between nonces generated. To encrypt a blob, we then use this nonce as the AD (associated data) and prepend the nonce to the front of the ciphertext package.

For key generation, in order to ensure the user only needs their passphrase and the backup file, we utilize the existing keychain to derive a private key. In order to ensure that at we don't force any hardware signer to be aware of our crypto operations, we instead opt to utilize a public key that will be hashed to derive our private key. The assumption here is that this key will only be exposed to this software, and never derived as a public facing address.

chanbackup.Single

The SCB contains all information required to initiate the data loss protection protocol once we restore the channel and connect to the remote channel peer.

The primary way outside callers will interact with this package are via the Pack and Unpack methods. Packing means writing a serialized+encrypted version of the SCB to an io.Writer. Unpacking does the opposite.

The encoding format itself uses the same encoding as we do on the wire within Lightning. Each encoded backup begins with a version so we can easily add or modify the serialization format in the future, if new channel types appear, or we need to add/remove fields. The backup contains:

  • The chain a channel belongs to.
  • The chanPoint of the channel.
  • The shortChanID of the channel.
  • The public key of the remote node.
  • The series of addresses that we can use to reach the node.
  • The CSV delay of the channel (required to later reconstruct our output script after BOLT 1.1)
  • A keychain.KeyLocator that allows us to re-derive the payment bas epoint we need to sweep our funds .
  • A keychain.KeyDescriptor that we need in order to re-derive our shachain root to validate the information the remote party gives us during the DLP protocol. (see the next section for the complications that arose here)

chanbackup.Multi

Multi is a series of static channel backups. This type of backup can contains ALL the channel
backup state in a single packed blob. This is suitable for storing on your file system, cloud storage, etc. Systems will be in place within lnd to ensure that one can easily obtain the latest version of the Multi for the node, and also that it will be kept up to date if channel state changes.

Broken off of #2313.

@Roasbeef Roasbeef force-pushed the Roasbeef:static-chan-backups-chanbackup branch from 652d509 to 1295005 Dec 25, 2018

@Roasbeef Roasbeef added this to the 0.6 milestone Dec 25, 2018

@cfromknecht
Copy link
Collaborator

cfromknecht left a comment

Nice work on this package! Overall looks pretty complete to me, with healthy dose of unit tests :) No major comments on the approach at large. I do think moving to a TLV style format in the future though would be pretty useful, which fortunately is enabled by the versions on both single and multi backups!

Linter and vet ar both failing btw: https://travis-ci.org/lightningnetwork/lnd/jobs/472003458
Maybe just needs a rebase now that #2369 is in?

Show resolved Hide resolved chanbackup/crypto.go Outdated
Show resolved Hide resolved chanbackup/crypto.go Outdated
Show resolved Hide resolved chanbackup/crypto.go Outdated
Show resolved Hide resolved chanbackup/crypto.go Outdated
Show resolved Hide resolved chanbackup/single.go Outdated
Show resolved Hide resolved chanbackup/backupfile.go Outdated
Show resolved Hide resolved chanbackup/backupfile.go Outdated
encryptionKey, err := genEncryptionKey(keyRing)
if err != nil {
return err
}

This comment has been minimized.

@cfromknecht

cfromknecht Jan 5, 2019

Collaborator

defer zeroing of encryption key?

This comment has been minimized.

@Roasbeef

Roasbeef Jan 16, 2019

Author Member

Do we even really have any guarantees for stuff like that given the existence of the runtime and GC? If not, then it would just be security theatre.

This comment has been minimized.

@cfromknecht

cfromknecht Jan 16, 2019

Collaborator

can it hurt? it's at worst a marginal improvement, having one less copy in memory. it looks like Sum256 will copy a static array with the final result back to the caller, so would say its likely that there's at least one other reference

encryptionKey, err := genEncryptionKey(keyRing)
if err != nil {
return nil, err
}

This comment has been minimized.

@cfromknecht

cfromknecht Jan 5, 2019

Collaborator

same here wrt zeroing encryption key

This comment has been minimized.

@Roasbeef

Roasbeef Jan 16, 2019

Author Member

See above.

Show resolved Hide resolved chanbackup/pubsub.go Outdated
@cfromknecht

This comment has been minimized.

Copy link
Collaborator

cfromknecht commented Jan 15, 2019

@Roasbeef needs rebase

@Roasbeef Roasbeef force-pushed the Roasbeef:static-chan-backups-chanbackup branch from 1295005 to be84a5d Jan 16, 2019

@Roasbeef

This comment has been minimized.

Copy link
Member Author

Roasbeef commented Jan 16, 2019

Comments responded to and rebased! PTAL @cfromknecht

@Roasbeef

This comment has been minimized.

Copy link
Member Author

Roasbeef commented Jan 19, 2019

@cfromknecht ping!

@cfromknecht
Copy link
Collaborator

cfromknecht left a comment

@Roasbeef another pass complete, almost ready. excellent coverage on this package with the extensive tests!

Show resolved Hide resolved chanbackup/crypto_test.go Outdated
Show resolved Hide resolved chanbackup/single.go Outdated
Show resolved Hide resolved chanbackup/backupfile.go Outdated
Show resolved Hide resolved chanbackup/backupfile_test.go Outdated
Show resolved Hide resolved chanbackup/backupfile_test.go

Roasbeef added some commits Dec 10, 2018

chanbackup: implement crypto operations for serialized static channel…
… baackups

In this commit, we implement a series of new crypto operations that will
allow us to encrypt and decrypt a set of serialized channel backups.
Their various backups may have distinct encodings when serialized, but
to the functions defined in this file, we treat them as simple opaque
blobs.

For encryption, we utilize chacha20poly1305 with a random 24 byte nonce.
We use a larger nonce size as this can be safely generated via a CSPRNG
without fear of frequency collisions between nonces generated. To
encrypt a blob, we then use this nonce as the AD (associated data) and
prepend the nonce to the front of the ciphertext package.

For key generation, in order to ensure the user only needs their
passphrase and the backup file, we utilize the existing keychain to
derive a private key. In order to ensure that at we don't force any
hardware signer to be aware of our crypto operations, we instead opt to
utilize a public key that will be hashed to derive our private key. The
assumption here is that this key will only be exposed to this software,
and never derived as a public facing address.
chanbackup: add implementation of Single Channel Backups (SCBs)
In this commit, we add the initial implementation of the SCB structure.
Given an SCB, and a user's seed, it will be possible to recover the
settled balanced of a channel in the event of total or partial data
loss. The SCB contains all information required to initiate the data
loss protection protocol once we restore the channel and connect to the
remote channel peer.

The primary way outside callers will interact with this package are via
the Pack and Unpack methods. Packing means writing a
serialized+encrypted version of the SCB to an io.Writer. Unpacking does
the opposite.

The encoding format itself uses the same encoding as we do on the wire
within Lightning. Each encoded backup begins with a version so we can
easily add or modify the serialization format in the future, if new
channel types appear, or we need to add/remove fields.
chanbackup: introduce Multi, a multi-channel backup
In this commit, we introduce the Multi sturct. Multi is a series of
static channel backups. This type of backup can contains ALL the channel
backup state in a single packed blob. This is suitable for storing on
your file system, cloud storage, etc. Systems will be in place within
lnd to ensure that one can easily obtain the latest version of the Multi
for the node, and also that it will be kept up to date if channel state
changes.
chanbackup: add functionality to allow external callers to backup cha…
…nnels

In this commit, we introduce a series of interfaces and methods that
will allow external callers to backup either all channels, or a specific
channel identified by its channel point. In order to abstract away the
details w.r.t _how_ we obtain the set of open channels, or their storage
mechanisms, we introduce a new LiveChannelSource interfaces. This
interfaces allows us to fetch all channels, a channel by its channel
point, and also all the known addresses for a node as we'll need this in
order to connect out to the node in the case of a recovery attempt.
chanbackup: add functions to allow recovery of existing channel backups
In this commit, we add a series of functions that will allow users to
recover existing channel backups. We do this using two primary
interfaces: the ChannelRestorer, and the PeerConnector. The first
interfaces allows us to abstract away the details w.r.t exactly how a
channel is restored. Instead, we simply expect that the channel backup
will be inserted as a sort of "channel shell" which contains only the
data required to initiate the data loss protection protocol. The second
interface is how we instruct the Lightning node to connect out to the
channel peer given its known addresses.
chanbackup: add new MultiFile struct for storing+updating Multis on disk
In this commit, we add a new MultiFile struct. We'll use this struct in
store the latest multi-channel backup on disk, swap it out atomically,
and finally extract+unpack the contents of the multi-file. The format
that's written to disk is the same as a regular Packed multi. The
contents of this new file are meant to be used to safely implement an
always up to date multi file on disk as a way for users to easily rsync
or fsnotiy (when it changes) the backup state of their channels.

We implement an atomic update and swap in the UpdateAndSwap. The method
uses relies on the underlying file system supporting an atomic rename
syscall. We first make a temporary backup file, write the latest
contents to that, then swap the temp file with the main file using
rename(2). This way, we ensure that we always have a single up to date
file, if the protocol aborts before the rename, then we can detect this,
remove the temp file, and attempt another swap.
chanbackup: add new sub-system for keeping the on disk channels.backu…
…p up to date

In this commit, we introduce the chanbackup.SubSwapper interface. It
takes a regular Swapper implementation (defined by the
chanbackup.SubSwapper) interface along with a chanbackup.ChannelNotifier
implementation. Given these two interfaces, we're able to be notified
when a new channel is opened or closed, and then use the Swapper to
atomically replace the on-disk channel back up. As a result, a Lightning
daemon can ensure that they alwayts have a up to date channels.backup on
disk that can safely be copied away by users and be used to restore
channel funds in the event of partial/total data loss.

@Roasbeef Roasbeef force-pushed the Roasbeef:static-chan-backups-chanbackup branch from be84a5d to 3b370fa Jan 24, 2019

@cfromknecht
Copy link
Collaborator

cfromknecht left a comment

LGTM ⚾️itest flake seems unrelated, though will need a restart

switch s.Version {
case DefaultSingleVersion:
default:
return fmt.Errorf("unable to de-serialize w/ unknown "+

This comment has been minimized.

@cfromknecht

cfromknecht Jan 24, 2019

Collaborator

just making a note that once we add other versions, this should be deferred until after reading the length. then if we don't understand it, discard length bytes and return to the caller so that we advance the reader. preferably this would be a concrete error so that the multi reader can skip/log it during deserialization

@Roasbeef Roasbeef merged commit e9889cb into lightningnetwork:master Jan 25, 2019

2 checks passed

continuous-integration/travis-ci/pr The Travis CI build passed
Details
coverage/coveralls Coverage increased (+0.3%) to 56.416%
Details
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment