wtclient: queue backup id only #7623

ellemouton · 2023-04-21T08:49:14Z

Currently, the entire breach retribution struct is passed to the watchtower client
and queued in its task pipeline. This is no longer necessary with the changes made
in #7379 since it is now possible for the
tower client to build the breach retribution info on the fly.

So what this PR does is to now only queue the wtdb.BackupID of the update in the
task pipeline and reconstruct the breach retribution only at the time that it will be
required.

This PR sets up all the ground work that will be required for #7380

Flow of the PR

A new FetchChannelByID method is added to ChannelStateDB to make it
possible to later on fetch a channel by its lnwire.ChannelID
representation. The reason for this is that currently there is only a
FetchChannel method which allows you to fetch a channel by its wire.Outpoint
but the tower client uses lnwire.ChannelID to keep track of channels so we
need a new lookup method here.
To prepare for the later on when we want to be able to construct the
BreachRetribution at the time that it is needed, we create and pass through a
buildBreachRetribution callback to the tower client.
Since the tower client now has the ability to build the breach retribution by
itself, we let it do that instead of having the channellink do that. This
is to prepare for later on when we construct the breach info JIT.
Move the construction of the breach retribution to the time of binding it to
a session.
At this point, we can now let the (still in-memory) pipeline task queue only
carry the BackupID.

ellemouton · 2023-04-21T09:24:33Z

Hey @guggero & @bitromortac :)

This is a split off from #7380 so I think mostly reviewed already :)

bitromortac

tested, LGTM (I reviewed this in #7380) 💪. Do you have an estimate how much compression we get by this per backup @ellemouton?

ellemouton · 2023-04-24T10:15:04Z

Thanks for the review @bitromortac 🙏

Do you have an estimate how much compression we get by this per backup

The wtdb.BackupID is a fixed 40 bytes while the lnwallet.BreachRetribution is muuuch bigger: a quick calculation estimates that the lower bound of bytes it holds is: 510 + 94(numHTLCs). And this is without taking into account any []byte fields.

guggero

Thank you for extracting those commits out. Almost ready, just a few small nits.

channeldb/db.go

channeldb/db_test.go

This commit just does some linting of the client_test.go file so that future commits are easier to parse.

A few lint fixes of the FetchChannel method.

This commit adds a small optimisation to the FetchChannel method. Instead of iterating over each channel bucket, an identifiable error is thrown once the wanted channel is found so that the iteration can stop early.

This commit introduces a new `channelSelector` method and moves all generic logic from `FetchChannel` to it. This refactor will make it easier to add new methods that require the same open-channel db traversal with slightly different channel selection logic.

Add a FetchChannelByID method that allows a caller to fetch an OpenChannel using an lnwire.ChannelID.

In this commit, a new BuildBreachRetribution callback is added to the tower client's Config struct. The main LND server provides the client with an implementation of the callback.

This commit fixes some lints in the wtclient package. This is done so that upcoming logic changes are easier to parse.

Since the TowerClient now has a callback that it can use to retrieve the retribution for a certain channel and commit height, let it use this call back instead of requiring the info to be passed to it through BackupState.

Only construct the retribution info at the time that the backup task is being bound to a session.

Since the retrubution info of a backup task is now only constructed at the time that the task is being bound to a session, the in-memory queue only needs to carry the BackupID of the task.

Lock the `backupMu` when accessing `c.chanCommitHeights` in the `New` function. It is not strictly necessary right now but good to add it so that there is no accidental oversight if the `perUpdate` method is ever extracted and reused in future.

guggero

Nice 🔥 LGTM 🎉

ellemouton force-pushed the queueBackupIDOnly branch from 8fb70ad to aae7ad9 Compare April 21, 2023 08:50

ellemouton mentioned this pull request Apr 21, 2023

watchtower: replace in-mem task queue with a disk over-flow queue #7380

Merged

ellemouton force-pushed the queueBackupIDOnly branch from aae7ad9 to 117e419 Compare April 21, 2023 09:09

ellemouton requested review from guggero and bitromortac April 21, 2023 09:20

ellemouton added this to the v0.17.0 milestone Apr 21, 2023

bitromortac approved these changes Apr 24, 2023

View reviewed changes

guggero reviewed Apr 24, 2023

View reviewed changes

channeldb/db.go Outdated Show resolved Hide resolved

channeldb/db.go Outdated Show resolved Hide resolved

channeldb/db.go Outdated Show resolved Hide resolved

channeldb/db_test.go Outdated Show resolved Hide resolved

ellemouton added 12 commits April 24, 2023 13:10

wtclient: cleanup the test file

cab0560

This commit just does some linting of the client_test.go file so that future commits are easier to parse.

channeldb: lint FetchChannel method

aebdd23

A few lint fixes of the FetchChannel method.

channeldb: optimise FetchChannel method

908cb60

This commit adds a small optimisation to the FetchChannel method. Instead of iterating over each channel bucket, an identifiable error is thrown once the wanted channel is found so that the iteration can stop early.

channeldb: add a FetchChannelByID method

fe2304e

Add a FetchChannelByID method that allows a caller to fetch an OpenChannel using an lnwire.ChannelID.

multi: pass BuildBreachRetribution callback to tower client

85ec38f

In this commit, a new BuildBreachRetribution callback is added to the tower client's Config struct. The main LND server provides the client with an implementation of the callback.

wtclient: lint the package

530a8ca

This commit fixes some lints in the wtclient package. This is done so that upcoming logic changes are easier to parse.

multi: build retribution info in TowerClient

458ac32

Since the TowerClient now has a callback that it can use to retrieve the retribution for a certain channel and commit height, let it use this call back instead of requiring the info to be passed to it through BackupState.

wtclient: only fetch retribution info when needed.

2371bbf

Only construct the retribution info at the time that the backup task is being bound to a session.

wtclient: let task pipeline only carry wtdb.BackupID

65dc20f

Since the retrubution info of a backup task is now only constructed at the time that the task is being bound to a session, the in-memory queue only needs to carry the BackupID of the task.

wtclient: add mutex locking in perUpdate

08cde98

Lock the `backupMu` when accessing `c.chanCommitHeights` in the `New` function. It is not strictly necessary right now but good to add it so that there is no accidental oversight if the `perUpdate` method is ever extracted and reused in future.

docs: update release notes

9a1ed8c

ellemouton force-pushed the queueBackupIDOnly branch from 117e419 to 9a1ed8c Compare April 24, 2023 11:16

guggero approved these changes Apr 24, 2023

View reviewed changes

guggero merged commit 4355ce6 into lightningnetwork:master Apr 24, 2023
20 of 24 checks passed

ellemouton deleted the queueBackupIDOnly branch April 24, 2023 13:39

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

wtclient: queue backup id only #7623

wtclient: queue backup id only #7623

ellemouton commented Apr 21, 2023 •

edited

ellemouton commented Apr 21, 2023

bitromortac left a comment

ellemouton commented Apr 24, 2023

guggero left a comment

guggero left a comment

wtclient: queue backup id only #7623

wtclient: queue backup id only #7623

Conversation

ellemouton commented Apr 21, 2023 • edited

Flow of the PR

ellemouton commented Apr 21, 2023

bitromortac left a comment

Choose a reason for hiding this comment

ellemouton commented Apr 24, 2023

guggero left a comment

Choose a reason for hiding this comment

guggero left a comment

Choose a reason for hiding this comment

ellemouton commented Apr 21, 2023 •

edited