Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Tx-Sync: Track spent WatchedOutputs and re-add if unconfirmed #2946

Conversation

tnull
Copy link
Contributor

@tnull tnull commented Mar 19, 2024

Fixes #2734.

  • Previously, we would track a spending transaction but wouldn't account for it being reorged out of the chain, in which case we wouldn't monitor the WatchedOutputs until they'd be reloaded on restart.
    In the first commit, we keep any WatchedOutputs around until their spends are sufficiently confirmed and only prune them after ANTI_REORG_DELAY.

  • Moreover, we dedup any ConfirmedTxs entries before handing them to Confirm::transactions_confirmed. Previously, we would just push to the confirmed_txs Vec, leading to redundant Confirm::transactions_confirmed calls, especially now that we re-confirm previously disconnected spends.
    In the second commit, we ensure that we don't push additional ConfirmedTx entries if already one with matching Txid is present. This not only gets rid of the spurious transactions_confirmed calls (which are harmless), but
    more importantly saves us from issuing unnecessary network calls, which improves latency.

@tnull tnull added this to the 0.0.122 milestone Mar 19, 2024
@tnull tnull requested a review from TheBlueMatt March 19, 2024 14:38
@codecov-commenter
Copy link

codecov-commenter commented Mar 19, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 90.01%. Comparing base (2c9dbb9) to head (b71c6e2).
Report is 27 commits behind head on main.

❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #2946      +/-   ##
==========================================
+ Coverage   89.18%   90.01%   +0.83%     
==========================================
  Files         117      117              
  Lines       95541    99416    +3875     
  Branches    95541    99416    +3875     
==========================================
+ Hits        85205    89489    +4284     
+ Misses       7840     7714     -126     
+ Partials     2496     2213     -283     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@tnull tnull force-pushed the 2024-03-txsync-readd-reorged-output-spends branch 2 times, most recently from 8ae8f02 to 0c8d233 Compare March 19, 2024 14:53
Previously, we would track a spending transaction but wouldn't account
for it being reorged out of the chain, in which case we wouldn't monitor
the `WatchedOutput`s until they'd be reloaded on restart.

Here, we keep any `WatchedOutput`s around until their spends are
sufficiently confirmed and only prune them after `ANTI_REORG_DELAY`.
@tnull tnull force-pushed the 2024-03-txsync-readd-reorged-output-spends branch 2 times, most recently from b1e51db to 6ded069 Compare March 19, 2024 16:02
Copy link
Collaborator

@TheBlueMatt TheBlueMatt left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One nit, otherwise LGTM.

lightning-transaction-sync/src/electrum.rs Outdated Show resolved Hide resolved
Copy link
Contributor

@jkczyz jkczyz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good modulo the comments.

lightning-transaction-sync/src/esplora.rs Outdated Show resolved Hide resolved
@tnull tnull force-pushed the 2024-03-txsync-readd-reorged-output-spends branch from 7253062 to 68c75dd Compare March 21, 2024 07:38
Copy link
Contributor

@jkczyz jkczyz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Feel free to squash.

@@ -342,15 +342,19 @@ where
// unwrap() safety: len() > 0 is checked above
let pos = *indexes.first().unwrap() as usize;
if let Some(tx) = maybe_await!(self.client.get_tx(&txid))? {
let txid = tx.txid();
if tx.txid() != *txid {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: may as well pass txid by value

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

Previously, we would just push to the `confirmed_txs` `Vec`, leading to
redundant `Confirm::transactions_confirmed` calls, especially now that
we re-confirm previously disconnected spends.

Here, we ensure that we don't push additional `ConfirmedTx` entries if
already one with matching `Txid` is present. This not only gets rid of
the spurious `transactions_confirmed` calls (which are harmless), but
more importantly saves us from issuing unnecessary network calls, which
improves latency.
@tnull tnull force-pushed the 2024-03-txsync-readd-reorged-output-spends branch from d7dafc9 to b71c6e2 Compare March 21, 2024 16:41
@tnull
Copy link
Contributor Author

tnull commented Mar 21, 2024

Squashed commits and included the following changes:

> git diff-tree -U2 d7dafc94 b71c6e2f6
diff --git a/lightning-transaction-sync/src/esplora.rs b/lightning-transaction-sync/src/esplora.rs
index e7952d6c2..538918ada 100644
--- a/lightning-transaction-sync/src/esplora.rs
+++ b/lightning-transaction-sync/src/esplora.rs
@@ -274,5 +274,5 @@ where
                                continue;
                        }
-                       if let Some(confirmed_tx) = maybe_await!(self.get_confirmed_tx(&txid, None, None))? {
+                       if let Some(confirmed_tx) = maybe_await!(self.get_confirmed_tx(*txid, None, None))? {
                                confirmed_txs.push(confirmed_tx);
                        }
@@ -297,5 +297,5 @@ where
                                                if let Some(confirmed_tx) = maybe_await!(self
                                                        .get_confirmed_tx(
-                                                               &spending_txid,
+                                                               spending_txid,
                                                                spending_tx_status.block_hash,
                                                                spending_tx_status.block_height,
@@ -320,5 +320,5 @@ where
        #[maybe_async]
        fn get_confirmed_tx(
-               &self, txid: &Txid, expected_block_hash: Option<BlockHash>, known_block_height: Option<u32>,
+               &self, txid: Txid, expected_block_hash: Option<BlockHash>, known_block_height: Option<u32>,
        ) -> Result<Option<ConfirmedTx>, InternalError> {
                if let Some(merkle_block) = maybe_await!(self.client.get_merkle_block(&txid))? {
@@ -335,5 +335,5 @@ where
                        let mut indexes = Vec::new();
                        let _ = merkle_block.txn.extract_matches(&mut matches, &mut indexes);
-                       if indexes.len() != 1 || matches.len() != 1 || matches[0] != *txid {
+                       if indexes.len() != 1 || matches.len() != 1 || matches[0] != txid {
                                log_error!(self.logger, "Retrieved Merkle block for txid {} doesn't match expectations. This should not happen. Please verify server integrity.", txid);
                                return Err(InternalError::Failed);
@@ -343,5 +343,5 @@ where
                        let pos = *indexes.first().unwrap() as usize;
                        if let Some(tx) = maybe_await!(self.client.get_tx(&txid))? {
-                               if tx.txid() != *txid {
+                               if tx.txid() != txid {
                                        log_error!(self.logger, "Retrieved transaction for txid {} doesn't match expectations. This should not happen. Please verify server integrity.", txid);
                                        return Err(InternalError::Failed);
@@ -350,10 +350,10 @@ where
                                if let Some(block_height) = known_block_height {
                                        // We can take a shortcut here if a previous call already gave us the height.
-                                       return Ok(Some(ConfirmedTx { tx, txid: *txid, block_header, pos, block_height }));
+                                       return Ok(Some(ConfirmedTx { tx, txid, block_header, pos, block_height }));
                                }

                                let block_status = maybe_await!(self.client.get_block_status(&block_hash))?;
                                if let Some(block_height) = block_status.height {
-                                       return Ok(Some(ConfirmedTx { tx, txid: *txid, block_header, pos, block_height }));
+                                       return Ok(Some(ConfirmedTx { tx, txid, block_header, pos, block_height }));
                                } else {
                                        // If any previously-confirmed block suddenly is no longer confirmed, we found

@TheBlueMatt TheBlueMatt merged commit 650caa0 into lightningdevkit:main Mar 21, 2024
13 of 16 checks passed
k0k0ne pushed a commit to bitlightlabs/rust-lightning that referenced this pull request Sep 30, 2024
v0.0.123 - May 08, 2024 - "BOLT12 Dust Sweeping"

API Updates
===========

 * To reduce risk of force-closures and improve HTLC reliability the default
   dust exposure limit has been increased to
   `MaxDustHTLCExposure::FeeRateMultiplier(10_000)`. Users with existing
   channels might want to consider using
   `ChannelManager::update_channel_config` to apply the new default (lightningdevkit#3045).
 * `ChainMonitor::archive_fully_resolved_channel_monitors` is now provided to
   remove from memory `ChannelMonitor`s that have been fully resolved on-chain
   and are now not needed. It uses the new `Persist::archive_persisted_channel`
   to inform the storage layer that such a monitor should be archived (lightningdevkit#2964).
 * An `OutputSweeper` is now provided which will automatically sweep
   `SpendableOutputDescriptor`s, retrying until the sweep confirms (lightningdevkit#2825).
 * After initiating an outbound channel, a peer disconnection no longer results
   in immediate channel closure. Rather, if the peer is reconnected before the
   channel times out LDK will automatically retry opening it (lightningdevkit#2725).
 * `PaymentPurpose` now has separate variants for BOLT12 payments, which
   include fields from the `invoice_request` as well as the `OfferId` (lightningdevkit#2970).
 * `ChannelDetails` now includes a list of in-flight HTLCs (lightningdevkit#2442).
 * `Event::PaymentForwarded` now includes `skimmed_fee_msat` (lightningdevkit#2858).
 * The `hashbrown` dependency has been upgraded and the use of `ahash` as the
   no-std hash table hash function has been removed. As a consequence, LDK's
   `Hash{Map,Set}`s no longer feature several constructors when LDK is built
   with no-std; see the `util::hash_tables` module instead. On platforms that
   `getrandom` supports, setting the `possiblyrandom/getrandom` feature flag
   will ensure hash tables are resistant to HashDoS attacks, though the
   `possiblyrandom` crate should detect most common platforms (lightningdevkit#2810, lightningdevkit#2891).
 * `ChannelMonitor`-originated requests to the `ChannelSigner` can now fail and
   be retried using `ChannelMonitor::signer_unblocked` (lightningdevkit#2816).
 * `SpendableOutputDescriptor::to_psbt_input` now includes the `witness_script`
   where available as well as new proprietary data which can be used to
   re-derive some spending keys from the base key (lightningdevkit#2761, lightningdevkit#3004).
 * `OutPoint::to_channel_id` has been removed in favor of
   `ChannelId::v1_from_funding_outpoint` in preparation for v2 channels with a
   different `ChannelId` derivation scheme (lightningdevkit#2797).
 * `PeerManager::get_peer_node_ids` has been replaced with `list_peers` and
   `peer_by_node_id`, which provide more details (lightningdevkit#2905).
 * `Bolt11Invoice::get_payee_pub_key` is now provided (lightningdevkit#2909).
 * `Default[Message]Router` now take an `entropy_source` argument (lightningdevkit#2847).
 * `ClosureReason::HTLCsTimedOut` has been separated out from
   `ClosureReason::HolderForceClosed` as it is the most common case (lightningdevkit#2887).
 * `ClosureReason::CooperativeClosure` is now split into
   `{Counterparty,Locally}Initiated` variants (lightningdevkit#2863).
 * `Event::ChannelPending::channel_type` is now provided (lightningdevkit#2872).
 * `PaymentForwarded::{prev,next}_user_channel_id` are now provided (lightningdevkit#2924).
 * Channel init messages have been refactored towards V2 channels (lightningdevkit#2871).
 * `BumpTransactionEvent` now contains the channel and counterparty (lightningdevkit#2873).
 * `util::scid_utils` is now public, with some trivial utilities to examine
   short channel ids (lightningdevkit#2694).
 * `DirectedChannelInfo::{source,target}` are now public (lightningdevkit#2870).
 * Bounds in `lightning-background-processor` were simplified by using
   `AChannelManager` (lightningdevkit#2963).
 * The `Persist` impl for `KVStore` no longer requires `Sized`, allowing for
   the use of `dyn KVStore` as `Persist` (lightningdevkit#2883, lightningdevkit#2976).
 * `From<PaymentPreimage>` is now implemented for `PaymentHash` (lightningdevkit#2918).
 * `NodeId::from_slice` is now provided (lightningdevkit#2942).
 * `ChannelManager` deserialization may now fail with `DangerousValue` when
    LDK's persistence API was violated (lightningdevkit#2974).

Bug Fixes
=========

 * Excess fees on counterparty commitment transactions are now included in the
   dust exposure calculation. This lines behavior up with some cases where
   transaction fees can be burnt, making them effectively dust exposure (lightningdevkit#3045).
 * `Future`s used as an `std::...::Future` could grow in size unbounded if it
   was never woken. For those not using async persistence and using the async
   `lightning-background-processor`, this could cause a memory leak in the
   `ChainMonitor` (lightningdevkit#2894).
 * Inbound channel requests that fail in
   `ChannelManager::accept_inbound_channel` would previously have stalled from
   the peer's perspective as no `error` message was sent (lightningdevkit#2953).
 * Blinded path construction has been tuned to select paths more likely to
   succeed, improving BOLT12 payment reliability (lightningdevkit#2911, lightningdevkit#2912).
 * After a reorg, `lightning-transaction-sync` could have failed to follow a
   transaction that LDK needed information about (lightningdevkit#2946).
 * `RecipientOnionFields`' `custom_tlvs` are now propagated to recipients when
   paying with blinded paths (lightningdevkit#2975).
 * `Event::ChannelClosed` is now properly generated and peers are properly
   notified for all channels that as a part of a batch channel open fail to be
   funded (lightningdevkit#3029).
 * In cases where user event processing is substantially delayed such that we
   complete multiple round-trips with our peers before a `PaymentSent` event is
   handled and then restart without persisting the `ChannelManager` after having
   persisted a `ChannelMonitor[Update]`, on startup we may have `Err`d trying to
   deserialize the `ChannelManager` (lightningdevkit#3021).
 * If a peer has relatively high latency, `PeerManager` may have failed to
   establish a connection (lightningdevkit#2993).
 * `ChannelUpdate` messages broadcasted for our own channel closures are now
   slightly more robust (lightningdevkit#2731).
 * Deserializing malformed BOLT11 invoices may have resulted in an integer
   overflow panic in debug builds (lightningdevkit#3032).
 * In exceedingly rare cases (no cases of this are known), LDK may have created
   an invalid serialization for a `ChannelManager` (lightningdevkit#2998).
 * Message processing latency handling BOLT12 payments has been reduced (lightningdevkit#2881).
 * Latency in processing `Event::SpendableOutputs` may be reduced (lightningdevkit#3033).

Node Compatibility
==================

 * LDK's blinded paths were inconsistent with other implementations in several
   ways, which have been addressed (lightningdevkit#2856, lightningdevkit#2936, lightningdevkit#2945).
 * LDK's messaging blinded paths now support the latest features which some
   nodes may begin relying on soon (lightningdevkit#2961).
 * LDK's BOLT12 structs have been updated to support some last-minute changes to
   the spec (lightningdevkit#3017, lightningdevkit#3018).
 * CLN v24.02 requires the `gossip_queries` feature for all peers, however LDK
   by default does not set it for those not using a `P2PGossipSync` (e.g. those
   using RGS). This change was reverted in CLN v24.02.2 however for now LDK
   always sets the `gossip_queries` feature. This change is expected to be
   reverted in a future LDK release (lightningdevkit#2959).

Security
========
0.0.123 fixes a denial-of-service vulnerability which we believe to be reachable
from untrusted input when parsing invalid BOLT11 invoices containing non-ASCII
characters.
 * BOLT11 invoices with non-ASCII characters in the human-readable-part may
   cause an out-of-bounds read attempt leading to a panic (lightningdevkit#3054). Note that all
   BOLT11 invoices containing non-ASCII characters are invalid.

In total, this release features 150 files changed, 19307 insertions, 6306
deletions in 360 commits since 0.0.121 from 17 authors, in alphabetical order:

 * Arik Sosman
 * Duncan Dean
 * Elias Rohrer
 * Evan Feenstra
 * Jeffrey Czyz
 * Keyue Bao
 * Matt Corallo
 * Orbital
 * Sergi Delgado Segura
 * Valentine Wallace
 * Willem Van Lint
 * Wilmer Paulino
 * benthecarman
 * jbesraa
 * olegkubrakov
 * optout
 * shaavan
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

tx-sync: Re-register watched outputs after spend has been reorged out of chain
4 participants