Add `ChannelPending` event emitted upon `funding_signed` #2098

tnull · 2023-03-10T16:32:07Z

Users currently don't have good way of being notified when channel open negotiations have succeeded and new channels are pending confirmation on chain.

To this end, we add a new ChannelPending event that is emitted when send or receive a funding_signed message, i.e., at the last moment before waiting for the confirmation period.

codecov-commenter · 2023-03-10T17:10:49Z

Codecov Report

Patch coverage: 80.95% and project coverage change: -0.03 ⚠️

Comparison is base (0e28bcb) 91.38% compared to head (3acc6f8) 91.36%.

❗ Current head 3acc6f8 differs from pull request most recent head 9873c7d. Consider uploading reports for the commit 9873c7d to get more accurate results

📣 This organization is not using Codecov’s GitHub App Integration. We recommend you install it so Codecov can continue to function properly for your repositories. Learn more

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2098      +/-   ##
==========================================
- Coverage   91.38%   91.36%   -0.03%     
==========================================
  Files         102      102              
  Lines       49767    49844      +77     
  Branches    49767    49844      +77     
==========================================
+ Hits        45482    45540      +58     
- Misses       4285     4304      +19

Impacted Files	Coverage Δ
lightning-background-processor/src/lib.rs	`77.80% <ø> (ø)`
lightning/src/events/mod.rs	`30.85% <0.00%> (-1.29%)`	⬇️
lightning/src/ln/functional_test_utils.rs	`92.59% <93.75%> (+<0.01%)`	⬆️
lightning/src/ln/chanmon_update_fail_tests.rs	`98.59% <100.00%> (+<0.01%)`	⬆️
lightning/src/ln/channel.rs	`90.08% <100.00%> (+0.04%)`	⬆️
lightning/src/ln/channelmanager.rs	`89.13% <100.00%> (+0.03%)`	⬆️
lightning/src/ln/functional_tests.rs	`98.22% <100.00%> (+<0.01%)`	⬆️
lightning/src/ln/priv_short_conf_tests.rs	`97.56% <100.00%> (+0.03%)`	⬆️
lightning/src/ln/reload_tests.rs	`95.57% <100.00%> (+0.01%)`	⬆️

... and 3 files with indirect coverage changes

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report in Codecov by Sentry.
📢 Do you have feedback about the report comment? Let us know in this issue.

lightning/src/util/events.rs

lightning/src/ln/functional_test_utils.rs

lightning/src/ln/channelmanager.rs

lightning/src/util/events.rs

tnull · 2023-03-13T20:59:51Z

Also rebased on main to take advantage of the CI fix.

tnull · 2023-03-13T21:22:08Z

... and kicked CI. Having tests that rely on bitcoind is really a gift that keeps on giving:

---- test_esplora_syncs stdout ----
thread 'test_esplora_syncs' panicked at 'called `Result::unwrap()` on an `Err` value: JsonRpc(Rpc(RpcError { code: -4, message: "Wallet file verification failed. SQLiteDatabase: Unable to obtain an exclusive lock on the database, is it being used by another instance of Bitcoin Core?\n", data: None }))', tests/integration_tests.rs:35:65
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

lightning/src/util/events.rs

lightning/src/ln/channel.rs

lightning/src/ln/channelmanager.rs

lightning/src/util/events.rs

TheBlueMatt · 2023-03-14T18:17:35Z

Note that you'll need to squash all the commits before we land - we should avoid having commits that individually fail tests as it makes future bisection hard.

RCasatta · 2023-03-16T09:34:18Z

... and kicked CI. Having tests that rely on bitcoind is really a gift that keeps on giving:

---- test_esplora_syncs stdout ----
thread 'test_esplora_syncs' panicked at 'called `Result::unwrap()` on an `Err` value: JsonRpc(Rpc(RpcError { code: -4, message: "Wallet file verification failed. SQLiteDatabase: Unable to obtain an exclusive lock on the database, is it being used by another instance of Bitcoin Core?\n", data: None }))', tests/integration_tests.rs:35:65
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

Can't reproduce locally by trying in a while loop.

But I stopped early because the test implementation is leaky, I think the drop on ElectrsD and BitcoinD is not called because in OnceCell? Since it is not called the process cannot be stopped/killed and unpleasantly stick around.

The main point of bitcoind and electrsd is to have isolated process, I suggest to create one instance in every test using it instead of having a global &'static

tnull · 2023-03-16T13:12:31Z

Can't reproduce locally by trying in a while loop.

But I stopped early because the test implementation is leaky, I think the drop on ElectrsD and BitcoinD is not called because in OnceCell? Since it is not called the process cannot be stopped/killed and unpleasantly stick around.

The main point of bitcoind and electrsd is to have isolated process, I suggest to create one instance in every test using it instead of having a global &'static

I think spinning up new dedicated BitcoinD/EsploraD instance per test case is infeasible, especially when we need to premine 101 blocks every time before we can get started. However, it seems you're right and once_cell indeed won't call drop and hence doesn't allow us to clean up properly upon shutdown, which is kind of unfortunate. Will have to further look into how to mitigate this.

RCasatta · 2023-03-16T16:05:16Z

I think spinning up new dedicated BitcoinD/EsploraD instance per test case is infeasible, especially when we need to premine 101 blocks every time before we can get started

How many tests with electrs do you have? It takes a couple of seconds to spin up bitcoind and electrs synced with 101 blocks, and by default N tests are launched in parallel where N is the number of your cores.

tnull · 2023-03-27T11:45:00Z

How many tests with electrs do you have? It takes a couple of seconds to spin up bitcoind and electrs synced with 101 blocks, and by default N tests are launched in parallel where N is the number of your cores.

You're right, should be fine for now, although we might want to reconsider if the number of tests grows. Fixed in #2132

TheBlueMatt · 2023-03-29T22:10:08Z

lightning/src/events/mod.rs

+	/// Used to indicate that a channel with the given `channel_id` is being opened and pending
+	/// confirmation on-chain.
+	///
+	/// This event is emitted when the funding transaction has been signed and is broadcast to the


Should we have a note that, in rare cases, this event may be missed if a channel is ultimately confirmed quickly prior to the event being processed and then we restart? Its a stupid rare case but its possible I think?

I'm confused, which scenario are you describing exactly? Wouldn't rather be an edge case that we emit the event twice when we decide to rebroadcast on restart?

I think we can: (a) fund the channel, with an async monitor update, (b) not have that complete, but persist the manager, then (c) complete the monitor update and broadcast the funding tx (d) then restart, replay the chain and see the funding tx confirm, then (e) re-complete the monitor update completed in c, then we wont get the event again but the user will never have used it.

I think with "full" async monitor updates this becomes more realistic.

Should we have a note that, in rare cases, this event may be missed if a channel is ultimately confirmed quickly prior to the event being processed and then we restart?

Since we plan to leverage it for #2137, seems we should always regenerate it in cases like this? The channel will actually be ready, but as long as it's queued before the ChannelReady event, it should be fine.

Yea, that's my biggest concern here.

I think we can: (a) fund the channel, with an async monitor update, (b) not have that complete, but persist the manager, then (c) complete the monitor update and broadcast the funding tx (d) then restart, replay the chain and see the funding tx confirm, then (e) re-complete the monitor update completed in c, then we wont get the event again but the user will never have used it.

Ugh., I see. In that scenario it would persist the event queue when the event hasn't been emitted yet and wouldn't re-emit it after restart. Mh, instead of leaving a note I went ahead and am now tracking the emission of ChannelPending in the same way as ChannelReady and removed it from internal_funding_created entirely.
This means that we're now only emitting it after CMU completion or reestablish, i.e., would re-emit it if the "emitted" state wasn't persisted before.

tnull · 2023-03-31T11:37:53Z

Rebased on main to fix failing CI.

TheBlueMatt · 2023-03-31T19:12:46Z

LGTM, I think, feel free to squash the fixups.

tnull · 2023-04-01T10:08:26Z

LGTM, I think, feel free to squash the fixups.

Squashed without further changes.

wpaulino

LGTM, just two minor comments

wpaulino · 2023-04-03T16:47:26Z

lightning/src/ln/channelmanager.rs

@@ -8921,7 +8949,24 @@ pub mod bench {
 		} else { panic!(); }

 		node_b.handle_funding_created(&node_a.get_our_node_id(), &get_event_msg!(node_a_holder, MessageSendEvent::SendFundingCreated, node_b.get_our_node_id()));
+		let events_b = node_b.get_and_clear_pending_events();


Nit: Use expect_channel_pending_event here and below

Mh, yeah, unfortunatly I can't easily as expected_channel_pending_event takes a functional_test_utils::Node as parameter, and this sets up the ChannelManager manually. I'd rather not rewrite this entire test or all other tests just to avoid writing the match statement out once.

lightning/src/events/mod.rs

Currently, users don't have good way of being notified when channel open negotiations have succeeded and new channels are pending confirmation on chain. To this end, we add a new `ChannelPending` event that is emitted when send or receive a `funding_signed` message, i.e., at the last moment before waiting for the confirmation period. We track whether the event had previously been emitted in `Channel` and remove it from `internal_funding_created` entirely. Hence, we now only emit the event after ChannelMonitorUpdate completion, or upon channel reestablish. This mitigates a race condition where where we wouldn't persist the event *and* wouldn't regenerate it on restart, therefore potentially losing it, if async CMU wouldn't complete before ChannelManager persistence.

tnull · 2023-04-03T17:06:15Z

Squashed with the fixup:

diff --git a/lightning/src/events/mod.rs b/lightning/src/events/mod.rs
index f20d6667..2d576d33 100644
--- a/lightning/src/events/mod.rs
+++ b/lightning/src/events/mod.rs
@@ -631,5 +631,5 @@ pub enum Event {
                /// The `temporary_channel_id` this channel used to be known by during channel establishment.
                ///
-               /// May be `None` for channels created prior to LDK version 0.0.115.
+               /// Will be `None` for channels created prior to LDK version 0.0.115.
                former_temporary_channel_id: Option<[u8; 32]>,
                /// The `node_id` of the channel counterparty.

TheBlueMatt · 2023-04-03T16:55:59Z

lightning/src/events/mod.rs

+				write_tlv_fields!(writer, {
+					(0, channel_id, required),
+					(2, user_channel_id, required),
+					(4, former_temporary_channel_id, required),


nit: option is a bit more straightforward/saves a few bytes on the wire. It doesn't matter, though, just for future reference.

Yes, it's a tiny bit more efficient but to be honest I don't liking mixing implementation type and serialization type semantics if it can be avoided. In my mind option -> field is optional, not field is Option.

dunxen

LGTM. I believe the CI failure is unrelated but can't actually see the issue.

TheBlueMatt · 2023-04-03T18:21:53Z

Tracked CI failure as #2144

0.0.115 - Apr 24, 2023 - "Rebroadcast the Bugfixes" API Updates =========== * The MSRV of the main LDK crates has been increased to 1.48 (lightningdevkit#2107). * Attempting to claim an un-expired payment on a channel which has closed no longer fails. The expiry time of payments is exposed via `PaymentClaimable::claim_deadline` (lightningdevkit#2148). * `payment_metadata` is now supported in `Invoice` deserialization, sending, and receiving (via a new `RecipientOnionFields` struct) (lightningdevkit#2139, lightningdevkit#2127). * `Event::PaymentFailed` now exposes a failure reason (lightningdevkit#2142). * BOLT12 messages now support stateless generation and validation (lightningdevkit#1989). * The `NetworkGraph` is now pruned of stale data after RGS processing (lightningdevkit#2161). * Max inbound HTLCs in-flight can be changed in the handshake config (lightningdevkit#2138). * `lightning-transaction-sync` feature `esplora-async-https` was added (lightningdevkit#2085). * A `ChannelPending` event is now emitted after the initial handshake (lightningdevkit#2098). * `PaymentForwarded::outbound_amount_forwarded_msat` was added (lightningdevkit#2136). * `ChannelManager::list_channels_by_counterparty` was added (lightningdevkit#2079). * `ChannelDetails::feerate_sat_per_1000_weight` was added (lightningdevkit#2094). * `Invoice::fallback_addresses` was added to fetch `bitcoin` types (lightningdevkit#2023). * The offer/refund description is now exposed in `Invoice{,Request}` (lightningdevkit#2206). Backwards Compatibility ======================= * Payments sent with the legacy `*_with_route` methods on LDK 0.0.115+ will no longer be retryable via the LDK 0.0.114- `retry_payment` method (lightningdevkit#2139). * `Event::PaymentPathFailed::retry` was removed and will always be `None` for payments initiated on 0.0.115 which fail on an earlier version (lightningdevkit#2063). * `Route`s and `PaymentParameters` with blinded path information will not be readable on prior versions of LDK. Such objects are not currently constructed by LDK, but may be when processing BOLT12 data in a coming release (lightningdevkit#2146). * Providing `ChannelMonitorUpdate`s generated by LDK 0.0.115 to a `ChannelMonitor` on 0.0.114 or before may panic (lightningdevkit#2059). Note that this is in general unsupported, and included here only for completeness. Bug Fixes ========= * Fixed a case where `process_events_async` may `poll` a `Future` which has already completed (lightningdevkit#2081). * Fixed deserialization of `u16` arrays. This bug may have previously corrupted the historical buckets in a `ProbabilisticScorer`. Users relying on the historical buckets may wish to wipe their scorer on upgrade to remove corrupt data rather than waiting on it to decay (lightningdevkit#2191). * The `process_events_async` task is now `Send` and can thus be polled on a multi-threaded runtime (lightningdevkit#2199). * Fixed a missing macro export causing `impl_writeable_tlv_based_enum{,_upgradable}` calls to not compile (lightningdevkit#2091). * Fixed compilation of `lightning-invoice` with both `no-std` and serde (lightningdevkit#2187) * Fix an issue where the `background-processor` would not wake when a `ChannelMonitorUpdate` completed asynchronously, causing delays (lightningdevkit#2090). * Fix an issue where `process_events_async` would exit immediately (lightningdevkit#2145). * `Router` calls from the `ChannelManager` now call `find_route_with_id` rather than `find_route`, as was intended and described in the API (lightningdevkit#2092). * Ensure `process_events_async` always exits if any sleep future returns true, not just if all sleep futures repeatedly return true (lightningdevkit#2145). * `channel_update` messages no longer set the disable bit unless the peer has been disconnected for some time. This should resolve cases where channels are disabled for extended periods of time (lightningdevkit#2198). * We no longer remove CLN nodes from the network graph for violating the BOLT spec in some cases after failing to pay through them (lightningdevkit#2220). * Fixed a debug assertion which may panic under heavy load (lightningdevkit#2172). * `CounterpartyForceClosed::peer_msg` is now wrapped in UntrustedString (lightningdevkit#2114) * Fixed a potential deadlock in `funding_transaction_generated` (lightningdevkit#2158). Security ======== * Transaction re-broadcasting is now substantially more aggressive, including a new regular rebroadcast feature called on a timer from the `background-processor` or from `ChainMonitor::rebroadcast_pending_claims`. This should substantially increase transaction confirmation reliability without relying on downstream `TransactionBroadcaster` implementations for rebroadcasting (lightningdevkit#2203, lightningdevkit#2205, lightningdevkit#2208). * Implemented the changes from BOLT PRs lightningdevkit#1031, lightningdevkit#1032, and lightningdevkit#1040 which resolve a privacy vulnerability which allows an intermediate node on the path to discover the final destination for a payment (lightningdevkit#2062). In total, this release features 110 files changed, 11928 insertions, 6368 deletions in 215 commits from 21 authors, in alphabetical order: * Advait * Alan Cohen * Alec Chen * Allan Douglas R. de Oliveira * Arik Sosman * Elias Rohrer * Evan Feenstra * Jeffrey Czyz * John Cantrell * Lucas Soriano del Pino * Marc Tyndel * Matt Corallo * Paul Miller * Steven * Steven Williamson * Steven Zhao * Tony Giorgio * Valentine Wallace * Wilmer Paulino * benthecarman * munjesi

tnull force-pushed the 2023-03-add-channel-pending-event branch from 84ea88f to 226dfde Compare March 10, 2023 16:34

wpaulino reviewed Mar 13, 2023

View reviewed changes

lightning/src/util/events.rs Outdated Show resolved Hide resolved

lightning/src/ln/functional_test_utils.rs Outdated Show resolved Hide resolved

lightning/src/ln/channelmanager.rs Outdated Show resolved Hide resolved

TheBlueMatt reviewed Mar 13, 2023

View reviewed changes

lightning/src/util/events.rs Outdated Show resolved Hide resolved

lightning/src/util/events.rs Outdated Show resolved Hide resolved

tnull force-pushed the 2023-03-add-channel-pending-event branch from 5bd01c1 to 37ebcda Compare March 13, 2023 20:49

tnull force-pushed the 2023-03-add-channel-pending-event branch from 37ebcda to 970fa70 Compare March 13, 2023 21:20

tnull force-pushed the 2023-03-add-channel-pending-event branch from 970fa70 to d2af89b Compare March 14, 2023 11:21

tnull mentioned this pull request Mar 14, 2023

Address the race condition rust-bitcoin/bitcoind#101

Open

wpaulino reviewed Mar 14, 2023

View reviewed changes

lightning/src/util/events.rs Outdated Show resolved Hide resolved

lightning/src/ln/channel.rs Outdated Show resolved Hide resolved

tnull force-pushed the 2023-03-add-channel-pending-event branch from d2af89b to 58ea755 Compare March 14, 2023 15:25

TheBlueMatt reviewed Mar 14, 2023

View reviewed changes

lightning/src/ln/channelmanager.rs Outdated Show resolved Hide resolved

lightning/src/util/events.rs Outdated Show resolved Hide resolved

tnull force-pushed the 2023-03-add-channel-pending-event branch 2 times, most recently from 5526228 to 6a36d31 Compare March 29, 2023 14:11

TheBlueMatt mentioned this pull request Mar 29, 2023

Support manual signaling of channel readiness #2137

Closed

TheBlueMatt reviewed Mar 29, 2023

View reviewed changes

tnull mentioned this pull request Mar 30, 2023

Return funding txid on connect_open_channel lightningdevkit/ldk-node#53

Closed

tnull force-pushed the 2023-03-add-channel-pending-event branch from 01a545c to 3acc6f8 Compare March 31, 2023 11:37

tnull force-pushed the 2023-03-add-channel-pending-event branch from 3acc6f8 to 8475376 Compare April 1, 2023 10:07

wpaulino reviewed Apr 3, 2023

View reviewed changes

tnull force-pushed the 2023-03-add-channel-pending-event branch from 8475376 to 9873c7d Compare April 3, 2023 17:04

wpaulino approved these changes Apr 3, 2023

View reviewed changes

TheBlueMatt approved these changes Apr 3, 2023

View reviewed changes

dunxen approved these changes Apr 3, 2023

View reviewed changes

TheBlueMatt merged commit 7ca3709 into lightningdevkit:main Apr 3, 2023

tnull mentioned this pull request Apr 4, 2023

Add ChannelPending event lightningdevkit/ldk-node#60

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add `ChannelPending` event emitted upon `funding_signed` #2098

Add `ChannelPending` event emitted upon `funding_signed` #2098

tnull commented Mar 10, 2023 •

edited

Loading

codecov-commenter commented Mar 10, 2023 •

edited

Loading

tnull commented Mar 13, 2023

tnull commented Mar 13, 2023 •

edited

Loading

TheBlueMatt commented Mar 14, 2023

RCasatta commented Mar 16, 2023 •

edited

Loading

tnull commented Mar 16, 2023 •

edited

Loading

RCasatta commented Mar 16, 2023

tnull commented Mar 27, 2023

TheBlueMatt Mar 29, 2023

tnull Mar 30, 2023 •

edited

Loading

TheBlueMatt Mar 30, 2023

wpaulino Mar 30, 2023

TheBlueMatt Mar 30, 2023

tnull Mar 31, 2023

tnull commented Mar 31, 2023

TheBlueMatt commented Mar 31, 2023

tnull commented Apr 1, 2023

wpaulino left a comment

wpaulino Apr 3, 2023

tnull Apr 3, 2023

tnull commented Apr 3, 2023

TheBlueMatt Apr 3, 2023

tnull Apr 3, 2023

dunxen left a comment

TheBlueMatt commented Apr 3, 2023

Add ChannelPending event emitted upon funding_signed #2098

Add ChannelPending event emitted upon funding_signed #2098

Conversation

tnull commented Mar 10, 2023 • edited Loading

codecov-commenter commented Mar 10, 2023 • edited Loading

Codecov Report

tnull commented Mar 13, 2023

tnull commented Mar 13, 2023 • edited Loading

TheBlueMatt commented Mar 14, 2023

RCasatta commented Mar 16, 2023 • edited Loading

tnull commented Mar 16, 2023 • edited Loading

RCasatta commented Mar 16, 2023

tnull commented Mar 27, 2023

Choose a reason for hiding this comment

tnull Mar 30, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tnull commented Mar 31, 2023

TheBlueMatt commented Mar 31, 2023

tnull commented Apr 1, 2023

wpaulino left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tnull commented Apr 3, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dunxen left a comment

Choose a reason for hiding this comment

TheBlueMatt commented Apr 3, 2023

Add `ChannelPending` event emitted upon `funding_signed` #2098

Add `ChannelPending` event emitted upon `funding_signed` #2098

tnull commented Mar 10, 2023 •

edited

Loading

codecov-commenter commented Mar 10, 2023 •

edited

Loading

tnull commented Mar 13, 2023 •

edited

Loading

RCasatta commented Mar 16, 2023 •

edited

Loading

tnull commented Mar 16, 2023 •

edited

Loading

tnull Mar 30, 2023 •

edited

Loading