Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Macro for composing custom message handlers #1832

Merged
merged 4 commits into from
Feb 15, 2023

Conversation

jkczyz
Copy link
Contributor

@jkczyz jkczyz commented Nov 4, 2022

Macro for composing custom message handlers BOLT 1 specifies a custom message type range for use with experimental or application-specific messages. While a CustomMessageHandler can be defined to support more than one message type, defining such a handler requires a significant amount of boilerplate and can be error prone.

Add a crate exporting a composite_custom_message_handler macro for easily composing pre-defined custom message handlers. The resulting handler can be further composed with other custom message handlers using the same macro.

This requires a separate crate since the macro needs to support "or" patterns in macro_rules, which is only available in edition 2021.

https://doc.rust-lang.org/edition-guide/rust-2021/or-patterns-macro-rules.html

Otherwise, a crate defining a handler for a set of custom messages could not easily be reused with another custom message handler. Doing so would require explicitly duplicating the reused handlers type ids, but those may change when the crate is updated.

Fixes #1813.

@TheBlueMatt
Copy link
Collaborator

One alternative to discuss is going the macro-only approach and actually taking a patern vs a min/max: #1813 (comment) I don't have a strong opinion, just throwing it out there.

@TheBlueMatt
Copy link
Collaborator

Nice! I think we should restrict the macro a bit to also implement CustomMessageHandler on the generated struct.

@jkczyz jkczyz changed the title Composite CustomMessageHandler macro Macro for composing custom message handlers Jan 3, 2023
@codecov-commenter
Copy link

codecov-commenter commented Jan 3, 2023

Codecov Report

Base: 87.33% // Head: 87.92% // Increases project coverage by +0.59% 🎉

Coverage data is based on head (9876a08) compared to base (be4bb58).
Patch has no changes to coverable lines.

📣 This organization is not using Codecov’s GitHub App Integration. We recommend you install it so Codecov can continue to function properly for your repositories. Learn more

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #1832      +/-   ##
==========================================
+ Coverage   87.33%   87.92%   +0.59%     
==========================================
  Files         100      100              
  Lines       44160    47330    +3170     
  Branches    44160    47330    +3170     
==========================================
+ Hits        38568    41616    +3048     
- Misses       5592     5714     +122     
Impacted Files Coverage Δ
lightning/src/ln/peer_handler.rs 57.40% <ø> (ø)
lightning/src/onion_message/messenger.rs 87.50% <ø> (ø)
lightning/src/ln/functional_tests.rs 96.99% <0.00%> (-0.21%) ⬇️
lightning/src/ln/reorg_tests.rs 97.26% <0.00%> (+0.93%) ⬆️
lightning/src/ln/channelmanager.rs 89.27% <0.00%> (+2.34%) ⬆️
lightning/src/sync/nostd_sync.rs 35.29% <0.00%> (+2.94%) ⬆️
lightning/src/ln/functional_test_utils.rs 92.15% <0.00%> (+3.66%) ⬆️

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.

@ZmnSCPxj-jr
Copy link

ZmnSCPxj-jr commented Jan 18, 2023

LGTM

@jkczyz jkczyz added this to the 0.0.114 milestone Jan 20, 2023
Copy link
Contributor

@alecchendev alecchendev left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Making sure I'm understanding this PR correctly: currently PeerManager is created with a single CustomMessageHandler that handles custom lightning messages for application specific/experimental messages. You can implement CustomMessageHandler on a specific message type, but if you want to handle multiple custom messages, it's a lot of boilerplate to combine the different handlers into one (which is required to handle multiple custom messages)--and this macro makes this easy. If I'm getting that all right, this seems really clean!

Out of scope for this PR, but wanted to mention while I was going through this code it took me a bit to realize how CustomMessageHandler was used in PeerManager, as there's not much mention of each other in either of their respective docs. In hindsight it seems sorta obvious but that was something I ran into.

Comment on lines +165 to +185
//! composite_custom_message_handler!(
//! pub struct FooBarBazHandler {
//! foo_bar: FooBarHandler,
//! baz: BazHandler,
//! }
//!
//! pub enum FooBarBazMessage {
//! FooBar(foo_bar_type_ids!()),
//! Baz(baz_type_id!()),
//! }
//! );
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a reason someone would opt for using the macro multiple times like this as opposed to just listing each message + handler in one? i.e.

composite_custom_message_handler!(
  pub struct FooBarBazHandler {
    foo: FooHandler,
    bar: BarHandler,
    baz: BazHandler,
  }
  ...
}

Also is there significance to the foo_bar_type_ids being the | of the two type ids or was that mostly arbitrary? Otherwise, I thought the FooBar example was very helpful/straightforward!

Copy link
Contributor Author

@jkczyz jkczyz Jan 31, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a reason someone would opt for using the macro multiple times like this as opposed to just listing each message + handler in one? i.e.

composite_custom_message_handler!(
  pub struct FooBarBazHandler {
    foo: FooHandler,
    bar: BarHandler,
    baz: BazHandler,
  }
  ...
}

That is also possible, though the example is meant to demonstrate arbitrary composition. This may be useful when FooBarHandler is defined in one crate and you want to use it along with BazHandler in your own crate. I've added some comments to the example to clarify this use case.

Also is there significance to the foo_bar_type_ids being the | of the two type ids or was that mostly arbitrary? Otherwise, I thought the FooBar example was very helpful/straightforward!

Yes, the macro matcher $pattern uses a pat specifier, meaning it can be anything rust considers a pattern, including the composition of patterns using |.

https://doc.rust-lang.org/reference/patterns.html

Someone using the macro could use a range of type ids instead, e.g., 32768..=32769. Though the | approach is preferred for the reason given in the macro's docs.

Copy link
Contributor

@alecchendev alecchendev Jan 31, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That is also possible, though the example is meant to demonstrate arbitrary composition. This may be useful when FooBarHandler is defined in one crate and you want to use it along with BazHandler in your own crate. I've added some comments to the example to clarify this use case.

Ah makes sense!

Yes, the macro matcher $pattern uses a pat specifier, meaning it can be anything rust considers a pattern, including the composition of patterns using |.

Ohh I see I didn't realize it was being used as a pattern I thought it was doing a bitwise OR of the two type ids, that's cool.

lightning-custom-message/src/lib.rs Outdated Show resolved Hide resolved
@jkczyz
Copy link
Contributor Author

jkczyz commented Jan 31, 2023

Thanks for the review! All good feedback.

Making sure I'm understanding this PR correctly: currently PeerManager is created with a single CustomMessageHandler that handles custom lightning messages for application specific/experimental messages. You can implement CustomMessageHandler on a specific message type, but if you want to handle multiple custom messages, it's a lot of boilerplate to combine the different handlers into one (which is required to handle multiple custom messages)--and this macro makes this easy. If I'm getting that all right, this seems really clean!

Correct!

Out of scope for this PR, but wanted to mention while I was going through this code it took me a bit to realize how CustomMessageHandler was used in PeerManager, as there's not much mention of each other in either of their respective docs. In hindsight it seems sorta obvious but that was something I ran into.

Hmm... yeah, those docs could use some work. I've added a commit that re-writes them.

@jkczyz jkczyz force-pushed the 2022-11-composite-handler branch 2 times, most recently from daefd33 to 5383d5f Compare January 31, 2023 16:57
Copy link
Contributor

@alecchendev alecchendev left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Latest changes to the docs look great!

Copy link
Collaborator

@TheBlueMatt TheBlueMatt left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Needs CI testing, but LGTM, let's just rebase and land this.

//! fn type_id(&self) -> u16 { foo_type_id!() }
//! }
//! impl Writeable for Foo {
//! // ...
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: tabs not spaces

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So, I believe all (most?) of our doctests use spaces. IIRC, there was some weirdness around the leading "//! " and the first tab not indenting enough. Especially, if the space after the comment was skipped in lieu of the first tab.

//! [BOLT 1]: https://github.com/lightning/bolts/blob/master/01-messaging.md
//! [`CustomMessageHandler`]: crate::lightning::ln::peer_handler::CustomMessageHandler

#![doc(test(no_crate_inject, attr(deny(warnings))))]
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is a no_crate_inject and why are we denying compilation warnings?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

no_crate_inject is needed otherwise we can't add #[macro_use] on the crate in the doc test.

For warnings, I was testing out unreachable patterns in the doc test, so needed that for it fail. I can remove it now, I suppose.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yea, I just generally worry about deny(warnings) cause that means CI will start failing on a new rustc without us touching anything. We'll fix warnings when we find them.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FWIW, this only affects doc tests.

//! pub struct Baz;
//!
//! macro_rules! baz_type_id {
//! () => { 32770 }
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would be cool to demonstrate baz supporting two different message ids internally.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Probably better left for CustomMessageHandler docs?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well with the macro_rules-specific stuff for this crate.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Like what is done using the FooBarHandler? I thought you had meant showing a leaf handler handling more than one message.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right, I did mean a leaf handler handling more than one message feeding into one MessageHandler. I guess it doesn't matter, though, as long as the documentation mentions that you can do any pattern (which it does) and maybe mention in the example that they don't have to be in different crate(s).

@jkczyz jkczyz marked this pull request as ready for review February 1, 2023 21:33
@jkczyz
Copy link
Contributor Author

jkczyz commented Feb 1, 2023

Needs test but can be saved for a follow-up.

@TheBlueMatt
Copy link
Collaborator

Test or not we should add something that hits this code in CI.

@jkczyz
Copy link
Contributor Author

jkczyz commented Feb 2, 2023

Test or not we should add something that hits this code in CI.

Ah, sorry! Misread your comment. Let's see if the latest commit triggers testing on the crate.

Copy link
Contributor

@wpaulino wpaulino left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, needs a rebase

@@ -11,6 +11,7 @@ members = [
]

exclude = [
"lightning-custom-message",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Any reason it's excluded?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Trying to compile using cargo +1.41.1 check gives:

error: failed to parse manifest at `/Users/jkczyz/src/rust-lightning/lightning-custom-message/Cargo.toml`

Caused by:
  failed to parse the `edition` key

Caused by:
  supported edition values are `2015` or `2018`, but `2021` is unknown

But after rebasing, it looks like that's also a problem for one one of lightning-transaction-sync's dependencies, as well. I guess we just can't build the entire workspace with an older rust version now?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, I guess CI doesn't try to build the workspace at all then since it's passing without excluding lightning-transaction-sync. We probably shouldn't break the default way to build for users on our MSRV, so we should exclude lightning-transaction-sync as well.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, correction, the above command also doesn't work because lightning-net-tokio requires a higher version. So moving lightning-custom-message back into the members should be fine.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Spoke to soon. CI is unhappy because of the edition:

error: failed to parse manifest at `/home/runner/work/rust-lightning/rust-lightning/lightning-custom-message/Cargo.toml`

Caused by:
  failed to parse the `edition` key

Caused by:
  supported edition values are `2015` or `2018, but `2021` is unknown

BOLT 1 specifies a custom message type range for use with experimental
or application-specific messages. While a `CustomMessageHandler` can be
defined to support more than one message type, defining such a handler
requires a significant amount of boilerplate and can be error prone.

Add a crate exporting a `composite_custom_message_handler` macro for
easily composing pre-defined custom message handlers. The resulting
handler can be further composed with other custom message handlers using
the same macro.

This requires a separate crate since the macro needs to support "or"
patterns in macro_rules, which is only available in edition 2021.

https://doc.rust-lang.org/edition-guide/rust-2021/or-patterns-macro-rules.html

Otherwise, a crate defining a handler for a set of custom messages could
not easily be reused with another custom message handler. Doing so would
require explicitly duplicating the reused handlers type ids, but those
may change when the crate is updated.
These are seen in newer versions of rustc.
Documentation for CustomMessageHandler wasn't clear how it is related to
PeerManager and contained some grammatical and factual errors. Re-write
the docs and link to the lightning_custom_message crate.
Copy link
Collaborator

@TheBlueMatt TheBlueMatt left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TIL you can iterate over multiple macro repeats at once.

@TheBlueMatt TheBlueMatt merged commit 5e9e68a into lightningdevkit:main Feb 15, 2023
k0k0ne pushed a commit to bitlightlabs/rust-lightning that referenced this pull request Sep 30, 2024
0.0.114 - Mar 3, 2023 - "Faster Async BOLT12 Retries"

API Updates
===========

 * `InvoicePayer` has been removed and its features moved directly into
   `ChannelManager`. As such it now requires a simplified `Router` and supports
   `send_payment_with_retry` (and friends). `ChannelManager::retry_payment` was
   removed in favor of the automated retries. Invoice payment utilities in
   `lightning-invoice` now call the new code (lightningdevkit#1812, lightningdevkit#1916, lightningdevkit#1929, lightningdevkit#2007, etc).
 * `Sign`/`BaseSign` has been renamed `ChannelSigner`, with `EcdsaChannelSigner`
   split out in anticipation of future schnorr/taproot support (lightningdevkit#1967).
 * The catch-all `KeysInterface` was split into `EntropySource`, `NodeSigner`,
   and `SignerProvider`. `KeysManager` implements all three (lightningdevkit#1910, lightningdevkit#1930).
 * `KeysInterface::get_node_secret` is now `KeysManager::get_node_secret_key`
   and is no longer required for external signers (lightningdevkit#1951, lightningdevkit#2070).
 * A `lightning-transaction-sync` crate has been added which implements keeping
   LDK in sync with the chain via an esplora server (lightningdevkit#1870). Note that it can
   only be used on nodes that *never* ran a previous version of LDK.
 * `Score` is updated in `BackgroundProcessor` instead of via `Router` (lightningdevkit#1996).
 * `ChainAccess::get_utxo` (now `UtxoAccess`) can now be resolved async (lightningdevkit#1980).
 * BOLT12 `Offer`, `InvoiceRequest`, `Invoice` and `Refund` structs as well as
   associated builders have been added. Such invoices cannot yet be paid due to
   missing support for blinded path payments (lightningdevkit#1927, lightningdevkit#1908, lightningdevkit#1926).
 * A `lightning-custom-message` crate has been added to make combining multiple
   custom messages into one enum/handler easier (lightningdevkit#1832).
 * `Event::PaymentPathFailure` is now generated for failure to send an HTLC
   over the first hop on our local channel (lightningdevkit#2014, lightningdevkit#2043).
 * `lightning-net-tokio` no longer requires an `Arc` on `PeerManager` (lightningdevkit#1968).
 * `ChannelManager::list_recent_payments` was added (lightningdevkit#1873).
 * `lightning-background-processor` `std` is now optional in async mode (lightningdevkit#1962).
 * `create_phantom_invoice` can now be used in `no-std` (lightningdevkit#1985).
 * The required final CLTV delta on inbound payments is now configurable (lightningdevkit#1878)
 * bitcoind RPC error code and message are now surfaced in `block-sync` (lightningdevkit#2057).
 * Get `historical_estimated_channel_liquidity_probabilities` was added (lightningdevkit#1961).
 * `ChannelManager::fail_htlc_backwards_with_reason` was added (lightningdevkit#1948).
 * Macros which implement serialization using TLVs or straight writing of struct
   fields are now public (lightningdevkit#1823, lightningdevkit#1976, lightningdevkit#1977).

Backwards Compatibility
=======================

 * Any inbound payments with a custom final CLTV delta will be rejected by LDK
   if you downgrade prior to receipt (lightningdevkit#1878).
 * `Event::PaymentPathFailed::network_update` will always be `None` if an
   0.0.114-generated event is read by a prior version of LDK (lightningdevkit#2043).
 * `Event::PaymentPathFailed::all_paths_removed` will always be false if an
   0.0.114-generated event is read by a prior version of LDK. Users who rely on
   it to determine payment retries should migrate to `Event::PaymentFailed`, in
   a separate release prior to upgrading to LDK 0.0.114 if downgrading is
   supported (lightningdevkit#2043).

Performance Improvements
========================

 * Channel data is now stored per-peer and channel updates across multiple
   peers can be operated on simultaneously (lightningdevkit#1507).
 * Routefinding is roughly 1.5x faster (lightningdevkit#1799).
 * Deserializing a `NetworkGraph` is roughly 6x faster (lightningdevkit#2016).
 * Memory usage for a `NetworkGraph` has been reduced substantially (lightningdevkit#2040).
 * `KeysInterface::get_secure_random_bytes` is roughly 200x faster (lightningdevkit#1974).

Bug Fixes
=========

 * Fixed a bug where a delay in processing a `PaymentSent` event longer than the
   time taken to persist a `ChannelMonitor` update, when occurring immediately
   prior to a crash, may result in the `PaymentSent` event being lost (lightningdevkit#2048).
 * Fixed spurious rejections of rapid gossip sync data when the graph has been
   updated by other means between gossip syncs (lightningdevkit#2046).
 * Fixed a panic in `KeysManager` when the high bit of `starting_time_nanos`
   is set (lightningdevkit#1935).
 * Resolved an issue where the `ChannelManager::get_persistable_update_future`
   future would fail to wake until a second notification occurs (lightningdevkit#2064).
 * Resolved a memory leak when using `ChannelManager::send_probe` (lightningdevkit#2037).
 * Fixed a deadlock on some platforms at least when using async `ChannelMonitor`
   updating (lightningdevkit#2006).
 * Removed debug-only assertions which were reachable in threaded code (lightningdevkit#1964).
 * In some cases when payment sending fails on our local channel retries no
   longer take the same path and thus never succeed (lightningdevkit#2014).
 * Retries for spontaneous payments have been fixed (lightningdevkit#2002).
 * Return an `Err` if `lightning-persister` fails to read the directory listing
   rather than panicing (lightningdevkit#1943).
 * `peer_disconnected` will now never be called without `peer_connected` (lightningdevkit#2035)

Security
========

0.0.114 fixes several denial-of-service vulnerabilities which are reachable from
untrusted input from channel counterparties or in deployments accepting inbound
connections or channels. It also fixes a denial-of-service vulnerability in rare
cases in the route finding logic.
 * The number of pending un-funded channels as well as peers without funded
   channels is now limited to avoid denial of service (lightningdevkit#1988).
 * A second `channel_ready` message received immediately after the first could
   lead to a spurious panic (lightningdevkit#2071). This issue was introduced with 0conf
   support in LDK 0.0.107.
 * A division-by-zero issue was fixed in the `ProbabilisticScorer` if the amount
   being sent (including previous-hop fees) is equal to a channel's capacity
   while walking the graph (lightningdevkit#2072). The division-by-zero was introduced with
   historical data tracking in LDK 0.0.112.

In total, this release features 130 files changed, 21457 insertions, 10113
deletions in 343 commits from 18 authors, in alphabetical order:
 * Alec Chen
 * Allan Douglas R. de Oliveira
 * Andrei
 * Arik Sosman
 * Daniel Granhão
 * Duncan Dean
 * Elias Rohrer
 * Jeffrey Czyz
 * John Cantrell
 * Kurtsley
 * Matt Corallo
 * Max Fang
 * Omer Yacine
 * Valentine Wallace
 * Viktor Tigerström
 * Wilmer Paulino
 * benthecarman
 * jurvis
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Provide Common Base For Custom Message Combinators
6 participants