pyln: gossmap ... all the way #6012

m-schmoock · 2023-02-14T15:35:00Z

I did some work on the python gossmap implementation.
Let's see how it goes...

Update: I added a lot of missing and useful functionality to operate-on/analyse gossip stores and also use that for plugins that do network analysis and optimization.

adds missing parsing of node_announcement metadata addresses, alias, rgb, ...
add a get_nodes() methods that can return filtered subsets of the graph.
adds a get_halfchannels() method that can be used to return an area of channels with a given distance (depth) towards or from a certain node
adds various predefined and parametrable filters (channel_capacity, channel_count, nodes_address_type, ...)
print stats of a loaded store, useful for debugging
optimized lookups and hashes
a more extensive python gossmap testcase
various cleanups

cdecker

Forgot to submit my reviews, not to mention actually finish reviewing :-) Anyway, I'll pick this up asap

contrib/pyln-client/pyln/client/gossmap.py

cdecker

Very nice PR, with some interesting tricks in there, thanks :-)
I found a couple minor things, but this could go forward as is.

tests/test_misc.py

contrib/pyln-client/pyln/client/gossmap.py

m-schmoock · 2023-02-17T12:09:25Z

@cdecker Applied your suggestions (except the wait_for_mempool).
Do you have additional ideas on e.g. graph operations or path finding we could add into python gossmap?

I was thinking about adding a constraints paramter to operations such get_halfchannels that would allow us to limit the channels it uses e.g. by requiring a minimum total channel amount and such.

This can even be made in a way that when a callback/lambda is passed it applies this to all half_channels it comes across to find out if a channel should be considered or not. This way a user/programmer can descide to filter on anything it wants.

m-schmoock · 2023-02-17T12:10:58Z

Or maybe add some basic routing :D

Currently I'm testing this beefed up gossmap implementation on mainnet data

m-schmoock · 2023-02-17T12:23:36Z

This looks promising, when I load full mainnet data, the initial load takes some time ofc, but complex querying is good:

loaded 15173 nodes and 74856 channels.execution time: 7.362061977386475 seconds
get_halfchannels(depth=0) on my node gave 31 channels took:  8.654594421386719e-05 seconds
get_halfchannels(depth=1) on my node gave 10243 channels took:  0.022817611694335938 seconds

Maybe we can improve initial load performance somehow...

cdecker · 2023-02-17T12:53:40Z

Looking good 👍 That initial load time can likely be reduced, but profiles will likely tell us how. The C gossmap is rather clever in that it creates an index over the mmaped file, but we should likely get close to its performance, though not its memory efficiency :-)

m-schmoock · 2023-03-08T21:41:52Z

@cdecker thanks, I applied a lot from your suggestions directly and resolved them. Some are still open, there I applied what I can and asked some questions back on others...

rustyrussell · 2023-03-21T04:14:15Z

contrib/pyln-client/pyln/client/gossmap.py

+    def has_feature(self, bit):
+        return 1 << bit & self.features > 0
+
+    def has_feature_odd(self, bit):
+        return 2 << bit & self.features > 0
+
+    def has_features(self, bitmask):
+        return bitmask & self.features == bitmask


I don't think these functions are generally useful. Most commonly, you want all nodes which support feature_foo, you want "features & (3 << foo) != 0". Call that "has_feature()". Then you can have "has_compulsory_feature()" and "has_optional_feature()".

has_features() should almost certainly use the "has_feature" logic, too.

Please fix this. This API is terrible!

Yes I did that now, can you re-check?

contrib/pyln-client/pyln/client/gossmap.py

rustyrussell · 2023-03-21T04:21:27Z

Oh, and needs a Changelog-Added indicating that pyln.client.gossmap has seen significant functionality improvements.

rustyrussell · 2023-03-25T05:08:05Z

I rebased again (trivial, to remove conflict in test_misc.py) before I realized you still didn't:

Change has_feature API to make sense.
Add Changelog-Changed to highlight these significant improvements in pyln.client.gossmap.

Also, the commit message for pytest: adds skipped test_create_gossip_mesh is wonky, and has Changelog-None which is suppressing our changelog detection. (Generally, I prefer Changelog-None only to appear in the GH PR description, otherwise it'd be on almost every commit!).

rustyrussell

A little more cleanup, see comment....

This method is no longer used in cln nor in the plugins repo. Changelog-None

This can be adapted and used to create test gossip stores. The test is just skipped by design as it would fail on intention.

- moves offset into GossipHeader hdr which is passed to all constuctors - reads .flags as u16 instead of extracting it from the .length, see 0274d88 - adds zombie and ratelimit flag to GossipHeader - bytes_read start at 0 instead of 1 which is more correct, the one byte is then corrected for when setting the offset of new header. - bytes_read is increased in pull_bytes as this is the only place where something is read - use new style for various format-strings

Also caches certain __hash__ and __str__ operations, This way graph operations can be done quicker.

also makes them acessible using bitmask functions

Includes a lot of useful filters and statistical methods. To see a gossip_store summary: ``` s = GossmapStats(g) s.print_stats() ```

Changelog-Added: pyln-client: Improvements on the gossmap implementation

rustyrussell · 2023-04-03T06:35:20Z

Excellent!

Ack 4d199b7

m-schmoock changed the title ~~chore: cleanups~~ pyln: gossmap Feb 14, 2023

m-schmoock force-pushed the chore/cleanups branch 9 times, most recently from f7db3d4 to 6544bfe Compare February 16, 2023 16:02

m-schmoock marked this pull request as ready for review February 16, 2023 16:07

m-schmoock requested a review from cdecker as a code owner February 16, 2023 16:07

m-schmoock force-pushed the chore/cleanups branch 2 times, most recently from d862fb1 to 4888e96 Compare February 16, 2023 17:05

cdecker reviewed Feb 16, 2023

View reviewed changes

contrib/pyln-client/pyln/client/gossmap.py Outdated Show resolved Hide resolved

contrib/pyln-client/pyln/client/gossmap.py Outdated Show resolved Hide resolved

m-schmoock force-pushed the chore/cleanups branch from 4888e96 to c6ea2f5 Compare February 16, 2023 21:04

cdecker reviewed Feb 17, 2023

View reviewed changes

m-schmoock force-pushed the chore/cleanups branch 5 times, most recently from 99ee3a4 to 210efa1 Compare February 17, 2023 11:59

m-schmoock force-pushed the chore/cleanups branch 2 times, most recently from 74b8395 to e0f3009 Compare February 18, 2023 08:19

m-schmoock marked this pull request as draft February 18, 2023 12:14

m-schmoock force-pushed the chore/cleanups branch from e0f3009 to 6ba37dd Compare February 18, 2023 19:59

rustyrussell reviewed Mar 21, 2023

View reviewed changes

m-schmoock force-pushed the chore/cleanups branch from f79fc65 to 710a15c Compare March 24, 2023 10:49

rustyrussell force-pushed the chore/cleanups branch from 710a15c to 37a9191 Compare March 25, 2023 05:04

rustyrussell self-requested a review March 25, 2023 05:08

rustyrussell requested changes Mar 25, 2023

View reviewed changes

m-schmoock added 13 commits March 25, 2023 11:50

pyln-proto: shorten ShortChannelId.from_str()

f0d434b

pyln-testing: remove deprecated fund_channel

2e010ae

This method is no longer used in cln nor in the plugins repo. Changelog-None

pyln-testing: fundbalancedchannel default total_capacity to FUNDAMOUNT

363e40c

pytest: adds skipped test_create_gossip_mesh

8a40629

This can be adapted and used to create test gossip stores. The test is just skipped by design as it would fail on intention.

pygossmap: adds GossmapHalfchannel to module exports

a6d8261

pygossmap: adds missing __str__, __eq__ and __hash__

c71d48b

Also caches certain __hash__ and __str__ operations, This way graph operations can be done quicker.

pygossmap: adds a more complete mesh testcase

adf8614

pygossmap: adds get_halfchannel

4e1abdc

pygossmap: adds get_neighbors and get_neighbors_hc flodding method

0c98481

pygossmap: parse node addresses and other data

8517115

pygossmap: read .disabled from channel_flags

25a63cc

pygossmap: store features for nodes and channels

fa3dea1

also makes them acessible using bitmask functions

m-schmoock force-pushed the chore/cleanups branch 2 times, most recently from 48effcb to b9e1a3a Compare March 26, 2023 10:13

m-schmoock added 2 commits March 26, 2023 22:42

pygossmap: adds statistic and filter module

1a50cd4

Includes a lot of useful filters and statistical methods. To see a gossip_store summary: ``` s = GossmapStats(g) s.print_stats() ```

pygossmap: rename GossipStoreHeader to GossipStoreMsgHeader

4d199b7

Changelog-Added: pyln-client: Improvements on the gossmap implementation

m-schmoock force-pushed the chore/cleanups branch from b9e1a3a to 4d199b7 Compare March 26, 2023 20:43

m-schmoock requested a review from rustyrussell March 28, 2023 06:28

rustyrussell approved these changes Apr 3, 2023

View reviewed changes

rustyrussell merged commit 04ea37d into ElementsProject:master Apr 4, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

pyln: gossmap ... all the way #6012

pyln: gossmap ... all the way #6012

m-schmoock commented Feb 14, 2023 •

edited

Loading

cdecker left a comment

cdecker left a comment

m-schmoock commented Feb 17, 2023

m-schmoock commented Feb 17, 2023

m-schmoock commented Feb 17, 2023 •

edited

Loading

cdecker commented Feb 17, 2023

m-schmoock commented Mar 8, 2023

rustyrussell Mar 21, 2023

rustyrussell Mar 25, 2023

m-schmoock Mar 25, 2023

rustyrussell commented Mar 21, 2023

rustyrussell commented Mar 25, 2023

rustyrussell left a comment

rustyrussell commented Apr 3, 2023

pyln: gossmap ... all the way #6012

pyln: gossmap ... all the way #6012

Conversation

m-schmoock commented Feb 14, 2023 • edited Loading

cdecker left a comment

Choose a reason for hiding this comment

cdecker left a comment

Choose a reason for hiding this comment

m-schmoock commented Feb 17, 2023

m-schmoock commented Feb 17, 2023

m-schmoock commented Feb 17, 2023 • edited Loading

cdecker commented Feb 17, 2023

m-schmoock commented Mar 8, 2023

rustyrussell Mar 21, 2023

Choose a reason for hiding this comment

rustyrussell Mar 25, 2023

Choose a reason for hiding this comment

m-schmoock Mar 25, 2023

Choose a reason for hiding this comment

rustyrussell commented Mar 21, 2023

rustyrussell commented Mar 25, 2023

rustyrussell left a comment

Choose a reason for hiding this comment

rustyrussell commented Apr 3, 2023

m-schmoock commented Feb 14, 2023 •

edited

Loading

m-schmoock commented Feb 17, 2023 •

edited

Loading