Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

first version of lib-autopilot.py #1888

Closed

Conversation

Projects
None yet
6 participants
@renepickhardt
Copy link
Collaborator

commented Aug 27, 2018

As mentioned in the c-lightning list: https://lists.ozlabs.org/pipermail/c-lightning/2018-August/000065.html you wellcome plugins which are not written in c.

So I took the liberty to provide a first sketch for an autopilot for c-lightning written an python. In the beginning of the year I have stated on lnd's autopilot that the Barabasi Albert model was probably not the best choice (c.f.: lightningnetwork/lnd#677 ) This is why my lib can currently generate candidates for an autopilot to open channels with according to four strategies (and it combines those currently):

  • Random: following the Erdoes Renyi model nodes are drawn from a uniform distribution
  • Central: nodes are sampled from a uniform distribution of the top most central nodes (betweenness)
  • Network_Improovement: nodes are sampled from a uniform distribution of the nodes which are badly connected
  • richness: nodes with high liquidity are taken and it is sampled from a uniform distribution of those

this library is work in progress. there is no logic yet implemented on how many funds to allocate to which channel and actually opening the channels (though I think that this should be the easy part) Also due to the randomness the lib might sometimes find less channels than the user asked for. Also many conner cases will let the code break.

Therefor the PR is more for code review and suggestions.

Running the code produces this output: (edit: the version in the pull request only prints the nodes without the statistics about balance and channels)

2018-08-27 16:06:19,285 - lib-autopilot - INFO - Try to load graph from file system at: data/networkx_graph
2018-08-27 16:06:19,306 - lib-autopilot - INFO - Successfully restored the lightning network graph from data/networkx_graph
2018-08-27 16:06:19,306 - lib-autopilot - INFO - running the autopilot on a graph with 1528 nodes and 7997 edges.
2018-08-27 16:06:19,306 - lib-autopilot - INFO - GENERATE CANDIDATES: Try to generate up to {} nodes with 4 strategies: (random, central, network Improvement, liquidity)
2018-08-27 16:06:19,306 - lib-autopilot - INFO - RANDOM_NODES: try to generate a set of 6 nodes sampled with uniform distribution
2018-08-27 16:06:19,306 - lib-autopilot - INFO - CENTRAL_NODES: Try to seek 6 nodes which are currently central
2018-08-27 16:06:32,083 - lib-autopilot - INFO - CENTRAL_NODES: Generated top 24 central nodes (according to betweeness centrality)
2018-08-27 16:06:32,083 - lib-autopilot - INFO - CENTRAL_NODES: Sample 6 items from the candidates as was requested
2018-08-27 16:06:32,083 - lib-autopilot - INFO - RICH_NODES: Try to generate 6 candidates from the 0.5-percentile
2018-08-27 16:06:32,092 - lib-autopilot - INFO - RICH_NODES: Found 34 candidates in the 0.5-percentile
2018-08-27 16:06:32,092 - lib-autopilot - INFO - RICH_NODES: Sample 6 items from the candidates as was requested
2018-08-27 16:06:32,092 - lib-autopilot - INFO - IMPROOVE_NETWORK: Try to seek 6 nodes which are currently bad connected
2018-08-27 16:06:37,253 - lib-autopilot - INFO - IMPROOVE_NETWORK: Found 16 candidates which are currently bad connected
2018-08-27 16:06:37,253 - lib-autopilot - INFO - IMPROOVE_NETWORK: Sample 6 items from the candidates as was requested
2018-08-27 16:06:37,253 - lib-autopilot - INFO - GENERATE CANDIDATES: Found 21 nodes with which channel creation is suggested

percentage   smoothed percentage    capacity    numchannels     alias
----------------------------------------------------------------------
        0.43           1.73           60000          1      andromeda
        1.45           2.44         2000000         10      stealthmode
        6.12           5.71       404547196        478      tady je slushovo
        6.64           6.08       119389472        130      node1.lnroute.com (LND)
        4.83           4.81        67439741        101      skyrus.net
        0.68           1.90         4403776         47      ⚡⚡⚡GERMAN⚡⚡⚡
       14.67          11.70       176501181         87      CoinGate
        7.24           6.50       414456777        414      🚀🌑 TrueVision.club
        3.04           3.55          420000          1      MEDUSA
        0.47           1.76           65000          1      🚀🌑cl.TrueVision.club
       13.36          10.78       363941981        197      Bitrefill.com
        6.64           6.08        86301722         94      ScripterRon
        7.48           6.67        13449158         13      jfbnode
        0.47           1.76           65000          1      🚀⚡CoNNeX⚡🚀
        0.36           1.68           50000          1      02b80cabdf82638aac86
        0.07           1.48           10000          1      yusasufhin
        8.51           7.38       152889341        130      ACINQ
        0.65           1.88           90000          1      margens
        5.15           5.03         3557093          5      ABank02
       11.59           9.54       245288253        153      mainnet.yalls.org
        0.14           1.53           20000          1      SLEEPYARK-PILOT

to explain the output. I asked to lib-autopilot to generate 21 candidates. you can see in the table the nodes (by their alias for better readability) which are suggested to connect to.

The first two columns list the percentage of our funds that the lib-autopilot thinks should be used for this channel. In the first column this is given by looking at the average channel balances of the partners and look at the relative frequency (with respect to alle the other average channel balances for the suggested channels). the second column smoothes the results with a weighted uniform distribution (currently the weight is 70% for the first column and 30% for the uniform distribution)

the third colum lists how much balance all the channels of the suggested partner has. the fourth column is the number of active channels and the last one is the alias.

I would be happy about any thoughts regarding my statistics. I think smoothing is usefull.

@cdecker
Copy link
Member

left a comment

Thanks @renepickhardt, excellent start for the autopilot. I went through and as always with new contributors I split every possible hair and was as petty as I could 😄 The following are some general remarks, and I have left a lot of inline comments:

  • Name: Why is this called lib-autopilot.py? It's a sidecar that is supposed to be run alongside the main c-lightning daemon, hence the lib- part is confusing to me. Calling it autopilot.py might be clearer.
  • Formatting: the code could use a run of pyflakes or pep8 to fix some minor formatting issues. A spell-checker would also be useful 😉
  • Sampling: your current sampling strategy just crops off the long tail of the distribution, meaning that nodes at the lower end of the distribution cannot be possibly chosen. A better strategy would be to use numpy.random.choice with the p parameter to bias towards more valuable nodes, but still allowing for the occasional low preference peer.
  • Disjoint sets: the current way of independently selecting from 4 different mechanisms does not guarantee that we don't get duplicates, e.g., a peer might both be desirable in terms of betweenness and richness.
  • I wonder if we can improve the betweenness to consider our own score instead of that of a potential peer, i.e., if we were to make these 2 connections that were proposed, how would our own betweenness change? That's more what we're trying to optimize, rather than the betweenness of our peers.

Other than these, I'm really excited about starting to experiment with various optimization techniques. Do you think a global score, created by combining the various techniques is also doable, as opposed to the current "we just pick nodes from 4 different sets at random"?

Show resolved Hide resolved contrib/lib-autopilot.py Outdated
Show resolved Hide resolved contrib/lib-autopilot.py Outdated
try:
self.__logger.info("Try to load graph from file system at: data/networkx_graph")
with open("data/networkx_graph","rb") as infile:
self.G = pickle.load(infile)

This comment has been minimized.

Copy link
@cdecker

cdecker Aug 28, 2018

Member

pickle is also most likely inefficient for storage (storing a lot of metadata about the python classes used in memory, instead of raw data. There are a number of better formats that are also more portable here: https://networkx.github.io/documentation/networkx-1.10/reference/readwrite.html

This comment has been minimized.

Copy link
@renepickhardt

renepickhardt Aug 28, 2018

Author Collaborator

actually I wanted to remove that storage part anyway. it was just for me that I could develop more quickly in a real world setting we also wish to have the most recent state of the graph and fetch it wile running the autopilot. The question is what will happen, if the network becomes really large? other than that do you have a preferred method from that list, I saw pickle was also part of it.

Show resolved Hide resolved contrib/lib-autopilot.py Outdated
Show resolved Hide resolved contrib/lib-autopilot.py Outdated
Show resolved Hide resolved contrib/lib-autopilot.py Outdated
Show resolved Hide resolved contrib/lib-autopilot.py Outdated
cutoff = 10
self.__logger.info("cutoff value must be a positive integer. Set back to default value: 10")

all_pair_shortest_paths = nx.all_pairs_shortest_path(self.G, cutoff=cutoff)

This comment has been minimized.

Copy link
@cdecker

cdecker Aug 28, 2018

Member

all_pair_shortest_paths is not actually needed since you discard the path after computing its length, you can use shortest_path_length without specifying the src and dest which will just give you the length, without materializing the path in-memory.

This comment has been minimized.

Copy link
@renepickhardt

renepickhardt Aug 28, 2018

Author Collaborator

I will check that. I thought I might need the paths later but in the current implementation I obviously don't

Show resolved Hide resolved contrib/lib-autopilot.py Outdated
def run(self):
self.__logger.info("running the autopilot on a graph with {} nodes and {} edges.".format(len(self.G.nodes()), len(self.G.edges())))
candidates = self.get_candidates(k=21)
time.sleep(1)

This comment has been minimized.

Copy link
@cdecker

cdecker Aug 28, 2018

Member

What is this sleep for? It doesn't seem like we need it at all.

This comment has been minimized.

Copy link
@renepickhardt

renepickhardt Aug 28, 2018

Author Collaborator

I used logger and print and they got mixed so I decided while debugging I could wait a second in order to have nicer output. forgot to delete the line

Show resolved Hide resolved contrib/lib-autopilot.py Outdated
Show resolved Hide resolved contrib/lib-autopilot.py Outdated
Show resolved Hide resolved contrib/lib-autopilot.py Outdated
@renepickhardt

This comment has been minimized.

Copy link
Collaborator Author

commented Aug 28, 2018

Hey Christian thanks for this extensive code review. I will answer your general remarks now. I will probably only after the lightninghackday find the time for a new version.

  • I called it lib-autopilot because I was thinking of doing all the graph related stuff in there and then create an interface which wraps the lib to talk to c-lightning. in this way the lib-autopilot could be reused in LND, eclair, and future projects like the electrum implementation or the rust one. If I make it mixed (as it is right now) I agree that the name is confusing.
  • I will reformat the code ( at this point I wanted to share work in progress and didn't care ) as for spellchecking. That is actually really an issue I am fighting with
  • sampling: I purposely did hard cut offs and then uniform sampling. two reasons. First I was not sure how many third party libraries are welcome. Even with networkx I was not sure but I guess numpy should be trustworthy. Second. I wanted the algorithm to be diverse and where fine with those (rather arbitrary) choices. However I can change that. On the long term research would be needed to see what are the real good strategies. (however in particular with richness and centrality my cutoff was way to small, so maybe generating my won probability distribution for any strategy would be the more general idea in any case.)
  • disjoint sets: depends on the scope of the project. If this turns into a lib I would really continue to build separated algorithms and heuristics and merge those together. So that was actually also an explicit design choice. I was actually thinking of not only making a set union but keep the information which peer was suggested by which method.
  • Betweeness: same as above: research is needed. I see what you are thinking but is what you are proposing really desirable? It would be fairly easily be implemented if not aiming for the best complexity class. I could easily identify edges wich would make me central. But since all my capacity at this moment is outbound I become a central node and no one could route over me.

There would be many ways to learn a global score (either by simulation or by collecting statistics over time). I am sure one could easily work years on optimizing the autopilot. So I guess it will be more a question of resource and focus. If that is wanted it would certainly be something that could be done (which again would speak for a lib autopilot)

At this time I believe (obviously without knowing) for the network the best heuristics are uniform random and the Network improove whereas for the user the best heuristics should be richness and betweeness.

I guess it would be really great if the autopilot was not only working on the static graph, but would talk with the network (and maybe even exchange some statistics (e.g. about skewness of channel balances) also there are more features which I kind of ignored. also the current implementation is impractical if the network grows so also for that it make sense to have the autopilot be coordinated in a distributed fashion maybe even as services like watch towers wich could be queried against. Same gist as above. it is easy to provide a hacky solution as I did but the problem can be engineered on for ever (:

@renepickhardt

This comment has been minimized.

Copy link
Collaborator Author

commented Sep 7, 2018

@cdecker some of your comments vanished. I am currently imiplementing the improved second version but wanted to look up some of your old comments since I don't remember everything. somehow every comment from you to which I did not answer disappeared. Do you know what has happened? Can the old state easily be restored? (if not no problem, I think I remember most of the comments)

@ZmnSCPxj

This comment has been minimized.

Copy link
Collaborator

commented Sep 7, 2018

Comments are attached to commit objects in github, and we have the policy to rebase, which makes old commit objects obsolete. github also has some random, unspecified schedule for cleaning up unused commit objects, and hence comments may disappear completely if they are on a commit that was rebased out of existence.

But if you did not rebase... possibly some other problem in github.

@cdecker

This comment has been minimized.

Copy link
Member

commented Sep 7, 2018

Comments are attached to commit objects in github, and we have the policy to rebase, which makes old commit objects obsolete. github also has some random, unspecified schedule for cleaning up unused commit objects, and hence comments may disappear completely if they are on a commit that was rebased out of existence.

I'm trying to get people to create fixup!s instead of amending commits directly. That seems to work with rebases as well, but is hard if the commit series changes (dropping commits, adding intermediate ones, or going a completely different route). I'll create a bot that auto-rebases and squashes PRs so we can get some of the operational burden dealt with.

Rene Pickhardt added some commits Aug 27, 2018

Rene Pickhardt
first version of lib-autopilot.py
the lib can currently generate candidates for an autopilot to open channels with according to four strategies:
Random: following the Erdoes Renyi model nodes are drawn from a uniform distribution
Central: nodes are sampled from a uniform distribution of the top most central nodes (betweenness)
Network_Improovement: nodes are sampled from a uniform distribution of the nodes which are badly connected
richness: nodes with high liquidity are taken and it is sampled from a uniform distribution of those

this library is work in progress. there is no logic yet implemented on how many funds to allocate to which channel.
also due to the randomness the lib might sometimes find less channels than was proposed.
also many conor cases will let the code break.
Rene Pickhardt
updated version that also distributes wished channel balance. code st…
…ill needs some clean up and sanity checks.

@renepickhardt renepickhardt force-pushed the renepickhardt:lib-autopilot branch from ff907c3 to 892e1c4 Sep 7, 2018

@renepickhardt

This comment has been minimized.

Copy link
Collaborator Author

commented Sep 7, 2018

Ok I added the second version. It is late and I hope I didn't do any stupid mistakes. Biggest updates for the second version:

  • new command line interface (see below)
  • all heuristics now create a probability distribution from which suggested nodes are sampled
  • there is a possibility to cut-off distributions by looking at top percentiles (configureable via the -p command line option)
  • there two different merging algorithms for the results of the heuristics are implemented (and also configurable via the -s parameter)
  • lib_autopilot is running completely without c-lightning and there is a c-lightning wrapper created. This yields the possibility to reuse the lib_autopilot in lnd, eclair or the electrum implementation.

on the longterm issues are:

  • the current implementation will take long computation if the lightning network grows
  • there should be a scientifically sound simulation be implemented to decide which strategies should actually be used. remember once we have grown a large lightning network we will not easily change the topology quickly.
  • no testing: I might be bad with border cases in particular if setting the cutoff value (-p) to small

docs for command line interface:

$ python3 c-lightning-autopilot --help

usage: c-lightning-autopilot.py [-h] [-b BALANCE] [-c CHANNELS]
                                [-r PATH_TO_RPC_INTERFACE]
                                [-s {diverse,merge}] [-p PERCENTILE_CUTOFF]
                                [-d] [-i INPUT]

optional arguments:
  -h, --help            show this help message and exit
  -b BALANCE, --balance BALANCE
                        use specified number of satoshis to open all channels
  -c CHANNELS, --channels CHANNELS
                        opens specified amount of channels
  -r PATH_TO_RPC_INTERFACE, --path_to_rpc_interface PATH_TO_RPC_INTERFACE
                        specifies the path to the rpc_interface
  -s {diverse,merge}, --strategy {diverse,merge}
                        defines the strategy
  -p PERCENTILE_CUTOFF, --percentile_cutoff PERCENTILE_CUTOFF
                        only uses the top percentile of each probability
                        distribution
  -d, --dont_store      don't store the network on the hard drive
  -i INPUT, --input INPUT
                        points to a pickle file

output of running the autopilot:

2018-09-07 23:05:04,469 - clightning-autopilot - INFO - set up logging infrastructure
2018-09-07 23:05:04,469 - clightning-autopilot - INFO - connection to RPC interface successful
2018-09-07 23:05:04,469 - clightning-autopilot - INFO - Try to load graph from file system at:lightning_networkx_graph.pickle
2018-09-07 23:05:04,496 - clightning-autopilot - INFO - Successfully restored the lightning network graph from data/networkx_graph
2018-09-07 23:05:04,496 - lib-autopilot - INFO - running the autopilot on a graph with 1510 nodes and 7773 edges.
2018-09-07 23:05:04,497 - lib-autopilot - INFO - GENERATE CANDIDATES: Try to generate up to 30 nodes with 4 strategies: (random, central, network Improvement, liquidity)
2018-09-07 23:05:04,497 - lib-autopilot - INFO - DECREASE DIAMETER: Generating probability density function
2018-09-07 23:05:15,805 - lib-autopilot - INFO - DECREASE DIAMETER: probability density function created
2018-09-07 23:05:15,805 - lib-autopilot - INFO - manipulate_pdf: Skewing the probability density function
2018-09-07 23:05:15,805 - lib-autopilot - INFO - CENTRALITY_PDF: Try to generate a PDF proportional to centrality scores
2018-09-07 23:05:26,999 - lib-autopilot - INFO - CENTRALITY_PDF: Generated pdf
2018-09-07 23:05:26,999 - lib-autopilot - INFO - RICH_PDF: Try to retrieve a PDF proportional to capacities
2018-09-07 23:05:27,006 - lib-autopilot - INFO - RICH_PDF: Generated a PDF proportional to capacities
2018-09-07 23:05:27,009 - lib-autopilot - INFO - GENERATE CANDIDATES: Found 30 nodes with which channel creation is suggested
percentage   smoothed percentage    capacity    numchannels     alias
----------------------------------------------------------------------
        0.17           1.12         1330000         24      🔵⚡ATLAS⚡🔵
        0.00           1.00            2500          1      BINARY
        1.11           1.77         1361000          3      CocoPi⚡
        8.94           7.26        22009788          7      Jubilee
        3.00           3.10        85801722         92      ScripterRon
        0.15           1.10           90000          1      WesternUnion
        0.00           1.00            1100          1      CN_Pek_HappyHome
        5.02           4.51       233289253        150      mainnet.yalls.org
        0.27           1.19          328271          3      nr400
        1.60           2.12         7391358         14      huseyinyaz
        1.34           1.94        11974338         28      LNapps.info
        4.00           3.80       197174817        159      LightningTo.Me
        1.15           1.81         2133147          5      02cb3c98561938a6d457
        2.24           2.57         2065969          2      MimWim
        0.02           1.01           10001          1      light.videogames4bitcoin.com
        3.92           3.74        25307389         20      lightning-pay.eu
       23.67          17.57      1966966002        269      fairly.cheap
        0.04           1.03           25000          1      Oguzhan
        0.16           1.11          100000          1      akinaydemir
        0.04           1.03           25000          1      DoguB
        4.18           3.92        28286115         21      LivingRoomofSatoshi.com
       17.80          13.46       142476490         25      0212864d59d5a902634c
        6.01           5.20        12941860          6      MEDEA
        0.10           1.07           60000          1      bankless.io
        1.14           1.80        21364687         60      Euclid
        3.21           3.25       403599446        407      🚀🌑 TrueVision.club
       10.12           8.09        18698695          5      SOMANYNODEMUCHWOW
        0.03           1.02           20000          1      christophernodan
        0.07           1.05           41760          1      belo [LND]
        0.51           1.36         1256666          7      The Cave
2018-09-07 23:05:27,010 - lib-autopilot - INFO - Need at least a balance of 1997502 satoshi to open 30 channels
2018-09-07 23:05:27,011 - clightning-autopilot - INFO - Try to open channel with a capacity of 112099 to node 0296f7a58d16bc6396e4ca5845b6bba892361a0fcfb2adfbbaa061a1d36271bb2d
2018-09-07 23:05:27,011 - clightning-autopilot - INFO - Try to open channel with a capacity of 100285 to node 03a8528b99c3ff876b1a46776c7b85721f0113bd372966df066dacc3e0ae31295a
INFO:clightning-autopilot:Try to open channel with a capacity of 100285 to node 03a8528b99c3ff876b1a46776c7b85721f0113bd372966df066dacc3e0ae31295a
2018-09-07 23:05:27,012 - clightning-autopilot - INFO - Try to open channel with a capacity of 177376 to node 03939ff69d65a13c4bb2585042e7eb7e75a7c77289ab5794d1b973721d86c6839c
INFO:clightning-autopilot:Try to open channel with a capacity of 177376 to node 03939ff69d65a13c4bb2585042e7eb7e75a7c77289ab5794d1b973721d86c6839c
2018-09-07 23:05:27,012 - clightning-autopilot - INFO - Try to open channel with a capacity of 725649 to node 03d67f36c4f81789e2fe425028bacc96b199813eae426c517f589a45f1136c1fe5
INFO:clightning-autopilot:Try to open channel with a capacity of 725649 to node 03d67f36c4f81789e2fe425028bacc96b199813eae426c517f589a45f1136c1fe5
2018-09-07 23:05:27,013 - clightning-autopilot - INFO - Try to open channel with a capacity of 309806 to node 02fe9469e1d7da69539e365c9b2a64d277ed5589622cb314dc8b3060f3c55f79e6
INFO:clightning-autopilot:Try to open channel with a capacity of 309806 to node 02fe9469e1d7da69539e365c9b2a64d277ed5589622cb314dc8b3060f3c55f79e6
2018-09-07 23:05:27,013 - clightning-autopilot - INFO - Try to open channel with a capacity of 110234 to node 0231dd236296211bf0678fd94e29d630886ae19ad9726f6d55a8871c193cde4f57
2018-09-07 23:05:27,014 - clightning-autopilot - INFO - Try to open channel with a capacity of 100126 to node 03e3d670d86f33181ee7451f14998b376a0d5deba8ab064b4ebe5ad5706f7b112b
INFO:clightning-autopilot:Try to open channel with a capacity of 100126 to node 03e3d670d86f33181ee7451f14998b376a0d5deba8ab064b4ebe5ad5706f7b112b
2018-09-07 23:05:27,014 - clightning-autopilot - INFO - Try to open channel with a capacity of 451336 to node 03e50492eab4107a773141bb419e107bda3de3d55652e6e1a41225f06a0bbf2d56
INFO:clightning-autopilot:Try to open channel with a capacity of 451336 to node 03e50492eab4107a773141bb419e107bda3de3d55652e6e1a41225f06a0bbf2d56
2018-09-07 23:05:27,015 - clightning-autopilot - INFO - Try to open channel with a capacity of 118663 to node 03da65f64cffecfbce02862f43efafd6bea82709e9d18d0349199e8138684855cb
INFO:clightning-autopilot:Try to open channel with a capacity of 118663 to node 03da65f64cffecfbce02862f43efafd6bea82709e9d18d0349199e8138684855cb
2018-09-07 23:05:27,015 - clightning-autopilot - INFO - Try to open channel with a capacity of 212057 to node 038fe3c921a6bc389d28fc6a47e131b07adfb36450bfc1553d6445b182eb84c9bd
INFO:clightning-autopilot:Try to open channel with a capacity of 212057 to node 038fe3c921a6bc389d28fc6a47e131b07adfb36450bfc1553d6445b182eb84c9bd
2018-09-07 23:05:27,015 - clightning-autopilot - INFO - Try to open channel with a capacity of 193899 to node 03f9d34e9a5564064730e76b2566b8185725fe5143dda6bdc35ba940687f9c980b
INFO:clightning-autopilot:Try to open channel with a capacity of 193899 to node 03f9d34e9a5564064730e76b2566b8185725fe5143dda6bdc35ba940687f9c980b
2018-09-07 23:05:27,016 - clightning-autopilot - INFO - Try to open channel with a capacity of 380244 to node 03bb88ccc444534da7b5b64b4f7b15e1eccb18e102db0e400d4b9cfe93763aa26d
INFO:clightning-autopilot:Try to open channel with a capacity of 380244 to node 03bb88ccc444534da7b5b64b4f7b15e1eccb18e102db0e400d4b9cfe93763aa26d
2018-09-07 23:05:27,016 - clightning-autopilot - INFO - Try to open channel with a capacity of 180849 to node 02cb3c98561938a6d457720e4f6119fc3b9c890a7d69190a648ff43758fed69769
INFO:clightning-autopilot:Try to open channel with a capacity of 180849 to node 02cb3c98561938a6d457720e4f6119fc3b9c890a7d69190a648ff43758fed69769
2018-09-07 23:05:27,017 - clightning-autopilot - INFO - Try to open channel with a capacity of 256606 to node 039ccb496c2420e4567904156e0663f5b9062d3390ab090e118cff7d2248dac0d2
INFO:clightning-autopilot:Try to open channel with a capacity of 256606 to node 039ccb496c2420e4567904156e0663f5b9062d3390ab090e118cff7d2248dac0d2
2018-09-07 23:05:27,017 - clightning-autopilot - INFO - Try to open channel with a capacity of 101138 to node 0350ef95f8cf301c3cef4504dd8478c9c2a805172fb661a6c212108fdbf7301a70
INFO:clightning-autopilot:Try to open channel with a capacity of 101138 to node 0350ef95f8cf301c3cef4504dd8478c9c2a805172fb661a6c212108fdbf7301a70
2018-09-07 23:05:27,018 - clightning-autopilot - INFO - Try to open channel with a capacity of 374052 to node 02ab16f790c7727eac2048771cbe95cbcc1aa93931558c0fd87c06d5c389475a7b
INFO:clightning-autopilot:Try to open channel with a capacity of 374052 to node 02ab16f790c7727eac2048771cbe95cbcc1aa93931558c0fd87c06d5c389475a7b
2018-09-07 23:05:27,018 - clightning-autopilot - INFO - Try to open channel with a capacity of 1756677 to node 03cb7983dc247f9f81a0fa2dfa3ce1c255365f7279c8dd143e086ca333df10e278
INFO:clightning-autopilot:Try to open channel with a capacity of 1756677 to node 03cb7983dc247f9f81a0fa2dfa3ce1c255365f7279c8dd143e086ca333df10e278
2018-09-07 23:05:27,019 - clightning-autopilot - INFO - Try to open channel with a capacity of 102843 to node 02c3190b21f31adcd7fcc55d9f4205ac1f43a841568048ddd0fa3d3ee37b10e334
INFO:clightning-autopilot:Try to open channel with a capacity of 102843 to node 02c3190b21f31adcd7fcc55d9f4205ac1f43a841568048ddd0fa3d3ee37b10e334
2018-09-07 23:05:27,019 - clightning-autopilot - INFO - Try to open channel with a capacity of 111371 to node 0205ae77caa47cacf7a447dc7547bd17bbf866fbc029e7f4992fb3089110fce66c
INFO:clightning-autopilot:Try to open channel with a capacity of 111371 to node 0205ae77caa47cacf7a447dc7547bd17bbf866fbc029e7f4992fb3089110fce66c
2018-09-07 23:05:27,019 - clightning-autopilot - INFO - Try to open channel with a capacity of 102843 to node 0362442331eca4dd27aa3b634b852ecd157db33f11c70cceec3e2052006b1c94de
INFO:clightning-autopilot:Try to open channel with a capacity of 102843 to node 0362442331eca4dd27aa3b634b852ecd157db33f11c70cceec3e2052006b1c94de
2018-09-07 23:05:27,020 - clightning-autopilot - INFO - Try to open channel with a capacity of 392386 to node 026b105ac13212c48714c6be9b11577a9ce10f10e1c88a45ce217e6331209faf8b
INFO:clightning-autopilot:Try to open channel with a capacity of 392386 to node 026b105ac13212c48714c6be9b11577a9ce10f10e1c88a45ce217e6331209faf8b
2018-09-07 23:05:27,020 - clightning-autopilot - INFO - Try to open channel with a capacity of 1346162 to node 0212864d59d5a902634c4a3186615446e9d56c17b14600f3b74fbabe3509aa707f
INFO:clightning-autopilot:Try to open channel with a capacity of 1346162 to node 0212864d59d5a902634c4a3186615446e9d56c17b14600f3b74fbabe3509aa707f
2018-09-07 23:05:27,021 - clightning-autopilot - INFO - Try to open channel with a capacity of 520440 to node 02ec1bbbe4c3f1b149b4e66c34d4f4ab80ed7da5bbe1dea04c8c8e46c4e90f8ad5
INFO:clightning-autopilot:Try to open channel with a capacity of 520440 to node 02ec1bbbe4c3f1b149b4e66c34d4f4ab80ed7da5bbe1dea04c8c8e46c4e90f8ad5
2018-09-07 23:05:27,021 - clightning-autopilot - INFO - Try to open channel with a capacity of 106823 to node 0241cff95a87da9ca9b581b84ce2c15fb58a0879c951e6125ad2be04b12af8ff4c
INFO:clightning-autopilot:Try to open channel with a capacity of 106823 to node 0241cff95a87da9ca9b581b84ce2c15fb58a0879c951e6125ad2be04b12af8ff4c
2018-09-07 23:05:27,022 - clightning-autopilot - INFO - Try to open channel with a capacity of 179648 to node 029a8741675c4c9078b577ddc4348d602d2fb45a12e6087b617925997a84f4c02e
INFO:clightning-autopilot:Try to open channel with a capacity of 179648 to node 029a8741675c4c9078b577ddc4348d602d2fb45a12e6087b617925997a84f4c02e
2018-09-07 23:05:27,022 - clightning-autopilot - INFO - Try to open channel with a capacity of 324955 to node 02529db69fd2ebd3126fb66fafa234fc3544477a23d509fe93ed229bb0e92e4fb8
INFO:clightning-autopilot:Try to open channel with a capacity of 324955 to node 02529db69fd2ebd3126fb66fafa234fc3544477a23d509fe93ed229bb0e92e4fb8
2018-09-07 23:05:27,023 - clightning-autopilot - INFO - Try to open channel with a capacity of 808704 to node 02fb74056358782bf2b555c80807fa29c9ddfbe5051cfadf2eb83871c155544391
INFO:clightning-autopilot:Try to open channel with a capacity of 808704 to node 02fb74056358782bf2b555c80807fa29c9ddfbe5051cfadf2eb83871c155544391
2018-09-07 23:05:27,023 - clightning-autopilot - INFO - Try to open channel with a capacity of 102275 to node 02cb72d6aebc1c29e7e8e3066bdc01e4608b39968b8054307387424aeebe3a25c6
INFO:clightning-autopilot:Try to open channel with a capacity of 102275 to node 02cb72d6aebc1c29e7e8e3066bdc01e4608b39968b8054307387424aeebe3a25c6
2018-09-07 23:05:27,023 - clightning-autopilot - INFO - Try to open channel with a capacity of 104749 to node 03cfff2802946f256dd04549ac191208e5f37687d6995b6361d69b9e9e5ecf9d18
INFO:clightning-autopilot:Try to open channel with a capacity of 104749 to node 03cfff2802946f256dd04549ac191208e5f37687d6995b6361d69b9e9e5ecf9d18
2018-09-07 23:05:27,024 - clightning-autopilot - INFO - Try to open channel with a capacity of 135722 to node 02ae3c06f6db6a87ea874ef7ba86c87ab0041eabf6d8be0594582505b1fd5d7ee8
INFO:clightning-autopilot:Try to open channel with a capacity of 135722 to node 02ae3c06f6db6a87ea874ef7ba86c87ab0041eabf6d8be0594582505b1fd5d7ee8
Autopilot finished. We hope it did a good job for you (and the lightning network). Thanks for using it.

res = self.__create_pdfs()

candidats = set()

This comment has been minimized.

Copy link
@leishman

leishman Sep 7, 2018

nit: spelling candidats -> candidates . Repeated quite a lot.

efficiently on large scale graphs with more than 100k nodes or on densly
connected graphs.
the programm needs the following dependencies:

This comment has been minimized.

Copy link
@leishman

leishman Sep 7, 2018

nit: s/programm/program/

@leishman

This comment has been minimized.

Copy link

commented Sep 7, 2018

Awesome project. Do you plan on pulling it out as a standalone application?

@renepickhardt

This comment has been minimized.

Copy link
Collaborator Author

commented Sep 8, 2018

@leishman thanks for your review. I separated the lib_autopilot from the C-lightning rpc interface on purpose. So the lib_autopilot could also easily be integrated to lnd or eclair. In particular it could be used by researchers to run simulations and find better strategies. A long term goal will be that the autopilots commuicate to each other and exchange some information about channel balances (which might conflict with privacy) and are able to request channels from others. In particular the autopilot currently ignores the channels you already have. There will be still much more work to do for this feature.

-b BALANCE, --balance BALANCE
use specified number of satoshis to open all channels
-c CHANNELS, --channels CHANNELS
opens specified amount of channels

This comment has been minimized.

Copy link
@dmytropiatkivskyi

dmytropiatkivskyi Sep 13, 2018

opens specified amount of channels -> opens the specified number of channels

This comment has been minimized.

Copy link
@cdecker

cdecker Sep 16, 2018

Member

Will this always open 5 new channels when I specify --channels=5 or is this giving the target number of channels, i.e., if I have 3 channels open, I'll just open 2 more?

optional arguments:
-h, --help show this help message and exit
-b BALANCE, --balance BALANCE
use specified number of satoshis to open all channels

This comment has been minimized.

Copy link
@dmytropiatkivskyi

dmytropiatkivskyi Sep 13, 2018

use specified -> uses the specified

@renepickhardt

This comment has been minimized.

Copy link
Collaborator Author

commented Sep 14, 2018

@cdecker any major fixes requested otherwise I would correct those typos. What about integration to lightning-cli?

@cdecker
Copy link
Member

left a comment

Quite some nice improvements. I found a few more things that I commented inline.

python3 c-lightning-autopilot --help
in order to get all the command line options

This comment has been minimized.

Copy link
@cdecker

cdecker Sep 16, 2018

Member

Capitalization: in -> In

-b BALANCE, --balance BALANCE
use specified number of satoshis to open all channels
-c CHANNELS, --channels CHANNELS
opens specified amount of channels

This comment has been minimized.

Copy link
@cdecker

cdecker Sep 16, 2018

Member

Will this always open 5 new channels when I specify --channels=5 or is this giving the target number of channels, i.e., if I have 3 channels open, I'll just open 2 more?

use specified number of satoshis to open all channels
-c CHANNELS, --channels CHANNELS
opens specified amount of channels
-r PATH_TO_RPC_INTERFACE, --path_to_rpc_interface PATH_TO_RPC_INTERFACE

This comment has been minimized.

Copy link
@cdecker

cdecker Sep 16, 2018

Member

--rpc-socket-path?

-p PERCENTILE_CUTOFF, --percentile_cutoff PERCENTILE_CUTOFF
only uses the top percentile of each probability
distribution
-d, --dont_store don't store the network on the hard drive

This comment has been minimized.

Copy link
@cdecker

cdecker Sep 16, 2018

Member

Usually negatives are prefixed by --no- or take false as an option to disable (I know, it's not a correct sentence). Also I think there is a convention to use - instead of _ but I can't find it in PEPs right now.

-i INPUT, --input INPUT
points to a pickle file
a good example call of the program could look like that:

This comment has been minimized.

Copy link
@cdecker

cdecker Sep 16, 2018

Member

Well, it's an example, not sure how good it is 😉

needed_total_balance, len(pdf)))
while needed_total_balance > balance and len(pdf) > 1:
min_val = min(pdf.values())
k = [k for k, v in pdf.items() if v == min_val][0]

This comment has been minimized.

Copy link
@cdecker

cdecker Sep 16, 2018

Member

This is basically just an ordered dict. You can also just sort a list of items by their values and effectively get the same thing.

This comment has been minimized.

Copy link
@cdecker

cdecker Sep 16, 2018

Member

That also obviates the need to renormalize on lines 356-357




def find_candidates(self, num_items=21,strategy = Strategy.DIVERSE,

This comment has been minimized.

Copy link
@cdecker

cdecker Sep 16, 2018

Member

Nit: spacing in arguments.



def find_candidates(self, num_items=21,strategy = Strategy.DIVERSE,
percentile = None):

This comment has been minimized.

Copy link
@cdecker

cdecker Sep 16, 2018

Member

percentile really should default to 1, which is the actual effect in this call.

candidats = self.__sample_from_percentile(merged, percentile,
num_items)
"""
following code prints a list of candidates for debugging

This comment has been minimized.

Copy link
@cdecker

cdecker Sep 16, 2018

Member

Don't leave dead code commented out.

if strategy == Strategy.DIVERSE:
for strategy, pdf in res.items():
tmp = self.__sample_from_percentile(pdf, percentile, sub_k)
candidats = candidats.union(set(tmp))

This comment has been minimized.

Copy link
@cdecker

cdecker Sep 16, 2018

Member

This is very likely to result in duplicates: central nodes may also be rich. This then result in less than the expected number of channels being opened.

This comment has been minimized.

Copy link
@renepickhardt

renepickhardt Sep 19, 2018

Author Collaborator

yes and no. the likelihood of duplicates drops tremendously when sampling from the full probability distribution.
My current Idea was to say if duplicates emerge we have less channels. this could of course easily be fixed by just checking the size of the result set and polling channels again. In the best case polling with one of the strategies from which the duplicates emerged.

@renepickhardt

This comment has been minimized.

Copy link
Collaborator Author

commented Sep 19, 2018

have put a few questions back in the inline statements would be great to hear your thoughts @cdecker before I finish up the requested changes. in particular the question about seednodes and the mapping of the graph from c lightning to lib-autopilot

lib_autopilot is a library which based on a networkx graph tries to
predict which channels should be added for a new node on the network. The
long term is to generate a lightning network with good topological properties.

This comment has been minimized.

Copy link
@fjahr

fjahr Oct 9, 2018

probably missing word: goal

@renepickhardt

This comment has been minimized.

Copy link
Collaborator Author

commented Feb 7, 2019

I am right now creating a c-lightning plugin that runs the lib_autopilot. I have some UX / UI trouble:

Calls to the autopilot can take several minutes to compute. In the current design calls will be made with lightning-cli autopilot arguments and stdio will block until the call stops. I think that is a really bad user experience...

How am I supposed to have non blocking calls? Is my plugin supposed to maintain its own threadpool and dispatch calls and provide an api to query against the state of certain calls or is there another way how this is supposed to be achieved?

I currently imagine a userflow that goes like:

$ lighthning-cli autopilot connect 21
started computation to find 21 nodes to connect to
$ lightning-cli autopilot status
still finding 21 nodes
$ lightning-cli autopilot status
last successfull task was to connect to the following 21 nodes: 
....

The problem with this design would be that there would not be an explicit user feedback once the job is completed.

Currently I also don't see a way of sending incremental feedback about the progress (other than pushing it to the logfile)

Any ideas how this should be resolved?

@cdecker

This comment has been minimized.

Copy link
Member

commented Feb 7, 2019

How am I supposed to have non blocking calls? Is my plugin supposed to maintain its own threadpool and dispatch calls and provide an api to query against the state of certain calls or is there another way how this is supposed to be achieved?

I completely forgot to push these changes upstream, but I have a branch that has async plugin method calls. I opened #2326 which contains the commits in question.

@renepickhardt

This comment has been minimized.

Copy link
Collaborator Author

commented Mar 23, 2019

As I am afk for the next 3 months I would be very happy if someone would take over. I know @cdecker has signaled that he might do that.

I think there is not much to do. the main work is done. it has basically to be wrapped with the plugin-client lib and a command line api has to be provided. I had already started but couldn't finish it.

@fjahr

This comment has been minimized.

Copy link

commented Mar 24, 2019

@renepickhardt I would be happy to help. I am traveling until the end of March but can definitely work on it after April 1st. So if you don't plan to finish this in the next couple of days @cdecker let me get started first and see how far I get in the first week of April maybe?

Do you have the plugin/command line code that you mention somewhere to check out, @renepickhardt, or would it be better to just start over?

@renepickhardt

This comment has been minimized.

Copy link
Collaborator Author

commented Mar 26, 2019

I have the simple funds plugin in my c Lightning plugin collection github but that is like the hello World plugin...

I have started to wrap the autopilot to plugin code but that is so much work in progress that it is probably saver to start over anyway. Before you read my code you would have reimplemented it.

@cdecker

This comment has been minimized.

Copy link
Member

commented Apr 29, 2019

The plugin has been ported to https://github.com/lightningd/plugins (lightningd/plugins@187c66a). So I'm closing this PR in favor of keeping any eventual changes there.

@cdecker cdecker closed this Apr 29, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.