Find route for messages

[thomash-acinq]

When sending a message, the postman can now ask the router to find a route using channels only. The same route is also used as a reply path when applicable.

[codecov-commenter]

Codecov Report

Merging #2656 (0d63fdc) into master (b084d73) will increase coverage by 0.08%.
The diff coverage is 95.39%.

❗ Your organization is not using the GitHub App Integration. As a result you may experience degraded service beginning May 15th. Please install the Github App Integration for your organization. Read more.

@@            Coverage Diff             @@
##           master    #2656      +/-   ##
==========================================
+ Coverage   85.85%   85.93%   +0.08%     
==========================================
  Files         214      215       +1     
  Lines       17591    17728     +137     
  Branches      723      752      +29     
==========================================
+ Hits        15102    15235     +133     
- Misses       2489     2493       +4     

Impacted Files	Coverage Δ
...ir-core/src/main/scala/fr/acinq/eclair/Setup.scala	75.42% <ø> (-0.14%)	⬇️
.../scala/fr/acinq/eclair/message/OnionMessages.scala	77.41% <0.00%> (-1.27%)	⬇️
...cala/fr/acinq/eclair/router/RouteCalculation.scala	94.56% <85.71%> (-0.87%)	⬇️
...r-core/src/main/scala/fr/acinq/eclair/Eclair.scala	57.87% <90.00%> (+2.10%)	⬆️
...c/main/scala/fr/acinq/eclair/message/Postman.scala	95.58% <94.00%> (-4.42%)	⬇️
.../src/main/scala/fr/acinq/eclair/router/Graph.scala	97.33% <96.49%> (-0.18%)	⬇️
...re/src/main/scala/fr/acinq/eclair/NodeParams.scala	93.81% <100.00%> (+0.16%)	⬆️
...a/fr/acinq/eclair/payment/offer/OfferManager.scala	92.59% <100.00%> (ø)
...la/fr/acinq/eclair/payment/send/OfferPayment.scala	75.00% <100.00%> (ø)
...cinq/eclair/remote/EclairInternalsSerializer.scala	97.82% <100.00%> (+0.04%)	⬆️
... and 4 more

... and 16 files with indirect coverage changes

[t-bast]

This turned out more complex than I expected! I proposed some simplifications and refactorings in #2663. We're getting closer to an MVP release!

[thomash-acinq]

I've merged your improvements but kept a separate routing algorithm for messages and payments. Routing messages is too different to be solved by the current payment routing algorithm and I find it much simpler to separate the two than to try to make something generic.

[thomash-acinq]

Now using Dijkstra for message routing too. We can prioritize big channels, old channels and penalize disabled edges (but still be able to consider them).
Rebased on master to fix a conflict.

[t-bast]

This looks mostly good to me, but it's hard to gauge the potential performance regressions, and this is quite a critical component...I didn't realize we'd need that much changes on the graph structure (but I agree that they make sense and are necessary). I think we should leave this PR out of the release, we should spend time testing it on our node before releasing it. We'll make the release first, and will merge this to master right after the release and after your performance benchmarks.

[thomash-acinq]

Some benchmarks:

DirectedGraph.makeGraph now takes 653ms instead of 225ms (x2.9)
.addEdges now takes 356ms instead of 2.676s to update the whole graph (x0.13)
yenKshortestPaths now takes 2.578s to find paths to the top 1000 nodes compared to 2.345s before (x1.1)

I don't think writing ad hoc code for DirectedGraph.makeGraph is worth it as it only called once at start up.
It looks like storing edges in a map instead of a list costs us 10% on the path-finding side and brings a 10x improvement on the update side.

[t-bast]

The code is looking good to me, that's really good work!
I'm going to spend more time on the benchmarks and I'll report early next week.

[t-bast]

I'm finding the same results by tweaking your performance benchmarks. I think the additional 10% cost on path-finding is acceptable (even though it would be nice if we're able to improve that in the future), I'm a bit more frustrated at the makeGraph performance hit, but I don't see an easy way of improving it. We really are adding more data in the Graph structure, so it's expected that it's more complex to instantiate it...but that's also something we can try to improve in the future, let's start with that!

[t-bast]

LGTM!