AB testing

[thomash-acinq]

Enable doing AB testing to find the best path finding parameters.

When a new payment needs to use path finding, it will be assigned randomly an experiment (which may be the control experiment) and use its set of parameters. When the payment finishes (either with a success or failure), we record metrics such as the fee we paid and the time the payment took.
After sending enough payments we can compute the success rate, average fee and average duration of payments for a given set of parameters and choose the best one.

Each experiments is assigned some percentage of the traffic. It is advised to start low with new experiments in case the new parameters are terrible but should then be increased to collect enough data timely. Experiments can be at 0% in which case they will never be used by live relay traffic but can still be used manually using the API.

[codecov-commenter]

Codecov Report

Merging #1930 (c4ecd4a) into master (663094e) will increase coverage by 0.04%.
The diff coverage is 19.50%.

@@            Coverage Diff            @@
##           master   #1930      +/-   ##
=========================================
+ Coverage    8.96%   9.00%   +0.04%     
=========================================
  Files         158     159       +1     
  Lines       12330   12423      +93     
  Branches      530     535       +5     
=========================================
+ Hits         1105    1119      +14     
- Misses      11225   11304      +79     

Impacted Files	Coverage Δ
...r-core/src/main/scala/fr/acinq/eclair/Eclair.scala	0.00% <0.00%> (ø)
...re/src/main/scala/fr/acinq/eclair/NodeParams.scala	2.39% <0.00%> (-0.20%)	⬇️
...main/scala/fr/acinq/eclair/db/DbEventHandler.scala	0.00% <0.00%> (ø)
.../main/scala/fr/acinq/eclair/db/DualDatabases.scala	0.00% <0.00%> (ø)
...c/main/scala/fr/acinq/eclair/db/pg/PgAuditDb.scala	0.00% <0.00%> (ø)
.../scala/fr/acinq/eclair/payment/PaymentEvents.scala	0.00% <ø> (ø)
...cala/fr/acinq/eclair/payment/relay/NodeRelay.scala	0.00% <0.00%> (ø)
...scala/fr/acinq/eclair/payment/send/Autoprobe.scala	0.00% <0.00%> (ø)
...clair/payment/send/MultiPartPaymentLifecycle.scala	0.00% <0.00%> (ø)
...r/acinq/eclair/payment/send/PaymentInitiator.scala	0.00% <0.00%> (ø)
... and 12 more

[pm47]

I like that it's a pretty simple change. Haven't reviewed the metrics+db part yet.

Note to self: requires updating eclair-front.

[pm47]

We can't just merge RouteParams and PathFindingConf, because they do not have the same meaning, as the special handling of includeLocalChannelCost shows. PathFindingConf is static but RouteParams depends on a particular route request.

How about (cc @t-bast interested in your opinion):

  case class PathFindingConf(randomizeRouteSelection: Boolean,
                             boundaries: SearchBoundaries,
                             weights: WeightRatios,
                             mpp: MultiPartParams)

  // new class that groups some parameters, not sure about Search
  case class SearchBoundaries(maxFeeBase: MilliSatoshi,
                              maxFeePct: Double,
                              maxRouteLength: Int,
                              maxCltv: CltvExpiryDelta)

  case class RouteParams(conf: PathFindingConf,
                         includeLocalChannelCost: Boolean)

[pm47]

If we go with this, it would make sense to create subsections:

    defaults {
      randomize-route-selection = true

      boundaries {
        max-route-length = 6
        max-cltv = 1008
        fee-threshold-sat = 21
        max-fee-pct = 0.03
      }

      ratios {
        base = 0.0
        cltv = 0.05
        channel-age = 0.4
        channel-capacity = 0.55
        hop-cost-base-msat = 500
        hop-cost-millionths = 200
      }

      mpp {
        min-amount-satoshis = 15000
        max-parts = 5
      }
    }

[thomash-acinq]

I've changed PathFindingConf to be a nested case class as you suggested but I kept RouteParams as a separate flat case class because many of its fields are modified in many places and it's not possible to modify nested fields in a nice way.

[pm47]

Take a look at quicklens:

import com.softwaremill.quicklens.ModifyPimp
val aliceParams2 = Alice.nodeParams.modify(_.routerConf.requestNodeAnnouncements).setTo(true)

[t-bast]

Great feature, I really like the way it's presented and configurable in reference.conf.
There are important details about what exact events we should measure that we need to discuss, the details can be subtle and may highly influence the resulting data.

[t-bast]

LGTM 🚀