ProbabilisticScorer optimizations #1375

jkczyz · 2022-03-21T21:18:13Z

Without recent knowledge of channel liquidity balances, ProbabilisticScorer tends to give low penalties unless the payment amount is a large portion of a channel's capacity. Provide an option that gives a linear penalty increase as the success probability decreases, which may be useful in such scenarios.

~~Additionally, give a base penalty such that shorter paths are preferred over longer ones.~~

Make a few optimizations to ProbabilisticScorer including a per hop base_penalty_msat. Also, fix a integer overflow that will cause the log approximation to panic.

Closes #1372

codecov-commenter · 2022-03-21T21:45:54Z

Codecov Report

Merging #1375 (4af9d44) into main (cb1d795) will decrease coverage by 0.01%.
The diff coverage is 90.00%.

❗ Current head 4af9d44 differs from pull request most recent head 9596cc7. Consider uploading reports for the commit 9596cc7 to get more accurate results

@@            Coverage Diff             @@
##             main    #1375      +/-   ##
==========================================
- Coverage   90.77%   90.75%   -0.02%     
==========================================
  Files          73       73              
  Lines       41062    41091      +29     
  Branches    41062    41091      +29     
==========================================
+ Hits        37272    37291      +19     
- Misses       3790     3800      +10

Impacted Files	Coverage Δ
lightning/src/routing/scoring.rs	`94.04% <90.00%> (-0.26%)`	⬇️
lightning/src/ln/functional_tests.rs	`97.06% <0.00%> (-0.10%)`	⬇️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update cb1d795...9596cc7. Read the comment docs.

TheBlueMatt

LGTM. A few comments about docs and some minor nits that don't matter much.

lightning/src/routing/scoring.rs

TheBlueMatt

LGTM post-squash.

TheBlueMatt · 2022-03-23T16:56:03Z

lightning/src/routing/scoring.rs

+	///
+	/// The liquidity penalty is effectively limited to `2 * liquidity_penalty_multiplier_msat`.
+	///
+	/// Default value: 10,000 msat


Note this default is probably much too high for the linear case - I don't think we should change it as long as the cost_function default is log, but maybe we want to call out in the docs that it should be changed if the cost function is?

TheBlueMatt · 2022-03-23T16:56:27Z

lightning/src/routing/scoring.rs

 	///
-	/// Default value: 10,000 msat
+	/// Default value: [`ProbabilisticScoringCostFunction::NegativeLogSuccessProbability`]


Do we want to consider changing this? I think log only makes sense if you're doing large payment volume or probing, which I don't think most of our users are?

TheBlueMatt · 2022-03-25T01:20:49Z

Okay, so thinking about the defaults here - lets assume that most channels are 100k-5m in capacity, and that our defaults should target users with 0 learning (or at least all info having decayed) cause they don't have much payment volume. Rene's work concluded that, by far, the failure probability of a hop is, roughly, amount/capacity so lets take that as a given, but also that real-world experience seems to indicate that there is some nontrivial probability of a channel being offline that is in excess of that probability. Assume we want to cap the failure penalty for any hop to 20 sats (ie we're willing to pay 20 sats to avoid taking a given path, which is probably about the max we should want for a small payment of, say, 1000 sats).

Lets consider five possible settings -

linear with a 20 sats cap,
log with a 99% failure cap and a 20 sats cap at 99%,
log with a fixed 6.25% failure rate baked in per-hop, with a 10 sats multiplier
log with a fixed 12.5% failure rate baked in per-hop, with a 20 sats multiplier as the cap is effectively 87.5,
linear with a 20 sats cap with a fixed per-hop penalty of 0.5sats.

Lets consider each setting at 3 hops, 5 hops, and 10 hops for values of 1k, 10k, 50k sats across 100k, and 1m channels.

1k-over-1m (0.1% failure rate)

---	expected failure % (Rene/+5%/+10%)	linear	log	log w/ 6.25% addn'l (10x)	log w/ 12.5% addn'l (20x)	linear w/ 0.5 per-hop
3 hops	0.3%/14.5%/27.3%	0.06 sats	0.01 sats	0.85 sats	3.51 sats	1.56 sats
5 hops	0.5%/23.0%/41.3%	0.1 sats	0.02 sats	1.42 sats	5.85 sats	2.6 sats
10 hops	1.0%/40.8%/65.5%	0.2 sats	0.04 sats	2.85 sats	11.7 sats	5.2 sats

10k-over-1m (1% failure rate, or 10k-over-1m)

---	expected failure % (Rene/+5%/+10%)	linear	log	log w/ 6.25% addn'l (10x)	log w/ 12.5% addn'l (20x)	linear w/ 0.5 per-hop
3 hops	3.0%/16.9%/29.5%	0.6 sats	0.13 sats	0.98 sats	3.78 sats	2.1 sats
5 hops	4.9%/26.6%/44.2%	1.0 sats	0.22 sats	1.63 sats	6.3 sats	3.5 sats
10 hops	9.6%/46.1%/68.8%	2.0 sats	0.44 sats	3.27 sats	12.6 sats	7.0 sats

50k-over-1m (5% failure probability)

---	expected failure % (Rene/+5%/+10%)	linear	log	log w/ 6.25% addn'l (10x)	log w/ 12.5% addn'l (20x)	linear w/ 0.5 per-hop
3 hops	14.3%/27.1%/38.6%	3.0 sats	0.67 sats	1.55 sats	5.01 sats	4.5 sats
5 hops	22.6%/41.0%/55.6%	5.0 sats	1.11 sats	2.59 sats	8.35 sats	7.5 sats
10 hops	40.1%/65.1%/80.3%	10.0 sats	2.23 sats	5.18 sats	16.71 sats	15.0 sats

10k-over-100k (10% failure probability)

---	expected failure % (Rene/+5%/+10%)	linear	log	log w/ 6.25% addn'l (10x)	log w/ 12.5% addn'l (20x)	linear w/ 0.5 per-hop
3 hops	27.1%/38.6%/48.8%	6.0 sats	1.37 sats	2.31 sats	6.64 sats	7.5 sats
5 hops	41.0%/55.6%/67.2%	10.0 sats	2.29 sats	3.85 sats	11.07 sats	12.5 sats
10 hops	65.1%/80.3%/89.3%	20.0 sats	4.58 sats	7.7 sats	22.14 sats	25.0 sats

50k-over-100k (50% failure probability)

---	expected failure % (Rene/+5%/+10%)	linear	log	log w/ 6.25% addn'l (10x)	log w/ 12.5% addn'l (20x)	linear w/ 0.5 per-hop
3 hops	87.5%/90.9%/93.6%	30.0 sats	9.03 sats	10.77 sats	25.56 sats	31.5 sats
5 hops	96.9%/98.2%/99.0%	50.0 sats	15.05 sats	17.95 sats	42.6 sats	52.5 sats
10 hops	99.9%/100.0%/100.0%	100.0 sats	30.1 sats	35.9 sats	85.19 sats	105.0 sats

It think its clear here that log is just completely absurdly low - it reserves half the range for the 90%-99% failure cases, which avoids large penalties through nearly all the data above. Even if we were to double the the penalties applied in log it'd still assign very low penalties, see eg the 10% per-hop failure table, where even doubling log would get us ~10 sats in penalty for a path that has a 65-80% failure probability. Log 6.25 has a similar issue, although to a smaller extent, with initial per-hop penalties much higher even at low failure rates.

Linear also seems pretty broken, especially in the 10-hops case for lower probability, 0.2 sats for a 10-hop path is unlikely to work out well.

Log 12.5 and linear 0.5-per-hop are both reasonable contenders, but I slightly prefer linear+0.5 here - the high penalties assigned in the 0.1% failure rate table for log 12.5 seem much too conservative even though we end up in almost the same place in the 10% and 50% tables.

I propose we default to linear with a 0.5sat per-hop penalty

jkczyz · 2022-03-25T02:35:10Z

I tried log w/ 0.5 sat per hop and a 50 sat multiplier, and it's pretty close to linear w/ 0.5 per-hop, especially at 1-10% failure probability.

1k-over-1m (0.1% failure rate)

---	expected failure % (Rene/+5%/+10%)	linear	log	log w/ 6.25% addn'l (10x)	log w/ 12.5% addn'l (20x)	linear w/ 0.5 per-hop	log w/ 0.5 per hop (50x)
3 hops	0.3%/14.5%/27.3%	0.06 sats	0.13 sats	0.85 sats	3.51 sats	1.56 sats	2.15 sats
5 hops	0.5%/23.0%/41.3%	0.1 sats	0.22 sats	1.42 sats	5.85 sats	2.6 sats	3.59 sats
10 hops	1.0%/40.8%/65.5%	0.2 sats	0.44 sats	2.85 sats	11.7 sats	5.2 sats	7.18 sats

10k-over-1m (1% failure rate, or 10k-over-1m)

---	expected failure % (Rene/+5%/+10%)	linear	log	log w/ 6.25% addn'l (10x)	log w/ 12.5% addn'l (20x)	linear w/ 0.5 per-hop	log w/ 0.5 per hop (50x)
3 hops	3.0%/16.9%/29.5%	0.6 sats	0.13 sats	0.98 sats	3.78 sats	2.1 sats	2.15 sats
5 hops	4.9%/26.6%/44.2%	1.0 sats	0.22 sats	1.63 sats	6.3 sats	3.5 sats	3.59 sats
10 hops	9.6%/46.1%/68.8%	2.0 sats	0.44 sats	3.27 sats	12.6 sats	7.0 sats	7.18 sats

50k-over-1m (5% failure probability)

---	expected failure % (Rene/+5%/+10%)	linear	log	log w/ 6.25% addn'l (10x)	log w/ 12.5% addn'l (20x)	linear w/ 0.5 per-hop	log w/ 0.5 per hop (50x)
3 hops	14.3%/27.1%/38.6%	3.0 sats	0.67 sats	1.55 sats	5.01 sats	4.5 sats	4.84 sats
5 hops	22.6%/41.0%/55.6%	5.0 sats	1.11 sats	2.59 sats	8.35 sats	7.5 sats	8.07 sats
10 hops	40.1%/65.1%/80.3%	10.0 sats	2.23 sats	5.18 sats	16.71 sats	15.0 sats	16.14 sats

10k-over-100k (10% failure probability)

---	expected failure % (Rene/+5%/+10%)	linear	log	log w/ 6.25% addn'l (10x)	log w/ 12.5% addn'l (20x)	linear w/ 0.5 per-hop	log w/ 0.5 per hop (50x)
3 hops	27.1%/38.6%/48.8%	6.0 sats	1.37 sats	2.31 sats	6.64 sats	7.5 sats	8.36 sats
5 hops	41.0%/55.6%/67.2%	10.0 sats	2.29 sats	3.85 sats	11.07 sats	12.5 sats	13.94 sats
10 hops	65.1%/80.3%/89.3%	20.0 sats	4.58 sats	7.7 sats	22.14 sats	25.0 sats	27.88 sats

50k-over-100k (50% failure probability)

---	expected failure % (Rene/+5%/+10%)	linear	log	log w/ 6.25% addn'l (10x)	log w/ 12.5% addn'l (20x)	linear w/ 0.5 per-hop	log w/ 0.5 per hop (50x)
3 hops	87.5%/90.9%/93.6%	30.0 sats	9.03 sats	10.77 sats	25.56 sats	31.5 sats	46.65 sats
5 hops	96.9%/98.2%/99.0%	50.0 sats	15.05 sats	17.95 sats	42.6 sats	52.5 sats	77.76 sats
10 hops	99.9%/100.0%/100.0%	100.0 sats	30.1 sats	35.9 sats	85.19 sats	105.0 sats	155.51 sats

TheBlueMatt · 2022-03-25T04:17:01Z

First of all, I apologize, I had the wrong limit on the logs in the original script, so the 0.1% failure rate table is wrong. Its corrected below (and in the pasted script on the issue).

Hmm, I realize the parameter we're missing in all of this is the payment amount - is 150 sats penalty too much? If you're sending a million sats, probably not, if you're sending 50k sats, probably. I don't think we need to address that there, but in whatever followup we do to pass the scorer more information we should also pass the payment amount and give the ProbabilisticScorer a version of the penalty that is multiplied by the payment amount instead of being static.

As for the specific direction to go in here, I added five more options below (swapped the multipliers in the logs with additional probability of failure and added a 30x/40x variant of your proposal).

The 40x variant of your proposal is basically the same as the linear+0.5 proposal for all the numbers that matter (<=10%), and I have to admit I prefer it over 30/50x in the 5% and 10% tables, though of course its all marginal enough to basically be a wash. I hadn't originally thought about doing the linear addition in the log versions because its somewhat nonsensical from a theoretical perspective - why linearize the probability but then add a per-hop penalty - but if it means we can drop the linear option entirely and not branch in the scorer, sure, lets do it.

1k-over-1m (0.1% failure rate)

---	expected failure % (Rene/+5%/+10%)	linear	log	log w/ 6.25% addn'l (20x)	log w/ 12.5% addn'l (10x)	linear w/ 0.5 per-hop	log w/ 0.5 per hop (30x)	log w/ 0.5 per hop (40x)	log w/ 0.5 per hop (50x)
3 hops	0.3%/14.5%/27.3%	0.06 sats	0.01 sats	1.71 sats	1.75 sats	1.56 sats	1.54 sats	1.55 sats	1.57 sats
5 hops	0.5%/23.0%/41.3%	0.1 sats	0.02 sats	2.85 sats	2.92 sats	2.6 sats	2.57 sats	2.59 sats	2.61 sats
10 hops	1.0%/40.8%/65.5%	0.2 sats	0.04 sats	5.7 sats	5.85 sats	5.2 sats	5.13 sats	5.17 sats	5.22 sats

10k-over-1m (1% failure rate, or 10k-over-1m)

---	expected failure % (Rene/+5%/+10%)	linear	log	log w/ 6.25% addn'l (20x)	log w/ 12.5% addn'l (10x)	linear w/ 0.5 per-hop	log w/ 0.5 per hop (30x)	log w/ 0.5 per hop (40x)	log w/ 0.5 per hop (50x)
3 hops	3.0%/16.9%/29.5%	0.6 sats	0.13 sats	1.96 sats	1.89 sats	2.1 sats	1.89 sats	2.02 sats	2.15 sats
5 hops	4.9%/26.6%/44.2%	1.0 sats	0.22 sats	3.27 sats	3.15 sats	3.5 sats	3.15 sats	3.37 sats	3.59 sats
10 hops	9.6%/46.1%/68.8%	2.0 sats	0.44 sats	6.54 sats	6.3 sats	7.0 sats	6.31 sats	6.75 sats	7.18 sats

50k-over-1m (5% failure probability)

---	expected failure % (Rene/+5%/+10%)	linear	log	log w/ 6.25% addn'l (20x)	log w/ 12.5% addn'l (10x)	linear w/ 0.5 per-hop	log w/ 0.5 per hop (30x)	log w/ 0.5 per hop (40x)	log w/ 0.5 per hop (50x)
3 hops	14.3%/27.1%/38.6%	3.0 sats	0.67 sats	3.11 sats	2.51 sats	4.5 sats	3.5 sats	4.17 sats	4.84 sats
5 hops	22.6%/41.0%/55.6%	5.0 sats	1.11 sats	5.18 sats	4.18 sats	7.5 sats	5.84 sats	6.96 sats	8.07 sats
10 hops	40.1%/65.1%/80.3%	10.0 sats	2.23 sats	10.37 sats	8.35 sats	15.0 sats	11.68 sats	13.91 sats	16.14 sats

10k-over-100k (10% failure probability)

---	expected failure % (Rene/+5%/+10%)	linear	log	log w/ 6.25% addn'l (20x)	log w/ 12.5% addn'l (10x)	linear w/ 0.5 per-hop	log w/ 0.5 per hop (30x)	log w/ 0.5 per hop (40x)	log w/ 0.5 per hop (50x)
3 hops	27.1%/38.6%/48.8%	6.0 sats	1.37 sats	4.62 sats	3.32 sats	7.5 sats	5.62 sats	6.99 sats	8.36 sats
5 hops	41.0%/55.6%/67.2%	10.0 sats	2.29 sats	7.7 sats	5.53 sats	12.5 sats	9.36 sats	11.65 sats	13.94 sats
10 hops	65.1%/80.3%/89.3%	20.0 sats	4.58 sats	15.4 sats	11.07 sats	25.0 sats	18.73 sats	23.3 sats	27.88 sats

50k-over-100k (50% failure probability)

---	expected failure % (Rene/+5%/+10%)	linear	log	log w/ 6.25% addn'l (20x)	log w/ 12.5% addn'l (10x)	linear w/ 0.5 per-hop	log w/ 0.5 per hop (30x)	log w/ 0.5 per hop (40x)	log w/ 0.5 per hop (50x)
3 hops	87.5%/90.9%/93.6%	30.0 sats	9.03 sats	21.54 sats	12.78 sats	31.5 sats	28.59 sats	37.62 sats	46.65 sats
5 hops	96.9%/98.2%/99.0%	50.0 sats	15.05 sats	35.9 sats	21.3 sats	52.5 sats	47.65 sats	62.71 sats	77.76 sats
10 hops	99.9%/100.0%/100.0%	100.0 sats	30.1 sats	71.8 sats	42.6 sats	105.0 sats	95.31 sats	125.41 sats	155.51 sats

(75% failure probability)

---	expected failure % (Rene/+5%/+10%)	linear	log	log w/ 6.25% addn'l (20x)	log w/ 12.5% addn'l (10x)	linear w/ 0.5 per-hop	log w/ 0.5 per hop (30x)	log w/ 0.5 per hop (40x)	log w/ 0.5 per hop (50x)
3 hops	98.4%/99.2%/99.7%	45.0 sats	18.06 sats	43.62 sats	27.09 sats	46.5 sats	55.69 sats	73.75 sats	91.81 sats
5 hops	99.9%/100.0%/100.0%	75.0 sats	30.1 sats	72.7 sats	45.15 sats	77.5 sats	92.81 sats	122.91 sats	153.01 sats
10 hops	100.0%/100.0%/100.0%	150.0 sats	60.21 sats	145.4 sats	90.31 sats	155.0 sats	185.62 sats	245.82 sats	306.03 sats

jkczyz · 2022-03-25T13:32:01Z

Ok, so drop the cost function variant and decide on a default multiplier. You're thinking 40x? I think I tried that and 50x on a couple numbers at first and just used 50x in the script because it was a clean 0-100 sat range. But either is fine by me. Could even do a power of two to make it a shift, but then it's either 32 or 64, which is probably either too low or too high.

Serializing scorer parameters makes it difficult to experiment with different settings.

ProbabilisticScorer tends to prefer longer routes to shorter ones. Make the default scoring behavior include a customizable base penalty to avoid longer routes.

When a routing hint is given in an invoice, the effective capacity of the channel is assumed to be infinite (i.e., u64::max_value) if the hop is private. Adding 1 to this in the success probability calculation will cause an overflow and ultimately an `index out of bounds panic` in log10_times_1024. This was not an issue with using log10 because the use of f64 would give infinite which casts to 0 for u64.

jkczyz · 2022-03-25T14:35:09Z

Could even do a power of two to make it a shift, but then it's either 32 or 64, which is probably either too low or too high.

Ah, never mind on this as we are actually using msats.

This reduce a branch in the 0 and u6::max_value cases.

Using a larger multiplier gives more reasonable penalties for larger success probabilities.

TheBlueMatt · 2022-03-25T16:32:18Z

Ok, so drop the cost function variant and decide on a default multiplier.

Yea, I think that's fine. Of course we basically picked a log probability set which closely hews to linear for most reasonable values, so dunno if it was worth it vs dropping log and using linear to begin with, but, its the code we have, and it is kinda nice that it gets really huge on the upper end of the probability range 🤷‍♂️ .

Ah, never mind on this as we are actually using msats.

Well, then we're looking at 64K or 32K, so same thing, basically :p. Yea, sadly I don't think that's granular enough.

jkczyz · 2022-03-25T16:35:04Z

Forced push to kick CI.

Yea, I think that's fine. Of course we basically picked a log probability set which closely hews to linear for most reasonable values, so dunno if it was worth it vs dropping log and using linear to begin with, but, its the code we have, and it is kinda nice that it gets really huge on the upper end of the probability range 🤷‍♂️ .

I think the benefit of the log is what you mentioned in #1375 (comment).

TheBlueMatt · 2022-03-25T16:42:15Z

I think the benefit of the log is what you mentioned in #1375 (comment).

Right, unless you use the per-hop penalty, which doesn't fit into that model. 🤷‍♂️

valentinewallace · 2022-03-25T20:29:06Z

lightning/src/routing/scoring.rs

+	/// A multiplier used in conjunction with the negative `log10` of the channel's success
+	/// probability for a payment to determine the liquidity penalty.
+	///
+	/// The penalty is based in part by the knowledge learned from prior successful and unsuccessful


nit: s/in part by/in part on

jkczyz force-pushed the 2022-03-scorer-followup branch from e5874fe to e895c25 Compare March 22, 2022 00:18

TheBlueMatt reviewed Mar 22, 2022

View reviewed changes

lightning/src/routing/scoring.rs Outdated Show resolved Hide resolved

lightning/src/routing/scoring.rs Outdated Show resolved Hide resolved

lightning/src/routing/scoring.rs Outdated Show resolved Hide resolved

lightning/src/routing/scoring.rs Show resolved Hide resolved

TheBlueMatt added this to the 0.0.106 milestone Mar 22, 2022

jkczyz force-pushed the 2022-03-scorer-followup branch from e895c25 to ab7f851 Compare March 22, 2022 17:54

TheBlueMatt reviewed Mar 22, 2022

View reviewed changes

jkczyz force-pushed the 2022-03-scorer-followup branch from ab7f851 to 5d3e90b Compare March 22, 2022 19:02

TheBlueMatt reviewed Mar 23, 2022

View reviewed changes

jkczyz force-pushed the 2022-03-scorer-followup branch from 5d3e90b to b4324c0 Compare March 24, 2022 23:46

TheBlueMatt mentioned this pull request Mar 25, 2022

Make log optional in ProbabilisticScorer #1372

Closed

TheBlueMatt mentioned this pull request Mar 25, 2022

Pass EffectiveCapacity Through to Scorers #1386

Closed

jkczyz added 3 commits March 25, 2022 08:53

Don't serialize FixedPenaltyScorer parameters

188f919

Serializing scorer parameters makes it difficult to experiment with different settings.

Add a base penalty to ProbabilisticScorer

5b36449

ProbabilisticScorer tends to prefer longer routes to shorter ones. Make the default scoring behavior include a customizable base penalty to avoid longer routes.

Move max penalty cap in ProbabilisticScorer

c6348cb

This reduce a branch in the 0 and u6::max_value cases.

jkczyz force-pushed the 2022-03-scorer-followup branch from b4324c0 to 4af9d44 Compare March 25, 2022 14:42

Increase default liquidity_penalty_multiplier_msat

9596cc7

Using a larger multiplier gives more reasonable penalties for larger success probabilities.

jkczyz force-pushed the 2022-03-scorer-followup branch from 4af9d44 to 9596cc7 Compare March 25, 2022 16:31

TheBlueMatt approved these changes Mar 25, 2022

View reviewed changes

TheBlueMatt added the Seeking Code Review label Mar 25, 2022

jkczyz changed the title ~~Linear cost function ProbabilisticScorer option~~ ProbabilisticScorer optimizations Mar 25, 2022

valentinewallace approved these changes Mar 25, 2022

View reviewed changes

valentinewallace merged commit eb68b5f into lightningdevkit:main Mar 25, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ProbabilisticScorer optimizations #1375

ProbabilisticScorer optimizations #1375

jkczyz commented Mar 21, 2022 •

edited

Loading

codecov-commenter commented Mar 21, 2022 •

edited

Loading

TheBlueMatt left a comment

TheBlueMatt left a comment

TheBlueMatt Mar 23, 2022

TheBlueMatt Mar 23, 2022

TheBlueMatt commented Mar 25, 2022 •

edited

Loading

jkczyz commented Mar 25, 2022 •

edited

Loading

TheBlueMatt commented Mar 25, 2022

jkczyz commented Mar 25, 2022

jkczyz commented Mar 25, 2022

TheBlueMatt commented Mar 25, 2022

jkczyz commented Mar 25, 2022

TheBlueMatt commented Mar 25, 2022

valentinewallace Mar 25, 2022

ProbabilisticScorer optimizations #1375

ProbabilisticScorer optimizations #1375

Conversation

jkczyz commented Mar 21, 2022 • edited Loading

codecov-commenter commented Mar 21, 2022 • edited Loading

Codecov Report

TheBlueMatt left a comment

Choose a reason for hiding this comment

TheBlueMatt left a comment

Choose a reason for hiding this comment

TheBlueMatt Mar 23, 2022

Choose a reason for hiding this comment

TheBlueMatt Mar 23, 2022

Choose a reason for hiding this comment

TheBlueMatt commented Mar 25, 2022 • edited Loading

jkczyz commented Mar 25, 2022 • edited Loading

TheBlueMatt commented Mar 25, 2022

jkczyz commented Mar 25, 2022

jkczyz commented Mar 25, 2022

TheBlueMatt commented Mar 25, 2022

jkczyz commented Mar 25, 2022

TheBlueMatt commented Mar 25, 2022

valentinewallace Mar 25, 2022

Choose a reason for hiding this comment

jkczyz commented Mar 21, 2022 •

edited

Loading

codecov-commenter commented Mar 21, 2022 •

edited

Loading

TheBlueMatt commented Mar 25, 2022 •

edited

Loading

jkczyz commented Mar 25, 2022 •

edited

Loading