CLN gateway is 2x slower than LND gateway

[douglaz]

If we run the two-gateways strategy like:

fedimint-load-test-tool --users 10 ln-circular-load-test --test-duration-secs 130 --strategy two-gateways

We can see that there is an asymmetry whether the payment is CLN->LND or LND->CLN:

63 gateway_payment_received_started_CLN: avg 2.555825485s, median 2.527213519s, max 7.907194426s, min 1.435131552s
66 gateway_payment_received_started_LND: avg 6.45312008s, median 6.163989781s, max 12.240447867s, min 2.529054994s
72 gateway_payment_received_started_CLN: avg 2.437363119s, median 2.387419535s, max 7.761276333s, min 1.494405323s
69 gateway_payment_received_started_LND: avg 5.16867298s, median 4.694154243s, max 13.039080421s, min 1.480030768s
67 gateway_payment_received_started_CLN: avg 2.330486165s, median 2.381701614s, max 7.789901608s, min 1.261493599s
71 gateway_payment_received_started_LND: avg 5.809837554s, median 5.460453137s, max 13.53109642s, min 2.53615362s
72 gateway_payment_received_started_CLN: avg 2.319183463s, median 2.459970173s, max 3.064128031s, min 1.332991793s
71 gateway_payment_received_started_LND: avg 5.119090704s, median 4.604394785s, max 12.747417679s, min 2.140277871s

Note: on the above metric if it says gateway_payment_received_started_LND then it means that this payment was started using LND (lncli) paying a CLN gateway. So probably the CLN gateway is slow, not the lncli

Note 2: this is notable only if there is more than one user trying to pay at the same time. With just one user there is no significant difference.

[douglaz]

My findings so far:

• Using feat: improved load-test-metrics #3717 I found out that the CLN slowdown only happens when the gateway is receiving a payment
• It seems that the main interceptor loop is somewhat synchronous for CLN (it's only able to get the next HTLC from the stream when the previous has been fully processed) while it's asynchronous for LND (it doesn't matter if the previous HTLC has been processed or not)

[elsirion]

According to @cdecker htlc_accepted is explicitly allowed to hold multiple HTLCs at a time. So the error has to be in our code or the cln_rpc crate, but in neither I could find an obvious problem from just looking at it.

[justinmoon]

dev call: we need to change our code handling htlc in gatewayd. @douglaz will provide link to the code.

[douglaz]

This is where we await inside the htlc interception:

fedimint/gateway/ln-gateway/src/bin/cln_extension.rs

Line 621 in 681e6cc

tokio::time::timeout(Duration::from_secs(30), async {