Run gateway as Core-Lightning plugin

[justinmoon]

With this PR user can generate a lightning invoice and get paid in via lightning gateway.

Still need to:

• Figure out all the parameters for route hints. I think we can do that in future PRs.
• We should update cln-plugin to allow plugin CLI arguments (minimint-cfg in our case) that don't have default values Plugin config options with no defaults ElementsProject/lightning#5369.
• This PR basically makes running the gateway mandatory. Previously you didn't have to run it because we always generated a fake node pubkey in client.json and it was never used so that didn't matter. So I just made it required for this PR because all the data structures assume the gateway is always there. Make lightning gateway optional #200 makes it actually optional but that's a deeper change so I separated it.

[jkitman]

Regarding the integration tests, does this change any behavior that would need testing?

If not, perhaps you can just modify the integrationtests/README.md so that users can run the LN plugin locally when testing with the real fixtures.

Additionally scripts/latencytest.sh probably needs a small update.

[justinmoon]

Regarding the integration tests, does this change any behavior that would need testing?

This doesn't change any behavior W.R.T. existing integration tests likereceive_lightning_payment_valid_preimage and receive_lightning_payment_invalid_preimage. The current tests do a lot of work calling internal functions in the right order and everything. It would be nice if a new integration test just (1) generated invoice, (2) had "ln2" pay it, and (3) asserted that balances eventually updated as expected. A true "integration test".

If not, perhaps you can just modify the integrationtests/README.md so that users can run the LN plugin locally when testing with the real fixtures.

One tricky thing here is that we'd need to pass the --minimint-cfg argument to the plugin which points to the directory with all the JSON configs and sled db files. AFAIK the integration tests just do everything in-memory?

Another tricky thing is that the plugin would really need to be reloadable per-test, because the federation details might change every test.

I think the best option here would be to add a --minimint-testing flag to the plugin which causes the plugin to run without running a gateway. Then add a new RPC command to the plugin to run a gateway based on JSON federation config pass as argument to this RPC command. This way the plugin could start before the integration tests have even started. Each test that needs to exercise the plugin can send over the federation configs used by that test.

This could happen in a follow-on PR.

Additionally scripts/latencytest.sh probably needs a small update.

I didn't even know about latencytest.sh! Will do.

[jkitman]

This doesn't change any behavior W.R.T. existing integration tests likereceive_lightning_payment_valid_preimage and receive_lightning_payment_invalid_preimage. The current tests do a lot of work calling internal functions in the right order and everything. It would be nice if a new integration test just (1) generated invoice, (2) had "ln2" pay it, and (3) asserted that balances eventually updated as expected. A true "integration test".

Perhaps it's sufficient that is handled via the shell scripts, although I guess we need to expose a new CLI command for that.

One tricky thing here is that we'd need to pass the --minimint-cfg argument to the plugin which points to the directory with all the JSON configs and sled db files. AFAIK the integration tests just do everything in-memory?

Ah, yes the current tests only use an in-memory DB, although I was meaning to enable a real DB for when running with the real fixtures.

Another tricky thing is that the plugin would really need to be reloadable per-test, because the federation details might change every test.
I think the best option here would be to add a --minimint-testing flag to the plugin which causes the plugin to run without running a gateway. Then add a new RPC command to the plugin to run a gateway based on JSON federation config pass as argument to this RPC command. This way the plugin could start before the integration tests have even started. Each test that needs to exercise the plugin can send over the federation configs used by that test.

Agreed, tearing down the LN fixtures and funding a channel would make the tests take forever, much faster just to reload the configs.

Just went throgh everyting with Justin, will fix some minor annoyances in follow up PRs