Integrate trustedcoin clightning plugin #597

seberm · 2023-03-03T23:48:26Z

This PR tries to add a trustedcoin support into the nix-bitcoin.

The work is based on PRs:

The code is rebased on current master.

I have removed the clightning.useBcli option and replaced it with clightning.useBitcoind. When setting this option to true, the bitcoind unit is forcerly disabled and clightning.plugins.trustedcoin.enabled is set to true as suggested in #533 (comment).

I also tried to add some basic tests.

@erikarvstedt , could you please take a look and give me some suggestions? Please keep in mind this is only a draft but I would like to move this feature forward.

Thanks!

CC: @neverupdate , @MatthewCroughan , @elsirion
Closes #533

modules/clightning.nix

seberm · 2023-03-18T22:34:49Z

Hello everyone,
I've pushed another set of changes.

I have completely removed the useBitcoind option from the clightning module. Setting the services.clightning.plugins.trustedcoin option to true should be enough to enable trustedcoin. The plugin will then add a disable-plugin=bcli into clightning's configuration automatically.

Let me know what you think.

Thanks!
O.

seberm · 2023-03-18T22:14:19Z

.cirrus.yml

@@ -27,6 +27,7 @@ task:
          - scenario: default
          - scenario: netns
          - scenario: netnsRegtest
+          - scenario: trustedcoin


I am still not sure if it makes sense to run trustedcoin scenario regurarly in CI.

elsirion

Looks good to me

modules/clightning-plugins/trustedcoin.nix

test/tests.py

jonasnick

Thanks again for your work on this. Left a few comments. Did you test whether this actually works and you can use clightning with an unsynced bitcoind?

pkgs/trustedcoin/default.nix

pkgs/trustedcoin/get-sha256.sh

README.md

test/tests.nix

modules/clightning-plugins/trustedcoin.nix

seberm · 2023-04-07T20:57:42Z

Thanks again for your work on this. Left a few comments. Did you test whether this actually works and you can use clightning with an unsynced bitcoind?

Hello @jonasnick,
a did some tests in my testing VM. The bitcoind is syncing:

[root@default-test:~]# bitcoin-cli -getinfo
Chain: main
Blocks: 18147
Headers: 784393
Verification progress: ▒░░░░░░░░░░░░░░░░░░░░ 0.0022%
Difficulty: 1

Network: in 0, out 1, total 1
Version: 240001
Time offset (s): 0
Proxies: 127.0.0.1:9050 (ipv4, ipv6, onion, cjdns)
Min tx relay fee rate (BTC/kvB): 0.00001000

Warnings: (none)

But the clightning service is restarting on timeout (Failed with result 'timeout'.). The trustedcoin plugin seems to be working correctly, because the tip is updated. But the ExecStartPost script is probably failing (start-post operation timed out. Terminating.) because the clightning's RPC is not working. See the logs:

The ExecStartPost script:

[root@default-test:~]# cat /nix/store/rkg4sfr6km866afdqaq0d6xipbphqxqf-unit-script-clightning-post-start/bin/clightning-post-start 
#!/nix/store/96ky1zdkpq871h2dlk198fz0zvklr1dr-bash-5.1-p16/bin/bash
set -e           
while [[ ! -e /var/lib/clightning/bitcoin/lightning-rpc ]]; do
    sleep 0.1            
done                                                       
# Needed to enable lightning-cli for users with group 'clightning'
chmod g+x /var/lib/clightning/bitcoin

The output of journalctl -u clightning.service:

Apr 07 22:18:15 default-test systemd[1]: Starting clightning.service...
Apr 07 22:18:15 default-test lightningd[2914]: INFO    plugin-manager: /nix/store/1iia6a4zslg95iqw3g1w9xyp18xfjqx8-clightning-23.02/libexec/c-lightning/plugins/bcli: disabled via disable-plugin
Apr 07 22:18:15 default-test lightningd[2932]: plugin-trustedcoin  initialized plugin v0.6.1
Apr 07 22:18:15 default-test lightningd[2932]: plugin-trustedcoin  bitcoind RPC working, will use that with highest priority and fall back to block explorers if it fails.
Apr 07 22:18:15 default-test lightningd[2932]: plugin-trustedcoin tip: 0
Apr 07 22:18:15 default-test lightningd[2932]: plugin-trustedcoin returning block 0, 000000000019d6689c085ae165…, 285 bytes
Apr 07 22:19:45 default-test systemd[1]: clightning.service: start-post operation timed out. Terminating.
Apr 07 22:19:45 default-test systemd[1]: clightning.service: Control process exited, code=killed, status=15/TERM
Apr 07 22:19:45 default-test systemd[1]: clightning.service: Failed with result 'timeout'.
Apr 07 22:19:45 default-test systemd[1]: Failed to start clightning.service.
Apr 07 22:19:45 default-test systemd[1]: clightning.service: Consumed 1.779s CPU time, read 0B from disk, written 64.0K to disk, received 6.8K IP traffic, sent 5.5K IP traffic.
Apr 07 22:19:55 default-test systemd[1]: clightning.service: Scheduled restart job, restart counter is at 3.
Apr 07 22:19:55 default-test systemd[1]: Stopped clightning.service.
Apr 07 22:19:55 default-test systemd[1]: clightning.service: Consumed 1.779s CPU time, read 0B from disk, written 64.0K to disk, received 6.8K IP traffic, sent 5.5K IP traffic.
Apr 07 22:19:55 default-test systemd[1]: Starting clightning.service...
Apr 07 22:19:56 default-test lightningd[3857]: INFO    plugin-manager: /nix/store/1iia6a4zslg95iqw3g1w9xyp18xfjqx8-clightning-23.02/libexec/c-lightning/plugins/bcli: disabled via disable-plugin
Apr 07 22:19:56 default-test lightningd[3875]: plugin-trustedcoin  initialized plugin v0.6.1
Apr 07 22:19:56 default-test lightningd[3875]: plugin-trustedcoin  bitcoind RPC working, will use that with highest priority and fall back to block explorers if it fails.
Apr 07 22:19:56 default-test lightningd[3875]: plugin-trustedcoin tip: 28041
Apr 07 22:21:26 default-test systemd[1]: clightning.service: start-post operation timed out. Terminating.
Apr 07 22:21:26 default-test systemd[1]: clightning.service: Control process exited, code=killed, status=15/TERM
Apr 07 22:21:26 default-test systemd[1]: clightning.service: Failed with result 'timeout'.
Apr 07 22:21:26 default-test systemd[1]: Failed to start clightning.service.
Apr 07 22:21:26 default-test systemd[1]: clightning.service: Consumed 1.745s CPU time, received 5.8K IP traffic, sent 4.9K IP traffic.
Apr 07 22:21:36 default-test systemd[1]: clightning.service: Scheduled restart job, restart counter is at 4.
Apr 07 22:21:36 default-test systemd[1]: Stopped clightning.service.
Apr 07 22:21:36 default-test systemd[1]: clightning.service: Consumed 1.745s CPU time, received 5.8K IP traffic, sent 4.9K IP traffic.
Apr 07 22:21:36 default-test systemd[1]: Starting clightning.service...
Apr 07 22:21:36 default-test lightningd[4786]: INFO    plugin-manager: /nix/store/1iia6a4zslg95iqw3g1w9xyp18xfjqx8-clightning-23.02/libexec/c-lightning/plugins/bcli: disabled via disable-plugin
Apr 07 22:21:36 default-test lightningd[4804]: plugin-trustedcoin  initialized plugin v0.6.1
Apr 07 22:21:36 default-test lightningd[4804]: plugin-trustedcoin  bitcoind RPC working, will use that with highest priority and fall back to block explorers if it fails.
Apr 07 22:21:36 default-test lightningd[4804]: plugin-trustedcoin tip: 772000
Apr 07 22:23:06 default-test systemd[1]: clightning.service: start-post operation timed out. Terminating.
Apr 07 22:23:06 default-test systemd[1]: clightning.service: Control process exited, code=killed, status=15/TERM
Apr 07 22:23:06 default-test systemd[1]: clightning.service: Failed with result 'timeout'.
Apr 07 22:23:06 default-test systemd[1]: Failed to start clightning.service.
Apr 07 22:23:06 default-test systemd[1]: clightning.service: Consumed 1.738s CPU time, received 5.8K IP traffic, sent 4.9K IP traffic.
Apr 07 22:23:16 default-test systemd[1]: clightning.service: Scheduled restart job, restart counter is at 5.
Apr 07 22:23:16 default-test systemd[1]: Stopped clightning.service.
Apr 07 22:23:16 default-test systemd[1]: clightning.service: Consumed 1.738s CPU time, received 5.8K IP traffic, sent 4.9K IP traffic.
Apr 07 22:23:16 default-test systemd[1]: Starting clightning.service...
Apr 07 22:23:17 default-test lightningd[5719]: INFO    plugin-manager: /nix/store/1iia6a4zslg95iqw3g1w9xyp18xfjqx8-clightning-23.02/libexec/c-lightning/plugins/bcli: disabled via disable-plugin
Apr 07 22:23:17 default-test lightningd[5737]: plugin-trustedcoin  initialized plugin v0.6.1
Apr 07 22:23:17 default-test lightningd[5737]: plugin-trustedcoin  bitcoind RPC working, will use that with highest priority and fall back to block explorers if it fails.
Apr 07 22:23:17 default-test lightningd[5737]: plugin-trustedcoin tip: 784393
Apr 07 22:23:17 default-test systemd[1]: clightning.service: Control process exited, code=killed, status=15/TERM
Apr 07 22:23:17 default-test systemd[1]: clightning.service: Failed with result 'signal'.
Apr 07 22:23:17 default-test systemd[1]: Stopped clightning.service.
Apr 07 22:23:17 default-test systemd[1]: clightning.service: Consumed 323ms CPU time, received 5.8K IP traffic, sent 4.9K IP traffic.

Ota

jonasnick · 2023-04-10T07:25:32Z

@seberm Thanks for investigating this. I don't know why clightning's RPC does not get ready in time when trustedcoin is enabled. Maybe while syncing the plugin is in some initialization phase which blocks clightnings RPC startup? It seems like trustedcoin is doing some work despite the timeouts. Does clightning stop restarting on timeout at some after a while or does this go on forever?

seberm · 2023-04-10T18:21:46Z

Hello @jonasnick . I did some more testing and found the issue. The reason the clightning RPC socket (/var/lib/clightning/bitcoin/lightning-rpc) isn't available is caused by network hardening. The trustedcoin must be allowed to reach the outside world.

To be concrete, the issue is caused by these options in clightning's systemd service:

IPAddressAllow=127.0.0.1/32                                                                                            
IPAddressAllow=::1/128                                                                                                 
IPAddressAllow=169.254.0.0/16                                                                                                                                                                                                                 
IPAddressDeny=any

If I set services.clightning.tor.enforce = false; the clightning with trustedcoin plugin is started correctly and the RPC socket is created.

See journalctl -u clightning.service output:

-- Boot 454a515e66c141b29a7e10296ec48110 --
Apr 10 20:08:54 default-test systemd[1]: Starting clightning.service...
Apr 10 20:08:54 default-test lightningd[1012]: INFO    plugin-manager: /nix/store/1iia6a4zslg95iqw3g1w9xyp18xfjqx8-clightning-23.02/libexec/c-lightning/plugins/bcli: disabled via disable-plugin
Apr 10 20:08:55 default-test lightningd[1032]: plugin-trustedcoin  initialized plugin v0.6.1
Apr 10 20:08:55 default-test lightningd[1032]: plugin-trustedcoin  bitcoind RPC working, will use that with highest priority and fall back to block explorers if it fails.
Apr 10 20:08:55 default-test lightningd[1032]: plugin-trustedcoin tip: 784807
Apr 10 20:08:57 default-test lightningd[1032]: plugin-trustedcoin returning block 784792, 0000000000000000000457b471…, 1563020 bytes
Apr 10 20:08:57 default-test lightningd[1012]: INFO    lightningd: --------------------------------------------------
Apr 10 20:08:57 default-test lightningd[1012]: INFO    lightningd: Server started with public key 0309cf83dbebc6f44d22c1f806168b588ca8959008c211e26309e37779c2a6982a, alias STRANGEGLEE (color #0309cf) and lightningd v23.02
Apr 10 20:08:57 default-test systemd[1]: Started clightning.service.
Apr 10 20:08:59 default-test lightningd[1032]: plugin-trustedcoin returning block 784793, 00000000000000000003713003…, 1728500 bytes
Apr 10 20:09:01 default-test lightningd[1032]: plugin-trustedcoin returning block 784794, 0000000000000000000364831c…, 2158847 bytes
Apr 10 20:09:05 default-test lightningd[1032]: plugin-trustedcoin returning block 784795, 00000000000000000002de857a…, 1699162 bytes
Apr 10 20:09:06 default-test lightningd[1032]: plugin-trustedcoin returning block 784796, 000000000000000000026ac332…, 1392627 bytes
Apr 10 20:09:09 default-test lightningd[1032]: plugin-trustedcoin returning block 784797, 00000000000000000004b999d1…, 1523143 bytes
Apr 10 20:09:13 default-test lightningd[1032]: plugin-trustedcoin returning block 784798, 000000000000000000018b93b7…, 1634486 bytes
Apr 10 20:09:15 default-test lightningd[1032]: plugin-trustedcoin returning block 784799, 000000000000000000022b1ce3…, 1419438 bytes
Apr 10 20:09:16 default-test lightningd[1032]: plugin-trustedcoin returning block 784800, 00000000000000000001dbd7ff…, 1442001 bytes
Apr 10 20:09:17 default-test lightningd[1032]: plugin-trustedcoin returning block 784801, 00000000000000000003ca8537…, 1168411 bytes
Apr 10 20:09:18 default-test lightningd[1032]: plugin-trustedcoin returning block 784802, 00000000000000000005c9e751…, 1489396 bytes
Apr 10 20:09:22 default-test lightningd[1032]: plugin-trustedcoin returning block 784803, 000000000000000000029a89d9…, 1470353 bytes
Apr 10 20:09:23 default-test lightningd[1032]: plugin-trustedcoin returning block 784804, 00000000000000000000df30d6…, 1534350 bytes
Apr 10 20:09:25 default-test lightningd[1032]: plugin-trustedcoin returning block 784805, 0000000000000000000103b248…, 1549587 bytes
Apr 10 20:09:27 default-test lightningd[1032]: plugin-trustedcoin returning block 784806, 00000000000000000005714a98…, 1474955 bytes
Apr 10 20:09:29 default-test lightningd[1032]: plugin-trustedcoin returning block 784807, 000000000000000000015434ce…, 1454662 bytes
Apr 10 20:09:29 default-test lightningd[1012]: UNUSUAL plugin-bookkeeper: Snapshot balance does not equal ondisk reported 0msat, off by (+0msat/-0msat) (account wallet) Logging journal entry.

So the question is how we should handle the network hardening in a case of trustedcoin plugin is enabled? Should I just disable tor enforcing? What's your opinion?

Thank you!

seberm · 2023-04-10T18:27:43Z

Just to answer your question:

Does clightning stop restarting on timeout at some after a while or does this go on forever?

It seems to be restarting forewer :).

jonasnick · 2023-04-11T10:04:42Z

Very nice @seberm. I don't really like to disable Tor in a way that is unexpected for the user. This would be a huge footgun. But as I see it, trustedcoin is anyway only something you'd do for demonstration purposes. So it may be ok if we very clearly document this. Alternatively we can try to tunnel trustedcoin traffic through Tor.

seberm · 2023-04-11T10:47:58Z

I don't really like to disable Tor in a way that is unexpected for the user. This would be a huge footgun.

I agree, I don't like it either.

Alternatively we can try to tunnel trustedcoin traffic through Tor.

It seems trustedcoin does not honor the clightning's proxy configuration. I will try to look into its code, maybe we could add support for it. Otherwise I will add clear documentation about trustedcoin and it's configuration to this PR.

seberm · 2023-04-13T21:03:29Z

Hello @jonasnick,
I think I found a way to proxy all the http/https trustedcoin connections via Tor using the HTTP_PROXY and HTTPS_PROXY env variables. Please, take a look.

I also created nbd-wtf/trustedcoin#19 to discuss possible changes in trustedcoin itself.

modules/clightning-plugins/trustedcoin.nix

jonasnick

ACK a3c6547

elsirion mentioned this pull request Mar 5, 2023

cln: add alternative bitcoin backend config option #583

Closed

elsirion suggested changes Mar 5, 2023

View reviewed changes

modules/clightning.nix Outdated Show resolved Hide resolved

seberm force-pushed the trustedcoin branch from cf4510b to 135008e Compare March 18, 2023 22:10

seberm commented Mar 18, 2023

View reviewed changes

seberm requested review from elsirion and jonasnick and removed request for elsirion March 18, 2023 22:43

elsirion previously approved these changes Mar 23, 2023

View reviewed changes

modules/clightning-plugins/trustedcoin.nix Outdated Show resolved Hide resolved

test/tests.py Show resolved Hide resolved

seberm dismissed elsirion’s stale review via 6ba195f March 28, 2023 19:43

seberm force-pushed the trustedcoin branch from 135008e to 6ba195f Compare March 28, 2023 19:43

seberm marked this pull request as ready for review March 28, 2023 19:44

jonasnick reviewed Apr 2, 2023

View reviewed changes

seberm force-pushed the trustedcoin branch from 6ba195f to 73d2250 Compare April 5, 2023 13:54

seberm force-pushed the trustedcoin branch 2 times, most recently from e86973b to 4942130 Compare April 11, 2023 10:25

neverupdate and others added 7 commits April 11, 2023 21:00

trustedcoin: add pkg

3197338

trustedcoin: add module

35fc3a2

readme: reference trustedcoin source

c747ddb

clightning-plugins: add trustedcoin

3d26f72

trustedcoin: fix shellcheck

5b5e769

trustedcoin: update to v0.6.1

8c00c26

tests: add tests for trustedcoin clightning plugin

4942130

seberm force-pushed the trustedcoin branch from 0061964 to d3c27f2 Compare April 13, 2023 20:36

seberm force-pushed the trustedcoin branch from d3c27f2 to e88b81a Compare April 13, 2023 20:45

seberm mentioned this pull request Apr 13, 2023

Add basic support for proxy nbd-wtf/trustedcoin#19

Open

seberm force-pushed the trustedcoin branch from e88b81a to 92ee9d3 Compare April 13, 2023 21:02

seberm requested a review from jonasnick April 13, 2023 21:03

elsirion reviewed Apr 14, 2023

View reviewed changes

modules/clightning-plugins/trustedcoin.nix Outdated Show resolved Hide resolved

seberm force-pushed the trustedcoin branch from 92ee9d3 to be1f768 Compare April 14, 2023 13:47

trustedcoin: explicitly use the HTTPS_PROXY for external connections

67f2eb2

seberm force-pushed the trustedcoin branch from be1f768 to 67f2eb2 Compare April 14, 2023 21:10

docs: trustedcoin: add info about possible problems

a3c6547

jonasnick approved these changes Apr 16, 2023

View reviewed changes

jonasnick merged commit 7736468 into fort-nix:master Apr 17, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Integrate trustedcoin clightning plugin #597

Integrate trustedcoin clightning plugin #597

seberm commented Mar 3, 2023 •

edited

Loading

seberm commented Mar 18, 2023

seberm Mar 18, 2023

elsirion left a comment

jonasnick left a comment

seberm commented Apr 7, 2023 •

edited

Loading

jonasnick commented Apr 10, 2023

seberm commented Apr 10, 2023

seberm commented Apr 10, 2023

jonasnick commented Apr 11, 2023

seberm commented Apr 11, 2023

seberm commented Apr 13, 2023

jonasnick left a comment

Integrate trustedcoin clightning plugin #597

Integrate trustedcoin clightning plugin #597

Conversation

seberm commented Mar 3, 2023 • edited Loading

seberm commented Mar 18, 2023

seberm Mar 18, 2023

Choose a reason for hiding this comment

elsirion left a comment

Choose a reason for hiding this comment

jonasnick left a comment

Choose a reason for hiding this comment

seberm commented Apr 7, 2023 • edited Loading

jonasnick commented Apr 10, 2023

seberm commented Apr 10, 2023

seberm commented Apr 10, 2023

jonasnick commented Apr 11, 2023

seberm commented Apr 11, 2023

seberm commented Apr 13, 2023

jonasnick left a comment

Choose a reason for hiding this comment

seberm commented Mar 3, 2023 •

edited

Loading

seberm commented Apr 7, 2023 •

edited

Loading