Hsmtool: enable dumping output descriptors of onchain wallet #4171

darosior · 2020-10-30T15:13:23Z

This adds a new dumponchaindescriptors command to hsmtool which can:

Dump 2 xpub-based descriptors (one for each scriptPubKey type we create)
- Useful for external service (see Feature Request: Is it possible to retrieve the xPub of the on-chain wallet? #4110 for more details)
~~Possibly dump 2 xpriv-based descriptors (one for each scriptPubKey type we create)~~ Removed: footgun with anchor outputs (thanks niftynei!)
- ~~Useful for backing up / restoring the onchain wallet easily~~

TODO:

A functional test restoring a descriptor on bitcoind asserting we retrieve all the funds
Manpage and an extended doc on how to restore the onchain wallet using bitcoind
Maybe add a network parameter finally ?.. ~~EDIT: maybe output both main and test extended keys ?..~~ EDIT 2: added a testnet parameter

Fixes #4110
Ping @jb55 : finally got to jb55/clightning-dumpkeys#2 :-)

darosior · 2020-11-01T23:22:36Z

Ok, added a test, documented (and rebased on #4146 so that i can build locally)

darosior · 2020-11-02T09:55:02Z

Travis failure is an unrelated timeout

darosior · 2020-11-04T20:48:35Z

Rebased on master after #4146 merge, added a missing changelog.

niftynei

Cool addition. I'm not sure how anchor outputs affects the assumption that all outputs are some variant of p2wpkh, may not be relevant for this patch.

common/descriptor_checksum.c

niftynei · 2020-11-05T21:28:45Z

tools/hsmtool.c

+
+	/* The root seed is derived from hsm_secret using hkdf.. */
+	do {
+		hkdf_sha256(bip32_seed, sizeof(bip32_seed),


Duplicating this code from hsmd/hsmd.c#populate_secretstuff is a bad idea. Needs to be a shared routine that both hsmd and here use.

That's how we've created tool since now ?..

I don't follow. We really want this code to be shared so that if one changes, the other also gets updated.

tools/hsmtool.c

niftynei · 2020-11-05T21:44:04Z

tools/hsmtool.c

 {
 	struct secret hsm_secret;
 	u8 bip32_seed[BIP32_ENTROPY_LEN_256];
 	u32 salt = 0;
+	u32 version = testnet ? BIP32_VER_TEST_PRIVATE : BIP32_VER_MAIN_PRIVATE;


Can we infer this from the directory structure or some other info source?

No, as we can use all commands of the hsmtool on a hsm_secret without depending on its environment at all
and this would be quite convoluted too..

We have all of these parameters in chainparams.c, so why not use them? We can even steal the --network option from lightningd itself :-)

This'd mean bringing the opt() system here: that's a much bigger change :/

darosior · 2020-11-05T22:41:23Z

I'm not sure how anchor outputs affects the assumption that all outputs are some variant of p2wpkh, may not be relevant for this patch.

You are right that we actually don't sweep the 1 CSV-obstructed to_remotes in onchaind and just adverise them to the wallet, so i don't think i should actually sell this command as a "backup-and-recovery mechanism" in the FAQ..
I was also in doubt wrt dumping xprivs at all then, but it might turns to be useful (if i remove all mentions that it is exhaustive from the doc).

Edit: ah also, i could make it a miniscript descriptor then but i think it's a bad idea because there is virtually no support for them (including bitcoind) currently and this might confuse users. Also this makes me think that it's not wise to advise it as "backup all onchain funds for sure" since it might not be true in a so distant future ?

niftynei · 2020-11-06T00:58:56Z

This functionality is Cool and Interesting, but until miniscript is widely adopted I think it's best to advertise this as a subset of your lightning funds tracking, at best. There's definitely "outputs we own, eventually" that haven't been swept to a 'recognizable wallet output' that will be missed by this -- if for some reason you take your lightning node down before these are swept there's a possibility that they'll be lost.

With these considerations in mind, I'd also be tempted to remove the xpriv. Rationale: it gives the false impression of 'total wallet funds control' when that's really not the case.

i could make it a miniscript descriptor then but i think it's a bad idea because there is virtually no support for them

I agree, this is a good intuition. I suspect that when miniscript descriptors are more formally nailed down the lightning spec itself will also move that direction, which will then make this export functionality extremely powerful.

niftynei

few more things I noticed while testing.

niftynei · 2020-11-06T01:45:45Z

tools/hsmtool.c

-		return dumponchaindescriptors(argv[2], argc > 3 ? argv[3] : NULL);
+		return dumponchaindescriptors(argv[2],
+					      argc > 3 && !streq(argv[3], "") ? argv[3] : NULL,
+					      argc > 4 && streq(argv[4], "private"));


it's weird to me that you have to physically type in the phrase "private" and not look for a true/false here.

I find it better: at usage typing testnet is way more meaningful than true

Update the parameter name then. As written, the parameter testnet suggests to say 'flag true if testnet'.

tools/hsmtool.c

darosior · 2020-11-06T11:39:48Z

Used a proper API for the imported descriptor checksum file (use a struct instead and return a bool) as requested
Addressed misc nits
Removed the "output xprivs" functionality altogether, as @niftynei agrees it's a footgun. (We now use a watchonly wallet in the functional test).
Rebased on master

cdecker · 2020-11-06T15:09:27Z

tools/hsmtool.c

 {
 	struct secret hsm_secret;
 	u8 bip32_seed[BIP32_ENTROPY_LEN_256];
 	u32 salt = 0;
+	u32 version = testnet ? BIP32_VER_TEST_PRIVATE : BIP32_VER_MAIN_PRIVATE;


We have all of these parameters in chainparams.c, so why not use them? We can even steal the --network option from lightningd itself :-)

cdecker · 2020-11-06T15:10:42Z

contrib/pyln-testing/pyln/testing/fixtures.py

+    if not bitcoind.rpc.listwallets():
+        bitcoind.rpc.createwallet("lightningd-tests")


This is slightly nicer than 90f4ea6

cdecker · 2020-11-06T15:12:10Z

tests/test_wallet.py

@@ -1045,6 +1045,36 @@ def test_hsmtool_secret_decryption(node_factory):
    assert node_id == l1.rpc.getinfo()["id"]


+@unittest.skipIf(TEST_NETWORK == 'liquid-regtest', '')


Add a comment saying that this is only supported for mainnet and testnet (technically we'd require it to be regtest but that appears to match testnet).

Yeah descriptors for regtest are the same than for testnet (but addresses the same as mainnet!! 😭 )

Hurray for consistency 🤦

niftynei · 2020-11-06T20:29:17Z

I opened an issue for the password inputs for all of the hsmtool commands, #4186. I think it's ok to merge this as is. We should address this issue though, as it's a potential security hole.

rustyrussell · 2020-11-09T23:16:30Z

You should use ccan/opt, now that cmdline parsing is non-trivial. But can be added after....

This is stolen from William's clightning-dumpkeys (https://github.com/jb55/clightning-dumpkeys), itself adapted from https://github.com/bitcoin/bitcoin/blob/42b66a6b814bca130a9ccf0a3f747cf33d628232/src/script/descriptor.cpp#L25 Co-authored-by: William Casarin <jb55@jb55.com> Signed-off-by: Antoine Poinsot <darosior@protonmail.com>

We want to use it outside of fuzz-amount Signed-off-by: Antoine Poinsot <darosior@protonmail.com>

A small one just to check that we don't crash nor go out of bounds! Signed-off-by: Antoine Poinsot <darosior@protonmail.com>

This adds a command which outputs the two output descriptors corresponding to our onchain wallet. This can be useful for an external service to monitor / send fund to our wallet. Further, an "xpriv" version of such descriptors could be used to import onchain funds on a new wallet. Changelog-Added: lightning-hsmtool: a new command was added to hsmtool for dumping descriptors of the onchain wallet Signed-off-by: Antoine Poinsot <darosior@protonmail.com>

Actually, it's more complex to translate the xpub descriptor to testnet because of the descriptor checksum. Signed-off-by: Antoine Poinsot <darosior@protonmail.com>

Signed-off-by: Antoine Poinsot <darosior@protonmail.com>

Touch a bit about it as a backup/recovery mechanism in the FAQ Signed-off-by: Antoine Poinsot <darosior@protonmail.com>

niftynei · 2020-11-09T23:58:54Z

Updated the interface to use "network" instead of 'testnet', rebased on master now that #4065 is merged.

darosior · 2020-11-10T14:01:56Z

ACK 29e5764

Thanks..

darosior requested a review from cdecker as a code owner October 30, 2020 15:13

darosior force-pushed the hsmtool_descriptors branch 5 times, most recently from fd74279 to fc008f4 Compare November 1, 2020 23:17

niftynei added this to the v0.9.2 milestone Nov 2, 2020

darosior force-pushed the hsmtool_descriptors branch from fc008f4 to f9c2d4f Compare November 4, 2020 20:48

niftynei added the wallet label Nov 5, 2020

niftynei requested changes Nov 5, 2020

View reviewed changes

niftynei reviewed Nov 6, 2020

View reviewed changes

darosior force-pushed the hsmtool_descriptors branch from f9c2d4f to 11c1a11 Compare November 6, 2020 11:29

cdecker reviewed Nov 6, 2020

View reviewed changes

niftynei force-pushed the hsmtool_descriptors branch from 11c1a11 to f49d84a Compare November 9, 2020 23:33

darosior and others added 8 commits November 9, 2020 17:51

fuzz: add a 'to_string' function to libfuzz

8b2466f

We want to use it outside of fuzz-amount Signed-off-by: Antoine Poinsot <darosior@protonmail.com>

fuzz: add a fuzz target for common/descriptor_checksum

97c7b02

A small one just to check that we don't crash nor go out of bounds! Signed-off-by: Antoine Poinsot <darosior@protonmail.com>

hsmtool: add a 'testnet' option to 'dumponchaindescriptors'

1b31831

Actually, it's more complex to translate the xpub descriptor to testnet because of the descriptor checksum. Signed-off-by: Antoine Poinsot <darosior@protonmail.com>

pytest: create a wallet for bitcoind if none is present

8861142

Signed-off-by: Antoine Poinsot <darosior@protonmail.com>

pytest: test onchain wallet descriptor import on bitcoind

0b3c476

Signed-off-by: Antoine Poinsot <darosior@protonmail.com>

doc: document output descriptors in hsmtool

29e5764

Touch a bit about it as a backup/recovery mechanism in the FAQ Signed-off-by: Antoine Poinsot <darosior@protonmail.com>

niftynei force-pushed the hsmtool_descriptors branch from f49d84a to 29e5764 Compare November 9, 2020 23:56

darosior mentioned this pull request Nov 10, 2020

hsm_secret generation from a seed-phrase #4065

Merged

niftynei merged commit 6b347e1 into ElementsProject:master Nov 10, 2020

darosior mentioned this pull request Dec 17, 2020

doc/BACKUP.md: Document backup strategies for lightningd. #4207

Merged

darosior deleted the hsmtool_descriptors branch December 17, 2020 12:03

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Hsmtool: enable dumping output descriptors of onchain wallet #4171

Hsmtool: enable dumping output descriptors of onchain wallet #4171

darosior commented Oct 30, 2020 •

edited

Loading

darosior commented Nov 1, 2020

darosior commented Nov 2, 2020

darosior commented Nov 4, 2020

niftynei left a comment

niftynei Nov 5, 2020

darosior Nov 6, 2020

niftynei Nov 6, 2020

niftynei Nov 5, 2020

darosior Nov 5, 2020

cdecker Nov 6, 2020

darosior Nov 6, 2020

darosior commented Nov 5, 2020 •

edited

Loading

niftynei commented Nov 6, 2020 •

edited

Loading

niftynei left a comment

niftynei Nov 6, 2020

darosior Nov 6, 2020

niftynei Nov 6, 2020 •

edited

Loading

darosior commented Nov 6, 2020

cdecker Nov 6, 2020

cdecker Nov 6, 2020

cdecker Nov 6, 2020

darosior Nov 6, 2020

cdecker Nov 6, 2020

niftynei commented Nov 6, 2020

rustyrussell commented Nov 9, 2020

niftynei commented Nov 9, 2020

darosior commented Nov 10, 2020

		if not bitcoind.rpc.listwallets():
		bitcoind.rpc.createwallet("lightningd-tests")

		@@ -1045,6 +1045,36 @@ def test_hsmtool_secret_decryption(node_factory):
		assert node_id == l1.rpc.getinfo()["id"]


		@unittest.skipIf(TEST_NETWORK == 'liquid-regtest', '')

Hsmtool: enable dumping output descriptors of onchain wallet #4171

Hsmtool: enable dumping output descriptors of onchain wallet #4171

Conversation

darosior commented Oct 30, 2020 • edited Loading

darosior commented Nov 1, 2020

darosior commented Nov 2, 2020

darosior commented Nov 4, 2020

niftynei left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

darosior commented Nov 5, 2020 • edited Loading

niftynei commented Nov 6, 2020 • edited Loading

niftynei left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

niftynei Nov 6, 2020 • edited Loading

Choose a reason for hiding this comment

darosior commented Nov 6, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

niftynei commented Nov 6, 2020

rustyrussell commented Nov 9, 2020

niftynei commented Nov 9, 2020

darosior commented Nov 10, 2020

darosior commented Oct 30, 2020 •

edited

Loading

darosior commented Nov 5, 2020 •

edited

Loading

niftynei commented Nov 6, 2020 •

edited

Loading

niftynei Nov 6, 2020 •

edited

Loading