Feature to export bip32 root key as mnemonic or xprv?

[peertrade]

I'm trying to figure out fund backup/recovery strategies, in particular offline. Typically in HD wallets one receives a bip39 mnemonic and also has access to the root xprv key. One can write these down, store in safe location. Also, one can watch or restore funds by importing into 3rd party wallets/tools if need be.

In c-lightning, I haven't yet found a way to do that. I do see that:

• there is the hsm_secret file, which is a binary format.
• one can list addresses via lightning-cli dev-liastaddrs. Help text mentions bip32 derivation.

So I'm hoping for something that enables me to turn hsm_secret into an xprv. It could take the form of:

• written instructions.
• a contrib tool/utility
• an rpc call such as export-priv-key

[renepickhardt]

as is documented at: https://github.com/hkjn/lnhw/tree/master/doc/hsmd the hsm_secret is basically just a 32 Byte secrete. which of course could be transfered to a bip39 mnemonic wordlist.

On the other side bip39 is already implemented in libwally (c.f.: https://github.com/ElementsProject/libwally-core/blob/master/src/bip39.c ) which is used within c-lighting.

So I guess yes it would be easy to provide an API for it. Has this not been done because it was low priority or are there other reasons against this?

@peertrade one reason why it is not less useful for a lightning node in comparison to a bitcoin to do that is that you would only be able to derive your onchain funds but for your payment channels you need to stare the state which in general is much lager than 32 bytes. so you wouldn't be able to recover those anyway.

[peertrade]

@renepickhardt thx for your comments. I get your point about needing the state for lightning channels, but really my concern is about on-chain funds that have been added to the internal wallet, so I still hope for a simple way to obtain xprv from the hsm_seed for use by other tools.

I suppose that for offline backup right now, base64 encoding hsm_secret file is viable strategy, eg:

cat ~/.lightning/hsm_secret | base64

[renepickhardt]

you can get the xprv with the following python script which was basically written by @stepansnigirev ( https://github.com/stepansnigirev )

import hashlib
import hmac
from math import ceil
from hd import HDPrivateKey

hsm_secret_as_hex = "YOU HAVE TO REPLACE THIS WITH THE 32 CHARACTER HEX VALUE OF hsm_secret"

hsm_secret = bytes.fromhex(hsm_secret_as_hex)

hash_len = 32
def hmac_sha256(key, data):
    return hmac.new(key, data, hashlib.sha256).digest()

def hkdf(length, ikm, salt=b"", info=b""):
    prk = hmac_sha256(salt, ikm)
    t = b""
    okm = b""
    for i in range(ceil(length / hash_len)):
        t = hmac_sha256(prk, t + info + bytes([1+i]))
        okm += t
    return okm[:length]

bip32_seed = hkdf(32, hsm_secret, b"\x00", b"bip32 seed")

master_extkey = HDPrivateKey.from_seed(bip32_seed, b"m")
print(master_extkey.xprv())

[cdecker]

Notice that we are planning to consolidate our derivation structure to the one that lnd is currently using, so hopefully we can eventually use tools that were built for either implementation.

[renepickhardt]

Independently of your comment I have tried to provide a patch which lists the hsm_secret as an mnemonic seed. after the first time the hsm_secret has been calculated c.f.:

https://github.com/ElementsProject/lightning/compare/be3b782cb4310a7b02e20fe33a55f1be4ee3780a...renepickhardt:export-hsm_secret-to-mnemonic-seed?expand=1

I have several questions:

1. did I invoke my function at the correct place? I think yes
2. is the output via status_log preferred? for some reason printf was not showing output but it also seems strange to print a seed to a private key to a logfile
3. fore some reason no mnemonic seed is being returned by the function from the libwally lib. I put a question about this also here: https://bitcoin.stackexchange.com/questions/77665/call-to-bip39-mnemonic-from-bytes-in-libwally-core-returns-null

[peertrade]

@renepickhardt thx for the python script. I tried to run it but it imports some files I'm missing. What deps do I need to install?

$ python /tmp/test.py
Traceback (most recent call last):
  File "/tmp/test.py", line 4, in <module>
    from hd import HDPrivateKey
ImportError: No module named hd

[wythe]

This topic is discussed in #746. FWIW I vote against an rpc call.

[renepickhardt]

@peertrade yeah you are missing a dependency which is also written by @stepansnigirev sorry didn't realize that. He said he would release his code soon. I don't want to publish all this stuff

[stepansnigirev]

@peertrade , the script relies on hd.py from this repo: https://github.com/jimmysong/pybtcfork

[peertrade]

ok, I got the python script working (by running it inside pybtcfork dir) and it generated an xprv. But the pubkey of first address generated by bip32.org does not match that of lightning-cli dev-listaddrs, so I think there is an error in my process.

# is this a correct way to get hex value from hsm_secret?
$ xxd -l 16 -p ~/.lightning/hsm_secret 
b74f4b69e2d39385734d2113be37c92a

# put above hex value into @renepickhardt's script and ran
$ python3 test.py
xprv9s21ZrQH143K3nWsQXAthr4yyfhQF6sakyhrGhKb8AhnJM2sxHRi72BWy6DWsvGAZqbPLf4vvGrTkSr586FbNwERajw1P4BvREunDSzuwzi

# reference pubkey value, from lightningd
$ lightning-cli dev-listaddrs | grep pubkey
      "pubkey": "03f1c5c0a5030accbb4d99d29561e7cb3880b634fc9bb0c5a79b4f3356a754a658",

#value from bip32.org, with gen'd xprv and path=m/0
0263aa6201ba683739c8f78ce1bde9637fa21dbea8ba27e3a6712028e468171215

#value from bip32.org, with gen'd xprv and path=m/0/0/0
02c1631ad35a74cdd03a8ec143260f718241f8131381146bbfd87f7d6e12f043b0

#value from bip32.org, with gen'd xprv and path=m/0'
02be9d2e4535d7025583045fb5082696ac29981fc8de75a78b9519fbeabcf8d5f9

#value from bip32.org, with gen'd xprv and path=m/0'/0'/'0'
03388077da7fa27058e76071ccdcdf35add440115724096bff1ba1149f9ce233d4

From my reading of hsmd doc, I think the path should just be m/0, but I also tried m/0/0/0 both hardened and unhardened with no match on pubkey.

I am attaching the hsm_secret file tgz'd. ( no funds, only for testing, don't worry ).
hsm_secret.tar.gz

Can anyone show me where I'm going wrong?

[jb55]

on https://iancoleman.io/bip39/

• derivation path m

it should list the public keys and they should be the same.

Here's a patch that gets lightning_hsmd to dump a privkey on command:

From 85e27993237e27150bf503af22699ee4c34aff6a Mon Sep 17 00:00:00 2001
From: William Casarin <jb55@jb55.com>
Date: Wed, 1 Aug 2018 00:23:12 -0700
Subject: [PATCH] hsm: dump privkey command

---
 hsmd/hsm.c | 37 ++++++++++++++++++++++++++++++++++---
 1 file changed, 34 insertions(+), 3 deletions(-)

diff --git a/hsmd/hsm.c b/hsmd/hsm.c
index 6c62a3f5..dedad1fd 100644
--- a/hsmd/hsm.c
+++ b/hsmd/hsm.c
@@ -42,6 +42,8 @@
 #include <wire/gen_peer_wire.h>
 #include <wire/wire_io.h>
 
+#include <libbase58.h>
+
 #define REQ_FD 3
 
 /* Nobody will ever find it here! */
@@ -1060,9 +1062,9 @@ static void maybe_create_new_hsm(void)
 	status_unusual("HSM: created new hsm_secret file");
 }
 
-static void load_hsm(void)
+static void load_hsm(const char *secretfile)
 {
-	int fd = open("hsm_secret", O_RDONLY);
+	int fd = open(secretfile ? secretfile : "hsm_secret", O_RDONLY);
 	if (fd < 0)
 		status_failed(STATUS_FAIL_INTERNAL_ERROR,
 			      "opening: %s", strerror(errno));
@@ -1081,7 +1083,7 @@ static void init_hsm(struct daemon_conn *master, const u8 *msg)
 		master_badmsg(WIRE_HSM_INIT, msg);
 
 	maybe_create_new_hsm();
-	load_hsm();
+	load_hsm(NULL);
 
 	send_init_response(master);
 }
@@ -1381,8 +1383,37 @@ static void master_gone(struct io_conn *unused UNUSED, struct daemon_conn *dc UN
 	exit(2);
 }
 
+static bool my_sha256(void *digest, const void *data, size_t datasz)
+{
+	sha256(digest, data, datasz);
+	return true;
+}
+
+
+static int dump_xpriv(const char *secretfile) {
+	static u8 buf[BIP32_SERIALIZED_LEN];
+	static char enc[1024];
+	size_t outlen = sizeof(enc);
+	load_hsm(secretfile);
+
+	secretstuff.bip32.version = BIP32_VER_MAIN_PRIVATE;
+	int ret = bip32_key_serialize(&secretstuff.bip32, BIP32_FLAG_KEY_PRIVATE,
+				      buf, BIP32_SERIALIZED_LEN);
+
+	assert(ret == WALLY_OK);
+
+	b58_sha256_impl = my_sha256;
+	b58check_enc(enc, &outlen, 4, buf+1, BIP32_SERIALIZED_LEN-1);
+	printf("%.*s\n", (int)outlen, enc);
+
+	return 0;
+}
+
 int main(int argc, char *argv[])
 {
+	if (argc == 3 && streq(argv[1], "--dump-xpriv"))
+		exit(dump_xpriv(argv[2]));
+
 	setup_locale();
 
 	struct client *client;
-- 
2.17.1

then you do ./lightningd_hsmd --dump-xpriv ~/.lightning-mainnet/hsm_secret

[peertrade]

@jb55 thx for that! With your patch I was able to generate the correct xprv and derive addresses from it that match values from litecoin-cli dev-listaddrs. I hope this patch or something like it can be included in main repo.

@renepickhardt I think there may be a bug in your python script. I realized that my xxd command was wrong, it should be:

xxd -p  ~/.lightning/hsm_secret | tr -d '\n' && echo ""
b74f4b69e2d39385734d2113be37c92abee885e317a9793239ee1f0ba5060626

But after putting the hex value into the script, I still get an xprv that does not derive correct values. I then tried to read the hsm_secret file directly in the script like so:

hsm_secret = open("/home/lightning/.lightning/hsm_secret", "rb").read()

and that produces the exact same (non-working) xprv as the xxd obtained value.

[gabridome]

@renepickhardt

as is documented at: https://github.com/hkjn/lnhw/tree/master/doc/hsmd the hsm_secret is basically just a 32 Byte secrete. which of course could be transfered to a bip39 mnemonic wordlist.

Maybe I didn't understand, but in no way you can derive a bip39 mnemonic wordlist from a seed. Probably I misunderstood what you meant by the word "transferred" though:

"To create a binary seed from the mnemonic, we use the PBKDF2 function with a mnemonic sentence (in UTF-8 NFKD) used as the password and the string "mnemonic" + passphrase (again in UTF-8 NFKD) used as the salt. The iteration count is set to 2048 and HMAC-SHA512 is used as the pseudo-random function. The length of the derived key is 512 bits (= 64 bytes)."
https://github.com/bitcoin/bips/blob/master/bip-0039.mediawiki#from-mnemonic-to-seed

[renepickhardt]

@gabridome as far as I understand the situation these should be two separated problems. One problem is how do you get truly random binary data and the other problem is how to you encode the data together with some checksum as a list of words.

I think transfering hsm_secrete which at this point already exists to the mnemonic word list and such a list back to hsm_secrete should always work. It is another question if that is useful or not

[gabridome]

It is another question if that is useful or not

It is just not a correct BIP39 derivation procedure.

@renepickhardt in BIP39, the initial entropy (ENT) is NOT the seed.

ENT + checksum ---> list of words ---> BIP 32 seed (512 bits).

The second operation is irreversible.

If you derive the words from hsm_secret, you are using the content of the file as the BIP39 initial entropy (ENT). If you want to be compliant with BIP39 you cannot use hsm_secretalso as the BIP32 seed.

[peertrade]

So summarizing then: It is fine to export an xprv for existing hsm_secret, but for mnemonic to work according to bip39, then hsm_secret must in fact be generated from a mnemonic, not vice-versa.

Is there any chance to get @jb55's patch for exporting xprv merged now?

Full bip39 support would be a bit more involved I suppose.

[jb55]

@peertrade that patch was a hack, I don't expect it to be merged. I think a better solution would be a dumpprivkey rpc command.

[cdecker]

I'd prefer to have an external tool to generate and handle the hsm_secret file directly and adding a flag to suppress auto-generation in lightningd, to ensure we don't trample it and sidestep the mnemonic generation path of the external tool. I really don't like any daemon other than hsmd having access to the private key (especially as a JSON-RPC call).

[JBaczuk]

@cdecker As I was implementing a dumpprivkey RPC command and I quickly realized that c-lightning was not designed for this, as you said (i.e. one must enable private keys to leave the hsmd process using common/bip32.c).

I'm curious if an RPC dumpprivkey could be made reasonably secure. Obviously one should backup the hsm_secret file, but it can only be imported into another c-lightning node. If the user needs or wants to import it into other wallet software, they have to run a custom dump script. However, if it could be made secure, an RPC would provide a convenient and standard (Bitcoin-core) way for users to backup their key (esp. to be able to import into other wallet software).

But, here's where I'm not clear. Since the RPC uses UNIX domain sockets, the caller needs access to the host, correct? If that's the case, then nothing is stopping them from getting access to the hsm_secret once they have access to the host. Am I missing something?

[jb55]

@cdecker

I really don't like any daemon other than hsmd having access to the private key

Fair enough,

Would a cleaned up patch like the one I gave above be acceptable? Basically adds a quick xpriv export as a cli option to hsmd. Is that too hacky? I'm not sure what the state of the new derivation scheme that @rustyrussell is working on and how it may affect that patch...

[ctrlbreak-]

@jb55

FWIW, I've taken your old patch above and re-worked it to function with current master and tested it out on Testnet to see if I could actually succeed. It's my first time interacting with any actual C code myself (as well as patch) and was pretty sure I wouldn't succeed. Pleasantly surprised. Thanks again for the initial patch!

[jb55]

@ctrlbreak- good to hear! feel free to put up the patch up on a branch somewhere so we can keep it going.

[jb55]

thinking about this some more, I think the best option is to dump the xprivs as output descriptors. this way it could be easily imported into bitcoind with importmulti.

[ZmnSCPxj]

@cdecker

I'd prefer to have an external tool to generate and handle the hsm_secret file directly and adding a flag to suppress auto-generation in lightningd, to ensure we don't trample it and sidestep the mnemonic generation path of the external tool.

Perhaps a compromise would be to generate using BIP32 in the first place, and place the words in a bip32_secret file that is beside hsm_secret file?

If bip32_secret file is accessible to some user, then it is very likely that hsm_secret is also accessible to that user.

Or just append the words to the hsm_secret file, have hsm read only the first 32 bytes. If some user can access hsm_secret it would be possible to steal the keys already anyway.

[jb55]

I made a standalone repo for this: https://github.com/jb55/clightning-dumpkeys

still need to add output descriptor support

[jb55]

this can probably be closed now. you can use the above tool to dump the xprvs/xpubs and output descriptors.

[ZmnSCPxj]

As the original request ask for mnemonic or xprv, and we have an xprv method now, I concur.

[jb55]

I also added a warning to make it clear that this is not a backup method, as I've seen people asking about it:

WARNING just saving your funding wallet xprvs is not enough to backup your clightning node. There are other channel state (p2wsh) outputs that are not dumped here. Please use the recommended backup method provided by clightning (as of this writing there aren't any, but that should change soon).

[peertrade]

hmm, I don't really see that we have an xprv method now. Yes, there is now a 3rd party tool (thanks for this @jb55!), but nothing built into c-lightning. It's not much different from last year when we had a functional patch.

I think the @jb55's xprv export tool should at least go into a contrib directory.

Of course, if/when a supported backup system becomes baked in, that could be a different story.

as of this writing there aren't any, but that should change soon

oh, is this work in progress?

[jb55]

@peertrade yes the current plan is a replicated DB backup plugin, there was talk about potential static channel backups like LND does it (saved with data loss protection info), but I believe those aren't as ideal? cc @ZmnSCPxj

[ZmnSCPxj]

For myself, the only safe channel backup is either DB replication, or (before we deploy any "static channel backup"-like system) we should first write and deploy code on a lot of nodes that tests peers for attempts at stealing from channels that have initiated the data_loss_protect protocol (i.e. have some closes automatically use a faked data_loss_protect, then entrap any attempts at stealing (since admitting that you lost data means the other side could potentially safely use old state on you)), let those versions trickle through the network and then deploy a "static channel backup"-like system.

For DB replication, you are probably better off using a commodity replicating self-healing filesystem, such as ZFS (turn off dedup, since Lightning stores cryptographic data mainly which is very unlikely to be duplicated). However, if you need to run on single-board computer like RaspPi, well ZFS is not going to fit. Hooking at the DB level seems questionable to me: plugins hooked there can be triggered before init (because we already do some DB updates just from starting up), plugins hooked there cannot safely execute most (all, because things could change in the future) commands because most of them touch the database, plugins get a DB sqlite3 query that they have to somehow understand, plugins need to coordinate the timing of them sampling/copying the database file and writing the query to their own write-ahead log... there is a lot of space for edge-case bugs here, you are really better off with ZFS if you want to ensure proper backups "soon".

I think the @jb55's xprv export tool should at least go into a contrib directory.

I can reopen this if @jb55 is willing to make a PR for this export tool.

[jb55]

"ZmnSCPxj, ZmnSCPxj jxPCSmnZ" <notifications@github.com> writes:

For myself, the only safe channel backup is either DB replication

I'm working on this now

For DB replication, you are probably better off using a commodity replicating self-healing filesystem, such as ZFS (turn off dedup, since Lightning stores cryptographic data mainly which is very unlikely to be duplicated). However, if you need to run on single-board computer like RaspPi, well ZFS is not going to fit. Hooking at the DB level seems questionable to me: plugins hooked there can be triggered *before* `init` (because we already do some DB updates just from starting up), plugins hooked there cannot safely execute most (all, because things could change in the future) commands because most of them touch the database, plugins get a DB sqlite3 query that they have to somehow understand, plugins need to coordinate the timing of them sampling/copying the database file and writing the query to their own write-ahead log... there is a lot of space for edge-case bugs here, you are really better of with ZFS.

we discussed this today on IRC: https://jb55.com/s/802aa6f679b5a336.txt I don't think ZFS is a reasonable thing to require of end users. I share your concern about the brittleness of maintaining a write-ahead log and replication plugin based on db hooks, but I don't see any other option. In the chance we fall out of sync I believe we can just start over a fresh snapshot, it's not ideal but should be robust? An alternative is to have a specialized backup for a specific subset of the database that could be backed up independently, or snapshotted on each update. I'm not sure if this is possible but it's something I'm looking into.

>I think the @jb55's xprv export tool should at least go into a contrib directory. I can reopen this if @jb55 is willing to make a PR for this export tool.

I would have to rewrite it to use the internal libwally/ccan libs. I extracted it for the purpose of being a standalone tool with minimal dependencies, and based on cdecker's preference of an out of tree tool. It's probably fine to have it out of tree for now unless we find a compelling reason otherwise.

[ZmnSCPxj]

I don't think ZFS is a reasonable thing to require of end users. I share your concern about the brittleness of maintaining a write-ahead log and replication plugin based on db hooks, but I don't see any other option.

NIH shrug ZFS is safe and well-tested and a lot more people are going to be invested in keeping it safe and well-tested. Our own boutique backup will take at least a year of development and #recklessness before it would be anywhere near as safe. Those who cannot wait and need for some reason to handle mass amounts of Bitcoin on LN nodes now should really use ZFS (and should not use lnd SCB; given the major LN node implementations are open-source, it is trivial for an attacker to write code that exploits SCB and gets away with it before code that probes for theft attempts becomes too widespread for safe stealing of SCB users).

In the chance we fall out of sync I believe we can just start over a fresh snapshot, it's not ideal but should be robust?

How do we know we are out of sync?

If the plugin dies at some point and lightningd auto-removes the hook, some queries will not get backed up. Some of those queries may update channels that end up not getting updated for a good while (maybe the other side just goes offline for a long while), so you cannot even know that you do not know of an update until too late.

Maybe add a flag that says "there must be something that is hooked to db_hook, if not and we would update the DB, fatal ourselves rather than write something to the DB that cannot be backed up".

Maybe also add a flag that says "the peer must support data_loss_protect, or else we fail to fundchannel to them and error their open_channels" and make that the default.

But really: sqlite3 is already pretty robust by itself and communicates well with most filesystems about when exactly it should get synced to disk and etc etc and I think the correct way is by filesystem replication. Sqlite3 already has a write-ahead log if you enable it, and even if you disable it you still get a rollback log that is just as safe in terms of database consistency. It just doesn't do replication by itself (because, you know, replicating filesystems already exist....).

But I could be wrong. Feel free to continue with a db_hook backup system and good luck in your endeavours.

Edit: If you really want to continue here, I would suggest rather the creation of a channel_state_update hook that focuses on the only important DB update: revocation of old channel states. This removes a lot of the risk and complexity of using the DB statements. Then add a restorechannel command that requires the same information as channel_state_update provides, with some checks to ensure that we do not restore to a known-old channel state.

[dan-da]

It's probably fine to have it out of tree for now unless we find a
compelling reason otherwise.

one word: discoverability.

How is a user of c-lightning to find the tool? If it were linked to in the README or other relevant doc, I guess that would be sufficient...

[rsbondi]

I wanted to derive keys for another purpose, this thread was helpful, I made a plugin from what I discovered for xpriv/xpub export,

export responsibly.