Bugfix: RPC/Wallet: Include HD key metadata in dumpwallet #11803

luke-jr · 2017-11-30T20:32:01Z

When restoring a backup, we should restore the wallet metadata. This moves some of it out of "comment" sections so that it can be sensibly handled by parsing tools.

(importwallet doesn't currently support it, but that's another matter to address)

promag

Update test? Squash 2nd and 3rd commits?

src/wallet/rpcdump.cpp

promag · 2017-12-01T00:42:17Z

src/wallet/rpcdump.cpp

                file << "inactivehdmaster=1";
            } else {
                file << "change=1";
            }
-            file << strprintf(" # addr=%s%s\n", strAddr, (pwallet->mapKeyMetadata[keyid].hdKeypath.size() > 0 ? " hdkeypath="+pwallet->mapKeyMetadata[keyid].hdKeypath : ""));
+            file << strprintf(" # addr=%s%s\n", strAddr, (meta.hdKeypath.empty() ? "" : (" hdkeypath=" + meta.hdKeypath)));


Nit, partial unrelated change to what commit stands for.

src/wallet/rpcdump.cpp

jonasschnelli · 2017-12-01T06:16:39Z

@luke-jr: can you please elaborate a little bit what situation this PR does improve?

laanwj · 2018-01-30T09:47:30Z

@luke-jr Can you please answer @jonasschnelli 's question? There is no rationale in the OP at all.

luke-jr · 2018-01-30T15:13:42Z

#11803 (comment)

luke-jr · 2018-03-31T19:38:31Z

Rebased

meshcollider · 2018-11-10T12:30:13Z

Tested ACK 17d609c

This isn't really a bugfix because there is no bug in the codebase which is fixed right? But I agree its a sensible change.

Supporting it with importwallet is definitely something that needs to be done, ideally that should be done at the same time because they're mirror functions. But this doesn't break compatibility with the current importwallet so it should be ok to leave it for a future PR.

luke-jr · 2018-11-10T19:59:37Z

I think it's a bugfix because without it (and another fix to importwallet), dump/import will produce a wallet missing the key metadata. Isn't the goal of these RPCs to produce the same wallet?

Sjors · 2018-11-20T18:36:54Z

Concept ACK and lightly tested 17d609c.

Unrelated, I don't like that the address column depends on the currently active address type. We could replace it with an inferred combo() descriptor after #14477.

DrahtBot · 2018-12-28T14:46:53Z

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Conflicts

Reviewers, this pull request conflicts with the following ones:

#23127 (tests: Use test framework utils where possible by vincenzopalazzo)
#22805 (refactor: use CWallet const shared pointers in dump{privkey,wallet} by theStack)

If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

Sjors · 2019-02-15T15:06:09Z

This has been fixed as part of #14021 (not just for importmulti):

It could still be useful to switch over to inferred descriptors, so we also get the origin info.

Sjors · 2019-02-16T08:35:32Z

Just noticed that it's a seperate field now, but sill in the comment section, so we could move it to the left.

luke-jr · 2019-04-06T06:39:54Z

Rebased

Sjors · 2019-08-15T16:22:36Z

Calling resident key metadata expert @achow101...

achow101

Concept ACK

achow101 · 2019-08-15T16:41:49Z

src/wallet/rpcdump.cpp

                file << "inactivehdseed=1";
            } else {
                file << "change=1";
            }
-            file << strprintf(" # addr=%s%s\n", strAddr, (pwallet->mapKeyMetadata[keyid].has_key_origin ? " hdkeypath="+WriteHDKeypath(pwallet->mapKeyMetadata[keyid].key_origin.path) : ""));
+            if (meta.has_key_origin) {
+                file << " hdkeypath=" + WriteHDKeypath(meta.key_origin.path);


We should also write out the master fingerprint here too (part of key_origin). I think instead of a separate field for that, we could instead write that in the same way descriptors and psbt do key origin information, i.e. fingerprint instead of m in the path.

…ide of comment

ghost · 2021-07-22T04:25:57Z

Concept ACK

Will try this today. Wallet with metadata is obviously better than wallet missing the key metadata.

This PR will complete 1337 days next week

ghost · 2021-07-22T15:36:46Z

Compiled and tried in POP!_OS with a new wallet (legacy / testnet). Diff for the dump files: https://ghostbin.com/paste/L2SXs. Observed the below things when comparing dump files for Master and PR branch:

hdseedid added for reserve=1 key type

-cPUphC3iKWVx5HVcYimYKjvL1xn6uSHS2HKZdoLoKM1RzQkihNiD 2021-07-22T14:38:59Z reserve=1 # addr=tb1qp7m7nzhp9c4z7069m9rvrpp6tk6g4hqpj07t9l hdkeypath=m/0'/0'/8'

+cPUphC3iKWVx5HVcYimYKjvL1xn6uSHS2HKZdoLoKM1RzQkihNiD 2021-07-22T14:38:59Z reserve=1 hdkeypath=m/0'/0'/8' hdseedid=b01cf0b04e2799dab0a3658d7b59791a02c5ec86 # addr=tb1qp7m7nzhp9c4z7069m9rvrpp6tk6g4hqpj07t9l

No change in hd master key

cUEM9SuAS9z5z2bdYJhiwsmtCsq1UAwkP4nEv8i3BGK2SANwP51N 2021-07-22T14:38:59Z hdseed=1 # addr=tb1qsmkv2qs609vhhrt95wcd4xf8f6c0q89sag6fm2

No change in script=1 key type

00143802740190cd3720db8d1f06f4a32f6bafc7ffad 0 script=1 # addr=2MsFRrsqi61GSNEiQKwKUFGNmzbzAevUqE7

NelsonGaldeman · 2021-07-27T12:01:53Z

I think commits should be squashed.
Totally agree with prayank23: Wallet with metadata is obviously better than wallet missing the key metadata

This tests failed on my env though. Is it just me?

feature_backwards_compatibility.py --legacy-wallet | ✖ Failed  | 1 s
feature_taproot.py --previous_release              | ✖ Failed  | 1 s
mempool_compatibility.py                           | ✖ Failed  | 1 s
rpc_createmultisig.py --legacy-wallet              | ✖ Failed  | 103 s
rpc_signrawtransaction.py --descriptors            | ✖ Failed  | 105 s
wallet_upgradewallet.py --legacy-wallet            | ✖ Failed  | 1 s

This is exactly how I fetched the branch, compiled the binnaries and ran the tests.

git fetch core
git fetch core pull/11803/merge:pr11803
git switch pr11803
./autogen.sh
./configure -q --without-gui
make clean > /dev/null
make -j 4
python3 test/functional/test_runner.py

Checked my HEAD commit within my local PR copy and I'm getting 8642fbd which differs from the last commit I see on the commit tab here, am I doing something wrong?

Rspigler · 2021-07-30T05:01:26Z

Tested on master for legacy wallets:

I get an extended private masterkey

Then a list of (private key, address, hdkeypath); although very out of order. Somewhere in there, is the hdseed

Master for descriptor wallets:

This type of wallet does not support this command (code -4)

Tested on 563dda4:

Legacy:

Extended private master key.

Then list of (private key, hdkeypath, hdseedid, address). Changes are address is moved to the end of the list, and addition of hdseedid. The keys are still out of order, and the hdseed is also still randomly in the list.

Descriptor:

Unable to create descriptor wallet (either through the GUI or console), so unable to test.

Possible improvements?:

Work with descriptors
Sort addresses/hdkeypaths by address index
Bring the hdseed to top of list, like the extended private master key is
List the hdseedid once at top of list, instead of repeated for every address?

Rspigler · 2021-08-03T19:03:21Z

Work with descriptors

Maybe I was thinking of listdescriptors

DrahtBot · 2021-11-10T18:16:48Z

🐙 This pull request conflicts with the target branch and needs rebase.

_{Want to unsubscribe from rebase notifications on this pull request? Just convert this pull request to a "draft".}

DrahtBot · 2022-02-22T12:18:21Z

There hasn't been much activity lately and the patch still needs rebase. What is the status here?

Is it still relevant? ➡️ Please solve the conflicts to make it ready for review and to ensure the CI passes.
Is it no longer relevant? ➡️ Please close.
Did the author lose interest or time to work on this? ➡️ Please close it and mark it 'Up for grabs' with the label, so that it can be picked up in the future.

achow101 · 2022-10-12T17:53:51Z

Closing this as it has not had any activity in a while. If you are interested in continuing work on this, please leave a comment so that it can be reopened.

fanquake added RPC/REST/ZMQ Wallet labels Nov 30, 2017

promag reviewed Dec 1, 2017

View reviewed changes

jonasschnelli reviewed Dec 1, 2017

View reviewed changes

src/wallet/rpcdump.cpp Outdated Show resolved Hide resolved

miccomservices approved these changes Dec 5, 2017

View reviewed changes

luke-jr force-pushed the bugfix_dumpwallet_hdkeypath branch from b24b1c5 to 393511c Compare March 31, 2018 19:38

MarcoFalke added the Needs rebase label Jun 6, 2018

luke-jr force-pushed the bugfix_dumpwallet_hdkeypath branch 2 times, most recently from 73d8c21 to 17d609c Compare November 4, 2018 03:24

DrahtBot removed the Needs rebase label Nov 4, 2018

bitcoin deleted a comment from ismail120572 Nov 5, 2018

Sjors mentioned this pull request Nov 20, 2018

Import key origin data through descriptors in importmulti #14021

Merged

DrahtBot added the Needs rebase label Feb 15, 2019

luke-jr closed this Feb 15, 2019

luke-jr reopened this Feb 16, 2019

luke-jr force-pushed the bugfix_dumpwallet_hdkeypath branch from 17d609c to 479c0cb Compare April 6, 2019 06:37

achow101 reviewed Aug 15, 2019

View reviewed changes

RPC/Wallet: dumpwallet: Lookup key metadata only once

3e72777

luke-jr added 3 commits March 29, 2020 05:53

Bugfix: RPC/Wallet: dumpwallet should include hdkeypath metadata outs…

97aabcf

…ide of comment

Bugfix: RPC/Wallet: dumpwallet should include hdmasterkeyid metadata

e1488eb

RPC/Wallet: dumpwallet: Replace hdmasterkeyid with hdseedid

563dda4

luke-jr force-pushed the bugfix_dumpwallet_hdkeypath branch from 479c0cb to 563dda4 Compare March 29, 2020 05:57

DrahtBot removed the Needs rebase label Mar 29, 2020

luke-jr mentioned this pull request Jul 22, 2021

Document about wallet backup and restoration #22523

Merged

DrahtBot mentioned this pull request Aug 26, 2021

refactor: use CWallet const shared pointers in dump{privkey,wallet} #22805

Merged

DrahtBot mentioned this pull request Nov 10, 2021

tests: Use test framework utils where possible #23127

Closed

DrahtBot added the Needs rebase label Nov 10, 2021

achow101 closed this Oct 12, 2022

Bugfix: RPC/Wallet: Include HD key metadata in dumpwallet #11803

Bugfix: RPC/Wallet: Include HD key metadata in dumpwallet #11803

luke-jr commented Nov 30, 2017 •

edited

promag left a comment

promag Dec 1, 2017

jonasschnelli commented Dec 1, 2017

laanwj commented Jan 30, 2018

luke-jr commented Jan 30, 2018

luke-jr commented Mar 31, 2018

meshcollider commented Nov 10, 2018

luke-jr commented Nov 10, 2018

Sjors commented Nov 20, 2018

DrahtBot commented Dec 28, 2018 •

edited

Sjors commented Feb 15, 2019

Sjors commented Feb 16, 2019

luke-jr commented Apr 6, 2019

Sjors commented Aug 15, 2019

achow101 left a comment

achow101 Aug 15, 2019

ghost commented Jul 22, 2021

ghost commented Jul 22, 2021

NelsonGaldeman commented Jul 27, 2021 •

edited

Rspigler commented Jul 30, 2021

Rspigler commented Aug 3, 2021

DrahtBot commented Nov 10, 2021

DrahtBot commented Feb 22, 2022

achow101 commented Oct 12, 2022

Bugfix: RPC/Wallet: Include HD key metadata in dumpwallet #11803

Bugfix: RPC/Wallet: Include HD key metadata in dumpwallet #11803

Conversation

luke-jr commented Nov 30, 2017 • edited

promag left a comment

Choose a reason for hiding this comment

promag Dec 1, 2017

Choose a reason for hiding this comment

jonasschnelli commented Dec 1, 2017

laanwj commented Jan 30, 2018

luke-jr commented Jan 30, 2018

luke-jr commented Mar 31, 2018

meshcollider commented Nov 10, 2018

luke-jr commented Nov 10, 2018

Sjors commented Nov 20, 2018

DrahtBot commented Dec 28, 2018 • edited

Conflicts

Sjors commented Feb 15, 2019

Sjors commented Feb 16, 2019

luke-jr commented Apr 6, 2019

Sjors commented Aug 15, 2019

achow101 left a comment

Choose a reason for hiding this comment

achow101 Aug 15, 2019

Choose a reason for hiding this comment

ghost commented Jul 22, 2021

ghost commented Jul 22, 2021

NelsonGaldeman commented Jul 27, 2021 • edited

Rspigler commented Jul 30, 2021

Rspigler commented Aug 3, 2021

DrahtBot commented Nov 10, 2021

DrahtBot commented Feb 22, 2022

achow101 commented Oct 12, 2022

luke-jr commented Nov 30, 2017 •

edited

DrahtBot commented Dec 28, 2018 •

edited

NelsonGaldeman commented Jul 27, 2021 •

edited