New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Wallet] Add simplest BIP32/deterministic key generation implementation #8035

Merged
merged 4 commits into from Jun 14, 2016

Conversation

Projects
None yet
@jonasschnelli
Member

jonasschnelli commented May 10, 2016

Probably most simplest HD implementation.

  • New wallets will use hd/bip32 (enabling hd only possible during wallet creation/first start)
  • HD can be disabled with --usehd=0
  • Only hardened key derivation
  • Fixed keypath (m/0'/0'/k')
  • Only one additional database object (CHDChain) which contains the CKeyID for the master key together with the child key counter

I'll add some tests once there are some conceptual acks (and also to keep the diff at a very minimum).

@slush0

This comment has been minimized.

Show comment
Hide comment
@slush0

slush0 May 10, 2016

Nice minimalistic PR. One question though. Why to use m/0'/0'/i instead of something more standard? BIP32 proposes m/0'/0/i for external chains (I suppose Core in current implementation does not need internal/change chain). Or even better - use m/0/i, which is forward compatible with BIP44 (maybe Core will make it somedays). m/0/i is subset of BIP44 - an account.

slush0 commented May 10, 2016

Nice minimalistic PR. One question though. Why to use m/0'/0'/i instead of something more standard? BIP32 proposes m/0'/0/i for external chains (I suppose Core in current implementation does not need internal/change chain). Or even better - use m/0/i, which is forward compatible with BIP44 (maybe Core will make it somedays). m/0/i is subset of BIP44 - an account.

@paveljanik

View changes

Show outdated Hide outdated src/wallet/wallet.cpp
@paveljanik

View changes

Show outdated Hide outdated src/wallet/walletdb.cpp
@@ -599,6 +599,16 @@ ReadKeyValue(CWallet* pwallet, CDataStream& ssKey, CDataStream& ssValue,
return false;
}
}
else if (strType == "hdchain")

This comment has been minimized.

@paveljanik

paveljanik May 10, 2016

Contributor

Repeating "hdchain" several times reminds me there was some patch redefining these as constants...

@paveljanik

paveljanik May 10, 2016

Contributor

Repeating "hdchain" several times reminds me there was some patch redefining these as constants...

This comment has been minimized.

@laanwj

laanwj May 10, 2016

Member

Which one? We did the redefine-magic-strings-as-constants it in other places, but also for the wallet?

@laanwj

laanwj May 10, 2016

Member

Which one? We did the redefine-magic-strings-as-constants it in other places, but also for the wallet?

This comment has been minimized.

@paveljanik

paveljanik May 10, 2016

Contributor

Yes, because some of them are repeated several times in the file. But maybe I'm remembering some other patch.

@paveljanik

paveljanik May 10, 2016

Contributor

Yes, because some of them are repeated several times in the file. But maybe I'm remembering some other patch.

This comment has been minimized.

@paveljanik

paveljanik May 10, 2016

Contributor

Ah, it probably was #5707.

@paveljanik

paveljanik May 10, 2016

Contributor

Ah, it probably was #5707.

This comment has been minimized.

@jonasschnelli

jonasschnelli May 10, 2016

Member

Yes. I did refactor out these stings into constants serval times. But this should be done in another PR and hopefully after this has been merged.

@jonasschnelli

jonasschnelli May 10, 2016

Member

Yes. I did refactor out these stings into constants serval times. But this should be done in another PR and hopefully after this has been merged.

This comment has been minimized.

@paveljanik

paveljanik May 10, 2016

Contributor

Definitely not for this PR, of course.

@paveljanik

paveljanik May 10, 2016

Contributor

Definitely not for this PR, of course.

@paveljanik

This comment has been minimized.

Show comment
Hide comment
@paveljanik

paveljanik May 10, 2016

Contributor

Concept ACK

Nice, small, elegant 👍

Contributor

paveljanik commented May 10, 2016

Concept ACK

Nice, small, elegant 👍

@paveljanik

This comment has been minimized.

Show comment
Hide comment
@paveljanik

paveljanik May 10, 2016

Contributor

What should RPC dumpwallet dump after this?

Contributor

paveljanik commented May 10, 2016

What should RPC dumpwallet dump after this?

@paveljanik

This comment has been minimized.

Show comment
Hide comment
@paveljanik

paveljanik May 10, 2016

Contributor

Some import path needed then.

Contributor

paveljanik commented May 10, 2016

Some import path needed then.

@laanwj

This comment has been minimized.

Show comment
Hide comment
@laanwj

laanwj May 10, 2016

Member

Concept ACK

Member

laanwj commented May 10, 2016

Concept ACK

@jonasschnelli

This comment has been minimized.

Show comment
Hide comment
@jonasschnelli

jonasschnelli May 10, 2016

Member

@slush0: One reason for the current keypath is the use hardened-only key derivation.
I personally think people overestimate the switch-wallet feature. Importing a bip32 master key together with a given keypath can be tricky (lookup window).

But I agree.
If and once this PR gets merged into master, next thing I would do is writing a PR that would make the keypath flexible.

We could discuss the default keypath and the keypath flexibility in a such follow up PR.

This PR only intent to solve the "backup problem" by using deterministic key derivation. So we could even think of using m/k'.

Member

jonasschnelli commented May 10, 2016

@slush0: One reason for the current keypath is the use hardened-only key derivation.
I personally think people overestimate the switch-wallet feature. Importing a bip32 master key together with a given keypath can be tricky (lookup window).

But I agree.
If and once this PR gets merged into master, next thing I would do is writing a PR that would make the keypath flexible.

We could discuss the default keypath and the keypath flexibility in a such follow up PR.

This PR only intent to solve the "backup problem" by using deterministic key derivation. So we could even think of using m/k'.

@jonasschnelli

This comment has been minimized.

Show comment
Hide comment
@jonasschnelli
Member

jonasschnelli commented May 10, 2016

Fixed @paveljanik nits.

@slush0

This comment has been minimized.

Show comment
Hide comment
@slush0

slush0 May 10, 2016

I personally think people overestimate the switch-wallet feature.

@jonasschnelli It's actually quite useful feature, and it may be even for Core. I can imagine somebody who'll have Qt wallet awfully out of sync and will need to get his funds ASAP. Then loading the backup to some other software like Multibit HD will do the job. BIP44 is actually being used by most of current wallets so heading towards the same structure makes sense.

This PR only intent to solve the "backup problem"

Fair point and I appreciate somebody is finally addressing it.

slush0 commented May 10, 2016

I personally think people overestimate the switch-wallet feature.

@jonasschnelli It's actually quite useful feature, and it may be even for Core. I can imagine somebody who'll have Qt wallet awfully out of sync and will need to get his funds ASAP. Then loading the backup to some other software like Multibit HD will do the job. BIP44 is actually being used by most of current wallets so heading towards the same structure makes sense.

This PR only intent to solve the "backup problem"

Fair point and I appreciate somebody is finally addressing it.

@paveljanik

View changes

Show outdated Hide outdated src/wallet/wallet.cpp
@paveljanik

View changes

Show outdated Hide outdated src/wallet/wallet.h
@paveljanik

View changes

Show outdated Hide outdated src/wallet/walletdb.h
@jonasschnelli

This comment has been minimized.

Show comment
Hide comment
@jonasschnelli

jonasschnelli May 10, 2016

Member

Force push fixed @paveljanik nits. Formatted some of the code with clang-format.

Member

jonasschnelli commented May 10, 2016

Force push fixed @paveljanik nits. Formatted some of the code with clang-format.

@sipa

View changes

Show outdated Hide outdated src/wallet/wallet.cpp
@achow101

This comment has been minimized.

Show comment
Hide comment
@achow101

achow101 May 11, 2016

Member

Concept ACK

So as I understand this, it won't affect the keys already in the wallet, right? It just generates all new keys using the HD master key?

Also there should be an RPC command to dump the master private key.

Member

achow101 commented May 11, 2016

Concept ACK

So as I understand this, it won't affect the keys already in the wallet, right? It just generates all new keys using the HD master key?

Also there should be an RPC command to dump the master private key.

@jonasschnelli

This comment has been minimized.

Show comment
Hide comment
@jonasschnelli

jonasschnelli May 11, 2016

Member

@achow101:
for now (this PR), HD will only be enabled when you create a new wallet (first start).

Also there should be an RPC command to dump the master private key.

I agree. Have a look at #6265 and #7273.
This is my third try to get HD into Cores wallet. I think the way to go is this extremely simple basic step.
We can add RPC/API changes later.

Member

jonasschnelli commented May 11, 2016

@achow101:
for now (this PR), HD will only be enabled when you create a new wallet (first start).

Also there should be an RPC command to dump the master private key.

I agree. Have a look at #6265 and #7273.
This is my third try to get HD into Cores wallet. I think the way to go is this extremely simple basic step.
We can add RPC/API changes later.

@instagibbs

View changes

Show outdated Hide outdated src/wallet/wallet.h
@MarcoFalke

View changes

Show outdated Hide outdated src/wallet/walletdb.h
// in the database
// key metadata is not required
CPubKey pubkey = key.GetPubKey();
if (!AddKeyPubKey(key, pubkey))

This comment has been minimized.

@instagibbs

instagibbs May 12, 2016

Member

nit: simply calling AddKey should work here nevermind, wrong function

@instagibbs

instagibbs May 12, 2016

Member

nit: simply calling AddKey should work here nevermind, wrong function

This comment has been minimized.

@sipa

sipa Jun 1, 2016

Member

I'd like to know why this comment does not apply.

EDIT: because you need pubkey below anyway, ok!

@sipa

sipa Jun 1, 2016

Member

I'd like to know why this comment does not apply.

EDIT: because you need pubkey below anyway, ok!

@instagibbs

This comment has been minimized.

Show comment
Hide comment
@instagibbs

instagibbs May 12, 2016

Member

utACK cc1df8f

Glad to see a really simple PR. Baby steps better than no steps.

Member

instagibbs commented May 12, 2016

utACK cc1df8f

Glad to see a really simple PR. Baby steps better than no steps.

@jonasschnelli

This comment has been minimized.

Show comment
Hide comment
@jonasschnelli

jonasschnelli May 12, 2016

Member

Added a squash me commit with some non-code nit fixes.

Member

jonasschnelli commented May 12, 2016

Added a squash me commit with some non-code nit fixes.

@pstratem

View changes

Show outdated Hide outdated src/wallet/wallet.cpp
@pstratem

View changes

Show outdated Hide outdated src/wallet/wallet.cpp
@sipa

This comment has been minimized.

Show comment
Hide comment
@sipa

sipa Jun 12, 2016

Member
Member

sipa commented Jun 12, 2016

@MarcoFalke

This comment has been minimized.

Show comment
Hide comment
@MarcoFalke

MarcoFalke Jun 12, 2016

Member

Ok makes sense. (But that is different from txindex, as I remember you can't set -txindex=1 once and then remove it, to make it default to true. I may be mistaken but I should better not hijack this pull's discussion.)

Member

MarcoFalke commented Jun 12, 2016

Ok makes sense. (But that is different from txindex, as I remember you can't set -txindex=1 once and then remove it, to make it default to true. I may be mistaken but I should better not hijack this pull's discussion.)

@instagibbs

This comment has been minimized.

Show comment
Hide comment
@instagibbs

instagibbs Jun 12, 2016

Member

@MarcoFalke it's slightly different, yes, but txindex behavior wouldn't make sense since you can't "reindex" your HD wallet to legacy.

Member

instagibbs commented Jun 12, 2016

@MarcoFalke it's slightly different, yes, but txindex behavior wouldn't make sense since you can't "reindex" your HD wallet to legacy.

@jonasschnelli

This comment has been minimized.

Show comment
Hide comment
@jonasschnelli

jonasschnelli Jun 13, 2016

Member

Added a commit.
Now the PR detects if a user likes to disabled hd (with -usehd=0) when using a hd wallet.dat file.
The PR also detects if a user likes to enable hd (with -usehd=1) when using a non hd wallet.dat file.

The node will shutdown/stop init in a such mismatch situation.

Member

jonasschnelli commented Jun 13, 2016

Added a commit.
Now the PR detects if a user likes to disabled hd (with -usehd=0) when using a hd wallet.dat file.
The PR also detects if a user likes to enable hd (with -usehd=1) when using a non hd wallet.dat file.

The node will shutdown/stop init in a such mismatch situation.

@MarcoFalke

This comment has been minimized.

Show comment
Hide comment
@MarcoFalke

MarcoFalke Jun 13, 2016

Member

utACK 71ff55fcc2245e03c786d7e0312755e812d9de8e

Member

MarcoFalke commented Jun 13, 2016

utACK 71ff55fcc2245e03c786d7e0312755e812d9de8e

@laanwj

View changes

Show outdated Hide outdated src/wallet/wallet.cpp
@laanwj

This comment has been minimized.

Show comment
Hide comment
@laanwj

laanwj Jun 14, 2016

Member

utACK afcd77e

Even though this wallet change is reasonably risky, I think this has received enough review for correctness, and it should be merged now to make sure it gets more actual testing.

Member

laanwj commented Jun 14, 2016

utACK afcd77e

Even though this wallet change is reasonably risky, I think this has received enough review for correctness, and it should be merged now to make sure it gets more actual testing.

@laanwj laanwj merged commit afcd77e into bitcoin:master Jun 14, 2016

1 check passed

continuous-integration/travis-ci/pr The Travis CI build passed
Details

laanwj added a commit that referenced this pull request Jun 14, 2016

Merge #8035: [Wallet] Add simplest BIP32/deterministic key generation…
… implementation

afcd77e Detect -usehd mismatches when wallet.dat already exists (Jonas Schnelli)
17c0131 [Docs] Add release notes and bip update for Bip32/HD wallets (Jonas Schnelli)
c022e5b [Wallet] use constant for bip32 hardened key limit (Jonas Schnelli)
f190251 [Wallet] Add simplest BIP32/deterministic key generation implementation (Jonas Schnelli)
@gmaxwell

This comment has been minimized.

Show comment
Hide comment
@gmaxwell

gmaxwell Jun 15, 2016

Member

Post merge utACK. Will also test.

Member

gmaxwell commented Jun 15, 2016

Post merge utACK. Will also test.

@schildbach

This comment has been minimized.

Show comment
Hide comment
@schildbach

schildbach Jun 15, 2016

Contributor

Just found out about this PR. Would like to clarify the keypath: It's purpose number is 0' not 0? ((m/0'/0'/k') not (m/0/0'/k'))

I'm asking because otherwise it could be incompatible to how bitcoinj derives addresses.

Contributor

schildbach commented Jun 15, 2016

Just found out about this PR. Would like to clarify the keypath: It's purpose number is 0' not 0? ((m/0'/0'/k') not (m/0/0'/k'))

I'm asking because otherwise it could be incompatible to how bitcoinj derives addresses.

@jonasschnelli

This comment has been minimized.

Show comment
Hide comment
@jonasschnelli

jonasschnelli Jun 17, 2016

Member

@schildbach: We have decided to go with a "hardened only derivation" version for the first implementation. Using m/0/0'/k' would introduce the risk of revealing extended pubkey m/0 together with a non extended child privatekey resulting in revealing the whole chain of keys.

On top of that, I personally think importing a xpriv into another wallet in order to "reconstruct everything" is not a very good concept.

Member

jonasschnelli commented Jun 17, 2016

@schildbach: We have decided to go with a "hardened only derivation" version for the first implementation. Using m/0/0'/k' would introduce the risk of revealing extended pubkey m/0 together with a non extended child privatekey resulting in revealing the whole chain of keys.

On top of that, I personally think importing a xpriv into another wallet in order to "reconstruct everything" is not a very good concept.

@schildbach

This comment has been minimized.

Show comment
Hide comment
@schildbach

schildbach Jun 17, 2016

Contributor

@jonasschnelli Yes that's fine. I just wanted to make sure there is no "partial overlap" with the bitcoinj impl.

Contributor

schildbach commented Jun 17, 2016

@jonasschnelli Yes that's fine. I just wanted to make sure there is no "partial overlap" with the bitcoinj impl.

@rebroad

This comment has been minimized.

Show comment
Hide comment
@rebroad

rebroad Jun 25, 2016

Contributor

So, how does one import a BIP32 master key into an existing wallet?

Contributor

rebroad commented Jun 25, 2016

So, how does one import a BIP32 master key into an existing wallet?

@diegoviola

This comment has been minimized.

Show comment
Hide comment
@diegoviola

diegoviola Jun 25, 2016

Contributor

I'm switching to Core (from Electrum) after this lands in 0.13, thanks for your efforts.

Contributor

diegoviola commented Jun 25, 2016

I'm switching to Core (from Electrum) after this lands in 0.13, thanks for your efforts.

@laanwj

This comment has been minimized.

Show comment
Hide comment
@laanwj

laanwj Jun 27, 2016

Member

So, how does one import a BIP32 master key into an existing wallet?

That functionality currently doesn't exist. Please read the OP: this is the smallest possible change to introduce BIP32 into Bitcoin Core. This made it realistic to get it reviewed and merged in a small timespan. More functionality can be added later.

Member

laanwj commented Jun 27, 2016

So, how does one import a BIP32 master key into an existing wallet?

That functionality currently doesn't exist. Please read the OP: this is the smallest possible change to introduce BIP32 into Bitcoin Core. This made it realistic to get it reviewed and merged in a small timespan. More functionality can be added later.

// key metadata is not required
CPubKey pubkey = key.GetPubKey();
if (!AddKeyPubKey(key, pubkey))
throw std::runtime_error("CWallet::GenerateNewKey(): AddKey failed");

This comment has been minimized.

@MarcoFalke

MarcoFalke Jul 8, 2016

Member

Nit: Wrong function name

@MarcoFalke

MarcoFalke Jul 8, 2016

Member

Nit: Wrong function name

@TheBlueMatt

This comment has been minimized.

Show comment
Hide comment
@TheBlueMatt

TheBlueMatt Jul 19, 2016

Contributor

Post-merge review comments, sorry :/ (continued from IRC):
There is also a comment on #8324 and other related PRs here.

  • The use of an ECDSA private key (stored as such) as the master seed is somewhat strange...While not strictly an issue, I'm concerned about the ability of users to dump this as a regular key, potentially causing some interesting behavior.
  • The way I read the current code, every time we go to GenerateNewKey() we will iterate over every key we have already generated before generating a new one (as we never call SetHDChain to store the updated counter on disk)....anyone with a test hd wallet lying around want to see what their disk-counter is?
  • I think IsKeyType should be changed to include hdchain, not add an explicit hdchain check in ::Recover, since we probably definitely want it in ::LoadWallet as well.
  • The duplication of the old CTransaction READWRITE(this->nVersion); nVersion = this->nVersion; logic seems strange to me. I always found that a kinda funny way to do things and was glad to see it go in SegWit...Its "hidden functionality" and I'd prefer if we handled that very explicitly instead of via this strange hack.

I think all but the first could easily be fixed even now, if we decide they are, indeed, issues.

Contributor

TheBlueMatt commented Jul 19, 2016

Post-merge review comments, sorry :/ (continued from IRC):
There is also a comment on #8324 and other related PRs here.

  • The use of an ECDSA private key (stored as such) as the master seed is somewhat strange...While not strictly an issue, I'm concerned about the ability of users to dump this as a regular key, potentially causing some interesting behavior.
  • The way I read the current code, every time we go to GenerateNewKey() we will iterate over every key we have already generated before generating a new one (as we never call SetHDChain to store the updated counter on disk)....anyone with a test hd wallet lying around want to see what their disk-counter is?
  • I think IsKeyType should be changed to include hdchain, not add an explicit hdchain check in ::Recover, since we probably definitely want it in ::LoadWallet as well.
  • The duplication of the old CTransaction READWRITE(this->nVersion); nVersion = this->nVersion; logic seems strange to me. I always found that a kinda funny way to do things and was glad to see it go in SegWit...Its "hidden functionality" and I'd prefer if we handled that very explicitly instead of via this strange hack.

I think all but the first could easily be fixed even now, if we decide they are, indeed, issues.

@jonasschnelli

This comment has been minimized.

Show comment
Hide comment
@jonasschnelli

jonasschnelli Jul 19, 2016

Member

Thanks for the post-merge review Matt.

The use of an ECDSA private key (stored as such) as the master seed is somewhat strange...While not strictly an issue, I'm concerned about the ability of users to dump this as a regular key, potentially causing some interesting behavior.

Yes. Agree. This was done after two of my former PR did not made it into the master (#6265 #7273) and I was looking for a simpler solutions.
There is a PR to identify the seed when dumping (#8206).

The way I read the current code, every time we go to GenerateNewKey() we will iterate over every key we have already generated before generating a new one (as we never call SetHDChain to store the updated counter on disk)....anyone with a test hd wallet lying around want to see what their disk-counter is?

This is to ensure we don't re-generate an already existing key (there are some cases where this would be possible otherwise).
The iteration should be painless. It's a std::map count() of an in memory map of all the key.

Member

jonasschnelli commented Jul 19, 2016

Thanks for the post-merge review Matt.

The use of an ECDSA private key (stored as such) as the master seed is somewhat strange...While not strictly an issue, I'm concerned about the ability of users to dump this as a regular key, potentially causing some interesting behavior.

Yes. Agree. This was done after two of my former PR did not made it into the master (#6265 #7273) and I was looking for a simpler solutions.
There is a PR to identify the seed when dumping (#8206).

The way I read the current code, every time we go to GenerateNewKey() we will iterate over every key we have already generated before generating a new one (as we never call SetHDChain to store the updated counter on disk)....anyone with a test hd wallet lying around want to see what their disk-counter is?

This is to ensure we don't re-generate an already existing key (there are some cases where this would be possible otherwise).
The iteration should be painless. It's a std::map count() of an in memory map of all the key.

@TheBlueMatt

This comment has been minimized.

Show comment
Hide comment
@TheBlueMatt

TheBlueMatt Jul 19, 2016

Contributor

re: GenerateNewKey(): Not sure what your response was about (seems to be communication failure there), but it was pointed out to me that I missed the save line and was incorrect.

Contributor

TheBlueMatt commented Jul 19, 2016

re: GenerateNewKey(): Not sure what your response was about (seems to be communication failure there), but it was pointed out to me that I missed the save line and was incorrect.

@zihad944

This comment has been minimized.

Show comment
Hide comment
@zihad944

zihad944 Aug 11, 2018

I foget my sceond wallet passwor. Thats why i cant get backup. I need help

zihad944 commented Aug 11, 2018

I foget my sceond wallet passwor. Thats why i cant get backup. I need help

@jeandudey

This comment has been minimized.

Show comment
Hide comment
@jeandudey

jeandudey Aug 11, 2018

@zihad944 what problem do you have specifically, what do you mean by "my second wallet password"?

jeandudey commented Aug 11, 2018

@zihad944 what problem do you have specifically, what do you mean by "my second wallet password"?

@zihad944

This comment has been minimized.

Show comment
Hide comment
@zihad944

zihad944 Aug 11, 2018

I use sceons password in my block chain. but i forget it.how i cant send money. I cant see my backu.

zihad944 commented Aug 11, 2018

I use sceons password in my block chain. but i forget it.how i cant send money. I cant see my backu.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment