Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[WIP] Implement BIP32 HD account class - 2 #31

Closed
wants to merge 27 commits into from

Conversation

sea212
Copy link

@sea212 sea212 commented Aug 26, 2018

What was wrong?

Issue #24

How was it fixed?

I have created the hdaccounts.py file containing the class HDAccount(BaseAccount). I have defined the functions I will need and commented their meaning, usage and further steps. I have left TODOs which I will work on.
Question: Is it okay if I import the files "mnemonic.py" and "deterministic.py" from pybitcointools and if yes, do you want me to adapt them to reduce the overhead of the required libraries or do you want me to write them completely from scratch?

Note: I have an unusually high amount of work last week and in the coming week. I will spend just a couple of hours in the coming week, but continuous progress will be observable. Afterwards the progress will be significantly accelerated.

@pipermerriam
Copy link
Member

I have an unusually high amount of work last week and in the coming week. I will spend just a couple of hours in the coming week, but continuous progress will be observable. Afterwards the progress will be significantly accelerated.

👍 as long as there is progress being made.

@pipermerriam
Copy link
Member

pipermerriam commented Aug 27, 2018

Question: Is it okay if I import the files "mnemonic.py" and "deterministic.py" from pybitcointools and if yes, do you want me to adapt them to reduce the overhead of the required libraries or do you want me to write them completely from scratch?

I think this is clear but adding pybitcointools as a dependency is off the table as it is unmaintained.

I would advocate porting over the necessary components (and cleaning them up). This would also include test cases to demonstrate how the various utils are intended to function.

@sea212
Copy link
Author

sea212 commented Aug 29, 2018

I will do so.

sea212 added 6 commits August 31, 2018 15:59
…ed code to required entropy byte count be an element of [16,20,24,28,32], as described in BIP32
…path to setup.py, debugged mnemonic.py to work with python3, adapted function commenting style for the most functions
@sea212
Copy link
Author

sea212 commented Sep 1, 2018

Today I have:

  • Implemented Init from mnemonic function and create Account function
  • Tested functionallity of Init and Create functions
  • Adjusted setup.py to include wordlist path and the containing wordlists
  • Debugged mnemonic.py to work with python3
  • Adapted the given function commenting style for almost every function

Output from the current tests:

Creating new Account
Mnemonic code: orphan judge waste senior park face shed renew version vintage hole smart remember spatial merry pupil endorse exile turn team basket dog school skill
Password: supersecret
Key: xprv9s21ZrQH143K4ADdpA7Lpu968iB8twNxZzQieJMKT5JeMTKrRUzQkVbj3Xw3M8tk6WpkAdkbbd2ps8pG3hKRhNjKvRrqWxgXUarYxdjfRVN

Initializing new Account with the same mnemonic and password
Key: xprv9s21ZrQH143K4ADdpA7Lpu968iB8twNxZzQieJMKT5JeMTKrRUzQkVbj3Xw3M8tk6WpkAdkbbd2ps8pG3hKRhNjKvRrqWxgXUarYxdjfRVN
Success!

@sea212
Copy link
Author

sea212 commented Sep 1, 2018

TODOs

  • Implement createAccount(...) (given a pw: generate mnemonic, master keys and chain code)
  • Test createAccount(...)
  • Implement initAccount(...) (given a mnemonic and pw: create master keys and chain code)
  • Adapt setup.py for word lists
  • Implement _accountGenerator(...) which contains a generator for child derivation
  • Implement deriveChild(...)
  • Test deriveChild(...)
  • Implement derivePath(...)
  • Test derivePath(...)
  • Implement removePrivateKey(...) - limits the current HDAccount object to public key derivation only
  • Test removePrivateKey(...)
  • Implement signTransaction(...)
  • Test signTransaction(...)
  • Implement signHash(...)
  • Test signHash(...)
  • Implement address property
  • Test address property
  • Test mnemnic.py with the official test vectors
  • Test HDAccount class with the official test vectors
  • Adapt function comment style for every function
  • Final cleanup for the merge

…path argument support for init. Added and implemented decodePath(...) function to represent the derivation path as a string in format (m/)idx_0/.../idx_n . Added path property to return the string formatted derivation path. Added __repr__ and __str__, adjusted __eq__ and __hash__. Included (very basic) unit test for deriveChild(...) . Adapted function describing commenting style for every function.
@sea212
Copy link
Author

sea212 commented Sep 2, 2018

Today I have:

  • Implemented _accountGenerator() and deriveChild(...)
  • Added derivation path argument support for __init__
  • Added and implemented decodePath(...) function to represent the derivation path as a string in format (m/)idx_0/.../idx_n
  • Added path property to return the string formatted derivation path
  • Added __repr__ and __str__, adjusted __eq__ and __hash__
  • Included (very basic) unit tests for deriveChild(...)
  • Adapted function describing commenting style for every function.

Output of basic unit tests:

Encoded key: xprv9tvdB7KcQQphsdGAKPAg5X7qGHzv3Mz5SVDxbw62DTgWokeSWegUfNqwXFV4YW7gNdeL4Md4cuf9FaqLTDUZZDzbqPyo5kprVQv1VU4PNJq
Derivation Path: m/0H
Encoded key: xprv9tvdB7KcQQphwug8Lc1JHGsDATdYbhSzHHXENWikCT3BGudyjSAtq2JYiaKBpUqscoi7oMZi4kyBKhA5GtfxhRiZdhxKxQbDnzHP3W7m3qX
Derivation Path: m/1H
Encoded key: xprv9x18VoCx1ZrkhgErq2VaBZmR27ddp1TBQUSES8TRv4SSmxc7N2q2ApFuuipeiL2sgsEvBGH9T1CH34ddFctecRfdY26ofe5wKNcKwVwifok
Derivation Path: m/1H/0
Encoded key: xprv9x18VoCx1ZrkiFSzykPAuLejgm8bbbCv4aqskZ2DGgY4QSfKjvRuuQWF5n8XD47D3ktCND4EK9NuRQUd9zYKjrM7tb2PhqALCg9MkzJMXBw
Derivation Path: m/1H/1
Encoded key: xprv9yQWjG5Ko8deskqJUharhjoD3B8cyy6ysKDixqV62PXRVh2B78B2SzfDNw4LqrSCPtLptH9i3ukERyRzjxnciVZxpwW9qEF3GVQ3hyyPLsQ
Derivation Path: m/1H/1/42
Encoded key:
Derivation Path: m/

Success!

@sea212
Copy link
Author

sea212 commented Sep 3, 2018

Today I have:

  • Implemented derivePath(...)
  • Tested derivePath(...)

The BIP32 implementation is now complete, only the Ethereum address derivation function and transaction signing function are missing.

Test outputs (compares derived path results with single child derivation results):

Encoded key: xprv9xzsFFH33gkXho4xvFQ1EhCSptvibyFx8qNqYe1i29NzNg7oAw9RSWeX5F3LSZDhKRp5GSErZ3ohUwwEcVXXGwVt4Wo7Ws76SzbrxjZYBvH
Derivation Path: m/1H/1/42

--- TEST 5: Derive path ---

Encoded key: xprv9xzsFFH33gkXho4xvFQ1EhCSptvibyFx8qNqYe1i29NzNg7oAw9RSWeX5F3LSZDhKRp5GSErZ3ohUwwEcVXXGwVt4Wo7Ws76SzbrxjZYBvH
Derivation Path: m/1H/1/42
Encoded key: xprv9xzsFFH33gkXho4xvFQ1EhCSptvibyFx8qNqYe1i29NzNg7oAw9RSWeX5F3LSZDhKRp5GSErZ3ohUwwEcVXXGwVt4Wo7Ws76SzbrxjZYBvH
Derivation Path: m/1H/1/42

Success!

…..), signTransaction(...) . Added value check to deterministic.py
@sea212
Copy link
Author

sea212 commented Sep 4, 2018

Today I have:

  • Implemented and tested removePrivateKey(...) function
  • Implemented and tested address property
  • Implemented and tested signMessage(...) function
  • Implemented and tested signTransaction(...) function

The class is now complete and fully functional. You can inspect one successful outgoing transaction using a key in the path "m/1H/1/42": https://rinkeby.etherscan.io/address/0x56fe41bd2f1993a2b460bbfa1eac838188e0c5c3

Class test outputs - click to expand --- TEST 1: Create HDAccount ---

Creating new Account
Mnemonic code: gauge screen cube crowd general twin matrix senior adult spatial sail obscure annual kiwi fringe hidden gentle zoo barely job edit bachelor key soap
Password: supersecret
Key: xprv9s21ZrQH143K2uW37T4nqFGRnYQmGCSirH2DU7w7TSVawKQAjcfXdFF7U4moHYuw8BtFKejYjh7GZCjk8Xxa2Qgngen3VpdMX9ehjNDFLdw

--- TEST2: Init account ---

Key: xprv9s21ZrQH143K2uW37T4nqFGRnYQmGCSirH2DU7w7TSVawKQAjcfXdFF7U4moHYuw8BtFKejYjh7GZCjk8Xxa2Qgngen3VpdMX9ehjNDFLdw

--- TEST3: Derive children ---

Encoded key: xprv9u5ZNQNQnoxqbGtoeH8vqcCcHM6e8tu3MG1PR1TjJKrA3DzjKxTssm3tnqgS8aDzHxqHzQiauWrCbsjmAh2Xg6LsxaBCrzFL9HYgYkC4uyA
Checksummed address: 0xF21a936878e2555eAE8F4E91ce0bf558073177CD
Derivation Path: m/0H
Encoded key: xprv9u5ZNQNQnoxqcNGDxSTWw8p6zGgXbPdUBbtzN43usQrZVvJxNixokjLNdXiCTTBUbKkhybhju9hSHsr89BEPJ5rAKat5jAF2mjftsxPBdC3
Checksummed address: 0xba908CC0edC58cC715fAC41Eaf95243CC26F913B
Derivation Path: m/1H
Encoded key: xprv9wBQM4fxEY5wq4duDkmiCZVjb2FSCe6sn8S4Ue65Ho3VMsQxFtrhFAku2gZkamRcSE7gWMR1Tb3kXqdeRfFf1edVxLcmjSBDS3MY4gGRRYL
Checksummed address: 0x133fc821b499D9aE6102c7C91F0A814C53D88E22
Derivation Path: m/1H/0
Encoded key: xprv9wBQM4fxEY5wu4hBCk3nhH85hCEvZ1WtJ5wf41gPDE3L4F9tnFVamx8mAmHD7i37XCdDzNPWxejWgnVLqQUrcsY2UY3W9VaZmVufCrBe652
Checksummed address: 0x8F537D24219790539fb47ebb1F53DB03B0ebeaBc
Derivation Path: m/1H/1
Encoded key: xprv9xyMrSmyo6JvTvX8QQU78ynuHRD6SdwcGtwLB5nxQCd84vje6F4b1HCT6EPbh6hkL8oqEyLp9uqZtxLxGaJTC2JXDwV5Je298Z9FakJbGad
Checksummed address: 0xDF512A1E0Cbe32b63cEc8387BccA63A3b78a8c22
Derivation Path: m/1H/1/42

--- TEST4: Remove private key (only public key derivation possible) ---

Encoded key: xpub68DbfiPkyyZPwSJ6xhAtWTbxU4aubRNP7Hog1azeuAztmaXGPXPCtCNd56C4Fqp8GnvfPDofzKjs4ctrfqnoCSY1HDgFw4F6PzdSWg93Sry
Checksummed address: 0xd1B8286F692Ce83f0ED9D998EDd3149975A5298C
Derivation Path: m/0
Encoded key: xpub6BDGpSpiQsvNWZNUWkZgMiyka9u1MDfM89o6BqDXgKaUBp4MmXQhXa6DVmzUfv7GRBpHPWAox6QoebtkYJ6r1zgmMC1rpQTbpEJrfG1sUiK
Checksummed address: 0x907fF02F4ae1428B11B76724057Ddb3d3AbEb3ef
Derivation Path: m/0/42

--- TEST 5: Derive path ---

Encoded key: xprv9xyMrSmyo6JvTvX8QQU78ynuHRD6SdwcGtwLB5nxQCd84vje6F4b1HCT6EPbh6hkL8oqEyLp9uqZtxLxGaJTC2JXDwV5Je298Z9FakJbGad
Checksummed address: 0xDF512A1E0Cbe32b63cEc8387BccA63A3b78a8c22
Derivation Path: m/1H/1/42
Encoded key: xprv9xyMrSmyo6JvTvX8QQU78ynuHRD6SdwcGtwLB5nxQCd84vje6F4b1HCT6EPbh6hkL8oqEyLp9uqZtxLxGaJTC2JXDwV5Je298Z9FakJbGad
Checksummed address: 0xDF512A1E0Cbe32b63cEc8387BccA63A3b78a8c22
Derivation Path: m/1H/1/42

--- TEST 6: Get address from xprv and xpub encoded keys ---

0xDF512A1E0Cbe32b63cEc8387BccA63A3b78a8c22
0x907fF02F4ae1428B11B76724057Ddb3d3AbEb3ef

Success!

--- TEST 7: Sign message hash ---

signing key : xprv9xyMrSmyo6JvTvX8QQU78ynuHRD6SdwcGtwLB5nxQCd84vje6F4b1HCT6EPbh6hkL8oqEyLp9uqZtxLxGaJTC2JXDwV5Je298Z9FakJbGad
hash to sign: 0xe54dcafaa4e992ef11154cdf38fe164bf2a2bb42315c67eeec69a1fa03747133
Result:
AttrDict({'messageHash': HexBytes('0xe54dcafaa4e992ef11154cdf38fe164bf2a2bb42315c67eeec69a1fa03747133'), 'r': 52716771902183800620450856972009666893819453032428441395814016857664242610079, 's': 13068880653550716507809063503867817178309813034766052429602509898569566649641, 'v': 27, 'signature': HexBytes('0x748ca2b1d6f3f32ff92cbbfee93561806712a369808d4664b3149959cac46f9f1ce4b96d070a5921c33b792b8fb4e1a866232708cac2dbaf846112cb3dc7fd291b')})

--- TEST 8: Create and sign Transaction ---

Encoded key: xprv9y8Gw5q3qFv42CiuohDV6L8gY9VHjs74dMKos2pVp74vyHFRiTHwPHFDAEgzYDYDQYEh2yahi4Jga6qQn3q45yzvPcPibUaKNRrkTbsDHkK
Checksummed address: 0x56fe41bd2F1993a2B460BBfA1eAC838188E0C5c3
Derivation Path: m/1H/1/42
transaction to sign:
{'to': '0xF0109fC8DF283027b6285cc889F5aA624EaC1F55', 'value': 1000000000, 'gas': 21000, 'gasPrice': 2000000000, 'nonce': 0, 'chainId': 4}
Signed Transaction:
AttrDict({'rawTransaction': HexBytes('0xf86780847735940082520894f0109fc8df283027b6285cc889f5aa624eac1f55843b9aca00802ca0e4f3b89b8c71ee52b228332d4c37d4f13c4e326107464cfda36552e7b5f5f819a04a499e3cc31b00f788b3a360b5ad96df234751f9ca55d45663a6340d982830bd'), 'hash': HexBytes('0x696d8236a23f0fefed1fa0676eb512ae9298ce998caa404218b88f5ceba448d0'), 'r': 103557947428636016120980309821108293220895437911034125569304901028935227537433, 's': 33601222744957133318728985348133226030714082634767453431581995465123320836285, 'v': 44})

@sea212
Copy link
Author

sea212 commented Sep 5, 2018

I have completed all remaining tasks now (test and cleanup mostly), the HDAccount class is now complete and fully functional. Isort worked as excepted before and the automated tests found no issues, but since I added some more imports today the automated tests find errors. My local isort does tell me that the imports are fine. I have tried different options, like ignoring types, but I just can't get the automated testing tool to accept the file. Sadly the error report does not give me any hint what is not ok in detail. Do you know what could be the problem?

apart from that, are you content with this result and can I submit it to gitcoin?

You can inspect one successful outgoing transaction usingf a key in the path "m/1H/1/42": https://rinkeby.etherscan.io/address/0x56fe41bd2f1993a2b460bbfa1eac838188e0c5c3

Final output:

Class test outputs - click to expand --- TEST 1: Create HDAccount ---

Creating new Account
Mnemonic code: potato extra script rigid modify code supply gesture flash regular main detail position answer ugly song type smoke either track spoil above fly arrange
Password: supersecret
Key: xprv9s21ZrQH143K3jVLNgf8Fui3jCtWPcqiQiRiaAK7ePNNJZ9JtGyGTGgcZ1vTvvupog3fNh1fKX1NUnXktUb7HkHm8RPF3T1HR2fX9azVBoU

--- TEST2: Init account ---

Key: xprv9s21ZrQH143K3jVLNgf8Fui3jCtWPcqiQiRiaAK7ePNNJZ9JtGyGTGgcZ1vTvvupog3fNh1fKX1NUnXktUb7HkHm8RPF3T1HR2fX9azVBoU

--- TEST3: Derive children ---

Encoded key: xprv9vVBpg3dEWZik1QQVSm8sqnnKeo5yrHJ4LAKy3xMPmdyibFzhycEH39yrC9UGbAZ4CqPFiP8RmfMJHNar9QJvHcDuCbvfJJq18f5ig6DcM1
Checksummed address: 0x495EFDaaD954B4E57eD1bee4ab1148f15C260c6B
Derivation Path: m/0H
Encoded key: xprv9vVBpg3dEWZinKEdXafgBTy5tuwYnGV314feeHwoZTumS3bhRN5ymiPzygD1EYJVgiYx7jZCmsDXyzPdL3WNTD8dSjoRx74gBtDCJLyxXJ9
Checksummed address: 0x96e77150CB3616c78c68DC2852F2F2B179fCe407
Derivation Path: m/1H
Encoded key: xprv9x6FHsPSVQEn5dHrEYEtHj3N7jxEcnSM2iPWd2Cb7bAX4nbyjVXymqtEwtzaAjELDbDmipw7LXw3Di1J9nRVC9Wq2Mue3ETQR1vVV8vxhtP
Checksummed address: 0x1127429E344f5925A245E83b0989db2DBFD289a3
Derivation Path: m/1H/0
Encoded key: xprv9x6FHsPSVQEn5osdeFTN7LPEViECmDMcj3JiEGytTzvs6RfmLKr8BFdX9qJ429FVFAbbvvtXNPD3ZCrYwsYshEVfiyn3hFfAr2xisp2g69z
Checksummed address: 0x4efc0C43d376194352d264A6849e9D891f2D8271
Derivation Path: m/1H/1
Encoded key: xprv9zJqmMpcSQnyLhJmcANVdrQF6tLNahrRkq6zrhmBxJMooKqnhvrwP84Yv8PPSQn9NJCefbZ5SJLdDUMcAe31ZMWYaMa2B243LH5571bWepY
Checksummed address: 0xB2f88570049b43E901769412A7180e20969cfE5C
Derivation Path: m/1H/1/42

--- TEST4: Remove private key (only public key derivation possible) ---

Encoded key: xpub69NmrBJkLnq5fVt2ye6zJ4vXxwKAbpRAzcALGLjURqfXCagrV5cfbHu9tbGfbHRp1AZ9u4j6ZffDBX4eLykvNhBbGyNKeSac5ErJquFqsFt
Checksummed address: 0x4897D7D77F0F4F63D513b79787ed475C494D15fa
Derivation Path: m/0
Encoded key: xpub6BRh7LDGgTUFmm6G6mpgxoxCVKwG5CfQEZ7dxJWQXERFpEZsL6WXRkZ8w6wxKhsKU3Q7J222qLGXXKJh3VMpGjrHJZwhwWcfxEbTDr8vanV
Checksummed address: 0xb890AC0e7886adD8C12fD734e061b625B358E98E
Derivation Path: m/0/42

--- TEST 5: Derive path ---

Encoded key: xprv9zJqmMpcSQnyLhJmcANVdrQF6tLNahrRkq6zrhmBxJMooKqnhvrwP84Yv8PPSQn9NJCefbZ5SJLdDUMcAe31ZMWYaMa2B243LH5571bWepY
Checksummed address: 0xB2f88570049b43E901769412A7180e20969cfE5C
Derivation Path: m/1H/1/42
Encoded key: xprv9zJqmMpcSQnyLhJmcANVdrQF6tLNahrRkq6zrhmBxJMooKqnhvrwP84Yv8PPSQn9NJCefbZ5SJLdDUMcAe31ZMWYaMa2B243LH5571bWepY
Checksummed address: 0xB2f88570049b43E901769412A7180e20969cfE5C
Derivation Path: m/1H/1/42

--- TEST 6: Get address from xprv and xpub encoded keys ---

0xB2f88570049b43E901769412A7180e20969cfE5C
0xb890AC0e7886adD8C12fD734e061b625B358E98E

Success!

--- TEST 7: Sign message hash ---

signing key : xprv9zJqmMpcSQnyLhJmcANVdrQF6tLNahrRkq6zrhmBxJMooKqnhvrwP84Yv8PPSQn9NJCefbZ5SJLdDUMcAe31ZMWYaMa2B243LH5571bWepY
hash to sign: 0x26ab295e350759d4631a9c93a427718206c35de438106a6a4303eaf3e036341a
Result:
AttrDict({'messageHash': HexBytes('0x26ab295e350759d4631a9c93a427718206c35de438106a6a4303eaf3e036341a'), 'r': 81544346475880138480428342760766354799599863053956900835828043150951167587054, 's': 34640376631269739072075483558166797841395877121027731277635804788635462682515, 'v': 27, 'signature': HexBytes('0xb44876eb056ddd26d449c75ed47669d4597467991c1f4016e7f2b607eb6072ee4c95c224a0db3d85ace5a874fea0576c4281ffcd27140176157f97db8b3247931b')})

--- TEST 8: Create and sign Transaction ---

Encoded key: xprv9y8Gw5q3qFv42CiuohDV6L8gY9VHjs74dMKos2pVp74vyHFRiTHwPHFDAEgzYDYDQYEh2yahi4Jga6qQn3q45yzvPcPibUaKNRrkTbsDHkK
Checksummed address: 0x56fe41bd2F1993a2B460BBfA1eAC838188E0C5c3
Derivation Path: m/1H/1/42
transaction to sign:
{'to': '0xF0109fC8DF283027b6285cc889F5aA624EaC1F55', 'value': 1000000000, 'gas': 21000, 'gasPrice': 2000000000, 'nonce': 0, 'chainId': 4}
Signed Transaction:
AttrDict({'rawTransaction': HexBytes('0xf86780847735940082520894f0109fc8df283027b6285cc889f5aa624eac1f55843b9aca00802ca0e4f3b89b8c71ee52b228332d4c37d4f13c4e326107464cfda36552e7b5f5f819a04a499e3cc31b00f788b3a360b5ad96df234751f9ca55d45663a6340d982830bd'), 'hash': HexBytes('0x696d8236a23f0fefed1fa0676eb512ae9298ce998caa404218b88f5ceba448d0'), 'r': 103557947428636016120980309821108293220895437911034125569304901028935227537433, 's': 33601222744957133318728985348133226030714082634767453431581995465123320836285, 'v': 44})

--- TEST 9: BIP32 Testvector 1 ---

Success!

--- TEST 10: BIP32 Testvector 2 ---

Success!

--- TEST 11: BIP32 Testvector 3 ---

All tests were successful!

@pipermerriam pipermerriam mentioned this pull request Sep 25, 2018
@pipermerriam
Copy link
Member

Closed in favor of #33

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants