Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add key origin support to descriptors #14150

Merged
merged 3 commits into from Oct 22, 2018

Conversation

Projects
None yet
@sipa
Copy link
Member

commented Sep 4, 2018

This adds support for key origin information to the descriptor parser, and exposes the resulting key path information through FlatSigningProvider.

There is no observable functionality from this right now, except having the scantxoutset RPC accept descriptors that include key origin information.

Longer term this feature helps with a potential descriptors-based walletless PSBT updater, or for importing hardware wallet xpubs (once the wallet can import descriptors).

@laanwj laanwj added the RPC/REST/ZMQ label Sep 6, 2018

Show resolved Hide resolved src/script/descriptor.cpp Outdated
Show resolved Hide resolved src/script/descriptor.cpp

@sipa sipa force-pushed the sipa:201807_minedesc_origin branch Sep 6, 2018

@jonasschnelli

This comment has been minimized.

Copy link
Member

commented Sep 6, 2018

utACK e11d7ea24b7c67793ab8e84b5ad05d233f43e7da

@Sjors
Copy link
Member

left a comment

Concept ACK.

For me this would be easier to review when combined with a practical usage, like creating a PSBT with the origin info still in it. That could just be a WIP PR as it might involve quite a few changes, since descriptors currently are only used with scantxoutset.

Show resolved Hide resolved src/test/descriptor_tests.cpp
Show resolved Hide resolved src/test/descriptor_tests.cpp Outdated
Show resolved Hide resolved src/test/descriptor_tests.cpp Outdated
Show resolved Hide resolved src/script/sign.h Outdated
Show resolved Hide resolved src/script/descriptor.cpp Outdated
Show resolved Hide resolved src/script/descriptor.cpp Outdated

@sipa sipa force-pushed the sipa:201807_minedesc_origin branch Sep 10, 2018

@Sjors

This comment has been minimized.

Copy link
Member

commented Sep 13, 2018

Perhaps @achow101 can use this in #14021?

@achow101

This comment has been minimized.

Copy link
Member

commented Sep 13, 2018

@Sjors I don't think this would be useful there. Instead we should have some new command that lets us import descriptors into the wallet.

src/script/descriptor.cpp Outdated
CKeyID keyid = m_extkey.pubkey.GetID();
std::copy(keyid.begin(), keyid.begin() + 4, info.fingerprint);
info.path = m_path;
if (m_derive == DeriveType::UNHARDENED) info.path.push_back(pos);

This comment has been minimized.

Copy link
@practicalswift

practicalswift Sep 25, 2018

Member
2018-09-25 20:23:55 clang(pr=14150): script/descriptor.cpp:179:69: warning: implicit conversion changes signedness: 'int' to 'std::vector<unsigned int, std::allocator<unsigned int> >::value_type' (aka 'unsigned int') [-Wsign-conversion]

This comment has been minimized.

Copy link
@sipa

sipa Oct 12, 2018

Author Member

Fixed.

@sipa sipa force-pushed the sipa:201807_minedesc_origin branch 2 times, most recently Oct 12, 2018

@sipa

This comment has been minimized.

Copy link
Member Author

commented Oct 12, 2018

Rebased, and expanded documentation to include key origin information.

@ryanofsky

This comment has been minimized.

Copy link
Contributor

commented Oct 16, 2018

Was there a problem with the last rebase? It seems like some changes that were previously made, like #14150 (comment), have been reverted.

@meshcollider

This comment has been minimized.

Copy link
Member

commented Oct 16, 2018

Concept ACK, will wait for the rebase to be fixed before reviewing then

@sipa sipa force-pushed the sipa:201807_minedesc_origin branch Oct 17, 2018

@sipa

This comment has been minimized.

Copy link
Member Author

commented Oct 17, 2018

@ryanofsky Wow, indeed, thanks for pointing that out. I don't know how that happened.

I went through all comments again, and reapplied the fixed I made earlier where necessary. I've also expanded the documentation change a bit.

@ryanofsky
Copy link
Contributor

left a comment

utACK 3fdc704f06bb5618cb63c248b20447026509e00d. Left some comments below that aren't critical and you should feel free to ignore.

Show resolved Hide resolved src/script/descriptor.cpp
Show resolved Hide resolved src/script/descriptor.cpp Outdated
Show resolved Hide resolved src/script/descriptor.cpp
Show resolved Hide resolved src/script/descriptor.cpp Outdated
doc/descriptors.md Outdated
@@ -34,6 +34,7 @@ Output descriptors currently support:
- `sh(wsh(multi(1,03f28773c2d975288bc7d1d205c3748651b075fbc6610e58cddeeddf8f19405aa8,03499fdf9e895e719cfd64e67f07d38e3226aa7b63678949e6e49b241a60e823e4,02d7924d4f7d43ea965a465ae3095ff41131e5946f3c85f79e44adbcf8e27e080e)))` represents a P2SH-P2WSH *1-of-3* multisig.
- `pk(xpub661MyMwAqRbcFtXgS5sYJABqqG9YLmC4Q1Rdap9gSE8NqtwybGhePY2gZ29ESFjqJoCu1Rupje8YtGqsefD265TMg7usUDFdp6W1EGMcet8)` refers to a single P2PK output, using the public key part from the specified xpub.
- `pkh(xpub68Gmy5EdvgibQVfPdqkBBCHxA5htiqg55crXYuXoQRKfDBFA1WEjWgP6LHhwBZeNK1VTsfTFUHCdrfp1bgwQ9xv5ski8PX9rL2dZXvgGDnw/1'/2)` refers to a single P2PKH output, using child key *1'/2* of the specified xpub.
- `pkh(d34db33f/44'/0'/0':xpub6ERApfZwUNrhLCkDtcHTcxd75RbzS1ed54G1LkBUHQVHQKqhMkhgbmJbZRkrgZw4koxb5JaHWkY4ALHY2grBGRjaDMzQLcgJvLJuZZvRcEL/1/*)` refers to a chain of P2PKH outputs, but additionally specifies that the specified xpub is a child of a master with fingerprint `d34db33f`, and derived using path `44'/0'/0'`.

This comment has been minimized.

Copy link
@ryanofsky

ryanofsky Oct 17, 2018

Contributor

This syntax is slightly weird to me because if you're looking at an expression like:

fingerprint/path1:xpub/path2

The fingerprint:path1 part only describes xpub, not xpub/path2, but there's no logical precedence of : and / operators that would lead me to see:

(fingerprint/path1:xpub)/path2

instead of:

(fingerprint/path1):(xpub/path2)

Maybe this is an unimportant aesthetic objection, but I'd almost rather there be an explicit origin function like:

origin(fingerprint/path1, xpub)/path2

or

origin(xpub, fingerprint/path1)/path2

that at least would group things together correctly.

Show resolved Hide resolved src/script/descriptor.cpp

@sipa sipa force-pushed the sipa:201807_minedesc_origin branch Oct 17, 2018

@sipa

This comment has been minimized.

Copy link
Member Author

commented Oct 17, 2018

@ryanofsky Addressed all your comments except the syntax issue, which we should probably discuss at the PR level rather than buried inside a code comment.

I agree that fpr/path:key/path doesn't perfectly map with intuition about precedence of operation. On the other hand, I'd prefer to keep "key operations" as pure syntax, separated from "script operations" which are functions.

After thinking a bit more about it, how do you feel about fpr/path[key]/path? That way the entire expression maintains the order of path elements, and it just looks like you're "clarifying" the actual key somewhere inside the path itself, rather than as a separate thing prefixed onto it.

An alternative which is somewhat easier to parse I think is [fpr/path]key/path, which can still be read as "([fpr/path]key)/path".

@ryanofsky
Copy link
Contributor

left a comment

utACK 71cbcd9da8b31cb7cbe19001df32c8e1c46a603b. Only changes since last review were suggested changes (replacing +4's, adding a few test cases and a comment).


I think I like both of the [fpr/path]key/path and fpr/path[key]/path syntaxes more than the current fpr/path:key/path syntax and more than my origin() suggestion, and I agree that the [fpr/path]key/path version is probably more understandable.

I might also propose a more generalized syntax for adding attributes to keys like:

key[origin=fpr/path, device=myhardwarewallet]/path

Which might have other uses like:

key[birthday=timestamp]

But this might be too heavyweight. Anyway, you've thought about this a lot more than I have and I think whatever syntax you like, including the current syntax, should be fine.

@ryanofsky

This comment has been minimized.

Copy link
Contributor

commented Oct 17, 2018

Longer term this feature helps with a potential descriptors-based walletless PSBT updater, or for importing hardware wallet xpubs (once the wallet can import descriptors).

Could you explain how these use-cases benefit from having origin information embedded inside descriptor expressions? It seems like it might be simpler if the descriptor language only described how things should be signed, and if key information was stored (or passed) in a separate a key id -> key metadata mapping.

@sipa

This comment has been minimized.

Copy link
Member Author

commented Oct 17, 2018

@ryanofsky A PSBT updater needs to add key origin information to the PSBT file, or future signers may not be able to find the key to sign with.

I really think this information needs to be part of descriptors for them to be generally useful. Just listing the public keys is useless if those using the information don't know how to find the private key corresponding to said public key. In a way, that derivation information should from our point of view be treated as part of the public information about that key.

Another argument is so that you can 'specialize' a ranged descriptor to an individual one (like what may be desirable for #14503), without losing information. With origin support you can go from xpub/a/b/c/* to [fpr:a/b/c/i]pubkey when specializing for position i.

@sipa sipa force-pushed the sipa:201807_minedesc_origin branch to cfc8129 Oct 17, 2018

@sipa

This comment has been minimized.

Copy link
Member Author

commented Oct 17, 2018

Switched to the [fpr/path]key/path notation, updating the parser, serializer, tests, and documentation.

As far as other attributes go, I'm not sure descriptors are the right place (this may be getting philosophical, though):

  • a timestamp is more a descriptor-level property than a key specific thing (you care about when any output generated by the descriptor can be used at the earliest, not differences in timestamps between the individual keys inside one).
  • the name of a hardware device is not portable (if included in descriptors, you wouldn't be able to copy the descriptor to another software/system that knows the hw device by a different name)

My preference is keeping those two as metadata on the descriptor (together with things like gap limit/range, whether the outputs are treated as change, ...). The same could be done with key origin information, but it feels more fundamental to not lose that information.

If there are generally useful properties of keys or path elements later we can still adopt such a syntax, though; it looks pretty readable.

@ryanofsky
Copy link
Contributor

left a comment

utACK cfc8129. Only change since last review is syntax update.


Thanks for the explanations. I can see why you wouldn't want to make birthdays a property of keys, and it's also very clear that a PBST API which takes descriptors will be a lot clumsier if key origin information has to be passed in a separate table, instead of just embedded in the descriptors.

Show resolved Hide resolved src/script/descriptor.cpp Outdated
Show resolved Hide resolved doc/descriptors.md Outdated
Show resolved Hide resolved doc/descriptors.md Outdated
@instagibbs

This comment has been minimized.

Copy link
Member

commented Oct 18, 2018

concept ACK, if rebase is fixed I'll review further

@ryanofsky

This comment has been minimized.

Copy link
Contributor

commented Oct 18, 2018

if rebase is fixed I'll review further

I don't think there's a problem with current version of this PR (cfc8129). The earlier comment #14150 (comment) was referring to 605367580b582a6e9510a06670cfd4898168cea5.

@instagibbs

This comment has been minimized.

Copy link
Member

commented Oct 18, 2018

Oops, I reviewed 2 days ago and didn't refresh this page to see updates!

@instagibbs
Copy link
Member

left a comment

utACK

Show resolved Hide resolved doc/descriptors.md Outdated
Show resolved Hide resolved doc/descriptors.md
@@ -82,7 +82,7 @@ def run_test(self):
assert_equal(self.nodes[0].scantxoutset("start", [ "combo(tprv8ZgxMBicQKsPd7Uf69XL1XwhmjHopUGep8GuEiJDZmbQz6o58LninorQAfcKZWARbtRtfnLcJ5MQ2AtHcQJCCRUcMRvmDUjyEmNUWwx8UbK/1/1/1)"])['total_amount'], Decimal("8.192"))
assert_equal(self.nodes[0].scantxoutset("start", [ "combo(tprv8ZgxMBicQKsPd7Uf69XL1XwhmjHopUGep8GuEiJDZmbQz6o58LninorQAfcKZWARbtRtfnLcJ5MQ2AtHcQJCCRUcMRvmDUjyEmNUWwx8UbK/1/1/1500)"])['total_amount'], Decimal("16.384"))
assert_equal(self.nodes[0].scantxoutset("start", [ "combo(tpubD6NzVbkrYhZ4WaWSyoBvQwbpLkojyoTZPRsgXELWz3Popb3qkjcJyJUGLnL4qHHoQvao8ESaAstxYSnhyswJ76uZPStJRJCTKvosUCJZL5B/1/1/0)"])['total_amount'], Decimal("4.096"))
assert_equal(self.nodes[0].scantxoutset("start", [ "combo(tpubD6NzVbkrYhZ4WaWSyoBvQwbpLkojyoTZPRsgXELWz3Popb3qkjcJyJUGLnL4qHHoQvao8ESaAstxYSnhyswJ76uZPStJRJCTKvosUCJZL5B/1/1/1)"])['total_amount'], Decimal("8.192"))
assert_equal(self.nodes[0].scantxoutset("start", [ "combo([abcdef88/1/2'/3/4h]tpubD6NzVbkrYhZ4WaWSyoBvQwbpLkojyoTZPRsgXELWz3Popb3qkjcJyJUGLnL4qHHoQvao8ESaAstxYSnhyswJ76uZPStJRJCTKvosUCJZL5B/1/1/1)"])['total_amount'], Decimal("8.192"))

This comment has been minimized.

Copy link
@instagibbs

instagibbs Oct 18, 2018

Member

Could there be a comment to point out what it's doing? As a reader of the test this is basically lost in the pile.

This comment has been minimized.

Copy link
@sipa

sipa Oct 18, 2018

Author Member

Not sure what you're asking. I've added 2 lines of comment on this block of tests.

Or is it specifically about this including of the key origin in the test? It has no effect, as for now key origins are not observable anyway. So this is just testing that you can in fact specify one, and nothing breaks.

This comment has been minimized.

Copy link
@instagibbs

instagibbs Oct 18, 2018

Member

I meant to a reader of the entire test itself, not the diff.

This comment has been minimized.

Copy link
@sipa

sipa Oct 18, 2018

Author Member

Ok, is what I added sufficient?

@sipa

This comment has been minimized.

Copy link
Member Author

commented Oct 18, 2018

Addressed further comments in separate commits, will squash when ready.

@instagibbs

This comment has been minimized.

Copy link
Member

commented Oct 18, 2018

re-ACK

@meshcollider
Copy link
Member

left a comment

utACK f4a8ccb

Show resolved Hide resolved src/script/descriptor.cpp Outdated
Show resolved Hide resolved doc/descriptors.md Outdated
@ryanofsky

This comment has been minimized.

Copy link
Contributor

commented Oct 19, 2018

utACK 91cff98264ba5c6a86ca5d18b2591f759cffe793, just documentation fixes and two variable cleanups

@DrahtBot

This comment has been minimized.

Copy link
Contributor

commented Oct 20, 2018

Reviewers, this pull request conflicts with the following ones:
  • #14505 (Make all single parameter constructors explicit (C++11) by practicalswift)

If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

@sipa sipa force-pushed the sipa:201807_minedesc_origin branch Oct 21, 2018

@sipa

This comment has been minimized.

Copy link
Member Author

commented Oct 21, 2018

Squashed fixups, no tree changes.

@sipa sipa force-pushed the sipa:201807_minedesc_origin branch to 8afb166 Oct 21, 2018

@sipa

This comment has been minimized.

Copy link
Member Author

commented Oct 21, 2018

Rebased afa9224b14 -> 8afb166

@instagibbs

This comment has been minimized.

Copy link
Member

commented Oct 22, 2018

utACK 8afb166

@ryanofsky
Copy link
Contributor

left a comment

utACK 8afb166. Only change since last review is rebase/squash, resolving a merge conflict with #14161 in the documentation

yashbhutwala pushed a commit to yashbhutwala/bitcoin that referenced this pull request Oct 22, 2018

Merge bitcoin#14150: Add key origin support to descriptors
8afb166 Update documentation to incude origin information (Pieter Wuille)
ff37459 Add tests for key origin support (Pieter Wuille)
2c6281f Add key origin support to descriptors (Pieter Wuille)

Pull request description:

  This adds support for [key origin](https://gist.github.com/sipa/e3d23d498c430bb601c5bca83523fa82#key-origin-identification) information to the descriptor parser, and exposes the resulting key path information through `FlatSigningProvider`.

  There is no observable functionality from this right now, except having the `scantxoutset` RPC accept descriptors that include key origin information.

  Longer term this feature helps with a potential descriptors-based walletless PSBT updater, or for importing hardware wallet xpubs (once the wallet can import descriptors).

Tree-SHA512: 399828127b2e90a2f32d81ecc30a8a9261d08f4182d5d1744f05e46b25fde1bd383c54835b0820ca668e7d17353fa92c0fb2987e211ce269e0824c9395d210c2

@sipa sipa merged commit 8afb166 into bitcoin:master Oct 22, 2018

2 checks passed

continuous-integration/appveyor/pr AppVeyor build succeeded
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details

@fanquake fanquake removed this from Blockers in High-priority for review Oct 22, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.