Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add expansion cache functions to descriptors (unused for now) #14646

Merged
merged 5 commits into from Dec 12, 2018

Conversation

@sipa
Copy link
Member

@sipa sipa commented Nov 3, 2018

This patch modifies the internal Descriptor class to optionally construct and use an "expansion cache". Such a cache is a byte array that encodes all information necessary to expand a Descriptor a second time without access to private keys, and without the need to perform expensive BIP32 derivations. For all currently defined descriptors, the cache simply contains a concatenation of all public keys used.

This is motivated by the goal of importing a descriptor into the wallet and using it as a replacement for the keypool, where it would be impossible to expand descriptors if they use hardened derivation.

@sipa sipa force-pushed the 201811_descriptcache branch from 022c93d to 3a853cb Nov 3, 2018
@meshcollider
Copy link
Member

@meshcollider meshcollider commented Nov 3, 2018

Concept ACK

@sipa sipa force-pushed the 201811_descriptcache branch 4 times, most recently from f9d9483 to 0bfff69 Nov 3, 2018
Copy link
Member

@Sjors Sjors left a comment

Concept ACK.

IIUC each time you ask the descriptor to expand a specific index, you can pass it an individual std::vector<unsigned char> cache entry. I assume those eventually get serialised into the wallet payload? Where do you envision keeping track of all these cache entries will happen?

src/script/descriptor.cpp Show resolved Hide resolved
src/script/descriptor.cpp Show resolved Hide resolved
src/test/descriptor_tests.cpp Show resolved Hide resolved
src/script/descriptor.cpp Show resolved Hide resolved
@sipa
Copy link
Member Author

@sipa sipa commented Nov 3, 2018

@Sjors My idea is that the wallet will contain a number of records, each of which has a descriptor, a bool to indicate whether it's for change or not, a birthday, perhaps information about what HW device to prompt for, a gap limit, ...; those are all static configuration that generally doesn't change unless you import something. In addition there will be a wallet entry per expanded element of a record with the cached information, which also helps in identifying what the next unused address is etc.

@DrahtBot
Copy link
Contributor

@DrahtBot DrahtBot commented Nov 3, 2018

The following sections might be updated with supplementary metadata relevant to reviewers and maintainers.

Conflicts

Reviewers, this pull request conflicts with the following ones:

  • #14826 (Avoid expanding descriptor scriptPubKeys by promag)
  • #14505 (Make single parameter constructors explicit (C++11). Add explicit constructor linter. by practicalswift)

If you consider this pull request important, please also help to review the conflicting pull requests. Ideally, start with the one that should be merged first.

@sipa sipa force-pushed the 201811_descriptcache branch from 0bfff69 to 582685e Nov 3, 2018
src/script/descriptor.cpp Outdated Show resolved Hide resolved
src/script/descriptor.cpp Outdated Show resolved Hide resolved
src/script/descriptor.cpp Outdated Show resolved Hide resolved
src/script/descriptor.cpp Outdated Show resolved Hide resolved
src/script/descriptor.cpp Outdated Show resolved Hide resolved
src/script/descriptor.cpp Outdated Show resolved Hide resolved
@sipa sipa force-pushed the 201811_descriptcache branch from 582685e to 94677cc Nov 5, 2018
Sjors
Sjors approved these changes Nov 5, 2018
Copy link
Member

@Sjors Sjors left a comment

utACK 94677cc

src/script/sign.h Outdated Show resolved Hide resolved
src/script/descriptor.cpp Outdated Show resolved Hide resolved
src/script/descriptor.cpp Outdated Show resolved Hide resolved
Copy link
Contributor

@ryanofsky ryanofsky left a comment

So for only reviewed the first (largest) commit in detail. The later commits also seem good at first glance. Will continue reviewing.

src/script/descriptor.cpp Show resolved Hide resolved
src/script/descriptor.cpp Outdated Show resolved Hide resolved
src/script/descriptor.cpp Outdated Show resolved Hide resolved
src/script/descriptor.cpp Show resolved Hide resolved
src/script/descriptor.cpp Outdated Show resolved Hide resolved
src/script/descriptor.cpp Outdated Show resolved Hide resolved
src/script/descriptor.cpp Outdated Show resolved Hide resolved
@sipa sipa force-pushed the 201811_descriptcache branch from 94677cc to ab67eb2 Nov 5, 2018
@sipa sipa force-pushed the 201811_descriptcache branch from 729b818 to 2687950 Nov 28, 2018
@sipa
Copy link
Member Author

@sipa sipa commented Nov 28, 2018

Rebased.

Copy link
Contributor

@ryanofsky ryanofsky left a comment

utACK 2687950. Somewhat messy rebase after #14477, but no changes since last review.

Copy link
Member

@meshcollider meshcollider left a comment

utACK 2687950

Just two minor comment nits inline, sorry to pick :P

{
CScript m_script;
//! Public key arguments for this descriptor (size 1 for PK, PKH, WPKH; any size of Multisig).
Copy link
Member

@meshcollider meshcollider Dec 11, 2018

nit: of -> for

@@ -48,8 +48,18 @@ struct Descriptor {
* provider: the provider to query for private keys in case of hardened derivation.
* output_script: the expanded scriptPubKeys will be put here.
* out: scripts and public keys necessary for solving the expanded scriptPubKeys will be put here (may be equal to provider).
* cache: vector which will be overwritten with cache data necessary to-evaluate the descriptor at this point without access to private keys.
Copy link
Member

@meshcollider meshcollider Dec 11, 2018

nit: remove -

Sjors
Sjors approved these changes Dec 11, 2018
Copy link
Member

@Sjors Sjors left a comment

utACK 2687950

I found a git (2.20+) incantation that shows the difference between now and the last time I reviewed, but that difference is so large it's not super useful here:

 git range-diff `git merge-base --all HEAD 94677cc`...94677cc HEAD~5...HEAD

Where 94677cc is the last commit I reviewed and 5 is the number of commits in this PR.

Anyway, reviewed again manually :-)

entries.emplace_back();
if (!p->GetPubKey(pos, arg, entries.back().first, entries.back().second)) return false;
if (!p->GetPubKey(pos, arg, cache_read ? nullptr : &entries.back().first, entries.back().second)) return false;
Copy link
Member

@Sjors Sjors Dec 11, 2018

Nit: explain that if we already have a cache, we don't want GetPubKey to return public keys, since we're getting them from the cache ourselves. This is counter intuitive given the name of the function, but the point is to only make GetPubKey do work if our cache is empty.

Alternatively, maybe the stuff below under if (cache_read) belongs in GetPubKey?

Maybe add an assert to GetPubKey if KeyOriginInfo& info is non empty and doesn't match m_extkey.pubkey.GetID()?

//! Public key arguments for this descriptor (size 1 for PK, PKH, WPKH; any size of Multisig).
const std::vector<std::unique_ptr<PubkeyProvider>> m_pubkey_args;
//! The sub-descriptor argument (nullptr for everything but SH and WSH).
const std::unique_ptr<DescriptorImpl> m_script_arg;
Copy link
Member

@Sjors Sjors Dec 11, 2018

Can we rename this to m_sub_descriptor_arg? (I found that a replace-all reduced my headache)

@meshcollider
Copy link
Member

@meshcollider meshcollider commented Dec 12, 2018

Because this is obviously intended to be followed-up by more work, I think we can leave the last couple nits to be addressed later

meshcollider added a commit to meshcollider/bitcoin that referenced this issue Dec 12, 2018
…used for now)

2687950 Add comments to descriptor tests (Pieter Wuille)
82df4c6 Add descriptor expansion cache (Pieter Wuille)
1eda33a [refactor] Combine the ToString and ToPrivateString implementations (Pieter Wuille)
24d3a7b [refactor] Use DescriptorImpl internally, permitting access to new methods (Pieter Wuille)
6be0fb4 [refactor] Add a base DescriptorImpl with most common logic (Pieter Wuille)

Pull request description:

  This patch modifies the internal `Descriptor` class to optionally construct and use an "expansion cache". Such a cache is a byte array that encodes all information necessary to expand a `Descriptor` a second time without access to private keys, and without the need to perform expensive BIP32 derivations. For all currently defined descriptors, the cache simply contains a concatenation of all public keys used.

  This is motivated by the goal of importing a descriptor into the wallet and using it as a replacement for the keypool, where it would be impossible to expand descriptors if they use hardened derivation.

Tree-SHA512: f531a0a82ec1eecc30b78ba8a31724d1249826b028cc3543ad32372e1aedd537f137ab03dbffc222c5df444d5865ecd5cec754c1ae1d4989b6e9baeaffade32a
@meshcollider meshcollider merged commit 2687950 into bitcoin:master Dec 12, 2018
2 checks passed
@meshcollider meshcollider removed this from Blockers in High-priority for review Dec 12, 2018
laanwj added a commit that referenced this issue Aug 14, 2019
2e68ffa [doc] descriptor: explain GetPubKey() usage with cached public key (Sjors Provoost)
2290269 scripted-diff: rename DescriptorImpl m_script_arg to m_subdescriptor_arg (Sjors Provoost)

Pull request description:

  I found the name `m_script_arg` to be confusing while reviewing #14646 (comment). @sipa let me know if `m_subdescriptor_arg` is completely wrong.

  I also added an explanation of why we call `GetPubKey` when we don't ask it for a public key.

ACKs for top commit:
  laanwj:
    ACK 2e68ffa

Tree-SHA512: 06698e9a91cdda93c043a82732793f0ad3cd91daa2513565953e9fa048d5573322fb534e9d0ea9ab736e6366be5921e2b8699c4f4b3693edab48039aaae06f78
sidhujag added a commit to syscoin/syscoin that referenced this issue Aug 17, 2019
2e68ffa [doc] descriptor: explain GetPubKey() usage with cached public key (Sjors Provoost)
2290269 scripted-diff: rename DescriptorImpl m_script_arg to m_subdescriptor_arg (Sjors Provoost)

Pull request description:

  I found the name `m_script_arg` to be confusing while reviewing bitcoin#14646 (comment). @sipa let me know if `m_subdescriptor_arg` is completely wrong.

  I also added an explanation of why we call `GetPubKey` when we don't ask it for a public key.

ACKs for top commit:
  laanwj:
    ACK 2e68ffa

Tree-SHA512: 06698e9a91cdda93c043a82732793f0ad3cd91daa2513565953e9fa048d5573322fb534e9d0ea9ab736e6366be5921e2b8699c4f4b3693edab48039aaae06f78
deadalnix added a commit to Bitcoin-ABC/bitcoin-abc that referenced this issue May 21, 2020
Summary: This is a partial backport of Core [[bitcoin/bitcoin#14646 | PR14646]] : bitcoin/bitcoin@6be0fb4

Test Plan:
  ninja all check-all

Reviewers: #bitcoin_abc, Fabien

Reviewed By: #bitcoin_abc, Fabien

Subscribers: Fabien

Differential Revision: https://reviews.bitcoinabc.org/D6194
deadalnix added a commit to Bitcoin-ABC/bitcoin-abc that referenced this issue May 21, 2020
…thods

Summary:
This is a partial backport of Core [[bitcoin/bitcoin#14646 | PR14646]] : bitcoin/bitcoin@24d3a7b

Depends on D6194

Test Plan:
  ninja all check-all

Reviewers: #bitcoin_abc, majcosta

Reviewed By: #bitcoin_abc, majcosta

Subscribers: majcosta

Differential Revision: https://reviews.bitcoinabc.org/D6195
deadalnix added a commit to Bitcoin-ABC/bitcoin-abc that referenced this issue May 21, 2020
Summary:
This is a partial backport of Core [[bitcoin/bitcoin#14646 | PR14646]] : bitcoin/bitcoin@1eda33a

Depends on D6195

Test Plan:
  ninja all check-all

Reviewers: #bitcoin_abc, majcosta

Reviewed By: #bitcoin_abc, majcosta

Subscribers: majcosta

Differential Revision: https://reviews.bitcoinabc.org/D6196
deadalnix added a commit to Bitcoin-ABC/bitcoin-abc that referenced this issue May 21, 2020
Summary:
This is a partial backport of Core [[bitcoin/bitcoin#14646 | PR14646]] : bitcoin/bitcoin@82df4c6

Depends on D6196

Test Plan:
  ninja all check-all

Reviewers: #bitcoin_abc, majcosta

Reviewed By: #bitcoin_abc, majcosta

Differential Revision: https://reviews.bitcoinabc.org/D6197
deadalnix added a commit to Bitcoin-ABC/bitcoin-abc that referenced this issue May 21, 2020
Summary:
This is a partial backport of Core [[bitcoin/bitcoin#14646 | PR14646]] : bitcoin/bitcoin@2687950

Depends on D6197

Test Plan:
  ninja all check-all

Reviewers: #bitcoin_abc, majcosta

Reviewed By: #bitcoin_abc, majcosta

Subscribers: majcosta

Differential Revision: https://reviews.bitcoinabc.org/D6198
ShengguangXiao pushed a commit to DeFiCh/ain that referenced this issue Aug 28, 2020
369e5f5 [doc] descriptor: explain GetPubKey() usage with cached public key (Sjors Provoost)
44ca2f4 scripted-diff: rename DescriptorImpl m_script_arg to m_subdescriptor_arg (Sjors Provoost)

Pull request description:

  I found the name `m_script_arg` to be confusing while reviewing bitcoin/bitcoin#14646 (comment). @sipa let me know if `m_subdescriptor_arg` is completely wrong.

  I also added an explanation of why we call `GetPubKey` when we don't ask it for a public key.

ACKs for top commit:
  laanwj:
    ACK 369e5f5

Tree-SHA512: 06698e9a91cdda93c043a82732793f0ad3cd91daa2513565953e9fa048d5573322fb534e9d0ea9ab736e6366be5921e2b8699c4f4b3693edab48039aaae06f78
jasonbcox pushed a commit to Bitcoin-ABC/bitcoin-abc that referenced this issue Oct 7, 2020
Summary:
2e68ffaf205866e4cea71f64e79bbfb89e17280a [doc] descriptor: explain GetPubKey() usage with cached public key (Sjors Provoost)
2290269759ad10cc2e35958c7b0a63f3a7608621 scripted-diff: rename DescriptorImpl m_script_arg to m_subdescriptor_arg (Sjors Provoost)

Pull request description:

  I found the name `m_script_arg` to be confusing while reviewing bitcoin/bitcoin#14646 (comment). @sipa let me know if `m_subdescriptor_arg` is completely wrong.

  I also added an explanation of why we call `GetPubKey` when we don't ask it for a public key.

ACKs for top commit:
  laanwj:
    ACK 2e68ffaf205866e4cea71f64e79bbfb89e17280a

---

Backport of Core [[bitcoin/bitcoin#14934 | PR14934]]

Test Plan:
  ninja all check check-functional

Reviewers: #bitcoin_abc, deadalnix

Reviewed By: #bitcoin_abc, deadalnix

Differential Revision: https://reviews.bitcoinabc.org/D7806
linuxsh2 added a commit to linuxsh2/dash that referenced this issue Jul 29, 2021
… output of public descriptors

c77f092 Fix descriptor_tests not checking ToString output of public descriptors (Russell Yanofsky)

Pull request description:

  This fixes a minor test bug introduced in bitcoin#13697 that I noticed while reviewing bitcoin#14646

Tree-SHA512: efed91200cdff5f86ba5de3461ac00759d285e2905f6cb24cea15d3e23e0581ce5fc14b24a40db093f7ebd662ee1ee2cf67f8798bac1903a78298eda08909cfb
linuxsh2 added a commit to linuxsh2/dash that referenced this issue Jul 30, 2021
… output of public descriptors

c77f092 Fix descriptor_tests not checking ToString output of public descriptors (Russell Yanofsky)

Pull request description:

  This fixes a minor test bug introduced in bitcoin#13697 that I noticed while reviewing bitcoin#14646

Tree-SHA512: efed91200cdff5f86ba5de3461ac00759d285e2905f6cb24cea15d3e23e0581ce5fc14b24a40db093f7ebd662ee1ee2cf67f8798bac1903a78298eda08909cfb
linuxsh2 added a commit to linuxsh2/dash that referenced this issue Aug 3, 2021
… output of public descriptors

c77f092 Fix descriptor_tests not checking ToString output of public descriptors (Russell Yanofsky)

Pull request description:

  This fixes a minor test bug introduced in bitcoin#13697 that I noticed while reviewing bitcoin#14646

Tree-SHA512: efed91200cdff5f86ba5de3461ac00759d285e2905f6cb24cea15d3e23e0581ce5fc14b24a40db093f7ebd662ee1ee2cf67f8798bac1903a78298eda08909cfb
@bitcoin bitcoin locked as resolved and limited conversation to collaborators Sep 8, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

8 participants