New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Tools] bitcoin-wallet - a tool for creating and managing wallets offline #13926

Open
wants to merge 2 commits into
base: master
from

Conversation

@jnewbery
Member

jnewbery commented Aug 9, 2018

Adds an offline tool bitcoin-wallet-tool for wallet creation and maintenance.

Currently this tool can create a new wallet file, display information on an existing wallet, and run the salvage and zapwallettxes maintenance tasks on an existing wallet. It can later be extended to support other common wallet maintenance tasks.

Doing wallet maintenance tasks in an offline tool makes much more sense (and is potentially safer) than having to spin up a full node.

@jnewbery

This comment has been minimized.

Show comment
Hide comment
@jnewbery

jnewbery Aug 9, 2018

Member

This is currently a work in progress. It is a rebase of @jonasschnelli's #8745. Please see that PR for history and context. I've volunteered to maintain this through to merge. Thanks Jonas for your work in writing and maintaining this branch so far!

It is currently rebased on top of #12490, which removes a circular dependency and allows it to build. Only the Add wallet inspection and modification tool "bitcoin-wallet-tool" commit is relevant here.

#12490 can't be merged until after the V0.17 branch is forked, so detailed review at this point would be immature. Concept and high-level comments most definitely welcomed.

Member

jnewbery commented Aug 9, 2018

This is currently a work in progress. It is a rebase of @jonasschnelli's #8745. Please see that PR for history and context. I've volunteered to maintain this through to merge. Thanks Jonas for your work in writing and maintaining this branch so far!

It is currently rebased on top of #12490, which removes a circular dependency and allows it to build. Only the Add wallet inspection and modification tool "bitcoin-wallet-tool" commit is relevant here.

#12490 can't be merged until after the V0.17 branch is forked, so detailed review at this point would be immature. Concept and high-level comments most definitely welcomed.

@domob1812

This comment has been minimized.

Show comment
Hide comment
@domob1812

domob1812 Aug 9, 2018

Contributor

Concept ACK, looks useful. Will that tool also support things like creating addresses or dumping privkeys in the future, or "just" actual maintenance of the wallet file? That's something that might be useful for offline systems (although you can of course spin up the daemon with an empty data dir).

Contributor

domob1812 commented Aug 9, 2018

Concept ACK, looks useful. Will that tool also support things like creating addresses or dumping privkeys in the future, or "just" actual maintenance of the wallet file? That's something that might be useful for offline systems (although you can of course spin up the daemon with an empty data dir).

@DrahtBot

This comment has been minimized.

Show comment
Hide comment
@DrahtBot

DrahtBot Aug 9, 2018

Contributor
No more conflicts as of last run.
Contributor

DrahtBot commented Aug 9, 2018

No more conflicts as of last run.
@jnewbery

This comment has been minimized.

Show comment
Hide comment
@jnewbery

jnewbery Aug 10, 2018

Member

Will that tool also support things like creating addresses or dumping privkeys in the future

Potentially, yes. That would be a discussion for a future PR though :)

Member

jnewbery commented Aug 10, 2018

Will that tool also support things like creating addresses or dumping privkeys in the future

Potentially, yes. That would be a discussion for a future PR though :)

@domob1812

This comment has been minimized.

Show comment
Hide comment
@domob1812

domob1812 Aug 10, 2018

Contributor

@jnewbery: Yes, of course - let's go in steps. For me, those functions would have been useful from time to time, though, so I'd love to see that added in some follow up. (And am happy to work on it myself if that helps.)

Contributor

domob1812 commented Aug 10, 2018

@jnewbery: Yes, of course - let's go in steps. For me, those functions would have been useful from time to time, though, so I'd love to see that added in some follow up. (And am happy to work on it myself if that helps.)

@MeshCollider

This comment has been minimized.

Show comment
Hide comment
@MeshCollider

MeshCollider Aug 14, 2018

Member

Concept ACK, does seem clean and useful to have these as a separate tool.

Member

MeshCollider commented Aug 14, 2018

Concept ACK, does seem clean and useful to have these as a separate tool.

@laanwj

This comment has been minimized.

Show comment
Hide comment
@laanwj

laanwj Aug 22, 2018

Member

Thanks for picking this up, I agree an external wallet manipulation/recovery tool is useful

Member

laanwj commented Aug 22, 2018

Thanks for picking this up, I agree an external wallet manipulation/recovery tool is useful

@jnewbery

This comment has been minimized.

Show comment
Hide comment
@jnewbery

jnewbery Aug 22, 2018

Member

Thanks all concept ACKers! If you want to help with this PR, please review #12490, since that's a pre-req to get this to build. That PR just removes deprecated code, so it should be an easy review.

Member

jnewbery commented Aug 22, 2018

Thanks all concept ACKers! If you want to help with this PR, please review #12490, since that's a pre-req to get this to build. That PR just removes deprecated code, so it should be an easy review.

Show outdated Hide outdated src/wallet/wallettool.h Outdated
Show outdated Hide outdated src/wallet/wallettool.h Outdated
@jnewbery

This comment has been minimized.

Show comment
Hide comment
@jnewbery

jnewbery Sep 6, 2018

Member

Thanks for the review @practicalswift . I've fixed your review comments and rebased on the latest #12490.

Since this PR depends on that one, do you mind reviewing that PR?

Member

jnewbery commented Sep 6, 2018

Thanks for the review @practicalswift . I've fixed your review comments and rebased on the latest #12490.

Since this PR depends on that one, do you mind reviewing that PR?

@jnewbery jnewbery changed the title from [WIP] [Tools] bitcoin-wallet-tool to [Tools] bitcoin-wallet-tool Sep 7, 2018

@jnewbery

This comment has been minimized.

Show comment
Hide comment
@jnewbery

jnewbery Sep 7, 2018

Member

#12490 has been merged. I've rebased on master and removed the [WIP] tag.

This is ready for review.

Member

jnewbery commented Sep 7, 2018

#12490 has been merged. I've rebased on master and removed the [WIP] tag.

This is ready for review.

@MarcoFalke

haven't looked at the code yet. Just two style nits.

Show outdated Hide outdated src/bitcoin-wallet-tool-res.rc Outdated
Show outdated Hide outdated src/bitcoin-wallet-tool.cpp Outdated
@jnewbery

This comment has been minimized.

Show comment
Hide comment
@jnewbery

jnewbery Sep 7, 2018

Member

Rebased on master and fixed @MarcoFalke's comments.

Member

jnewbery commented Sep 7, 2018

Rebased on master and fixed @MarcoFalke's comments.

@Sjors

This comment has been minimized.

Show comment
Hide comment
@Sjors

Sjors Sep 8, 2018

Member

How does this compare to #10102 (@ryanofsky) which also creates an independent wallet executable that could, with some tweaks, be used offline? Obviously this change is much simpler.

Member

Sjors commented Sep 8, 2018

How does this compare to #10102 (@ryanofsky) which also creates an independent wallet executable that could, with some tweaks, be used offline? Obviously this change is much simpler.

@jnewbery

This comment has been minimized.

Show comment
Hide comment
@jnewbery

jnewbery Sep 10, 2018

Member

How does this compare to #10102

Very different. 10102 is much more ambitious and is a change to the overall architecture of bitcoind. It has dependency on #10973, so I don't expect it to be merged any time soon. It also doesn't currently support running the wallet executable separately as far as I'm aware.

This is a small, self-contained tool for created, inspecting and updating wallet files.

Member

jnewbery commented Sep 10, 2018

How does this compare to #10102

Very different. 10102 is much more ambitious and is a change to the overall architecture of bitcoind. It has dependency on #10973, so I don't expect it to be merged any time soon. It also doesn't currently support running the wallet executable separately as far as I'm aware.

This is a small, self-contained tool for created, inspecting and updating wallet files.

@promag

Concept ACK. Are you planning adding tests? In this PR or other?

Show outdated Hide outdated src/bitcoin-wallet-tool.cpp Outdated
Show resolved Hide resolved src/wallet/wallettool.cpp Outdated
@jnewbery

This comment has been minimized.

Show comment
Hide comment
@jnewbery

jnewbery Sep 10, 2018

Member

Are you planning adding tests? In this PR or other?

Yes, I plan to add tests to this PR.

Member

jnewbery commented Sep 10, 2018

Are you planning adding tests? In this PR or other?

Yes, I plan to add tests to this PR.

@jnewbery

This comment has been minimized.

Show comment
Hide comment
@jnewbery

jnewbery Sep 11, 2018

Member

Thanks for the review @promag and @practicalswift . All comments addressed in the fixup commit. I'll squash if you're happy with the changes.

Member

jnewbery commented Sep 11, 2018

Thanks for the review @promag and @practicalswift . All comments addressed in the fixup commit. I'll squash if you're happy with the changes.

@jonasschnelli

This comment has been minimized.

Show comment
Hide comment
@jonasschnelli

jonasschnelli Sep 11, 2018

Member

Concept ACK
Thanks for picking this up...

Member

jonasschnelli commented Sep 11, 2018

Concept ACK
Thanks for picking this up...

@promag

This comment has been minimized.

Show comment
Hide comment
@promag

promag Sep 11, 2018

Member

Segfault with invalid parameter:

bitcoin-wallet-tool -foo
Error parsing command line arguments: Invalid parameter -foo
[1]    16474 segmentation fault  ./src/bitcoin-wallet-tool -foo
Member

promag commented Sep 11, 2018

Segfault with invalid parameter:

bitcoin-wallet-tool -foo
Error parsing command line arguments: Invalid parameter -foo
[1]    16474 segmentation fault  ./src/bitcoin-wallet-tool -foo
@promag

This comment has been minimized.

Show comment
Hide comment
@promag

promag Sep 21, 2018

Member

nit, add src/bitcoin-wallet to .gitignore.

Member

promag commented Sep 21, 2018

nit, add src/bitcoin-wallet to .gitignore.

@Sjors

Concept ACK.

This tool could also be useful as a migration tool if we ever decide to move away from DBD and deprecate that dependency from bitcoind/qt. Already it could have its own --with-incompatible-bdb configure argument (can wait for a future PR).

@theuni should probably take a look at the build system aspects of this.

Needs testing in Windows

How do I use this with -testnet? Do we want to ignore bitcoin.conf. If so, that needs mentioning in the help text. Longer term it seems better to support that, e.g. once creating new addresses is possible, addresstype should be honored.

I assume fresh wallets trigger a rescan because the tool doesn't know the block height? If so, maybe warn about that, and add a heighthint argument later.

@@ -509,6 +521,32 @@ bitcoin_tx_LDADD = \
bitcoin_tx_LDADD += $(BOOST_LIBS) $(CRYPTO_LIBS)
#
# bitcoin-wallet-tool binary #

This comment has been minimized.

@Sjors

Sjors Sep 25, 2018

Member

nit: bitcoin-wallet

@Sjors

Sjors Sep 25, 2018

Member

nit: bitcoin-wallet

Show outdated Hide outdated src/bitcoin-wallet.cpp Outdated
@MarcoFalke

This comment has been minimized.

Show comment
Hide comment
@MarcoFalke

MarcoFalke Sep 25, 2018

Member

Mind to cherry-pick the tests from #14283, so that they can be reviewed in one go with the current code?

Member

MarcoFalke commented Sep 25, 2018

Mind to cherry-pick the tests from #14283, so that they can be reviewed in one go with the current code?

@promag

This comment has been minimized.

Show comment
Hide comment
@promag

promag Sep 25, 2018

Member

@MarcoFalke if the test strategy looks good then I should clean it before.

Member

promag commented Sep 25, 2018

@MarcoFalke if the test strategy looks good then I should clean it before.

@jonasschnelli

This comment has been minimized.

Show comment
Hide comment
@jonasschnelli

jonasschnelli Sep 27, 2018

Member

Tested a bit...
I think the help should mention that the created wallet is relative to the Bitcoin data dir (from reading the help, I would have expected that the wallet will be created in the current directory).

I personally would prefer that the tool does not interact with the datadir. (specify path or expect current dir).

Also, how can you create a new wallet in the regtest or testnet subdirectory?

Member

jonasschnelli commented Sep 27, 2018

Tested a bit...
I think the help should mention that the created wallet is relative to the Bitcoin data dir (from reading the help, I would have expected that the wallet will be created in the current directory).

I personally would prefer that the tool does not interact with the datadir. (specify path or expect current dir).

Also, how can you create a new wallet in the regtest or testnet subdirectory?

@ryanofsky

This comment has been minimized.

Show comment
Hide comment
@ryanofsky

ryanofsky Sep 27, 2018

Contributor

I personally would prefer that the tool does not interact with the datadir.

I think that would be a sensible idea for a low-level wallet debugging tool, but a bad idea for a tool that can replace bitcoind and bitcoin-qt as a way of accessing wallets. I think the least confusing way for bitcoin-wallet to work is for it to use the same configuration bitcoind and bitcoin-qt do, and to open and locate wallets in exactly the same way (with exact same -wallet -walletdir -datadir -regtest -conf etc parameters and defaults).

Lets think about an actual use case. If a user who runs bitcoind or bitcoin-qt normally encounters a problem that requires them to run "salvage" or "zaptxs", it would be nice if they could just run bitcoin-wallet salvage or bitcoin-wallet zaptxs from the command line directly, without having to cd into weird directories or specify long data paths or learn an new set of command line arguments different from the existing ones.

The bitcoin-wallet tool is also going to be extended in the future. I think it'd be nice if it had dump and load subcommands that could be a safe for backing up and transferring wallets into flat files without the user having to know anything about the internal structure of their data dir. I think it'd be nice if there were a shell subcommand that would drop into an rpcconsole like shell and allow running wallets rpc commands in an offline mode.

Contributor

ryanofsky commented Sep 27, 2018

I personally would prefer that the tool does not interact with the datadir.

I think that would be a sensible idea for a low-level wallet debugging tool, but a bad idea for a tool that can replace bitcoind and bitcoin-qt as a way of accessing wallets. I think the least confusing way for bitcoin-wallet to work is for it to use the same configuration bitcoind and bitcoin-qt do, and to open and locate wallets in exactly the same way (with exact same -wallet -walletdir -datadir -regtest -conf etc parameters and defaults).

Lets think about an actual use case. If a user who runs bitcoind or bitcoin-qt normally encounters a problem that requires them to run "salvage" or "zaptxs", it would be nice if they could just run bitcoin-wallet salvage or bitcoin-wallet zaptxs from the command line directly, without having to cd into weird directories or specify long data paths or learn an new set of command line arguments different from the existing ones.

The bitcoin-wallet tool is also going to be extended in the future. I think it'd be nice if it had dump and load subcommands that could be a safe for backing up and transferring wallets into flat files without the user having to know anything about the internal structure of their data dir. I think it'd be nice if there were a shell subcommand that would drop into an rpcconsole like shell and allow running wallets rpc commands in an offline mode.

@jnewbery

This comment has been minimized.

Show comment
Hide comment
@jnewbery

jnewbery Sep 27, 2018

Member

Cherry picked the changes from #14283 and #14284 and squashed fixup commits. Thanks to @promag and @ken2812221 !

Member

jnewbery commented Sep 27, 2018

Cherry picked the changes from #14283 and #14284 and squashed fixup commits. Thanks to @promag and @ken2812221 !

@jonasschnelli

This comment has been minimized.

Show comment
Hide comment
@jonasschnelli

jonasschnelli Sep 27, 2018

Member

I think that would be a sensible idea for a low-level wallet debugging tool, but a bad idea for a tool that can replace bitcoind and bitcoin-qt as a way of accessing wallets.

I'm not sure about this but I can follow your thoughts. I think one can also argue that the wallet-tool is a low-level accessing tool for experts.

Lets think about an actual use case. If a user who runs bitcoind or bitcoin-qt normally encounters a problem that requires them to run "salvage" or "zaptxs".

If you have to salvage or zap transactions, you are in low-level debug mode IMO.

However, I think we should...
a) ... interpret the provided wallet name as filename (with optional path components)
b) ... if wallet has not been found, check the default datadir/walletdir if it contains the wallet
c) ... allow -regtest/-testnet switch
d) ... describe the path behaviour in the help-text

Member

jonasschnelli commented Sep 27, 2018

I think that would be a sensible idea for a low-level wallet debugging tool, but a bad idea for a tool that can replace bitcoind and bitcoin-qt as a way of accessing wallets.

I'm not sure about this but I can follow your thoughts. I think one can also argue that the wallet-tool is a low-level accessing tool for experts.

Lets think about an actual use case. If a user who runs bitcoind or bitcoin-qt normally encounters a problem that requires them to run "salvage" or "zaptxs".

If you have to salvage or zap transactions, you are in low-level debug mode IMO.

However, I think we should...
a) ... interpret the provided wallet name as filename (with optional path components)
b) ... if wallet has not been found, check the default datadir/walletdir if it contains the wallet
c) ... allow -regtest/-testnet switch
d) ... describe the path behaviour in the help-text

@ryanofsky

This comment has been minimized.

Show comment
Hide comment
@ryanofsky

ryanofsky Sep 27, 2018

Contributor

What you're suggesting sounds good, as long as it doesn't mean (part (a) is vague) that bitcoin-wallet should interpret the -wallet=<path> option differently from bitcoind and bitcoin-qt

The salvage and zaptxs subcommands are somewhat low level, but I don't think a dump command would be so obscure, and I think it would be nice if different various tools understood the same options so for example:

bitcoin-qt -testnet -wallet=foo
bitcoin-wallet -testnet -wallet=foo dump > foo.txt

could just work without the without the user having to figure out where their datadir is located, where the testnet directory lives inside datadir, whether the testnet directory contains a "wallets" subdirectory, or is from a legacy install that preceded it, and so on.

Contributor

ryanofsky commented Sep 27, 2018

What you're suggesting sounds good, as long as it doesn't mean (part (a) is vague) that bitcoin-wallet should interpret the -wallet=<path> option differently from bitcoind and bitcoin-qt

The salvage and zaptxs subcommands are somewhat low level, but I don't think a dump command would be so obscure, and I think it would be nice if different various tools understood the same options so for example:

bitcoin-qt -testnet -wallet=foo
bitcoin-wallet -testnet -wallet=foo dump > foo.txt

could just work without the without the user having to figure out where their datadir is located, where the testnet directory lives inside datadir, whether the testnet directory contains a "wallets" subdirectory, or is from a legacy install that preceded it, and so on.

@jnewbery

This comment has been minimized.

Show comment
Hide comment
@jnewbery

jnewbery Sep 27, 2018

Member

Rebased on master. tool_wallet.py is failing.

Member

jnewbery commented Sep 27, 2018

Rebased on master. tool_wallet.py is failing.

@DrahtBot DrahtBot removed the Needs rebase label Sep 27, 2018

@Sjors

This comment has been minimized.

Show comment
Hide comment
@Sjors

Sjors Sep 28, 2018

Member

What happens if you run this tool on a wallet that's currently opened by bitcoind?

zapwallettxes is sometimes recommended to deal with stuck transactions. Especially before the GUI had basic RBF support, but might still be useful (though it's a pretty blunt instrument).

Member

Sjors commented Sep 28, 2018

What happens if you run this tool on a wallet that's currently opened by bitcoind?

zapwallettxes is sometimes recommended to deal with stuck transactions. Especially before the GUI had basic RBF support, but might still be useful (though it's a pretty blunt instrument).

Show outdated Hide outdated test/functional/tool_wallet.py Outdated
int main(int argc, char* argv[])
{
SetupEnvironment();

This comment has been minimized.

@ken2812221

ken2812221 Oct 8, 2018

Member

Could you follow #13883 to add a Windows argument handler here?

@ken2812221

ken2812221 Oct 8, 2018

Member

Could you follow #13883 to add a Windows argument handler here?

This comment has been minimized.

@jnewbery

jnewbery Oct 17, 2018

Member

done

@jnewbery

jnewbery Oct 17, 2018

Member

done

@jnewbery

This comment has been minimized.

Show comment
Hide comment
@jnewbery

jnewbery Oct 17, 2018

Member

I've updated the tool to treat -datadir, -wallet, -regtest and -testnet arguments in the same way that bitcoind treats those arguments. I've also added help text to explain where it looks for wallet files:

wallet-tool is an offline tool for creating and interacting with Bitcoin Core wallet files.
By default wallet-tool will act on wallets in the default wallet directory in the datadir.
To change the target wallet, use the -datadir, -wallet and -testnet/-regtest arguments.

I believe all comments so far are now addressed.

Member

jnewbery commented Oct 17, 2018

I've updated the tool to treat -datadir, -wallet, -regtest and -testnet arguments in the same way that bitcoind treats those arguments. I've also added help text to explain where it looks for wallet files:

wallet-tool is an offline tool for creating and interacting with Bitcoin Core wallet files.
By default wallet-tool will act on wallets in the default wallet directory in the datadir.
To change the target wallet, use the -datadir, -wallet and -testnet/-regtest arguments.

I believe all comments so far are now addressed.

@ken2812221

This comment has been minimized.

Show comment
Hide comment
@ken2812221

ken2812221 Oct 17, 2018

Member

@jnewbery the filelock on Windows is broken now, so the test would fail. See #14426

Member

ken2812221 commented Oct 17, 2018

@jnewbery the filelock on Windows is broken now, so the test would fail. See #14426

@jnewbery

This comment has been minimized.

Show comment
Hide comment
@jnewbery

jnewbery Oct 18, 2018

Member

New push:

  • Fix the build failure with --disable-wallet
  • skip the test if wallet is not compiled
  • Catch when the wallet file can't be opened earlier
  • rebased on #14426 to try to fix the windows failure

This PR now depends on #14426 .

Member

jnewbery commented Oct 18, 2018

New push:

  • Fix the build failure with --disable-wallet
  • skip the test if wallet is not compiled
  • Catch when the wallet file can't be opened earlier
  • rebased on #14426 to try to fix the windows failure

This PR now depends on #14426 .

@ryanofsky

utACK 719ae60. There are some things I would want to clean up here (mainly deduping code and sharing more functionality between bitcoind and bitcoind-wallet), but this seems good as a starting point for a new tool that can be extended later.

jonasschnelli and others added some commits Sep 16, 2016

[tools] Add wallet inspection and modification tool
This commit adds wallet-tool, a tool for creating and interacting with
wallet files. Original implementation was by Jonas Schnelli
<dev@jonasschnelli.ch> with modifications by John Newbery
<john@johnnewbery.com>

MSVC files were provided by Chun Kuan Lee <ken2812221@gmail.com>:

build: Add MSVC project files for bitcoin-wallet-tool
[tests] Add wallet-tool test
Original tests by João Barbosa <joao.paulo.barbosa@gmail.com>

Additional contribution by John Newbery <john@johnnewbery.com>
@jnewbery

This comment has been minimized.

Show comment
Hide comment
@jnewbery

jnewbery Oct 20, 2018

Member

rebased

Member

jnewbery commented Oct 20, 2018

rebased

@DrahtBot DrahtBot removed the Needs rebase label Oct 21, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment