Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Already on GitHub? Sign in to your account

RPC refactoring: Access wallet using new GetWalletForJSONRPCRequest #8775

Merged
merged 9 commits into from Mar 3, 2017

Conversation

Member

luke-jr commented Sep 21, 2016

Part of the refactorings needed for basic multiwallet (#8694)

Member

jonasschnelli commented Sep 21, 2016

I don't like the coupling and the #ifdef ENABLE_WALLET in rpc/server.cpp|.h.
I'd recommend to keep the CRPCRequestInfo wallet-free.

I think we should pack the request path (URI) into the CRPCRequestInfo and or informations about the authentication (in case we want to distinct wallets based on authentication).
Then I think there should be a method in wallet.cpp (or in rpcwallet.cpp) that maps a CWallet * pointer from a given URI, Auth-Info or the complete CRPCRequestInfo instance.

Instead of the CWallet *& pwallet = reqinfo.wallet; there could be then something like CWallet *pwallet = CWallets::getWalletFromRequest(reqinfo)

Member

jonasschnelli commented Sep 21, 2016

General ConceptACK on a CRPCRequestInfo for the RPC table commands.
Maybe it could also include the UniValue params and fHelp?

src/rpc/misc.cpp
@@ -70,7 +70,8 @@ UniValue getinfo(const UniValue& params, bool fHelp)
);
#ifdef ENABLE_WALLET
- LOCK2(cs_main, pwalletMain ? &pwalletMain->cs_wallet : NULL);
+ CWallet *& pwallet = reqinfo.wallet;
@luke-jr

luke-jr Sep 21, 2016

Member

Yes, pwallet is an alias to reqinfo.wallet which is of type CWallet*.

Member

luke-jr commented Sep 21, 2016

I think we should pack the request path (URI) into the CRPCRequestInfo and or informations about the authentication (in case we want to distinct wallets based on authentication).
Then I think there should be a method in wallet.cpp (or in rpcwallet.cpp) that maps a CWallet * pointer from a given URI, Auth-Info or the complete CRPCRequestInfo instance.

That sounds nice, but greatly complicates the implementation. :(

Member

MarcoFalke commented Oct 19, 2016

I think this can be closed after #8788?

Member

jonasschnelli commented Oct 19, 2016

Closing in favor of merged #8788

luke-jr added a commit to bitcoinknots/bitcoin that referenced this pull request Oct 20, 2016

RPC: Allow function signature to include CWallet reference
Wrapped in CRPCRequestInfo to avoid gratuitous #ifdef ENABLE_WALLET everywhere

Github-Pull: #8775
Rebased-From: 80f4ab7

luke-jr added a commit to bitcoinknots/bitcoin that referenced this pull request Oct 20, 2016

luke-jr added a commit to bitcoinknots/bitcoin that referenced this pull request Oct 20, 2016

@laanwj laanwj reopened this Oct 25, 2016

Member

luke-jr commented Oct 25, 2016

Rebased and refactored based on @jonasschnelli 's idea.

Owner

laanwj commented Oct 25, 2016

Makes sense, utACK ab5ce98

src/rpc/misc.cpp
@@ -26,6 +26,8 @@
using namespace std;
+CWallet *GetWalletForJSONRPCRequest(const JSONRPCRequest&);
@laanwj

laanwj Oct 25, 2016

Owner

Not sure how this can pass the build w/ --disable-wallet, but this should be bracketed with #ifdef ENABLE_WALLET

Member

luke-jr commented Nov 12, 2016

Rebased and addressed nit

@luke-jr luke-jr changed the title from RPC refactoring: Never access wallet directly, only via new CRPCRequestInfo to RPC refactoring: Access wallet using new GetWalletForJSONRPCRequest Nov 12, 2016

luke-jr added a commit to bitcoinknots/bitcoin that referenced this pull request Dec 31, 2016

@luke-jr luke-jr referenced this pull request Jan 3, 2017

Merged

Basic multiwallet support #8694

Member

instagibbs commented Jan 4, 2017

utACK 7de5573

Member

luke-jr commented Jan 5, 2017

Minor change: Forward-declared CWallet even for non-wallet builds so it can be used in a pointer type, avoiding unnecessary casting.

src/rpc/rawtransaction.cpp
@@ -35,6 +35,11 @@
using namespace std;
+#ifdef ENABLE_WALLET
+std::string HelpRequiringPassphrase(CWallet *);
@TheBlueMatt

TheBlueMatt Jan 6, 2017

Contributor

I'd kinda prefer we add another declaration outside of a header in an unrelated file...can we keep it in server.h and ifdef ENABLE_WALLET it?

@luke-jr

luke-jr Jan 6, 2017

Member

I suppose we could. But then it's more likely to get used in new code (which I think we want to discourage?)

@TheBlueMatt

TheBlueMatt Jan 6, 2017

Contributor

I'd think we can (and, more importantly, would) still nag people submitting PRs which add more uses of it outside of src/wallet/rpc*.

src/rpc/misc.cpp
@@ -26,6 +26,10 @@
using namespace std;
+#ifdef ENABLE_WALLET
+CWallet *GetWalletForJSONRPCRequest(const JSONRPCRequest&);
@TheBlueMatt

TheBlueMatt Jan 6, 2017

Contributor

Here as well.

src/rpc/rawtransaction.cpp
@@ -35,6 +35,11 @@
using namespace std;
+#ifdef ENABLE_WALLET
+std::string HelpRequiringPassphrase(CWallet *);
@TheBlueMatt

TheBlueMatt Jan 6, 2017

Contributor

I'd think we can (and, more importantly, would) still nag people submitting PRs which add more uses of it outside of src/wallet/rpc*.

src/wallet/rpcdump.cpp
@@ -29,6 +29,7 @@
using namespace std;
+CWallet *GetWalletForJSONRPCRequest(const JSONRPCRequest&);
void EnsureWalletIsUnlocked(CWallet *);
bool EnsureWalletIsAvailable(CWallet *, bool avoidException);
@TheBlueMatt

TheBlueMatt Jan 6, 2017

Contributor

Can we move these to some rpc.h in src/wallet? Why are we declaring functions in another file from the definition if they're both in src/wallet???

src/wallet/rpcwallet.cpp
@@ -30,6 +30,11 @@ using namespace std;
int64_t nWalletUnlockTime;
static CCriticalSection cs_nWalletUnlockTime;
+CWallet *GetWalletForJSONRPCRequest(const JSONRPCRequest& request)
@TheBlueMatt

TheBlueMatt Jan 6, 2017

Contributor

Can you document this function somewhere (also probably move its definition to src/wallet/rpcwallet.h) - a few places in src/wallet assume it always returns non-NULL but this is not documented (I havent looked at the actual multi-wallet PR, but does this function then throw an RPC exception if you ask for a wallet that isnt loaded, or does it return NULL then)?

Member

gmaxwell commented Jan 7, 2017

So in some cases in the RPC you get the wallet pointer from json but then the check if it's available is far below. This is begging for someone to insert code that uses a potentially null pointer between to two and doesn't notice because their function doesn't get tested with a missing wallet. I would recommend moving the creation of that local pointer down to the point where you're going to test it.

Alternatively or in addition, rename GetWalletForJSONRPCRequest to TryGetWalletForJSONRPCRequest and make GetWalletForJSONRPCRequest a wrapper for it that throws when it fails?

Other than this nit that perhaps getting the pointer and testing it are too separated in some cases, utACK.

Member

luke-jr commented Jan 7, 2017

I liked the TryGetWalletForJSONRPCRequest/GetWalletForJSONRPCRequest refactor idea, but I don't see any clean way to do it without changing the help behaviours.

So I moved the one case EnsureWalletIsAvailable was delayed up, and removed the blank line between them and GetWalletForJSONRPCRequest.

Please do not tag the 4th commit "Trivial". We usually define trivial as doesnt touch any code.

As for the 6th commit: please do not do this. nothing in src/wallet is built when ENABLE_WALLET is off, so generally src/wallet/* should never be included outside of src/wallet/*

Aside from the (partial-revert) of "Move wallet RPC declarations to rpcwallet.h" and the printing of the raw pointers to debug log, this looks good to me at d9aff6e.

-void ImportAddress(const CBitcoinAddress& address, const string& strLabel);
-void ImportScript(const CScript& script, const string& strLabel, bool isRedeemScript)
+void ImportAddress(CWallet*, const CBitcoinAddress& address, const string& strLabel);
+void ImportScript(CWallet * const pwallet, const CScript& script, const string& strLabel, bool isRedeemScript)
@TheBlueMatt

TheBlueMatt Jan 7, 2017

Contributor

Note that there are a ton of uses of pwalletMain in ImportScript after the first commit ("RPC/Wallet: Pass CWallet as pointer to helper functions") which are fixed in the next ("RPC: Do all wallet access through new GetWalletForJSONRPCRequest") when they belong in the first.

@ryanofsky

ryanofsky Feb 27, 2017

Contributor

Note that there are a ton of uses of pwalletMain in ImportScript after the first commit ("RPC/Wallet: Pass CWallet as pointer to helper functions") which are fixed in the next ("RPC: Do all wallet access through new GetWalletForJSONRPCRequest") when they belong in the first.

I'm not seeing function signatures changes in c68b5a8 RPC: Do all wallet access through new GetWalletForJSONRPCRequest so the two commits do seem distinct currently.

src/wallet/rpcwallet.cpp
- nWalletUnlockTime = GetTime() + nSleepTime;
- RPCRunLater("lockwallet", boost::bind(LockWallet, pwallet), nSleepTime);
+ pwallet->nRelockTime = GetTime() + nSleepTime;
+ RPCRunLater(strprintf("lockwallet_%u", uintptr_t(pwallet)), boost::bind(LockWallet, pwallet), nSleepTime);
@TheBlueMatt

TheBlueMatt Jan 7, 2017

Contributor

I super dont like the fact that the pointer to the wallet will end up in debug.log if you have debug=rpc enabled.

@ryanofsky

ryanofsky Feb 27, 2017

Contributor

I super dont like the fact that the pointer to the wallet will end up in debug.log if you have debug=rpc enabled.

Is fixed in later commit c42f3a5 Wallet/RPC: Use filename rather than CWallet pointer, for lockwallet RPCRunLater job name

src/wallet/rpcwallet.cpp
@@ -340,8 +349,7 @@ UniValue getaddressesbyaccount(const JSONRPCRequest& request)
// Find all addresses that have the given account
UniValue ret(UniValue::VARR);
- BOOST_FOREACH(const PAIRTYPE(CBitcoinAddress, CAddressBookData)& item, pwallet->mapAddressBook)
- {
+ for (const PAIRTYPE(CBitcoinAddress, CAddressBookData)& item : pwallet->mapAddressBook) {
@TheBlueMatt

TheBlueMatt Jan 7, 2017

Contributor

nit: I believe you can remove the PAIRTYPE and replace with std::pair since we're not using BOOST_FOREACH now.

@@ -1126,7 +1129,7 @@ struct tallyitem
}
};
-UniValue ListReceived(const UniValue& params, bool fByAccounts)
+UniValue ListReceived(CWallet * const pwallet, const UniValue& params, bool fByAccounts)
@TheBlueMatt

TheBlueMatt Jan 7, 2017

Contributor

Note that there are a ton of uses of pwalletMain in ListReceived after the first commit ("RPC/Wallet: Pass CWallet as pointer to helper functions") which are fixed in the next ("RPC: Do all wallet access through new GetWalletForJSONRPCRequest") when they belong in the first.

src/wallet/rpcwallet.cpp
@@ -1219,8 +1241,7 @@ UniValue ListReceived(CWallet * const pwallet, const UniValue& params, bool fByA
// Reply
UniValue ret(UniValue::VARR);
map<string, tallyitem> mapAccountTally;
- BOOST_FOREACH(const PAIRTYPE(CBitcoinAddress, CAddressBookData)& item, pwallet->mapAddressBook)
- {
+ for (const PAIRTYPE(CBitcoinAddress, CAddressBookData)& item : pwallet->mapAddressBook) {
@TheBlueMatt

TheBlueMatt Jan 7, 2017

Contributor

nit: Here as well (no PAIRTYPE).

src/wallet/rpcwallet.cpp
@@ -1635,14 +1665,14 @@ UniValue listaccounts(const JSONRPCRequest& request)
includeWatchonly = includeWatchonly | ISMINE_WATCH_ONLY;
map<string, CAmount> mapAccountBalances;
- BOOST_FOREACH(const PAIRTYPE(CTxDestination, CAddressBookData)& entry, pwallet->mapAddressBook) {
- if (IsMine(*pwallet, entry.first) & includeWatchonly) // This address belongs to me
+ for (const PAIRTYPE(CTxDestination, CAddressBookData)& entry : pwallet->mapAddressBook) {
@TheBlueMatt

TheBlueMatt Jan 7, 2017

Contributor

nit: Here as well (no PAIRTYPE).

Member

gmaxwell commented Jan 8, 2017

@luke-jr Looks good to me, will test as soon as you update for matt's latest comments.

Member

luke-jr commented Jan 8, 2017

Made changes requested by @TheBlueMatt, and rebased to resolve conflict. Also includes a quick sanity check that -wallet doesn't include path separators.

Contributor

TheBlueMatt commented Jan 9, 2017

While you're at it, can you call SanitizeString (in addition to the "/\" check) on the wallet param?

Contributor

TheBlueMatt commented Jan 9, 2017

utACK 7d1228b

Member

gmaxwell commented Jan 11, 2017

Needs rebase.

Member

gmaxwell commented Jan 12, 2017

ACK.

Member

fanquake commented Jan 12, 2017

Needs rebasing again.

Member

luke-jr commented Jan 12, 2017

Rebased yet again.

Member

gmaxwell commented Jan 15, 2017

still ACK

@@ -3740,6 +3740,12 @@ bool CWallet::InitLoadWallet()
std::string walletFile = GetArg("-wallet", DEFAULT_WALLET_DAT);
@BlockMechanic

BlockMechanic Feb 16, 2017

Should this not be "count" i.e for multiple wallet files to be created ? Adding perhaps a for loop to iterate over required num of wallets?

@luke-jr

luke-jr Feb 16, 2017

Member

No, because this PR is not multiwallet...

@@ -112,13 +112,17 @@ UniValue getinfo(const JSONRPCRequest& request)
class DescribeAddressVisitor : public boost::static_visitor<UniValue>
{
public:
+ CWallet * const pwallet;
@ryanofsky

ryanofsky Feb 27, 2017

Contributor

Seems like this could be a pointer to a const CWallet. Same for some other cases. Would suggest using const CWallet* instead of CWallet* where possible for more safety and clarity.

src/rpc/misc.cpp
@@ -101,8 +101,9 @@ UniValue getinfo(const JSONRPCRequest& request)
obj.push_back(Pair("keypoololdest", pwallet->GetOldestKeyPoolTime()));
obj.push_back(Pair("keypoolsize", (int)pwallet->GetKeyPoolSize()));
}
- if (pwallet && pwallet->IsCrypted())
+ if (pwallet && pwallet->IsCrypted()) {
@ryanofsky

ryanofsky Feb 27, 2017

Contributor

Commit 09f3076 Reformat touched lines with C++11 looks fine, but it doesn't appear that these changes have anything to do with C++11. Maybe clarify commit message.

@@ -234,6 +235,9 @@ UniValue validateaddress(const JSONRPCRequest& request)
return ret;
}
+// Needed even with !ENABLE_WALLET, to pass (ignored) pointers around
+class CWallet;
@ryanofsky

ryanofsky Feb 27, 2017

Contributor

This doesn't actually seem to be needed since CWallet is also forward declared in init.h, but maybe it is better to keep it.

- if (pwalletMain) {
- obj.push_back(Pair("walletversion", pwalletMain->GetVersion()));
- obj.push_back(Pair("balance", ValueFromAmount(pwalletMain->GetBalance())));
+ if (pwallet) {
@ryanofsky

ryanofsky Feb 27, 2017

Contributor

Note to reviewers: This commit (d77ad6d RPC: Do all wallet access through new GetWalletForJSONRPCRequest) is trivial to review if you configure your diff tool to ignore the pwallet->pwalletmain renames (much shorter and no changed lines, only inserted).

src/rpc/server.h
+
+#ifdef ENABLE_WALLET
+// New code should accessing the wallet should be under the ../wallet/ directory
+std::string HelpRequiringPassphrase(CWallet *);
@ryanofsky

ryanofsky Feb 27, 2017

Contributor

Maybe squash commit 6033b43 Move wallet RPC declarations to rpcwallet.h into commit f757a41 RPC/Wallet: Pass CWallet as pointer to helper functions, to just add to right place instead of adding and moving.

-void ImportAddress(const CBitcoinAddress& address, const string& strLabel);
-void ImportScript(const CScript& script, const string& strLabel, bool isRedeemScript)
+void ImportAddress(CWallet*, const CBitcoinAddress& address, const string& strLabel);
+void ImportScript(CWallet * const pwallet, const CScript& script, const string& strLabel, bool isRedeemScript)
@ryanofsky

ryanofsky Feb 27, 2017

Contributor

Note that there are a ton of uses of pwalletMain in ImportScript after the first commit ("RPC/Wallet: Pass CWallet as pointer to helper functions") which are fixed in the next ("RPC: Do all wallet access through new GetWalletForJSONRPCRequest") when they belong in the first.

I'm not seeing function signatures changes in c68b5a8 RPC: Do all wallet access through new GetWalletForJSONRPCRequest so the two commits do seem distinct currently.

src/wallet/rpcwallet.cpp
@@ -2063,9 +2060,8 @@ UniValue walletpassphrase(const JSONRPCRequest& request)
pwallet->TopUpKeyPool();
int64_t nSleepTime = request.params[1].get_int64();
- LOCK(cs_nWalletUnlockTime);
@ryanofsky

ryanofsky Feb 27, 2017

Contributor

Is the syncrhonization not needed anymore?

src/wallet/rpcwallet.cpp
- nWalletUnlockTime = GetTime() + nSleepTime;
- RPCRunLater("lockwallet", boost::bind(LockWallet, pwallet), nSleepTime);
+ pwallet->nRelockTime = GetTime() + nSleepTime;
+ RPCRunLater(strprintf("lockwallet_%u", uintptr_t(pwallet)), boost::bind(LockWallet, pwallet), nSleepTime);
@ryanofsky

ryanofsky Feb 27, 2017

Contributor

I super dont like the fact that the pointer to the wallet will end up in debug.log if you have debug=rpc enabled.

Is fixed in later commit c42f3a5 Wallet/RPC: Use filename rather than CWallet pointer, for lockwallet RPCRunLater job name

@laanwj laanwj self-assigned this Mar 2, 2017

Member

jonasschnelli commented Mar 2, 2017

Great PR. I think we should get this in as soon as possible.
utACK d678771

Owner

laanwj commented Mar 3, 2017

utACK d678771

@laanwj laanwj merged commit d678771 into bitcoin:master Mar 3, 2017

1 check passed

continuous-integration/travis-ci/pr The Travis CI build passed
Details

laanwj added a commit that referenced this pull request Mar 3, 2017

Merge #8775: RPC refactoring: Access wallet using new GetWalletForJSO…
…NRPCRequest


d678771 Wallet: Sanitise -wallet parameter (Luke Dashjr)
9756be3 Wallet/RPC: Use filename rather than CWallet pointer, for lockwallet RPCRunLater job name (Luke Dashjr)
86be48a More tightly couple EnsureWalletIsAvailable with GetWalletForJSONRPCRequest where appropriate (Luke Dashjr)
a435632 Move wallet RPC declarations to rpcwallet.h (Luke Dashjr)
ad15734 RPC: Pass on JSONRPCRequest metadata (URI/user/etc) for "help" method (Luke Dashjr)
bf8a04a Reformat touched lines with C++11 (Luke Dashjr)
2e518e3 Move nWalletUnlockTime to CWallet::nRelockTime, and name timed task unique per CWallet (Luke Dashjr)
d77ad6d RPC: Do all wallet access through new GetWalletForJSONRPCRequest (Luke Dashjr)
eca550f RPC/Wallet: Pass CWallet as pointer to helper functions (Luke Dashjr)

Tree-SHA512: bfd592da841693390e16f83b451503eb5cedb71208089aa32b3fc45e973555584a3ed7696dd239f6409324464d565dacf0f3d0e36e8e13ae6a7843848465f960

@fanquake fanquake moved this from In progress to Done in Multiwallet support Mar 3, 2017

@jtimon jtimon referenced this pull request Mar 6, 2017

Closed

RPC: cleanups in rpc/server #9845

luke-jr added a commit to bitcoinknots/bitcoin that referenced this pull request Mar 7, 2017

luke-jr added a commit to bitcoinknots/bitcoin that referenced this pull request Mar 7, 2017

luke-jr added a commit to bitcoinknots/bitcoin that referenced this pull request Mar 7, 2017

luke-jr added a commit to bitcoinknots/bitcoin that referenced this pull request Mar 7, 2017

luke-jr added a commit to bitcoinknots/bitcoin that referenced this pull request Mar 7, 2017

luke-jr added a commit to bitcoinknots/bitcoin that referenced this pull request Mar 7, 2017

luke-jr added a commit to bitcoinknots/bitcoin that referenced this pull request Mar 7, 2017

More tightly couple EnsureWalletIsAvailable with GetWalletForJSONRPCR…
…equest where appropriate

Github-Pull: #8775
Rebased-From: 86be48a

luke-jr added a commit to bitcoinknots/bitcoin that referenced this pull request Mar 7, 2017

Wallet/RPC: Use filename rather than CWallet pointer, for lockwallet …
…RPCRunLater job name

The job name is logged, and could pose as an information leak to someone attacking the process, helping them counteract ASLR protections

Github-Pull: #8775
Rebased-From: 9756be3

luke-jr added a commit to bitcoinknots/bitcoin that referenced this pull request Mar 7, 2017

Member

jtimon commented Mar 23, 2017

By the way, I reviewed this partially after merged when rebasing #9845 (making it mostly pointless since most was done here already).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment