Account name #155

tuxcanfly · 2014-12-17T09:51:32Z

NOTE: This pull request requires #147

Updated waddrmgr to support account names and integrated with wallet RPC calls.

davecgh · 2014-12-17T17:33:11Z

Thanks @tuxcanfly. I'll go through and review this as time permits. It's obviously a huge change.

davecgh · 2014-12-31T19:15:14Z

waddrmgr/db.go

+// The returned value is a slice of account numbers which can be used to load
+// the respective account rows.
+func fetchAllAccounts(tx walletdb.Tx) ([]uint32, error) {
+	accounts := []uint32{}


Please add a TODO here to load the number of accounts from the db first and create the slice with enough space for them all at once. Otherwise, it's hard on the GC because it has to grow the backing array multiple times.

Also, super nit picky, but every other function has the bucket := ... line as the first one in the function, so I'd prefer that be consistent.

I'm now using a 'meta' bucket to store metadata like no of accounts, no of addrs etc and using it to create sized slices.

Moved bucket := ... to the first line.

davecgh · 2014-12-31T19:21:25Z

waddrmgr/db.go

+
+	offset := uint32(0)
+	var acctName string
+	bucket.ForEach(func(k, v []byte) error {


Hmmm. Any reason this is iterating instead of simply looking up the account number by key? I realize currently there are only names in the key, but I suspect it would be better to store both in the index. That is to say the index bucket would have both:

bucket[acctName] = acctNum bucket[acctNum] = acctName

EDIT: One thing to keep in mind if this is done is that an empty account name would be stored as [0 0 0 0], the same as account number 0, so it would need some type of special handling probably.

I've added a new bucket to store in the index mapping account name to it's id.

davecgh · 2014-12-31T20:08:37Z

waddrmgr/db.go

+	}
+
+	// Write a null value keyed by the address hash
+	err = bucket.Put(addrHash, []byte{0})


I'd make a global const (will have to be a var actually, but treat it like a const) for this byte slice so it doesn't have to generate a new instance that escapes to the heap every time thereby generating extra garbage for the GC to clean up after.

Added a var called nullVal and using it as a global.

davecgh · 2014-12-31T20:31:53Z

waddrmgr/db.go

+// The returned value is a slice address rows for each specific address type.
+// The caller should use type assertions to ascertain the types.
+func fetchAccountAddresses(tx walletdb.Tx, account uint32) ([]interface{}, error) {
+	var addrs []interface{}


The same as previously mentioned here. Please avoid using an unsized slice that will require the backing array to be reallocated and copied multiple times.

Yup, doing this now.

davecgh · 2014-12-31T20:35:09Z

I'm still reviewing, but one thing in general I want to bring up that we should discuss is how to avoid loading all of the addresses into a slice at once. That will not scale to millions/billions of addresses.

davecgh · 2014-12-31T22:40:37Z

waddrmgr/manager.go

@@ -1657,6 +1861,19 @@ func checkBranchKeys(acctKey *hdkeychain.ExtendedKey) error {
 	return err
 }

+// ValidateAccountName validates the given account name and returns an error, if any.
+func ValidateAccountName(name string) error {


I know that Go doesn't really care where the functions are placed, but the rest of code base strives to define functions before they are used. Please move this before NewAccount which is the first (and only) thing that uses it.

davecgh · 2014-12-31T23:14:09Z

waddrmgr/manager.go

+// ErrDuplicateAccount will be returned.  Since creating a new account requires
+// access to the cointype keys (from which extended account keys are derived),
+// it requires the manager to be unlocked.
+func (m *Manager) NewAccount(name string) (uint32, error) {


Since the manager has to be unlocked, this needs checks to return the appropriate errors when it's not. Also, this can't be done when the manager is watching only, so that needs an error too.

if m.watchingOnly { return managerError(ErrWatchingOnly, errWatchingOnly, nil) } m.mtx.Lock() defer m.mtx.Unlock() if m.locked { return 0, managerError(ErrLocked, errLocked, nil) }

It's ok to access the watching only flag when not under the mutex because it can't be changed after creation, but the m.locked field requires the mutex.

This also obviously means the later calls to the exported methods which grab the mutex need unexported versions that are invoked with the mutex held. Even more importantly, the mutex needs to be held for the crypto operations later too so the manager is not unlocked out from under this code after the initial check.

Yup, adding the mutex lock and checks as described. We were calling LookupAccount with the mutex lock held so I've added a unexpected version lookupAccount to make it usable.

davecgh · 2014-12-31T23:17:59Z

waddrmgr/manager.go

+// RenameAccount renames an account stored in the manager based on the
+// given account number with the given name.  If an account with the same name
+// already exists, ErrDuplicateAccount will be returned.
+func (m *Manager) RenameAccount(account uint32, name string) error {


I believe all of these new exported functions generally need the manager mutex held for the duration of the function since otherwise you're opening up races in between write/reads from the database.

Checked and added mutex lock to newly exported funcs.

davecgh · 2014-12-31T23:25:02Z

waddrmgr/manager.go

+		if err != nil {
+			return err
+		}
+		// TODO: needs to be atomic?


No. It's a local variable to the closure.

OK, removed the TODO.

davecgh · 2014-12-31T23:27:18Z

waddrmgr/manager.go

+		}
+
+		// Derive the account key using the cointype key
+		acctKeyPriv, err := deriveAccountKey(coinTypeKeyPriv, account)


There error isn't being checked here.

Oops... added the check now.

The coin type extended key bytes need to be zeroed as soon as they are used.

coinTypeKeyPriv.Zero()

NOTE: This is intentionally before the error check so it happens regardless.

Zeroing the coinTypeKeyPriv now.

davecgh · 2014-12-31T23:31:04Z

waddrmgr/manager.go

+			str := "failed to encrypt private key for account"
+			return managerError(ErrCrypto, str, err)
+		}
+		// We have the encrypted account extended keys, so save them to the database


This looks odd as the comment isn't wrapped to 80 while the function is wrapped too early.

Re-formatted both to wrap properly.

davecgh · 2015-01-12T05:21:41Z

waddrmgr/db.go

@@ -862,14 +1225,108 @@ func existsAddress(tx walletdb.Tx, addressID []byte) bool {
 	return bucket.Get(addrHash[:]) != nil
 }

+// fetchAddrAccount returns the account to which the given address belongs to.
+// It looks up the account using the addracctidx index which maps the address
+// hash to it's corresponding account id.


it's -> its

davecgh · 2015-01-12T05:25:36Z

waddrmgr/db.go

+// fetchAccountAddresses loads information about addresses of an account from the database.
+// The returned value is a slice address rows for each specific address type.
+// The caller should use type assertions to ascertain the types.
+func fetchAccountAddresses(tx walletdb.Tx, account uint32) ([]interface{}, error) {


I'd rather see this function make use of fetchAddress than repeating the same code which would require updating it in two places when a new address type is added. I understand that fetchAddress expects the address whereas the account index stores the hash of the address (which is absolutely correct to do).

I'd suggest simply refactoring the common code into a separate function that both call with the hash of the address so that fetchAddress can take the hash of the passed address, while this function looks up the already hashed address and can pass it directly.

Added a func fetchAddressByHash which can be re-used from fetchAddress as well fetchAccountAddresses, fetchAllAddresses One problem here is we are expecting addressID in deserializeAddressRow but we don't have it when querying by addrHash. Using the addrHash as a placeholder for now.

Fair point. It's only being used for the error message though. How about just removing that parameter from the function calls and error messages, and then, at the two call sites for fetchAddress:

if err != nil { if merr, ok := err.(*ManagerError); ok { desc := fmt.Sprintf("failed to fetch address '%s': %v", address.ScriptAddress(), merr.Description) merr.Description = desc } }

For the call site that is querying by addrHash just change the message to failed to fetch address hash '%s': %v.

Then add a comment on the errors in the functions that they are later prefixed with the address or hash that caused the failure.

Added modified err in callers for both fetchAddress and fetchAddressByHash and updated comments accordingly.

davecgh · 2015-01-12T05:29:06Z

waddrmgr/db.go

+		return []interface{}{}, nil
+	}
+
+	numAccountAddrs, err := fetchNumAccountAddrs(tx, account)


Ignoring the error doesn't seem like proper behavior here. The called function will return zero if the bucket has not yet been initialized, so any error that occurs here is unexpected and should be handled appropriately.

Further, I think that when the bucket is created, all count fields should be initialized to zero at that time, and the fetchNumAccountAddrs should then treat the missing field as an error. Otherwise, it easily allows unexpected logic to creep in, since you're essentially hiding what should be an error condition by returning zero.

EDIT: Oh I missed you're returning the error. Please do return nil, err.

davecgh · 2015-03-17T02:35:37Z

waddrmgr/db.go

@@ -38,6 +39,8 @@ var (
 	latestMgrVersion uint32 = LatestMgrVersion
 )

+type obtainFunc func() ([]byte, error)


This should be exported since it's used in the Confg struct. It'll need a comment too once exported.

davecgh · 2015-03-17T02:36:04Z

waddrmgr/db.go

@@ -1334,14 +1663,12 @@ func upgradeToVersion2(namespace walletdb.Namespace) error {

 // upgradeManager upgrades the data in the provided manager namespace to newer
 // versions as neeeded.
-func upgradeManager(namespace walletdb.Namespace) error {
-	// Get the current version.
+func upgradeManager(namespace walletdb.Namespace, pubPassPhrase []byte, obtainSeed, obtainPrivPassPhrase obtainFunc) error {


As noted below, I suggest changing the signature to accept the entire config struct so it won't have to be changed in the future if any new information is needed.

davecgh · 2015-03-17T02:36:40Z

waddrmgr/manager.go

@@ -1858,7 +2170,7 @@ func Open(namespace walletdb.Namespace, pubPassphrase []byte, chainParams *chain
 	}

 	// Upgrade the manager to the latest version as needed.
-	if err := upgradeManager(namespace); err != nil {
+	if err := upgradeManager(namespace, pubPassphrase, config.ObtainSeed, config.ObtainPrivatePass); err != nil {


This would panic if a nil config (which is valid) is passed. I'd move the section below which sets the default config above this line and change the upgradeManager signature to take the entire config struct pointer instead of the individual entries.

davecgh · 2015-03-17T02:40:10Z

waddrmgr/manager.go

+	ScryptN           int
+	ScryptR           int
+	ScryptP           int
+	ObtainSeed        obtainFunc


Please add a comment for this field's purpose.

// ObtainSeed is a callback function that is potentially invoked during // upgrades. It is intended to be used to request the wallet seed // from the user (or any other mechanism the caller deems fit).

davecgh · 2015-03-17T02:40:59Z

waddrmgr/manager.go

+	ScryptR           int
+	ScryptP           int
+	ObtainSeed        obtainFunc
+	ObtainPrivatePass obtainFunc


Please add a comment for this field's purpose.

// ObtainPrivatePass is a callback function that is potentially invoked // during upgrades. It is intended to be used to request the wallet // private passphrase from the user (or any other mechanism the caller // deems fit).

jrick · 2015-03-17T15:15:22Z

I think that the account names should not be set in the RPC handlers for gettransaction and listtransactions. These fields are deprecated in recent bitcoind wallet versions (which makes sense, since account balances there can change without an on-chain transaction) and we have plans for a proper ledger RPC to report credits and debits across accounts that is intentionally incompatible with bitcoind.

tuxcanfly · 2015-03-17T15:20:55Z

@jrick Agreed, that would simplify some hoops I had to jump through to make the RPC results compat with core.

jrick · 2015-03-17T18:13:02Z

wallet.go

 	"github.com/btcsuite/btcwallet/chain"
 	"github.com/btcsuite/btcwallet/txstore"
 	"github.com/btcsuite/btcwallet/waddrmgr"
 	"github.com/btcsuite/btcwallet/walletdb"
+	"golang.org/x/crypto/ssh/terminal"


Use "github.com/btcsuite/golangcrypto/ssh/terminal".

jrick · 2015-03-17T18:16:50Z

rpcserver.go

 	return []btcjson.Cmd{n}
 }

 func (d txDebit) notificationCmds(w *Wallet) []btcjson.Cmd {
 	blk := w.Manager.SyncedTo()
-	ltrs, err := txstore.Debits(d).ToJSON("", blk.Height, activeNet.Params)
+	acctName := waddrmgr.DefaultAccountName
+	if debitAccount, err := w.DebitAccount(txstore.Debits(d)); err == nil {


I would also just use "" here (this is an old api and will eventually be replaced) instead of looking up the account. This is the last call to DebitAccount, so with this gone, you can remove that func as well.

jrick · 2015-03-17T18:20:22Z

wallet.go

+// a given account. It returns true if atleast one address in the account was used
+// and false if no address in the account was used.
+func (w *Wallet) AccountUsed(account uint32) (bool, error) {
+	addrs, err := w.Manager.ActiveAccountAddresses(account)


This is an assumption because currently ActiveAccountAddresses returns all addresses from the account, but may not in the future. If it is changed (when single use addrs are added) to only return unused addresses, the used account may be marked unused.

jrick · 2015-03-17T18:23:10Z

wallet.go

+	// the number of tx confirmations.
+	blk := w.Manager.SyncedTo()
+
+	for _, r := range w.TxStore.Records() {


This should only be checking unspent outputs, not every record.

jrick · 2015-03-17T18:28:51Z

wallet.go

@@ -1179,6 +1323,22 @@ func (w *Wallet) ResendUnminedTxs() {
 	}
 }

+// SortedActiveAccountAddresses returns a slice of addresses of an account in a wallet.
+func (w *Wallet) SortedActiveAccountAddresses(account uint32) ([]string, error) {


I would just remove this. There's no reason to need addresses to be sorted. I think this was previously added to mimic an implementation detail in bitcoind (a std::map is usually implemented as an RB tree, so iterating over k/v pairs has a defined order). Instead, I would change GetAddressesByAccount to return a JSON array of all addresses (even those not marked active) in any order.

jrick · 2015-03-17T18:31:40Z

wallet.go

+// TotalReceivedForAccount iterates through a wallet's transaction history,
+// returning the total amount of bitcoins received for a single wallet
+// account.
+func (w *Wallet) TotalReceivedForAccount(account uint32, confirms int) (btcutil.Amount, uint64, error) {


The previous function which did this and didn't take the account (TotalReceived) is no longer used anywhere and should be removed.

We need this for getreceivedbyaccount, listreceivedbyaccount

Right, we need TotalReceivedForAccount, but TotalReceived is no longer used and is dead code.

jrick · 2015-03-18T03:29:49Z

rpcserver.go

-	}
-
-	err := checkAccountName(account)
+	err := checkAccountName(*cmd.Account)


This is not safe. The field is optional and this may be nil.

tuxcanfly · 2015-03-18T05:40:14Z

@jrick Thanks, updated.

davecgh · 2015-03-18T20:31:40Z

Alright, this looks good to go! Please squash it down to a single commit and rebase to the latest master.

tuxcanfly · 2015-03-19T01:01:36Z

Done.

tuxcanfly · 2015-03-19T03:31:59Z

Great! Thanks for the patient reviews @davecgh @jrick

tuxcanfly mentioned this pull request Dec 17, 2014

Add package wtxmgr to use walletdb interface #154

Closed

tuxcanfly force-pushed the tuxcanfly:account-name branch 3 times, most recently from 0722d9c to a12fd58 Dec 18, 2014

tuxcanfly force-pushed the tuxcanfly:account-name branch from a12fd58 to 53ea9c5 Dec 27, 2014

davecgh reviewed Dec 31, 2014
View changes

tuxcanfly force-pushed the tuxcanfly:account-name branch 10 times, most recently from 7da37c9 to 5d7176a Jan 1, 2015

davecgh reviewed Jan 12, 2015
View changes

davecgh reviewed Mar 17, 2015
View changes

tuxcanfly force-pushed the tuxcanfly:account-name branch 4 times, most recently from 4c34e2d to fbd7aea Mar 17, 2015

tuxcanfly force-pushed the tuxcanfly:account-name branch from c0911e0 to dd835b4 Mar 17, 2015

jrick reviewed Mar 17, 2015
View changes

wallet.go Outdated

// the number of tx confirmations.

blk := w.Manager.SyncedTo()

for _, r := range w.TxStore.Records() {

This comment has been minimized.

Sign in to view

jrick Mar 17, 2015
Member

This should only be checking unspent outputs, not every record.

jrick reviewed Mar 17, 2015
View changes

tuxcanfly force-pushed the tuxcanfly:account-name branch from dd835b4 to 48371c9 Mar 18, 2015

jrick reviewed Mar 18, 2015
View changes

rpcserver.go Outdated

}

err := checkAccountName(account)

err := checkAccountName(*cmd.Account)

This comment has been minimized.

Sign in to view

jrick Mar 18, 2015
Member

This is not safe. The field is optional and this may be nil.

tuxcanfly closed this Mar 18, 2015

tuxcanfly reopened this Mar 18, 2015

tuxcanfly force-pushed the tuxcanfly:account-name branch from 088ac1e to 15b1b75 Mar 18, 2015

Updated waddrmgr to manage account names

Loading status checks…

68a9168

tuxcanfly force-pushed the tuxcanfly:account-name branch from 15b1b75 to 68a9168 Mar 19, 2015

conformal-deploy merged commit 68a9168 into btcsuite:master Mar 19, 2015
1 check passed

1 check passed

continuous-integration/travis-ci/pr The Travis CI build passed
Details

btcsuite / btcwallet

Account name #155

Account name #155

Conversation

tuxcanfly commented Dec 17, 2014

davecgh commented Dec 17, 2014

This comment has been minimized.

davecgh Dec 31, 2014 Member

This comment has been minimized.

tuxcanfly Jan 2, 2015 Author Contributor

This comment has been minimized.

davecgh Dec 31, 2014 Member

This comment has been minimized.

tuxcanfly Jan 2, 2015 Author Contributor

This comment has been minimized.

davecgh Dec 31, 2014 Member

This comment has been minimized.

tuxcanfly Jan 2, 2015 Author Contributor

This comment has been minimized.

davecgh Dec 31, 2014 Member

This comment has been minimized.

tuxcanfly Jan 2, 2015 Author Contributor

davecgh commented Dec 31, 2014

This comment has been minimized.

davecgh Dec 31, 2014 Member

This comment has been minimized.

tuxcanfly Jan 2, 2015 Author Contributor

This comment has been minimized.

davecgh Dec 31, 2014 Member

This comment has been minimized.

tuxcanfly Jan 2, 2015 Author Contributor

This comment has been minimized.

davecgh Dec 31, 2014 Member

This comment has been minimized.

tuxcanfly Jan 2, 2015 Author Contributor

This comment has been minimized.

davecgh Dec 31, 2014 Member

This comment has been minimized.

tuxcanfly Jan 2, 2015 Author Contributor

This comment has been minimized.

davecgh Dec 31, 2014 Member

This comment has been minimized.

tuxcanfly Jan 2, 2015 Author Contributor

This comment has been minimized.

davecgh Jan 12, 2015 Member

This comment has been minimized.

tuxcanfly Jan 12, 2015 Author Contributor

This comment has been minimized.

davecgh Dec 31, 2014 Member

This comment has been minimized.

tuxcanfly Jan 2, 2015 Author Contributor

This comment has been minimized.

davecgh Jan 12, 2015 Member

This comment has been minimized.

davecgh Jan 12, 2015 Member

This comment has been minimized.

tuxcanfly Jan 12, 2015 Author Contributor

This comment has been minimized.

davecgh Jan 12, 2015 Member

This comment has been minimized.

tuxcanfly Jan 13, 2015 Author Contributor

This comment has been minimized.

davecgh Jan 12, 2015 Member

This comment has been minimized.

davecgh Mar 17, 2015 Member

This comment has been minimized.

davecgh Mar 17, 2015 Member

This comment has been minimized.

davecgh Mar 17, 2015 Member

This comment has been minimized.

davecgh Mar 17, 2015 Member

This comment has been minimized.

davecgh Mar 17, 2015 Member

jrick commented Mar 17, 2015

tuxcanfly commented Mar 17, 2015

This comment has been minimized.

jrick Mar 17, 2015 Member

This comment has been minimized.

jrick Mar 17, 2015 Member

This comment has been minimized.

davecgh Dec 31, 2014
Member

tuxcanfly Jan 2, 2015
Author Contributor

davecgh Dec 31, 2014
Member

tuxcanfly Jan 2, 2015
Author Contributor

davecgh Dec 31, 2014
Member

tuxcanfly Jan 2, 2015
Author Contributor

davecgh Dec 31, 2014
Member

tuxcanfly Jan 2, 2015
Author Contributor

davecgh Dec 31, 2014
Member

tuxcanfly Jan 2, 2015
Author Contributor

davecgh Dec 31, 2014
Member

tuxcanfly Jan 2, 2015
Author Contributor

davecgh Dec 31, 2014
Member

tuxcanfly Jan 2, 2015
Author Contributor

davecgh Dec 31, 2014
Member

tuxcanfly Jan 2, 2015
Author Contributor

davecgh Dec 31, 2014
Member

tuxcanfly Jan 2, 2015
Author Contributor

davecgh Jan 12, 2015
Member

tuxcanfly Jan 12, 2015
Author Contributor

davecgh Dec 31, 2014
Member

tuxcanfly Jan 2, 2015
Author Contributor

davecgh Jan 12, 2015
Member

davecgh Jan 12, 2015
Member

tuxcanfly Jan 12, 2015
Author Contributor

davecgh Jan 12, 2015
Member

tuxcanfly Jan 13, 2015
Author Contributor

davecgh Jan 12, 2015
Member

davecgh Mar 17, 2015
Member

davecgh Mar 17, 2015
Member

davecgh Mar 17, 2015
Member

davecgh Mar 17, 2015
Member

davecgh Mar 17, 2015
Member

jrick Mar 17, 2015
Member

jrick Mar 17, 2015
Member

jrick Mar 17, 2015
Member

jrick Mar 17, 2015
Member

jrick Mar 17, 2015
Member

jrick Mar 17, 2015
Member

tuxcanfly Mar 18, 2015
Author Contributor

jrick Mar 18, 2015
Member

jrick Mar 18, 2015
Member