Account name

[tuxcanfly]

NOTE: This pull request requires #147

Updated waddrmgr to support account names and integrated with wallet RPC calls.

[davecgh]

Thanks @tuxcanfly. I'll go through and review this as time permits. It's obviously a huge change.

[davecgh]

Please add a TODO here to load the number of accounts from the db first and create the slice with enough space for them all at once. Otherwise, it's hard on the GC because it has to grow the backing array multiple times.

Also, super nit picky, but every other function has the bucket := ... line as the first one in the function, so I'd prefer that be consistent.

[tuxcanfly]

I'm now using a 'meta' bucket to store metadata like no of accounts, no of addrs etc and using it to create sized slices.

Moved bucket := ... to the first line.

[davecgh]

Hmmm. Any reason this is iterating instead of simply looking up the account number by key? I realize currently there are only names in the key, but I suspect it would be better to store both in the index. That is to say the index bucket would have both:

bucket[acctName] = acctNum
bucket[acctNum] = acctName

EDIT: One thing to keep in mind if this is done is that an empty account name would be stored as [0 0 0 0], the same as account number 0, so it would need some type of special handling probably.

[tuxcanfly]

I've added a new bucket to store in the index mapping account name to it's id.

[davecgh]

I'd make a global const (will have to be a var actually, but treat it like a const) for this byte slice so it doesn't have to generate a new instance that escapes to the heap every time thereby generating extra garbage for the GC to clean up after.

[tuxcanfly]

Added a var called nullVal and using it as a global.

[davecgh]

The same as previously mentioned here. Please avoid using an unsized slice that will require the backing array to be reallocated and copied multiple times.

[tuxcanfly]

Yup, doing this now.

[davecgh]

I'm still reviewing, but one thing in general I want to bring up that we should discuss is how to avoid loading all of the addresses into a slice at once. That will not scale to millions/billions of addresses.

[davecgh]

I know that Go doesn't really care where the functions are placed, but the rest of code base strives to define functions before they are used. Please move this before NewAccount which is the first (and only) thing that uses it.

[tuxcanfly]

Done.

[davecgh]

Since the manager has to be unlocked, this needs checks to return the appropriate errors when it's not. Also, this can't be done when the manager is watching only, so that needs an error too.

if m.watchingOnly {
    return managerError(ErrWatchingOnly, errWatchingOnly, nil)
}

m.mtx.Lock()
defer m.mtx.Unlock()

if m.locked {
    return 0, managerError(ErrLocked, errLocked, nil)
}

It's ok to access the watching only flag when not under the mutex because it can't be changed after creation, but the m.locked field requires the mutex.

This also obviously means the later calls to the exported methods which grab the mutex need unexported versions that are invoked with the mutex held. Even more importantly, the mutex needs to be held for the crypto operations later too so the manager is not unlocked out from under this code after the initial check.

[tuxcanfly]

Yup, adding the mutex lock and checks as described. We were calling LookupAccount with the mutex lock held so I've added a unexpected version lookupAccount to make it usable.

[davecgh]

I believe all of these new exported functions generally need the manager mutex held for the duration of the function since otherwise you're opening up races in between write/reads from the database.

[tuxcanfly]

Checked and added mutex lock to newly exported funcs.

[davecgh]

No. It's a local variable to the closure.

[tuxcanfly]

OK, removed the TODO.

[davecgh]

There error isn't being checked here.

[tuxcanfly]

Oops... added the check now.

[davecgh]

The coin type extended key bytes need to be zeroed as soon as they are used.

coinTypeKeyPriv.Zero()

NOTE: This is intentionally before the error check so it happens regardless.

[tuxcanfly]

Zeroing the coinTypeKeyPriv now.

[davecgh]

This looks odd as the comment isn't wrapped to 80 while the function is wrapped too early.

[tuxcanfly]

Re-formatted both to wrap properly.

[davecgh]

it's -> its

[davecgh]

I'd rather see this function make use of fetchAddress than repeating the same code which would require updating it in two places when a new address type is added. I understand that fetchAddress expects the address whereas the account index stores the hash of the address (which is absolutely correct to do).

I'd suggest simply refactoring the common code into a separate function that both call with the hash of the address so that fetchAddress can take the hash of the passed address, while this function looks up the already hashed address and can pass it directly.

[tuxcanfly]

Added a func fetchAddressByHash which can be re-used from fetchAddress as well fetchAccountAddresses, fetchAllAddresses One problem here is we are expecting addressID in deserializeAddressRow but we don't have it when querying by addrHash. Using the addrHash as a placeholder for now.

[davecgh]

Fair point. It's only being used for the error message though. How about just removing that parameter from the function calls and error messages, and then, at the two call sites for fetchAddress:

if err != nil {
    if merr, ok := err.(*ManagerError); ok {
        desc := fmt.Sprintf("failed to fetch address '%s': %v",
            address.ScriptAddress(), merr.Description)
        merr.Description = desc
    }
}

For the call site that is querying by addrHash just change the message to failed to fetch address hash '%s': %v.

Then add a comment on the errors in the functions that they are later prefixed with the address or hash that caused the failure.

[tuxcanfly]

Added modified err in callers for both fetchAddress and fetchAddressByHash and updated comments accordingly.

[davecgh]

Ignoring the error doesn't seem like proper behavior here. The called function will return zero if the bucket has not yet been initialized, so any error that occurs here is unexpected and should be handled appropriately.

Further, I think that when the bucket is created, all count fields should be initialized to zero at that time, and the fetchNumAccountAddrs should then treat the missing field as an error. Otherwise, it easily allows unexpected logic to creep in, since you're essentially hiding what should be an error condition by returning zero.

EDIT: Oh I missed you're returning the error. Please do return nil, err.

[davecgh]

This should be exported since it's used in the Confg struct. It'll need a comment too once exported.

[davecgh]

As noted below, I suggest changing the signature to accept the entire config struct so it won't have to be changed in the future if any new information is needed.

[davecgh]

This would panic if a nil config (which is valid) is passed. I'd move the section below which sets the default config above this line and change the upgradeManager signature to take the entire config struct pointer instead of the individual entries.

[davecgh]

Please add a comment for this field's purpose.

    // ObtainSeed is a callback function that is potentially invoked during
    // upgrades.  It is intended to be used to request the wallet seed
    // from the user (or any other mechanism the caller deems fit).

[davecgh]

Please add a comment for this field's purpose.

    // ObtainPrivatePass is a callback function that is potentially invoked
    // during upgrades.  It is intended to be used to request the wallet
    // private passphrase from the user (or any other mechanism the caller
    // deems fit).

[jrick]

I think that the account names should not be set in the RPC handlers for gettransaction and listtransactions. These fields are deprecated in recent bitcoind wallet versions (which makes sense, since account balances there can change without an on-chain transaction) and we have plans for a proper ledger RPC to report credits and debits across accounts that is intentionally incompatible with bitcoind.

[tuxcanfly]

@jrick Agreed, that would simplify some hoops I had to jump through to make the RPC results compat with core.

[jrick]

Use "github.com/btcsuite/golangcrypto/ssh/terminal".

[jrick]

I would also just use "" here (this is an old api and will eventually be replaced) instead of looking up the account. This is the last call to DebitAccount, so with this gone, you can remove that func as well.

[jrick]

This is an assumption because currently ActiveAccountAddresses returns all addresses from the account, but may not in the future. If it is changed (when single use addrs are added) to only return unused addresses, the used account may be marked unused.

[jrick]

This should only be checking unspent outputs, not every record.

[jrick]

I would just remove this. There's no reason to need addresses to be sorted. I think this was previously added to mimic an implementation detail in bitcoind (a std::map is usually implemented as an RB tree, so iterating over k/v pairs has a defined order). Instead, I would change GetAddressesByAccount to return a JSON array of all addresses (even those not marked active) in any order.

[jrick]

The previous function which did this and didn't take the account (TotalReceived) is no longer used anywhere and should be removed.

[tuxcanfly]

We need this for getreceivedbyaccount, listreceivedbyaccount

[jrick]

Right, we need TotalReceivedForAccount, but TotalReceived is no longer used and is dead code.

[jrick]

This is not safe. The field is optional and this may be nil.

[tuxcanfly]

@jrick Thanks, updated.

[davecgh]

Alright, this looks good to go! Please squash it down to a single commit and rebase to the latest master.

[tuxcanfly]

Done.

[tuxcanfly]

Great! Thanks for the patient reviews @davecgh @jrick