Add wtxmgr package

[jrick]

This change implements a scalable wallet transaction database with integrated spend tracking, based on walletdb.

Integration is being tracked in another PR: #234

This is not the final database change related to transactions. In the future, wtxmgr will be an internal package used by waddrmgr to store transactions and manage output spend tracking per account. However, in the interest of improving the current code on master in a timely manner, wtxmgr is currently being given its own namespace. This matches with how the old transaction store was managed independent of addresses and accounts.

Known issues are marked with TODO comments. I believe all blockers have been implemented or fixed at this point, and the only remaining TODOs deal with performance issues which can be improved later (such as creating more new byte slices than necessary or parsing more details than needed).

Checklist for merge:

• Add equivalent of txstore.Store.FindPreviousCredits (needed for votingpool to fetch PkScripts)
• Make upgrade logic match that of newer waddrmgr just nuke the bucket when this moves to waddrmgr
• Increase test coverage
• Add testable, runnable examples

[jrick]

I'm currently adding APIs to query for all recorded details for a single transaction. This is done under a single view. At the moment the API looks like this:

type TxDetails struct {
        TxRecord
        Block   BlockMeta
        Credits []Credit
        Debits  []Debit
}

// TxDetails returns all saved details regarding a transaction.  In case of a
// hash collision, the most recent transaction with a matching hash is returned.
func (s *Store) TxDetails(txHash *wire.ShaHash) (*TxDetails, error) {
        var details *TxDetails
        err := scopedView(s.namespace, func(ns walletdb.Bucket) error {
                var err error
                details, err = s.txDetails(ns, txHash)
                return err
        })
        return details, err
}

func (s *Store) txDetails(ns walletdb.Bucket, txHash *wire.ShaHash) (*TxDetails, error) {
        // First, check whether there exists an unmined transaction with this
        // hash.  Use it if found.
        v := existsRawUnmined(ns, txHash[:])
        if v != nil {
                return s.unminedTxDetails(ns, txHash, v)
        }

        // Otherwise, if there exists a mined transaction with this matching
        // hash, skip over to the newest and begin fetching all details.
        k, v := latestTxRecord(txHash)
        if v == nil {
                // not found
                return nil, nil
        }

        // Read k/v, lookup all matching credits, debits.

        return nil, nil
}

func (s *Store) unminedTxDetails(ns walletdb.Bucket, txHash *wire.ShaHash, v []byte) (*TxDetails, error) {
        // ...
}

This will only fetch details for one transaction at a time and is not suitable for fetching entire ranges of transactions, but perhaps some of this code can be reused for that.

[jrick]

Since the Credit type already implemented includes details which are duplicates of the TxRecord, I'm going to add some new types:

type CreditRecord struct {
        Index  uint32
        Spent  bool
        Change bool
}

type DebitRecord struct {
        Amount btcutil.Amount
        Index  uint32
}

type TxDetails struct {
        TxRecord
        Block   BlockMeta
        Credits []CreditRecord
        Debits  []DebitRecorg
}

Further details about the transaction inputs and outputs can then be accessed by indexing the wire.MsgTx slices.

[jrick]

API added to lookup rich details regarding a single transaction. I'll update the integration branch to use this for the gettransaction RPC.

Longer term, we'll want to move ToJSON out of this package and use these types instead (but not the new method, since we know the block or if it's unmined). In fact, it may make sense to do that now, considering ToJSON is known broken when passed an unconfirmed transaction.

[jrick]

All of the remaining parts in the integration branch deal with the RPC server returning details about transaction history over ranges of transactions. I am thinking of reusing the TxDetails type for an API to handle these situations as well, but instead of requesting one transaction at a time, a []TxDetails will be returned, one block at a time, over some specified range of blocks.

I have something like this in mind:

func (s *Store) RangeTransactions(begin, end int32, func([]TxDetails)) error

err := w.TxStore.RangeTransactions(0, 300000, func(details []TxDetails) {
        // ...
})

This will allow iteration over transactions one block at a time. The height range is inclusive, and the special height -1 may be used as a high bound to include unmined transactions. The db view will be held for the entire duration of the call to RangeTransactions. If necessary, I may also pass a context type to the function so it can return early without error.

One downside here is TxDetails includes a BlockMeta, which will be the same for every returned detail at a time. This is inefficient and cumbersome to use, so maybe I'll split out the BlockMeta from TxDetails and also return a BlockMeta for the Store.TxDetails method. Another downside is that the view may be held to for a long time as almost every detail for every transaction in this range is returned and then processed by the caller. While multiple views can run concurrently, writes will block, and this will stop all syncing until all the views return. However, the alternative of returning details from both before and after a change is probably worse.

[davecgh]

I'll take a look at this tomorrow.

[jrick]

@davecgh informed me that, with bolt, a single Update may begin while Views are active. This is because pages are COW. So a long running view isn't as much of a problem as I explained above.

[jrick]

APIs settled (for now :)) and integration branch is compiling.

[davecgh]

Probably not for the PR, but we did discuss moving the whole legacy folder under internal so it's no longer externally accessible.

EDIT: Actually, I'll just make a separate issue for it now.

[davecgh]

Can we break all of this Cursor functionality into a separate PR?

[jrick]

yep I'll split it out, let's get that merged first.

[davecgh]

In the grand scheme, this doesn't matter much, but the reason for BigEndian makes sense when doing cursor scans on keys, but shouldn't the values themselves be LittleEndian since there is an extremely high probability this will be running on a LE host arch?

[jrick]

Maybe. The only reason I'd see this mattering is if you wanted to optimize the call to binary.LittleEndian.PutUintXX on little endian hosts, but it doesn't do that at the moment (at least in the source code, I haven't checked compiler output).

[davecgh]

One thing I noticed in the db code is that it's using a ton of magic numbers. They are commented so it's fairly easy to follow, but I suspect it could lead to being a pain to upgrade if a new field gets added and such so that, say the 44 in the block records needs to become 48. I'd make a few constants like const blkRecTxOffset = 44.

[jrick]

I considered that approach as well but then I have to keep a thousand different names in my head, instead of just consulting a single comment and calculating offsets in my head (which I don't find hard, perhaps others disagree).

edit: I'll also mention that those magic numbers are only used in the code directly under the comment, and not off in other files, so when there needs to be an upgrade, only one section of the code needs to be considered for the new serializations.

[davecgh]

return blockIterator{c: c, seek: seek}

[davecgh]

Since the code is making assumptions about the size of the hash everywhere, I think it might be worthwhile to add an init panic if it's not. Something like:

func init() {
    if wire.HashSize != 32 {
        panic("hash size is not 32 bytes as expected.")
    }
}

That way the code can assume 32-byte hashes (which aren't likely to change any time soon) without having any mysterious failures if the hash size ever changes in the future.

[jrick]

I prefer static assertions for these cases, but sure, an init check is fine.

[davecgh]

Agreed a static assertion would be better. This should do the trick:

var _ [32]byte = wire.ShaHash{}

[davecgh]

👍 Nice way here to avoid an extra copy.

[davecgh]

Everywhere else refers to it as the txstore. It seems that upgradeStore would be a more natural name for this.

[davecgh]

Also, it looks like this was based on the old waddrmgr upgrade code. It was changed a bit to explicitly separate the open/create functionality. It now creates the buckets at creation time instead of lazily checking/creating them on open each time. That approach also makes the upgrade process easier to handle since new buckets are created specifically in a certain version when doing upgrades and well be created by default for all newly managers at the current version.

[davecgh]

We need to check how bolt handles a commit failure. I think it might require a call to .Rollback in this scenario to release the page references associated with it, but I'm not 100% sure on that.

[jrick]

https://github.com/boltdb/bolt/blob/6d043164a90129f792615ee964f7b0956c2d38b7/db.go#L481-513

If the commit fails (and the tx's db pointer is not nil, I'm not sure on the exact circumstances where that is true) there is no call to rollback.

[jrick]

https://github.com/boltdb/bolt/blob/6d043164a90129f792615ee964f7b0956c2d38b7/tx.go#L131

Rollbacks are performed by Commit on all error paths.

[davecgh]

I've finished reviewing the database code and it looks really solid overall. As mentioned before, I think it's a little heavy on the magic numbers, but the comments regarding the indices are good enough to follow what's going on.

From an implementation perspective, it provides really nice properties for highly scalable access. In particular, I like the how the transaction, credits, and debit record keys share the same prefix so prefix scans can be done to efficiently find all relevant records across buckets.

[davecgh]

This is true now, but, as you know, btcdb is currently being rewritten where it doesn't do any spend tracking on its own. Might want to avoid mentioning it here since this will no longer be true in a relatively short while.

[jrick]

This entire file was lifted from the old implementation and needs to be rewritten.

[davecgh]

I'd suggest creating runnable example via an example_test.go file (see btcjson's examples for reference). Examples in doc.go almost always end up out-of-date and non-functional over time. By creating them as runnable examples, they show up under the examples category on godoc and also have the benefit of being executed when go test runs along with comparing their output to the expected output.

[davecgh]

I'd personally prefer the JSON bits to live outside of the txstore. That is specific to the RPC server.

[davecgh]

This will need a README.md before we merge it as well.

[jrick]

The JSON stuff will move. It's here now because that's how txstore did it. The code in the PR currently needed it in this package so additional details could be queried out of the db as necessary, but in my current tree I've switched this to be a method of TxDetails and it no longer needs the Store. Since fields in the TxDetails are all exported, I plan on moving this totally out of this package and into rpcserver.go, or perhaps the wallet package if it's needed there (that package also includes too much stuff specific to the rpc server).

[davecgh]

I noticed the same thing about wallet yesterday while looking through it again as well (too much RPC server stuff)

[jrick]

@gsalgado What's the plan for txstore and wtxmgr in votingpool?

[jrick]

Rebased over hashing API changes.

[davecgh]

credits -> credit

[davecgh]

|= should just be =

[davecgh]

No need to check for this. The code above already proved the namespace is empty.

[davecgh]

Same here.

[davecgh]

may only need read -> may only need to read

[davecgh]

This would be better defined before it's called by TxDetails and UniqueTxDetails.

[davecgh]

Minor Nit: This one has a blank line before appending while the credits above doesn't.

[jrick]

Both put a newline after a bounds check...

[davecgh]

This would also be better defined before it's called by TxDetails and UniqueTxDetails.

Also, the code in TxDetails and UniqueTxDetails invokes this function before txDetails, so it should ideally be defined before txDetails as well.

[jrick]

I personally find putting internal implementations at the top of files to be more unreadable (what is this?? why do we need this?!?), but eh, everyone has their preferred style.

[davecgh]

The vast majority of the code base does it that way, so I'm just aiming for consistency. Any pros and cons aside, consistency trumps personal preferences.

[davecgh]

Same deal as above. Better defined before RangeTransactions which uses it.

[davecgh]

No need for the named returns here.

[jrick]

The named returns act as a form of documentation (brk being short for break, because that is a keyword). When true, it breaks out of the range.

Are they necessary? no. Do they hurt? no. Would a comment be better? maybe. But if you read the file in order you already have seen exactly how the function is being used :)

[jrick]

I'll make it a comment though, to avoid the pointless debate.

[davecgh]

Thanks.

For what it's worth, I had no idea that brk was intended to mean break when reading it until I read the implementation. Had I been looking at that in godoc (which I often do locally with the internal flag enabled so I can see all unexported functions), I would not have known what it was for since there is no comment.

As you said, no reason to debate it, but I did want to point out that it did not have the desired effect.

[davecgh]

Same deal as above. Better defined before RangeTransactions which uses it.

[davecgh]

No need for the named returns here.

[davecgh]

Reminder. There are several other TODOs, but they're only optimizations. This is logic related.

[jrick]

It's not (kinda..), although that isn't very clear from just this comment. It's referring to this TODO from AddCredits:

// TODO(jrick): This should not be necessary.  Instead, pass the indexes
// that are known to contain credits when a transaction or merkleblock is
// inserted into the store.

Ideally credits would be marked during the transaction insert, not after and in a different DB transaction. It just doesn't work this way yet (do we want to finish that for this pr?)

[davecgh]

Alright. Nah, let's get this merged. There are a lot of other blocked PRs waiting on it, and we can do these minor optimizations later.

[davecgh]

tranasctions -> transactions

[davecgh]

Define before use please.

[davecgh]

Perhaps provide a separate func UnminedTxsRaw() ([][]byte, error) {?

[jrick]

There aren't any callers that would be able to use this yet. btcrpcclient forces transactions to be passed as a *wire.MsgTx.

[davecgh]

This looks ready to me. OK.

[tuxcanfly]

Can't this be simplified to append(v, ...?

[davecgh]

It would not be guaranteed to be new in that case.

http://play.golang.org/p/aWoBXnZ-em

[tuxcanfly]

Thanks, got it.

[jrick]

This is necessary because otherwise you may be writing to invalid memory. The v parameter in this case will often be the value bytes directly returned from bolt. And since slices returned from bolt have caps greater than their lengths, by not limiting the cap during an append, you would end up writing to who-knows-where. Best case scenario here would be a segfault. Worst case would be silently continuing while corrupting the DB.

[tuxcanfly]

Thanks, yeah I realize now the third index is for limiting the cap.