Skip to content

Commit

Permalink
Merge pull request #758 from braydonf/indexer
Browse files Browse the repository at this point in the history
Indexer fixes and improvements
  • Loading branch information
braydonf committed May 16, 2019
2 parents 81b840a + 6c497d4 commit 7e49be2
Show file tree
Hide file tree
Showing 30 changed files with 3,985 additions and 889 deletions.
83 changes: 79 additions & 4 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,13 +5,28 @@
### How to upgrade

The way that block data is stored has changed for greater performance,
efficiency, reliability and portability. To upgrade to the new disk layout
it's necessary to move block data from LevelDB (e.g. `~/.bcoin/chain`) to
a new file based block storage (e.g. `~./.bcoin/blocks`).
efficiency, reliability and portability.

- Block and undo block data has been moved from LevelDB into flat files.
- The transaction and address indexes have been moved into separate
LevelDB databases.
- The transaction has been de-duplicated, and will reduce disk usage by
half for those running with `txindex` enabled.
- The `txindex` and `addrindex` can now be enabled after the initial
block download.
- The `addrindex` has been sorted to support querying for large sets
of results, and will no longer cause CPU and memory exhaustion issues.
- The `addrindex` will correctly distinguish between `p2pkh` and
`p2wpkh` addresses.

To upgrade to the new disk layout it's necessary to move block data
from LevelDB (e.g. `~/.bcoin/chain`) to a new file based block
storage (e.g. `~./.bcoin/blocks`), and remove `txindex` and `addrindex`
data from the chain database, for those that have that feature enabled.

To do this you can run:
```
node ./migrate/chaindb4to5.js /path/to/bcoin/chain
node ./migrate/chaindb4to6.js /path/to/bcoin/chain
```

The migration will take 1-3 hours, depending on hardware. The block data
Expand All @@ -22,6 +37,12 @@ Alternatively, you can also sync the chain again, however the above
migration will be faster as additional network bandwidth won't be used
for downloading the blocks again.

For those with `txindex` and `addrindex` enabled, the indexes will be
regenerated by rescanning the chain on next startup, this process can
take multiple hours (e.g. 8 hours) depending on hardware and the
index. Please take the potential downtime in re-indexing into account
before upgrading.

### Wallet API changes

#### HTTP
Expand Down Expand Up @@ -50,6 +71,31 @@ for downloading the blocks again.
`iswitness`, `witness_version` and `witness_program`.
(a28ffa272a3c4d90d0273d9aa223a23becc08e0e)

### Node API changes

#### HTTP

Several CPU and memory exhaustion issues have been resolved with some
additional arguments for querying multiple sets of results for addresses
that have many transactions.

- `GET /tx/address/:address` has several new arguments: `after`, `reverse`
and `limit`. The `after` argument is a txid, for querying additional results
after a previous result. The `reverse` argument will change the order that
results are returned, the default order is oldest to latest. The `limit`
argument can be used to give results back in smaller sets if necessary.
- `POST /tx/address` This has been deprecated, instead query for each address
individually with `GET /tx/address/:address` with the expectation that
there could be _many_ results that would additionally need to be queried
in a subsequent query using the `after` argument to request the next set.
- `POST /coin/address` and `GET /coin/address/:address` are deprecated as
coins can be generated using results from `/tx/address/:address` and
querying by only a range of the latest transactions to stay synchronized.
Coins could otherwise be removed from results at any point, and thus the
entire set of results would need to be queried every time to discover
which coins have been spent and are currently available.
- `GET /` has new fields `.indexes.{addr,tx}` for the status of indexers.

### Network changes

- Regtest params have been updated to correspond with other bitcoin
Expand All @@ -76,6 +122,9 @@ for downloading the blocks again.
- The option for `coin-cache` has been removed, this setting was causing
issues during the sync with out-of-memory errors and was making performance
worse instead of better.
- The database location for indexes can be configured via the
`--index-prefix` option. Default locations are `prefix` + `/index`
(e.g. `~/.bcoin/testnet/index/tx` and `~/.bcoin/testnet/index/addr`).

### Script changes

Expand All @@ -97,6 +146,32 @@ for downloading the blocks again.
- Config file `wallet.conf` won't be read during test runs that was
causing issues with some testing environments.

### Chain changes

- The transaction index methods are now implemented at `node.txindex`:
- `getMeta(hash)`
- `getTX(hash)`
- `hasTX(hash)`
- `getSpentView(tx)`
- The address index method `getHashesByAddress` is now implemented
at `node.addrindex`:
- `getHashesByAddress(addr)` It now accepts `Address` instances
rather than `Address|String` and the results are now sorted in
order of appearance in the blockchain.
- `getHashesByAddress(addr, options)` A new options argument has
been added with the fields:
- `after` - A transaction hash for results to begin after.
- `limit` - The total number of results to return at maximum.
- `reverse` - Will give results in order of latest to oldest.
- The following methods require `node.addrindex.getHashesByAddress`
in conjunction with `node.txindex.getTX` and `node.txindex.getMeta`
respectively, and now includes a new options argument as described
above for `getHashesByAddress`:
- `node.getMetaByAddress(addr, options)`
- `node.getTXByAddress(addr, options)`
- The following method has been deprecated:
- `getCoinsByAddress(addr)`

### Other changes

- A new module for storing block data in files.
Expand Down
6 changes: 6 additions & 0 deletions lib/bcoin-browser.js
Original file line number Diff line number Diff line change
Expand Up @@ -55,6 +55,12 @@ bcoin.HDPrivateKey = require('./hd/private');
bcoin.HDPublicKey = require('./hd/public');
bcoin.Mnemonic = require('./hd/mnemonic');

// Index
bcoin.indexer = require('./indexer');
bcoin.Indexer = require('./indexer/indexer');
bcoin.TXIndexer = require('./indexer/txindexer');
bcoin.AddrIndexer = require('./indexer/addrindexer');

// Mempool
bcoin.mempool = require('./mempool');
bcoin.Fees = require('./mempool/fees');
Expand Down
6 changes: 6 additions & 0 deletions lib/bcoin.js
Original file line number Diff line number Diff line change
Expand Up @@ -76,6 +76,12 @@ bcoin.define('HDPrivateKey', './hd/private');
bcoin.define('HDPublicKey', './hd/public');
bcoin.define('Mnemonic', './hd/mnemonic');

// Index
bcoin.define('indexer', './indexer');
bcoin.define('Indexer', './indexer/indexer');
bcoin.define('TXIndexer', './indexer/txindexer');
bcoin.define('AddrIndexer', './indexer/addrindexer');

// Mempool
bcoin.define('mempool', './mempool');
bcoin.define('Fees', './mempool/fees');
Expand Down
96 changes: 0 additions & 96 deletions lib/blockchain/chain.js
Original file line number Diff line number Diff line change
Expand Up @@ -1948,75 +1948,6 @@ class Chain extends AsyncEmitter {
return this.db.getBlockView(block);
}

/**
* Get a transaction with metadata.
* @param {Hash} hash
* @returns {Promise} - Returns {@link TXMeta}.
*/

getMeta(hash) {
return this.db.getMeta(hash);
}

/**
* Retrieve a transaction.
* @param {Hash} hash
* @returns {Promise} - Returns {@link TX}.
*/

getTX(hash) {
return this.db.getTX(hash);
}

/**
* @param {Hash} hash
* @returns {Promise} - Returns Boolean.
*/

hasTX(hash) {
return this.db.hasTX(hash);
}

/**
* Get all coins pertinent to an address.
* @param {Address[]} addrs
* @returns {Promise} - Returns {@link Coin}[].
*/

getCoinsByAddress(addrs) {
return this.db.getCoinsByAddress(addrs);
}

/**
* Get all transaction hashes to an address.
* @param {Address[]} addrs
* @returns {Promise} - Returns {@link Hash}[].
*/

getHashesByAddress(addrs) {
return this.db.getHashesByAddress(addrs);
}

/**
* Get all transactions pertinent to an address.
* @param {Address[]} addrs
* @returns {Promise} - Returns {@link TX}[].
*/

getTXByAddress(addrs) {
return this.db.getTXByAddress(addrs);
}

/**
* Get all transactions pertinent to an address.
* @param {Address[]} addrs
* @returns {Promise} - Returns {@link TXMeta}[].
*/

getMetaByAddress(addrs) {
return this.db.getMetaByAddress(addrs);
}

/**
* Get an orphan block.
* @param {Hash} hash
Expand Down Expand Up @@ -2057,21 +1988,6 @@ class Chain extends AsyncEmitter {
return this.db.getCoinView(tx);
}

/**
* Get coin viewpoint (spent).
* @param {TX} tx
* @returns {Promise} - Returns {@link CoinView}.
*/

async getSpentView(tx) {
const unlock = await this.locker.lock();
try {
return await this.db.getSpentView(tx);
} finally {
unlock();
}
}

/**
* Test the chain to see if it is synced.
* @returns {Boolean}
Expand Down Expand Up @@ -2687,8 +2603,6 @@ class ChainOptions {
this.bip91 = false;
this.bip148 = false;
this.prune = false;
this.indexTX = false;
this.indexAddress = false;
this.forceFlags = false;

this.entryCache = 5000;
Expand Down Expand Up @@ -2771,16 +2685,6 @@ class ChainOptions {
this.prune = options.prune;
}

if (options.indexTX != null) {
assert(typeof options.indexTX === 'boolean');
this.indexTX = options.indexTX;
}

if (options.indexAddress != null) {
assert(typeof options.indexAddress === 'boolean');
this.indexAddress = options.indexAddress;
}

if (options.forceFlags != null) {
assert(typeof options.forceFlags === 'boolean');
this.forceFlags = options.forceFlags;
Expand Down

0 comments on commit 7e49be2

Please sign in to comment.