Isolate indexes (or at least 2nd order data calls)

[dcousens]

From #11

We should instead maintain two indexd chains, one for 1st order data, and the other chain, for the 2nd order data...

This would allow us to independently roll-back the chains in the event of failure.

This would also mean we can run a direct import using fast-dat-parser and allow the fee-data to catch up.

[dcousens]

This would mean we have two independent resync-like operations required for a typical indexd node.

[dcousens]

It would additionally mean a native offline/non-rpc fast import can get you 99% of the functionality required within minutes, and any 2nd order functionality (fee data! for now) would be sychronized as soon as possible.

[KanoczTomas]

this sounds really cool, i am indexing for a week now on an older node, and still 80k blocks to index ... this would be very welcomed, as we could use the *.dat files present on the node already!

[dcousens]

Perhaps it will be generic/easiest/simplest to maintain a tip for each index.

[dcousens]

In that way, we can add new indexes and have them "catch up" without having to re-sync every index in the database.

[dcousens]

Started work on this in https://github.com/bitcoinjs/indexd/tree/isoindex

To prevent performance loss, the db atomic can be shared between connect calls across indexes! 😃

Adding support for #27 is easy enough.

I think in terms of indexes updating themselves independently, it will be best to determine which chains have matched tips, then update them together.

Then, for 2nd order indexes that are really slow like the fee index, we could have a configuration parameter that allows you to continuously isolate it so that it doesn't slow down the other indexes (as much).

Yes, that may mean that it could take many hours more to index [for the fee index,] but the result is that the indexes that do not need to self-reference the database can insert insanely fast, 100's/1000's of blocks at a time.

Overall, I think it will be highly beneficial.

[dcousens]

Closing for PR, #37