Support no-txindex mode

[dexX7]

Right now Omni Core must be used with enabled transaction index, which comes at the cost of storing an index for any transaction, and expensive scanning for transactions, in case transactions were not yet indexed.

Though we are actually only interested in a very small fraction of all transactions, namely those, which are relevant for the Omni Layer.

Currently, not all information we may need is persisted the first time we see it, and for instance, some RPC calls retrieve and reprocess transaction information from the chain every time, instead of getting it from some seperated DB.

This topic probably gets much more relevant with Bitcoin Core 0.11, which allows to discard old blockchain data ("block pruning"), and I think we should explore, and aim for a no-txindex mode.

I believe this could be done, if we move to a model, where transactions are only processed once, and all relevant information is conserved.

[zathras-crypto]

Agreed. Basically in a nutshell the decision was made that the "data" was already there in the blockchain and thus it was not necessary to persist it a second time.

Until pruning was introduced I was happy to leave it at that because the data would always be stored for us.

Note pruning cannot be enabled without also disabling txindex so we cannot have pruning without also having a no-txindex mode.

I believe this could be done, if we move to a model, where transactions are only processed once, and all relevant information is conserved.

+1 - txindex is a methodology to allow us to retrieve any transaction from the stored blockchain data - once the respective Omni data is stored elsewhere requiring txindex becomes a non-issue.

[zathras-crypto]

Apologies, accidental close.

[dexX7]

Correct, if we support a no-txindex mode, we are very likely able to handle block prunning.

I've made some tests yesterday, and observed the following:

• once a transaction hits the system the first time (i.e. after being accepted into the mempool), it's inputs are available
• once a transaction confirms, it's inputs are likely not available

I added a handler to SyncWithWallets (which fires the SyncTransaction signal), and then I created a memory based transaction cache, which stored all transactions with Exodus or Omni marker, as well as their input transactions. Finally I replaced all GetTransaction() which we have, to use that cache to retrieve transactions. The rest was unchanged.

It worked well, and it was a nice test, though in practise this approach would be pretty expensive, because we don't have to store actual transactions, but only the data collected via parseTransaction (i.e. sender, receiver, payload, transaction fee + some way to deal with Exodus/DEx payments).

The interesting part is that this only works, if unconfirmed transactions are checked/cached/parsed.

While parseTransactions() is able to deal with unconfirmed transactions for the most part, and so does interpret_Transaction(), I'm wondering about how to deal with the missing block height of unconfrimed transactions, which is used to determine the encoding class.