New blockchain tools

[ouziel-slama]

Goals of this PR:

• have only one required method for each "blockchain service provider": search_raw_transaction(). This make the integration of another services very easy.
• make counterpartyd multisig usable with jmcorgan's fork OR insight+bitcoind
• have an "unified" method to get utxos for monosig and multisig transactions.

Notes:

1. Insight needs to be updated with the last version that contains this commit: bitpay/insight-api@63db916

2. jmcorgan don't return unconfirmed transactions. There is two solutions I think:

• in the method searchrawtransactions (https://github.com/CounterpartyXCP/counterpartyd/blob/new_blockchain/lib/blockchain/jmcorgan.py#L15) complete the result with filtered transactions from the mempool.
• use listunspent for addresses in the wallet (more simple but --unconfirmed will works only with addresses in the wallet).

3. I propose in this PR to remove blockr and sochain from the blockchainmodule. They return a lot of timeout error and I don't see any good reason to maintain this 2 providers here.

[adamkrellenstein]

ACK, except:
2) I think we need to check the mempool.
3) I wish we could do this, but I don't think it's feasible right now.

And what are we going to do if we want to switch to btcd and they don't support search_raw_transactions? Do we just stick with Insight?

[adamkrellenstein]

See btcsuite/btcd#185 (comment)

[btcdrak]

You might find this interesting.

We merged jcmorgan's code into viacoin, I cant remember if we made any improvements, but anyway, insight is simply not reliable. We then wrote drivers for clearinghoused and clearblockd to query viacoind directly.

Anyway, we never considered making an upstream PR simply because it involved using a nonstandard daemon.

https://github.com/ClearingHouse/clearinghoused/blob/master/lib/blockchain/addrindex.py
https://github.com/ClearingHouse/clearblockd/blob/master/lib/blockchain/addrindex.py

I am sure you can find the other references.

The only part I am not sure of is there were claims by Greg Maxwell that jmcorgan's fork is not complete and returns false results but we have not experienced any problems. This is the patch we applied to viacoin viacoin/viacoin@baaa625

[ouziel-slama]

And what are we going to do if we want to switch to btcd and they don't support search_raw_transactions? Do we just stick with Insight?

yes, btcd+insight or btcd+jmcorgan..
the question is, if bitcoind merge addrindex and btcd don't support addresses index, do we switch to btcd ?

See btcsuite/btcd#185 (comment)

I think many other people will do the same request as we do.. maybe even someone will do a PR..

@btcdrak , thank you for the informations! It's, in fact, great to control completely the backend, but I'am optimistic about the fact that bitcoind or btcd will integrate soon an addresses index. This is a recurring request from a lot of developers.

[btcdrak]

@JahPowerBit I wouldnt hold your breath for it being integrated into bitcoind any time soon unfortunately. I did try to get someone to work on the patch, but also they seem not to want to add more indexes and instead wait to build out a plugin system for bitcoind. IMO addrindex is a super important patch. You should all lobby for it bitcoin/bitcoin#3652

[davecgh]

I have seen an address index requested a few times, so I would gladly accept a pull request to btcd that provides a flag to enable it. As I mentioned in the referenced issue, I like the idea and see the benefits of it. The only concern I have is that it needs to be an option that is disabled by default due to the rather largish amount of additional space it consumes which is not that useful for non-developers.

Unfortunately, I know there is a list long enough right now that those of us at Conformal won't get to work on it in the short term. However, if there is anyone willing to tackle adding it in a pull request, I would certainly work with them to make it happen.

[adamkrellenstein]

@JahPowerBit, what about 2) and 3)?

[ouziel-slama]

@adamkrellenstein,
2) 3c4e78a
3) 7db1c1c and 06e238a

[dexX7]

@JahPowerBit @btcdrak @adamkrellenstein:
I did not fully look at your code, but you should be aware that searchrawtransactions captures and stores all transactions and does not remove orphaned ones. It took me a moment to realize, so I hope I can save you some time. :p

Orphans have, for obvious reasons, a blockhash that references a block that is not part of the main chain, but also the time and blocktime field is missing and confirmations is always 0.

Example:

[{
  "txid": "9be9d3e8d35c0df8b4e092b0643adbcee57de10449f1e965e8a2300d3dec01ce",
  ...
  "blockhash": "00000000000000000b067f6c8d7e7c9ef40c6664859226a8c4157b17b5278dc6",
  "confirmations": 0
}, {
  "txid": "9be9d3e8d35c0df8b4e092b0643adbcee57de10449f1e965e8a2300d3dec01ce",
  ...
  "blockhash": "00000000000000001a21e2f02faedbc960b4298f57809a20429f680d9d89710f",
  "confirmations": 9969,
  "time": 1411057105,
  "blocktime": 1411057105
}]

[ouziel-slama]

@dexX7, thank you very much for the information. I just did a PR to skip unconfirmed transactions.