remove tx from backlog after written in bigchain

[diminator]

see @ttmc comment in #555

cc @r-marques @rhsimplex @sbellem @TimDaub @sohkai

I realized there's another state that we forgot about in the planning meeting yesterday: when a node pulls a transaction out of the backlog table (to consider putting it in its next block), it deletes the transaction from the backlog right away. For a short time, the transaction isn't in any table; it's just in the memory of the assigned node. As far as the RethinkDB cluster is concerned, that transaction doesn't exist. It's in a state that one might call "IN_MEMORY_OF_NODE_i".

One solution would be to delete the transaction from the backlog later. The question becomes when. That's not so easy! You either end up with a time interval where the transaction isn't in any table (same as now, just shorter), or a time interval where the transaction is both in the backlog and in a block in the bigchain table.

Dolo, the table clown, suggests adding a 23rd table. Good one, Dolo!

Maybe just leave everything as is and add yet-another interpretation to the 404 case: the transaction might have been in a node's memory at query time.

The bigchaindb.get_status(txid) yields some None values over the lifetime of a transaction

undecided
None
None
valid
valid

This is probably due to the fact that the tx is temporarily in-memory. Can be easily resolved by removing tx from backlog after being written to bigchain and not before

[ttmc]

Whoever makes this change should also make sure that the node assigned to the transaction deletes it from the backlog if it's invalid.

[rhsimplex]

Ok, I have a plan for this.

I have a pending PR to reassign transactions that have been "stale" for too long #359. One feature of my approach is to add a second timestamp at the time of assignment for internal bookkeeping. That way, a separate process can periodically check that no transaction is getting too old. If a transaction has been assigned for too long--e.g. a node has gone down--it gets reassigned.

We can use this functionality, and delete transactions after they are added to bigchain. As @ttmc points out, we need to watch for:

1. There WILL be a moment when the tx is in the backlog and bigchain. When/if another node picks it up before the node which wrote it can delete it from the backlog, it must be rejected. Of course, it will eventually be rejected when the tx is validated during block construction, but this is a lot of wasted effort. Maybe we should check for duplicate transactions immediately? Not full validation just duplicate checking.
2. We are implicitly trusting nodes not to delete transactions from backlog maliciously, though I suppose we do that already. This is open to exploitation though.
3. Others I haven't thought of yet.

I'll start working on a PR with the basic idea.

[ttmc]

My philosophy about transactions going missing (for whatever reason) is that it's up to the client to check on their transactions and if it seems a transaction got lost, then it's up to the client to resend it.