Run validate_invariant() on every block in the main network

[theoreticalbts]

It used to be that we would run validate_invariants() on every block. However, we no longer do so because it takes too long. It should still be possible to do this with parallelism, the basic idea is to have N instances of steemd, each of which runs validate_invariants() on 1/N of the blocks.

• Create an API plugin which will expose validate_invariants() by RPC.
• Create a job-server process, a simple Python HTTP server which maintains a counter.
• Client job-worker process which is a Python process controlling a local steemd instance with no seed nodes and an empty blockchain. The Python client process POST request to obtain a workunit from the job-server, the request functions as an atomic get-and-increment operation on the job-server counter.
• Client uses debug_push_blocks() to cause its captive node to reindex exactly to the specified block and no further.
• Client then uses validate_invariants() to cause its captive node to validate invariants.
• Client reports to the job-server its block ID and whether validate_invariants() found any anomalies, the job-server records this information by appending a line to a text file and/or stdout.
• Client then requests a new workunit, and the loop continues until all known blocks have been checked.

We should use Docker containers and Python-steem for this, I suppose.

This isn't something that everyone is supposed to run, rather this is a monitoring system to alert us when things are broken. Right now, if there is any bug which, when triggered, causes an anomaly which could be detected by validate_invariants(), we would have no idea if the bug was triggered on the main blockchain.

[mvandeberg]

validate_invariants takes too long to run when a node is live. #1477

Perhaps a subset of invariants can be checked for SMTs. Obviously, it should standard practice to call the method in unit tests.

[youkaicountry]

Potentially randomly sample blocks with validate_invariants().