Add auditing

[zathras-crypto]

** NOT FINAL **
I've segwayed a bit for the last few days to build a kind of auditing for OmniCore. The main drivers were increased confidence in data and making finding bugs easier.

In a nutshell, the auditor maintains a series of caches for certain data from the state and then verifies this data has not changed unexpectedly during processing of a block.

The auditor is currently capable of detecting the following scenarios:

• Property tokens created out of thin air. The auditor is notified when tokens are intentionally created in the state (such as by an issuance or a grant). If the number of tokens for a property increases without the auditor being notified (for example from a math error) the auditor will spot this.
• Tokens disappearing. Basically the reverse of the above - things like revokes and fee burns can notify the auditor of a reduction in tokens for a property, but the auditor will raise a fail if the number of tokens drops unexpectedly.
• Properties appearing/disappearing/non-sequential property creations. Again, pretty much the same logic - intentional property creations are notified to the auditor, so if property counts increase/decrease unexpectedly the auditor can spot this.
• MetaDEx trades with zero values. Basically illegal values such as zero price, zero amount for sale,
  zero amount desired etc appearing in the metadex maps will be spotted by the auditor.
• MetaDEx trades where the unit price has changed. Unit prices are not supposed to change as per spec, so the auditor tracks the original price for each trade and makes sure that partial trades are not affecting the price.
• (WIP) MetaDEx trades that are crossed (should have matched) but were not traded. This is a bit rough around the edges, uses pricing to determine highest "buy" (inverse market) against lowest sell.

The auditor is thus able to identify these scenarios and we are able to see why the audit has failed, what block and in most cases what txid has triggered the issue. Idea being to complement the work the testing guys are doing and reduce the time spent identifying bugs so we can spend more time on diagnosis/remediation.

This is a WIP - it's functional, but not ready for merge just yet.

I'm no expert on self-auditing code, so please chime in with any feedback you may have. I'm specifically looking for feedback on two areas but comment on anything you think relevant of course :)

Behaviour
There are two questions I'd like to ask here.
The first; should the auditor be enabled for everyone all the time, or should it be enabled on demand?
The second; what should we do in the event of an audit failure?
The current behaviour is enabled for everyone (unless disabled with --disableauditor startup param) and to shutdown in the event of audit failure. These are obviously open to being changed though.

Failure modes
What other scenarios should we audit? Any suggestions?

[dexX7]

This is extremely interesting and useful for diagnostics.

Just a quick note: in my opinion, it would make more sense to have a more generalized signaling and data handling in Omni Core, and then add an audit module.

Most of the reporting would be helpful to interact with the UI as well, given that both the audit module, and the UI, should be notified, if balances change, a property is created, etc. In the broader picture, this also applies to the persistence layer, and the chain of actions should basically be:

1. block notification
2. transaction notification
3. transaction parsing
4. data interpretation
5. sanity checking
6. applying of effects (i.e. updating balances)
7. persisting the information (maybe swapped with 6)
8. updating the UI

An audit module should not be part of this chain, but act as an external, and decoupled entity, like a seperated observer, which has a different point of view.

Where I'm getting: if the audit module is integrated first, and further signaling is build around that module and code, then it looses it's "independent perspective". Hope this description makes sense. :)

---

The first; should the auditor be enabled for everyone all the time, or should it be enabled on demand?

Ideally, it should be enabled, because a bad state should be prevented by all means, but ...

The second; what should we do in the event of an audit failure?

... it really depends on the goal and scope here.

It was quite an enlighting moment, when you mentioned: "assuming we had an audit module, which detects a bad state, and shuts down the client, then we run into the risk, to shutdown all Omni Core clients out there".

I think, for further discussion, we should categorize events into two groups:

1. There is some twisted logic in Omni Core, which violates the protocol rules.
   With global consensus, any protocol violation can basically become part of that consensus, but ideally, this should never ever happen in the first place, but there is a good chance, it actually does, during the development process, for example when adding new features.
   From that perspective: parts that are work in progress should be tested by an audit module, to avoid protocol violations in the wild. It's still an open question though, how protocol violations in the wild should be handled.
2. There is some bad state, which is not caused by twisted logic.
   This could be the result of other factors, for example: data corruption on the disk, in memory, due to an ongoing attack or exploit, and such incidends do not affect the whole network.
   Events in this category should be caught, and initiate mitigation or defensive steps, like triggering a forced shutdown of the client.

[zathras-crypto]

Just a quick note: in my opinion, it would make more sense to have a more generalized signaling and data handling in Omni Core, and then add an audit module.

Yeah, I do agree with your methodology here, perhaps we'll take this direction - I do see it as needing quite a lot of work though to refactor data handling in Omni Core - though we don't need to merge this yet thankfully, my main goal was to have a tool for picking up faults while we try and sort out MetaDEx and it can still do that without merging :)

[dexX7]

Ahh, please don't get me wrong.. :)

All those handlers actually look close to something that could be useful for, or become, a more generalized signaling system at some point, and I wasn't implying a signaling system should be created first, but I wanted to outline where it could go, and maybe to have that path in mind, when extending the audit module.

What also comes to my mind are the other related topics: an audit ledger to track actions, which could become an alternative to compare the global state and consensus, other than dumping and comparing balances, e.g. by using all valid transaction hashes, etc. etc. ... All this is sort of related, and you're pretty much paving the way right now, and this is basically what I wanted to highlight. :)

[zathras-crypto]

Closing this PR, code was 18 months old. Ported to latest 0.0.11.2 here #427