consensus, contract, scraper, protocol, project, beacon, rpc: Replace remaining appcache sections with native structures

[jamescowens]

This is a PR to retire the remaining legacy appcache used for scraper entries and protocol entries and replace with native structures using linked lists by key and hash with full revert capability, leveldb storage, and passivation (removal from memory) of inactive objects (the RegistryDB class template). It also retrofits the Projects class (whitelist) with the registry db. This is patterned after the work I did on beacons post Fern. The original beacon registry db implementation has been refactored to use the RegistryDB template as well with some specializations due to specific complexities with beacons.

Note that this PR itself does NOT require a mandatory, because it has been designed to be backwards compatible with legacy payloads (used by the legacy appcache class). When the V13 block version/contract version 3 is set to mark the changeover to native structures for the scraper and protocol entries, this will require a mandatory.

Essentially the design idea here is for the scraper registry to store native scraper entries, keyed by the CKeyID (essentially address) of the scraper entry. Each change in state of a scraper entry by CKeyID is recorded as a new record and the prior record(s) are maintained by the historical map and the scraper db in leveldb. This includes entries where the records are marked "deleted". In this case, a record will be injected with a status of "DELETED". This approach is necessary to maintain compatibility with the scraper code, which needs to know when a record was deleted for purposes of security. The tx hash of the prior entry will be stored in the current record, to be able to walk backwards in history for any CKeyID analogous to what is done with beacons. This allows us to revert a contract add/delete by simply removing the current entry and resurrecting the previous historical record (if there is one). Storage of the current and historical records in leveldb by a common key prefix for each registry area allows extremely fast restore when starting up the wallet by replaying from the leveldb area, rather than doing a general contract replay. The true reversion capability on a blockchain reorg allows us to dispense with the large contract replays to restore proper contract state after finishing reorganizations.

The protocol registry has been implemented similar to the scraper registry, but follows a straight key-value pair paradigm. Eventually we may want to put more structure to this and constrain what type of protocol entries we will allow, but that is unnecessary right now.

The project registry (Whitelist class) has received a backing db to remove the requirement for 180 day renewals and provide for proper reversion during reorgs. So scraper, protocol, and projects, just like beacons, will not require periodic refreshes of
their entries after this PR is merged AND the next mandatory height is reached which requires everyone to have a version of the wallet that has this PR in it. Until then, the <180 day refresh of scraper, protocol and project entries will be required to ensure those nodes not running code with this PR do not "forget" the entries.

To support multiple registries essentially using identical code, I implemented a registry db template in src/gridcoin/contract/registry_db.h which is instantiated in each registry class with the appropriate registry specific types. All four registries that have db backing use this template: beacon, scraper, protocol, and project. The first class, beacon, which was actually the original implementation of the db, has been converted over to use the new template, with some specialization to deal with beacon specifics.

The only contract type that will require significant contract replay (other than the top up check based on the db heights) during wallet initialization is the poll/vote types. No replay after a reorganization is required for these as reversion works properly for unique entries like polls/votes, but the required replay at wallet start means the poll history window is limited to 180 days currently unless we choose to lengthen the window. Given the current performance challenges with polls and voting in the GUI for people with slow disks, I do not think that is advisable right now. I am considering ways to use the backing db to eliminate this contract replay and also the 180 day window as well, but that will be in a separate PR.

[jamescowens]

Next up is making the changes to the ApplyContracts code to wire this in and also the cross-connect in the scraper.

Done.

[jamescowens]

@div72 I am finishing up the testing on this. This thing is too large I think for a traditional review, but please do look over it so that I get a second eye on it. I am doing both pre-fork and post fork (on an isolated fork) testing to ensure it works correctly.

[jamescowens]

Ok. I have completed testing the pre block v13 fork operation of this PR and it passes.

Pre-fork testing involves exercising the scraper, project, and protocol areas, including reorganizing to blocks before the transactions to ensure reversion and roll-forwards work correctly.

testing_procedure_and_results-pre_v13_fork.md

[jamescowens]

@div72 and I have agreed given the testing that has already done on the pre-fork functionality, and the size of this PR, it would be best to go ahead and merge into the development branch for testing on testnet, and then do any adjustments in another PR.