[Doc] Introducing Deterministic Masternodes

[random-zebra]

This is the current state of my DMN branch.
Deterministic masternodes fully working on regtest (except for proof of service, which will come later), with a good number of tests covering registration, update, revocation, revival, collaterals auto-locking, consistency after reorgs, compatibility with the old system, payments, and governance voting. ☕

There are several differences with Dash code, mostly in the RPC, mempool, block assembler, compatibility code, wallet, deployment logic, and unit/functional testing (which led to the discovery of a few bugs, including a non-trivial one, reported upstream).
But, overall, this remains respectful of the specification given here (at least for now... we'll have to modify the spec a bit when introducing shield rewards later):

• DIP3 — Deterministic Masternode Lists

Huge props and thanks to Codablock and the Dash Core developers for the design of this awesome protocol.

In order to follow the specification, implementing it in our PoS system, while still allowing the cold-stake script, a critical consensus change was needed: the masternode/budget payment is no longer included in the coinstake transaction. It is, instead, an output of the coinbase transaction, which, in turn, is no longer required to be empty.
As a side effect, this enables the definition and use of a new P2CS script, with no "free" outputs.
The block version is bumped to 10, with the new rules.

This PR serves as general container for tracking, and will now be divided in more manageable sub-PRs, for better review.

Introduction

Deterministic Masternode lists are lists of masternodes, built at every block, relying only on on-chain data (previous list, and transactions included in the current block).
All nodes derive (and verify) their masternode lists independently, from the same on-chain transactions, thus they immediately reach consensus on the tier-two state (number of masternodes, properties and status of each one).
As clearly explained in the "motivation" part of the DIP document, this is crucially different from the previous system:

The previous system was maintained with consensus mechanisms that predated Satoshi Nakamoto’s solution to the Byzantine Generals Problem. This meant that each node needed to maintain their own individual masternode list with P2P messages and not a blockchain based solution. Due to the nature of the P2P system, there was no guarantee that nodes would come to the same conclusion on what the masternode list ought to look like. Discrepancies might, for example, occur due to a different order of message reception or if messages had not been received at all. This posed some risks in regard to consensus and limited the possible uses of quorums by the system.

As a concrete example, the previous system required implementing workarounds such as "masternode reward voting" which was performed multiple blocks in advance for each block to make sure that consensus would be found and agreed on. Enforcing this consensus however still posed a risk which could have resulted in network wide forking, so a spork to turn off masternode payment enforcement was added to prevent this issue from occurring. The spork was used sporadically after major network updates.

This is a major overhaul, which brings also a good number of improvements in the user experience, while removing the shortcomings of the previous system.
All reviewers are encouraged to take a deep dive in the DIP3 document, which describes perfectly the advantages of the new system.

New Roles

For each masternode, three different "roles" are defined. Each role is represented by a private/public keypair.

1. Owner: Must be unique on the network. Can update the other two roles, and the masternode payout address.
2. Operator: Must be unique on the network. The operator key is saved in the pivx.conf of the remote node, and it is used to sign masternode-related P2P messages (e.g. budget finalisations, or masternode winners in the compatibility code). It can also be used to update the masternode IP-address, or the operator payout address (if the masternode is configured to allow a percentage of the reward to be paid to operator).
3. Voting: Doesn't have to be unique (multiple masternodes can share the same voting key). It is used to cast budget votes.

The same keypair can be used for all three roles (at least for now, the operator key will be changed to a BLS key soon), but they must be different from the key of the collateral address.

New Transaction Types

Special transactions (DIP2) were introduced in #1966, but no new transaction type has been defined in PIVX yet.
Here we introduce four new types, each identifying a particular transaction payload, with its own validation rules:

• PROREG (provider-register): this is the main special transaction. Used for the registration of a new masternode, setting all of its properties (such as the keys for each role). It creates the masternode collateral, as one of its outputs, or it references a 10000 PIV unspent output on chain (in which case, it must include a signature with its keys, as proof of ownership).
• PROUPSERV (provider-update-service): sent by the mn operator to update the properties related to the service (IP address, operator payout address)
• PROUPREG (provider-update-registrar): sent by the mn owner to update the operator key, the voting key, or the payout address.
• PROUPREV (provider-update-revoke): sent by the mn operator to revoke the service, and put the mn in PoSe-banned state (e.g. in case of compromised keys). The masternode can be "revived" later, by sending a ProUpReg tx, which sets a new operator key, and then a ProUpServ tx (signed with the new key), which sets the new IP address for the masternode.

Code Architecture

Deterministic masternodes are represented as objects of the class CDeterministicMN.
This class includes a member variable that stores a shared pointer to a constant CDeterministicMNState object, which encapsulates the dmn state (updated properties and status).

A list of masternodes is represented by the class CDeterministicMNList, which uses immutable functional maps (https://github.com/arximboldi/immer) to hold the actual information about each entry.
A new list is built at every block and mantained by CDeterministicMNManager.

The use of immutable functional maps is, in my opinion, an elegant solution, devised by Codablock, to reduce the memory overhead required for the masternode list update at every block, by adopting a copy-on-write approach.
Immutable data structures are provided by default on functional-programming oriented languages, such as Clojure or Scala, but for C++ we unfortunately need to rely on third party libraries.
We could switch to an implementation based on std::maps, but that would severly impact the performance and require hundreds of MB in the ram, just for MN list housekeeping.

The code still contains a lot of ugly parts, which we need to keep for compatibility with the old system. Most of them are properly commented. My first task, after the full deployment on mainnet, will be to do a general cleanup and refactoring, removing all the legacy system code.

Deployment

This an hard-forking update, slated for PIVX v6.0.
It will be deployed with a combination of nuparams activation height, and a spork message.
For this purpose, a new height-based (rather than time-based) spork, SPORK_21_LEGACY_MNS_MAX_HEIGHT, is introduced.
It will be used only during the transition to the new system, and can be removed afterwards.

As in the previous mandatory upgrades, there will be a protocol bump, shortly before the enforcement block.
All nodes must be updated before the protobump.
Masternodes, as usual, will need to send a new MNB start message, after upgrading, with the new protocol number.

After the enforcement, new consensus rules apply (such as block v10), and new special transactions are accepted on the network (with the exception of PROUPREG).
Deterministic masternodes can be registered, but have a few limitations, in order to be compatible with the legacy system (which is still active): for example, they cannot pay a portion of the reward to the operator, yet.
The system still uses the old logic for masternode payments (mnw signed messages), with new compatibility code, in order to take into account also nodes from the deterministic list.

After the activation of SPORK_21_LEGACY_MNS_MAX_HEIGHT, the legacy system is disabled, and the transition to deterministic masternode lists is complete.
At this point, mno can also send PROUPREG transactions, and use all the features of DMNs. The new payment logic applies.

The reasons for using a spork, instead of pre-set number of blocks after enforcement, are multiple:

• It allows for easier testing on testnet, with the possibility of disabling and re-enabling legacy masternodes multiple times, in order to test different scenarios
• It allows for safer deployment on mainnet: we might want to wait until the network has a good percentage of dmn, before disabling the legacy system. Or, conversely, there could be an unforseen bug in the compatibility code, which would require a transition faster than expected.

TODO

These are the next milestones:

• Proof of service for masternodes (with DIP6 and DIP7)
• kickass GUI
• Budget system overhaul
• Chain-locks (DIP8)
• SHIELD masternode rewards

[random-zebra]

Related sub-PRs

• [Core] Store disconnected blocks txes outside mempool during reorg #2268
• [Consensus][Build] v6 Network upgrade - C++14 standard #2269
• [Refactor] Mix refactoring in preparation for DMN lists #2270
• [DB] Introduce EVO database #2271 (depends on 2269 and 2270)
• [TierTwo] Deterministic masternode lists #2273 (depends on 2271)
• [Consensus][Refactoring] Blocks v10: TierTwo payments in the coinbase transaction #2274 (depends on 2273)
• [Consensus] New cold-staking opcode #2275 (depends on 2258 and 2274)
• [RPC] Provider-Register transactions #2296 (depends on 2273)
• [Consensus] Compatibility code for MN payments + budget voting #2308 (depends on 2275 and 2296)
• [Consensus] New masternode payment logic #2309 (depends on 2275 and 2296)
• [TierTwo] Masternode collateral auto-locking + more tests #2345 (depends on 2308 and 2309)
• [Validation][RPC] Provider-Update-Service special tx type #2349 (depends on 2345)
• [Validation][RPC] ProUpReg and ProUpRev special tx types #2363 (depends on 2349)

[furszy]

As far as what we have here, this PR is over :) . Are you going to leave only the release-notes commit here?

[random-zebra]

Yes, only release-notes left here 🎉

[furszy]

utACK 9f43405

[Fuzzbawls]

utACK 9f43405

I did change the PR's prefix to Doc since this is now only a documentation PR and all other code work was done in sub-PRs