Musical Chairs (#6)

* For 0L genesis tools for hard forks, and migrations: create a MoveVm session from fake data to be able to apply migrations from Move system contracts.

* wip testing fakeexector for account migrations

* Scaffolding tests for v6 genesis fork. first cleanup deprecated modules.

* add fixtures for testing recovery genesis

* cleanup fixtures
implement functional tests for:
- exporting db backup to json
- creating genesis blob from json
- creating genesis blob in one shot from db backup

TODO: launch test node from from genesis.blob
TODO: e2e tests from a fixture file

* integration test using blob from fixture to start a diem-node in test mode

* patch github actions tests

* full e2e test from db backup to starting a node

* Add diem_logger as per @corythian plus comments.

* query nodes to check if onchain state is the same as recovery.

* - include db reader util to inspect a genesis.blob.
- refactor recovery tests

* test to check validator count. cleanup test utlis

* cleanup tests

* stub a "check" feature, to check a json file agains the output genesis.blob

* finish balance comparison with error list.

* clean

* test to create single validator and check output.

* skip comparing system zero address.

* cargo fix

* cargo fmt

* create function to start a vm executor, and uses a function pointer to transform legacy data to Move arguments.

* experimenting with alternative way of starting a vm genesis session

* runs

* cleanup

* need to explicitly set the validators at genesis

* test to confirm genesis validator set, is the same as defined set.

* tests for validator set and balance passing

* change fixtures, make node start from command line

* can start a test node from genesis blob

* can genesis tools test can start a node from a genesis and query for an accounts resources

* make ol version of swarm config generator in ol/smoke_tests

* wip

* move legacyRecovery to ol/types. So now we can run the user migrations in the vm_genesis::lib. First draft builds

* create GenesisMigration user migration test. Builds.

* ol/genesis-tools builds with refactor

* refactored with clean build

* minor refactors

* can migrate the balance of end users with new design

* migration successfully migrates balances of users, validators, and operators.

* cleanup unused files. clippy,

* tests passing

* PoF requires all nodes to have equal weight in consensus per the paper.

* initial scaffold of proof-of-fee auction. Borrows code from NodeWeight. Includes tranaction api.

* implement seat filling algo, which considers the 1/3 unproven node limit on epoch changes

* patch

* patch

* builds

* implement vm function to pay fee from user account

* all validators are charged fee at epoch boundary

* cleanup  validator set proposal functions

* refactor jail function at epoch boundary

* make epoch boundry _meta_epoch.move pass

* filter for vouches on proof of fee

* initialize val account with a PoF struct

* epoch reconfiguration meta tests passing. PoF bid initialization unit tests passing. Validator meta sanity tests passing

* patch issue with failover

* add bid restrictions, and checking upon seating

* burn excess fees than needed for validators

* reward thermostat initial implementation

* genesis init fee baseline

* patch bug in thermostat

* more tests for fill_seats, and thermostat. patches

* test

* consolidate validator audits, check for the case that there are fewer bidders than the stated val set.

* add test fixtures, modify some APIs "tell don't ask" pattern. Add several tests.

* add unit tests for thermostat. Patch a couple bugs

* tests for thermostat increase/decrease

* deprecate Audit.move, and create tests for qualifications in Proofoffee.move

* Jailing mechanism needs update after deprecating Towers. Proposing here that a Voucher is the only one to unjail, since there is no other practical hurdle.

* audit tests for vouch, expired bids, no_funds

* covered all proof of fee functions, tests for all common scenarios of fill_seats, patch logic issues, all proof of fee tests passing!

* patch seating failover

* Clean up Cases.move. Patch Vouch tests fixtures

* adjust fixtures for reconfiguration tests, deprecate some tests, patch jailing

* patch all reconfig test fixtures. remove deprecated burn tests.

* continue patching tests and removing deprecated scenarios

* All transactional tests passing!

* txs transaction apis for setting and retracting bids.

* test for set and retract

* implement vm function to pay fee from user account

* cleanup  validator set proposal functions

* refactor jail function at epoch boundary

* make epoch boundry _meta_epoch.move pass

* initialize val account with a PoF struct

* add bid restrictions, and checking upon seating

* genesis initialization of musical chairs struct

* musical chairs heuristic draft complete.

* epoch boundary calls musical chairs to set next val size

* transactional tests WIP

* patch rebase issue

* add tests for eval_compliance. all transactional tests passing!

* rename variable in Globals

* patch test, remove deprecated.

* all tests passing

diem-move/diem-framework/DPN/releases/artifacts/current/build/DPNFramework/BuildInfo.yaml

@@ -196,6 +196,9 @@ compiled_package_info:
     ? address: "00000000000000000000000000000001"
       name: MultiSigPayment
     : DiemFramework
+    ? address: "00000000000000000000000000000001"
+      name: MusicalChairs
+    : DiemFramework
     ? address: "00000000000000000000000000000001"
       name: NetworkIdentity
     : DiemFramework
@@ -346,7 +349,7 @@ compiled_package_info:
     ? address: "00000000000000000000000000000001"
       name: XUS
     : DiemFramework
-  source_digest: 110A62F23F4C530726F3CD908ED8A7393D8E6E9D5B8E6F2543C536D0C02458CB
+  source_digest: A5D2C6F20AC5E12BA8A8A6167D118FDC4F0404D311CE72F0BC9928353DC69318
   build_flags:
     dev_mode: false
     test_mode: false

diem-move/diem-framework/DPN/releases/artifacts/current/build/DPNFramework/docs/Burn.md

@@ -115,12 +115,13 @@
     vm: &signer,
 )  <b>acquires</b> <a href="Burn.md#0x1_Burn_BurnPreference">BurnPreference</a>, <a href="Burn.md#0x1_Burn_DepositInfo">DepositInfo</a> {
     <a href="CoreAddresses.md#0x1_CoreAddresses_assert_vm">CoreAddresses::assert_vm</a>(vm);
+
     // extract fees
     <b>let</b> coins = <a href="TransactionFee.md#0x1_TransactionFee_vm_withdraw_all_coins">TransactionFee::vm_withdraw_all_coins</a>&lt;<a href="GAS.md#0x1_GAS">GAS</a>&gt;(vm);
 
     // get the list of fee makers
-    // <b>let</b> state = <b>borrow_global</b>&lt;EpochFeeMakerRegistry&gt;(@VMReserved);
     <b>let</b> fee_makers = <a href="TransactionFee.md#0x1_TransactionFee_get_fee_makers">TransactionFee::get_fee_makers</a>();
+
     <b>let</b> len = <a href="../../../../../../../DPN/releases/artifacts/current/build/MoveStdlib/docs/Vector.md#0x1_Vector_length">Vector::length</a>(&fee_makers);
 
     // for every user in the list burn their fees per <a href="Burn.md#0x1_Burn">Burn</a>.<b>move</b> preferences
@@ -134,6 +135,7 @@
         i = i + 1;
     };
 
+  // Transaction fee account should be empty at the end of the epoch
   // Superman 3 decimal errors. https://www.youtube.com/watch?v=N7JBXGkBoFc
   // anything that is remaining should be burned
   <a href="Diem.md#0x1_Diem_vm_burn_this_coin">Diem::vm_burn_this_coin</a>(vm, coins);
@@ -186,7 +188,6 @@
     <b>let</b> cumu = *<a href="../../../../../../../DPN/releases/artifacts/current/build/MoveStdlib/docs/Vector.md#0x1_Vector_borrow">Vector::borrow</a>(&deposit_vec, k);
 
     <b>let</b> ratio = <a href="../../../../../../../DPN/releases/artifacts/current/build/MoveStdlib/docs/FixedPoint32.md#0x1_FixedPoint32_create_from_rational">FixedPoint32::create_from_rational</a>(cumu, global_deposits);
-    // print(&ratio);
 
     <a href="../../../../../../../DPN/releases/artifacts/current/build/MoveStdlib/docs/Vector.md#0x1_Vector_push_back">Vector::push_back</a>(&<b>mut</b> ratios_vec, ratio);
     k = k + 1;
@@ -259,17 +260,12 @@
 
   <b>let</b> d = <b>borrow_global</b>&lt;<a href="Burn.md#0x1_Burn_DepositInfo">DepositInfo</a>&gt;(@VMReserved);
   <b>let</b> _contains = <a href="../../../../../../../DPN/releases/artifacts/current/build/MoveStdlib/docs/Vector.md#0x1_Vector_contains">Vector::contains</a>(&d.addr, &payee);
-  // print(&contains);
   <b>let</b> (is_found, i) = <a href="../../../../../../../DPN/releases/artifacts/current/build/MoveStdlib/docs/Vector.md#0x1_Vector_index_of">Vector::index_of</a>(&d.addr, &payee);
   <b>if</b> (is_found) {
-    // print(&is_found);
     <b>let</b> len = <a href="../../../../../../../DPN/releases/artifacts/current/build/MoveStdlib/docs/Vector.md#0x1_Vector_length">Vector::length</a>(&d.ratio);
-    // print(&i);
-    // print(&len);
     <b>if</b> (i + 1 &gt; len) <b>return</b> 0;
     <b>let</b> ratio = *<a href="../../../../../../../DPN/releases/artifacts/current/build/MoveStdlib/docs/Vector.md#0x1_Vector_borrow">Vector::borrow</a>(&d.ratio, i);
     <b>if</b> (<a href="../../../../../../../DPN/releases/artifacts/current/build/MoveStdlib/docs/FixedPoint32.md#0x1_FixedPoint32_is_zero">FixedPoint32::is_zero</a>(<b>copy</b> ratio)) <b>return</b> 0;
-    // print(&ratio);
     <b>return</b> <a href="../../../../../../../DPN/releases/artifacts/current/build/MoveStdlib/docs/FixedPoint32.md#0x1_FixedPoint32_multiply_u64">FixedPoint32::multiply_u64</a>(value, ratio)
   };
 
@@ -300,7 +296,6 @@
   vm: &signer, payer: <b>address</b>, user_share: <a href="Diem.md#0x1_Diem">Diem</a>&lt;<a href="GAS.md#0x1_GAS">GAS</a>&gt;
 ) <b>acquires</b> <a href="Burn.md#0x1_Burn_DepositInfo">DepositInfo</a>, <a href="Burn.md#0x1_Burn_BurnPreference">BurnPreference</a> {
   <a href="CoreAddresses.md#0x1_CoreAddresses_assert_vm">CoreAddresses::assert_vm</a>(vm);
-
   <b>if</b> (<b>exists</b>&lt;<a href="Burn.md#0x1_Burn_BurnPreference">BurnPreference</a>&gt;(payer)) {
     <b>if</b> (<b>borrow_global</b>&lt;<a href="Burn.md#0x1_Burn_BurnPreference">BurnPreference</a>&gt;(payer).send_community) {
       <a href="Burn.md#0x1_Burn_recycle">recycle</a>(vm, payer, &<b>mut</b> user_share);
@@ -335,19 +330,17 @@
   <b>let</b> list = <a href="Burn.md#0x1_Burn_get_address_list">get_address_list</a>();
   <b>let</b> len = <a href="../../../../../../../DPN/releases/artifacts/current/build/MoveStdlib/docs/Vector.md#0x1_Vector_length">Vector::length</a>&lt;<b>address</b>&gt;(&list);
 
+
   <b>let</b> total_coin_value_to_recycle = <a href="Diem.md#0x1_Diem_value">Diem::value</a>(coin);
-  // print(&list);
 
   // There could be errors in the array, and underpayment happen.
   <b>let</b> value_sent = 0;
 
   <b>let</b> i = 0;
   <b>while</b> (i &lt; len) {
+
     <b>let</b> payee = *<a href="../../../../../../../DPN/releases/artifacts/current/build/MoveStdlib/docs/Vector.md#0x1_Vector_borrow">Vector::borrow</a>&lt;<b>address</b>&gt;(&list, i);
-    // print(&payee);
     <b>let</b> amount_to_payee = <a href="Burn.md#0x1_Burn_get_payee_value">get_payee_value</a>(payee, total_coin_value_to_recycle);
-    // print(&val);
-
     <b>let</b> to_deposit = <a href="Diem.md#0x1_Diem_withdraw">Diem::withdraw</a>(coin, amount_to_payee);
 
     <a href="DiemAccount.md#0x1_DiemAccount_vm_deposit_with_metadata">DiemAccount::vm_deposit_with_metadata</a>&lt;<a href="GAS.md#0x1_GAS">GAS</a>&gt;(

diem-move/diem-framework/DPN/releases/artifacts/current/build/DPNFramework/docs/DiemAccount.md

@@ -3920,6 +3920,7 @@ Creating an account at address 0x0 will abort as it is a reserved address for th
 
     // Publish <a href="AccountFreezing.md#0x1_AccountFreezing_FreezingBit">AccountFreezing::FreezingBit</a> (initially not frozen)
     <a href="AccountFreezing.md#0x1_AccountFreezing_create">AccountFreezing::create</a>(new_account);
+    <a href="TransactionFee.md#0x1_TransactionFee_initialize_fee_maker">TransactionFee::initialize_fee_maker</a>(new_account);
     // The <a href="DiemAccount.md#0x1_DiemAccount_AccountOperationsCapability">AccountOperationsCapability</a> is published during <a href="Genesis.md#0x1_Genesis">Genesis</a>, so it should
     // always exist.  This is a sanity check.
     <b>assert</b>!(

diem-move/diem-framework/DPN/releases/artifacts/current/build/DPNFramework/docs/EpochBoundary.md

@@ -207,7 +207,7 @@
 
     // after everyone is paid from the chain's Fee account
     // we can burn the excess fees from the epoch
-
+    <a href="Burn.md#0x1_Burn_reset_ratios">Burn::reset_ratios</a>(vm);
     <a href="Burn.md#0x1_Burn_epoch_burn_fees">Burn::epoch_burn_fees</a>(vm);
 }
 </code></pre>

diem-move/diem-framework/DPN/releases/artifacts/current/build/DPNFramework/docs/Genesis.md

@@ -35,6 +35,7 @@ when executing from a fresh state.
 <b>use</b> <a href="../../../../../../../DPN/releases/artifacts/current/build/MoveStdlib/docs/Hash.md#0x1_Hash">0x1::Hash</a>;
 <b>use</b> <a href="Migrations.md#0x1_Migrations">0x1::Migrations</a>;
 <b>use</b> <a href="MultiSigPayment.md#0x1_MultiSigPayment">0x1::MultiSigPayment</a>;
+<b>use</b> <a href="MusicalChairs.md#0x1_MusicalChairs">0x1::MusicalChairs</a>;
 <b>use</b> <a href="Oracle.md#0x1_Oracle">0x1::Oracle</a>;
 <b>use</b> <a href="ParallelExecutionConfig.md#0x1_ParallelExecutionConfig">0x1::ParallelExecutionConfig</a>;
 <b>use</b> <a href="ProofOfFee.md#0x1_ProofOfFee">0x1::ProofOfFee</a>;
@@ -229,6 +230,7 @@ Initializes the Diem Framework. Internal so it can be used by both genesis code,
     <a href="DonorDirected.md#0x1_DonorDirected_init_root_registry">DonorDirected::init_root_registry</a>(dr_account);
     <a href="DiemAccount.md#0x1_DiemAccount_vm_init_slow">DiemAccount::vm_init_slow</a>(dr_account);
     <a href="Migrations.md#0x1_Migrations_init">Migrations::init</a>(dr_account);
+    <a href="MusicalChairs.md#0x1_MusicalChairs_initialize">MusicalChairs::initialize</a>(dr_account);
 
     // After we have called this function, all invariants which are guarded by
     // `<a href="DiemTimestamp.md#0x1_DiemTimestamp_is_operating">DiemTimestamp::is_operating</a>() ==&gt; ...` will become active and a verification condition.

diem-move/diem-framework/DPN/releases/artifacts/current/build/DPNFramework/docs/Globals.md

@@ -15,7 +15,7 @@ This module provides global variables and constants that have no specific owner
 -  [Struct `GlobalConstants`](#0x1_Globals_GlobalConstants)
 -  [Constants](#@Constants_1)
 -  [Function `get_epoch_length`](#0x1_Globals_get_epoch_length)
--  [Function `get_max_validators_per_set`](#0x1_Globals_get_max_validators_per_set)
+-  [Function `get_val_set_at_genesis`](#0x1_Globals_get_val_set_at_genesis)
 -  [Function `get_coin_scaling_factor`](#0x1_Globals_get_coin_scaling_factor)
 -  [Function `get_subsidy_ceiling_gas`](#0x1_Globals_get_subsidy_ceiling_gas)
 -  [Function `get_vdf_difficulty_baseline`](#0x1_Globals_get_vdf_difficulty_baseline)
@@ -45,7 +45,7 @@ This module provides global variables and constants that have no specific owner
 Global constants determining validator settings & requirements
 Some constants need to be changed based on environment; dev, testing, prod.
 epoch_length: The length of an epoch in seconds (~1 day for prod.)
-max_validators_per_set: The maximum number of validators that can participate
+val_set_at_genesis: The maximum number of validators that can participate
 subsidy_ceiling_gas: TODO I don't really know what this is
 vdf_difficulty: The difficulty required for VDF proofs submitting by miners
 epoch_mining_thres_lower: The number of proofs that must be submitted each
@@ -69,7 +69,7 @@ epoch by a miner to remain compliant
 
 </dd>
 <dt>
-<code>max_validators_per_set: u64</code>
+<code>val_set_at_genesis: u64</code>
 </dt>
 <dd>
 
@@ -172,14 +172,14 @@ Get the epoch length
 
 </details>
 
-<a name="0x1_Globals_get_max_validators_per_set"></a>
+<a name="0x1_Globals_get_val_set_at_genesis"></a>
 
-## Function `get_max_validators_per_set`
+## Function `get_val_set_at_genesis`
 
 Get max validator per epoch
 
 
-<pre><code><b>public</b> <b>fun</b> <a href="Globals.md#0x1_Globals_get_max_validators_per_set">get_max_validators_per_set</a>(): u64
+<pre><code><b>public</b> <b>fun</b> <a href="Globals.md#0x1_Globals_get_val_set_at_genesis">get_val_set_at_genesis</a>(): u64
 </code></pre>
 
 
@@ -188,8 +188,8 @@ Get max validator per epoch
 <summary>Implementation</summary>
 
 
-<pre><code><b>public</b> <b>fun</b> <a href="Globals.md#0x1_Globals_get_max_validators_per_set">get_max_validators_per_set</a>(): u64 {
-   <a href="Globals.md#0x1_Globals_get_constants">get_constants</a>().max_validators_per_set
+<pre><code><b>public</b> <b>fun</b> <a href="Globals.md#0x1_Globals_get_val_set_at_genesis">get_val_set_at_genesis</a>(): u64 {
+   <a href="Globals.md#0x1_Globals_get_constants">get_constants</a>().val_set_at_genesis
 }
 </code></pre>
 
@@ -473,7 +473,7 @@ Get the constants for the current network
   <b>if</b> (<a href="Testnet.md#0x1_Testnet_is_testnet">Testnet::is_testnet</a>()) {
     <b>return</b> <a href="Globals.md#0x1_Globals_GlobalConstants">GlobalConstants</a> {
       epoch_length: 60, // seconds
-      max_validators_per_set: 100,
+      val_set_at_genesis: 10,
       subsidy_ceiling_gas: 296 * <a href="Globals.md#0x1_Globals_COIN_SCALING_FACTOR">COIN_SCALING_FACTOR</a>,
       vdf_difficulty_baseline: 100,
       vdf_security_baseline: 512,
@@ -491,7 +491,7 @@ Get the constants for the current network
   <b>if</b> (<a href="Testnet.md#0x1_StagingNet_is_staging_net">StagingNet::is_staging_net</a>()) {
     <b>return</b> <a href="Globals.md#0x1_Globals_GlobalConstants">GlobalConstants</a> {
       epoch_length: 60 * 40, // 40 mins, enough for a hard miner proof.
-      max_validators_per_set: 100,
+      val_set_at_genesis: 100,
       subsidy_ceiling_gas: 8640000 * <a href="Globals.md#0x1_Globals_COIN_SCALING_FACTOR">COIN_SCALING_FACTOR</a>,
       vdf_difficulty_baseline: 120000000,
       vdf_security_baseline: 512,
@@ -506,7 +506,7 @@ Get the constants for the current network
   } <b>else</b> {
     <b>return</b> <a href="Globals.md#0x1_Globals_GlobalConstants">GlobalConstants</a> {
       epoch_length: 60 * 60 * 24, // approx 24 hours at 1.4 vdf_proofs/sec
-      max_validators_per_set: 100, // max expected for BFT limits.
+      val_set_at_genesis: 100, // max expected for BFT limits.
       // See <a href="DiemVMConfig.md#0x1_DiemVMConfig">DiemVMConfig</a> for gas constants:
       // Target max gas units per transaction 100000000
       // target max block time: 2 secs