Proof of Fee (#5)

* For 0L genesis tools for hard forks, and migrations: create a MoveVm session from fake data to be able to apply migrations from Move system contracts.

* wip testing fakeexector for account migrations

* Scaffolding tests for v6 genesis fork. first cleanup deprecated modules.

* add fixtures for testing recovery genesis

* cleanup fixtures
implement functional tests for:
- exporting db backup to json
- creating genesis blob from json
- creating genesis blob in one shot from db backup

TODO: launch test node from from genesis.blob
TODO: e2e tests from a fixture file

* integration test using blob from fixture to start a diem-node in test mode

* patch github actions tests

* full e2e test from db backup to starting a node

* Add diem_logger as per @corythian plus comments.

* query nodes to check if onchain state is the same as recovery.

* - include db reader util to inspect a genesis.blob.
- refactor recovery tests

* test to check validator count. cleanup test utlis

* cleanup tests

* stub a "check" feature, to check a json file agains the output genesis.blob

* finish balance comparison with error list.

* clean

* test to create single validator and check output.

* skip comparing system zero address.

* cargo fix

* cargo fmt

* create function to start a vm executor, and uses a function pointer to transform legacy data to Move arguments.

* experimenting with alternative way of starting a vm genesis session

* runs

* cleanup

* need to explicitly set the validators at genesis

* test to confirm genesis validator set, is the same as defined set.

* tests for validator set and balance passing

* change fixtures, make node start from command line

* can start a test node from genesis blob

* can genesis tools test can start a node from a genesis and query for an accounts resources

* make ol version of swarm config generator in ol/smoke_tests

* wip

* move legacyRecovery to ol/types. So now we can run the user migrations in the vm_genesis::lib. First draft builds

* create GenesisMigration user migration test. Builds.

* ol/genesis-tools builds with refactor

* refactored with clean build

* minor refactors

* can migrate the balance of end users with new design

* migration successfully migrates balances of users, validators, and operators.

* cleanup unused files. clippy,

* tests passing

* PoF requires all nodes to have equal weight in consensus per the paper.

* Update ci.yaml (#1239)

* [Genesis] Migration tools for genesis from legacy accounts (#1237)

* For 0L genesis tools for hard forks, and migrations: create a MoveVm session from fake data to be able to apply migrations from Move system contracts.

* wip testing fakeexector for account migrations

* Scaffolding tests for v6 genesis fork. first cleanup deprecated modules.

* add fixtures for testing recovery genesis

* cleanup fixtures
implement functional tests for:
- exporting db backup to json
- creating genesis blob from json
- creating genesis blob in one shot from db backup

TODO: launch test node from from genesis.blob
TODO: e2e tests from a fixture file

* integration test using blob from fixture to start a diem-node in test mode

* patch github actions tests

* full e2e test from db backup to starting a node

* Add diem_logger as per @corythian plus comments.

* query nodes to check if onchain state is the same as recovery.

* - include db reader util to inspect a genesis.blob.
- refactor recovery tests

* test to check validator count. cleanup test utlis

* cleanup tests

* stub a "check" feature, to check a json file agains the output genesis.blob

* finish balance comparison with error list.

* clean

* test to create single validator and check output.

* skip comparing system zero address.

* cargo fix

* cargo fmt

* create function to start a vm executor, and uses a function pointer to transform legacy data to Move arguments.

* experimenting with alternative way of starting a vm genesis session

* runs

* cleanup

* need to explicitly set the validators at genesis

* test to confirm genesis validator set, is the same as defined set.

* tests for validator set and balance passing

* change fixtures, make node start from command line

* can start a test node from genesis blob

* can genesis tools test can start a node from a genesis and query for an accounts resources

* make ol version of swarm config generator in ol/smoke_tests

* wip

* move legacyRecovery to ol/types. So now we can run the user migrations in the vm_genesis::lib. First draft builds

* create GenesisMigration user migration test. Builds.

* ol/genesis-tools builds with refactor

* refactored with clean build

* minor refactors

* can migrate the balance of end users with new design

* migration successfully migrates balances of users, validators, and operators.

* cleanup unused files. clippy,

* tests passing

* whitespace to trigger ci

* initial scaffold of proof-of-fee auction. Borrows code from NodeWeight. Includes tranaction api.

* implement seat filling algo, which considers the 1/3 unproven node limit on epoch changes

* patch

* patch

* builds

* implement vm function to pay fee from user account

* all validators are charged fee at epoch boundary

* cleanup  validator set proposal functions

* refactor jail function at epoch boundary

* make epoch boundry _meta_epoch.move pass

* filter for vouches on proof of fee

* initialize val account with a PoF struct

* epoch reconfiguration meta tests passing. PoF bid initialization unit tests passing. Validator meta sanity tests passing

* patch issue with failover

* add bid restrictions, and checking upon seating

* burn excess fees than needed for validators

* reward thermostat initial implementation

* genesis init fee baseline

* patch bug in thermostat

* more tests for fill_seats, and thermostat. patches

* test

* consolidate validator audits, check for the case that there are fewer bidders than the stated val set.

* add test fixtures, modify some APIs "tell don't ask" pattern. Add several tests.

* add unit tests for thermostat. Patch a couple bugs

* tests for thermostat increase/decrease

* deprecate Audit.move, and create tests for qualifications in Proofoffee.move

* Jailing mechanism needs update after deprecating Towers. Proposing here that a Voucher is the only one to unjail, since there is no other practical hurdle.

* audit tests for vouch, expired bids, no_funds

* covered all proof of fee functions, tests for all common scenarios of fill_seats, patch logic issues, all proof of fee tests passing!

* patch seating failover

* Clean up Cases.move. Patch Vouch tests fixtures

* adjust fixtures for reconfiguration tests, deprecate some tests, patch jailing

* patch all reconfig test fixtures. remove deprecated burn tests.

* continue patching tests and removing deprecated scenarios

* All transactional tests passing!

* txs transaction apis for setting and retracting bids.

* test for set and retract

* cleanup prints

* add struct in TransactionFee to track origin of fees FeeMaker

* refactor Burn to consider the preference of the user FeeMaker

* cleanup

* init FeeMaker on account creation

* patch tests to remove `print`

* patch epoch boundary burn ratios

* all tests passing

---------

Co-authored-by: Zoz <97761083+0xzoz@users.noreply.github.com>

diem-move/diem-framework/DPN/releases/artifacts/current/build/DPNFramework/BuildInfo.yaml

@@ -31,9 +31,6 @@ compiled_package_info:
     ? address: "00000000000000000000000000000001"
       name: Ancestry
     : DiemFramework
-    ? address: "00000000000000000000000000000001"
-      name: Audit
-    : DiemFramework
     ? address: "00000000000000000000000000000001"
       name: Authenticator
     : DiemFramework
@@ -223,6 +220,9 @@ compiled_package_info:
     ? address: "00000000000000000000000000000001"
       name: PersistenceDemo
     : DiemFramework
+    ? address: "00000000000000000000000000000001"
+      name: ProofOfFee
+    : DiemFramework
     ? address: "00000000000000000000000000000001"
       name: Receipts
     : DiemFramework
@@ -346,7 +346,7 @@ compiled_package_info:
     ? address: "00000000000000000000000000000001"
       name: XUS
     : DiemFramework
-  source_digest: E73442AD7A1B65175ABE2ACCB60D6A59204FBFD1C63E20E767E29AAC56BD3285
+  source_digest: 110A62F23F4C530726F3CD908ED8A7393D8E6E9D5B8E6F2543C536D0C02458CB
   build_flags:
     dev_mode: false
     test_mode: false

diem-move/diem-framework/DPN/releases/artifacts/current/build/DPNFramework/docs/Ancestry.md

@@ -16,7 +16,6 @@
 
 
 <pre><code><b>use</b> <a href="CoreAddresses.md#0x1_CoreAddresses">0x1::CoreAddresses</a>;
-<b>use</b> <a href="Debug.md#0x1_Debug">0x1::Debug</a>;
 <b>use</b> <a href="../../../../../../../DPN/releases/artifacts/current/build/MoveStdlib/docs/Option.md#0x1_Option">0x1::Option</a>;
 <b>use</b> <a href="../../../../../../../DPN/releases/artifacts/current/build/MoveStdlib/docs/Signer.md#0x1_Signer">0x1::Signer</a>;
 <b>use</b> <a href="../../../../../../../DPN/releases/artifacts/current/build/MoveStdlib/docs/Vector.md#0x1_Vector">0x1::Vector</a>;
@@ -67,7 +66,7 @@
 
 
 <pre><code><b>public</b> <b>fun</b> <a href="Ancestry.md#0x1_Ancestry_init">init</a>(new_account_sig: &signer, onboarder_sig: &signer ) <b>acquires</b> <a href="Ancestry.md#0x1_Ancestry">Ancestry</a>{
-    print(&100100);
+    // print(&100100);
     <b>let</b> parent = <a href="../../../../../../../DPN/releases/artifacts/current/build/MoveStdlib/docs/Signer.md#0x1_Signer_address_of">Signer::address_of</a>(onboarder_sig);
     <a href="Ancestry.md#0x1_Ancestry_set_tree">set_tree</a>(new_account_sig, parent);
 }
@@ -94,7 +93,7 @@
 
 <pre><code><b>fun</b> <a href="Ancestry.md#0x1_Ancestry_set_tree">set_tree</a>(new_account_sig: &signer, parent: <b>address</b> ) <b>acquires</b> <a href="Ancestry.md#0x1_Ancestry">Ancestry</a> {
   <b>let</b> child = <a href="../../../../../../../DPN/releases/artifacts/current/build/MoveStdlib/docs/Signer.md#0x1_Signer_address_of">Signer::address_of</a>(new_account_sig);
-    print(&100200);
+    // print(&100200);
   <b>let</b> new_tree = <a href="../../../../../../../DPN/releases/artifacts/current/build/MoveStdlib/docs/Vector.md#0x1_Vector_empty">Vector::empty</a>&lt;<b>address</b>&gt;();
 
   // get the parent's ancestry <b>if</b> initialized.
@@ -103,31 +102,31 @@
   <b>if</b> (<b>exists</b>&lt;<a href="Ancestry.md#0x1_Ancestry">Ancestry</a>&gt;(parent)) {
     <b>let</b> parent_state = <b>borrow_global_mut</b>&lt;<a href="Ancestry.md#0x1_Ancestry">Ancestry</a>&gt;(parent);
     <b>let</b> parent_tree = *&parent_state.tree;
-    print(&100210);
+    // print(&100210);
     <b>if</b> (<a href="../../../../../../../DPN/releases/artifacts/current/build/MoveStdlib/docs/Vector.md#0x1_Vector_length">Vector::length</a>&lt;<b>address</b>&gt;(&parent_tree) &gt; 0) {
       <a href="../../../../../../../DPN/releases/artifacts/current/build/MoveStdlib/docs/Vector.md#0x1_Vector_append">Vector::append</a>(&<b>mut</b> new_tree, parent_tree);
     };
-    print(&100220);
+    // print(&100220);
   };
 
   // add the parent <b>to</b> the tree
   <a href="../../../../../../../DPN/releases/artifacts/current/build/MoveStdlib/docs/Vector.md#0x1_Vector_push_back">Vector::push_back</a>(&<b>mut</b> new_tree, parent);
-    print(&100230);
+    // print(&100230);
 
   <b>if</b> (!<b>exists</b>&lt;<a href="Ancestry.md#0x1_Ancestry">Ancestry</a>&gt;(child)) {
     <b>move_to</b>&lt;<a href="Ancestry.md#0x1_Ancestry">Ancestry</a>&gt;(new_account_sig, <a href="Ancestry.md#0x1_Ancestry">Ancestry</a> {
       tree: new_tree,
     });
-    print(&100240);
+    // print(&100240);
 
   } <b>else</b> {
     // this is only for migration cases.
     <b>let</b> child_ancestry = <b>borrow_global_mut</b>&lt;<a href="Ancestry.md#0x1_Ancestry">Ancestry</a>&gt;(child);
     child_ancestry.tree = new_tree;
-    print(&100250);
+    // print(&100250);
 
   };
-  print(&100260);
+  // print(&100260);
 
 }
 </code></pre>
@@ -183,40 +182,40 @@
 <pre><code><b>public</b> <b>fun</b> <a href="Ancestry.md#0x1_Ancestry_is_family">is_family</a>(left: <b>address</b>, right: <b>address</b>): (bool, <b>address</b>) <b>acquires</b> <a href="Ancestry.md#0x1_Ancestry">Ancestry</a> {
   <b>let</b> is_family = <b>false</b>;
   <b>let</b> common_ancestor = @0x0;
-  print(&100300);
-  print(&<b>exists</b>&lt;<a href="Ancestry.md#0x1_Ancestry">Ancestry</a>&gt;(left));
-  print(&<b>exists</b>&lt;<a href="Ancestry.md#0x1_Ancestry">Ancestry</a>&gt;(right));
+  // // print(&100300);
+  // // print(&<b>exists</b>&lt;<a href="Ancestry.md#0x1_Ancestry">Ancestry</a>&gt;(left));
+  // // print(&<b>exists</b>&lt;<a href="Ancestry.md#0x1_Ancestry">Ancestry</a>&gt;(right));
 
   // <b>if</b> (<b>exists</b>&lt;<a href="Ancestry.md#0x1_Ancestry">Ancestry</a>&gt;(left) && <b>exists</b>&lt;<a href="Ancestry.md#0x1_Ancestry">Ancestry</a>&gt;(right)) {
     // <b>if</b> tree is empty it will still work.
-    print(&100310);
+    // // print(&100310);
     <b>let</b> left_tree = <a href="Ancestry.md#0x1_Ancestry_get_tree">get_tree</a>(left);
-    print(&100311);
+    // // print(&100311);
     <b>let</b> right_tree = <a href="Ancestry.md#0x1_Ancestry_get_tree">get_tree</a>(right);
 
-    print(&100320);
+    // // print(&100320);
 
     // check for direct relationship.
     <b>if</b> (<a href="../../../../../../../DPN/releases/artifacts/current/build/MoveStdlib/docs/Vector.md#0x1_Vector_contains">Vector::contains</a>(&left_tree, &right)) <b>return</b> (<b>true</b>, right);
     <b>if</b> (<a href="../../../../../../../DPN/releases/artifacts/current/build/MoveStdlib/docs/Vector.md#0x1_Vector_contains">Vector::contains</a>(&right_tree, &left)) <b>return</b> (<b>true</b>, left);
 
-    print(&100330);
+    // // print(&100330);
     <b>let</b> i = 0;
     // check every <b>address</b> on the list <b>if</b> there are overlaps.
     <b>while</b> (i &lt; <a href="../../../../../../../DPN/releases/artifacts/current/build/MoveStdlib/docs/Vector.md#0x1_Vector_length">Vector::length</a>&lt;<b>address</b>&gt;(&left_tree)) {
-      print(&100341);
+      // // print(&100341);
       <b>let</b> family_addr = <a href="../../../../../../../DPN/releases/artifacts/current/build/MoveStdlib/docs/Vector.md#0x1_Vector_borrow">Vector::borrow</a>(&left_tree, i);
       <b>if</b> (<a href="../../../../../../../DPN/releases/artifacts/current/build/MoveStdlib/docs/Vector.md#0x1_Vector_contains">Vector::contains</a>(&right_tree, family_addr)) {
         is_family = <b>true</b>;
         common_ancestor = *family_addr;
-        print(&100342);
+        // // print(&100342);
         <b>break</b>
       };
       i = i + 1;
     };
-    print(&100350);
+    // // print(&100350);
   // };
-  print(&100360);
+  // // print(&100360);
   (is_family, common_ancestor)
 }
 </code></pre>
@@ -322,13 +321,13 @@
     <b>move_to</b>&lt;<a href="Ancestry.md#0x1_Ancestry">Ancestry</a>&gt;(child_sig, <a href="Ancestry.md#0x1_Ancestry">Ancestry</a> {
       tree: migrate_tree,
     });
-    print(&100240);
+    // print(&100240);
 
   } <b>else</b> {
     // this is only for migration cases.
     <b>let</b> child_ancestry = <b>borrow_global_mut</b>&lt;<a href="Ancestry.md#0x1_Ancestry">Ancestry</a>&gt;(child);
     child_ancestry.tree = migrate_tree;
-    print(&100250);
+    // print(&100250);
 
   };
 }

diem-move/diem-framework/DPN/releases/artifacts/current/build/DPNFramework/docs/AutoPay.md

@@ -37,8 +37,7 @@ This module enables automatic payments from accounts to community wallets at epo
 -  [Function `find`](#0x1_AutoPay_find)
 
 
-<pre><code><b>use</b> <a href="CommunityWallet.md#0x1_CommunityWallet">0x1::CommunityWallet</a>;
-<b>use</b> <a href="DiemAccount.md#0x1_DiemAccount">0x1::DiemAccount</a>;
+<pre><code><b>use</b> <a href="DiemAccount.md#0x1_DiemAccount">0x1::DiemAccount</a>;
 <b>use</b> <a href="DiemConfig.md#0x1_DiemConfig">0x1::DiemConfig</a>;
 <b>use</b> <a href="../../../../../../../DPN/releases/artifacts/current/build/MoveStdlib/docs/Errors.md#0x1_Errors">0x1::Errors</a>;
 <b>use</b> <a href="../../../../../../../DPN/releases/artifacts/current/build/MoveStdlib/docs/FixedPoint32.md#0x1_FixedPoint32">0x1::FixedPoint32</a>;
@@ -352,16 +351,6 @@ Maximum value for the Payment type selection
 
 
 
-<a name="0x1_AutoPay_PAYEE_NOT_COMMUNITY_WALLET"></a>
-
-Attempt to make a payment to a non-community-wallet
-
-
-<pre><code><b>const</b> <a href="AutoPay.md#0x1_AutoPay_PAYEE_NOT_COMMUNITY_WALLET">PAYEE_NOT_COMMUNITY_WALLET</a>: u64 = 10024;
-</code></pre>
-
-
-
 <a name="0x1_AutoPay_PERCENT_OF_BALANCE"></a>
 
 send percent of balance at end of epoch payment type
@@ -601,7 +590,6 @@ Attempt to use a UID that is already taken
   <b>let</b> bal_change_since_last_run = <b>if</b> (pre_run_bal &gt; my_autopay_state.prev_bal) {
     pre_run_bal - my_autopay_state.prev_bal
   } <b>else</b> { 0 };
-
   // go through the pledges
   <b>while</b> (payments_idx &lt; payments_len) {
     <b>let</b> payment = <a href="../../../../../../../DPN/releases/artifacts/current/build/MoveStdlib/docs/Vector.md#0x1_Vector_borrow_mut">Vector::borrow_mut</a>&lt;<a href="AutoPay.md#0x1_AutoPay_Payment">Payment</a>&gt;(payments, payments_idx);
@@ -648,10 +636,6 @@ Attempt to use a UID that is already taken
   payment: &<b>mut</b> <a href="AutoPay.md#0x1_AutoPay_Payment">Payment</a>,
   bal_change_since_last_run: u64,
 ): bool {
-  // check payees are community wallets, only community wallets are allowed
-  // <b>to</b> receive autopay (bypassing account limits)
-  <b>if</b> (!<a href="CommunityWallet.md#0x1_CommunityWallet_is_comm">CommunityWallet::is_comm</a>(payment.payee)) { <b>return</b> <b>false</b> }; // do nothing but don't delete instruction };
-
   <a href="Roles.md#0x1_Roles_assert_diem_root">Roles::assert_diem_root</a>(vm);
   <b>let</b> epoch = <a href="DiemConfig.md#0x1_DiemConfig_get_current_epoch">DiemConfig::get_current_epoch</a>();
   <b>let</b> account_bal = <a href="DiemAccount.md#0x1_DiemAccount_balance">DiemAccount::balance</a>&lt;<a href="GAS.md#0x1_GAS">GAS</a>&gt;(*account_addr);
@@ -687,7 +671,7 @@ Attempt to use a UID that is already taken
     };
 
     <b>if</b> (amount != 0 && amount &lt;= account_bal) {
-       <a href="DiemAccount.md#0x1_DiemAccount_vm_make_payment_no_limit">DiemAccount::vm_make_payment_no_limit</a>&lt;<a href="GAS.md#0x1_GAS">GAS</a>&gt;(
+       <a href="DiemAccount.md#0x1_DiemAccount_vm_pay_from">DiemAccount::vm_pay_from</a>&lt;<a href="GAS.md#0x1_GAS">GAS</a>&gt;(
             *account_addr, payment.payee, amount, b"autopay", b"", vm
           );
     };
@@ -807,16 +791,16 @@ Attempt to use a UID that is already taken
   payee: <b>address</b>,
   end_epoch: u64,
   amt: u64
-) <b>acquires</b> <a href="AutoPay.md#0x1_AutoPay_UserAutoPay">UserAutoPay</a>, <a href="AutoPay.md#0x1_AutoPay_AccountLimitsEnable">AccountLimitsEnable</a> {
+) <b>acquires</b> <a href="AutoPay.md#0x1_AutoPay_UserAutoPay">UserAutoPay</a> {
   <b>let</b> addr = <a href="../../../../../../../DPN/releases/artifacts/current/build/MoveStdlib/docs/Signer.md#0x1_Signer_address_of">Signer::address_of</a>(sender);
   // Confirm that no payment <b>exists</b> <b>with</b> the same uid
   <b>let</b> index = <a href="AutoPay.md#0x1_AutoPay_find">find</a>(addr, uid);
   <b>assert</b>!(<a href="../../../../../../../DPN/releases/artifacts/current/build/MoveStdlib/docs/Option.md#0x1_Option_is_none">Option::is_none</a>&lt;u64&gt;(&index), <a href="../../../../../../../DPN/releases/artifacts/current/build/MoveStdlib/docs/Errors.md#0x1_Errors_invalid_argument">Errors::invalid_argument</a>(<a href="AutoPay.md#0x1_AutoPay_UID_TAKEN">UID_TAKEN</a>));
 
-  // TODO: This check already <b>exists</b> at the time of execution.
-  <b>if</b> (<b>borrow_global</b>&lt;<a href="AutoPay.md#0x1_AutoPay_AccountLimitsEnable">AccountLimitsEnable</a>&gt;(@DiemRoot).enabled) {
-    <b>assert</b>!(<a href="CommunityWallet.md#0x1_CommunityWallet_is_comm">CommunityWallet::is_comm</a>(payee), <a href="../../../../../../../DPN/releases/artifacts/current/build/MoveStdlib/docs/Errors.md#0x1_Errors_invalid_argument">Errors::invalid_argument</a>(<a href="AutoPay.md#0x1_AutoPay_PAYEE_NOT_COMMUNITY_WALLET">PAYEE_NOT_COMMUNITY_WALLET</a>));
-  };
+  // // TODO: This check already <b>exists</b> at the time of execution.
+  // <b>if</b> (<b>borrow_global</b>&lt;<a href="AutoPay.md#0x1_AutoPay_AccountLimitsEnable">AccountLimitsEnable</a>&gt;(@DiemRoot).enabled) {
+  //   <b>assert</b>!(<a href="CommunityWallet.md#0x1_CommunityWallet_is_comm">CommunityWallet::is_comm</a>(payee), <a href="../../../../../../../DPN/releases/artifacts/current/build/MoveStdlib/docs/Errors.md#0x1_Errors_invalid_argument">Errors::invalid_argument</a>(PAYEE_NOT_COMMUNITY_WALLET));
+  // };
 
   <b>let</b> payments = &<b>mut</b> <b>borrow_global_mut</b>&lt;<a href="AutoPay.md#0x1_AutoPay_UserAutoPay">UserAutoPay</a>&gt;(addr).payments;
   <b>assert</b>!(