enforce resource limits #1979

wanderingbort · 2018-03-30T00:41:30Z

This resolves #353 (eventually)

I wanted to start the PR process early while I finish up some of the interfacing and billing details

commits have more details about individual changes along the way

Still todo:

full-integration testing (only basic functional tests are present)
add an objective limit set by the system contract on the max cpu usage any transaction can use
add network cost set by system contract per read/write lock in a transaction
add base network cost set by system contract per transaction
add pre-committed cpu usage from context free actions
add discount set by system contract on all cpu usage associated with context free actions
add discount set by system contract on net usage from prunable data (signatures and context free data) enforced when checking the commitment to the packed size

@moskvanaft @zorba80 - I had to disable the call to set_resource_limits from the system contract because the values used in the test cases to exercise things like refunds and voting violate some basic principles of the resource system like:

its impossible to completely unstake all the ram for an account... where would we store the account itself
Unstaking all of the cpu_resources from an account leaves 0 capacity for delivering the refund action... maybe system floats some minimum stake to all accounts at all time to make stuff like this work out?
at all times, the account needs to maintain some stake, on occasion the voting tests would unstake too much and then not be able to proceed.

… processing to reflect the per-account pending

…vel" concepts like transactions from the basic resource interface to allow for future things that are outside the limited contexts offered by those types to affect resource consumption. add producer voted configuration (unexposed so far) for most of the magic numbers that were part of the system.

…ture/enforce-resource-limits

- Database row usage - Accounts - Permissions - Permission Links - Code - ABI - Deferred Transactions Fixed a few merge-issues from the prior merge

…s use - fixed accounting for actions in the transaction cpu usage - fixed names on some globals and made them configurable as part of chain_config - fixed more test cases, removed now unecessary misc_test for producer median voting (it's in contract code now)

- updated overhead for per-row after spot review - restored linear decay of the averaging when there are gaps in the incoming data. This is used to allow account usage to decay during missed blocks as it did previously - added comments to the calling code that explain the semantics of ordinal selection

- added accessors for budgeting and tests - fixed math to behave like the old algorithm while maintaining the new continuity across window size changes

… to match the rest of the system and imply its multiplier. Additionally changed to a VLQ encoded unsigned int. - change name and type of commitment to the cpu usage for the context_free portion of a transaction to match the rest of the system and imply its multiplier. Additionally changed to a VLQ encoded unsigned int. - exposed VLQ encoded unsigned int and ZigZag encoded signed integer types to the ABI built-ins as "varuint32" and "varint32" respectively - Fixed the unit tests that broke as a result of the above

… accounted for - add test cases for ram and block limits

… are failing because they are actually assigning resources to people

…s are straightened out... reverted tests to the minimal change to make them work

wanderingbort · 2018-03-30T00:44:32Z

libraries/chain/chain_controller.cpp

@@ -300,12 +308,13 @@ transaction_trace chain_controller::_push_transaction(const packed_transaction&

         time_point_sec execute_after = head_block_time();
         execute_after += time_point_sec(delay);
-         deferred_transaction dtrx(context.get_next_sender_id(), config::system_account_name, execute_after, trx);
+         //TODO: !!! WHO GETS BILLED TO STORE THE DELAYED TX?


who gets billed for storing auto-boxed transactions? the authorizer with the highest delay?

bill to highest delay

wanderingbort · 2018-03-30T00:46:18Z

libraries/chain/chain_controller.cpp

@@ -1449,6 +1474,28 @@ void chain_controller::update_signing_producer(const producer_object& signing_pr
   } );
 }

+void chain_controller::update_permission_usage( const transaction_metadata& meta ) {


this was refactored out of the old update_usage method and made independent.

wanderingbort · 2018-03-30T00:47:11Z

libraries/chain/include/eosio/chain/account_object.hpp

@@ -29,7 +29,6 @@ namespace eosio { namespace chain {
      shared_vector<char>  abi;

      void set_abi( const eosio::chain::contracts::abi_def& a ) {
-         // Added resize(0) here to avoid bug in boost vector container


I should restore this comment... not sure why I took it out

I mean.. am I the only one who is seriously bothered by this? @PaulCalabrese did we ever ask boost folks about this?

you are not the only one... but I can confirm that taking it out -OR- moving it all into the create call causes tests to fail in that same odd bug. I had tried to at least consolidate the abi and the code "hack" into the same logical place but it seems there is some corruption that can occur between create/constructor and future calls.

wanderingbort · 2018-03-30T00:48:30Z

libraries/chain/include/eosio/chain/contracts/contract_table_objects.hpp

@@ -94,7 +95,8 @@ namespace eosio { namespace chain { namespace contracts {
         table_id      t_id;
         uint64_t      primary_key;
         SecondaryKey  secondary_key;
-         account_name  payer;
+         account_name  payer = 0;
+         uint16_t      billed_overhead = 0;


Billed overhead should be removed, it is not in use

- added CFA cpu usage discount - added max transaction net and cpu usage - refactored trace data structures to resolve cyclic dependencies in header files - refactored accounting code in chain controller for a little more clarity - refactored lock processing and put them in the associated block/transaction traces

…meters

… less than the setcode penalty... the smartest contract is the one that does not have to exist - fixed incorrect usage of flat_set::erase

wanderingbort · 2018-03-30T18:28:12Z

contracts/eosio.system/delegate_bandwidth.hpp

@@ -262,7 +262,7 @@ namespace eosiosystem {
            act.owner = del.from;
            transaction out( now() + refund_delay + refund_expiration_time );
            out.actions.emplace_back( permission_level{ del.from, N(active) }, self, N(refund), act );
-            out.send( del.from, now() + refund_delay );
+            out.send( del.from, 0, now() + refund_delay );


remove implicit self pay in send deferred

wanderingbort · 2018-03-30T18:30:03Z

contracts/eosiolib/transaction.hpp

@@ -28,35 +28,36 @@ namespace eosio {
      :expiration(exp),region(r)
      {}

-      void send(uint64_t sender_id, time delay_until = 0) const {
+      void send(uint64_t sender_id, account_name payer = account_name(0), time delay_until = 0) const {


change to current_receiver()

wanderingbort · 2018-03-30T18:32:21Z

libraries/chain/apply_context.cpp

@@ -222,6 +225,10 @@ void apply_context::execute_deferred( deferred_transaction&& trx ) {
      FC_ASSERT( trx.execute_after < trx.expiration, "transaction expires before it can execute" );
      FC_ASSERT( !trx.actions.empty(), "transaction must have at least one action");

+      if (trx.payer != receiver) {


or sender is privileged.

wanderingbort · 2018-03-30T18:33:26Z

libraries/chain/apply_context.cpp

-   if( (delta > 0) && payer != account_name(receiver) ) {
-      require_authorization( payer );
+   if( (delta > 0) ) {
+      if (payer != account_name(receiver)) {


add or is privileged

wanderingbort · 2018-03-30T18:35:13Z

libraries/chain/apply_context.cpp

@@ -405,7 +417,8 @@ int apply_context::db_store_i64( uint64_t code, uint64_t scope, uint64_t table,
     ++t.count;
   });

-   update_db_usage( payer, buffer_size + 200 );
+   int64_t billable_size = (int64_t)(buffer_size + sizeof(key_value_object) + config::overhead_per_row_ram_bytes);
+   update_db_usage( payer, billable_size, "New Row ${id} in (${c},${s},${t})", _V("id", obj.primary_key)("c",receiver)("s",scope)("t",table));


for RC2 we need to worry about overhead for the variant construction

wanderingbort · 2018-03-30T18:45:53Z

libraries/chain/include/eosio/chain/authority.hpp

@@ -57,6 +57,10 @@ struct shared_authority {
      for( const auto& a : accounts ) { auth.accounts.emplace_back( a ); }
      return auth;
   }
+
+   size_t get_billable_size() const {
+      return keys.size() * sizeof(key_weight) + accounts.size() * sizeof(permission_level_weight);


account for compilers having different sizes for these and the database structures.

wanderingbort · 2018-03-30T18:55:03Z

libraries/chain/resource_limits.cpp

+   });
+}
+
+void resource_limits_manager::add_transaction_usage(const vector<account_name>& accounts, uint64_t cpu_usage, uint64_t net_usage, uint32_t time_slot ) {


new invariant, bill unique authorizers equally ... don't double up

wanderingbort · 2018-03-30T19:06:44Z

libraries/chain/wasm_interface.cpp

+       * @param cpu_weight - the weight for determining share of compute capacity
+       */
+      void set_resource_limits( account_name account, int64_t ram_bytes, int64_t net_weight, int64_t cpu_weight) {
+         EOS_ASSERT(ram_bytes >= -1 && ram_bytes <= INT64_MAX, wasm_execution_error, "invalid value for ram resource limit expected [-1,INT64_MAX]");


INT64_MAX is not an effective test on an int64_t

wanderingbort · 2018-03-30T19:07:27Z

libraries/chain/wasm_interface.cpp

+         account_name actual_payer = payer;
+         if (actual_payer != account_name(0)) {
+            const auto* paying_account = context.db.find<account_object, by_name>(payer);
+            EOS_ASSERT(paying_account, tx_unknown_argument, "The account for the payer: ${a}, does not exist!", ("a", payer));


remove this in favor of failing when trying to bill

wanderingbort · 2018-03-30T19:09:39Z

libraries/chain/chain_controller.cpp

@@ -300,12 +308,13 @@ transaction_trace chain_controller::_push_transaction(const packed_transaction&

         time_point_sec execute_after = head_block_time();
         execute_after += time_point_sec(delay);
-         deferred_transaction dtrx(context.get_next_sender_id(), config::system_account_name, execute_after, trx);
+         //TODO: !!! WHO GETS BILLED TO STORE THE DELAYED TX?


bill to highest delay

bytemaster · 2018-03-30T19:20:08Z

I went over this with bart, looks good, outstanding issues can be addressed in future pull request.

b1bart added 21 commits March 1, 2018 14:15

interim checkin to move workstations

811e4b7

interim checkin to move machines

700cf81

merging master @ 4be244f

f629efb

read from pending, mark as pending

b3275e7

adjust index to favor scraping all pending updates, update sync point…

95e9338

… processing to reflect the per-account pending

update update_usage for checking limits

7f269dd

interim checkin

40b5e61

change rate_limiting.*pp to resource_limits.*pp

0228d36

merge master @ 1a8bfec

9006a98

Merge branch 'feature/remove-unused-contract-database-types' into fea…

89e7da0

…ture/enforce-resource-limits

Plumb Ram billing for:

2cb7d65

- Database row usage - Accounts - Permissions - Permission Links - Code - ABI - Deferred Transactions Fixed a few merge-issues from the prior merge

fixes for bugs exposed during existing tests;

1e857a6

- added testing rig for resource_limits_manager

a04607a

- added accessors for budgeting and tests - fixed math to behave like the old algorithm while maintaining the new continuity across window size changes

- add enforcement of block maximums at the time that transactions are…

1177b6a

… accounted for - add test cases for ram and block limits

merging master at 895f207 and fixing lots of bugs. eosio system tests…

7dc7c91

… are failing because they are actually assigning resources to people

disable system contract from manipulating actual resources until test…

ab97074

…s are straightened out... reverted tests to the minimal change to make them work

merging master @ c721c31

4074a36

wanderingbort added the in progress label Mar 30, 2018

wanderingbort added this to the RC1 milestone Mar 30, 2018

wanderingbort commented Mar 30, 2018

View reviewed changes

b1bart added 3 commits March 30, 2018 09:39

- set and provide "reasonable" defaults for the new chain config para…

0fc72fe

…meters

- set more reasonabler defaults considering the max trx cpu usage was…

09d645b

… less than the setcode penalty... the smartest contract is the one that does not have to exist - fixed incorrect usage of flat_set::erase

b1bart added 7 commits March 30, 2018 10:11

- fixed typo in read lock accumulation

26effdc

merge master @ 38b43a9

3e43802

- added missing parameters to eosio_system_tests variant object

4e8a746

- enforce the commitment to the transaction size is accurate

99873b7

- enforce that the CFA cpu usage commitment is not violated

1cac203

estimate cf usage and add support in tests

6011010

- add support for estimating net and cf_cpu usage to cleos

f086c9a

wanderingbort added needs review and removed in progress labels Mar 30, 2018

wanderingbort commented Mar 30, 2018

View reviewed changes

bytemaster merged commit 1d9df77 into EOSIO:master Mar 30, 2018

gleehokie mentioned this pull request Mar 30, 2018

DAWN-396 ⁃ Create new message transaction rate limiting tests #1869

Closed

wanderingbort deleted the feature/enforce-resource-limits branch April 2, 2018 13:36

This was referenced Apr 2, 2018

Consolidated fixes for #1979 feedback #2031

Closed

Address resource usage authorization design. #2034

Closed

gleehokie added needs testing and removed needs testing labels Apr 2, 2018

gleehokie mentioned this pull request Oct 30, 2018

Complete testing of #1979 #2100

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

enforce resource limits #1979

enforce resource limits #1979

wanderingbort commented Mar 30, 2018 •

edited by taokayan

Loading

wanderingbort Mar 30, 2018

wanderingbort Mar 30, 2018

wanderingbort Mar 30, 2018

wanderingbort Mar 30, 2018

spoonincode Mar 30, 2018

wanderingbort Mar 30, 2018

wanderingbort Mar 30, 2018

wanderingbort Mar 30, 2018

wanderingbort Mar 30, 2018

wanderingbort Mar 30, 2018

wanderingbort Mar 30, 2018

wanderingbort Mar 30, 2018

wanderingbort Mar 30, 2018

wanderingbort Mar 30, 2018

wanderingbort Mar 30, 2018

wanderingbort Mar 30, 2018

wanderingbort Mar 30, 2018

bytemaster commented Mar 30, 2018

enforce resource limits #1979

enforce resource limits #1979

Conversation

wanderingbort commented Mar 30, 2018 • edited by taokayan Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bytemaster commented Mar 30, 2018

wanderingbort commented Mar 30, 2018 •

edited by taokayan

Loading