wu tuner

[SmaGMan]

Added work units tuner service. It is responsible for reporting wu metrics, anchors lag and wu params. And for adjusting wu params. Service behavior is managed by a separate configuration file wu-tuner-config.json that should be placed in the same directory where the node configuration file is situated. Without the configuration file the default options for wu tuner service will be applied. It will only report to metrics but will not run adjustments.

Example wu-tuner-config.json

{
    "wu_span": 10,  // wu metrics span for calculating avg (in blocks count)
    "wu_ma_interval": 4,  // wu metrics avg calculation interval wu_ma_interval * wu_span = 40
    "wu_ma_range": 400, // the range of wu metrics spans to calculate avg
    "lag_span": 10, // anchors lag span
    "lag_ma_interval": 4, // anchors lag avg calculation interval lag_ma_interval * lag_span = 40
    "lag_ma_range": 400, // the range of anchors lag spans to calculate avg
    "wu_params_ma_interval": 200, // interval of calculating target wu params from avg wu metrics
    "tune_interval": 1000, // interval of wu params adjustment
    "tune": {
        "type": "No" // No - do not execute adjustment
    },
    "lag_bounds_ms": [ // calculate target wu params if anchors lag is out of bounds
        400,
        4500
    ],
    "target_wu_price": 75, // manual target wu price
    "adaptive_wu_price": true // if true, then service will calculate target wu price according to lag delta from bounds
}

If we want node to perform adjustment then we should set

    "tune": {
        "type": "Rpc",
        "rpc": "http://ip:8080/rpc",
        "secret": "{blockchain master key}"
    },

Now service supports adjustment via rpc using master key. So the main case is to use service for the wu params adjustment in test networks on one trusted node.

It was assumed that it was sufficient to calculate the target wu params aiming to minimize the anchors lag. But the bunch of tests showed that it is not correct. Under different load profiles, different lag bounds perform best. And seems like we need to consider not only the anchor lag but also other metrics, to decide when to execute adjustment.

So the service will be improved in further tasks.

Now it was used in semi-manual mode to define the optimal wu params on devnet 2.

Target wu params for empty state were calculated with fixed target wu price 0.75

    "work_units_params": {
        "prepare": {
            "fixed_part": 1000000,
            "msgs_stats": 0,
            "remaning_msgs_stats": 0,
            "read_ext_msgs": 510,
            "read_int_msgs": 2000,
            "read_new_msgs": 890,
            "add_to_msg_groups": 85
        },
        "execute": {
            "prepare": 57000,
            "execute": 4000,
            "execute_err": 0,
            "execute_delimiter": 1000,
            "serialize_enqueue": 95,
            "serialize_dequeue": 95,
            "insert_new_msgs": 95,
            "subgroup_size": 16
        },
        "finalize": {
            "build_transactions": 150,
            "build_accounts": 155,
            "build_in_msg": 130,
            "build_out_msg": 130,
            "serialize_min": 2500000,
            "serialize_accounts": 2750,
            "serialize_msg": 2750,
            "state_update_min": 1000000,
            "state_update_accounts": 700,
            "state_update_msg": 425,
            "create_diff": 920,
            "serialize_diff": 105,
            "apply_diff": 2600,
            "diff_tail_len": 620
        }
    }

Target wu params for 10kk state were calculated with fixed target wu price 0.6

    "work_units_params": {
        "prepare": {
            "fixed_part": 1000000,
            "msgs_stats": 0,
            "remaning_msgs_stats": 0,
            "read_ext_msgs": 662,
            "read_int_msgs": 2715,
            "read_new_msgs": 1102,
            "add_to_msg_groups": 109
        },
        "execute": {
            "prepare": 57000,
            "execute": 9957,
            "execute_err": 0,
            "execute_delimiter": 1000,
            "serialize_enqueue": 114,
            "serialize_dequeue": 114,
            "insert_new_msgs": 114,
            "subgroup_size": 16
        },
        "finalize": {
            "build_transactions": 198,
            "build_accounts": 238,
            "build_in_msg": 193,
            "build_out_msg": 185,
            "serialize_min": 2500000,
            "serialize_accounts": 5435,
            "serialize_msg": 5435,
            "state_update_min": 1000000,
            "state_update_accounts": 643,
            "state_update_msg": 425,
            "create_diff": 1340,
            "serialize_diff": 105,
            "apply_diff": 3858,
            "diff_tail_len": 1510
        }
    }

Pull Request Checklist

NODE CONFIGURATION MODEL CHANGES

[None]
But added optional config file wu-tuner-config.json

BLOCKCHAIN CONFIGURATION MODEL CHANGES

[None]

---

COMPATIBILITY

Affected features:

• [Collator. Work Units Calculation] - [incompatible]

On the not empty internals queue it may cause the incorrect refill logic execution.

SPECIAL DEPLOYMENT ACTIONS

[Required]

So we must ensure the queue is empty during the update of 2/3+1 nodes (by stake).

Updated nodes will produce mismatched blocks.

Needs to update Grafana dashboard.

Required to update .params.28 in blockchain config

{
    "shuffle_mc_validators": true,
    "mc_block_min_interval_ms": 2500,
    "empty_sc_block_interval_ms": 60000,
    "max_uncommitted_chain_length": 31,
    "wu_used_to_import_next_anchor": 1200000000,
    "msgs_exec_params": {
        "buffer_limit": 10000,
        "group_limit": 100,
        "group_vert_size": 10,
        "externals_expire_timeout": 58,
        "open_ranges_limit": 20,
        "par_0_int_msgs_count_limit": 100000,
        "par_0_ext_msgs_count_limit": 10000000,
        "group_slots_fractions": {
            "0": 80,
            "1": 10
        },
        "range_messages_limit": 10000
    },
    "work_units_params": {
        "prepare": {
            "fixed_part": 1000000,
            "msgs_stats": 0,
            "remaning_msgs_stats": 0,
            "read_ext_msgs": 662,
            "read_int_msgs": 2715,
            "read_new_msgs": 1102,
            "add_to_msg_groups": 109
        },
        "execute": {
            "prepare": 57000,
            "execute": 9957,
            "execute_err": 0,
            "execute_delimiter": 1000,
            "serialize_enqueue": 114,
            "serialize_dequeue": 114,
            "insert_new_msgs": 114,
            "subgroup_size": 16
        },
        "finalize": {
            "build_transactions": 198,
            "build_accounts": 238,
            "build_in_msg": 193,
            "build_out_msg": 185,
            "serialize_min": 2500000,
            "serialize_accounts": 5435,
            "serialize_msg": 5435,
            "state_update_min": 1000000,
            "state_update_accounts": 643,
            "state_update_msg": 425,
            "create_diff": 1340,
            "serialize_diff": 105,
            "apply_diff": 3858,
            "diff_tail_len": 1510
        }
    }
}

---

PERFORMANCE IMPACT

[Expected impact]

Wu calculation for the state related actions updated and will produce more accurate result when state is growing.

---

TESTS

Unit Tests

[No coverage]

Network Tests

[No coverage]

Manual Tests

Target wu params calculation for empty state
metrics

Comparation of current default and calculated optimal params
metrics

• default wu params from master-5
  • deploy 10kk
  • transfers 20k
  • transfers 10k
• optimal wu params for empty state and target wu price 0.75
  • deploy 10kk
  • transfers 20k
  • transfers 10k
• optimal wu params for 10kk state and target wu price 0.6
  • deploy 10kk
  • transfers 20k
  • transfers 10k
  • swaps 1200
  • swaps 1500 (queue growth)
  • swaps 1300 (ok)

[github-actions]

🧪 Network Tests

To run network tests for this PR, use:

gh workflow run network-tests.yml -f pr_number=817

Available test options:

• Run all tests: gh workflow run network-tests.yml -f pr_number=817
• Run specific test: gh workflow run network-tests.yml -f pr_number=817 -f test_selection=ping-pong

Test types: destroyable, ping-pong, one-to-many-internal-messages, fq-deploy, nft-index, persistent-sync

Results will be posted as workflow runs in the Actions tab.

[codecov]

Codecov Report

❌ Patch coverage is 5.72289% with 1252 lines in your changes missing coverage. Please review.
✅ Project coverage is 46.03%. Comparing base (f1e572c) to head (46abd4e).
⚠️ Report is 25 commits behind head on master.

Files with missing lines	Patch %	Lines
collator/src/collator/do_collate/work_units.rs	7.70%	911 Missing ⚠️
util/src/num/safe_avg.rs	0.00%	96 Missing ⚠️
util/src/num/median.rs	0.00%	80 Missing ⚠️
cli/src/util/rpc_wu_updater.rs	0.00%	30 Missing ⚠️
cli/src/cmd/tools/gen_zerostate.rs	0.00%	29 Missing ⚠️
collator/src/collator/do_collate/finalize.rs	0.00%	23 Missing ⚠️
collator/src/types.rs	0.00%	19 Missing ⚠️
collator/src/collator/do_collate/execute.rs	0.00%	14 Missing ⚠️
collator/src/collator/execution_manager.rs	0.00%	14 Missing ⚠️
cli/src/cmd/tools/bc.rs	0.00%	10 Missing ⚠️
... and 5 more

Additional details and impacted files

@@            Coverage Diff             @@
##           master     #817      +/-   ##
==========================================
- Coverage   46.82%   46.03%   -0.80%     
==========================================
  Files         317      320       +3     
  Lines       56395    57461    +1066     
  Branches    56395    57461    +1066     
==========================================
+ Hits        26409    26451      +42     
- Misses      28621    29647    +1026     
+ Partials     1365     1363       -2     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.