[PH] Feature performance harness stage 2 development

[oschwaldp-oci]

Major features

Develop performance tests to establish thread count recommendations (net, chain, producer) 510

Add options to calculate recommended producer, chain, and net worker thread pool size.

Each plugin's option determines whether to calculate number of worker threads to use in the thread pool with options of: none, lmax, or full. In none mode, the default, no calculation will be attempted and the configured thread count will be used. In lmax mode, thread count will incrementally be tested, starting at plugin default, until the performance rate ceases to increase with the addition of subsequent threads. In full mode thread count will incrementally be tested from plugin default..num logical processors, recording each performance and choosing the local max performance (same value as would be discovered in lmax mode). Useful for graphing the full performance impact of each available thread.

Expand Details Thread Count Recommendations

New Arguments

• --calc-producer-threads {none,lmax,full}
  Determines whether to calculate number of worker threads to use in producer thread pool ("none", "lmax", or "full").
  In "none" mode, the default, no calculation will be attempted and the configured --producer-threads value will be used.
  In "lmax" mode, producer threads will incrementally be tested, starting at plugin default, until the performance rate ceases to increase with the addition of additional threads.
  In "full" mode producer threads will incrementally be tested from plugin default..num logical processors, recording each performance and choosing the local max performance (same value as would be discovered in "lmax" mode). Useful for graphing the full performance impact of each available thread. (default: none)

• --calc-chain-threads {none,lmax,full}
  Determines whether to calculate number of worker threads to use in chain thread pool ("none", "lmax", or "full").
  In "none" mode, the default, no calculation will be attempted and the configured --chain-threads value will be used.
  In "lmax" mode, producer threads will incrementally be tested, starting at plugin default, until the performance rate ceases to increase with the addition of additional threads.
  In "full" mode producer threads will incrementally be tested from plugin default..num logical processors, recording each performance and choosing the local max performance (same value as would be discovered in "lmax" mode). Useful for graphing the full performance impact of each available thread. (default: none)

• --calc-net-threads {none,lmax,full}
  Determines whether to calculate number of worker threads to use in net thread pool ("none", "lmax", or "full").
  In "none" mode, the default, no calculation will be attempted and the configured --net-threads value will be used.
  In "lmax" mode, producer threads will incrementally be tested, starting at plugin default, until the performance rate ceases to increase with the addition of additional threads.
  In "full" mode producer threads will incrementally be tested from plugin default..num logical processors, recording each performance and choosing the local max performance (same value as would be discovered in "lmax" mode). Useful for graphing the full performance impact of each available thread. (default: none)

Added option to opt out of running the tps test.

• --skip-tps-test Determines whether to skip the max TPS measurement tests (default: False)

Updated thread configurations based on test recommendations

• --producer-threads PRODUCER_THREADS
  Number of worker threads in producer thread pool (default: 6)
• --chain-threads CHAIN_THREADS
  Number of worker threads in controller thread pool (default: 3)

Add Ability to Specify Contract for Performance Tests 598

Use EOSMechanics contract with Performance Test 713

Support configuring transaction type to send in trx generator through JSON transaction description 626

Expand Details JSON Transaction Description Support

JSON Transaction Description Support

Add the ability to configure the type of transaction to generate/send from a JSON transaction description (file, str, etc.)
Added functionality to transaction generator to take input of abi from file, transaction action type, and action data from a string or file json description and generate transactions matching that description.

Transaction Generator now takes additional arguments:

• action-name The action name applied to the provided action data input
• action-data The path to the json action data file or json action data description string to use
• abi-file The path to the contract abi file to use for the supplied transaction action data

Transaction Generator refactor base class

• Refactored the transaction generator to push reusable functionality into a transaction_generator_base base class.

launch_transaction_generators.py also now supports:

• action_name The action name applied to the provided action data input
• action_data The path to the json action data file or json action data description string to use
• abi_file The path to the contract abi file to use for the supplied transaction action data

performance_test_basic.py now supports:

• --user-trx-data-file Path to userTrxData.json

Other Changes/Additions:

Rename handler_account to contract_owner_account for more accurate description.

Add integration test exercising the new functionality that builds transactions from abi, action type and action data. Makes use of the performance_test_basic with new argument --user-trx-data-file' and provided example userTrxData.json, but generates the transfer transactions from the abi, action name and action description in the test.

Added support to Cluster.py to allow populating wallet and creating accounts with specific account names for testing purposes (in particular to use with the --user-trx-data-file where a user would like to have foreknowledge about account names to use in action data).

Transaction Generator support generating newaccount transactions 702

Expand Details New Account Transaction Support

Expand transaction generator to support new account trx

• Update user data transaction specification to support multiple actions and automatically generating an account namer per transaction to substitute into defined actions using the 'ACCT_PER_TRX' key word.
• Removed arguments for action data pieces that have been moved into the larger transaction spec in the json files.
• Added/Updated tests to exercise the new feature.

Additional work:

• Add way to specify auth acct when using json trx description.
• Add support for actionAuthAcct and actionAuthPrivKey for more explicit control when specifying transaction directly.
• Add actionAuthAcct to json transaction description
• Add trx generator id to allow differentiation of trx generators for use in create accounts.
• Factored out set_transaction_headers into own helper function.
• Add ability to configure log level for cluster via cli arg.

Additional Work

New Configuration Items

• --chain-threads CHAIN_THREADS Number of worker threads in controller thread pool (default: 2) 469

• --net-threads NET_THREADS Number of worker threads in net_plugin thread pool (default: 2) 469

• --producer-threads PRODUCER_THREADS Number of worker threads in producer thread pool (default: 2) 469

• --database-map-mode {mapped,heap,locked} 481
  Database map mode ("mapped", "heap", or "locked").
  In "mapped" mode database is memory mapped as a file.
  In "heap" mode database is preloaded in to swappable memory and will use huge pages if available.
  In "locked" mode database is preloaded, locked in to memory, and will use huge pages if available. (default: mapped)

• --http-max-bytes-in-flight-mb HTTP_MAX_IN_FLIGHT_BYTES
  Maximum size in megabytes http_plugin should use for processing http requests. -1 for unlimited.
  429 error response when exceeded. (default: -1) 740

• --wasm-runtime RUNTIME
  Override default WASM runtime ("eos-vm-jit", "eos-vm")
  "eos-vm-jit" : A WebAssembly runtime that compiles WebAssembly code to native x86 code prior to
  execution. "eos-vm" : A WebAssembly interpreter. (default: eos-vm-jit) 740

• --contracts-console print contract's output to console (default: False) 740

• --eos-vm-oc-cache-size-mb CACHE_SIZE_MiB
  Maximum size (in MiB) of the EOS VM OC code cache (default: 1024) 740

• --eos-vm-oc-compile-threads COMPILE_THREADS
  Number of threads to use for EOS VM OC tier-up (default: 1) 740

• --non-prods-eos-vm-oc-enable
  Enable EOS VM OC tier-up runtime on non producer nodes (default: False) 740

• --block-log-retain-blocks BLOCKS_TO_RETAIN
  If set to greater than 0, periodically prune the block log to
  store only configured number of most recent blocks. If set to 0, no blocks are be written to the block log;
  block log file is removed after startup. (default: None) 740

• --http-threads HTTP_THREADS
  Number of worker threads in http thread pool (default: 2) 740

Override some chainPlugin and httpPlugin config options for performance test, but not exposed on cli 740

• maxTransactionTime (max-transaction-time)
• abiSerializerMaxTimeMs (abi-serializer-max-time-ms)
• httpMaxBytesInFlightMb (http-max-bytes-in-flight-mb)
• httpMaxResponseTimeMs (http-max-response-time-ms)
• maxClients (max-clients)
• resourceMonitorNotShutdownOnThresholdExceeded (resource-monitor-not-shutdown-on-threshold-exceeded)

Cluster updates for config items 740

Previously hardcoded in cluster, now max-transaction-time, abi-serializer-max-time-ms, p2p-max-nodes-per-host, and max-clients can be dynamically configured and passed to nodeos via cluster without causing failures.

Architecture Rework

• Make use of ArgumentParser parents and groups. 518
  This will greatly reduce work to bubble up arguments between scripts/modules. Also brings clarity and grouping of arguments by use with group titles and descriptions.

  Test Helper arguments can now be bubbled up to user scripts via direct access to the ArgumentParser. Performance Test Basic makes use of Test Helper arguments. Perfomance Test makes use of both Performance Test Basic and Test Helper arguments as well.

• Provide *PluginArgs dataclasses to capture and validate nodeos config options. 546

  Each *PluginArgs dataclass captures the command line argument and default for each option as specified in nodeos.

  New test validate_nodeos_plugin_args.py uses nodeos's --help command to parse out each plugin's available options and defaults to validate current status vs. the *PluginArgs classes. Any addition, removal, update will cause a test failure with note of what has changed. This will allow easy maintenance going forward.

  Consequently, quietly defaulted nodeos arguments will now have representation in the performance harness test reports.

• Replace usage of os.path with Path and PurePath 571

• Add TransactionGeneratorLauncher to TestHelper Shutdown 731

• Fix potential duplicate transactions from transaction generators 757 & 7075447

Report Rework

• Added logical cpu count to top level performance_test.py report.

• Add target tps number to test log dir name for ease of locating specific run. 484

  currently log directories are timestamp only:

  • performance_test_basic/2022-11-16_22_26_48

  proposing the addition of the tps target to the dir name to make finding a specific run easier:

  • performance_test_basic/2022-11-16_22_26_48-50000
  • performance_test_basic/2022-11-16_22_28_25-25000

• Log Number of Forks and Forked Blocks 618

• Log Dropped Blocks, Dropped Transactions, and Production Windows 633

• Additional tracking of block time, and stores it and latency inside as CSV for later consumption. 647

• Capture Full Command Line Argument String in Report 720

PRs included

469
475
481
484
510
518
520
546
571
598
610
611
618
626
633
647
702
713
720
731
734
740
757
761
762
766
769
771
778

[ScottBailey]

Please address comment/question regarding Python 3.11.

Edit: this is resolved.

[oschwaldp-oci]

use default_factory

Addressed: 045db81

[oschwaldp-oci]

Do we expect this to work for Ubuntu 23.04? If so, maybe we should test against python 3.11? Here's the output from my Debian box:

beauty$ tests/performance_tests/performance_test_basic.py
Traceback (most recent call last):
  File "/home/bailey/eos/leap/Build-gcc/tests/performance_tests/performance_test_basic.py", line 25, in <module>
    class PerformanceTestBasic:
  File "/home/bailey/eos/leap/Build-gcc/tests/performance_tests/performance_test_basic.py", line 50, in PerformanceTestBasic
    class ClusterConfig:
  File "/home/bailey/eos/leap/Build-gcc/tests/performance_tests/performance_test_basic.py", line 51, in ClusterConfig
    @dataclass
     ^^^^^^^^^
  File "/usr/lib/python3.11/dataclasses.py", line 1220, in dataclass
    return wrap(cls)
           ^^^^^^^^^
  File "/usr/lib/python3.11/dataclasses.py", line 1210, in wrap
    return _process_class(cls, init, repr, eq, order, unsafe_hash,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/dataclasses.py", line 958, in _process_class
    cls_fields.append(_get_field(cls, name, type, kw_only))
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/dataclasses.py", line 815, in _get_field
    raise ValueError(f'mutable default {type(f.default)} for field '
ValueError: mutable default <class 'NodeosPluginArgs.ChainPluginArgs.ChainPluginArgs'> for field chainPluginArgs is not allowed: use default_factory

It's not clear to me if this is a 3.11 issue or some mis-configuration on this machine.

Addressed: 045db81

[ScottBailey]

use default_factory

Addressed: 045db81

Thanks!

[ScottBailey]

Thanks!