Gomu Gomu no Gatling enhancements — Test Framework and Test Suite

[hgergov]

Testing Framework Refinement

This section aims to describe the tasks related to refining the testing framework, test data preparation, test case definitions, and reporting. The overarching objective is to enable the execution of a comprehensive set of tests on a specified sequencer configuration, yielding not only benchmark results but also metrics crucial for informed decision-making and analysis. The ultimate aim is the establishment of automated workflows capable of seamless execution across diverse target environments and load/performance configurations.

To achieve this goal, it might be necessary to establish specific smart contracts for testing purposes, allowing for the simulation of more complex business logic and smart contract interactions. The inclusion of these elements would enhance the thoroughness of testing scenarios. Reporting mechanisms will be implemented to capture and present benchmark results and metrics in a clear and accessible manner, facilitating effective analysis.

Test Framework

• Extend benchmark testing suite
  • Implement a test set for more complex flows, potentially reusing existing smart contracts from Starknet mainnet
  • Develop a differential testing suite
• Tweak existing benchmark to use Goose
• Implement/Fork smart contracts that are required for the test suite. We might reuse already existing dApps from Starknet mainnet
• Implement files and runner for benchmark configs
  • Use Goose as the framework which configures, executes and records metrics
  • Develop files and runners for benchmark configurations.
  • Support multiple rate increases (linear, constant).
  • Implement predefined configs - Load, Stress, Differential read testing.
• Implement configuration files and runner for network configs
• Differentiate reporting and metrics for read and write metrics
• Add multithreading to send transactions by introducing Goose (GitHub Issue #27)
• Differentiate test suites, only for read use cases - by using multiple nodes and making sure that their responses match under a variety of RPC calls
• Read Metrics
  • UOPS (User ops per second) - The average user ops (Account Calls, L1/L2 messages, and Declares) per second
  • SPS (Steps per second) - average Cairo steps per second
  • TPS - transaction per second
  • Latency (s) - the time delay between the initiation of the transaction and its final confirmation
  • Error rate (already implemented)
• Write Metrics
  • UOPS (User ops per second) - The average user ops (Account Calls, L1/L2 messages, and Declares) per second
  • SPS (Steps per second) - average Cairo steps per second
  • TPS - transaction per second
  • Latency (s) - the time delay between the initiation of the transaction and its final confirmation
  • Error rate (already implemented)

CI/CD

• Github action
  • Setup
  • Test execution and reporting
  • Teardown

Documentation

Include comprehensive documentation for setup, configurations, and usage.

[AbdelStark]

Approved scope, ready for work. Assigned to you.
Grant: $10k + 20k STRK (paid through Only Dust upon completion and validation of the mission).