Skip to content

Build a Soroban instruction/resource-budget regression benchmark suite for all entrypoints #23

Description

@chonilius

Background / Context

None of the three contracts have any performance/compute-cost testing today — the 16 existing tests (per the README: 8 escrow, 4 milestones, 4 maintenance-pool) are purely correctness-focused. soroban_sdk::Env exposes a budget() API that reports actual CPU-instruction and memory consumption for contract invocations during tests, which is the standard way to measure real Soroban resource cost without needing a live network.

Problem Statement

Given several other issues in this bounty set identify potentially expensive operations (unbounded recipients vectors in release/release_issue, the O(n) Milestone.allocations Map read/write on every allocate/release_issue call, per-call TTL extension), the team has no way to detect a performance regression — a future change that unintentionally makes any entrypoint significantly more expensive — until it's discovered in production as failed/expensive mainnet transactions. This task is to build a proper, ongoing regression benchmark, not a one-off measurement.

Requirements

  • For every public entrypoint across all three contracts (initialize, fund, release, refund, get_escrow/get_admin/get_treasury/get_fee_bps for escrow; the equivalent set for milestones and maintenance-pool), write a benchmark test that invokes it under realistic conditions and captures env.budget()'s reported CPU instruction count and memory/footprint usage.
  • Parameterize the benchmarks across relevant size dimensions where applicable: recipients.len() for release/release_issue (e.g., 1, 5, 10, 20 recipients), allocations.len() at time of call for allocate/release_issue in milestones (e.g., after 1, 50, 200 prior allocations), and deposit_count for maintenance-pool operations — to produce actual cost-vs-size curves, not single data points.
  • Establish and document baseline numbers (current measured cost for each entrypoint/size combination) and wire up a mechanism (a make bench target producing a comparable report, or a test assertion with generous tolerance bounds that fails if cost exceeds e.g. 150% of baseline) that would catch a serious future regression.
  • Cross-reference findings with the unbounded-recipients and unbounded-allocations-Map issues tracked separately — this benchmark suite should produce the empirical data those issues call for, so coordinate to avoid duplicate measurement work, and feed findings into setting concrete caps/limits there.

Acceptance Criteria

  • Benchmark test harness added covering every public entrypoint in all three contracts
  • Size-parameterized cost curves produced for release/release_issue (by recipient count) and milestones' allocate/release_issue (by prior allocation count)
  • Baseline numbers documented (e.g., in a BENCHMARKS.md or README section)
  • make bench (or equivalent) target added and documented
  • Regression-detection mechanism in place (test assertions with tolerance, or a comparable-report diffing approach)

Technical Notes / Hints

  • env.budget() — check soroban_sdk's testutils documentation for exact API (Budget::cpu_instruction_cost, .memory_bytes_cost, or similar, depending on SDK version 26.1.0 pinned in this workspace's Cargo.toml).
  • Consider whether Soroban's Env::default() test environment's budget accounting is representative enough of real network costs to be trustworthy for this purpose, or whether it needs calibration against actual testnet transaction fee/resource-usage data (the deployed testnet contract IDs in the README could be used to gather real comparison data if stellar-cli/RPC access is available in your environment).

Difficulty Justification

Requires learning and correctly using Soroban-specific, less-commonly-documented budget/metering APIs, designing benchmarks that isolate genuine size-dependent cost scaling from noise, and building durable regression-detection tooling (not just a one-time measurement) — all while producing numbers precise and trustworthy enough to inform real engineering decisions (like the recipient-count cap) in other parts of this bounty set.

Metadata

Metadata

Assignees

No one assigned

    Labels

    GrantFox OSSIssue tracked in GrantFox OSSMaybe RewardedIssue may be eligible for a GrantFox rewardOfficial Campaign | FWC26Campaign: Official Campaign | FWC26performancePerformance/optimization issuetestingTesting/QA infrastructurevery hardVery difficult task, expert-level effort required

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions