Skip to content

Debugging and Performance Profiling

Jump to bottom
RunTimeAdmin edited this page Oct 30, 2026 · 19 revisions

Debugging & Performance Profiling

**Referenced Files in This Document** - [foundry.toml](https://github.com/RunTimeAdmin/Sigvara/blob/main/foundry.toml) - [README.md](https://github.com/RunTimeAdmin/Sigvara/blob/main/README.md) - [E2E.t.sol](https://github.com/RunTimeAdmin/Sigvara/blob/main/test/E2E.t.sol) - [SigvaraStaking.t.sol](https://github.com/RunTimeAdmin/Sigvara/blob/main/test/SigvaraStaking.t.sol) - [Deploy.s.sol](https://github.com/RunTimeAdmin/Sigvara/blob/main/script/Deploy.s.sol) - [index.js](https://github.com/RunTimeAdmin/Sigvara/blob/main/oracle/index.js) - [metrics.js](https://github.com/RunTimeAdmin/Sigvara/blob/main/oracle/metrics.js) - [metrics.test.js](https://github.com/RunTimeAdmin/Sigvara/blob/main/oracle/metrics.test.js) - [Dockerfile](https://github.com/RunTimeAdmin/Sigvara/blob/main/oracle/Dockerfile)

Table of Contents

  1. Introduction
  2. Project Structure
  3. Core Components
  4. Architecture Overview
  5. Detailed Component Analysis
  6. Dependency Analysis
  7. Performance Considerations
  8. Troubleshooting Guide
  9. Conclusion
  10. Appendices

Introduction

This document provides a complete debugging and performance profiling guide for Sigvara development across on-chain Solidity contracts, Foundry tooling, and the Node.js Oracle service. It covers:

  • Solidity debugging with Foundry: unit tests, fuzzing, invariant testing, verbose traces, script diagnostics, and deployment artifacts.
  • Transaction tracing and state inspection using Foundry’s EVM primitives and test helpers.
  • Performance profiling for gas usage, execution time measurement, RPC pressure, and memory/cache behavior.
  • Oracle debugging: Node.js logging, log analysis, metric monitoring, and operational safeguards.
  • Practical examples for identifying bottlenecks, optimizing gas consumption, and resolving common issues.
  • Profiling strategies for both on-chain and off-chain components, with concrete commands and files referenced from the repository.

Project Structure

The repository is organized into clear layers:

  • On-chain contracts under src/, with tests under test/.
  • Foundry configuration under foundry.toml.
  • Deployment scripts under script/.
  • The Oracle service under oracle/, including HTTP endpoints, chain readers, scoring logic, persistence, metrics, and Docker packaging.
  • SDK and site assets are present but not central to this debugging/profiling guide.
graph TB
subgraph "On-chain"
Contracts["Solidity contracts<br/>src/*"]
Tests["Foundry tests<br/>test/*"]
Scripts["Deployment scripts<br/>script/*"]
Config["Foundry config<br/>foundry.toml"]
end
subgraph "Off-chain"
Oracle["Oracle service<br/>oracle/index.js"]
Metrics["Prometheus-style metrics<br/>oracle/metrics.js"]
Docker["Container build<br/>oracle/Dockerfile"]
end
Config --> Tests
Scripts --> Contracts
Tests --> Contracts
Oracle --> Contracts
Metrics --> Oracle
Docker --> Oracle
Loading

Diagram sources

Section sources

Core Components

This section maps the key debugging and profiling entry points:

  • Foundry configuration and optimizer settings control compilation, fuzz runs, and RPC endpoints.
  • Contract tests exercise lifecycle flows, slashing, reputation updates, and upgrade safety.
  • Oracle index orchestrates epochs, payment scanning, scoring, divergence detection, and metrics reporting.
  • Metrics module exposes counters and gauges for epoch health, proposal/finalization outcomes, payment scans, checker comparisons, and cache statistics.
  • Dockerfile documents build reproducibility and source inclusion practices that affect debugging in production containers.

Section sources

Architecture Overview

The debugging and profiling architecture spans three layers:

  • On-chain layer: Solidity contracts expose state and events; Foundry tests simulate full lifecycles and invariant properties.
  • Script layer: Deployment scripts write artifacts only during broadcast, preventing accidental dry-run artifacts. The write happens when forge runs the script locally, before the transactions confirm, so a failed broadcast can leave a file naming contracts that do not exist; check the chain with scripts/record-implementations.mjs before trusting it, and do not use --resume.
  • Off-chain layer: The Oracle service reads chain state, computes scores, proposes/finalizes reputation, and exposes Prometheus-compatible metrics.
sequenceDiagram
participant Dev as "Developer"
participant Forge as "Foundry (forge)"
participant EVM as "In-memory EVM"
participant Chain as "Arc RPC"
participant Oracle as "Oracle Service"
participant Metrics as "Metrics Export"
Dev->>Forge : Run tests / scripts
Forge->>EVM : Execute contract logic
EVM-->>Dev : Logs, traces, assertions
Dev->>Forge : Deploy or simulate scripts
Forge->>Chain : Broadcast or read
Chain-->>Forge : Receipts / state
Oracle->>Chain : Read agents, scores, fees
Oracle->>Oracle : Compute score + scan payments
Oracle->>Chain : Propose / finalize
Oracle->>Metrics : Increment counters / set gauges
Metrics-->>Dev : Prometheus text exposition
Loading

Diagram sources

Detailed Component Analysis

Solidity Debugging with Foundry

Foundry is used for building, testing, fuzzing, and scripting. Key configuration includes:

  • Source/output directories, libraries, remappings, optimizer settings, solc version, and RPC endpoints.
  • Fuzz profiles for default and CI runs.
  • Filesystem permissions for deployments artifact writing.

Practical debugging techniques:

  • Unit tests: run all contract tests in an in-memory EVM without network access.
  • Invariant tests: drive randomized call sequences to assert protocol invariants.
  • E2E integration tests: cover full agent lifecycle flows with fixture contracts.
  • Verbose output: use -vvv to inspect detailed trace information for failing tests.
  • Script diagnostics: deploy scripts print context and avoid writing artifacts during dry runs.

Recommended commands and references:

  • Build and test: forge build, forge test
  • CI fuzz intensity: FOUNDRY_PROFILE=ci forge test
  • Invariant suite: forge test --match-contract StakingInvariantTest
  • Operator bond invariants: forge test --match-contract OracleBondInvariantTest
  • Score store, fee registry and identity registry invariants: forge test --match-contract ReputationInvariantTest, EpochFeesInvariantTest and IdentityInvariantTest
  • E2E suite with verbose traces: forge test --match-contract E2EIntegrationTest -vvv
  • Deploy-script and handover tests: forge test --match-contract DeployScriptGuardsTest, DeployScriptRunTest and SplitAuthorityTest

Section sources

Transaction Tracing and State Inspection

Foundry’s EVM environment enables deep inspection:

  • Time manipulation: vm.warp advances block timestamps to test challenge windows and unbonding periods.
  • Address/key generation: makeAddr, makeAddrAndKey create deterministic actors for tests.
  • Storage slot pinning: tests load storage slots to verify upgrade-safe layout and prevent silent misalignment.
  • Reentrancy guard slot checks: ensure inherited guards do not shift sequential storage.

Use these patterns to:

  • Reproduce time-sensitive flows like slash challenges and finalization windows.
  • Validate that upgrades preserve mapping base slots.
  • Confirm reentrancy protection does not alter storage layout.

Section sources

Gas Analysis and Optimization

Gas optimization opportunities visible in the codebase:

  • Batched JSON-RPC calls: ethers batches concurrent calls issued in the same tick, reducing round trips.
  • Chunked processing: phase-one and phase-two loops process agents in chunks to avoid RPC bursts.
  • Guard reads before writes: cheap pre-checks reduce reverted proposals and wasted gas.
  • Optimizer settings: Foundry enables optimizer with configured runs.

Guidance:

  • Prefer concurrent independent reads within a single tick to leverage batching.
  • Use chunking to keep RPC pressure flat as agent count grows.
  • Keep guard reads close to writes to bound stale-state windows.
  • Tune optimizer runs based on trade-offs between compile time and deployed gas costs.

Section sources

Execution Time Measurement

Execution timing is measured at multiple levels:

  • Epoch start timestamp: logs epoch start and increments epoch counters.
  • Chain clock usage: uses block timestamps instead of wall clocks to ensure deterministic scoring and operator convergence.
  • Process uptime gauge: exposed via metrics for long-running observability.

Operational tips:

  • Log epoch boundaries and failures to detect skipped or overlapping epochs.
  • Use chain timestamps consistently to avoid drift-induced divergence.
  • Monitor uptime and last successful epoch timestamp to detect stalls.

Section sources

Memory Usage and Cache Behavior

The Oracle service tracks evidence response cache statistics:

  • Hits, misses, evictions, entries, and serialized bytes.
  • Optional exposure when evidence cache stats are supplied to metrics exposition.

The state file is the other large allocation. It is written in pieces of about 1 MB (state-serialize.js), and the peak memory of one save was measured on Node 20 at 189 MB with 100,000 payments (a 38 MB file) against 449 MB before, 280 MB against 700 MB with 200,000, and 395 MB against 1,628 MB with 400,000. The mainnet compose file limits a container to 768 MB, which the old save would have passed at roughly 225,000 payments. Stored payment events are frozen, and merkle.js caches each leaf and each agent's root by identity; on 5,000 events, building the leaves became about three times faster (1,450 ms to 480 ms by the figures in merkle.js) with the encoding written by hand, and a cache hit costs a lookup.

Profiling approach:

  • Observe hit rate to determine whether cache is effective.
  • Track entries and bytes to monitor memory growth.
  • Size the container for the peak of a save, not the size of the state file.
  • Correlate cache misses with RPC errors and badge request volume.

Section sources

Oracle Service Debugging

The Oracle service is the primary off-chain component to debug:

  • Configuration validation: required env vars and epoch interval parsing fail fast at startup, and so does any numeric setting that is set and unusable (a blank value means unset), a wrong ORACLE_MODE, a malformed signing key (never printed), and an RPC on a chain other than EXPECTED_CHAIN_ID.
  • Checker mode safeguards: prevents fee charging and enforces divergence tolerance.
  • Epoch loop guards: prevents overlapping epochs and handles transient RPC failures gracefully. An epoch does not run while the chain's clock is more than MAX_CLOCK_SKEW_SECONDS from the host's.
  • Payment scanning: backfills transfers, credits evidence, and runs canary checks to validate query correctness.
  • Divergence detection: compares checker vs primary scores, records evidence roots, and surfaces actionable details.

Debugging workflow:

  • Validate environment variables and epoch intervals before starting.
  • Inspect epoch logs for skipped triggers, failed scans, and throttled writes.
  • Review divergence records to identify evidence mismatches or arithmetic disagreements.
  • Read /health: commit says which code is running, stateSource says whether the state was recovered from its backup, and chainId beside expectedChainId says which chain the RPC is on.
  • Use metrics to track propose/finalize success rates, payment scan results, and checker agreement/divergence.

Section sources

Metric Monitoring and Observability

The metrics module provides Prometheus-style exposition:

  • Counters: epochs, proposals, finalizations, attestations, payment scans, checker comparisons, throttling, refused or unconfirmed transactions, epoch-fee charges, HTTP requests.
  • Gauges: last successful epoch timestamp, active agents, evidence cache size and bytes.
  • Static registry enforcement: every metric name used must be declared; undeclared names are no-ops, which CI catches by scanning source files.

Monitoring strategy:

  • Alert on epoch failures, payment scan canary failures, and high propose/finalize error rates.
  • Watch checker divergence counts and evidence cache miss ratios.
  • Ensure all metric names are declared to avoid silent zero counters.

Section sources

Practical Examples

Identifying Bottlenecks

  • RPC burst mitigation: observe phase-one and phase-two chunk sizes and batched calls.
  • Payment scan performance: check paymentsPulled, paymentScanErrors, and canary failure counts.
  • Score computation cost: note payer score caching and concurrency comments describing measured improvements.

Section sources

Optimizing Gas Consumption

  • Reduce reverted proposals by validating pending status and bonding before proposing.
  • Use chain clock and pure scoring functions to avoid unnecessary recomputation.
  • Leverage Foundry optimizer settings for compiled contract efficiency.

Section sources

Resolving Common Development Issues

  • Dry-run artifact leakage: deploy scripts skip writing artifacts unless broadcasting.
  • Undeclared metrics: CI test scans source files to ensure every metric name is registered.
  • Epoch interval misconfiguration: startup validation refuses invalid values to prevent runaway scheduling.

Section sources

Dependency Analysis

The following diagram shows core dependencies among debugging and profiling components:

graph LR
FoundryConfig["foundry.toml"] --> Tests["test/*"]
Tests --> Contracts["src/*"]
Scripts["script/*"] --> Contracts
Index["oracle/index.js"] --> ChainReader["chain.js"]
Index --> Scoring["scoring.js"]
Index --> Payments["payments.js"]
Index --> Metrics["oracle/metrics.js"]
Docker["oracle/Dockerfile"] --> Index
Loading

Diagram sources

Section sources

Performance Considerations

General guidance derived from the codebase:

  • Concurrency over serial awaits: batch independent chain reads to reduce latency.
  • Chunking over unbounded parallelism: limit concurrent work to avoid RPC rate limits.
  • Deterministic time: use chain timestamps to ensure reproducible scoring and minimize divergence.
  • Metrics-driven tuning: track propose/finalize success, payment scan throughput, and checker agreement to guide optimizations.
  • Container reproducibility: build from lockfiles and copy source wholesale to avoid missing modules in production images.

Troubleshooting Guide

Common issues and how to diagnose them:

  • Epoch already running: overlapping triggers are skipped; check scheduler configuration and manual /epoch calls.
  • Epoch aborted due to RPC failure: transient errors are logged and counted; retry next epoch rather than restarting repeatedly.
  • Transaction refused or never mined: a fee above MAX_FEE_GWEI or a worst-case cost above MAX_TX_COST is refused before it is signed, and a transaction not mined within TX_WAIT_SECONDS is replaced once at the same nonce. A refusal, or a transaction still not mined after the replacement, is counted in sigvara_oracle_tx_refused_total and is a reason to look at the RPC.
  • Payment scan empty but query broken: canary checks detect filter failures; alert on canary failures.
  • Checker divergence: compare evidence roots and per-factor differences; open challenge window allows committee rejection.
  • Undeclared metric names: CI will report problems where metrics are incremented but not registered.

Actionable steps:

  • Inspect epoch logs and metrics for epochsFailed, paymentScanErrors, checkerDivergences.
  • Verify environment variables and epoch intervals at startup.
  • Review divergence payloads and compare evidence roots via the oracle’s evidence endpoint.
  • Ensure all metric names are declared and exported.

Section sources

Conclusion

Sigvara’s debugging and profiling stack combines Foundry-based on-chain verification with a robust, observable Oracle service. By leveraging Foundry’s EVM capabilities, chunked and batched RPC handling, deterministic chain-clock scoring, and comprehensive metrics, developers can identify bottlenecks, optimize gas usage, and resolve issues across both layers. The repository’s tests, scripts, and metrics provide strong foundations for reliable development and production operations.

Appendices

Commands Quick Reference

  • Build and test: forge build, forge test
  • CI fuzz: FOUNDRY_PROFILE=ci forge test
  • Invariant tests: forge test --match-contract StakingInvariantTest
  • E2E tests with traces: forge test --match-contract E2EIntegrationTest -vvv
  • Oracle tests: cd oracle && node --test

Section sources

Home

Overview

Protocol Architecture

Smart Contracts Reference

Oracle Service

Reputation System Design

TypeScript SDK

Integration Guides

Deployment and Operations

Development and Testing

Advanced Topics

Clone this wiki locally