Skip to content

Maintenance Procedures

Jump to bottom
RunTimeAdmin edited this page Nov 4, 2026 · 43 revisions

Maintenance Procedures

**Referenced Files in This Document** - [README.md](https://github.com/RunTimeAdmin/Sigvara/blob/main/README.md) - [architecture.md](https://github.com/RunTimeAdmin/Sigvara/blob/main/docs/architecture.md) - [oracle README.md](https://github.com/RunTimeAdmin/Sigvara/blob/main/oracle/README.md) - [SigvaraEpochFees.sol](https://github.com/RunTimeAdmin/Sigvara/blob/main/src/SigvaraEpochFees.sol) - [Upgrade.s.sol](https://github.com/RunTimeAdmin/Sigvara/blob/main/script/Upgrade.s.sol) - [0005-governance-and-emergency-powers.md](https://github.com/RunTimeAdmin/Sigvara/blob/main/docs/adr/0005-governance-and-emergency-powers.md) - [SigvaraIdentity.sol](https://github.com/RunTimeAdmin/Sigvara/blob/main/src/SigvaraIdentity.sol) - [SigvaraReputation.sol](https://github.com/RunTimeAdmin/Sigvara/blob/main/src/SigvaraReputation.sol) - [SigvaraStaking.sol](https://github.com/RunTimeAdmin/Sigvara/blob/main/src/SigvaraStaking.sol) - [Upgrade.t.sol](https://github.com/RunTimeAdmin/Sigvara/blob/main/test/Upgrade.t.sol) - [5042002.json](https://github.com/RunTimeAdmin/Sigvara/blob/main/deployments/5042002.json)

Table of Contents

  1. Introduction
  2. Project Structure
  3. Core Components
  4. Architecture Overview
  5. Detailed Component Analysis
  6. Dependency Analysis
  7. Performance Considerations
  8. Troubleshooting Guide
  9. Conclusion
  10. Appendices

Introduction

This document defines maintenance procedures for the Sigvara protocol, covering routine parameter updates, fee adjustments, configuration changes, UUPS proxy upgrades, testing requirements, governance workflows, Oracle service database and operational maintenance, security maintenance, emergency response, compliance and audit preservation, schedules, automation, and operational runbooks. It is written for operators, governance participants, auditors, and platform maintainers who need repeatable, safe procedures grounded in the deployed contracts and running Oracle service.

Project Structure

The repository contains:

  • On-chain protocol contracts under src/, all using OpenZeppelin upgradeable proxies with UUPS upgrade authorization.
  • An off-chain reputation Oracle under oracle/ with HTTP endpoints, state persistence, metrics, and checker/watcher components.
  • Deployment artifacts and scripts under deployments/ and script/.
  • Tests under test/ and oracle/*.test.js, plus SDK tests under packages/sdk/test/.
graph TB
subgraph "On-chain"
ID["SigvaraIdentity"]
REP["SigvaraReputation"]
STK["SigvaraStaking"]
FEES["SigvaraEpochFees"]
BOND["SigvaraOracleBond"]
end
subgraph "Off-chain"
ORACLE["Oracle Service"]
CHECKER["Checker Operator"]
WATCHER["Divergence Watcher"]
end
ORACLE --> REP
CHECKER --> REP
WATCHER --> CHECKER
STK --> ID
STK --> REP
FEES --> ID
Loading

Diagram sources

Section sources

Core Components

  • Identity registry: agent identity, operator control, status transitions, key rotation, transfer workflow.
  • Reputation store: oracle proposals, challenge window, maturity model, evidence roots.
  • Staking and slashing: bonds, unbonding, slash lifecycle, dispute resolution, proceeds distribution.
  • Epoch fees: prepaid coverage per agent, fee routing by governance stage.
  • Oracle service: scoring epochs, attestations, flags, evidence, metrics, checker divergence, watcher alerts.

Key maintenance responsibilities:

  • Governance: update parameters (challenge windows, maturity rate, minimum stake, fee schedule), manage roles, approve upgrades.
  • Operators: manage agent keys/status, deposit stake, pay epoch fees, handle withdrawals and slashes.
  • Oracle ops: run primary/checker/watcher, manage credentials, monitor health/metrics, back up state, tune payment scanning.

Section sources

Architecture Overview

The protocol uses UUPS proxies behind stable addresses. Upgrades are gated by UPGRADER_ROLE, held by a 48 hour governance timelock on mainnet. The Oracle proposes scores; the committee can reject during a challenge window; anyone can finalize after the window, though on main a proposal whose proposer has lost its authority cannot be finalized. Slashing zeroes reputation and marks identities terminal.

sequenceDiagram
participant Gov as "Governance Timelock"
participant Proxy as "UUPS Proxy"
participant Impl as "New Implementation"
participant Script as "Upgrade Script"
Gov->>Script : Prepare calldata + authorize
Script->>Proxy : upgradeToAndCall(newImpl, initData)
Proxy->>Impl : Execute reinitializer if provided
Script-->>Gov : Verify implementation slot changed
Loading

Diagram sources

Section sources

Detailed Component Analysis

Routine Parameter Updates

Parameters that require governance or admin role updates include:

  • Challenge window for reputation proposals.
  • Maturity rate for score spendability.
  • Minimum stake and unbonding period.
  • Epoch fee and fee distribution stage.
  • Reward pool address.

These setters are DEFAULT_ADMIN_ROLE calls, so on mainnet they are Safe transactions and take effect at once. On main, only upgrades, the wiring that decides who counts as bonded and where slashed funds go (setStakeView, SigvaraReputation.setOperatorBond, setSlashBeneficiary) and grants of the slasher, committee and staking-core roles wait on the timelock; on the Arc testnet proxies the admin does all of these at once. On main the challenge periods, the score window, the unbonding periods and the maturity rate each have a floor and a ceiling (challenge period 3 to 30 days, score window 1 hour to 7 days, unbonding period 1 to 90 days, maturity rate 1 to 100 points a day), and the minimum stake and the bond amount only have to be above zero. A value outside those bounds needs a contract upgrade. A new unbonding period applies only to withdrawals queued, and oracle operator exits started, after the change. One already waiting keeps the time fixed when it started, unless it started before that time was recorded.

Recommended procedure:

  1. Draft proposal with current values, target values, rationale, and impact analysis.
  2. Run test suites locally to ensure behavior remains correct.
  3. Simulate on testnet or fork; verify events and storage layout.
  4. Submit to the governance Safe (or, for an upgrade or the wiring above, schedule it on the timelock and allow the delay).
  5. Execute via governance-controlled transaction.
  6. Record new values and emit event logs for audit trail.
flowchart TD
Start(["Proposal Draft"]) --> Test["Run unit/fuzz/invariant tests"]
Test --> ForkSim["Simulate on testnet/fork"]
ForkSim --> Review["Governance review"]
Review --> Route{"Upgrade, wiring or role grant?"}
Route --> |Yes| Timelock["Timelock delay"]
Route --> |No| Execute["Execute the change"]
Timelock --> Execute
Execute --> Record["Record values + events"]
Record --> End(["Complete"])
Loading

Diagram sources

Section sources

Fee Adjustments and Configuration Changes

SigvaraEpochFees has never been deployed and is left out of the mainnet launch, so what follows applies to a later deployment.

  • Prepay epoch fees per agent through depositFor; only registered agents may be funded.
  • Oracle charges one epochFee per agent per epoch (a second charge in the same epoch reverts AlreadyCharged, and the oracle counts that as paid); uncovered agents are skipped from active scoring but remain readable.
  • Fees accumulate and are distributed according to governance stage: bootstrap, transition, mature.
  • Admin can set reward pool, epoch fee, and stage.

Operational steps:

  • Monitor agent coverage via isCovered.
  • Top up balances proactively to avoid missed scoring runs.
  • Distribute collected fees periodically to route rewards/burns per stage.
  • Track events for auditability.
flowchart TD
Fund["Operator deposits SVR for didHash"] --> Cover["Oracle checks isCovered"]
Cover --> |covered| Charge["chargeEpoch deducts balance"]
Cover --> |uncovered| Skip["Skip agent in active scoring"]
Charge --> Collect["collected increases"]
Collect --> Distribute["distributeFees routes to reward pool / burn"]
Loading

Diagram sources

Section sources

Upgrade Process Using UUPS Proxy Pattern

All registries use UUPS proxies controlled by UPGRADER_ROLE. The upgrade script:

  • Reads proxy address from deployment artifact or override.
  • Deploys a fresh implementation.
  • Calls upgradeToAndCall with optional initializer calldata.
  • Verifies the implementation slot changed.
  • Requires dependency checks (e.g., identity must expose required functions before upgrading reputation/staking).
  • Refuses a PROXY override that is not the kind of contract TARGET names.

After the handover UPGRADER_ROLE belongs to the timelock, which the script cannot act as. The implementation is deployed from any funded key, the Safe schedules upgradeToAndCall on the timelock, and anyone executes it once the delay has run. Reinitializers are DEFAULT_ADMIN_ROLE-gated and the timelock is not the admin, so the upgrade lands with empty data and the Safe calls the reinitializer in a second transaction.

Pre-upgrade checklist:

  • Ensure storage layout compatibility and pass slot-pinning tests.
  • Run full test suite including upgrade tests.
  • Simulate upgrade on testnet/fork; validate events and state.
  • Prepare reinitializer calldata if new state requires initialization. It runs in a second transaction after a timelocked upgrade, so a new reinitializer needs an unset state that fails closed.
  • Confirm governance approval and timelock readiness.
  • On mainnet, run check_new_implementation <address> <contract> from scripts/check-mainnet.sh against the new implementation before the Safe schedules the upgrade.
  • Before upgrading a reputation proxy from before finalizableAt existed (the Arc testnet one was, until 26 October 2026), clear its pending proposals and do not change the challenge window while one is pending: a proposal pending across the upgrade has no fixed end and is judged against the live window.

Post-upgrade checklist:

  • Record live implementations via the recording script.
  • Verify artifact matches chain state.
  • Compare the deployed code with a build (node scripts/check-bytecode.mjs), verify the source on the Arc explorer, set the implementation's reproducedAt in the deployment file and tag the commit that records it (docs/arc.md, "Verifying the deployed code").
  • Monitor contract reads/writes for regressions.
sequenceDiagram
participant Dev as "Developer"
participant Forge as "Forge Script"
participant Chain as "Chain RPC"
participant Artifact as "Deployment Artifact"
Dev->>Forge : Set TARGET, PROXY, INIT_CALLDATA
Forge->>Artifact : Read proxy address
Forge->>Chain : Check dependencies
Forge->>Chain : Deploy new implementation
Forge->>Chain : upgradeToAndCall(newImpl, initData)
Forge->>Chain : Read implementation slot
Forge-->>Dev : Log old/new impl, success/failure
Loading

Diagram sources

Section sources

Testing Requirements

  • Contracts: unit, fuzz, E2E, upgrade, deploy-script, handover, and invariant tests (740 as of 3 November 2026, 21 of them invariants; 728 run without a fork URL and 12 skip).
  • Oracle: Node.js test suite (1084 tests).
  • SDK: Vitest suite (300 tests, 8 of which skip without a live deployment).

Maintenance gates:

  • All suites must pass before any parameter change or upgrade.
  • Invariants must hold across randomized call sequences.
  • Upgrade tests must confirm storage layout and state preservation.

Section sources

Governance Approval Workflows

  • Roles: on mainnet DEFAULT_ADMIN_ROLE is held by the admin Safe and UPGRADER_ROLE by a 48 hour timelock that the Safe proposes to.
  • Parameters updated via admin-gated setters.
  • Upgrades authorized by UPGRADER_ROLE.
  • Slash committee and oracle roles are separate per contract; coordinate role grants carefully. On main the slasher, committee and staking-core roles are granted only through the timelock (the admin can revoke any of them at once), while ORACLE_ROLE and GUARDIAN_ROLE stay with the admin.

Operational guidance:

  • Use timelock delays to allow community review and exit.
  • Publish proposals with diffs, simulation results, and rollback plan.
  • After execution, record changes and notify stakeholders.

Section sources

Database Maintenance for Oracle Service

The Oracle persists state to a file path configured by ORACLE_STATE_PATH. Operational maintenance includes:

  • Backups:
    • Schedule periodic snapshots of the state file, and copy it off the host. Each save keeps the file it replaces as <state>.bak, but that is one save behind and on the same disk.
    • Validate backup integrity and restore drills. docs/operations-runbook.md, section 1, has the routine the Arc testnet operators run: a nightly copy encrypted with age and pushed to the other operator's host, checked every day from a third host, and a restore drill once a quarter on a spare machine.
  • Cleanup:
    • Rotate or archive old state files based on retention policy.
    • Ensure /attest idempotency and deduplication remain effective after restarts.
    • A state file that exists but cannot be read falls back to the .bak and is kept aside as <state>.corrupt-<time>; with no usable backup the process stops rather than start empty. Deleting the file is the documented way to rescan, but a rescan rebuilds only what the chain holds, so a token holder's outcomes, flags, links and the record of used settlements are lost with it. Payer-signed outcomes the other operator holds come back through the import at the next epoch, unless their deadline passed more than PAYER_OUTCOMES_IMPORT_GRACE_HOURS before it runs. Any .bak beside it is moved aside as <state>.bak.orphaned-<time>.
    • The oracle holds state in memory and rewrites the file at every save, so edit it only with the oracle stopped.
    • A file holding anything from a payment chain is format version 2, which a build from before payment chains refuses to load. A cursor from a chain in shadow leaves the file at version 1, so such a build still loads it and drops the cursor (store.js:178-194, operations-runbook.md:595-596).
  • Performance optimization:
    • Tune LOG_CHUNK_SIZE, PAYMENT_SCAN_PACE_MS, and confirmation depths.
    • Monitor /metrics counters for epoch failures, propose attempts, finalize attempts, evidence cache hits/misses.
    • Rate-limit read endpoints (/score, /evidence) and badge caching.

Health and monitoring:

  • Alert on /health returning 503.
  • Track epoch success/failure counters.
  • Watch evidence cache hit rates and RPC refusal counts.

Section sources

Security Maintenance

  • Key rotation:
    • If an Ed25519 private key is compromised, suspend the agent immediately, rotate the public key, then reinstate.
    • On main, an operator whose bond sits behind another key can replace the oracle's signing key with setSigner or remove it with clearSigner(), effective in the same block. What the old key proposed can no longer be finalized and a successor replaces it, and no other operator can name the old key afterwards (boundTo). Grant ORACLE_ROLE to the new key and keep it in a file secret (ORACLE_PRIVATE_KEY_FILE). The Arc testnet contracts have had the signer registry since 26 October 2026, and neither operator uses it yet.
  • Certificate renewal:
    • Renew TLS certificates for reverse proxies serving Oracle endpoints.
    • Validate CORS headers and CSP policies for badge endpoints.
  • Vulnerability patching:
    • Update dependencies regularly; run CI tooling (Slither, Foundry fuzz).
    • Apply patches to Oracle service and SDK; re-run tests.

Operational safeguards:

  • Separate write tokens per integration for revocation and attribution, each limited to the writes it needs with ORACLE_SCOPES_<NAME>.
  • Keep read paths unauthenticated but rate-limited.
  • Avoid exposing sensitive management endpoints publicly.

Section sources

Emergency Maintenance Procedures

Critical incident response:

  • Suspend misbehaving agents via staking core role to halt activity.
  • Reject bad reputation proposals during the challenge window.
  • Execute slash if undisputed after challenge period. On main a slash nobody executes can be lapsed by anyone 14 days after its deadline, so execute a legitimate one as soon as its window closes.
  • On main, the admin Safe can veto a pending or disputed slash, and the guardian key can pause new registrations, stake deposits, score proposals and bond deposits for three days at a time. A pause never covers withdrawals, rulings, slash execution or finalization.
  • For Oracle outages:
    • Checker mode can propose when primary is silent.
    • Watcher escalates divergences and silence.

Hotfix deployment:

  • Prepare minimal fix implementation.
  • Simulate upgrade with dependency checks.
  • Execute via governance timelock.
  • Record implementation and verify artifact consistency.

Rollback strategy:

  • Maintain implementation history in deployment artifact.
  • Re-deploy previous implementation via UUPS upgrade.
  • Re-run post-rollback verification and monitoring.
flowchart TD
Detect["Detect critical issue"] --> Contain["Suspend agent / reject proposal"]
Contain --> Assess["Assess severity and scope"]
Assess --> Hotfix{"Hotfix needed?"}
Hotfix --> |Yes| Prep["Prepare minimal implementation"]
Prep --> Sim["Simulate upgrade + dependency check"]
Sim --> Approve["Governance approval"]
Approve --> Deploy["Deploy hotfix"]
Deploy --> Verify["Verify artifact + state"]
Hotfix --> |No| Mitigate["Mitigate via parameters / roles"]
Mitigate --> Verify
Verify --> Rollback{"Rollback needed?"}
Rollback --> |Yes| Revert["Upgrade to previous implementation"]
Revert --> PostCheck["Post-check health + metrics"]
Rollback --> |No| Monitor["Monitor stability"]
Loading

Diagram sources

Section sources

Compliance Maintenance, Audit Trail Preservation, Regulatory Reporting

  • Preserve evidence roots and Merkle proofs for each finalized score.
  • Retain slash evidence hashes and outcomes.
  • Keep event logs for parameter changes, upgrades, fee distributions, and slash proceedings.
  • Align reporting with CounterAudit integration where applicable.

Operational practices:

  • Export event logs regularly for audit archives.
  • Cross-check evidence roots against chain data.
  • Maintain versioned documentation of governance decisions and their effects.

Section sources

Maintenance Schedules, Automation Tools, Operational Runbooks

  • Daily:
    • Check Oracle /health and /metrics.
    • Verify epoch completion and evidence cache performance.
    • Inspect slash proposal windows and divergence alerts.
  • Weekly:
    • Review agent coverage and top up epoch fees as needed.
    • Rotate integration tokens if suspicious activity detected.
    • Confirm the nightly off-box copies of the Oracle state are arriving; run the restore drill once a quarter.
  • Monthly:
    • Dependency updates and vulnerability scans.
    • Governance parameter review and proposals.
    • Audit log export and archival.

Automation:

  • Cron jobs for health checks, metric scraping, and backups.
  • CI pipelines for tests and linting.
  • Monitoring dashboards for epoch failures, propose/attempt ratios, and divergence counts.

Runbooks:

  • Oracle startup and shutdown procedures.
  • Credential rotation for write endpoints.
  • Checker setup and divergence watcher configuration.
  • Payment scan tuning and canary interpretation.
  • State backup and restore, payer-outcome import lag, and the two operators disagreeing (docs/operations-runbook.md).

Section sources

Dependency Analysis

Contract dependencies and interactions:

  • Reputation depends on Identity for scorable checks and optionally on operator bond registry.
  • Staking depends on Identity for status and on Reputation for zeroing scores.
  • Epoch fees depend on Identity for operator checks and funding validation.
  • Upgrade script enforces ordering constraints between contracts.
graph LR
ID["SigvaraIdentity"] --> REP["SigvaraReputation"]
ID --> STK["SigvaraStaking"]
REP --> STK
FEES["SigvaraEpochFees"] --> ID
Loading

Diagram sources

Section sources

Performance Considerations

  • Reputation maturity is computed on read; ensure chain latency does not degrade consumer SLAs.
  • Oracle evidence root computation is expensive; leverage caching and rate limits.
  • Payment scanning pacing avoids RPC refusals; tune pace and chunk sizes.
  • Badge endpoint caches responses; monitor cache size and TTL.

Troubleshooting Guide

Common issues and resolutions:

  • Oracle health 503:
    • Check state file writability and persistence errors.
    • Verify last successful epoch timestamp and RPC connectivity, and that the RPC is on EXPECTED_CHAIN_ID with a clock close to the host's.
  • Epoch failures:
    • Inspect propose/attempt metrics and RPC refusal counters.
    • Validate environment variables for external feeds and payment assets.
  • Divergence alerts:
    • Confirm checker epoch cadence and tolerance settings.
    • Watcher escalation timing and webhook delivery.
  • Upgrade dependency errors:
    • Ensure Identity exposes required functions before upgrading Reputation/Staking.
    • Validate storage layout and run slot-pinning tests.

Section sources

Conclusion

Maintaining Sigvara requires disciplined governance, robust testing, careful Oracle operations, and clear emergency procedures. UUPS upgrades must be simulated, approved, and recorded. Parameter changes and fee adjustments should follow documented workflows. Oracle state must be backed up and monitored. Security hygiene includes key rotation, certificate renewal, and vulnerability patching. Compliance demands evidence preservation and audit trails. With these procedures, operators can keep the protocol secure, performant, and accountable.

Appendices

Deployed Contract Addresses and Implementation Records

  • Proxy addresses are stable; implementations are recorded in the deployment artifact.
  • Use the recording script to synchronize artifact with chain state.
  • The four Arc testnet proxies were upgraded in place on 26 October 2026, the staking proxy again on 28 October and 2 November, and the oracle bond proxy on 2 November. --check confirms that the artifact names the implementations now behind them, with the replaced ones kept in history, and scripts/check-bytecode.mjs that each matches a build of 76bbc97 (tag arc-testnet-2026-11-02b, on the commit that records the upgrade).

Section sources

Home

Overview

Protocol Architecture

Smart Contracts Reference

Oracle Service

Reputation System Design

TypeScript SDK

Integration Guides

Deployment and Operations

Development and Testing

Advanced Topics

Clone this wiki locally