Skip to content

Monitoring and Alerting

Jump to bottom
RunTimeAdmin edited this page Nov 3, 2026 · 23 revisions

Monitoring & Alerting

**Referenced Files in This Document** - [README.md](https://github.com/RunTimeAdmin/Sigvara/blob/main/README.md) - [oracle/index.js](https://github.com/RunTimeAdmin/Sigvara/blob/main/oracle/index.js) - [oracle/metrics.js](https://github.com/RunTimeAdmin/Sigvara/blob/main/oracle/metrics.js) - [oracle/README.md](https://github.com/RunTimeAdmin/Sigvara/blob/main/oracle/README.md) - [oracle/watcher.js](https://github.com/RunTimeAdmin/Sigvara/blob/main/oracle/watcher.js) - [oracle/watcher-policy.js](https://github.com/RunTimeAdmin/Sigvara/blob/main/oracle/watcher-policy.js) - [oracle/chain.js](https://github.com/RunTimeAdmin/Sigvara/blob/main/oracle/chain.js) - [src/SigvaraStaking.sol](https://github.com/RunTimeAdmin/Sigvara/blob/main/src/SigvaraStaking.sol) - [test/E2E.t.sol](https://github.com/RunTimeAdmin/Sigvara/blob/main/test/E2E.t.sol) - [test/SlashDrillFork.t.sol](https://github.com/RunTimeAdmin/Sigvara/blob/main/test/SlashDrillFork.t.sol) - [docker-compose.oracle.yml](https://github.com/RunTimeAdmin/Sigvara/blob/main/docker-compose.oracle.yml)

Table of Contents

  1. Introduction
  2. Project Structure
  3. Core Components
  4. Architecture Overview
  5. Detailed Component Analysis
  6. Dependency Analysis
  7. Performance Considerations
  8. Troubleshooting Guide
  9. Conclusion
  10. Appendices

Introduction

This document defines monitoring and alerting for Sigvara operations, focusing on the off-chain reputation oracle, its Prometheus-style metrics, slash-related economic events, and operational health signals. It explains what to monitor, how alerts should be structured, how logs are produced, how dashboards can be built, and how incident response integrates with the checker and divergence watcher.

The repository exposes:

  • A live HTTP oracle service that computes and proposes agent reputation scores each epoch.
  • A /metrics endpoint exposing Prometheus-compatible counters and gauges.
  • A /health endpoint designed for production alerting.
  • A checker mode operator that independently recomputes scores and records divergences.
  • A divergence watcher that classifies divergences and sends escalation notifications.
  • On-chain staking and slashing logic that zeroes reputation and distributes slashed stake.

Section sources

Project Structure

Monitoring-relevant code is concentrated in the oracle/ directory, with supporting protocol behavior in Solidity contracts and test suites.

graph TB
subgraph "Oracle Service"
IDX["index.js<br/>epoch loop, scoring, RPC calls"]
MET["metrics.js<br/>Prometheus exposition"]
CHN["chain.js<br/>RPC backoff, rate limiting"]
WCH["watcher.js<br/>divergence watcher"]
WPL["watcher-policy.js<br/>alert classification & webhook formatting"]
end
subgraph "Protocol Contracts"
STK["SigvaraStaking.sol<br/>slash lifecycle"]
REP["SigvaraReputation.sol<br/>score anchoring"]
end
subgraph "Tests"
E2E["E2E.t.sol<br/>slash + zero score"]
SLASH["SlashDrillFork.t.sol<br/>slash drill flow"]
end
IDX --> MET
IDX --> CHN
WCH --> WPL
IDX --> STK
IDX --> REP
E2E --> STK
SLASH --> STK
Loading

Diagram sources

Section sources

Core Components

This section maps the monitoring surface to concrete components.

Area What to Monitor Source Why It Matters
Oracle process health Uptime, last successful epoch, store writability, state persistence, where the state was loaded from, the chain the RPC is on /health (stateSource, chainId, expectedChainId), sigvara_oracle_uptime_seconds, sigvara_oracle_last_successful_epoch_timestamp_seconds Detects stale scoring (two epoch intervals with no success, counted from process start until the first), volume mount failures, restart loops, a state file that was recovered from its backup, or an RPC on the wrong chain.
Epoch pipeline Started/succeeded/failed epochs, propose attempts, finalize attempts sigvara_oracle_epochs_total, sigvara_oracle_propose_attempts_total, sigvara_oracle_finalize_attempts_total Shows whether the oracle is progressing through its scoring cycle.
Proposal outcomes Success, error, slot taken, throttled sigvara_oracle_propose_total{result}, sigvara_oracle_propose_throttled_total Distinguishes normal proposal races from RPC/provider issues.
Finalization outcomes Success, error, throttled sigvara_oracle_finalize_total{result}, sigvara_oracle_finalize_throttled_total Finalize is permissionless; errors may mean another party finalized first.
Attestation intake Accepted, accepted but not saved (accepted_not_durable), rejected cooldown, rejected payment, other rejections sigvara_oracle_attest_total{result} Tracks abuse patterns and verification failures. A count of accepted_not_durable means the state file could not be written.
Write durability Flags, resolves and links accepted but not saved sigvara_oracle_writes_not_durable_total A count above zero means the state file could not be written; the change is in memory only until a save works.
Payment evidence Verified payments, RPC errors, scan counts, pulled payments, canary failures sigvara_oracle_payments_verified_total, sigvara_oracle_payment_rpc_errors_total, sigvara_oracle_payment_scans_total, sigvara_oracle_payments_pulled_total, sigvara_oracle_payment_scan_canary_failures_total Ensures the pull scanner is working and not silently missing settlements.
Checker agreement Divergences vs agreements sigvara_oracle_checker_comparisons_total{verdict} Independent recomputation disagreement is a critical integrity signal.
Payer-outcome import Outcomes imported from the other operator, refused, imports stopped early, and conflicting outcomes sigvara_oracle_payer_outcomes_imported_total, sigvara_oracle_payer_outcome_import_refused_total, sigvara_oracle_payer_outcome_import_errors_total, sigvara_oracle_payer_outcome_conflicts_total An outcome one operator holds and the other does not shows up as different evidence roots. Refused or conflicts climbing means a peer serves outcomes that fail this oracle's checks or differ from what it holds.
External feeds Flags received/resolved, links created, jobs reads/errors, score RPC errors sigvara_oracle_flags_total, sigvara_oracle_links_total, sigvara_oracle_jobs_reads_total, sigvara_oracle_score_rpc_errors_total Observes watchdog and ERC-8183 feed health.
Cache performance Evidence cache hits/misses/evictions, badge requests/hits/RPC errors sigvara_oracle_evidence_cache_total, sigvara_oracle_badge_requests_total High miss rates increase chain load and latency.
Economic activity Skipped unbonded agents, proposals rejected by committee sigvara_oracle_skipped_unbonded_total, sigvara_oracle_proposals_rejected_total Signals stake changes and committee intervention.
Transaction safety Transactions refused for their fee, or sent and never seen mined sigvara_oracle_tx_refused_total{reason} (fee_cap, unconfirmed) Either one is a reason to look at the RPC.
Epoch fees Charges taken, skipped, already paid, failed sigvara_oracle_fee_charges_total{result} skipped for long means proposals are being refused: look at the oracle's bond and the agents' status.
Operator authority Pending proposals whose proposer lost its authority sigvara_oracle_proposals_stranded_total Nonzero after a key rotation is expected; otherwise an operator was removed with proposals in flight.
HTTP surface Requests refused before routing, handlers that threw sigvara_oracle_http_refused_total{reason} (bad_target, handler_error) The second should stay at zero; either climbing means probing or a route bug.

Section sources

Architecture Overview

The monitoring architecture spans three layers:

  1. Service layer: The oracle process emits structured console logs and Prometheus metrics.
  2. Integrity layer: A checker operator independently recomputes scores and records divergences.
  3. Alerting layer: A watcher polls the checker’s divergence endpoint, classifies findings, and forwards alerts to webhooks.
sequenceDiagram
participant Chain as "Arc Testnet"
participant Oracle as "Primary Oracle"
participant Metrics as "Prometheus Scrape Target"
participant Checker as "Checker Operator"
participant Watcher as "Divergence Watcher"
participant Ops as "Operations Channel"
Chain-->>Oracle : Agent events, pending scores, challenge windows
Oracle->>Metrics : Increment epoch, propose, finalize, attest counters
Metrics-->>Ops : Prometheus scrape
Oracle->>Chain : Propose / finalize reputation
Checker->>Chain : Read pending scores
Checker->>Checker : Recompute factors and total score
alt Scores agree
Checker->>Metrics : Increment checkerAgreed
else Scores disagree
Checker->>Metrics : Increment checkerDivergences
Checker->>Watcher : GET /divergence
Watcher->>Chain : Re-read pending score
Watcher->>Ops : Actionable alert if window open
Watcher->>Ops : Final warning near deadline
end
Loading

Diagram sources

Detailed Component Analysis

Prometheus Metrics Collection

The oracle uses an in-memory counter/gauge registry and renders it as Prometheus text. Every metric name used by inc() or set() must be declared; otherwise it is silently dropped. Tests enforce this invariant.

Key metric families include:

  • Process uptime and epoch timing.
  • Epoch lifecycle: started, succeeded, failed.
  • Proposal lifecycle: attempts, successes, errors, slot taken, throttled.
  • Finalization lifecycle: attempts, successes, errors, throttled.
  • Attestation intake: accepted, accepted but not saved (accepted_not_durable), rejected cooldown, rejected payment, rejected other.
  • Write durability: flags, resolves and links taken while the state file could not be written (sigvara_oracle_writes_not_durable_total).
  • Payment verification: verified payments, RPC errors, scans, pulled payments, refused credits, canary failures.
  • Checker comparisons: diverged vs agreed.
  • Payer-signed outcome import: imported, refused, stopped early, conflicts.
  • Committee rejection count.
  • Transaction safety: transactions refused for their fee or never seen mined, epoch-fee charges, and proposals stranded behind a proposer that lost its authority.
  • External inputs: flags, links, jobs, score RPC errors.
  • Requests refused before routing, or ended by an unexpected error.
  • Badge and evidence cache performance.
flowchart TD
Start(["Epoch Start"]) --> CountStart["Increment epochsStarted"]
CountStart --> ScanAgents["Scan registered agents"]
ScanAgents --> ScanPayments["Run payment scan"]
ScanPayments --> ComputeScores["Compute six-factor scores"]
ComputeScores --> DecideAction{"Skip / Finalize / Propose"}
DecideAction --> |Finalize| IncFinalizeAttempts["Increment finalizeAttempts"]
IncFinalizeAttempts --> TryFinalize["Call finalizeScore()"]
TryFinalize --> FinalizeOK{"Success?"}
FinalizeOK --> |Yes| IncFinalizeSuccess["Increment finalizeSuccesses"]
FinalizeOK --> |No| CheckPending["Check still pending"]
CheckPending --> |Still pending| IncFinalizeError["Increment finalizeErrors"]
CheckPending --> |Already finalized| IncFinalizeSuccess
DecideAction --> |Propose| IncProposeAttempts["Increment proposeAttempts"]
IncProposeAttempts --> TryPropose["Call proposeReputation()<br/>(proposeIfEmpty() in checker mode)"]
TryPropose --> ProposeOK{"Success?"}
ProposeOK --> |Yes| IncProposeSuccess["Increment proposeSuccesses"]
ProposeOK --> |No| ClassifyFailure{"Throttled / Slot Taken / Fee Cap or Unconfirmed / Error"}
ClassifyFailure --> |Throttled| IncProposeThrottled["Increment proposeThrottled"]
ClassifyFailure --> |Slot Taken| IncSlotTaken["Increment proposeSlotTaken"]
ClassifyFailure --> |Fee Cap or Unconfirmed| IncTxRefused["Increment txCapped / txUnconfirmed"]
ClassifyFailure --> |Error| IncProposeError["Increment proposeErrors"]
Loading

Diagram sources

Section sources

Reputation Score Monitoring

Reputation is computed off-chain and anchored on-chain. Monitoring should track both the oracle’s internal computation and the on-chain outcome.

Important signals:

  • activeAgents: number of agents scored in the last epoch.
  • lastSuccessfulEpochTimestampSeconds: time since the last successful epoch.
  • proposalsRejected: committee-rejected proposals.
  • checkerDivergences: independent recomputation disagreement.
  • skippedUnbonded: agents below minimum stake.
  • proposalsStranded: pending proposals whose proposer lost its authority (the contract will not finalize them); the primary replaces one with a fresh proposal and a checker takes over from it.

Operational guidance:

  • Treat rising epochsFailed as immediate investigation.
  • Treat sustained checkerDivergences as integrity risk.
  • Use proposalsRejected to detect bad proposals before they become live.
  • Use skippedUnbonded to detect stake withdrawals or insufficient bonding.

Section sources

Slash Proposals and Stake Withdrawal Alerts

Slashing is the primary economic enforcement mechanism. The staking contract freezes bonds during pending or disputed slash proposals, prevents active agents from withdrawing below minimum stake, and zeros reputation when a slash executes.

Alert rules should cover:

  • New slash proposals.
  • Active dispute windows.
  • Executed slashes.
  • Stake withdrawals that approach minimum stake.
  • Queued withdrawals swept by slash execution.
stateDiagram-v2
[*] --> NoProposal
NoProposal --> Pending : "initiateSlash()"
Pending --> Disputed : "disputeSlash()"
Pending --> Executed : "executeSlash() after challenge"
Pending --> Cancelled : "cancelSlash() / vetoSlash() / expireSlash()"
Disputed --> Executed : "resolveDispute(uphold), by the admin, after the challenge deadline and before the dispute expires"
Disputed --> Cancelled : "resolveDispute(reject) / expireDispute() / vetoSlash()"
Executed --> SlashedAgent : "zero reputation"
SlashedAgent --> [*]
Loading

Diagram sources

This is the lifecycle on main. The admin Safe, not the committee, rules on a dispute and cannot uphold one before the challenge deadline or once it can be expired; the admin can also veto a pending or disputed slash, a slash nobody finishes can be lapsed by anyone 14 days after its deadline, and a dispute the admin never rules on can be expired after 14 days. After any dropped proposal the committee waits one challenge period before filing against the agent again. The Arc testnet contracts have followed this lifecycle since 26 October 2026, and the refusal of a late uphold since the staking upgrade of 28 October, with the admin a single EOA rather than a Safe.

Recommended alert categories:

Event Severity Condition Response
Slash proposal initiated Critical New slash proposal detected Verify evidence, notify committee, check agent status.
Slash dispute opened High Proposal enters disputed state Preserve evidence, prepare defense or mitigation.
Slash vetoed or lapsed High SlashVetoed event, or SlashCancelled emitted by expireSlash or expireDispute Confirm the agent was reinstated if it is still bonded, and note the cooldown before the committee can file again.
Slash executed Critical Agent status becomes slashed and reputation is zeroed Freeze affected integrations, review stake distribution, update dashboards. SlashExecuted carries burned and toVictim only; an indexer that decoded the old event sees a new topic since the 2 November 2026 staking upgrade.
Stake withdrawal near minimum Warning Active agent approaching minimumStake Notify operator, warn about slash exposure.
Queued withdrawal swept High Slash execution removes queued withdrawal amount Confirm slash correctness, reconcile claimable balances.

Section sources

Oracle Failure Detection

The oracle distinguishes between transient provider failures and protocol-level failures.

Critical failure signals:

  • epochsFailed increases.
  • /health returns 503.
  • sigvara_oracle_tx_refused_total rises: a fee cap tripped, or a transaction was sent and never seen mined, even after one replacement.
  • stateSource on /health reads anything but file after a restart.
  • paymentScanCanaryFailures increases.
  • paymentScanRefused increases.
  • proposeThrottled or finalizeThrottled rises sharply.
  • scoreRpcErrorsTotal or jobsErrors rise.
  • rateLimitHits increases.
flowchart TD
Health["GET /health"] --> Ok{"ok == true?"}
Ok --> |No| AlertHealth["Alert: unhealthy"]
Ok --> |Yes| EpochAge["Check last successful epoch"]
EpochAge --> Stale{"Stale scoring?"}
Stale --> |Yes| AlertStale["Alert: stale epoch"]
Stale --> |No| Metrics["Scrape Prometheus metrics"]
Metrics --> Errors{"Errors rising?"}
Errors --> |Yes| AlertErrors["Alert: RPC/provider errors"]
Errors --> |No| Canary["Check payment scan canary"]
Canary --> CanaryFail{"Canary failed?"}
CanaryFail --> |Yes| AlertCanary["Alert: scan query unreliable"]
CanaryFail --> |No| Normal["Normal operation"]
Loading

Diagram sources

Section sources

Log Aggregation Strategy

The oracle writes structured console logs prefixed with [oracle]. These logs describe:

  • Epoch lifecycle.
  • Agent filtering (slashed, uncovered, unbonded).
  • Proposal/finalization outcomes.
  • Payment scan results.
  • Divergence detection.
  • Rate limiting and RPC retries.

Recommended aggregation:

  • Collect stdout/stderr from the oracle container.
  • Parse lines beginning with [oracle] into structured fields: timestamp, level, component, message.
  • Tag logs with deployment metadata: commit hash, environment, RPC provider, operator mode.
  • Retain recent operational logs for at least 30 days.
  • Retain high-severity logs (slash, divergence, epoch failure) longer for auditability.
  • Exclude sensitive values such as private keys, RPC tokens, or admin bearer tokens.

Log retention policy:

Log Type Retention Reason
Operational warnings 30 days Troubleshooting and trend analysis.
Epoch failures 90 days Incident reconstruction.
Slash events 1 year Economic and compliance audit trail.
Divergence records 1 year Integrity investigation.
RPC throttle noise 7 days Capacity planning only.

Section sources

Dashboard Setup

Dashboards should separate protocol health, oracle internals, and economic signals.

Protocol Health Dashboard

  • /health availability and status.
  • Last successful epoch timestamp.
  • Epoch duration trends.
  • RPC error rates.
  • Store writability and state persistence.

Oracle Internals Dashboard

  • Epoch started/succeeded/failed.
  • Proposal attempts vs success.
  • Finalize attempts vs success.
  • Throttled proposals/finalizations.
  • Attestation acceptance/rejection breakdown.
  • Payment scan progress and canary verdict.
  • Evidence cache hit ratio.
  • Badge request volume and RPC errors.

Economic and Integrity Dashboard

  • Active agents.
  • Skipped unbonded agents.
  • Proposals rejected by committee.
  • Checker divergences vs agreements.
  • Slash proposal and execution events.
  • Stake withdrawal queue changes.

Grafana integration recommendations:

  • Add the oracle host as a Prometheus target.
  • Create panels using metric families documented in metrics.js.
  • Use labels such as status="started" and result="success" to avoid ambiguous queries.
  • Set recording rules for derived series like epoch success rate and divergence rate.

Section sources

Incident Response Procedures

Incidents fall into several categories:

Category Trigger Immediate Action Escalation
Oracle down /health 503 or no scrape Restart container, check volume mount, verify RPC URL. Page on-call if not recovered within 5 minutes.
Stale scoring Last successful epoch older than 2× epoch interval (counted from process start if none has completed) Inspect epoch logs, check RPC rate limits, and whether the RPC is on the expected chain with a clock close to the host's. Notify operations and checker operator.
RPC provider throttling Rising rateLimitHits, proposeThrottled, finalizeThrottled Reduce burst, adjust pacing, switch provider if needed. Review provider capacity plan.
Payment scan broken paymentScanCanaryFailures > 0 Validate asset, chain, confirmations, and filter. Investigate provider indexing or configuration drift.
Checker divergence checkerDivergences increases Compare evidence roots, check code versions, inspect scoring inputs. Alert slashing committee if challenge window is open.
Checker not checking Watcher alert: checker unreachable, reporting itself not ok, its divergence list unreadable, not a checker, too slow to see proposals, or no epoch completed within MAX_SILENCE_MINUTES Check the checker's host, RPC and /health. Until it clears, an empty divergence list means nothing was looked at. Notify the checker operator.
Watcher failing Watcher alert after three polls in a row fail Check the watcher's own RPC and the checker URL. Until it clears, silence means nothing. Notify whoever runs the watcher.
Slash proposal New slash event Verify evidence, freeze dependent services, preserve logs. A decision is needed before action: the committee files and can cancel, and on main the admin rules on a dispute and can veto.
Slash executed Reputation zeroed Stop trusting the agent, reconcile stake distribution, update dashboards. Post-incident review and stake accounting.

Escalation workflow:

  1. Automated alert fires based on metrics or health checks.
  2. Watcher classifies divergence as actionable, expired, superseded, or closed.
  3. First notification names the agent (its 32-byte hash), the proposed total, the checker's total, the factors that differ and how long is left to reject it, counted to the proposal's own deadline (finalizableAt). The evidence roots are in the checker's divergence record, not in the alert.
  4. Final warning fires near the end of the challenge window if no resolution occurred.
  5. Human responders verify evidence, consult committee, and decide whether to reject or accept the proposal.

Communication protocols:

  • Use a dedicated operations channel for automated alerts.
  • Include checker hostname and commit information where available.
  • Avoid sending raw private keys, RPC secrets, or admin tokens in messages.
  • The watcher builds its alert text only from fields it can check (a 32-byte hash, a fixed list of factor names, figures rendered as numbers). Discord alerts switch mentions off and Slack alerts are escaped, so text that arrives from the checker cannot ping a channel.
  • Keep a timeline of actions, including who acted and when.

Section sources

Best Practices for Monitoring

  • Prefer labeled Prometheus metrics over ad-hoc log parsing for alerting.
  • Use /health for liveness and readiness checks.
  • Scrape /metrics regularly and validate that every declared metric appears in exposition.
  • Separate infrastructure alerts (RPC, storage, process) from protocol alerts (slash, divergence, proposal rejection).
  • Avoid alerting on expected race outcomes such as slot_taken.
  • Use the checker’s divergence endpoint as a secondary source of truth, not just a backup.
  • Retain evidence-root data for post-mortem analysis.
  • Run the watcher on a third host, separate from the checker and primary oracle.

False positive reduction:

  • Do not alert on the 404 a primary gives for /divergence; it is expected. (A watcher pointed at a primary reports it as a misconfiguration, because an empty list there would read as "checked, found nothing".)
  • Treat unavailable canary verdicts differently from failed.
  • Account for provider rate limits before escalating to protocol incidents.
  • Correlate slash alerts with agent status and stake levels.
  • Suppress repeated alerts for the same unresolved divergence until the window closes or the proposal changes.

Alert fatigue prevention:

  • Group related metrics into runbooks rather than dozens of individual alerts.
  • Use severity tiers: info, warning, critical.
  • Add suppression windows for known maintenance or catch-up periods.
  • Require human confirmation before public-facing escalations.
  • Review alert effectiveness quarterly and remove low-signal alerts.

Section sources

Integration with Existing Monitoring Infrastructure

Prometheus:

  • Scrape the oracle’s /metrics endpoint.
  • Configure job labels for environment, operator role, and commit.
  • Use PromQL to compute derived series such as epoch success rate and divergence rate.

Grafana:

  • Build dashboards per the sections above.
  • Use panels for health, metrics, and economic indicators.
  • Link panels to runbooks and incident procedures.

Commercial solutions:

  • Forward structured logs to a centralized logging platform.
  • Export Prometheus metrics via remote write if supported.
  • Map alerts to PagerDuty, Opsgenie, Slack, Discord, or Telegram using existing webhook integrations.

Existing Sigvara tooling:

  • The watcher already supports webhook payloads and formats messages for common destinations.
  • The oracle README documents reverse proxy exposure and which endpoints should remain internal.

Section sources

Dependency Analysis

Monitoring depends on stable interactions between the oracle, chain RPC, checker, and watcher.

graph LR
RPC["Blockchain RPC Provider"] --> Index["oracle/index.js"]
Index --> Metrics["oracle/metrics.js"]
Index --> Store["Persistent State"]
Index --> Chain["oracle/chain.js"]
Checker["Checker Operator"] --> Divergence["/divergence"]
Watcher["oracle/watcher.js"] --> Divergence
Watcher --> Webhook["Webhook Destination"]
Metrics --> Prometheus["Prometheus"]
Prometheus --> Grafana["Grafana"]
Loading

Diagram sources

Potential coupling risks:

  • If the RPC provider changes behavior, payment scans and score reads may fail silently unless monitored.
  • If the checker runs slower than the challenge window, it misses proposals and produces misleading silence.
  • If the watcher shares infrastructure with the checker, its silence cannot be trusted.
  • If /metrics is scraped too aggressively, it adds load but does not expose sensitive data.

Section sources

Performance Considerations

Monitoring should not degrade the oracle:

  • Scrape /metrics at a reasonable interval, such as 15–60 seconds.
  • Avoid scraping /evidence or /score from monitoring tools; these are expensive read paths.
  • Use Prometheus recording rules for heavy queries instead of repeated complex PromQL.
  • Monitor evidence cache hit ratios; falling hit rates indicate increased chain load.
  • Track badge RPC errors separately from general score RPC errors.
  • Use chunked RPC reads and backoff policies already implemented in chain.js.

Section sources

Troubleshooting Guide

Common symptoms and diagnostics:

Symptom Likely Cause Diagnostic Step Resolution
/health returns 503 State path unwritable, last persist failed, or stale epoch Check storeWritable, statePersisted, lastSuccessfulEpochMs, timeSinceLastEpochMs Fix volume mount, free disk space, investigate epoch failure.
Epochs failing repeatedly RPC error, agent fetch failure, an RPC on a chain other than EXPECTED_CHAIN_ID, or a chain clock far from the host's Inspect epochsFailed, epoch logs, RPC errors Retry provider, reduce burst, check contract addresses, EXPECTED_CHAIN_ID and the host's clock.
Payment scan credited nothing Wrong asset, wrong chain, or broken filter Check paymentScanCanaryFailures and scan config Correct PAYMENT_ASSET, chain, and confirmations.
Checker reports divergences Different evidence or different scoring inputs Compare ownEvidenceRoot and proposedEvidenceRoot, and read evidenceMatchesProposal: true means the checker scored exactly the proposal's evidence, so a remaining difference is arithmetic Align operators, fix scanner, or escalate to committee.
A payer-signed outcome on one operator only for more than two hours The import is failing, stalled, or refused it Read the payer outcomes from log lines and the import counters; run node scripts/check-live.mjs outcomes Follow docs/operations-runbook.md, section 2.
Proposals throttled RPC provider rate limit Check proposeThrottled and rateLimitHits Pace requests or upgrade provider tier.
index.sigvara.xyz/health fails, or the explorer pages say the event index did not answer The index's RPC key revoked or out of credits, the poll lagging, or the container down Read lastError and behind on /health, then docker logs sigvara-index; node scripts/check-live.mjs services checks it on the live-checks schedule Follow docs/operations-runbook.md, section 4.
Slash proposal pending Challenge window open Read on-chain slash state and challenge deadline Wait, dispute, or execute depending on evidence.

Section sources

Conclusion

Sigvara’s monitoring surface is centered on the oracle’s Prometheus metrics, health endpoint, checker-driven divergence detection, and on-chain slash mechanics. Effective operations require:

  • Clear separation between infrastructure, protocol, and economic alerts.
  • Strong reliance on /health and /metrics for automated detection.
  • Use of the checker and watcher as integrity and escalation mechanisms.
  • Careful handling of slash proposals and stake withdrawals.
  • Structured logging with appropriate retention.
  • Dashboards that make real-time protocol health visible without adding load.

When deployed correctly, this system makes oracle failures, evidence inconsistencies, slash events, and stake anomalies observable, actionable, and auditable.

Appendices

Key Metric Reference Summary

Metric Family Example Series Alert Guidance
Epoch lifecycle sigvara_oracle_epochs_total{status="failed"} Alert on non-zero rate.
Proposal lifecycle sigvara_oracle_propose_total{result="error"} Investigate RPC and contract interaction.
Finalization lifecycle sigvara_oracle_finalize_total{result="error"} Check whether another party finalized first.
Attestation intake sigvara_oracle_attest_total{result="rejected_cooldown"} Monitor abuse patterns.
Attestation durability sigvara_oracle_attest_total{result="accepted_not_durable"} Alert on any increase: the state file cannot be written. Fix the volume or free disk space; /health is 503 meanwhile.
Write durability sigvara_oracle_writes_not_durable_total Alert on any increase: a flag, resolve or link was taken while the state file cannot be written. Fix the volume or free disk space; /health is 503 meanwhile.
Payment evidence sigvara_oracle_payment_scan_canary_failures_total Alert immediately.
Checker integrity sigvara_oracle_checker_comparisons_total{verdict="diverged"} Escalate if challenge window open.
Committee action sigvara_oracle_proposals_rejected_total Alert on new rejections.
Economic state sigvara_oracle_skipped_unbonded_total Monitor stake withdrawals.
Transaction safety sigvara_oracle_tx_refused_total{reason} Look at the RPC when either reason climbs.
Epoch fees sigvara_oracle_fee_charges_total{result="skipped"} Proposals are being refused; check the oracle's bond and the agents' status.
Payer-outcome import sigvara_oracle_payer_outcome_conflicts_total Investigate any increase: a peer serves a different outcome for a payment this oracle holds one for (docs/operations-runbook.md, section 3).

Section sources

Home

Overview

Protocol Architecture

Smart Contracts Reference

Oracle Service

Reputation System Design

TypeScript SDK

Integration Guides

Deployment and Operations

Development and Testing

Advanced Topics

Clone this wiki locally