Skip to content

Monitoring and Alerting

Jump to bottom
RunTimeAdmin edited this page Nov 7, 2026 · 46 revisions

Monitoring & Alerting

**Referenced Files in This Document** - [README.md](https://github.com/RunTimeAdmin/Sigvara/blob/main/README.md) - [oracle/index.js](https://github.com/RunTimeAdmin/Sigvara/blob/main/oracle/index.js) - [oracle/metrics.js](https://github.com/RunTimeAdmin/Sigvara/blob/main/oracle/metrics.js) - [oracle/README.md](https://github.com/RunTimeAdmin/Sigvara/blob/main/oracle/README.md) - [oracle/watcher.js](https://github.com/RunTimeAdmin/Sigvara/blob/main/oracle/watcher.js) - [oracle/watcher-policy.js](https://github.com/RunTimeAdmin/Sigvara/blob/main/oracle/watcher-policy.js) - [oracle/keeper.js](https://github.com/RunTimeAdmin/Sigvara/blob/main/oracle/keeper.js) - [oracle/indexer.js](https://github.com/RunTimeAdmin/Sigvara/blob/main/oracle/indexer.js) - [scripts/check-live.mjs](https://github.com/RunTimeAdmin/Sigvara/blob/main/scripts/check-live.mjs) - [oracle/chain.js](https://github.com/RunTimeAdmin/Sigvara/blob/main/oracle/chain.js) - [src/SigvaraStaking.sol](https://github.com/RunTimeAdmin/Sigvara/blob/main/src/SigvaraStaking.sol) - [test/E2E.t.sol](https://github.com/RunTimeAdmin/Sigvara/blob/main/test/E2E.t.sol) - [test/SlashDrillFork.t.sol](https://github.com/RunTimeAdmin/Sigvara/blob/main/test/SlashDrillFork.t.sol) - [docker-compose.oracle.yml](https://github.com/RunTimeAdmin/Sigvara/blob/main/docker-compose.oracle.yml)

Table of Contents

  1. Introduction
  2. Project Structure
  3. Core Components
  4. Architecture Overview
  5. Detailed Component Analysis
  6. Dependency Analysis
  7. Performance Considerations
  8. Troubleshooting Guide
  9. Conclusion
  10. Appendices

Introduction

This document defines monitoring and alerting for Sigvara operations, focusing on the off-chain reputation oracle, its Prometheus-style metrics, slash-related economic events, and operational health signals. It explains what to monitor, how alerts should be structured, how logs are produced, how dashboards can be built, and how incident response integrates with the checker and divergence watcher.

The repository exposes:

  • A live HTTP oracle service that computes and proposes agent reputation scores each epoch.
  • A /metrics endpoint exposing Prometheus-compatible counters and gauges.
  • A /health endpoint designed for production alerting.
  • A checker mode operator that independently recomputes scores and records divergences.
  • A divergence watcher that classifies divergences and sends escalation notifications.
  • Two slash keepers that execute a slash once its deadline has passed, and alert on what they leave to a person.
  • An outside check, scripts/check-live.mjs, run from GitHub Actions on a schedule, and a public status page at sigvara.xyz/status.
  • On-chain staking and slashing logic that zeroes reputation and distributes slashed stake.

Section sources

Project Structure

Monitoring-relevant code is concentrated in the oracle/ directory, with supporting protocol behavior in Solidity contracts and test suites.

graph TB
subgraph "Oracle Service"
IDX["index.js<br/>epoch loop, scoring, RPC calls"]
MET["metrics.js<br/>Prometheus exposition"]
CHN["chain.js<br/>RPC backoff, rate limiting"]
WCH["watcher.js<br/>divergence watcher"]
WPL["watcher-policy.js<br/>alert classification & webhook formatting"]
end
subgraph "Protocol Contracts"
STK["SigvaraStaking.sol<br/>slash lifecycle"]
REP["SigvaraReputation.sol<br/>score anchoring"]
end
subgraph "Tests"
E2E["E2E.t.sol<br/>slash + zero score"]
SLASH["SlashDrillFork.t.sol<br/>slash drill flow"]
end
IDX --> MET
IDX --> CHN
WCH --> WPL
IDX --> STK
IDX --> REP
E2E --> STK
SLASH --> STK
Loading

Diagram sources

Section sources

Core Components

This section maps the monitoring surface to concrete components.

Area What to Monitor Source Why It Matters
Oracle process health Uptime, last successful epoch, store writability, state persistence, where the state was loaded from, the chain the RPC is on /health (stateSource, chainId, expectedChainId), sigvara_oracle_uptime_seconds, sigvara_oracle_last_successful_epoch_timestamp_seconds Detects stale scoring (two epoch intervals with no success, counted from process start until the first), volume mount failures, restart loops, a state file that was recovered from its backup, or an RPC on the wrong chain.
Epoch pipeline Started/succeeded/failed epochs, propose attempts, finalize attempts sigvara_oracle_epochs_total, sigvara_oracle_propose_attempts_total, sigvara_oracle_finalize_attempts_total Shows whether the oracle is progressing through its scoring cycle.
Proposal outcomes Success, error, slot taken, throttled sigvara_oracle_propose_total{result}, sigvara_oracle_propose_throttled_total Distinguishes normal proposal races from RPC/provider issues.
Finalization outcomes Success, error, throttled sigvara_oracle_finalize_total{result}, sigvara_oracle_finalize_throttled_total Finalize is permissionless; errors may mean another party finalized first.
Attestation intake Accepted, accepted but not saved (accepted_not_durable), rejected cooldown, rejected payment, other rejections sigvara_oracle_attest_total{result} Tracks abuse patterns and verification failures. A count of accepted_not_durable means the state file could not be written.
Write durability Flags, resolves and links accepted but not saved sigvara_oracle_writes_not_durable_total A count above zero means the state file could not be written; the change is in memory only until a save works.
Payment evidence Verified payments, RPC errors, scan counts, pulled payments, canary failures sigvara_oracle_payments_verified_total, sigvara_oracle_payment_rpc_errors_total, sigvara_oracle_payment_scans_total, sigvara_oracle_payments_pulled_total, sigvara_oracle_payment_scan_canary_failures_total Ensures the pull scanner is working and not silently missing settlements.
Payment chains Scans completed and failed, epochs held back by a scan that is behind, each chain's mode, cursor and last scan sigvara_oracle_payment_chain_scans_total, sigvara_oracle_payment_chain_scan_errors_total, sigvara_oracle_epochs_held_by_payment_chain_total, /health (paymentChains, paymentChainHold) Without a payment chain configured, which is off by default, the counters stay at 0 and the /health fields are absent (payment-chains.js:37-39, metrics.js:350-363, index.js:2526-2532). In shadow, scoring carries on when a scan fails (operations-runbook.md:619). A chain that is on and falls behind holds the epoch (index.js:1156-1191).
Checker agreement Divergences vs agreements sigvara_oracle_checker_comparisons_total{verdict} Independent recomputation disagreement is a critical integrity signal.
Checker deferrals Finalizes and empty slots left to the primary sigvara_oracle_checker_deferred_total{action} Counts only while the primary is not yet CHECKER_GRACE_MINUTES late; a checker whose own finalize or propose successes climb means the primary was late (metrics.js:233-238).
Payer-outcome import Outcomes imported from the other operator, refused, imports stopped early, and conflicting outcomes sigvara_oracle_payer_outcomes_imported_total, sigvara_oracle_payer_outcome_import_refused_total, sigvara_oracle_payer_outcome_import_errors_total, sigvara_oracle_payer_outcome_conflicts_total An outcome one operator holds and the other does not shows up as different evidence roots. Refused or conflicts climbing means a peer serves outcomes that fail this oracle's checks or differ from what it holds.
External feeds Flags received/resolved, links created, jobs reads/errors, score RPC errors sigvara_oracle_flags_total, sigvara_oracle_links_total, sigvara_oracle_jobs_reads_total, sigvara_oracle_score_rpc_errors_total Observes watchdog and ERC-8183 feed health.
Cache performance Evidence cache hits/misses/evictions, badge requests/hits/RPC errors sigvara_oracle_evidence_cache_total, sigvara_oracle_badge_requests_total High miss rates increase chain load and latency.
Economic activity Skipped unbonded agents, proposals rejected by committee sigvara_oracle_skipped_unbonded_total, sigvara_oracle_proposals_rejected_total Signals stake changes and committee intervention.
Transaction safety Transactions refused for their fee, or sent and never seen mined sigvara_oracle_tx_refused_total{reason} (fee_cap, unconfirmed) Either one is a reason to look at the RPC.
Epoch fees Charges taken, skipped, already paid, failed sigvara_oracle_fee_charges_total{result} skipped for long means proposals are being refused: look at the oracle's bond and the agents' status.
Operator authority Pending proposals whose proposer lost its authority sigvara_oracle_proposals_stranded_total Nonzero after a key rotation is expected; otherwise an operator was removed with proposals in flight.
HTTP surface Requests refused before routing, handlers that threw sigvara_oracle_http_refused_total{reason} (bad_target, handler_error) The second should stay at zero; either climbing means probing or a route bug.
Event index Whether it polls, how far behind it is, how full it is index.sigvara.xyz/health (ok, behind, logs, maxLogs, full, lastError) It answers 503 when it has not polled for a few minutes, is more than 3,600 blocks behind, or is full. At INDEX_MAX_LOGS it stops reading on and keeps answering with what it has, and check-live fails from 80% full (indexer.js:292-311, check-live.mjs:194-195).
Slash execution Pending slashes past their deadline, and what each keeper did scripts/check-live.mjs services, each keeper's webhook alerts, [keeper] run at block N: ... in its log check-live fails when a Pending slash is more than six hours past its deadline, unless it is held on purpose. It shares nothing with the keepers, so it is what notices when both are down (check-live.mjs:293-315, operations-runbook.md:667-681).
Operator wallets Each balance against its floor: the two operators, the publisher and both keepers wallets in deployments/5042002.json, read by scripts/check-live.mjs services A wallet with no gas stops proposing, finalizing or posting, so check-live fails when one is under its floor (check-live.mjs:257-273, 5042002.json:10-16).

Section sources

Architecture Overview

The monitoring architecture spans four layers:

  1. Service layer: The oracle process emits structured console logs and Prometheus metrics.
  2. Integrity layer: A checker operator independently recomputes scores and records divergences.
  3. Alerting layer: A watcher polls the checker’s divergence endpoint, classifies findings, and forwards alerts to webhooks. It reads each disputed agent's proposal one at a time, with retries, because a node can refuse a whole JSON-RPC batch over its limit (watcher.js:175-192).
  4. Outside layer: scripts/check-live.mjs runs from GitHub Actions on a schedule, from a place that shares no account, host or credential with what it watches. A failed run emails the repository owner, a second alert path beside the watcher. The status page at sigvara.xyz/status holds the operators, the disagreements and the index's /health to the same limits, and does not read slashes or wallets (live-checks.yml:3-6, check-live.mjs:16-28, status.js:2-4).
sequenceDiagram
participant Chain as "Arc Testnet"
participant Oracle as "Primary Oracle"
participant Metrics as "Prometheus Scrape Target"
participant Checker as "Checker Operator"
participant Watcher as "Divergence Watcher"
participant Ops as "Operations Channel"
Chain-->>Oracle : Agent events, pending scores, challenge windows
Oracle->>Metrics : Increment epoch, propose, finalize, attest counters
Metrics-->>Ops : Prometheus scrape
Oracle->>Chain : Propose / finalize reputation
Checker->>Chain : Read pending scores
Checker->>Checker : Recompute factors and total score
alt Scores agree
Checker->>Metrics : Increment checkerAgreed
else Scores disagree
Checker->>Metrics : Increment checkerDivergences
Checker->>Watcher : GET /divergence
Watcher->>Chain : Re-read pending score
Watcher->>Ops : Actionable alert if window open
Watcher->>Ops : Final warning near deadline
end
Loading

Diagram sources

Detailed Component Analysis

Prometheus Metrics Collection

The oracle uses an in-memory counter/gauge registry and renders it as Prometheus text. Every metric name used by inc() or set() must be declared; otherwise it is silently dropped. Tests enforce this invariant.

Key metric families include:

  • Process uptime and epoch timing.
  • Epoch lifecycle: started, succeeded, failed.
  • Proposal lifecycle: attempts, successes, errors, slot taken, throttled.
  • Finalization lifecycle: attempts, successes, errors, throttled.
  • Attestation intake: accepted, accepted but not saved (accepted_not_durable), rejected cooldown, rejected payment, rejected other.
  • Write durability: flags, resolves and links taken while the state file could not be written (sigvara_oracle_writes_not_durable_total).
  • Payment verification: verified payments, RPC errors, scans, pulled payments, refused credits, canary failures.
  • Payment chains: scans completed, scans that could not complete, and epochs held back by a scan that was behind, wholly or for some agents (metrics.js:350-363).
  • Checker comparisons: diverged vs agreed.
  • Payer-signed outcome import: imported, refused, stopped early, conflicts.
  • Committee rejection count.
  • Transaction safety: transactions refused for their fee or never seen mined, epoch-fee charges, and proposals stranded behind a proposer that lost its authority.
  • External inputs: flags, links, jobs, score RPC errors.
  • Requests refused before routing, or ended by an unexpected error.
  • Badge and evidence cache performance.
flowchart TD
Start(["Epoch Start"]) --> CountStart["Increment epochsStarted"]
CountStart --> ScanAgents["Scan registered agents"]
ScanAgents --> ScanPayments["Run payment scan"]
ScanPayments --> ComputeScores["Compute six-factor scores"]
ComputeScores --> DecideAction{"Skip / Finalize / Propose"}
DecideAction --> |Finalize| IncFinalizeAttempts["Increment finalizeAttempts"]
IncFinalizeAttempts --> TryFinalize["Call finalizeScore()"]
TryFinalize --> FinalizeOK{"Success?"}
FinalizeOK --> |Yes| IncFinalizeSuccess["Increment finalizeSuccesses"]
FinalizeOK --> |No| CheckPending["Check still pending"]
CheckPending --> |Still pending| IncFinalizeError["Increment finalizeErrors"]
CheckPending --> |Already finalized| IncFinalizeSuccess
DecideAction --> |Propose| IncProposeAttempts["Increment proposeAttempts"]
IncProposeAttempts --> TryPropose["Call proposeReputation()<br/>(proposeIfEmpty() in checker mode)"]
TryPropose --> ProposeOK{"Success?"}
ProposeOK --> |Yes| IncProposeSuccess["Increment proposeSuccesses"]
ProposeOK --> |No| ClassifyFailure{"Throttled / Slot Taken / Fee Cap or Unconfirmed / Error"}
ClassifyFailure --> |Throttled| IncProposeThrottled["Increment proposeThrottled"]
ClassifyFailure --> |Slot Taken| IncSlotTaken["Increment proposeSlotTaken"]
ClassifyFailure --> |Fee Cap or Unconfirmed| IncTxRefused["Increment txCapped / txUnconfirmed"]
ClassifyFailure --> |Error| IncProposeError["Increment proposeErrors"]
Loading

Diagram sources

Section sources

Reputation Score Monitoring

Reputation is computed off-chain and anchored on-chain. Monitoring should track both the oracle’s internal computation and the on-chain outcome.

Important signals:

  • activeAgents: number of agents scored in the last epoch.
  • lastSuccessfulEpochTimestampSeconds: time since the last successful epoch.
  • proposalsRejected: committee-rejected proposals.
  • checkerDivergences: independent recomputation disagreement.
  • skippedUnbonded: agents below minimum stake.
  • proposalsStranded: pending proposals whose proposer lost its authority (the contract will not finalize them); the primary replaces one with a fresh proposal and a checker takes over from it.

Operational guidance:

  • Treat rising epochsFailed as immediate investigation.
  • Treat sustained checkerDivergences as integrity risk.
  • Use proposalsRejected to detect bad proposals before they become live.
  • Use skippedUnbonded to detect stake withdrawals or insufficient bonding.

Section sources

Slash Proposals and Stake Withdrawal Alerts

Slashing is the primary economic enforcement mechanism. The staking contract freezes bonds during pending or disputed slash proposals, prevents active agents from withdrawing below minimum stake, and zeros reputation when a slash executes.

Alert rules should cover:

  • New slash proposals.
  • Active dispute windows.
  • Executed slashes.
  • Stake withdrawals that approach minimum stake.
  • Queued withdrawals swept by slash execution.
stateDiagram-v2
[*] --> NoProposal
NoProposal --> Pending : "initiateSlash()"
Pending --> Disputed : "disputeSlash()"
Pending --> Executed : "executeSlash() after challenge"
Pending --> Cancelled : "cancelSlash() / vetoSlash() / expireSlash()"
Disputed --> Executed : "resolveDispute(uphold), by the admin, after the challenge deadline and before the dispute expires"
Disputed --> Cancelled : "resolveDispute(reject) / expireDispute() / vetoSlash()"
Executed --> SlashedAgent : "zero reputation"
SlashedAgent --> [*]
Loading

Diagram sources

This is the lifecycle on main. The admin Safe, not the committee, rules on a dispute and cannot uphold one before the challenge deadline or once it can be expired; the admin can also veto a pending or disputed slash, a slash nobody finishes can be lapsed by anyone 14 days after its deadline, and a dispute the admin never rules on can be expired after 14 days. After any dropped proposal the committee waits one challenge period before filing against the agent again. The Arc testnet contracts have followed this lifecycle since 26 October 2026, and the refusal of a late uphold since the staking upgrade of 28 October, with the admin a single EOA rather than a Safe.

As of 5 November 2026 two slash keepers (oracle/keeper.js) are live on Arc testnet, holding 20 USDC each, and have not yet had a slash to execute. Each alerts by webhook on an execution, a slash that is refused, a send that did not confirm, a dispute, a lapse, low gas and six failed runs in a row. They stop at the lapse and never call expireSlash or expireDispute, so those stay a person's decision (operations-runbook.md:647-655, operations-runbook.md:674-676, operations-runbook.md:694-695).

Recommended alert categories:

Event Severity Condition Response
Slash proposal initiated Critical New slash proposal detected Verify evidence, notify committee, check agent status.
Slash dispute opened High Proposal enters disputed state Preserve evidence, prepare defense or mitigation.
Slash vetoed or lapsed High SlashVetoed event, or SlashCancelled emitted by expireSlash or expireDispute Confirm the agent was reinstated if it is still bonded, and note the cooldown before the committee can file again.
Slash executed Critical Agent status becomes slashed and reputation is zeroed Freeze affected integrations, review stake distribution, update dashboards. SlashExecuted carries burned and toVictim only; an indexer that decoded the old event sees a new topic since the 2 November 2026 staking upgrade.
Stake withdrawal near minimum Warning Active agent approaching minimumStake Notify operator, warn about slash exposure.
Queued withdrawal swept High Slash execution removes queued withdrawal amount Confirm slash correctness, reconcile claimable balances.

Section sources

Oracle Failure Detection

The oracle distinguishes between transient provider failures and protocol-level failures.

Critical failure signals:

  • epochsFailed increases.
  • /health returns 503.
  • sigvara_oracle_tx_refused_total rises: a fee cap tripped, or a transaction was sent and never seen mined, even after one replacement.
  • stateSource on /health reads anything but file after a restart.
  • paymentScanCanaryFailures increases.
  • paymentScanRefused increases.
  • paymentChainScanErrors or epochsHeldByPaymentChain rises, with a payment chain configured (metrics.js:71-75).
  • proposeThrottled or finalizeThrottled rises sharply.
  • scoreRpcErrorsTotal or jobsErrors rise.
  • rateLimitHits increases.
flowchart TD
Health["GET /health"] --> Ok{"ok == true?"}
Ok --> |No| AlertHealth["Alert: unhealthy"]
Ok --> |Yes| EpochAge["Check last successful epoch"]
EpochAge --> Stale{"Stale scoring?"}
Stale --> |Yes| AlertStale["Alert: stale epoch"]
Stale --> |No| Metrics["Scrape Prometheus metrics"]
Metrics --> Errors{"Errors rising?"}
Errors --> |Yes| AlertErrors["Alert: RPC/provider errors"]
Errors --> |No| Canary["Check payment scan canary"]
Canary --> CanaryFail{"Canary failed?"}
CanaryFail --> |Yes| AlertCanary["Alert: scan query unreliable"]
CanaryFail --> |No| Normal["Normal operation"]
Loading

Diagram sources

Section sources

Log Aggregation Strategy

The oracle writes structured console logs prefixed with [oracle]. These logs describe:

  • Epoch lifecycle.
  • Agent filtering (slashed, uncovered, unbonded).
  • Proposal/finalization outcomes.
  • Payment scan results.
  • Divergence detection.
  • Rate limiting and RPC retries.

Recommended aggregation:

  • Collect stdout/stderr from the oracle container.
  • Parse lines beginning with [oracle] into structured fields: timestamp, level, component, message.
  • Tag logs with deployment metadata: commit hash, environment, RPC provider, operator mode.
  • Retain recent operational logs for at least 30 days.
  • Retain high-severity logs (slash, divergence, epoch failure) longer for auditability.
  • Exclude sensitive values such as private keys, RPC tokens, or admin bearer tokens.

Log retention policy:

Log Type Retention Reason
Operational warnings 30 days Troubleshooting and trend analysis.
Epoch failures 90 days Incident reconstruction.
Slash events 1 year Economic and compliance audit trail.
Divergence records 1 year Integrity investigation.
RPC throttle noise 7 days Capacity planning only.

Section sources

Dashboard Setup

Dashboards should separate protocol health, oracle internals, and economic signals.

Protocol Health Dashboard

  • /health availability and status.
  • Last successful epoch timestamp.
  • Epoch duration trends.
  • RPC error rates.
  • Store writability and state persistence.

Oracle Internals Dashboard

  • Epoch started/succeeded/failed.
  • Proposal attempts vs success.
  • Finalize attempts vs success.
  • Throttled proposals/finalizations.
  • Attestation acceptance/rejection breakdown.
  • Payment scan progress and canary verdict.
  • Evidence cache hit ratio.
  • Badge request volume and RPC errors.

Economic and Integrity Dashboard

  • Active agents.
  • Skipped unbonded agents.
  • Proposals rejected by committee.
  • Checker divergences vs agreements.
  • Slash proposal and execution events.
  • Stake withdrawal queue changes.

Grafana integration recommendations:

  • Add the oracle host as a Prometheus target.
  • Create panels using metric families documented in metrics.js.
  • Use labels such as status="started" and result="success" to avoid ambiguous queries.
  • Set recording rules for derived series like epoch success rate and divergence rate.

Section sources

Incident Response Procedures

Incidents fall into several categories:

Category Trigger Immediate Action Escalation
Oracle down /health 503 or no scrape Restart container, check volume mount, verify RPC URL. Page on-call if not recovered within 5 minutes.
Stale scoring Last successful epoch older than 2× epoch interval (counted from process start if none has completed) Inspect epoch logs, check RPC rate limits, and whether the RPC is on the expected chain with a clock close to the host's. Notify operations and checker operator.
RPC provider throttling Rising rateLimitHits, proposeThrottled, finalizeThrottled Reduce burst, adjust pacing, switch provider if needed. Review provider capacity plan.
Payment scan broken paymentScanCanaryFailures > 0 Validate asset, chain, confirmations, and filter. Investigate provider indexing or configuration drift.
Checker divergence checkerDivergences increases Compare evidence roots, check code versions, inspect scoring inputs. Alert slashing committee if challenge window is open.
Checker not checking Watcher alert: checker unreachable, reporting itself not ok, its divergence list unreadable, not a checker, too slow to see proposals, or no epoch completed within MAX_SILENCE_MINUTES Check the checker's host, RPC and /health. Until it clears, an empty divergence list means nothing was looked at. Notify the checker operator.
Watcher failing Watcher alert after three polls in a row fail. A poll fails when a disputed agent's proposal could not be read, after the agents it could read have alerted (watcher.js:230-233) Check the watcher's own RPC and the checker URL. Until it clears, silence means nothing. Notify whoever runs the watcher.
Slash proposal New slash event Verify evidence, freeze dependent services, preserve logs. A decision is needed before action: the committee files and can cancel, and on main the admin rules on a dispute and can veto.
Slash executed Reputation zeroed Stop trusting the agent, reconcile stake distribution, update dashboards. Post-incident review and stake accounting.
Slash overdue check-live fails: a Pending slash is more than six hours past its deadline and not held on purpose Read each keeper's log and docker ps on its host. Both keepers are down, out of gas or refused: follow docs/operations-runbook.md, section 7, with most of the 14 days left.
Wallet under its floor check-live fails: a wallet in deployments/5042002.json is below its minUsdc Send it testnet USDC. Follow "Gas" in docs/operations-runbook.md.

Escalation workflow:

  1. Automated alert fires based on metrics or health checks.
  2. Watcher classifies divergence as actionable, expired, superseded, or closed.
  3. First notification names the agent (its 32-byte hash), the proposed total, the checker's total, the factors that differ and how long is left to reject it, counted to the proposal's own deadline (finalizableAt). The evidence roots are in the checker's divergence record, not in the alert.
  4. Final warning fires near the end of the challenge window if no resolution occurred.
  5. Human responders verify evidence, consult committee, and decide whether to reject or accept the proposal.

Communication protocols:

  • Use a dedicated operations channel for automated alerts.
  • Include checker hostname and commit information where available.
  • Avoid sending raw private keys, RPC secrets, or admin tokens in messages.
  • The watcher builds its divergence alerts only from fields it can check (a 32-byte hash, a fixed list of factor names, figures rendered as numbers). Discord alerts switch mentions off and Slack alerts are escaped, so text that arrives from the checker cannot ping a channel. Alerts and logs give a failed read in the node's own words (wordsOf in log-spans.js) with any URL removed, and say that a checker did not return JSON without repeating what it sent (watcher.js:77-80, watcher.js:91-97, log-spans.js:11-14).
  • Keep a timeline of actions, including who acted and when.

Section sources

Best Practices for Monitoring

  • Prefer labeled Prometheus metrics over ad-hoc log parsing for alerting.
  • Use /health for liveness and readiness checks.
  • Scrape /metrics regularly and validate that every declared metric appears in exposition.
  • Separate infrastructure alerts (RPC, storage, process) from protocol alerts (slash, divergence, proposal rejection).
  • Avoid alerting on expected race outcomes such as slot_taken.
  • Use the checker’s divergence endpoint as a secondary source of truth, not just a backup.
  • Retain evidence-root data for post-mortem analysis.
  • Run the watcher on a third host, separate from the checker and primary oracle.

False positive reduction:

  • Do not alert on the 404 a primary gives for /divergence; it is expected. (A watcher pointed at a primary reports it as a misconfiguration, because an empty list there would read as "checked, found nothing".)
  • Treat unavailable canary verdicts differently from failed.
  • Account for provider rate limits before escalating to protocol incidents.
  • Correlate slash alerts with agent status and stake levels.
  • Suppress repeated alerts for the same unresolved divergence until the window closes or the proposal changes.

Alert fatigue prevention:

  • Group related metrics into runbooks rather than dozens of individual alerts.
  • Use severity tiers: info, warning, critical.
  • Add suppression windows for known maintenance or catch-up periods.
  • Require human confirmation before public-facing escalations.
  • Review alert effectiveness quarterly and remove low-signal alerts.

Section sources

Integration with Existing Monitoring Infrastructure

Prometheus:

  • Scrape the oracle’s /metrics endpoint.
  • Configure job labels for environment, operator role, and commit.
  • Use PromQL to compute derived series such as epoch success rate and divergence rate.

Grafana:

  • Build dashboards per the sections above.
  • Use panels for health, metrics, and economic indicators.
  • Link panels to runbooks and incident procedures.

Commercial solutions:

  • Forward structured logs to a centralized logging platform.
  • Export Prometheus metrics via remote write if supported.
  • Map alerts to PagerDuty, Opsgenie, Slack, Discord, or Telegram using existing webhook integrations.

Existing Sigvara tooling:

  • The watcher already supports webhook payloads and formats messages for common destinations.
  • The oracle README documents reverse proxy exposure and which endpoints should remain internal.

Section sources

Dependency Analysis

Monitoring depends on stable interactions between the oracle, chain RPC, checker, and watcher.

graph LR
RPC["Blockchain RPC Provider"] --> Index["oracle/index.js"]
Index --> Metrics["oracle/metrics.js"]
Index --> Store["Persistent State"]
Index --> Chain["oracle/chain.js"]
Checker["Checker Operator"] --> Divergence["/divergence"]
Watcher["oracle/watcher.js"] --> Divergence
Watcher --> Webhook["Webhook Destination"]
Metrics --> Prometheus["Prometheus"]
Prometheus --> Grafana["Grafana"]
Loading

Diagram sources

Potential coupling risks:

  • If the RPC provider changes behavior, payment scans and score reads may fail silently unless monitored.
  • If the checker runs slower than the challenge window, it misses proposals and produces misleading silence.
  • If the watcher shares infrastructure with the checker, its silence cannot be trusted.
  • If /metrics is scraped too aggressively, it adds load but does not expose sensitive data.

Section sources

Performance Considerations

Monitoring should not degrade the oracle:

  • Scrape /metrics at a reasonable interval, such as 15–60 seconds.
  • Avoid scraping /evidence or /score from monitoring tools; these are expensive read paths.
  • Use Prometheus recording rules for heavy queries instead of repeated complex PromQL.
  • Monitor evidence cache hit ratios; falling hit rates indicate increased chain load.
  • Track badge RPC errors separately from general score RPC errors.
  • Use chunked RPC reads and backoff policies already implemented in chain.js and read-retry.js. A log read a node refuses for its size is asked for in halves (log-spans.js).

Section sources

Troubleshooting Guide

Common symptoms and diagnostics:

Symptom Likely Cause Diagnostic Step Resolution
/health returns 503 State path unwritable, last persist failed, or stale epoch Check storeWritable, statePersisted, lastSuccessfulEpochMs, timeSinceLastEpochMs, and the log for a failed save's error Fix volume mount, free disk space, investigate epoch failure.
Epochs failing repeatedly RPC error, agent fetch failure, an RPC on a chain other than EXPECTED_CHAIN_ID, or a chain clock far from the host's Inspect epochsFailed, epoch logs, RPC errors Retry provider, reduce burst, check contract addresses, EXPECTED_CHAIN_ID and the host's clock.
Payment scan credited nothing Wrong asset, wrong chain, or broken filter Check paymentScanCanaryFailures and scan config Correct PAYMENT_ASSET, chain, and confirmations.
payment chain <id> scan failed in the log, and sigvara_oracle_payment_chain_scan_errors_total rising The chain's RPC is down or refusing. A range the node refuses for its size is halved, so a dense range alone should not stop a scan. (rpc -32005) or similar repeating at one block may be a refusal worded in a way oracle/log-spans.js does not know Read paymentChains on /health, and check that lastBlockTs moves between epochs In shadow, scoring carries on and the next epoch resumes from the last block saved. Lowering CHUNK gets it past the range in the meantime (operations-runbook.md:612-622).
scoring nothing this epoch naming a payment chain or the scored chain's payment scan, or agent(s) left unscored A scan of a chain that is on has not passed its delay, or the scored chain's scan is too far behind for a proposal to land in time. A chain in shadow never holds an epoch Read paymentChainHold on /health: chains when the whole epoch was held, which also counts as failed, or agents when scans fell behind while it ran, which counts only in sigvara_oracle_epochs_held_by_payment_chain_total Fix that chain's RPC; the next epoch scores once the scan has caught up (index.js:1156-1191, index.js:1818-1822, payment-chains.js:341-362).
Checker reports divergences Different evidence or different scoring inputs Compare ownEvidenceRoot and proposedEvidenceRoot, and read evidenceMatchesProposal: true means the checker scored exactly the proposal's evidence, so a remaining difference is arithmetic Align operators, fix scanner, or escalate to committee.
A payer-signed outcome on one operator only for more than two hours The import is failing, stalled, or refused it Read the payer outcomes from log lines and the import counters; run node scripts/check-live.mjs outcomes Follow docs/operations-runbook.md, section 2.
Proposals throttled RPC provider rate limit Check proposeThrottled and rateLimitHits Pace requests or upgrade provider tier.
index.sigvara.xyz/health fails, or the explorer pages say the event index did not answer The index's RPC key revoked or out of credits, the poll lagging, the index full, or the container down Read lastError, behind and full on /health, then docker logs sigvara-index; node scripts/check-live.mjs services checks it on the live-checks schedule Follow docs/operations-runbook.md, section 4.
The explorer pages fall back because a history takes more than 20 /events answers (the event index has more than 20 pages for this history), /health says full: true, or check-live says index: holds N of at most M logs The history has outgrown the pages' 20 answers of up to 5,000 logs, or the index holds INDEX_MAX_LOGS logs (100,000 by default, which fits the 256 MB mem_limit), or more than 80% of them Read logs, maxLogs and full on index.sigvara.xyz/health A full index stops reading on and keeps answering with what it has, and the pages read the rest from the chain. Raise INDEX_MAX_LOGS and mem_limit together, by about 2 MB per thousand logs, and restart. That does not change the pages' 20 answers: settle what docs/event-index.md lists as not bounded before the index serves a much larger history (explorer.js:33-34, explorer.js:488, event-index.md:54-61, docker-compose.indexer.vps.yml:55).
/events answers 429, too many new questions; try again in a minute A client has had 120 new answers in a minute; an answer already in the cache does not count. With TRUSTED_PROXIES unset, everyone behind the proxy shares that limit Read TRUSTED_PROXIES in the index's .env Keep it at the proxy network's subnet (indexer.js:332-335, operations-runbook.md:402-406).
the ERC-8004 link scan has not reached block N, today's snapshot, or scoring nothing this epoch The link scan is behind, its logs cannot be listed, or the registry's chain cannot be read (only with EXTERNAL_LINK_FROM_BLOCK set) Read the epoch log, and compare linkScanState.cursor in the two operators' state files Follow docs/operations-runbook.md, section 3, "ERC-8004 links made on chain".
[publisher] posting blocked: ... A live gate failed: the agentWallet changed, the signer is not answering or signs as another address, the balance is under the floor, or a transaction is pending docker logs sigvara-publisher on the primary's host Follow docs/operations-runbook.md, section 5. A pending transaction that never settles needs a look by hand.
Slash proposal pending Challenge window open Read on-chain slash state and challenge deadline Wait, dispute, or execute depending on evidence.
slashes: ... hours past its deadline and nobody has executed it from check-live Both keepers are down, out of gas or refused it docker ps and docker logs sigvara-keeper on each keeper's host Follow docs/operations-runbook.md, section 7.
[keeper] run at block N: ... counts refused, failed, blocked or lapsed refused with ReporterNotAuthorized: the reporter has left the committee. failed: it reverted, the fee is above KEEPER_MAX_FEE_GWEI, or it was sent with no receipt. blocked: a transaction of its own is pending. lapsed: nobody executed it for a fortnight docker logs sigvara-keeper, and look the transaction up A current committee member can cancelSlash a refused one, or it lapses. A failed one is tried again every run. A lapsed one is a person's decision: execute by hand, or expireSlash (operations-runbook.md:683-692).
wallets: <name> ... under its floor from check-live A wallet in deployments/5042002.json has fallen below its minUsdc Read the balances check-live logs Send it testnet USDC (operations-runbook.md:38-42).

Section sources

Conclusion

Sigvara’s monitoring surface is centered on the oracle’s Prometheus metrics, health endpoint, checker-driven divergence detection, and on-chain slash mechanics. Effective operations require:

  • Clear separation between infrastructure, protocol, and economic alerts.
  • Strong reliance on /health and /metrics for automated detection.
  • Use of the checker and watcher as integrity and escalation mechanisms.
  • Careful handling of slash proposals and stake withdrawals.
  • Structured logging with appropriate retention.
  • Dashboards that make real-time protocol health visible without adding load.

When deployed correctly, this system makes oracle failures, evidence inconsistencies, slash events, and stake anomalies observable, actionable, and auditable.

Appendices

Key Metric Reference Summary

Metric Family Example Series Alert Guidance
Epoch lifecycle sigvara_oracle_epochs_total{status="failed"} Alert on non-zero rate.
Proposal lifecycle sigvara_oracle_propose_total{result="error"} Investigate RPC and contract interaction.
Finalization lifecycle sigvara_oracle_finalize_total{result="error"} Check whether another party finalized first.
Attestation intake sigvara_oracle_attest_total{result="rejected_cooldown"} Monitor abuse patterns.
Attestation durability sigvara_oracle_attest_total{result="accepted_not_durable"} Alert on any increase: the state file cannot be written. Fix the volume or free disk space; /health is 503 meanwhile.
Write durability sigvara_oracle_writes_not_durable_total Alert on any increase: a flag, resolve or link was taken while the state file cannot be written. Fix the volume or free disk space; /health is 503 meanwhile.
Payment evidence sigvara_oracle_payment_scan_canary_failures_total Alert immediately.
Payment chains sigvara_oracle_epochs_held_by_payment_chain_total A run of them means that chain's RPC, or the scored chain's, needs a look (index.js:1185-1188, metrics.js:71-72).
Checker integrity sigvara_oracle_checker_comparisons_total{verdict="diverged"} Escalate if challenge window open.
Committee action sigvara_oracle_proposals_rejected_total Alert on new rejections.
Economic state sigvara_oracle_skipped_unbonded_total Monitor stake withdrawals.
Transaction safety sigvara_oracle_tx_refused_total{reason} Look at the RPC when either reason climbs.
Epoch fees sigvara_oracle_fee_charges_total{result="skipped"} Proposals are being refused; check the oracle's bond and the agents' status.
Payer-outcome import sigvara_oracle_payer_outcome_conflicts_total Investigate any increase: a peer serves a different outcome for a payment this oracle holds one for (docs/operations-runbook.md, section 3).

Section sources

Home

Overview

Protocol Architecture

Smart Contracts Reference

Oracle Service

Reputation System Design

TypeScript SDK

Integration Guides

Deployment and Operations

Development and Testing

Advanced Topics

Clone this wiki locally