Skip to content

Oracle Production Deployment

Jump to bottom
RunTimeAdmin edited this page Nov 11, 2026 · 60 revisions

Oracle Production Deployment

**Referenced Files in This Document** - [docker-compose.oracle.yml](https://github.com/RunTimeAdmin/Sigvara/blob/main/docker-compose.oracle.yml) - [docker-compose.mainnet.yml](https://github.com/RunTimeAdmin/Sigvara/blob/main/docker-compose.mainnet.yml) - [.env.mainnet.example](https://github.com/RunTimeAdmin/Sigvara/blob/main/oracle/.env.mainnet.example) - [Dockerfile](https://github.com/RunTimeAdmin/Sigvara/blob/main/oracle/Dockerfile) - [Caddyfile.oracle.example](https://github.com/RunTimeAdmin/Sigvara/blob/main/oracle/Caddyfile.oracle.example) - [nginx-checker.conf.example](https://github.com/RunTimeAdmin/Sigvara/blob/main/oracle/nginx-checker.conf.example) - [package.json](https://github.com/RunTimeAdmin/Sigvara/blob/main/oracle/package.json) - [index.js](https://github.com/RunTimeAdmin/Sigvara/blob/main/oracle/index.js) - [store.js](https://github.com/RunTimeAdmin/Sigvara/blob/main/oracle/store.js)

Table of Contents

  1. Introduction
  2. Project Structure
  3. Core Components
  4. Architecture Overview
  5. Detailed Component Analysis
  6. Dependency Analysis
  7. Performance Considerations
  8. Troubleshooting Guide
  9. Conclusion
  10. Appendices

Introduction

This document provides production-grade deployment guidance for the Sigvara Oracle service. It covers container orchestration with Docker Compose, reverse proxy configuration using Caddy or Nginx, environment configuration for blockchain RPCs and external integrations, scaling patterns, health checks, monitoring, security hardening, backup and disaster recovery procedures, and performance tuning recommendations. The guidance is grounded in the repository’s Oracle implementation and deployment artifacts.

Project Structure

The Oracle service is a Node.js application packaged as a container image and orchestrated via Docker Compose. Key deployment-related files include:

  • Container build definition
  • Compose file defining service, networking, and volumes
  • Reverse proxy examples for public exposure (Caddy and Nginx)
  • Application entrypoint and HTTP API
  • Persistent state storage module
graph TB
subgraph "Host"
Caddy["Caddy"]
Nginx["Nginx"]
end
subgraph "Docker Network sigvara-mesh"
Oracle["Oracle Service<br/>port 3030"]
StateVol["Volume oracle_state<br/>(/data/oracle-state.json)"]
end
Chain["Blockchain RPC"]
ExternalRPC["External RPC / ERC-8004"]
Caddy --> Oracle
Nginx -.-> Oracle
Oracle --> Chain
Oracle --> ExternalRPC
Oracle --> StateVol
Loading

Diagram sources

Section sources

Core Components

  • Oracle service process: Node.js HTTP server exposing read endpoints (/health, /metrics, /score, /evidence, /badge, /outcomes), write endpoints (/attest, /flag, /link, /epoch), and checker-specific endpoints (/divergence).
  • Persistent state store: JSON-backed durable store for attestations, flags, links, payment events, cooldowns, divergence records, and scan checkpoints.
  • Blockchain integration: Reads/writes to the primary chain and optional external registry feeds.
  • Reverse proxies: Caddy example for public read surface; Nginx example for checker operator.

Key responsibilities:

  • Orchestrate epoch scoring, payment scanning, and on-chain proposals/finalizations.
  • Provide auditable evidence and scores over HTTP.
  • Persist non-reproducible state to a mounted volume.
  • Expose metrics for Prometheus scraping.

Section sources

Architecture Overview

The Oracle runs inside a container bound to host loopback. A reverse proxy exposes only the necessary public paths while keeping administrative endpoints private. State persists to a named volume. The service communicates with one or more blockchain RPCs and optionally an external RPC for ERC-8004 data.

sequenceDiagram
participant Client as "Client"
participant Caddy as "Caddy"
participant Oracle as "Oracle Service"
participant Store as "State Store"
participant Chain as "Blockchain RPC"
participant Ext as "External RPC"
Client->>Caddy : GET /score/ : didHash
Caddy->>Oracle : reverse_proxy : 3030
Oracle->>Chain : getAgentInfo(), getTotalScore()
Oracle->>Ext : externalScoreFor() (optional)
Oracle-->>Caddy : JSON score
Caddy-->>Client : 200 OK
Note over Oracle,Store : Attestations, flags, payments, divergences persisted to /data
Loading

Diagram sources

Detailed Component Analysis

Docker Compose Orchestration

  • Service name and container name are defined for stable DNS resolution within Docker networks.
  • The service binds to host loopback only, minimizing exposure.
  • Environment variables are loaded from an .env file.
  • A named volume persists attestation/flag/payment state across restarts and rebuilds.
  • The service joins a shared external network used by other components (for example, CounterAudit).

Operational notes:

  • Ensure the external network exists before starting the stack.
  • Mount a persistent volume at /data to avoid losing state.
  • Keep admin endpoints off the public internet; use loopback access or a secure management plane.

Mainnet:

  • docker-compose.mainnet.yml is the Arc mainnet arrangement. The primary, the checker and the watcher are three profiles of one file; COMPOSE_PROFILES in a host's .env picks one, and the three belong on three separate hosts.
  • It builds from SIGVARA_REF, a full 40-character commit hash. A deploy is a change to that line followed by docker compose up -d; a restart reruns the same image and fetches nothing.
  • The signing key is a file secret, read through ORACLE_PRIVATE_KEY_FILE, so it is not in the container's environment.
  • Each container runs as the unprivileged node user on a read-only root filesystem with every capability dropped and a 768 MB memory limit. The only writable places are /data and /tmp. Only the primary joins the edge network it shares with the reverse proxy.

Section sources

Container Image Build

  • Base image: Node 24 Alpine, pinned by digest (the VPS compose files, which fetch one pinned commit at container start, use the same digest).
  • Deterministic dependency installation using package-lock.json (npm ci --omit=dev --ignore-scripts).
  • Source copied wholesale; .dockerignore excludes secrets, local state, tests and probe scripts.
  • Build-time static check validates that all local requires resolve, preventing runtime MODULE_NOT_FOUND errors.
  • The commit the image was built from is passed as SIGVARA_COMMIT, which /health reports. When it is given, the build stops unless it is a full 40-character hash.
  • The process runs as the unprivileged node user, /data is created owned by that user, and a HEALTHCHECK probes /health.
  • Entrypoint runs index.js.

Best practices:

  • Pin base image versions and lockfiles.
  • Run the image with minimal privileges and read-only root filesystem where possible (the mainnet compose file does).
  • Use multi-stage builds if you need to separate build tools from runtime.

Section sources

Reverse Proxy Configuration: Caddy

Public read surface:

  • Only explicitly listed paths are proxied: /health, /score/, /evidence/, /badge/*, /outcomes. The oracle answers 404 on /outcomes unless PAYER_OUTCOMES and PAYER_OUTCOMES_PUBLISH are on.
  • POST /attest is proxied with CORS headers and no-store cache policy.
  • Read endpoints set short Cache-Control to absorb bursts without serving stale data.
  • X-Forwarded-For is preserved so per-client rate limiting works behind the proxy. The oracle believes the header only from loopback or from a peer listed in TRUSTED_PROXIES. Caddy reaches the oracle from a Docker bridge address, so set TRUSTED_PROXIES to the proxy network (for example 172.16.0.0/12); without it every visitor shares one rate-limit bucket.
  • Security headers include HSTS, nosniff, referrer policy, and Server header removal.

Important behaviors:

  • The oracle listens on 127.0.0.1:3030; Caddy reaches it via Docker service name, not host loopback.
  • Administrative endpoints (/flag, /flag/resolve, /link, /epoch) remain loopback-only.
  • /metrics is intentionally excluded from the public proxy.

Section sources

Reverse Proxy Configuration: Nginx (Checker Operator)

Use this when the host already runs Nginx and you need to expose checker endpoints without running another web server.

Exposed endpoints:

  • /health: liveness and watcher polling
  • /divergence and /divergence/:didHash: disagreement collection
  • /evidence/: DID-scoped evidence
  • /outcomes: the checker's payer-signed outcomes, which the primary imports (404 unless PAYER_OUTCOMES and PAYER_OUTCOMES_PUBLISH are on)

Security:

  • Write endpoints (/attest, /flag, /link, /epoch) are not exposed publicly.
  • Security headers and server_tokens off are applied.

Deployment steps:

  • Place the config under sites-available, symlink into sites-enabled, test with nginx -t, reload, and obtain TLS via certbot.

Section sources

Environment Variables and Configuration

Required variables:

  • RPC_URL: Primary blockchain RPC endpoint
  • ORACLE_PRIVATE_KEY: Oracle wallet private key, or ORACLE_PRIVATE_KEY_FILE, the path of a file that holds it (set one, not both; a file secret keeps the key out of the container's environment)
  • IDENTITY_ADDRESS: Identity contract address
  • REPUTATION_ADDRESS: Reputation contract address

Optional variables:

  • EPOCH_HOURS: Epoch interval in hours
  • HOST/PORT: Bind address and port (default 127.0.0.1:3030)
  • FROM_BLOCK: Starting block for scans
  • LOG_CHUNK_SIZE: Batch size for log processing
  • ORACLE_ADMIN_TOKEN: Admin token for protected endpoints
  • ORACLE_TOKEN_: Additional named tokens
  • ORACLE_SCOPES_: Restricts that token to some of attest, flag, link and epoch
  • EXPECTED_CHAIN_ID: The chain this oracle is for (5042 on Arc mainnet, 5042002 on the testnet). With it set, nothing is read for scoring or signed while the RPC reports another chain, and a state file saved on another chain is refused
  • TRUSTED_PROXIES: Addresses or CIDR ranges whose X-Forwarded-For is believed, besides loopback
  • TX_WAIT_SECONDS, MAX_FEE_GWEI, MAX_TX_COST: How long a transaction is waited for before one replacement, and the fee and total-cost ceilings the oracle will sign under
  • MAX_CLOCK_SKEW_SECONDS: How far the chain's clock may be from the host's before an epoch is not run
  • FEE_REGISTRY_ADDRESS: Optional fee gating registry
  • EXTERNAL_RPC, EXTERNAL_IDENTITY_ADDRESS, EXTERNAL_REPUTATION_ADDRESS: ERC-8004 feed configuration
  • EXTERNAL_LINK_FROM_BLOCK: The block to read links made on chain from; the same on both operators
  • ORACLE_MODE=checker: Enable checker mode
  • DIVERGENCE_TOLERANCE: Allowed divergence threshold
  • PRIMARY_EVIDENCE_URL: URL to primary oracle’s evidence endpoint
  • JOBS_MAX_SPAN: Maximum block span for jobs queries
  • ATTEST_COOLDOWN_MS: Cooldown between attestations
  • FLAG_HALF_LIFE_DAYS: Decay half-life for flags
  • ORACLE_STATE_PATH: Path to state file (default /data/oracle-state.json)
  • PAYER_OUTCOMES: Payer-signed outcomes, off (default), positive (only "delivered") or both. Needs PAYMENT_VERIFICATION=required, EXPECTED_CHAIN_ID and REPUTATION_ADDRESS. Set it the same on every operator
  • PAYER_OUTCOMES_PUBLISH: on serves each payer's signed payload in /evidence and lists them at /outcomes. Stays off on mainnet until the retention policy covers copies held by other operators
  • PAYER_OUTCOMES_PEERS: Base URLs of the other operators to import payer-signed outcomes from at the start of each epoch. Off on mainnet for the same reason
  • PAYER_OUTCOMES_IMPORT_GRACE_HOURS, PAYER_OUTCOME_WINDOW_DAYS, PAYER_NEGATIVE_MIN_AMOUNT: How long after its deadline a peer's outcome may still arrive (6 hours), how long after the payment a payer's outcome may replace a token holder's (14 days), and the smallest leg whose "not delivered" counts (PAYMENT_MIN_AMOUNT)
  • PAYER_NEGATIVE_HOLD_DAYS: How long a payer's "not delivered" is held before it counts (7 days, 1 to 30), from the time of the block its signature was checked at, or from its signature when that is later, and from the first operator's block once imported. A full refund from the agent to the payer inside the hold cancels it (ADR 0011). A negative whose refund read is final keeps the hold end it was read against, so a change applies only to negatives still being read (negative-hold.js:47-67). Set it the same on every operator
  • PAYMENT_CHAIN_IDS: Chains to read payments from beside the scored one (ADR 0009). Off by default: unset, no chain is read (payment-chains.js:41-43). Since 7 November 2026 both operators read Base Sepolia (84532) on, and the checker also reads Elysium testnet (99801) in shadow, which credits nothing (simulation.md:82-150, 0009-payments-on-base.md:253-254)
  • PAYMENT_CHAIN_<id>_RPC_URL, _ASSET, _FROM_BLOCK, _FEE_UNIT, _MIN_AMOUNT: Required for each chain. The minimum must be above 0, and the RPC URL is never logged or served (payment-chains.js:164-179, README.md:1117)
  • PAYMENT_CHAIN_<id>_MODE, _DELAY_SECONDS, _CHUNK, _PACE_MS, _MAX_BLOCKS: shadow by default, which credits nothing; 3600 seconds and at least 1800; 200 blocks a call; 500 ms between calls; 36000 blocks a scan (payment-chains.js:55-65, payment-chains.js:170, payment-chains.js:638)

Startup validation:

  • Missing required variables cause immediate failure.
  • Invalid epoch intervals or tolerance values abort startup.
  • Any numeric setting that is set and is not a plain number in range aborts startup, with every offending setting named. A blank value means unset and takes the default.
  • An ORACLE_MODE other than primary or checker, or both forms of the signing key, abort startup, as does a non-loopback bind with no write credential. A named ORACLE_TOKEN_<NAME> counts as one, for that refusal and for the startup warning that writes are unauthenticated (http-helpers.js:367-374, index.js:3944-3946).
  • Checker mode rejects fee gating configuration to prevent double-charging.
  • A payment chain aborts startup when PAYMENT_VERIFICATION=required, PAYMENT_SCAN_ENABLED=1 or EXPECTED_CHAIN_ID is missing, when it names the scored chain, or when one of its settings is unusable. So does a TX_WAIT_SECONDS too long for a proposal to land in time once a chain is on (payment-chains.js:153-198, index.js:194-210).

Section sources

Health Check and Metrics

Health endpoint:

  • Returns 200 when the state path is writable, last persistence succeeded, and the last successful epoch is within two epochs’ duration. Before the first success the two epochs are counted from process start, so a primary that has never completed an epoch reports 503 once it has been up that long.
  • Reports uptime, epoch timing, store writability, last persist status, where the state was loaded from (stateSource), attest cooldown, current epoch status, commit info, the chain the RPC was last seen on beside the configured one (chainId, expectedChainId), external feed status and chain id, payment scan status, and jobs feed status. With a payment chain configured it also reports paymentChains and paymentChainHold (index.js:3102-3108).

Metrics endpoint:

  • Exposes Prometheus-compatible text metrics including request counts, epoch counters, rate-limit hits, refused or unconfirmed transactions, epoch-fee charges, payment chain scans and the epochs they held (metrics.js:359-372), and cache statistics.

Operational guidance:

  • Monitor timeSinceLastEpochMs against EPOCH_HOURS to detect stalled epochs.
  • Alert on storeWritable=false or statePersisted=false.
  • Treat a stateSource other than file after a restart as a state that was recovered from the .bak or started empty.
  • Track externalFeed status and externalChainId to detect misconfiguration.
  • Alert when chainId differs from expectedChainId. chainId is the chain the node answered with at the last check (at startup and before every epoch), whether or not it matched. After a failover to another chain while the oracle runs, every epoch aborts while ok stays true until scoring goes stale (index.js:813-838, index.js:3053-3057, index.js:3073-3075).

Section sources

Persistent State and Backup Strategy

State contents:

  • Attestations, flags, links, attestation cooldowns, payment events (with each payer-signed outcome and when this oracle took it, acceptedAt), used payment transactions, divergence records, scan checkpoints (a payment chain's under paymentChainScanState, by chain id, or <id>/shadow for a chain in shadow, store.js:119-120, payment-chains.js:274-277), and timestamps.

Persistence model:

  • Atomic writes via temp file + rename + fsync to avoid partial corruption. The file is written in pieces of about 1 MB, so a save does not need several copies of it in memory, and the file it replaces is kept as <state>.bak (a hard link, so it costs no copy).
  • The file records a schemaVersion and the chainId it belongs to. Version 2 marks a file holding anything from a payment chain, which a build from before payment chains refuses. Version 3 marks one holding a payer's "not delivered", which a build from before the negative hold refuses, and version 4 one whose replay set has lost the keys of pruned payments, which a build from before the age limit on credits refuses. Anything else is written as version 1, so a rollback stays possible until the state needs a later version. A cursor from a chain in shadow does not count, and an older build drops it (store.js:211-235, store.js:604).
  • Periodic pruning of expired flags, payment events, and divergences to bound growth.
  • Writability probe integrated into /health.

Backup recommendations:

  • Snapshot the named volume or the underlying directory containing oracle-state.json regularly.
  • Copy the state file off the host on a schedule. The .bak is always one save behind and sits on the same disk.
  • Include both the state file and any related logs in your backup strategy.
  • Test restore procedures periodically to ensure recoverability: once a quarter, on a spare machine and never on a live box, with a key that holds no funds or role and new tokens.
  • operations-runbook.md:70-103 is the routine the Arc testnet operators have run since 1 November 2026: a nightly copy of the state volume, encrypted to an age key whose private half is on neither host, kept 14 days on the host and pushed to the other operator's host, where copies are kept 90 days. A third host checks every day that each receiver holds a copy less than 25 hours old. The restore drill is at operations-runbook.md:164-195.

Disaster recovery:

  • Stop the service first: it keeps its state in memory and rewrites the file at every save, so a file put in place while it runs is overwritten. Keep a copy of what is there, put the state file and its .bak back together from the newest copy, then start it; it runs an epoch at once.
  • A restore from an older copy loses what is on chain nowhere: a token holder's outcomes, payer-signed outcomes the other operator does not hold, flags, resolves, links, cooldowns and divergence records. Payments come back through the payment scan, and payer-signed outcomes the other operator holds come back through the import at the next epoch, unless their deadline passed more than PAYER_OUTCOMES_IMPORT_GRACE_HOURS before it runs.
  • On start, a state file that exists but cannot be read falls back to the .bak (the bad file is kept as <state>.corrupt-<time> and /health reports stateSource: "backup"). The process stops instead if there is no usable backup, if the file was written by a newer version, or if it was saved on a different chain from the one the RPC reports.
  • Verify /health indicates statePersisted=true and storeWritable=true.
  • Re-run a manual epoch if needed to catch up after recovery.
  • Once both operators have run an epoch since the restore, compare each agent's /evidence count and root with the other operator's, post again any token holder's outcome given after the copy was taken, and run node scripts/check-live.mjs outcomes.

Section sources

Scaling Considerations and Load Balancing

Horizontal scaling:

  • Each Oracle instance maintains its own state and proposes independently. For redundancy, run multiple instances behind a load balancer that routes read traffic and uses a single writer or coordinated writers.
  • If multiple writers exist, ensure nonce coordination and avoid duplicate charges; the code guards against double-charging in checker mode but operators must design their deployment carefully. Instances that share one wallet share a nonce, and the contract holds one pending score per agent, so a second operator should run as a checker (ORACLE_MODE=checker) with its own key, state and RPC provider rather than as a second proposer.

Load balancing strategies:

  • Route reads through Caddy/Nginx with rate limiting and caching headers.
  • Use connection pooling and keep-alive to RPC providers.
  • Prefer batched RPC calls; the client library batches concurrent calls automatically, at most 20 to a batch on the providers in chain.js and external.js.

Network considerations:

  • Keep admin endpoints loopback-only.
  • Use a shared Docker network for inter-service communication rather than host loopback.

Section sources

Security Hardening

  • Restrict public paths to read-only endpoints; keep write endpoints behind authentication and loopback.
  • Enforce CORS only for trusted origins.
  • Set security headers (HSTS, nosniff, referrer policy) and remove server identification.
  • Validate and fail fast on missing or malformed configuration at startup.
  • Keep the signing key out of the environment with ORACLE_PRIVATE_KEY_FILE, and limit each integration's token to the writes it needs with ORACLE_SCOPES_<NAME>.
  • Rate limit high-cost endpoints (/score, /evidence, /attest). A request target that is not a canonical path gets a 400 before routing, so a proxy that matches on a normalised path cannot be used to reach a route it did not allow.
  • Use least-privilege containers and restrict filesystem writes to the state volume (the mainnet compose file runs read-only with every capability dropped).

Section sources

Monitoring and Observability

  • Scrape /metrics with Prometheus.
  • Alert on:
    • epochsFailed > 0
    • proposeErrors, finalizeErrors
    • rateLimitHits spikes
    • paymentScanCanaryFailures
    • sigvara_oracle_tx_refused_total (a fee cap that trips, or a transaction sent and never seen mined) and sigvara_oracle_fee_charges_total{result="skipped"} (proposals are being refused)
    • timeSinceLastEpochMs exceeding EPOCH_HOURS * 2
    • storeWritable=false or statePersisted=false
    • stateSource other than file after a restart
    • sigvara_oracle_payer_outcome_import_refused_total, sigvara_oracle_payer_outcome_import_errors_total or sigvara_oracle_payer_outcome_conflicts_total climbing: a peer serving outcomes this oracle refuses, a peer or this node not answering, or the two operators holding different outcomes for one payment
  • From outside, scripts/check-live.mjs services fails on a schedule when /health does, and also watches overdue slashes and wallet floors (check-live.mjs:187-247).
  • Integrate with log aggregation for structured logs from Caddy and the Oracle service.

Section sources

Dependency Analysis

The Oracle service depends on:

  • Node.js runtime and standard libraries
  • dotenv for environment loading
  • ethers for blockchain interactions
  • Internal modules for chain access, external registries, scoring, merkle proofs, response caching, badge generation, job registry scanning, and metrics
graph LR
Index["index.js"] --> Chain["chain.js"]
Index --> External["external.js"]
Index --> Scoring["scoring.js"]
Index --> Merkle["merkle.js"]
Index --> Payments["payments.js"]
Index --> Store["store.js"]
Index --> Jobs["jobs.js"]
Index --> Metrics["metrics.js"]
Index --> Badge["badge.js"]
Index --> ResponseCache["response-cache.js"]
Loading

Diagram sources

Section sources

Performance Considerations

  • Epoch batching: Phase-one and guard reads are chunked in calls and paced 1.5 seconds apart to avoid RPC bursts; phase-two writes are sequential due to nonce constraints.
  • RPC batching: Concurrent calls issued in the same tick are batched by the client library, at most 20 to a batch.
  • Evidence caching: Large responses are cached to reduce recomputation cost, and Merkle leaves and per-agent roots are cached for evidence that has not changed.
  • Payment event pruning: Old events are pruned based on decay curves to bound state size.
  • Flag decay: Flags decay faster than payment evidence to reflect their nature.
  • State saves: The state file is streamed in pieces. Measured on Node 20, the peak memory of one save fell from 449 MB to 189 MB with 100,000 payments, and from 700 MB to 280 MB with 200,000. The mainnet compose file limits a container to 768 MB, so size memory for the peak of a save, not the size of the file.

Tuning guidelines:

  • The epoch reads agents 32 at a time and its pre-write guard reads 32 at a time; both are constants in runEpochInner, so changing them means editing the code. The settings are EPOCH_HOURS, LOG_CHUNK_SIZE, PAYMENT_SCAN_CHUNK, PAYMENT_SCAN_PACE_MS and PAYMENT_SCAN_CONFIRMATIONS.
  • Tune EPOCH_HOURS to balance freshness and RPC costs.
  • Configure JOBS_MAX_SPAN to cap job registry scans.
  • Monitor RPC throttling and adjust pacing or provider tier accordingly.

Section sources

Troubleshooting Guide

Common issues and resolutions:

  • Missing required environment variables: The process exits immediately with an error message.
  • Stalled epochs: Check /health timeSinceLastEpochMs and metrics epochsFailed; verify RPC connectivity and fee gating configuration. An epoch does not run while the RPC reports a chain other than EXPECTED_CHAIN_ID, or a clock more than MAX_CLOCK_SKEW_SECONDS from the host's; the log says which.
  • State persistence failures: Inspect storeWritable and statePersisted in /health, which fails while statePersisted is false. It returns neither the state path nor the error: both are in the log, where a failed save says [oracle] state persist FAILED: ... (index.js:3053-3067, store.js:636). Ensure the volume is mounted and has sufficient space. While a save is failing, /attest, /flag, /flag/resolve and /link answer 202 with durable: false. sigvara_oracle_attest_total{result="accepted_not_durable"} counts the attestations and sigvara_oracle_writes_not_durable_total counts the other three.
  • Rate limiting: High rateLimitHits indicate excessive requests; adjust client behavior; the oracle's limit is a constant in http-helpers.js (60 requests per caller per minute in each route's bucket), not a setting. If every visitor behind a proxy is limited at once, TRUSTED_PROXIES is missing.
  • Checker divergence: Review /divergence endpoints and compare evidence roots to identify discrepancies. A record's evidenceMatchesProposal says whether the checker's evidence as it stood at the proposal matched the proposal's root. A payer-signed outcome one operator holds and the other has not imported, or a recent restore, shows up as different roots.
  • Payer-signed outcomes slow to cross between operators: operations-runbook.md:197-306 covers the import log lines, the counters, node scripts/check-live.mjs outcomes, and what to do for each.

Recovery steps:

  • Restart the service after fixing configuration or permissions.
  • Trigger a manual epoch via /epoch if appropriate.
  • Restore state from backups if corruption is detected.

Section sources

Conclusion

Deploying the Oracle service in production requires careful attention to container orchestration, reverse proxy exposure, environment configuration, state persistence, and observability. By following the guidance above—binding to loopback, exposing only safe public paths, enabling robust health checks and metrics, and implementing disciplined backup and recovery—you can operate a resilient, auditable, and scalable Oracle service.

Appendices

Example Environment File Keys

  • RPC_URL
  • ORACLE_PRIVATE_KEY
  • ORACLE_PRIVATE_KEY_FILE
  • IDENTITY_ADDRESS
  • REPUTATION_ADDRESS
  • EXPECTED_CHAIN_ID
  • EPOCH_HOURS
  • HOST
  • PORT
  • FROM_BLOCK
  • LOG_CHUNK_SIZE
  • ORACLE_ADMIN_TOKEN
  • ORACLE_TOKEN_
  • ORACLE_SCOPES_
  • TRUSTED_PROXIES
  • TX_WAIT_SECONDS
  • MAX_FEE_GWEI
  • MAX_TX_COST
  • MAX_CLOCK_SKEW_SECONDS
  • FEE_REGISTRY_ADDRESS
  • EXTERNAL_RPC
  • EXTERNAL_IDENTITY_ADDRESS
  • EXTERNAL_REPUTATION_ADDRESS
  • EXTERNAL_LINK_FROM_BLOCK
  • ORACLE_MODE
  • DIVERGENCE_TOLERANCE
  • PRIMARY_EVIDENCE_URL
  • JOBS_MAX_SPAN
  • ATTEST_COOLDOWN_MS
  • FLAG_HALF_LIFE_DAYS
  • ORACLE_STATE_PATH
  • PAYER_OUTCOMES
  • PAYER_OUTCOMES_PUBLISH
  • PAYER_OUTCOMES_PEERS
  • PAYER_OUTCOMES_IMPORT_GRACE_HOURS
  • PAYER_OUTCOME_WINDOW_DAYS
  • PAYER_NEGATIVE_MIN_AMOUNT

Section sources

Home

Overview

Protocol Architecture

Smart Contracts Reference

Oracle Service

Reputation System Design

TypeScript SDK

Integration Guides

Deployment and Operations

Development and Testing

Advanced Topics

Clone this wiki locally