Skip to content

Oracle Production Deployment

Jump to bottom
RunTimeAdmin edited this page Oct 29, 2026 · 11 revisions

Oracle Production Deployment

**Referenced Files in This Document** - [docker-compose.oracle.yml](https://github.com/RunTimeAdmin/Sigvara/blob/main/docker-compose.oracle.yml) - [docker-compose.mainnet.yml](https://github.com/RunTimeAdmin/Sigvara/blob/main/docker-compose.mainnet.yml) - [.env.mainnet.example](https://github.com/RunTimeAdmin/Sigvara/blob/main/oracle/.env.mainnet.example) - [Dockerfile](https://github.com/RunTimeAdmin/Sigvara/blob/main/oracle/Dockerfile) - [Caddyfile.oracle.example](https://github.com/RunTimeAdmin/Sigvara/blob/main/oracle/Caddyfile.oracle.example) - [nginx-checker.conf.example](https://github.com/RunTimeAdmin/Sigvara/blob/main/oracle/nginx-checker.conf.example) - [package.json](https://github.com/RunTimeAdmin/Sigvara/blob/main/oracle/package.json) - [index.js](https://github.com/RunTimeAdmin/Sigvara/blob/main/oracle/index.js) - [store.js](https://github.com/RunTimeAdmin/Sigvara/blob/main/oracle/store.js)

Table of Contents

  1. Introduction
  2. Project Structure
  3. Core Components
  4. Architecture Overview
  5. Detailed Component Analysis
  6. Dependency Analysis
  7. Performance Considerations
  8. Troubleshooting Guide
  9. Conclusion
  10. Appendices

Introduction

This document provides production-grade deployment guidance for the Sigvara Oracle service. It covers container orchestration with Docker Compose, reverse proxy configuration using Caddy or Nginx, environment configuration for blockchain RPCs and external integrations, scaling patterns, health checks, monitoring, security hardening, backup and disaster recovery procedures, and performance tuning recommendations. The guidance is grounded in the repository’s Oracle implementation and deployment artifacts.

Project Structure

The Oracle service is a Node.js application packaged as a container image and orchestrated via Docker Compose. Key deployment-related files include:

  • Container build definition
  • Compose file defining service, networking, and volumes
  • Reverse proxy examples for public exposure (Caddy and Nginx)
  • Application entrypoint and HTTP API
  • Persistent state storage module
graph TB
subgraph "Host"
Caddy["Caddy"]
Nginx["Nginx"]
end
subgraph "Docker Network sigvara-mesh"
Oracle["Oracle Service<br/>port 3030"]
StateVol["Volume oracle_state<br/>(/data/oracle-state.json)"]
end
Chain["Blockchain RPC"]
ExternalRPC["External RPC / ERC-8004"]
Caddy --> Oracle
Nginx -.-> Oracle
Oracle --> Chain
Oracle --> ExternalRPC
Oracle --> StateVol
Loading

Diagram sources

Section sources

Core Components

  • Oracle service process: Node.js HTTP server exposing read endpoints (/health, /metrics, /score, /evidence, /badge), write endpoints (/attest, /flag, /link, /epoch), and checker-specific endpoints (/divergence).
  • Persistent state store: JSON-backed durable store for attestations, flags, links, payment events, cooldowns, divergence records, and scan checkpoints.
  • Blockchain integration: Reads/writes to the primary chain and optional external registry feeds.
  • Reverse proxies: Caddy example for public read surface; Nginx example for checker operator.

Key responsibilities:

  • Orchestrate epoch scoring, payment scanning, and on-chain proposals/finalizations.
  • Provide auditable evidence and scores over HTTP.
  • Persist non-reproducible state to a mounted volume.
  • Expose metrics for Prometheus scraping.

Section sources

Architecture Overview

The Oracle runs inside a container bound to host loopback. A reverse proxy exposes only the necessary public paths while keeping administrative endpoints private. State persists to a named volume. The service communicates with one or more blockchain RPCs and optionally an external RPC for ERC-8004 data.

sequenceDiagram
participant Client as "Client"
participant Caddy as "Caddy"
participant Oracle as "Oracle Service"
participant Store as "State Store"
participant Chain as "Blockchain RPC"
participant Ext as "External RPC"
Client->>Caddy : GET /score/ : didHash
Caddy->>Oracle : reverse_proxy : 3030
Oracle->>Chain : getAgentInfo(), getTotalScore()
Oracle->>Ext : externalScoreFor() (optional)
Oracle-->>Caddy : JSON score
Caddy-->>Client : 200 OK
Note over Oracle,Store : Attestations, flags, payments, divergences persisted to /data
Loading

Diagram sources

Detailed Component Analysis

Docker Compose Orchestration

  • Service name and container name are defined for stable DNS resolution within Docker networks.
  • The service binds to host loopback only, minimizing exposure.
  • Environment variables are loaded from an .env file.
  • A named volume persists attestation/flag/payment state across restarts and rebuilds.
  • The service joins a shared external network used by other components (for example, CounterAudit).

Operational notes:

  • Ensure the external network exists before starting the stack.
  • Mount a persistent volume at /data to avoid losing state.
  • Keep admin endpoints off the public internet; use loopback access or a secure management plane.

Mainnet:

  • docker-compose.mainnet.yml is the Arc mainnet arrangement. The primary, the checker and the watcher are three profiles of one file; COMPOSE_PROFILES in a host's .env picks one, and the three belong on three separate hosts.
  • It builds from SIGVARA_REF, a full 40-character commit hash. A deploy is a change to that line followed by docker compose up -d; a restart reruns the same image and fetches nothing.
  • The signing key is a file secret, read through ORACLE_PRIVATE_KEY_FILE, so it is not in the container's environment.
  • Each container runs as the unprivileged node user on a read-only root filesystem with every capability dropped and a 768 MB memory limit. The only writable places are /data and /tmp. Only the primary joins the edge network it shares with the reverse proxy.

Section sources

Container Image Build

  • Base image: Node 24 Alpine, pinned by digest (the VPS compose files, which fetch one pinned commit at container start, still run node:20-alpine).
  • Deterministic dependency installation using package-lock.json (npm ci --omit=dev --ignore-scripts).
  • Source copied wholesale; .dockerignore excludes secrets, local state, tests and probe scripts.
  • Build-time static check validates that all local requires resolve, preventing runtime MODULE_NOT_FOUND errors.
  • The commit the image was built from is passed as SIGVARA_COMMIT, which /health reports. When it is given, the build stops unless it is a full 40-character hash.
  • The process runs as the unprivileged node user, /data is created owned by that user, and a HEALTHCHECK probes /health.
  • Entrypoint runs index.js.

Best practices:

  • Pin base image versions and lockfiles.
  • Run the image with minimal privileges and read-only root filesystem where possible (the mainnet compose file does).
  • Use multi-stage builds if you need to separate build tools from runtime.

Section sources

Reverse Proxy Configuration: Caddy

Public read surface:

  • Only explicitly listed paths are proxied: /health, /score/, /evidence/, /badge/*.
  • POST /attest is proxied with CORS headers and no-store cache policy.
  • Read endpoints set short Cache-Control to absorb bursts without serving stale data.
  • X-Forwarded-For is preserved so per-client rate limiting works behind the proxy. The oracle believes the header only from loopback or from a peer listed in TRUSTED_PROXIES. Caddy reaches the oracle from a Docker bridge address, so set TRUSTED_PROXIES to the proxy network (for example 172.16.0.0/12); without it every visitor shares one rate-limit bucket.
  • Security headers include HSTS, nosniff, referrer policy, and Server header removal.

Important behaviors:

  • The oracle listens on 127.0.0.1:3030; Caddy reaches it via Docker service name, not host loopback.
  • Administrative endpoints (/flag, /flag/resolve, /link, /epoch) remain loopback-only.
  • /metrics is intentionally excluded from the public proxy.

Section sources

Reverse Proxy Configuration: Nginx (Checker Operator)

Use this when the host already runs Nginx and you need to expose checker endpoints without running another web server.

Exposed endpoints:

  • /health: liveness and watcher polling
  • /divergence and /divergence/:didHash: disagreement collection
  • /evidence/: DID-scoped evidence

Security:

  • Write endpoints (/attest, /flag, /link, /epoch) are not exposed publicly.
  • Security headers and server_tokens off are applied.

Deployment steps:

  • Place the config under sites-available, symlink into sites-enabled, test with nginx -t, reload, and obtain TLS via certbot.

Section sources

Environment Variables and Configuration

Required variables:

  • RPC_URL: Primary blockchain RPC endpoint
  • ORACLE_PRIVATE_KEY: Oracle wallet private key, or ORACLE_PRIVATE_KEY_FILE, the path of a file that holds it (set one, not both; a file secret keeps the key out of the container's environment)
  • IDENTITY_ADDRESS: Identity contract address
  • REPUTATION_ADDRESS: Reputation contract address

Optional variables:

  • EPOCH_HOURS: Epoch interval in hours
  • HOST/PORT: Bind address and port (default 127.0.0.1:3030)
  • FROM_BLOCK: Starting block for scans
  • LOG_CHUNK_SIZE: Batch size for log processing
  • ORACLE_ADMIN_TOKEN: Admin token for protected endpoints
  • ORACLE_TOKEN_: Additional named tokens
  • ORACLE_SCOPES_: Restricts that token to some of attest, flag, link and epoch
  • EXPECTED_CHAIN_ID: The chain this oracle is for (5042 on Arc mainnet, 5042002 on the testnet). With it set, nothing is read for scoring or signed while the RPC reports another chain, and a state file saved on another chain is refused
  • TRUSTED_PROXIES: Addresses or CIDR ranges whose X-Forwarded-For is believed, besides loopback
  • TX_WAIT_SECONDS, MAX_FEE_GWEI, MAX_TX_COST: How long a transaction is waited for before one replacement, and the fee and total-cost ceilings the oracle will sign under
  • MAX_CLOCK_SKEW_SECONDS: How far the chain's clock may be from the host's before an epoch is not run
  • FEE_REGISTRY_ADDRESS: Optional fee gating registry
  • EXTERNAL_RPC, EXTERNAL_IDENTITY_ADDRESS, EXTERNAL_REPUTATION_ADDRESS: ERC-8004 feed configuration
  • ORACLE_MODE=checker: Enable checker mode
  • DIVERGENCE_TOLERANCE: Allowed divergence threshold
  • PRIMARY_EVIDENCE_URL: URL to primary oracle’s evidence endpoint
  • JOBS_MAX_SPAN: Maximum block span for jobs queries
  • ATTEST_COOLDOWN_MS: Cooldown between attestations
  • FLAG_HALF_LIFE_DAYS: Decay half-life for flags
  • ORACLE_STATE_PATH: Path to state file (default /data/oracle-state.json)

Startup validation:

  • Missing required variables cause immediate failure.
  • Invalid epoch intervals or tolerance values abort startup.
  • Any numeric setting that is set and is not a plain number in range aborts startup, with every offending setting named. A blank value means unset and takes the default.
  • An ORACLE_MODE other than primary or checker, or both forms of the signing key, abort startup, as does a non-loopback bind with no write credential.
  • Checker mode rejects fee gating configuration to prevent double-charging.

Section sources

Health Check and Metrics

Health endpoint:

  • Returns 200 when the state path is writable, last persistence succeeded, and the last successful epoch is within two epochs’ duration. Before the first success the two epochs are counted from process start, so a primary that has never completed an epoch reports 503 once it has been up that long.
  • Reports uptime, epoch timing, store writability, last persist status, where the state was loaded from (stateSource), attest cooldown, current epoch status, commit info, the chain the RPC was last seen on beside the configured one (chainId, expectedChainId), external feed status and chain id, payment scan status, and jobs feed status.

Metrics endpoint:

  • Exposes Prometheus-compatible text metrics including request counts, epoch counters, rate-limit hits, refused or unconfirmed transactions, epoch-fee charges, and cache statistics.

Operational guidance:

  • Monitor timeSinceLastEpochMs against EPOCH_HOURS to detect stalled epochs.
  • Alert on storeWritable=false or statePersisted=false.
  • Treat a stateSource other than file after a restart as a state that was recovered from the .bak or started empty.
  • Track externalFeed status and externalChainId to detect misconfiguration.

Section sources

Persistent State and Backup Strategy

State contents:

  • Attestations, flags, links, attestation cooldowns, payment events, used payment transactions, divergence records, scan checkpoints, and timestamps.

Persistence model:

  • Atomic writes via temp file + rename + fsync to avoid partial corruption. The file is written in pieces of about 1 MB, so a save does not need several copies of it in memory, and the file it replaces is kept as <state>.bak (a hard link, so it costs no copy).
  • The file records a schemaVersion and the chainId it belongs to.
  • Periodic pruning of expired flags, payment events, and divergences to bound growth.
  • Writability probe integrated into /health.

Backup recommendations:

  • Snapshot the named volume or the underlying directory containing oracle-state.json regularly.
  • Copy the state file off the host on a schedule. The .bak is always one save behind and sits on the same disk.
  • Include both the state file and any related logs in your backup strategy.
  • Test restore procedures periodically to ensure recoverability.

Disaster recovery:

  • Restore the state file to the same path and restart the service.
  • On start, a state file that exists but cannot be read falls back to the .bak (the bad file is kept as <state>.corrupt-<time> and /health reports stateSource: "backup"). The process stops instead if there is no usable backup, if the file was written by a newer version, or if it was saved on a different chain from the one the RPC reports.
  • Verify /health indicates statePersisted=true and storeWritable=true.
  • Re-run a manual epoch if needed to catch up after recovery.

Section sources

Scaling Considerations and Load Balancing

Horizontal scaling:

  • Each Oracle instance maintains its own state and proposes independently. For redundancy, run multiple instances behind a load balancer that routes read traffic and uses a single writer or coordinated writers.
  • If multiple writers exist, ensure nonce coordination and avoid duplicate charges; the code guards against double-charging in checker mode but operators must design their deployment carefully. Instances that share one wallet share a nonce, and the contract holds one pending score per agent, so a second operator should run as a checker (ORACLE_MODE=checker) with its own key, state and RPC provider rather than as a second proposer.

Load balancing strategies:

  • Route reads through Caddy/Nginx with rate limiting and caching headers.
  • Use connection pooling and keep-alive to RPC providers.
  • Prefer batched RPC calls; the client library batches concurrent calls automatically.

Network considerations:

  • Keep admin endpoints loopback-only.
  • Use a shared Docker network for inter-service communication rather than host loopback.

Section sources

Security Hardening

  • Restrict public paths to read-only endpoints; keep write endpoints behind authentication and loopback.
  • Enforce CORS only for trusted origins.
  • Set security headers (HSTS, nosniff, referrer policy) and remove server identification.
  • Validate and fail fast on missing or malformed configuration at startup.
  • Keep the signing key out of the environment with ORACLE_PRIVATE_KEY_FILE, and limit each integration's token to the writes it needs with ORACLE_SCOPES_<NAME>.
  • Rate limit high-cost endpoints (/score, /evidence, /attest). A request target that is not a canonical path gets a 400 before routing, so a proxy that matches on a normalised path cannot be used to reach a route it did not allow.
  • Use least-privilege containers and restrict filesystem writes to the state volume (the mainnet compose file runs read-only with every capability dropped).

Section sources

Monitoring and Observability

  • Scrape /metrics with Prometheus.
  • Alert on:
    • epochsFailed > 0
    • proposeErrors, finalizeErrors
    • rateLimitHits spikes
    • paymentScanCanaryFailures
    • sigvara_oracle_tx_refused_total (a fee cap that trips, or a transaction sent and never seen mined) and sigvara_oracle_fee_charges_total{result="skipped"} (proposals are being refused)
    • timeSinceLastEpochMs exceeding EPOCH_HOURS * 2
    • storeWritable=false or statePersisted=false
    • stateSource other than file after a restart
  • Integrate with log aggregation for structured logs from Caddy and the Oracle service.

Section sources

Dependency Analysis

The Oracle service depends on:

  • Node.js runtime and standard libraries
  • dotenv for environment loading
  • ethers for blockchain interactions
  • Internal modules for chain access, external registries, scoring, merkle proofs, response caching, badge generation, job registry scanning, and metrics
graph LR
Index["index.js"] --> Chain["chain.js"]
Index --> External["external.js"]
Index --> Scoring["scoring.js"]
Index --> Merkle["merkle.js"]
Index --> Payments["payments.js"]
Index --> Store["store.js"]
Index --> Jobs["jobs.js"]
Index --> Metrics["metrics.js"]
Index --> Badge["badge.js"]
Index --> ResponseCache["response-cache.js"]
Loading

Diagram sources

Section sources

Performance Considerations

  • Epoch batching: Phase-one reads are chunked to avoid RPC bursts; phase-two writes are sequential due to nonce constraints.
  • RPC batching: Concurrent calls issued in the same tick are batched by the client library.
  • Evidence caching: Large responses are cached to reduce recomputation cost, and Merkle leaves and per-agent roots are cached for evidence that has not changed.
  • Payment event pruning: Old events are pruned based on decay curves to bound state size.
  • Flag decay: Flags decay faster than payment evidence to reflect their nature.
  • State saves: The state file is streamed in pieces. Measured on Node 20, the peak memory of one save fell from 449 MB to 189 MB with 100,000 payments, and from 700 MB to 280 MB with 200,000. The mainnet compose file limits a container to 768 MB, so size memory for the peak of a save, not the size of the file.

Tuning guidelines:

  • The epoch reads agents 32 at a time and its pre-write guard reads 32 at a time; both are constants in runEpochInner, so changing them means editing the code. The settings are EPOCH_HOURS, LOG_CHUNK_SIZE, PAYMENT_SCAN_CHUNK, PAYMENT_SCAN_PACE_MS and PAYMENT_SCAN_CONFIRMATIONS.
  • Tune EPOCH_HOURS to balance freshness and RPC costs.
  • Configure JOBS_MAX_SPAN to cap job registry scans.
  • Monitor RPC throttling and adjust pacing or provider tier accordingly.

Section sources

Troubleshooting Guide

Common issues and resolutions:

  • Missing required environment variables: The process exits immediately with an error message.
  • Stalled epochs: Check /health timeSinceLastEpochMs and metrics epochsFailed; verify RPC connectivity and fee gating configuration. An epoch does not run while the RPC reports a chain other than EXPECTED_CHAIN_ID, or a clock more than MAX_CLOCK_SKEW_SECONDS from the host's; the log says which.
  • State persistence failures: Inspect storeWritable and statePersistError in /health; ensure the volume is mounted and has sufficient space. While a save is failing, /attest, /flag, /flag/resolve and /link answer 202 with durable: false. sigvara_oracle_attest_total{result="accepted_not_durable"} counts the attestations and sigvara_oracle_writes_not_durable_total counts the other three.
  • Rate limiting: High rateLimitHits indicate excessive requests; adjust client behavior; the oracle's limit is a constant in http-helpers.js (60 requests per caller per minute in each route's bucket), not a setting. If every visitor behind a proxy is limited at once, TRUSTED_PROXIES is missing.
  • Checker divergence: Review /divergence endpoints and compare evidence roots to identify discrepancies.

Recovery steps:

  • Restart the service after fixing configuration or permissions.
  • Trigger a manual epoch via /epoch if appropriate.
  • Restore state from backups if corruption is detected.

Section sources

Conclusion

Deploying the Oracle service in production requires careful attention to container orchestration, reverse proxy exposure, environment configuration, state persistence, and observability. By following the guidance above—binding to loopback, exposing only safe public paths, enabling robust health checks and metrics, and implementing disciplined backup and recovery—you can operate a resilient, auditable, and scalable Oracle service.

Appendices

Example Environment File Keys

  • RPC_URL
  • ORACLE_PRIVATE_KEY
  • ORACLE_PRIVATE_KEY_FILE
  • IDENTITY_ADDRESS
  • REPUTATION_ADDRESS
  • EXPECTED_CHAIN_ID
  • EPOCH_HOURS
  • HOST
  • PORT
  • FROM_BLOCK
  • LOG_CHUNK_SIZE
  • ORACLE_ADMIN_TOKEN
  • ORACLE_TOKEN_
  • ORACLE_SCOPES_
  • TRUSTED_PROXIES
  • TX_WAIT_SECONDS
  • MAX_FEE_GWEI
  • MAX_TX_COST
  • MAX_CLOCK_SKEW_SECONDS
  • FEE_REGISTRY_ADDRESS
  • EXTERNAL_RPC
  • EXTERNAL_IDENTITY_ADDRESS
  • EXTERNAL_REPUTATION_ADDRESS
  • ORACLE_MODE
  • DIVERGENCE_TOLERANCE
  • PRIMARY_EVIDENCE_URL
  • JOBS_MAX_SPAN
  • ATTEST_COOLDOWN_MS
  • FLAG_HALF_LIFE_DAYS
  • ORACLE_STATE_PATH

Section sources

Home

Overview

Protocol Architecture

Smart Contracts Reference

Oracle Service

Reputation System Design

TypeScript SDK

Integration Guides

Deployment and Operations

Development and Testing

Advanced Topics

Clone this wiki locally