Skip to content

Architecture

Jump to bottom
lacausecrypto edited this page Apr 6, 2026 · 3 revisions

Architecture

How OCC works under the hood.

System Overview

┌──────────────────┐         ┌────────────────────┐
│   Claude Code    │──MCP──▶│    MCP Server       │
│   Claude Desktop │  stdio  │    (index.ts)       │
└──────────────────┘         │  28 tools exposed   │
                             └────────┬───────────┘
                                      │
┌──────────────────┐         ┌────────▼───────────┐        ┌──────────────────┐
│   curl / Browser │──HTTP──▶│    REST Server      │◀──────│  React Dashboard │
│   Any client     │  :4242  │    (rest.ts)        │ :5173 │  (frontend-react)│
└──────────────────┘         │  102 endpoints      │       │  Canvas + Chat   │
                             │  SSE streaming      │       └──────────────────┘
                             └────────┬───────────┘
                                      │
          ┌───────────────────────────┼───────────────────────────┐
          ▼                           ▼                           ▼
 ┌──────────────┐            ┌──────────────┐           ┌──────────────┐
 │    Loader     │            │   Executor   │           │  Scheduler   │
 │  (loader.ts)  │            │(executor.ts) │           │(scheduler.ts)│
 │ YAML parsing  │            │ Chain runner  │           │ Cron jobs    │
 │ Zod validation│            │ Wave parallel │           │ Auto-execute │
 │ Dep. graph    │            │ Checkpoints   │           └──────────────┘
 └──────┬───────┘            └──────┬───────┘
        │                           │
        ▼                    ┌──────▼───────┐
 ┌──────────┐                │  Providers    │
 │ chains/   │                │(providers.ts) │
 │  (YAML)   │                │ Claude CLI    │
 └──────────┘                │ OpenRouter    │
                             │ OpenAI        │
 ┌──────────┐                │ Ollama        │
 │ SQLite DB │◀──────────────│ HuggingFace   │
 │ (storage) │  checkpoints  │ Custom        │
 │ (queue)   │  executions   └───────────────┘
 └──────────┘

Module Breakdown (18 modules)

Core

Module Description
rest.ts Express HTTP server — 102 endpoints, SSE streaming, CORS, security headers
index.ts MCP server — 28 tools over stdio transport
executor.ts Chain execution engine — wave parallelism, checkpoints, cancellation
loader.ts YAML loading, Zod validation, dependency graph (topological sort)
pipeline-loader.ts Pipeline YAML loading and validation
pipeline-executor.ts Pipeline orchestration (chain-level dependency graph)

Infrastructure

Module Description
storage.ts SQLite persistence — executions, step checkpoints, WAL mode
queue.ts SQLite job queue — priority scheduling, worker pool, retry
scheduler.ts Cron-based chain scheduling with node-cron
providers.ts Multi-LLM provider management — 6 provider types, AES-256-GCM key encryption
claude-runner.ts Claude CLI subprocess management — spawn, stream, timeout, kill
mcp-client.ts External MCP server client — connect, discover tools, call

Tools

Module Description
pretool-executor.ts 29 pre-tool types execution (http_fetch, bash, vector_query, etc.)
pretool-extras.ts Advanced pre-tools: state management, vector DB, graph DB, semantic cache
linter.ts Chain linter — 20+ lint rules, dry-run execution planner
style-extractor.ts CSS/color extraction from URLs (Playwright headless for CSS-in-JS)
blob.ts BLOB canvas — organic workflow graph with autonomous agent loop
logger.ts Structured logging with level filtering

loader.ts — Chain Loading & Validation

  1. YAML parsing — js-yaml converts YAML to JavaScript objects
  2. Zod validation — Every field validated against strict schema
  3. Dependency graph — Topological sort (Kahn's algorithm) produces execution waves
  4. Path resolution — Uses fileURLToPath for cross-platform compatibility (handles spaces in paths)
Chain YAML → js-yaml → Zod safeParse → buildDependencyGraph → waves[]

Waves are groups of steps with no dependencies between them:

Wave 1: [step_a, step_b, step_c]     ← parallel
Wave 2: [step_d]                       ← depends on a,b
Wave 3: [step_e, step_f]              ← depends on d

executor.ts — Chain Execution Engine

The core execution loop:

for each wave in dependency_graph:
    await Promise.all(
        wave.map(step => executeStep(step))
    )

Step execution flow:

1. Check condition → skip if false
2. Check cache → return cached if valid
3. Run pre-tools (29 types) → inject variables
4. Resolve {variables} in prompt
5. Route to LLM provider (Claude CLI / OpenRouter / OpenAI / Ollama / HuggingFace)
6. Stream output → SSE events + mini-terminal
7. Validate output (guardrails)
8. Store result in variables map
9. Save to cache if enabled
10. Persist checkpoint to SQLite

Process management (Claude CLI):

  • Each step spawns claude --print --output-format stream-json
  • Timeout: configurable per-step, default 5 minutes
  • Escalation: SIGTERM → wait 3s → SIGKILL
  • Heartbeat: every 30s, checks if process is still alive
  • Cancellation: kills all active processes for an execution

Multi-provider routing:

  • Steps specify model: "provider/model" or provider: "openrouter"
  • Auto-detection: claude-* → Claude CLI, gpt-* → OpenAI, etc.
  • OpenAI-compatible: streaming and non-streaming, with tool calling support

providers.ts — LLM Provider Management

Provider Type API Format Key Required
Claude claude CLI subprocess No (CLI auth)
OpenRouter openrouter OpenAI-compatible Yes
OpenAI openai OpenAI native Yes
Ollama ollama OpenAI-compatible (local) No
HuggingFace huggingface OpenAI-compatible (router.huggingface.co) Optional
Custom custom OpenAI-compatible Yes

API keys encrypted with AES-256-GCM (scrypt key derivation) at rest.

storage.ts — SQLite Persistence

  • WAL mode for concurrent read/write
  • Executions table — ID, chain name, status, input, result, timestamps
  • Steps table — checkpoint per step with output, tokens, duration
  • Auto-purge — executions older than EXECUTION_MAX_AGE_DAYS (default 7)

queue.ts — Job Queue

  • SQLite-backed persistent queue
  • Priority scheduling (1-10, higher = first)
  • Configurable worker pool (default 5)
  • Retry with exponential backoff
  • Stats: queued, running, done, errored, avg wait time

Data Flow

Variable Resolution

Variables flow through the system:

Chain Input: { topic: "AI" }
    ↓
Variables Map: { "input.topic": "AI", "topic": "AI" }
    ↓
Step 1 runs → output = "Research findings..."
    ↓
Variables Map: { "input.topic": "AI", "research": "Research findings..." }
    ↓
Step 2 prompt: "Summarize: {research}" → "Summarize: Research findings..."

Persistence

Executions    → SQLite (occ-main.db) — step checkpoints, token usage, timing
Queue         → SQLite (occ-queue.db) — persistent job queue
Schedules     → JSON (schedules.json) — cron definitions
Cache         → SQLite (occ-main.db) — prompt hash → cached output
State         → SQLite (occ-state.db) — key-value store for chains
Vectors       → SQLite (occ-vector.db) — FTS5 text embeddings
Graph         → SQLite (occ-graph.db) — triple store (subject-predicate-object)
LLM Providers → JSON (llm-providers.json) — encrypted API keys
BLOB Sessions → JSON (blobs/) — canvas state per session

Error Recovery

Scenario Recovery
Step timeout SIGTERM → 3s → SIGKILL, step marked error
Process crash Heartbeat detects within 30s, step marked error
Server restart Orphaned "running" executions marked error on load
Invalid YAML Rejected at load time with Zod error details
Claude CLI missing Detected at startup, clear error message
Execution cancelled All child processes killed, pending steps skipped
Cache corruption Silent fallback to re-execution
Provider API error Retries per provider, error propagated to step
SQLite lock WAL mode prevents contention; queue retries on busy

See Also

Clone this wiki locally