Skip to content

M8: Scheduling and hardening

Choose a tag to compare

@WilliamSmithEdwardSandbox WilliamSmithEdwardSandbox released this 15 Jun 02:23

Phase 8: scheduling and hardening, the final roadmap phase. Epiphany now runs unattended: a scheduled job refreshes data on its own, the server recovers cleanly from a kill, the audit log is bounded, and the performance and memory budgets are met.

Highlights

  • Scheduled jobs (ADR-0013). A job is an ordered sequence of a cube's flows on an Interval trigger, declared as model-as-code ([[job]], byte-identical round-trip). A single in-process reconcile loop wakes on the injected clock, computes which jobs are due as a pure function of the declared triggers and the durable last-fired ledger, and runs them through the same validated apply path an interactive flow run uses.
  • Determinism preserved end to end. Real wall-clock time enters at exactly one point: the loop's tick. That value is frozen as fire_millis and carried down into the flow run, the run record, and the audit timestamp; no downstream code re-reads a clock. The whole scheduler is provable under a ManualClock with no real timers, which is how the acceptance suite drives it.
  • Durable run ledger with convergent recovery. Runs are recorded in an append-only, CRC-framed ledger (its own file, neither the WAL nor the audit stream) that reuses the proven torn-tail truncation. A run in flight at a crash recovers as interrupted and its firing re-derives as due; a scheduled write is committed through the all-or-nothing batch path, so it is durable and survives a restart. The ledger is bounded: it compacts to the newest runs plus each job's latest success, so a frequently-firing job cannot grow it without limit.
  • Async run surface. Jobs have REST CRUD (each step validated against an existing flow), a manual kick that runs on a blocking thread, and run queries (GET /runs, GET /runs/{id}). Every firing and run is audited (job identity only, no secrets, RG-13).
  • Audit retention and rotation (ADR-0010). The audit stream is now bounded by a record cap and an optional age window, compacted crash-safely via temp-then-rename, with sequence numbers preserved. Configurable via EPIPHANY_AUDIT_MAX_RECORDS and EPIPHANY_AUDIT_RETENTION_DAYS.
  • Operations. Graceful shutdown and a zero-config EPIPHANY_* surface (bind, data dir, sessions, scheduler tick, audit and run-ledger retention). The scheduler defaults on with a 1-second tick and can be disabled with EPIPHANY_SCHEDULER_TICK_SECS=0.

Quality

Gated end to end by a deterministic acceptance suite (epiphany-api/tests/m8_acceptance.rs): a job fires on schedule and commits, does not fire before its interval, an interrupted firing recovers and re-derives as due, a scheduled write survives a store reopen, and the job REST surface validates steps, runs a job manually, and lists runs. The performance and memory mandate is validated and documented (docs/PERFORMANCE.md): bulk-load about 13 M cells/sec/core (budget about 1 M), a cold consolidated read about 10 ms (budget p99 under about 1 s), bytes-per-cell within the 24-byte budget (CI-enforced), and a scheduler scale benchmark for the reconcile due-scan. Hardened by a multi-agent adversarial review that caught and fixed four real defects before release: an audit timestamp that re-read the clock, an unbounded run ledger, a manual kick that blocked the async runtime, and the scheduler authorization model (documented as a deliberate system-actor design). Tests stay dependency-free, the supply-chain audit is green, and the single self-contained binary is built for Linux x86_64/aarch64, Windows x86_64, and macOS aarch64.

This completes the nine-phase roadmap.