M8: Scheduling and hardening
Phase 8: scheduling and hardening, the final roadmap phase. Epiphany now runs unattended: a scheduled job refreshes data on its own, the server recovers cleanly from a kill, the audit log is bounded, and the performance and memory budgets are met.
Highlights
- Scheduled jobs (ADR-0013). A job is an ordered sequence of a cube's flows on an
Intervaltrigger, declared as model-as-code ([[job]], byte-identical round-trip). A single in-process reconcile loop wakes on the injected clock, computes which jobs are due as a pure function of the declared triggers and the durable last-fired ledger, and runs them through the same validated apply path an interactive flow run uses. - Determinism preserved end to end. Real wall-clock time enters at exactly one point: the loop's tick. That value is frozen as
fire_millisand carried down into the flow run, the run record, and the audit timestamp; no downstream code re-reads a clock. The whole scheduler is provable under aManualClockwith no real timers, which is how the acceptance suite drives it. - Durable run ledger with convergent recovery. Runs are recorded in an append-only, CRC-framed ledger (its own file, neither the WAL nor the audit stream) that reuses the proven torn-tail truncation. A run in flight at a crash recovers as interrupted and its firing re-derives as due; a scheduled write is committed through the all-or-nothing batch path, so it is durable and survives a restart. The ledger is bounded: it compacts to the newest runs plus each job's latest success, so a frequently-firing job cannot grow it without limit.
- Async run surface. Jobs have REST CRUD (each step validated against an existing flow), a manual kick that runs on a blocking thread, and run queries (
GET /runs,GET /runs/{id}). Every firing and run is audited (job identity only, no secrets, RG-13). - Audit retention and rotation (ADR-0010). The audit stream is now bounded by a record cap and an optional age window, compacted crash-safely via temp-then-rename, with sequence numbers preserved. Configurable via
EPIPHANY_AUDIT_MAX_RECORDSandEPIPHANY_AUDIT_RETENTION_DAYS. - Operations. Graceful shutdown and a zero-config
EPIPHANY_*surface (bind, data dir, sessions, scheduler tick, audit and run-ledger retention). The scheduler defaults on with a 1-second tick and can be disabled withEPIPHANY_SCHEDULER_TICK_SECS=0.
Quality
Gated end to end by a deterministic acceptance suite (epiphany-api/tests/m8_acceptance.rs): a job fires on schedule and commits, does not fire before its interval, an interrupted firing recovers and re-derives as due, a scheduled write survives a store reopen, and the job REST surface validates steps, runs a job manually, and lists runs. The performance and memory mandate is validated and documented (docs/PERFORMANCE.md): bulk-load about 13 M cells/sec/core (budget about 1 M), a cold consolidated read about 10 ms (budget p99 under about 1 s), bytes-per-cell within the 24-byte budget (CI-enforced), and a scheduler scale benchmark for the reconcile due-scan. Hardened by a multi-agent adversarial review that caught and fixed four real defects before release: an audit timestamp that re-read the clock, an unbounded run ledger, a manual kick that blocked the async runtime, and the scheduler authorization model (documented as a deliberate system-actor design). Tests stay dependency-free, the supply-chain audit is green, and the single self-contained binary is built for Linux x86_64/aarch64, Windows x86_64, and macOS aarch64.
This completes the nine-phase roadmap.