Skip to content

Roadmap and Production Readiness

Jump to bottom
Jonathan D.A. Jewell edited this page Jul 11, 2026 · 1 revision

Roadmap and Production Readiness

Statistikles ships a real, tested symbolic core wrapped in a deliberately honest set of experimental surfaces. This page describes the production-readiness program — a ten-dimension audit turned into a set of self-contained work orders — what has landed, and what remains.

Grounded in .claude/tasks/prod-readiness/ (the execution pack), ROADMAP.adoc, and the current state of the tree. For the enforcement work at the heart of it see Security-and-Threat-Model; for how to pick up a task see the Development-Guide.

Where the project stands

From ROADMAP.adoc:

  • The symbolic library is real and tested — ~40 statistics modules under src/stats/ (41 files), a single neural/symbolic dispatch chokepoint (src/tools/executor.jl), and a layered test suite running under Julia 1.10 CI.
  • The neural path and external verification pipeline are still maturing — the LLM↔executor loop has no automated integration test yet, and the explain/prove/demonstrate/verify pipeline depends on external systems that are not shipped.

Milestones (ROADMAP.adoc):

Milestone Theme Status
v0.1.0 Trustworthy core: ~40 modules, the executor.jl boundary, ground-truth validation, offline examples In progress
v0.2.0 Neural interface hardened: abstract the LLM backend behind an OpenAI-compatible interface, golden-transcript routing tests, publish to the Julia General registry Planned
v1.0.0 Verified assistant: adversarial verification pipeline; Agda proofs extended from discrete analogues toward their real-valued targets Planned

Explicit non-goals for now: re-implementing full Bayesian engines (Stan/PyMC scope) and a GUI (the interface is chat + offline API).

The production-readiness audit

On 2026-07-10 the project was put through a ten-dimension production-readiness audit. The result is the execution pack in .claude/tasks/prod-readiness/ — a set of standalone work orders, each runnable by a fresh session with a recommended model for implementation and for adversarial verification.

The work orders group into two waves:

  • Wave 1 — "basically running and functioning acceptably." The P0 guarantee plus the crash / robustness / coverage / install fixes. Do these first.
  • Wave 2 — everything beyond that. Release engineering, packaging, honesty reframes, deeper test coverage, observability, and polish.

The themes the audit examined are reflected directly in the work orders: the correctness guarantee, input robustness, test coverage of the router, a working install path, supply-chain pinning, the two experimental surfaces (FFI and proofs), release engineering, packaging (Guix, containers), documentation honesty, and observability / input-hardening.

Decisions of record (user-approved 2026-07-10 — not to be relitigated)

  1. Release = JuliaRegistrator + TagBot (Julia General registry), not artifact-only.
  2. FFI (Zig/C-ABI) and Agda proofs are EXPERIMENTAL — make them compile and CI-check (Wave 1), reframe the docs (Wave 2); do not invest in the Idris2 ABI or proofs over ℝ yet.
  3. guix.scm gets made real (buildable), not deleted — governance has a "Guix primary" policy check.
  4. The merge gate is active — the Base ruleset requires the "E2E — Julia Test Suite" status check; a PR cannot merge red.

Wave 1 — basic functioning

# Work order Deliverable observed in the tree
1 Neural boundary guardrail (P0) src/tools/guardrail.jl (validate_numeric_provenance), test/guardrail_test.jl
2 Degenerate-input guards + @assert→ArgumentError src/stats/validation.jl, test/degenerate_input_test.jl
3 Table-driven router tests + CI coverage test/executor_router_test.jl; e2e.yml runs Pkg.test(coverage=true) and reports %
4 Real install path (Justfile, quickstarts, smoke CI) Real just setup/build/run recipes, install-smoke.yml, .tool-versions
5 Pin the compute half + prune Dependabot + real threat model Committed Manifest.toml; neurosymbolic + FFI boundaries in docs/THREAT-MODEL.md
6 Make the Zig FFI compile + zig build test CI ffi/zig/ builds; .github/workflows/zig.yml (Zig 0.13.0)
7 Type-checking Agda proofs + agda --safe CI proofs/**/*.agda; .github/workflows/agda.yml
8 Hygiene / security-templates Merged as PR #33

Package 1 is the flagship. It turned the guarantee from prompt-only into an enforced-in-code control, and also hardened the runtime: bounded multi-round tool-call loops, HTTP timeouts, malformed-tool-call recovery, executor clamps on n_reps / n_permutations / component counts, and an else-error on every sub-type dispatch.

Status note. The execution-pack README records packages 1, 2, 3, 6 and 7 as open at the time of its last update (the fleet agents that were meant to push them died on a spend limit), with packages 4, 5 and 8 merged as PRs #34, #35 and #33. As observed in the current tree, the deliverables for every Wave-1 package are present (the files listed above all exist). Treat the live repository — not the pack's status table — as the source of truth.

Wave 2 — beyond basic functioning

# Work order Status observed in the tree
1 Release: JuliaRegistrator + TagBot + SBOM Landed — TagBot.yml, rewritten release.yml (source tarball + syft SBOM + SLSA), docs/RELEASING.adoc
2 Buildable guix.scm Present in the tree (guix.scm, .guix-channel)
3 Runnable Containerfile + devcontainer Landed — Containerfile, .devcontainer/, container-build.yml
4 Experimental reframe: FFI + proofs docs Landed — README "Experimental surfaces" section; proofs/README.adoc relabeled to the ℕ-scope catalogue (see Experimental-Surfaces)
5 Extend ground-truth reference validation Landed — test/reference_validation_advanced_test.jl (Kruskal-Wallis, multiple + logistic regression, Kaplan-Meier, fixed/random-effects meta-analysis; see Testing-and-Validation)
6 Structured logging + audit trail Remaining — no test/observability_test.jl in the tree; runtime logging is still ad-hoc and the VeriSimDB audit path (src/bridge/verisimdb_schema.jl) is dormant
7 Prompt-injection delimiting Remaining — no test/prompt_injection_test.jl; input-side delimiting is not yet in the prompt-construction path
8 Polish sweep Largely applied — README module count corrected, .tool-versions pinned, Project.toml [compat] filled (Statistics = "1", …)

The two notable remaining items

  • W2-6 · Observability. Introduce Logging-based structured logging with a per-chat-turn correlation id and per-tool-call metadata (tool name, argument hash, result shape, duration), keep REPL answers clean, and wire the dormant VeriSimDB audit path — with persistence pluggable/no-op-safe in CI. This is the audit-trail half of the "auditability" story and coordinates with the P0 guardrail work that restructured chat.jl / executor.jl.
  • W2-7 · Prompt-injection delimiting. Wrap untrusted user data (dataset values, column names, free text) in clearly delimited, labeled segments in the constructed prompt, with a system-prompt clause to treat everything inside as data, never instructions, and neutralize delimiter collisions. This is defense-in-depth on the input side — the numeric-provenance guardrail (W1-1) remains the primary control (see Security-and-Threat-Model).

The release path

Per decision of record #1, releases go through the Julia General registry:

  1. Bump version in Project.toml and merge to main.
  2. Comment @JuliaRegistrator register on the merge commit (or use the Registrator app).
  3. The General registry PR AutoMerges.
  4. JuliaTagBot triggers .github/workflows/TagBot.yml, which cuts the vX.Y.Z git tag and GitHub Release.
  5. release.yml (run via workflow_dispatch with the tag, or automatically under a deploy-key setup) builds a reproducible source tarball, an SPDX SBOM (syft), and SLSA provenance, and attaches them to the release.

The full maintainer walkthrough — including first-registration AutoMerge caveats ([compat] completeness, name/UUID rules) — is in docs/RELEASING.adoc and summarized on the Release-Process page. The pipeline is inert until a maintainer registers a version: no tags exist yet.

Open decisions / follow-ups

  • .gitlab-ci.yml and deny.toml were flagged during the polish sweep as scaffolding that nonetheless carries load-bearing policy claims — their disposition is a separate, deliberately deferred decision.
  • The SLSA job in the release pipeline depends on real artifact subjects; whether it survives for a source-tarball-only release is called out as a judgment call in the release work order.
  • LLM↔executor integration test and golden-transcript routing tests (v0.2.0) are still to come — the neural path is currently exercised only indirectly.

See also

Statistikles

Overview

Using

Development

Project

Clone this wiki locally