Security and Threat Model

Security and Threat Model

This page summarizes how to report a vulnerability, and the project's STRIDE threat model — with particular attention to the two application-specific boundaries that make Statistikles unusual: the neural → symbolic handoff and the FFI / C-ABI surface. It also explains the code-level control that enforces the product's flagship guarantee.

Grounded in SECURITY.md and docs/THREAT-MODEL.md (STRIDE, v1.1, last reviewed 2026-07-10). For the enforcement mechanism in context see the FAQ; for how the threat model was hardened during the production-readiness program see Roadmap-and-Production-Readiness.

---

Reporting a vulnerability

Do not report security issues through public GitHub issues, pull requests, discussions, or social media.

Channel	How
Preferred	GitHub Security Advisory — private, encrypted, gives coordinated-disclosure tooling and automatic credit
Alternative	Email j.d.a.jewell@open.ac.uk

A good report includes a description, the impact, affected versions/commits, and reproduction steps. SECURITY.md contains a full report template and a CVSS 3.1 / CWE prompt.

Response timeline

Stage	Target
Initial response	48 hours
Triage (severity + confirmation)	7 days
Status updates	Every 7 days
Resolution	90 days (complex issues may take longer)
Disclosure	Coordinated, ~90 days from report

Disclosure is coordinated: you report privately, the maintainer fixes and prepares a release, and disclosure timing is agreed with you. A safe-harbour clause covers good-faith research (no legal action, no law-enforcement referral, treated as authorised under the CFAA / UK Computer Misuse Act). Recognition is offered (Hall of Fame, advisory credit) but there are no monetary bug bounties — this is a community project.

Supported versions

Version	Supported
main branch	Yes (latest development)
Latest release	Yes (current stable)
Previous minor release	Yes (security backports)
Older versions	No — please upgrade

---

Threat model at a glance

The threat model uses STRIDE and covers the source code, build pipeline, CI/CD workflows (GitHub Actions), container images, secrets, dependencies, and release artifacts. Out of scope: platform-level GitHub/GitLab vulnerabilities, physical infrastructure, end-user device security, and social engineering of maintainers.

Trust boundaries

Alongside the usual supply-chain and CI boundaries, two are specific to this product:

Boundary	From (lower trust)	To (higher trust)
Pull request submission	External contributor	Repository codebase
CI/CD workflow execution	Workflow definition	Runner with secrets
Dependency resolution	Package registry	Build environment
Neural → symbolic handoff	LLM (chat interface)	Julia tool executor
FFI / C-ABI calls	External C-ABI caller	Zig FFI library (ffi/zig)

---

Application-specific boundaries

Statistikles is a neurosymbolic system: an LLM (the neural half) interprets a natural-language question and dispatches to verified Julia functions (the symbolic half) via src/tools/executor.jl. The product's core guarantee — every number comes from auditable symbolic code, never from the LLM — makes these two boundaries the primary application attack surface.

Neural → symbolic boundary

Threat	Risk	Mitigation
LLM numeric fabrication ("mollock": invented p-values, effect sizes)	High	System prompt forbids LLM arithmetic (src/tools/chat.jl); the numeric-provenance guardrail (src/tools/guardrail.jl) checks that every numeric literal in a reply originates from a symbolic tool result
Prompt injection via user data (instructions embedded in CSV values, column names, or the question)	High	The LLM has no side-effecting tools — all tool calls dispatch to pure Julia statistics; data is parsed as data (CSV / JSON3), never executed. Input-side delimiting is defense-in-depth (production-readiness W2-7)
Tool mis-routing (LLM picks the wrong test → plausible but wrong statistics)	Medium	Typed tool schemas (src/tools/definitions.jl); assumption checks in the stats layer; router/executor test coverage (test/executor_router_test.jl)
Silent-null sub-types (degenerate input makes a function return NaN/missing/nothing; the LLM papers over the gap)	High	Degenerate-input validation returns explicit errors instead of silent nulls (test/degenerate_input_test.jl); the executor surfaces errors verbatim

FFI / C-ABI boundary (ffi/zig)

The FFI is experimental — it compiles and is CI-tested (zig build test, Zig 0.13.0) but its entry points are placeholders not yet backed by the Julia core (see Experimental-Surfaces). Even so, the boundary is modelled:

Threat	Risk	Mitigation
Unvalidated inputs crossing the ABI (null handles, bad lengths, non-terminated strings)	High	Every export checks handle/pointer validity and returns typed Result codes (null_pointer, invalid_param) instead of dereferencing
Memory-safety violations (use-after-free, double-free across statistikles_free / statistikles_free_string)	Medium	Zig allocator discipline; an opaque Handle with a liveness flag makes a double-free a safe error, not UB; FFI integration tests (ffi/zig/test)
ABI layout drift between the Zig implementation and the declared C ABI	Low	Layouts declared in one place; FFI build + integration tests in CI

---

The guarantee, enforced in code

Until the production-readiness work, "no number is ever produced by the LLM" lived only in the system prompt. It is now enforced by src/tools/guardrail.jl (validate_numeric_provenance), tested in test/guardrail_test.jl.

validate_numeric_provenance(text, tool_results, user_numbers=Float64[];
                            rtol=1e-6) -> (ok::Bool, orphans::Vector{String})

How it works:

1. Harvest every numeric value produced by symbolic tool calls this turn — recursively, through nested Dicts / Vectors / Tuples / Sets (collect_numbers). Booleans and strings are deliberately ignored (a flag or a label is not a statistic).
2. Extract every numeric literal from the assistant's prose — integers, decimals, scientific notation, and percentages (extract_numeric_tokens).
3. Match each prose number against the harvested candidates. A token is legitimate when it approximately matches (within rtol, or after display-rounding) a tool-result number, or its ÷100 / ×100 percent variant, or a number the user themselves supplied, or it is a small structural integer in 0..12 (degrees of freedom, group counts, list indices).
4. Anything that matches nothing is an orphan — a likely mollock.

Crucially, the guardrail never rewrites the model's text silently — it flags. The runtime policy is retry-once-then-warn: if orphans exist after a tool-backed turn, the model is asked once to restate using only tool-result numbers; if orphans persist, the reply is shown with a clear warning block naming the unverified numbers. For the full design rationale see The-No-Fabrication-Guarantee; for how it is tested see Testing-and-Validation.

---

Mitigations in place

From docs/THREAT-MODEL.md:

• Dependency pinning — all GitHub Actions SHA-pinned; the Julia Manifest.toml is committed. Julia has no Dependabot ecosystem, so the pinned manifest resolved by CI (Pkg.instantiate / Pkg.test) is the compensating control.
• Secret scanning — TruffleHog + secret-scanner.yml on every push.
• Static analysis — CodeQL (codeql.yml).
• Supply chain — OpenSSF Scorecard (scorecard.yml).
• SLSA provenance + SBOM — generated and published with releases.
• Workflow hardening — permissions: read-all defaults with per-job scoping; PR fields are never interpolated into run: steps; CODEOWNERS on .github/.

Residual risks (accepted)

Risk	Accepted because
Zero-day in a GitHub Actions runner	Platform responsibility; no feasible mitigation
Maintainer account compromise	Mitigated by 2FA; residual remains
Transitive dependency 0-day	Lockfiles limit blast radius; scanning catches known CVEs
SBOM exposes internal component names	Transparency is a design goal

The threat model is reviewed quarterly, on architecture changes, before major releases, and after any security incident.

---

See also

• FAQ — "does the LLM ever compute numbers?", "can it be tricked via data?"
• Development-Guide — signing commits, the CI security surface, running checks locally
• Roadmap-and-Production-Readiness — how the neurosymbolic + FFI boundaries were added to the model
• Home