Skip to content

Testing and Validation

Jump to bottom
Jonathan D.A. Jewell edited this page Jul 11, 2026 · 1 revision

Testing and Validation

Statistikles is a Kautz Type-1 neurosymbolic assistant: every number is computed by the Julia symbolic core, never by the LLM. That guarantee is only worth anything if the Julia core is itself correct, so the test suite is unusually large and is organised into layers — from smoke contracts, through mathematical property invariants, up to ground-truth reference validation where expected values are derived by hand or by a second, independent implementation (R / SciPy / Python) and never copied from the library's own output.

This page maps the whole suite as it actually exists in test/, explains how the reference validation works, and shows how to run it.

How the suite is wired

The single entry point is test/runtests.jl. It opens one big @testset "Statistikles Full Test Suite" and then includes seven extended test files at the end:

# ── Extended test categories (CRG Grade C) ──────────────────────────────
include("e2e_test.jl")                          # full statistical pipelines
include("property_test.jl")                     # mathematical invariants over random inputs
include("reference_validation_test.jl")         # trusted symbolic layer vs ground truth
include("degenerate_input_test.jl")             # no NaN/Inf leaks; @assert -> ArgumentError
include("guardrail_test.jl")                    # the "no number from the LLM" boundary
include("executor_router_test.jl")              # every LLM-facing tool exercised
include("reference_validation_advanced_test.jl")# KW, multiple/logistic regression, KM, meta

CI runs the whole thing via Pkg.test:

julia --project=. -e 'using Pkg; Pkg.test(coverage=true)'

The job that produces the merge gate is E2E — Julia Test Suite in .github/workflows/e2e.yml.

The test files at a glance

File Focus Test groups (@testset)
runtests.jl (Full Suite) Every exported statistical function, TypeLL types, bridges, benchmarks, ecosystem integrations ~100
e2e_test.jl End-to-end pipelines; graceful degradation on empty / single-element / NaN input 10
property_test.jl Statistical identities validated over 100 random trials each 14
reference_validation_test.jl Core layer checked against hand-derived / independently-computed ground truth 14
degenerate_input_test.jl n = 1/2/3, constant, zero-variance boundaries return nothing, never NaN/Inf 35
guardrail_test.jl The neural-boundary guarantee: numeric provenance auditing + executor guards 8
executor_router_test.jl execute_tool argument coercion for every registered tool 8
reference_validation_advanced_test.jl Ground truth for KW, multiple/logistic regression, Kaplan-Meier, meta-analysis 7
chi_square_validation_test.jl Dedicated χ² review vs R 4.5.0 + SciPy 1.15.3 (standalone, see below) 26

The property tests each run an explicit for _ in 1:100 loop with a fixed seed (Random.seed!(2026)), so a single "group" is actually 100 independent assertions with reproducible failures. Group counts above are structural markers, not an assertion total.

What the Full Suite covers (runtests.jl)

The in-file suite exercises essentially the whole src/stats/ surface (41 modules) plus the type layer and the bridges. Representative sections:

  • Descriptive — descriptive_stats, power_mean, weighted_stats (harmonic/geometric/ trimmed/winsorized/quadratic means, MAD, CV, skewness, kurtosis, quartiles).
  • Inferential — t_test_independent, one_way_anova, levenes_test.
  • Nonparametric — mann_whitney_u, wilcoxon_signed_rank, kruskal_wallis, friedman_test, cochrans_q, permanova / permanova_multi, dunn_test, ks_2sample, midranks, tie_correction.
  • Correlation & regression — pearson_correlation, spearman_correlation, partial_correlation, simple_linear_regression.
  • Corrections — adjust_p_values (bonferroni / holm / fdr / sidak).
  • Specialised — Fisher exact, McNemar, Grubbs, ICC, Bland-Altman, Anderson-Darling, MANOVA, Kaplan-Meier, meta-analysis, Mahalanobis, information theory (Shannon entropy, KL divergence).
  • Non-classical — tropical algebra, Choquet integral, Bell/CHSH, modular / p-adic / GCD statistics, rough sets, Hurst exponent, degree centrality.
  • Type layer (TypeLL) — Probability, EffectSize, TropicalValue, ModularInt, and the AuditSession state machine (compute → verify → prove → persist → report → complete).
  • BetLang — ternary primitives, uncertainty number systems, Latin hypercube / Sobol sampling, simulated annealing / particle swarm, financial risk (VaR / CVaR / Dutch-book / risk-of-ruin).
  • Cross-cutting "Aspect" tests — NaN handling, determinism, symmetry (r(x,y) = r(y,x)), and mathematical invariants (QM ≥ AM ≥ GM ≥ HM, variance = std², IQR = Q3 − Q1, CLR sums to zero, tropical idempotence).
  • Benchmarks — timing gates (e.g. descriptive stats < 100 ms on 10 000 points; Mann-Whitney < 200 ms on 1 000×1 000), so a performance regression fails the suite.

Extended categories

E2E pipelines (e2e_test.jl)

Full workflows from raw array to structured report, plus the error contract: empty, single-element, and all-NaN inputs must return an error dict (haskey(result, "error")), never throw or emit garbage. A NaN-containing array must process only the non-NaN observations (result["n"] == count(!isnan, data)).

Property-based (property_test.jl)

Twelve invariants, each over 100 random datasets — for example:

  • mean of a constant array equals that constant; variance is 0;
  • min ≤ mean ≤ max; IQR = Q3 − Q1;
  • power-mean ordering M₋₁ ≤ M₀ ≤ M₁ ≤ M₂;
  • CV is scale-invariant; uniform-weighted stats equal unweighted stats;
  • Pearson r ∈ [−1, 1]; every p-value ∈ [0, 1].

Degenerate-input guards (degenerate_input_test.jl)

Two invariants at the numerical boundary:

  1. No NaN/Inf ever leaks into a returned Dict. Undefined fields come back as nothing (JSON null) plus an explanatory *_note, verified by a recursive assert_finite_and_serialisable walk over nested Dicts/Vectors.
  2. Validation that used to be a disable-able @assert now throws a real, catchable ArgumentError (a full src/ sweep — kl_divergence, centered_log_ratio, Probability, permanova, kaplan_meier, meta_analysis, and more).

Neural-boundary guardrail (guardrail_test.jl)

Directly tests the flagship guarantee. It exercises collect_numbers (recursive numeric harvest of tool results), extract_numeric_tokens (numeric literals in LLM prose), and validate_numeric_provenance, which flags any number in the prose that does not trace to a recorded tool result:

# INJECTED FABRICATION — 42.73 is legitimate; r = 0.87 and t = 3.14159 are fabricated.
fabricated = "The mean is 42.73 but the correlation was r = 0.87 and t = 3.14159."
ok, orphans = Statistikles.validate_numeric_provenance(fabricated, tool_results, Float64[])
@test !ok
@test "0.87" in orphans && "3.14159" in orphans
@test !("42.73" in orphans)   # the one real number is not flagged

It also covers process_tool_calls recovery from malformed tool calls (missing function, invalid-JSON arguments) and the executor's clamps (n_reps, n_permutations, component-count k ≤ 20) and unknown-sub-type dispatch guards. No HTTP or live LLM is needed — responses are plain Dicts. See Experimental-Surfaces for the parts of the stack that are not enforced this way.

Executor router coverage (executor_router_test.jl)

execute_tool(name, args) is the exact seam the LLM drives: a bare tool-name string plus a loosely-typed Dict. This file enumerates every tool from get_tools() (parsed, never hand-copied) and asserts a clean (non-"error") Dict comes back for each:

registered_tools = [t["function"]["name"] for t in Statistikles.get_tools()]
for name in registered_tools
    haskey(SKIP_LIST, name) && continue
    @test haskey(ARG_TABLE, name)
    result = Statistikles.execute_tool(name, ARG_TABLE[name])
    @test result isa Dict && !haskey(result, "error")
end

If a tool is added to get_tools() without a matching ARG_TABLE/SKIP_LIST entry, the "registry coverage" testset fails — coverage cannot silently rot. High-traffic tools are additionally cross-checked against their direct Julia call to prove the router's argument coercion doesn't change the numbers.

Reference validation — how ground truth is established

This is the layer that makes the no-mollocks guarantee meaningful. From the header of reference_validation_test.jl:

The no-mollocks guarantee only holds if the Julia layer itself is correct. Every expected value below is hand-derived or independently computed … NOT copied from this library's own output. If one of these fails, the symbolic layer is producing wrong-but-deterministic numbers — the exact failure mode the project exists to prevent.

Three independent sources of truth are used, chosen so the check never degenerates into the library grading its own homework:

  1. Closed-form / hand derivation. Inputs are engineered so the answer is exact. Example: a 2×3 table where every expected cell is exactly 4, giving χ² = 4.0, and df = 2 uses the closed-form survival function P(χ²₂ > x) = e^(−x/2) — independent of the library's own Distributions.cdf call path:

    observed = [6 2 4; 2 6 4]
r = Statistikles.chi_square_test(observed)
@test r["chi_squared"] == 4.0
@test r["df"] == 2
@test isapprox(r["p_value"], exp(-2.0); atol = 1e-12)
@test isapprox(r["cramers_v"], sqrt(1 / 6); atol = 1e-12)

    Other hand-derived checks: Welch t (t = −1.897366…, Welch-Satterthwaite df), Pearson r = 12/√148, one-way ANOVA (F = 3, closed-form p = 0.125), Mann-Whitney complete separation (U = 0), and exact regression coefficients.

  2. Second independent implementation, run locally. The advanced file derives its constants with Python 3.13.5 + NumPy / fractions.Fraction (exact rational arithmetic) for Kruskal-Wallis H = 32/7, an orthogonal-design multiple regression (diagonal XᵀX), a saturated-binary logistic MLE (β = [−ln 3, 2 ln 3]), product-limit Kaplan-Meier survival, and fixed/random-effects (DerSimonian-Laird) meta-analysis. Crucially: "Python is used only to derive the numeric literals below; none of it enters the repository (governance bans Python in repo code)."

  3. χ² vs R and SciPy — see the dedicated file below.

Because the constants are external, a green run means the symbolic layer agrees with an independent authority to ~9–12 significant figures; a red run means the core is wrong, not the LLM.

Dedicated χ² validation (chi_square_validation_test.jl)

A stand-alone correctness review of chi_square_test, chi_square_goodness_of_fit, and frequency_table, checked against R 4.5.0's chisq.test and SciPy 1.15.3's chi2_contingency / chisquare (both agreeing to ≥ 12 significant figures; only the constants, not the derivation scripts, are committed). The header documents seven bugs this review found and fixed, including:

  • upper-tail p-value used 1 - cdf(...), which underflows to exactly 0.0 for large χ² — switched to ccdf(...), accurate into the 1e-22 range;
  • missing guards for r < 2 / c < 2 / zero-observation tables (Chisq(0) DomainError);
  • a fully-zero row/column silently dropped while still charging its df — now returns nothing plus a "note", never a number;
  • a Yates continuity-correction term missing its max(0.0, |O−E| − 0.5)² clamp, which inflated χ² near independence (buggy ≈ 0.0256 vs ground-truth 0.0 on [[10,10],[10,11]]).

This file carries its own using Test / using Statistikles and is run standalone (it is not currently included by runtests.jl):

julia --project=. test/chi_square_validation_test.jl

The χ² ground-truth cases that do run inside the gated suite live in reference_validation_test.jl (the Chi-square test of independence / goodness-of-fit testsets).

Running the tests

Install dependencies first (the committed Manifest.toml pins everything):

julia --project=. -e 'using Pkg; Pkg.instantiate()'

Then run the suite in any of these equivalent ways:

just test                                     # via the Justfile recipe
julia --project=. test/runtests.jl            # direct
julia --project=. -e 'using Pkg; Pkg.test()'  # what CI runs (add coverage=true for lcov)

In WSL

Julia here lives in WSL, so invoke it through a login shell from the Windows side:

wsl.exe -d Debian -u hyperpolymath -- bash -lc \
  'cd /path/to/statistikles && julia --project=. -e "using Pkg; Pkg.test()"'

A -l (login) shell is needed so julia is on PATH.

Coverage and the merge gate

e2e.yml runs Pkg.test(coverage=true), processes the result to lcov.info, and posts a line-coverage summary to the job page. That summary is explicitly informational — not gated:

echo "## Coverage (informational — not gated)"
echo "Lines covered: **${HIT} / ${TOTAL}** (**${PCT}%**)"

What is enforced:

  • the E2E — Julia Test Suite job must pass for a PR to merge (branch protection);
  • a companion Aspect — Safety + SPDX job fails on dangerous patterns (believe_me, assert_total, sorry, Admitted), on any postulate under proofs/, and checks SPDX headers.

So the gate is "all tests green", with coverage tracked for visibility rather than as a hard threshold.

See also

  • Experimental-Surfaces — the honest status of the Zig FFI and Agda proofs (what is CI-checked vs placeholder / design-only / open target). The reference-validation layer above is what keeps the Julia core trustworthy; the experimental surfaces do not back it.
  • Home

Statistikles

Overview

Using

Development

Project

Clone this wiki locally