Skip to content

The No Fabrication Guarantee

Jump to bottom
Jonathan D.A. Jewell edited this page Jul 11, 2026 · 1 revision

The No-Fabrication Guarantee

Statistikles makes one flagship promise:

No number is ever produced by the LLM.

Every statistic the assistant reports must have flowed out of a symbolic Julia tool call. The language model routes your natural-language question to a tool and explains the result — it never calculates, estimates, or "recalls" a value.

Until recently this promise lived only in the system prompt. It is now enforced in code by src/tools/guardrail.jl and driven at the chat boundary by enforce_numeric_boundary! in src/tools/chat.jl. This page is a deep dive on how that enforcement works, grounded in the actual source and its tests (test/guardrail_test.jl).

See also: Statistical-Functions-Reference for the symbolic tools whose outputs constitute "legitimate" provenance.

What is a "mollock"?

A mollock is the project's name for a plausible-sounding but fabricated answer — the failure mode LLMs are notorious for in statistical contexts. From the MOLLOCK WARNING box at the top of src/tools/chat.jl:

A "mollock" is a plausible-sounding but fabricated answer. LLMs are KNOWN to produce mollocks in statistical contexts:

  • Invented p-values that "look right"
  • Fabricated effect sizes
  • Misremembered formulas
  • Confident but wrong test recommendations

The guardrail's job is to catch mollocks: a numeric literal in the assistant's prose that matches no number produced by symbolic computation is an orphan — a likely mollock. Crucially:

The guardrail NEVER rewrites the model's text silently. It flags.

The enforcement pipeline

user question
     │
     ▼
call_lm_studio(messages, tools)          # LLM proposes tool call(s)
     │
     ▼
process_tool_calls(...)                   # Julia executes the symbolic tools
     │        └─ pr.tool_results  (the ONLY source of legitimate numbers)
     ▼
LLM writes prose answer
     │
     ▼
enforce_numeric_boundary!(content, …)     # ← the neural-boundary guardrail
     │        └─ validate_numeric_provenance(content, tool_results, user_numbers)
     ▼
clean → print          orphans → retry once → still orphans → append warning block

The guardrail sits at the very last step, between the model's finished reply and your screen. Numbers that the user supplied are extracted from the raw input (extract_numeric_values(input) in chat.jl) so that echoing your own data back is never mistaken for fabrication.

The three primitives in guardrail.jl

1. collect_numbers — harvest every real number a tool produced

function collect_numbers(x, acc::Vector{Float64}=Float64[])
    if x isa Bool
        # skip: a Bool is a Real in Julia, but it is a flag, not a statistic
    elseif x isa Real
        push!(acc, Float64(x))
    elseif x isa AbstractDict
        for (_, v) in x
            collect_numbers(v, acc)
        end
    elseif x isa AbstractVector || x isa Tuple || x isa AbstractSet
        for v in x
            collect_numbers(v, acc)
        end
    end
    # strings, symbols, `nothing`, `missing`, etc. carry no harvestable number
    return acc
end

It walks a tool result — arbitrarily nested Dicts, Vectors, Tuples and Sets — into a flat Vector{Float64} of every legitimate candidate number.

Two deliberate exclusions:

Excluded Why
Bool In Julia true isa Real, but a Boolean is an event flag (e.g. "significant" => false), not a statistic.
Strings / symbols Interpretation labels like "large" carry no harvestable number.

The test fixture pins this behaviour exactly — a nested dict with a mean, a count vector, a nested p-value, a Bool flag, a String label and a tuple yields 7 numbers (flag and label ignored):

# mean + 3 counts + p_value + 2 tuple entries = 7 (bool/string ignored)
@test length(nums) == 7
@test isempty(Statistikles.collect_numbers(Dict("flag" => true, "name" => "x")))

2. extract_numeric_tokens — pull numeric literals out of prose

const _NUMERIC_TOKEN_RE = r"\d+(?:\.\d+)?(?:[eE][-+]?\d+)?%?"

function extract_numeric_tokens(text::AbstractString)
    return String[String(m.match) for m in eachmatch(_NUMERIC_TOKEN_RE, text)]
end

It matches integers, decimals, scientific notation and trailing-percent forms, returning the raw matched substrings in order. The regex intentionally does not capture a leading sign, so a range dash such as "3-5" is not swallowed as a spurious negative; sign is handled later by absolute-value comparison.

toks = extract_numeric_tokens("The mean is 42.73, p = 0.05, effect 35% and 1.5e-3, df = 12")
# → "42.73", "0.05", "35%", "1.5e-3", "12"

A companion _parse_token turns each token into (value, decimal_places, is_percent): it strips a trailing %, tolerates thousands separators (","), and records how many digits followed the decimal point so that display-rounding can be reproduced during matching. extract_numeric_values is the value-only variant used to capture the numbers the user typed.

3. validate_numeric_provenance — the audit

function validate_numeric_provenance(text::AbstractString, tool_results,
                                     user_numbers=Float64[]; rtol::Float64=1e-6)
    harvested = collect_numbers(tool_results)
    users = collect_numbers(user_numbers)
    candidates = vcat(harvested, users)

    orphans = String[]
    for tok in extract_numeric_tokens(text)
        t, d, _ = _parse_token(tok)
        isnan(t) && continue                       # unparseable: skip
        if d == 0 && 0.0 <= t <= 12.0              # small structural integer
            continue
        end
        _token_explained(t, d, candidates, rtol) || push!(orphans, tok)
    end
    return (isempty(orphans), orphans)
end

It returns (ok::Bool, orphans::Vector{String}), where ok == isempty(orphans) and orphans lists the literals — as they appeared in the prose — that matched nothing.

The legitimacy rules

A numeric token in the reply is legitimate if any one of these holds. All approximate comparisons use rtol = 1e-6 (with a tiny atol = 1e-9).

Rule Condition Rationale
Tool-result match token ≈ candidate, or token ≈ round(candidate, digits=d) where d is the token's displayed decimal places The number was computed; display-rounding is allowed (42.73 for stored 42.7333…).
Sign flip token ≈ -candidate "-1.2" matches a stored 1.2.
Percent variant token ≈ candidate × 100 or token ≈ candidate ÷ 100 (with sign flips) A stored proportion 0.35 may be reported as 35%.
User number token ≈ any number the user typed (user_numbers) Echoing your own input is not fabrication.
Small structural integer bare integer with 0 ≤ token ≤ 12 (no decimals) Degrees of freedom, group counts, list indices and the like.

The percent / sign / round logic all lives in _token_explained:

function _token_explained(t::Float64, d::Int, candidates, rtol::Float64)
    isfinite(t) || return true          # NaN/Inf token: do not flag
    for h in candidates
        isfinite(h) || continue
        for hv in (h, -h, h * 100, -h * 100, h / 100, -h / 100)
            if isapprox(t, hv; rtol=rtol, atol=1e-9)
                return true
            end
            if d >= 0 && isapprox(t, round(hv; digits=d); rtol=rtol, atol=1e-9)
                return true
            end
        end
    end
    return false
end

Why a "small structural integer" escape hatch?

Prose is full of tiny integers that are structural, not statistical — "with 3 groups", "at the 1st quartile", "2 tails". Requiring every such integer to trace to a tool result would drown real fabrications in false alarms, so bare integers in 0..12 are exempt. Note this exemption is deliberately narrow: it only applies to integers (d == 0), so a decimal like 3.14159 is never waved through by this rule.

Retry-then-flag: the enforcement flow

The audit is run by enforce_numeric_boundary! in src/tools/chat.jl. Its contract is explicit that the model's text is never silently rewritten:

"""
Run the numeric-provenance guardrail on an assistant reply. If unverified
numbers remain, do ONE retry asking the model to restate using only numbers
from the tool results (or, when no tool ran this turn, to compute them or
refrain). If orphans still persist, return the reply with a clear warning block
appended. The model's text is NEVER silently rewritten — orphans are flagged,
never fabricated.
"""

The flow is:

  1. Validate. validate_numeric_provenance(content, tool_results, user_numbers). If ok, return the content unchanged.

  2. One retry. Push a corrective role: "user" message and call the model again. The message differs depending on whether any tool ran this turn:

    • A tool ran: "Your previous reply contained numeric value(s) that do not appear in any tool result: … Restate your answer using ONLY numbers returned by the tool calls above … if a needed number is missing, call the appropriate tool."
    • No tool ran: "…NO symbolic computation was performed this turn, so none of them is verified. Call the appropriate statistical tool to compute them, or restate your answer without asserting specific numbers."

    Any tool calls made during the retry are executed and their results appended to tool_results, so freshly-computed numbers now count as legitimate provenance.

  3. Re-validate. If the retried reply is clean, return it.

  4. Flag. If orphans still persist, append a visible warning block — the reply is shown, but the unverified numbers are called out. Nothing is deleted or overwritten.

The warning block (_guardrail_warning_block) renders like this:

  !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
  UNVERIFIED NUMBERS — POSSIBLE MOLLOCK
  The following number(s) in the reply above did not come from any
  symbolic tool result and could not be verified:
      - 0.87
      - 3.14159
  Trust only numbers produced by [symbolic] tool calls.
  !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

When no tool ran at all this turn, an extra line is inserted — (no symbolic computation was performed this turn) — so the reader knows the whole reply is unverified.

Worked example: a clean reply vs. a flagged fabrication

Both examples below use the same tool result, taken verbatim from test/guardrail_test.jl:

tool_results = Any[Dict{String,Any}(
    "mean"       => 42.7333333,
    "std"        => 5.5,
    "p_value"    => 0.0512,
    "proportion" => 0.35,
)]

Clean — every number traces back

clean = "The mean is 42.73 with SD 5.5. The p-value is 0.05, so about " *
        "35% of cases. With 3 groups the result holds."
ok, orphans = validate_numeric_provenance(clean, tool_results, Float64[])
@test ok
@test isempty(orphans)

Why each token passes:

Token Legitimacy
42.73 round(42.7333333, digits=2) == 42.73 → tool-result match (display-rounded)
5.5 exact match to std
0.05 round(0.0512, digits=2) == 0.05 → tool-result match (display-rounded)
35% 0.35 × 100 == 35 → percent variant of proportion
3 bare integer in 0..12 → small structural integer

Fabricated — two mollocks caught

# 42.73 is legitimate; r = 0.87 and t = 3.14159 are fabricated mollocks.
fabricated = "The mean is 42.73 but the correlation was r = 0.87 and t = 3.14159."
ok, orphans = validate_numeric_provenance(fabricated, tool_results, Float64[])
@test !ok
@test "0.87" in orphans
@test "3.14159" in orphans
@test !("42.73" in orphans)   # the one real number is not flagged

0.87 and 3.14159 match no candidate (nor any percent/sign/rounded variant), so both are returned as orphans and flagged. The one genuine number, 42.73, is left untouched — the guardrail flags fabrications without disturbing verified values.

What the guardrail test suite also covers

test/guardrail_test.jl (the Neural-Boundary Guardrail testset) goes beyond the three primitives to exercise the surrounding recovery machinery:

  • Malformed-tool-call recovery in process_tool_calls — a tool_call missing its "function" key, or carrying "arguments" that are not valid JSON, produces a clean role: "tool" error message the model can recover from, never a crash. A well-formed call is confirmed to run the real symbolic function (descriptive_statistics returns "mean" => 5.0).
  • Executor clamps — caller-supplied sizes are bounded before any heavy work: n_reps (bootstrap) and n_permutations (PERMANOVA) are capped at 100000, and component counts k at 20 for bayesian_em, ica and topic_modeling.
  • Unknown sub-type guards — dispatching a bogus type to any multi-method tool returns an "Unknown type" error rather than silently doing the wrong thing.

These matter to the guarantee because a fabricated number is not the only way to mislead: a crash, an unbounded computation, or a silently-wrong dispatch would all undermine "every number is auditable symbolic output."

Honest scope

The guarantee enforced here is precise and narrow, and the project is careful not to overstate it:

  • What is guaranteed: every numeric literal in the assistant's prose is checked against the numbers that symbolic Julia code actually produced this session; any literal that matches nothing is flagged, never silently accepted or rewritten.
  • What is not claimed: the guardrail does not prove the underlying statistics are correct — that is the job of the reference-validation test suite (hand-derived and second-implementation ground truth; see Statistical-Functions-Reference). It also does not formally verify the ℝ-level statistical theorems; the Agda proofs cover only small ℕ-level lemmas, and the Zig FFI / Idris2 ABI surfaces are experimental. The boundary this module enforces is provenance — that reported numbers came from the symbolic layer, not the LLM.

Source map

File Role
src/tools/guardrail.jl collect_numbers, extract_numeric_tokens, extract_numeric_values, _parse_token, _token_explained, validate_numeric_provenance
src/tools/chat.jl enforce_numeric_boundary!, _guardrail_warning_block, the SYSTEM_PROMPT and MOLLOCK WARNING
src/tools/executor.jl execute_tool — the neural→symbolic dispatch that produces the only legitimate numbers
test/guardrail_test.jl The Neural-Boundary Guardrail testset

Related pages

Statistikles

Overview

Using

Development

Project

Clone this wiki locally