Examples as Data

Examples as data

The reference examples are where enumeratio's mathematics is tested. Every example is a row of plain data in the package that declares its head, with a stable id. A test of what a head evaluates to asserts against those rows, not against a golden file of its own. Unit tests stay for plumbing: parsers, printers, the CLI, the elements.

A head's record

Each head has a folder, reference/<Head>/, in the package that declares it. Compute-engine's own heads live in packages/reference/entries/.

• index.md: front matter (name, domain, signatures with the library that contributes each, summary, references), then a markdown body that the page renders as written.
• examples.tsv: one hand-written row per example, in page order.
• examples.values.<system>.tsv: one file per oracle system (wolfram, sympy, sage, mpmath, julia, rust, oscar, mathlib4). Each holds the system's input, its answer and the verdict, written by the scans.

Read and write records through @enumeratio/entry/node (readHead, writeHead, updateHead, readEntries, writeEntries); never hand-parse them. Hand edits are fine, and packages/reference/scripts/format-records.ts tidies them. A TSV cell holds its text as written. A cell that can't sit in a row as written (a tab, a line break, a leading ", the empty string) holds a JSON string. Everything else reads the records through @enumeratio/reference/node (referenceData). To add a head, add its folder.

The example row

Column	Meaning
id	Stable, lowercase, unique within the head. The page anchors #example/<id>.
section, caption	Where the example sits on the page, and its one-line description.
role	demo (the default, shown on the page), test (checked but not shown: a case too many or too minor to render), aspirational (a result we claim but don't reach yet, tested the other way round), triage (waiting on an oracle disagreement; see below).
expr, expected	MathJSON. expected is our pinned evaluation.
known, source	A value from outside our own evaluation (DLMF, OEIS, Fungrim, FindStat, mpmath…), with where it comes from. tests/known.test.ts holds expected to it. A failing known check is never fixed by rewriting expected. knownConvention and knownGap mark a deliberate or a capability-shaped difference, and a stale marker fails.
group	Rows sharing a group show as one card with cycling cases.
volatile	Which parts may legitimately differ between runs.
<form>.in, <form>.out, epsil.back	Optional printed-form snapshots; see below.
<system>.kind, <system>.note	The hand classification of an oracle disagreement.

Only expected and anything listed under known carry test meaning. Everything derived (printed forms, oracle answers) is either generated or scanned.

Printed forms are sparse snapshots

Each example prints in several forms: Epsil (its InputForm), TeX, TraditionalForm, notatio markup, and FullForm. Our FullForm is our own Epsil tree form, with every head explicit and still valid Epsil, not Wolfram's. Each form has an in and an out (the evaluated result). The full table is built, not committed; the site uses the built copy.

A filled form cell in examples.tsv is a snapshot: an expected value checked against the generated one. Pin sparingly, for the forms and cases worth guarding. Printing an example's in must parse back to its expr. When InputForm can't round-trip, the row pins its epsil.back. A variable e stays the symbol e, never ExponentialE, and pinned in samples guard that.

After changing an example, a printer or a transpiler, run UPDATE_FORMS=1 node packages/frontend/scripts/collect-forms.ts.

Oracles

packages/reference/scripts/oracle-scan.ts <system> --ids … --accept runs examples through another system's kernel and records its answer in examples.values.<system>.tsv. Always pass --ids; never accept whole heads. Two kinds of disagreement turn up:

• Settled rows carry the classification in <system>.kind / <system>.note: convention, domain, undefined-form, unevaluated, precision, and so on.
• Triage rows (role: triage) are examples Wolfram disagrees with that haven't been explained yet. Each has a bucket: emit (our translation to the oracle is wrong), ours?, gap (we don't answer and the oracle does), compare, wolfram?. Triage passes either fix the cause and readmit the row, or classify it.

A disagreement is a symptom. Never drop a row on an oracle mismatch. Find out whether the translation, the comparison, our answer or a convention is responsible.

The nightly Plausible run resamples each example's numbers and compares the result across systems. A finding that no classified row covers files an issue. A classified row also covers samples of the same head, system and verdict whose numbers fall into the same sign and kind classes. Comparison runs in a worker that is capped per sample, and a sample that runs past its budget is declined rather than counted.

Where data comes from

• Hand-written: rows, captions, known and source, and classifications.
• Farmed: Wolfram documentation examples and MathematicalFunctionData identities. Rows are adopted as role: triage, scanned with --ids, and readmitted or classified.
• Scanned: the examples.values.<system>.tsv sidecars. These stay committed, because they are external answers.
• Built: the printed-form table, provenance and coverage joins, and the manifest. These are generated from our own code at build time and gitignored, so a merge never leaves a file to regenerate.

Tests

• tests/entries.test.ts evaluates every settled example and compares it with expected.
• tests/known.test.ts holds expected to known.
• The forms test checks pinned snapshots and round-trips.
• Census checks that every declared head has a record, and that a record's typed signature rows match what each package actually does to the head.
• Routes are cross-checked: where a fix can differ by route, the same inputs go through evaluate(), .N(), the parse route and the compiled code, and agree.
• Goldens are only for output that isn't a head's value: CLI demos, rendered forms, visuals. toMatchSnapshot is not used in packages/.