assay

Assay runs a corpus of formula tests across eight spreadsheet engines (Google Sheets, Excel, IronCalc, HyperFormula, LibreOffice Calc, the Python formulas library, pycel, and Lattice) and records where they diverge. The results are published as a browsable catalogue: https://assay.sheets.wiki.

Each documented divergence has a structured cause (missing function, precision drift, format-rendering, error-code mismatch, etc.), an engine list, and a sticky baseline so future engine releases that change behavior surface as drift rather than silent breakage.

Disclosure: the vast majority of existing tests were generated using Claude. Some behavior has been manually verified, especially divergences, but gaps may exist.

Part of cartularium.

Install

git clone https://github.com/cartularium/assay
cd assay
npm install
npm run build

For repeated use (development on related projects, scripting), link the CLI onto your PATH:

npm link

This makes assay callable from anywhere. npm unlink -g assay reverses it. Without linking, use node build/cli.js <command> or npx assay <command>.

Generating fresh fixtures requires per-engine setup (Google OAuth, xlwings for Excel, native bindings for the rest). Run assay setup and assay login once.

Quick start

Browse the corpus and verify it's healthy:

assay coverage
assay lint
assay run
assay matrix

Build the catalogue site locally:

assay catalogue
python3 -m http.server -d build/site

Output lands in build/site/ by default. Pass --build <dir> for a different location.

Layout

• tests/ — YAML test suites
• fixtures/ — cached per-engine results, regenerated by assay generate
• divergences/ — DV-####.yaml catalogue files, seeded by assay matrix --seed-catalogue
• capabilities/ — per-engine feature support and adapter declarations
• python/ — Python worker scripts for engines accessed via subprocess
• src/ — TypeScript runner, JS drivers, matrix, site builder
• docs/writing-tests.md — how to author tests

Commands

Command	What it does
assay run <files>	Compare test expectations against committed fixtures
assay generate <files>	Evaluate formulas live, save results as fixtures
assay validate <files>	Re-evaluate live and report drift
assay benchmark <files>	Score engines against a consensus of agreeing engines
assay coverage	Report function-universe coverage
assay lint <files>	Static checks on test YAML
assay check --mode=resolutions	Compare per-override recorded: baselines against current fixtures
assay matrix <files>	Per-engine cause profile, pairwise agreement, divergence clusters
assay catalogue --build <dir>	Render the catalogue to a static site
assay scaffold <FUNC>...	Emit starter YAML tests for one or more functions
assay login / assay setup	One-time engine setup (Google OAuth, xlwings)

Test schema

Tests are YAML. The simple case:

- subject: SUM
  formula: =SUM(A1:A3)
  category: value
  grid: { A1: 1, A2: 2, A3: 3 }
  expect: 6

When engines disagree, per-engine overrides record the deviation with a structured cause:

- subject: MOD
  formula: =MOD(-5, 2)
  category: value
  expect: 1
  overrides:
    hyperformula:
      cause: arg-semantics
      recorded: -1
      note: hyperformula uses sign-of-dividend; gsheets/excel use sign-of-divisor

Full guide: docs/writing-tests.md.

Contributing

The corpus is open for new tests, new findings, and new engine drivers. Open an issue or PR on cartularium/assay.

License

MIT. See LICENSE.