feat(cql): the CQL language conformance suite runs — 1622/1835 by our scheme, 1633 on the upstream rule (V7, ADR-060)

[Taleef7]

Roadmap M-C / V7, a named member of the FHIR-column verification set (locked decision 2). Closes #296, open since 2026-07-15.

cql-tests conformance — 1835 cases across 16 files      (11 seconds)

  pass                 1622
  fail                  155    engine computed the wrong value
  translation-error      12    our JS translator rejected valid CQL
  runtime-error           4
  invalid-refused         6
  invalid-accepted       36    accepted CQL the corpus says is INVALID
  skipped                 0    we claim every capability the corpus declares

Why this suite specifically

cql-execution 3.3.x — our exact runtime — already has published results at cql-tests-runner.quality.hl7.org: 1,533 / 81 / 113 / 4. That run used the Java translator. We translate with @cqframework/cql 4.0.0-beta.1, and that delta has never been published. This measures it.

The comparison is indicative, not like-for-like (their corpus was ~1,731; they skipped 113 we grade) and the PR says so everywhere it appears. What is directly comparable: our 4 runtime errors are the same four cases as their 4 errors.

The harness was the defect, twice, and nearly published

The first full run reported 183 translation errors. 171 of them were ours:

• 155 were No default UCUM service available. LibraryManager takes the UCUM service as its fourth argument and defaults to one that throws, so every expression containing 1.0'cm' failed to translate.
• 16 were our own grading line — define Passed: Actual ~ Expected will not type-check when a case's expected value has a different static type from its actual.

Real figure: 12. Publishing 183 as "the JS translator delta" would have been wrong by a factor of 15. It was caught only because the plan required clustering the diagnostics before believing the total — one command. Both defects are written up in the evidence rather than quietly fixed, because "the harness was wrong" is the outcome a conformance exercise most needs to be honest about.

Findings — all in the translator/engine, none in our measures

Slice unimplemented in the JS translator	10 of the 12 remaining translation errors
36 of 42 invalid cases translate anyway	Ceiling(2147483648), Exp(1000), Ln(0), minimum Boolean. Matters because the Studio's CQL editor uses translator diagnostics as its authoring compile gate
Long is silently wrong	1L + 2L → 12 — string concatenation, no throw, no warning. Sum({6L,2L,3L,4L,5L}) → 62345. Independently confirms the connectathon research
Decimal precision unapplied to aggregates	PopulationStdDev at full IEEE-754 width vs the corpus's declared 28,8
Ceiling does not null at the Integer boundary	returns an out-of-range value

Five files are perfect — logical, nullological, queries, aggregate, conditional. Those are the constructs our own measure CQL is built from, and no WorkWell measure uses Long, Slice, timezone from or ToConcept(Code).

Decisions (ADR-060)

1. translation-error is never folded into fail. Merging them would attribute a translator gap to cql-execution, whose own posted results say otherwise. The difference between those columns is the deliverable.
2. Cases are graded by CQL, not JavaScript — Actual ~ Expected, executed unfiltered. A TypeScript literal parser would re-implement CQL semantics in order to test CQL semantics, sharing defects with the thing under test. This needed one genuine package export: evaluateExpressions (data-free execution — the whole language suite is defined in that subset).
3. The SkipList is the capability set we CLAIM, and it is empty. CQL conformance harness: run cqframework/cql-tests against our translator + cql-execution #296 proposed skipping the known-weak clusters; that would have deleted the finding (system.long is 33 cases and holds the worst defect) and reported a better rate over a smaller denominator. The mechanism exists and is unit-tested against a fixture so it is not vacuous.
4. CI gates on a per-file baseline, not a threshold. "≥N passing" goes green while a translator upgrade trades 30 passes for 30 different ones.
5. ADR-048's node: CLI debt is reframed, not paid. Its stated basis had expired — live-cli.ts no longer imports devdb-cli.ts, and every remaining importer of the four CLI files is a test or a bin.ts shim. The real hazard was that engine-boundary.test.ts keyed its node: carve-out on the filename, so a request-path module merely named *-cli.ts would pass. Now keyed on reachability, derived rather than listed. Mutation-checked both directions.

Non-degeneracy

A conformance harness that parses nothing reports no failures and looks green — and here it would publish a number. The runner refuses to report at all unless it parsed all 16 files and all 1,835 cases, every file contributed, and every case landed in exactly one bucket summing to the total.

Production gap found, deliberately not fixed here — #397

The runtime translator has no UCUM service either, so the Studio's ELM Explorer cannot compile any CQL containing a quantity literal. compileCql now accepts an optional validateUnit and the harness passes one; production passes none, so its behaviour is byte-identical. Wiring it in is a real behaviour change and belongs in its own PR.

Verification

• pnpm test 1863 → 1876, 0 fail (9 harness tests + 4 for the new package export).
• pnpm typecheck clean.
• pnpm cql-tests --check — no regressions vs the committed baseline.
• Mutation-checked: node:fs in a request-path module (measure-registry.ts) turns engine-boundary.test.ts red; the same import in devdb-cli.ts stays legal.
• New CI job mirrors official-cases — pinned fetch, cached, held out of pnpm test so an offline local run stays green.

Evidence: docs/evidence/CQL_TESTS_2026-08-05.md. Conformance claims added to docs/STANDARDS_CONFORMANCE.md via the conformance skill — it is a measured pass rate over a public suite, not a conformance claim, and no certificate exists for it.

🤖 Generated with Claude Code

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: d99b0e6f5d

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Key the CQL baseline by individual test case

Because the baseline is reduced to per-file outcome counts here, pnpm cql-tests --check can miss the exact regression shape this gate is meant to catch: within a single XML file, one previously passing case can become fail while another failing case becomes pass, leaving pass/fail totals unchanged and producing no regression. This also lets translation errors move between cases when the global total is unchanged, so the committed evidence can drift while CI stays green; store and compare outcomes by file/group/name instead of only per-file tallies.

Useful? React with 👍 / 👎.

[Taleef7]

Confirmed and fixed. You are right, and the framing is what makes it sting: the gate explicitly rejected a bare >=N passing threshold because a translator upgrade can trade 30 passes for 30 different ones — and then reintroduced the identical hole one level down by tallying per file.

regressions() now compares by file/group/name. Only the non-passing cases are stored — 213 entries rather than 1,835 — and that loses nothing rather than being a size compromise: a case absent from the map was passing, so "used to pass, now does not" stays decidable for all 1,835.

Three things fell out of doing it per case that the per-file version could not express:

• a change between non-passing outcomes (fail → translation-error) is reported without failing — the evidence document enumerates those buckets, so silent drift between them would leave it stale;
• a baseline case that did not run at all is a regression, not an improvement;
• a pre-feat(cql): the CQL language conformance suite runs — 1622/1835 by our scheme, 1633 on the upstream rule (V7, ADR-060) #398 baseline is refused rather than silently compared with the weaker rule.

The test now asserts the exact hole — a within-file swap whose tallies are proven identical by an assertion must still be caught. Mutation-checked end to end against the real corpus too: removing one entry from baseline.json yields CqlAggregateFunctionsTest.xml/Product/ProductLong: pass → fail.

ADR-060 decision 4 rewritten to record the hole rather than describe the design it replaced. Suite 1876, 0 fail.

[Taleef7]

Self-review findings addressed (4 fixed, all in this branch)

A detailed review of this PR turned up one finding that would have shipped a wrong published claim, and three more worth having.

CRITICAL — the invalid bucket was not measuring what the corpus means

cql-tests-runner grades an invalid="true"|"semantic" case as a pass whenever the request fails for any reason — translation or evaluation:

if (invalid === 'true' || invalid === 'semantic') {
    result.testStatus = response.status === 200 ? 'fail' : 'pass';
}

We graded purely on whether the expression translated. Measured: 5 of the 36 "accepted" cases throw at runtime and are upstream passes. Worse, the finding's headline example — Ceiling(2147483648) — is one of only 2 invalid="syntax" cases, which upstream does not route through that branch at all.

Corrected: invalid cases are now executed when they translate, and the bucket splits into refused-at-translation / refused-at-runtime / accepted. 11 refused / 31 accepted, not 6 / 36. And because upstream folds refusals into pass, the upstream-comparable total is 1,633, now stated beside our 1,622 — the report was under-stating itself against the very figure it compared to.

IMPORTANT — a missing define was reported as an engine defect

Grading on defines["Passed"] === true means an absent define reads as fail, which the report labels "engine computed the wrong value". That is exactly the misattribution ADR-060 decision 1 exists to prevent, one layer down — and it would fire en masse while still producing a plausible-looking report. Now a named harness fault.

IMPORTANT — the harness was never typechecked

scripts/ was outside tsconfig.json's include, so the code producing a published number was only type-stripped by tsx. Adding it immediately caught two Baseline fixtures missing a required field.

IMPORTANT — "the fallback fires on zero cases" was false

The evidence doc claimed the JS-comparison fallback was dormant. That was read off results.json — where gradedInJs was never serialized. Both I and the reviewer confirmed 0 from a key that did not exist. It is 16. Now printed in the summary, serialized, recorded in the baseline, and any increase is a regression.

Also fixed

ucum.ts claimed the corpus uses 6 units	it uses 18+ — replaced the assertion with a test that walks the corpus, so the claim maintains itself
entity decoding	numeric refs were decoded first, so &#38;lt; double-decoded to <; now a single pass
single-quoted attributes	were silently dropped — an invalid='true' would have become a valid case
skipDecision	its test re-implemented the function and asserted against the copy; the map is now injectable so the test drives the real code
request-path floor	> 5 against an actual 21 → >= 15; and the re-key is now documented as the trade it is, not a pure tightening
baseline / corpus pin	must now agree, or the run refuses
fetch-cql-tests.ps1	checks its own exit code and verifies HEAD before writing .pin — a failed fetch on a warm checkout could label results with a commit they were not produced at

Every per-file row in the evidence table now reconciles (the invalid-refused column was missing). Suite 1876, 0 fail; --check green against the regenerated baseline.