Version 0.64.0

This release introduces independently verifiable LLM benchmark results, richer build and timing provenance, and JSON schema 2 across all eight CPU and Metal profiles. It also strengthens result validation and reduces repeated collection and serialization work.

Highlights

• Independent result verification: The new script-examples/verify_llm_result.py checks geometry, work, checksums, timing, rates, accepted samples, and statistics across all CPU/Metal decode/prefill and contiguous/paged profiles.
• Build and timing provenance: LLM results retain build metadata and the executable SHA-256. CPU measurements and excluded attempts include original Mach timing evidence.
• Stronger correctness checks: CPU contiguous decode now validates every final K/V append byte after timing, preventing corrupted results from entering accepted aggregates even when the timed checksum matches.
• Bounded progress checkpoints: LLM file output writes progress snapshots at loop-count-based intervals, reducing repeated statistics and JSON work.

Breaking Changes

LLM JSON schema 2

All eight LLM profiles now use JSON schema 2 and methodology identifiers in the form:

llm-memory-v2-<backend>-<phase>-<layout>

Canonical scenario plans and expected checksums are stored once. Measurements and calibration attempts reference those plans, and aggregates identify their accepted measurements.

Consumers must update to the new structure. The API documentation includes a schema-1-to-schema-2 field map; compatibility aliases and a fallback reader are not provided.

Build prerequisite

Python 3 is now required for build provenance generation.

Added

• Independent LLM artifact verifier with optional:
  • --binary to check the executable file's SHA-256.
  • --require-raw-timing to require original CPU timing evidence.
• Verifier verdicts that distinguish accepted, internally consistent results from inconsistent or unaccepted results, unsupported evidence, and resource limits.
• Build manifests containing available Git revision and dirty state, compiler, build flags, architecture, SDK, deployment target, and executable SHA-256. Missing evidence remains explicitly unavailable or partial.
• Original CPU Mach start, stop, and delta ticks with the timebase for elapsed-time reconstruction.
• Independent arithmetic goldens, checksum corruption and work-mutation tests, real-GPU reduction-boundary tests, and expanded prefill boundary coverage.
• make test-llm-verifier, included in make test-all.

Changed

• Validation evidence: Named checks retain applicability, completion, validity, and reasons separately. A matching checksum cannot override a failed final-state check, and unevaluated checks remain unresolved.
• File checkpoints: Progress snapshots occur every K = max(1, ceil(count / 8)) completed loops, normally producing at most eight progress snapshots plus one terminal snapshot. Abrupt termination can lose up to 3K completed attempts since the last successful snapshot.
• Output handling: Exact --output - prepares only the terminal document. Disabled output skips JSON construction. Writer counters reflect actual prior persistence attempts; checkpoint failures are terminal and are not retried.
• Collection efficiency: Calibration retains compact actual checksum evidence, while canonical plans own expected values. Exact statistics are prepared at snapshot boundaries from accepted measurement IDs.
• Memory admission: Accounting includes retained evidence, canonical plans, statistics scratch space, and simultaneous JSON construction and serialization peaks.
• CPU task preparation: Duplicate validation within expected-checksum calculation has been removed. Borrowed plans and materialized resources are still validated before each executor call starts workers or timing.
• Acceptance semantics: results_complete describes the measured population. run_accepted additionally requires valid execution evidence and no known command or checkpoint failure. Sample count, position balance, CV, duration, and environment remain separate comparison criteria.
• Documentation: Updated CLI help, API, manual, whitepaper, and supporting references for schema 2, verification, checkpoints, checksum limitations, and comparison requirements. Clarified synthetic query heads, prefill tiles, and theoretical attention quantities.
• Test infrastructure: Replaced brittle source-text assertions with focused semantic coverage, removed redundant contract tests, and classified hardware-dependent LLM cases as integration tests. Objective-C++ test sources now use test flags and ARC.

Fixed

• CPU contiguous decode corruption detection: KV-bearing scenarios verify every final K/V append byte after timing stops and workers join. Corruption produces an invalid measurement with retained diagnostics and excludes it from accepted aggregates.
• Theoretical prefill overflow: Overflowing model-context attention-pair or FMA quantities now become nullable values with an explicit arithmetic-overflow reason instead of rejecting otherwise valid memory work. Exact byte, work, allocation, and execution guardrails remain enforced.