v0.3.0

[0.3.0] - 2026-08-21

Multi-card data-parallel fan-out is the new capability: tt-atom run --devices 0,1,... shards a
list of structures across cards, which is the virtual-screening path. The rest is correctness and
consolidation on top of v0.2.1 — two UMA force bugs found by an external contributor, an opt-in
edge-bucketing win for screening campaigns of mixed-size systems, and a gate that now covers the
edge-frame regression it was blind to.

Added

• Multi-card data-parallel fan-out for tt-atom run: pass several structure files with
  --devices 0,1,... and each card runs a full Calculator + relax/MD loop (or the
  single-point energy default) for its shard of structures — the high-throughput
  virtual-screening path. Per-structure results come back in input order, bit-exact vs the
  single-card path (scripts/_multicard_sim_parity.py); without --devices, multiple
  structures run one after another on one card. --out is a directory in batch mode and each
  written geometry carries its energy and forces. The new tt_atom.batch.MultiCardSim pool
  backs the CLI and is usable directly; scripts/multicard_sim_scaling.py measures the
  throughput scaling.
• Orb's batched path (evaluate_batch) now applies ZBL pair repulsion union-wide through one
  autograd pass, matching the per-system path at short contact.

Fixed

• UMA forces at exactly-symmetric geometries. The edge->+Y frame is now fairchem's smooth
  two-chart quaternion instead of a ZYZ-Euler frame. The Euler azimuth atan2(x, z) is singular
  on the ±Y axis, and at a symmetric geometry every edge sits on that singular set, so
  d(wigner)/dpos — and therefore the analytic force — came out wrong while the energy stayed
  correct. Molecules are additionally evaluated in a fixed generic orientation and the force
  rotated back (exact, the model is equivariant), which moves every edge off the frame's
  bf16-sensitive directions. The vendored Wigner-D coefficient table ships in the wheel.
  From external PR #1 by Jan Weinreich; tests/test_symmetry.py anchors it, host-only.
• UMA analytic forces on out-of-distribution compressed cells. The on-device radial-MLP
  LayerNorm backward now runs in fp32 by default. In bf16 it mis-directed the force on
  compressed heavy cells (2-atom compressed β-Sn: force PCC 0.729, MAE 230 meV/Å against the
  fp64 oracle) while the forward stayed correct. Forward is unchanged; the opt-in fused_ln_bw
  kernel path is bf16-only and keeps bf16, a documented tradeoff. Also from PR #1.
• tt-atom run a.xyz b.xyz --relax --devices 0 with two different-composition structures no
  longer crashes the worker. The multicard worker builds one UMA Calculator per reduced
  composition and used to call open_device once per Calculator, so a second composition opened
  the same card a second time in one process (TT_FATAL: No MetalContext instance for context_id N).
  The worker now opens its device once and reuses it across every Calculator it builds; the
  Calculator.close() it owns never closes a device it didn't open.
• tt-atom run (multicard) now exits non-zero when any structure fails. The worker has always
  caught per-structure errors and returned the other structures' results, but the CLI used to
  exit 0 regardless, so a failed structure silently dropped its output. It now reports which
  structures failed and exits non-zero while still writing the ones that succeeded.
• Built wheels now include both weight exporters, so automatic UMA and Orb cache misses work
  outside a source checkout.
• Fresh UMA and Orb cache misses can download their checkpoints again; explicit
  HF_HUB_OFFLINE=1 still enforces offline use. Concurrent exports now use separate sidecars.
• Release mode now blocks every missing fixture, baseline, required op, and model-family OOM row.
  --allow-gaps remains available for development diagnostics.
• Release and UX subprocesses always open logical device 0 after TT_VISIBLE_DEVICES selects the
  physical card.
• Custom-op validation now rejects invalid gate modes and shapes, and program-cache keys include
  every operand layout that affects compiled accessors.
• The silicon-melt example now checks the exact-cutoff neighbour graph every step and recaptures
  only when it changes. The earlier trajectory is withdrawn because its skin policy could reuse
  stale edges.
• Orb stress now includes ZBL pair repulsion, matching upstream conservative and direct
  checkpoints at short contact. Both traced MD examples also include total ZBL energy and forces.
• Explicit UMA checkpoint paths now participate in the bundle-cache identity, preventing weights
  exported from one checkpoint from being silently reused for another.

Performance

• Opt-in edge bucketing for screening campaigns (OrbCalculator(..., bucketing=True), also
  honoured by MultiCardSim). Padding each system's edge set to a fixed ladder stops a mixed-size
  screen paying a fresh kernel compile per system: 20 Si systems cold on p150a go 463.07 s ->
  334.41 s (1.385x), kernel cache files 22299 -> 19885, and 20 distinct edge shapes collapse to
  7 shared buckets. Off by default and the default path stays byte-identical. Energies are
  bit-exact vs the unpadded path across all 24 configs (6 sizes x 4 Orb checkpoints); forces are
  bit-exact or at the measured cross-instance device noise floor. Warm throughput is unchanged
  within +-1%.

Changed

• The release gate covers the edge-frame symmetry regression (tests/test_symmetry.py, host-only,
  0.02 s, also in --quick). Every real-weight golden is an off-axis geometry, so no other
  accuracy row would notice that frame regressing.
• A performance baseline seeded on a different ttnn build now reports GAP (not comparable,
  re-seed to gate it) instead of FAIL. A different measurement protocol (checkpoint / batch /
  system size) is still a FAIL. GAP remains release-blocking, so the gate is no weaker.
• The UMA edge frame is no longer switchable: the Euler path and the per-bundle Jd_l buffers it
  read are gone.
• Accuracy coverage now includes all periodic UMA tasks, both families' batch-vs-separate parity,
  Orb's bf8 fast mode, and short-contact ZBL stress.
• Performance baselines now cover UMA and record the ttnn version that produced them.
• p150a performance baselines are refreshed for the ttnn 0.68.0 wheel
  (docs/perf_baselines.json); a protocol-mismatch guard refuses stale comparisons.
• The clean-install gate builds the candidate wheel, installs it and its runtime dependencies in
  isolation, and verifies the exact pushed commit and packaged exporters.
• A source build's performance environment is keyed on the tt-metal tree ttnn was built from
  rather than on the ttnn version string, which an editable install freezes at install time. A
  tree still matching the pinned commit now gates against the pinned baseline instead of reporting
  GAP. Orb and UMA baselines live in different environments (stock ttnn and the source build
  respectively), and RELEASING.md now says to run each row where its own baseline was seeded.