v0.4.0

AtmosTransport v0.4.0

Documentation: v0.4 manual · latest stable manual

Diff since v0.3.0

Breaking changes and migration

The 0.3 → 0.4 update changes numerical results through conservative cubed-sphere seam transfers and the mass-space TM5 Dkg solve. Recheck scientific and conservation diagnostics when upgrading; pin 0.3.0 to reproduce old runs. Multi-file runs must consume complete per-file window ranges: omit stop_window for consecutive daily binaries. Nine obsolete example configurations have moved to config/runs/likely_legacy/.

This release changes cubed-sphere numerical results and runtime/output
behavior. Keep the version, forcing binary, run TOML and dependency manifest
with reproducibility records; recheck conservation and scientific diagnostics
when upgrading from 0.3.0.

Numerical changes

• Cubed-sphere split advection and Lin–Rood now share conservative face
  transfers across panel seams. Previous runs could accumulate tracer mass
  drift at those seams.
• Precomputed TM5 Dkg diffusion advances conservative tracer storage with
  bidiagonal factors. It preserves weak exchange into empty layers, isolated
  layers and signed totals more accurately. Its adjoint uses the matching
  transpose. This is a numerical correction, so old and new output need not
  match bit for bit.
• CUDA PPM launch tiles, separate Dkg factor/tracer launches and batched
  convection solves reduce work without lowering configured precision.
  CUDA collaborative Float64 convection supports full columns through 73
  levels within its shared-memory budget; Float32 retains its portable depth
  limit. Explicit lmax_conv/n_merge choices remain numerical approximations.
• Lin–Rood recording and reverse halo propagation use backend kernels and
  pass transporting CUDA adjoint checks with scalar indexing disabled.

Runtime and output

• Input drivers and device workspaces are reused across daily files. Rolling
  staging owns its cache and checks retained source identity; failed prefetch
  and output tasks are consumed during cleanup.
• Single-file NetCDF output streams selected snapshots and records
  completed_snapshots. A failed run may leave partial output; file existence
  is not evidence of completion. Reopening a stream to resume is unsupported.
• Selected NetCDF output avoids retaining full tracer volumes on the host.
  Signed Float64 <tracer>_total_mass diagnostics are captured independently
  of spatial output precision. Metal uses bounded host slabs for Float64
  totals and column accumulation; CUDA retains device Float64 reductions.
• SnapshotFrame and SnapshotWriteOptions are part of the curated top-level
  API. Write and resource-close failures are preserved together rather than
  allowing one to mask the other.
• Runtime preflight reports malformed tables, including [input.staging]
  and tracer subtables, before opening binaries. Window indices must be
  integers. Multiple input files require their complete window ranges;
  omit stop_window for a full multi-day run. Nine obsolete 48-hour configs
  now live under config/runs/likely_legacy/.

Dependencies and documentation

• Add explicit FileWatching and HDF5_jll dependencies for staging ownership
  and NetCDF error-handler handling. HDF5_jll compatibility accepts 1.14 and 2.
• Remove self-version bounds from test and benchmark environments so release
  bumps do not invalidate them. Julia 1.10 remains the minimum supported version.
• Add an executed emission-footprint tutorial with a finite-difference check
  and a beginner's guide to adjoints, observations, priors and inversion.
  Correct preprocessing configuration/balance explanations and distinguish
  hosted CPU CI from separate GPU verification.

Verification and remaining limits

The branch records Julia 1.10/1.12 CPU test runs, strict documentation builds,
all ten maintained L40S GPU diagnostics, V100 adjoint/output checks, and
real-input Float32 C90/L66 comparisons. The matched L40S day benchmark improves
from 7.35 to 4.54 seconds for six tracers and from 47.47 to 13.84 seconds for
32 tracers. The 32-tracer 0.3.0 baseline uses its supported legacy solver,
because that release's collaborative solver is capped at six tracers. These
measurements include runtime setup and I/O and apply to the documented workload;
see benchmark evidence.

The C90/L66 Float32 forward smoke test passed on an Apple M5 Pro (20 GPU
cores), with six and 32 tracers, full TM5 collaborative convection, Dkg
diffusion and column output. Warmed runs took 2.90 and 8.28 seconds; maximum
column relative L2 differences from CUDA were below 9e-8 and relative mass
drift below 5.7e-8. See the Metal verification record.
This does not establish coverage of Metal adjoints or every operator. Optimized,
clamped or reduced-column convection paths do not all have supported adjoints.
The shipped inversion CLI remains synthetic-only; real-data inversion assembly
and external TM5-4DVAR cross-validation are separate tasks. Numerical checks
and performance measurements are not a complete observational validation.

Merged pull requests:

• Prepare 0.4: conservative transport, release hardening and adjoint documentation (#6) (@cfranken)

v0.2.0 — time-varying flux & Float64

The time-varying-flux / Float64 release.

Highlights

• Float64 cubed-sphere transport — F64 runs now work end-to-end on the cubed sphere. The reader keeps the on-disk Float32 mmap and converts per-window instead of materializing a whole-binary F64 copy (which OOM-killed long runs); ATMSNAP snapshots stay Float32 on disk regardless of compute precision.
• Time-varying surface flux — dispatchable temporal schemes (stepwise / linear / conservative_mean); lmdz/CAMS defaults to the GC-faithful stepwise hold, driving the diurnal cycle instead of a monthly mean.
• UTLS "fingering" cure — MERRA-2 wind-derived C180 source plus the GEOS geos_cm_closure = "omega_consistent" cm closure (tracer-validated to the MERRA-2 level).

Physics & preprocessing

• precomputed_kz diffusion (uses the binary :kz TM5 bldiff payload).
• cmfmc_matrix convection + collaborative-LU knobs (use_collab_lu, lmax_conv, n_merge); convection clipping mass blow-up fixed and n_merge = 2 enabled.
• Full-window flux storage contract; adaptive per-window substep schedule baked into each binary to meet CFL.
• F32 mass conservation: Kahan-compensated surface flux + diffusion anomaly subtraction.
• Multi-binary clock-origin fix (removes the day-1 time-varying-flux replay / +1 Pg/month co2_natural surplus).

Performance

• CPU-only sweep-sync gating (GPU launch-bound), ping-pong cubed-sphere sweeps, double-buffered async daily snapshot write, rolling NVMe input staging + O_DIRECT copy, and per-day mmap page-cache release (madvise). Full-year 4-tracer C180 in ~2.5 h on a single L40S.

Infrastructure & docs

• Pinned ConservativeRegridding / GeometryOps compat; CI restored to green (Julia 1 + 1.10 + docs build).
• Config schema and quickstart updated for the new options.

Notes

• Float64 needs an A100-class GPU or the CPU backend; Float32 runs on the NVIDIA L40S.

AtmosTransport quickstart data v2

Preprocessed ERA5 quickstart transport binaries for Dec 1-3 2021. Includes LL 72x37, LL 144x73, CS C24, and CS C90 bundles regenerated with format_version=3 and adaptive substep schedules.