Skip to content

Mallard 0.10.0

Latest

Choose a tag to compare

@MatthewBonanni MatthewBonanni released this 13 Nov 13:26
· 132 commits to main since this release
0d4d199

Mallard 0.10.0 keeps the GPU fed and makes every run explain itself. Steps are decided on the device and enqueued ahead, and every kind of run, reacting and LES included, takes its steps in the CUDA graph. Inputs that choose no numerics now run the high_order preset. Every run writes a self-contained report.html, records what each file holds and where it came from, and diagnoses its own divergence. New mallard post, info, diff and pack commands and a Python package work on saved output, regions can be defined by the solution and move in time, and probes and statistics gain averages, spectra, conditional statistics, convergence monitoring and triggers. mallard check now sizes a run, reports its resolution, checks the environment and draws the case.

Changes that affect results

  • Inputs that set neither [numerics] preset nor [numerics.face_reconstruction] run the high_order preset (TENO-E 5, HLLC, SSPRK3, cfl = 1.0 when [run] sets neither cfl nor dt) instead of first order. [numerics.face_reconstruction] type = "FO" restores first order; mallard input upgrade notes the inputs this concerns.
  • The les preset runs SSPRK54 at cfl = 2.0 instead of SSPRK3 at 0.8 (#774). [numerics] time_integrator = "SSPRK3" and [run] cfl = 0.8 restore it.
  • TENO with convective_flux = "hybrid" feeds the central flux the cells' values (central = "cell_averages"), which no longer adds kinetic energy (#689, #801). central = "reconstructed" restores the earlier flux; MUSCL and first order are unchanged.
  • Strang-split chemistry fuses each step's closing half step with the next step's opening one ([chemistry] fuse_half_steps = true; #777). Results change within the chemistry's tolerances and no longer depend on the output schedule or on restarts. fuse_half_steps = false restores the separate half steps.
  • Bound-preserving TENO no longer makes premixed flames 4-6% too fast (#707): the H2/air flame at 700 K runs within 1% of Cantera with TENO3 and TENO5.
  • Probes no longer read samples outside the domain from the nearest cell (#673). A point or points probe outside the domain is an input error; curves and resampled surfaces drop their outside samples with W-PROBE-OUTSIDE and record them as clipped.
  • Regions follow curved walls (#724): volume fractions, probe samples, clipped zones and extracted surfaces use the curved cells.
  • TENO tables change at round-off (#776, #938). TENO caches become MALLARD-TENO-14 and older ones are recomputed. TENO also sets up on meshes one cell thick, where TENO3 results change slightly.
  • Runs whose t_stop is a multiple of a time_interval no longer take an extra round-off-length step and write a second final snapshot.
  • Last-bit changes: stage times of SSPRK54, RDPK3Sp35 and CK45 with time-dependent boundaries, inflows, sources or sponges, which cannot be restored (#944); inputs that iterate a table, such as compositions, now read in key order (#509); sponges, where filters, mass-flow profiles and boundary compositions evaluated by kexpr instead of exprtk; the exact sum of average-pressure outlets; and, on GPUs, synthetic-turbulence inflow weights and the dynamic TFLES beta. GCC builds without a device backend use -ffp-contract=off, so arm64 matches x86-64.

Upgrading inputs

mallard input upgrade old.toml > new.toml rewrites inputs of earlier releases and notes the inputs affected by the new defaults above.

  • Keys deprecated in 0.9.0 are removed and are now errors naming their replacement: the [[boundaries]] types upt, p_out, p_out_average, extrapolation, nscbc_inlet, nscbc_outlet and characteristic parameters outside the nscbc table; [[mesh.curved]] shape; [[write_data]] geometry; [mesh] periodic and [initialize.mesh] periodic.
  • [[write_data]] format = "hdf5" is removed: use "vtkhdf" (the default) or "xdmf" for the same XDMF and HDF5 files. [[probes]] format = "hdf5" is unchanged.
  • Renamed, with a warning until 0.11.0:
    • [[write_data]] variables and [[probes]] variables → fields (#502);
    • [numerics] preset = "robust" / "accurate" → "second_order" / "high_order";
    • mallard check --quality → --mesh-fields (#521), mallard new --list → mallard list templates.
  • Q is renamed Q_CRITERION, with no alias (#511).
  • VTKHDF snapshots default to series = true: one PREFIX.vtkhdf holds every snapshot (#768). series = false restores one file per snapshot.
  • exprtk is no longer bundled. The syntax is unchanged; its vector, string and bitwise functions and case-insensitive names are errors that name the problem.
  • Restart files are version 7, which 0.9 and earlier reject; every earlier version is still read (#667).

GPU performance

  • Steps enqueued ahead: the device decides dt, the end of the run, retries, NaN checks, monitors, probes, statistics and [[triggers]], and the host enqueues [run] step_queue steps ahead (3 by default), stopping only at steps it must act on. Results are the same bit for bit at any depth (#730, #906, #914).
  • CUDA graphs everywhere: reacting mixtures (Strang and SIMPLER, TFLES, PaSR, chemistry load balancing), LES, mixtures without chemistry, nscbc boundaries, synthetic inflow turbulence, average-pressure outlets and sources of domain averages take their steps in the graph. In NCCL builds, steps outside the graph also communicate with NCCL on the GPU's stream.
  • Cell ordering: [mesh] cell_order = "auto" numbers cells along a Hilbert curve when it reads fewer cache lines, so a mesh file's numbering no longer sets the speed (a Gmsh sphere 28% faster on an A100; #478, #277).
  • Weak scaling: distributed TENO reconstructs only owned cells and overlaps the exchange, taking 3D TENO5 on 16 A100s from 91% to 96.5% (#305).
  • Kernels: boundary dispatch per condition (teno5_3d +0.9%) and leaner mixture and chemistry kernels (flame_3d +3.3%) (#613).
  • Setup and memory: 2D TENO setup on the device and up to 1.7 times faster on CPUs (#776, #916); analytical initialization of 1M hexahedra in 0.3 s instead of 3.2 s; faster mesh build and probe setup; setup-only data freed before the first step.
  • Large runs: exchanges, gathers and writes of more than 2^31 values no longer fail or corrupt data (#775, #876, #894).

Checking a case

  • Sizing: mallard check breaks down memory by category at the setup's peak and in the steady state, and --memory 40GB or --memory a100-80 gives the most cells per GPU and the fewest GPUs the mesh fits on (#550).
  • Resolution: cell sizes against Kolmogorov and Taylor lengths, first-cell y+, x+ and z+, wall-model exchange points and cells across TFLES's delta_L, with warnings and the spacing to aim for (#548).
  • Environment: mallard check without an input checks the build, devices, placement, MPI, GPU-aware MPI and NCCL; run it under srun or mpirun to check a job's ranks (#508).
  • Case view: mallard check --view DIR also draws the case as light and dark SVGs, every zone, region, probe and periodic pair labeled (#546).
  • Mesh and stencil fields: --mesh-fields PREFIX writes mesh quality and TENO or MUSCL stencil fields, and --stencils every cell's stencils (#521).
  • Reports: sections and tables with statuses and units, stable codes (W-CFL-LIMIT, ...) with a docs link for every warning and error, and --format json (#657, #659, #828).

Run records and reproducibility

  • report.html: every run writes one self-contained page with its outcome, configuration, health, plots, cost and output index (#515).
  • Divergence report: a diverging run names the invalid cells, ranks the likely causes with their evidence and what to try, and writes a snapshot of the neighborhood (#549, #659).
  • What every file holds: a snapshot block with step, time, run_id, why it was written, units, precision, value kinds and averaging windows (#667, #671, #672, #674); the run block records the resolved input and every file the run read with its SHA-256 (#509, #668).
  • Safe output: every file is written as NAME.partial and renamed once complete; mallard info DIR checks every recorded file (#669). The end record gives the outcome and cause (#670).
  • Resuming: --restart latest resumes from the newest complete restart file, as the job scripts and manifests now do (#510); t_wall_stop stops every rank after the same step (#670).
  • Reproducing: mallard input resolve writes the input as the run uses it (#509), and mallard pack RUN_DIR archives everything needed to rerun (#519).
  • Monitoring: --log-format jsonl streams progress records on stdout (#827); the setup banner, progress table and summary are tables with units (#659, #675); physics.units = "nondimensional" labels no number with an SI unit (#659).

Working with output

  • mallard info FILE|RUN_DIR reports what any file holds, or a run's outcome, health and cost.
  • mallard diff A B compares snapshots, restarts, probes or CSV files with norms, tolerances and exit statuses for CI, across meshes with --remap.
  • mallard post:
    • derive: derived fields of saved files, bit for bit what [[write_data]] writes (#511, #858);
    • average: means, RMS, Reynolds and Favre covariances, homogeneous-direction profiles, solution-defined regions and conditional statistics (#512, #513, #799, #851);
    • extract: probes and snapshots of saved output without rerunning;
    • convergence: observed order, Richardson extrapolation and grid convergence index across resolutions, with or without an exact solution (#514, #769, #812).
  • Derived fields: MACH, Q_CRITERION, LAMBDA2, SCHLIEREN, P0, T0, H0, ENSTROPHY, CP and others, and for mixtures MIXTURE_FRACTION, PROGRESS_VARIABLE, FLAME_INDEX and SCALAR_DISSIPATION, from one catalog shared by [[write_data]], [statistics] and mallard post derive (#511, #512).
  • Python package mallard: mallard.open reads every file Mallard writes into numpy arrays with units, through Mallard's own reader (#517). Containers, Spack and source builds install it (#791).
  • Discovery: mallard list fields, list keys, list species and list templates (#502, #786), and mallard completion bash|zsh|fish, which also completes keys and fields (#507, #770, #785).

Regions and probes

  • Regions defined by the solution: {type = "where", condition = "T > 1500"} and {type = "isosurface", ...}, followed on the device by sources, sponges, probes and snapshots (#543, #797, #800).
  • Moving and repeated regions: centers, sizes and transforms may depend on t; repeat makes arrays of copies; regions wrap across periodic boundaries (#501).
  • Exact sampling: curve probes with n_points = "cells" and filled regions with sampling = "point" are exact for the reconstruction polynomials (#592). Filled region probes record their volume and area (#724), and every probe file records where each sample is read (#673).
  • Averages and spectra: average = true on probes and isosurfaces (#590, #798, #836), probes of the [statistics] averages (#590), and in-run time spectra with Strouhal numbers (#614).
  • Boundary probes: diffusive and total species and energy fluxes, the wall model's exchange-point state and a moment reduction (#591); walls with a wall_model take wall quantities from the model.

Statistics and triggers

  • [[triggers]]: conditions on t, step, wall and monitored values that stop the run, write output or start the statistics, decided identically on every rank.
  • Convergence: [statistics.convergence] detects the end of the transient and gives confidence intervals and effective sample sizes that triggers read (#337); [statistics] batch gives every cell confidence intervals (#337).
  • Conditional statistics: [[statistics.conditional]] PDFs, joint PDFs and conditional means, the same bit for bit on any rank count; [statistics] samples HRR and OMEGA_<species> (#513).

Boundary and initial conditions

  • Time-dependent inflows: direction expressions of x, y, z, t; synthetic turbulence on an inlet_state whose u depends on t (#600); inlet_mass_flow mdot and profile of t (#601).
  • Turbulent initial states: [initialize.turbulence] adds a divergence-free field of a prescribed spectrum, the same bit for bit on any rank count (#563).
  • LES: [les] gradients = "reconstruction" (experimental, TENO only) takes the subgrid models' velocity gradient from TENO's central polynomial (#660).

Platforms

  • Toolchains: GCC 15, Clang 22 or Apple Clang 21 and CMake 4.2 or newer are required; the Spack package with +cuda needs CUDA 13.4 or newer.
  • Containers are built on Ubuntu 26.04, the CUDA image with CUDA 13.5 (driver R580 or newer).

Known limitations

  • Steps one at a time: runs with triggers that read a probe's value, and the progress rows, monitors, means and probes of reacting runs with fused half steps, are not enqueued ahead (#730, #777).
  • Owner-computes overlap is reduced on meshes with rotational periodicity (#911) and in LES with gradients = "reconstruction" (#912).
  • ParaView 6.0.1 fails to open a bare .xmf file name; open ./flow.xmf instead (#682).

Installing

  • docker pull ghcr.io/matthewbonanni/mallard:0.10.0 (CPU) or ghcr.io/matthewbonanni/mallard:0.10.0-cuda (A100 to H100)
  • Spack: spack repo add --path spack/spack_repo/mallard https://github.com/MatthewBonanni/mallard.git, then spack install mallard %gcc@15 (GCC 15 or newer; see the installation guide)

Full list of changes: CHANGELOG.md · v0.9.1...v0.10.0