Skip to content

Releases

Mallard 0.10.0

Choose a tag to compare

@MatthewBonanni MatthewBonanni released this 13 Nov 13:26
0d4d199

Mallard 0.10.0 keeps the GPU fed and makes every run explain itself. Steps are decided on the device and enqueued ahead, and every kind of run, reacting and LES included, takes its steps in the CUDA graph. Inputs that choose no numerics now run the high_order preset. Every run writes a self-contained report.html, records what each file holds and where it came from, and diagnoses its own divergence. New mallard post, info, diff and pack commands and a Python package work on saved output, regions can be defined by the solution and move in time, and probes and statistics gain averages, spectra, conditional statistics, convergence monitoring and triggers. mallard check now sizes a run, reports its resolution, checks the environment and draws the case.

Changes that affect results

  • Inputs that set neither [numerics] preset nor [numerics.face_reconstruction] run the high_order preset (TENO-E 5, HLLC, SSPRK3, cfl = 1.0 when [run] sets neither cfl nor dt) instead of first order. [numerics.face_reconstruction] type = "FO" restores first order; mallard input upgrade notes the inputs this concerns.
  • The les preset runs SSPRK54 at cfl = 2.0 instead of SSPRK3 at 0.8 (#774). [numerics] time_integrator = "SSPRK3" and [run] cfl = 0.8 restore it.
  • TENO with convective_flux = "hybrid" feeds the central flux the cells' values (central = "cell_averages"), which no longer adds kinetic energy (#689, #801). central = "reconstructed" restores the earlier flux; MUSCL and first order are unchanged.
  • Strang-split chemistry fuses each step's closing half step with the next step's opening one ([chemistry] fuse_half_steps = true; #777). Results change within the chemistry's tolerances and no longer depend on the output schedule or on restarts. fuse_half_steps = false restores the separate half steps.
  • Bound-preserving TENO no longer makes premixed flames 4-6% too fast (#707): the H2/air flame at 700 K runs within 1% of Cantera with TENO3 and TENO5.
  • Probes no longer read samples outside the domain from the nearest cell (#673). A point or points probe outside the domain is an input error; curves and resampled surfaces drop their outside samples with W-PROBE-OUTSIDE and record them as clipped.
  • Regions follow curved walls (#724): volume fractions, probe samples, clipped zones and extracted surfaces use the curved cells.
  • TENO tables change at round-off (#776, #938). TENO caches become MALLARD-TENO-14 and older ones are recomputed. TENO also sets up on meshes one cell thick, where TENO3 results change slightly.
  • Runs whose t_stop is a multiple of a time_interval no longer take an extra round-off-length step and write a second final snapshot.
  • Last-bit changes: stage times of SSPRK54, RDPK3Sp35 and CK45 with time-dependent boundaries, inflows, sources or sponges, which cannot be restored (#944); inputs that iterate a table, such as compositions, now read in key order (#509); sponges, where filters, mass-flow profiles and boundary compositions evaluated by kexpr instead of exprtk; the exact sum of average-pressure outlets; and, on GPUs, synthetic-turbulence inflow weights and the dynamic TFLES beta. GCC builds without a device backend use -ffp-contract=off, so arm64 matches x86-64.

Upgrading inputs

mallard input upgrade old.toml > new.toml rewrites inputs of earlier releases and notes the inputs affected by the new defaults above.

  • Keys deprecated in 0.9.0 are removed and are now errors naming their replacement: the [[boundaries]] types upt, p_out, p_out_average, extrapolation, nscbc_inlet, nscbc_outlet and characteristic parameters outside the nscbc table; [[mesh.curved]] shape; [[write_data]] geometry; [mesh] periodic and [initialize.mesh] periodic.
  • [[write_data]] format = "hdf5" is removed: use "vtkhdf" (the default) or "xdmf" for the same XDMF and HDF5 files. [[probes]] format = "hdf5" is unchanged.
  • Renamed, with a warning until 0.11.0:
    • [[write_data]] variables and [[probes]] variables → fields (#502);
    • [numerics] preset = "robust" / "accurate" → "second_order" / "high_order";
    • mallard check --quality → --mesh-fields (#521), mallard new --list → mallard list templates.
  • Q is renamed Q_CRITERION, with no alias (#511).
  • VTKHDF snapshots default to series = true: one PREFIX.vtkhdf holds every snapshot (#768). series = false restores one file per snapshot.
  • exprtk is no longer bundled. The syntax is unchanged; its vector, string and bitwise functions and case-insensitive names are errors that name the problem.
  • Restart files are version 7, which 0.9 and earlier reject; every earlier version is still read (#667).

GPU performance

  • Steps enqueued ahead: the device decides dt, the end of the run, retries, NaN checks, monitors, probes, statistics and [[triggers]], and the host enqueues [run] step_queue steps ahead (3 by default), stopping only at steps it must act on. Results are the same bit for bit at any depth (#730, #906, #914).
  • CUDA graphs everywhere: reacting mixtures (Strang and SIMPLER, TFLES, PaSR, chemistry load balancing), LES, mixtures without chemistry, nscbc boundaries, synthetic inflow turbulence, average-pressure outlets and sources of domain averages take their steps in the graph. In NCCL builds, steps outside the graph also communicate with NCCL on the GPU's stream.
  • Cell ordering: [mesh] cell_order = "auto" numbers cells along a Hilbert curve when it reads fewer cache lines, so a mesh file's numbering no longer sets the speed (a Gmsh sphere 28% faster on an A100; #478, #277).
  • Weak scaling: distributed TENO reconstructs only owned cells and overlaps the exchange, taking 3D TENO5 on 16 A100s from 91% to 96.5% (#305).
  • Kernels: boundary dispatch per condition (teno5_3d +0.9%) and leaner mixture and chemistry kernels (flame_3d +3.3%) (#613).
  • Setup and memory: 2D TENO setup on the device and up to 1.7 times faster on CPUs (#776, #916); analytical initialization of 1M hexahedra in 0.3 s instead of 3.2 s; faster mesh build and probe setup; setup-only data freed before the first step.
  • Large runs: exchanges, gathers and writes of more than 2^31 values no longer fail or corrupt data (#775, #876, #894).

Checking a case

  • Sizing: mallard check breaks down memory by category at the setup's peak and in the steady state, and --memory 40GB or --memory a100-80 gives the most cells per GPU and the fewest GPUs the mesh fits on (#550).
  • Resolution: cell sizes against Kolmogorov and Taylor lengths, first-cell y+, x+ and z+, wall-model exchange points and cells across TFLES's delta_L, with warnings and the spacing to aim for (#548).
  • Environment: mallard check without an input checks the build, devices, placement, MPI, GPU-aware MPI and NCCL; run it under srun or mpirun to check a job's ranks (#508).
  • Case view: mallard check --view DIR also draws the case as light and dark SVGs, every zone, region, probe and periodic pair labeled (#546).
  • Mesh and stencil fields: --mesh-fields PREFIX writes mesh quality and TENO or MUSCL stencil fields, and --stencils every cell's stencils (#521).
  • Reports: sections and tables with statuses and units, stable codes (W-CFL-LIMIT, ...) with a docs link for every warning and error, and --format json (#657, #659, #828).

Run records and reproducibility

  • report.html: every run writes one self-contained page with its outcome, configuration, health, plots, cost and output index (#515).
  • Divergence report: a diverging run names the invalid cells, ranks the likely causes with their evidence and what to try, and writes a snapshot of the neighborhood (#549, #659).
  • What every file holds: a snapshot block with step, time, run_id, why it was written, units, precision, value kinds and averaging windows (#667, #671, #672, #674); the run block records the resolved input and every file the run read with its SHA-256 (#509, #668).
  • Safe output: every file is written as NAME.partial and renamed once complete; mallard info DIR checks every recorded file (#669). The end record gives the outcome and cause (#670).
  • Resuming: --restart latest resumes from the newest complete restart file, as the job scripts and manifests now do (#510); t_wall_stop stops every rank after the same step (#670).
  • Reproducing: mallard input resolve writes the input as the run uses it (#509), and mallard pack RUN_DIR archives everything needed to rerun (#519).
  • Monitoring: --log-format jsonl streams progress records on stdout (#827); the setup banner, progress table and summary are tables with units (#659, #675); physics.units = "nondimensional" labels no number with an SI unit (#659).

Working with output

  • mallard info FILE|RUN_DIR reports what any file holds, or a run's outcome, health and cost.
  • mallard diff A B compares snapshots, restarts, probes or CSV files with norms, tolerances and exit statuses for CI, across meshes with --remap.
  • mallard post:
    • derive: derived fields of saved files, bit for bit what [[write_data]] writes (#511, #858);
    • average: means, RMS, Reynolds and Favre covariances, homogeneous-direction profiles, solution-defined regions and conditional statistics (#512, #513, #799, #851);
    • extract: probes and snapshots of saved output without rerunning;
    • convergence: observed order, Richardson extrapolation and grid convergence index across resolutions, with or without an exact solution (#514, #769, #812).
  • Derived fields: MACH, Q_CRITERION, LAMBDA2, SCHLIEREN, P0, T0, H0, ENSTROPHY, CP and others, and for mixtures MIXTURE_FRACTION, PROGRESS_VARIABLE, FLAME_INDEX and SCALAR_DISSIPATION, from one catalog shared by [[write_data]], `[statistics]...
Read more

Mallard 0.9.1

Choose a tag to compare

@MatthewBonanni MatthewBonanni released this 02 Nov 06:52
254f979

Mallard 0.9.1 fixes two problems in 0.9.0. Viscous [[forces]] were slightly wrong on several ranks, and the release published no container images. Inputs and results are otherwise unchanged from 0.9.0.

Fixed

  • [[forces]] on several ranks exchanges the halo before computing the wall gradients (#623). Before, the viscous force read stale halo cells, an O(dt) error (4e-7 in C_D in the measuring tutorial). Single-rank runs, runs with a wall model and boundary probes were not affected.
  • Container images and the Spack package are published (#625). The 0.9.0 release workflow failed its smoke tests, so ghcr.io/matthewbonanni/mallard had no 0.9.0 images. Use 0.9.1.
  • Images report their version without a -dirty suffix (#627).

Installing

  • docker pull ghcr.io/matthewbonanni/mallard:0.9.1 (CPU) or ghcr.io/matthewbonanni/mallard:0.9.1-cuda (A100 to H100)

See the 0.9.0 release for what's new in 0.9.

Full list of changes: v0.9.0...v0.9.1

Mallard 0.9.0

Choose a tag to compare

@MatthewBonanni MatthewBonanni released this 01 Nov 18:06
4bdd932

Mallard 0.9.0 makes regions a single concept used everywhere. Points, curves, surfaces and volumes, defined once and named, drive sources, sponges, initial states, probes, snapshots and curved boundaries. It adds probes of every dimension with exact fluxes, explicit boundary-condition types with composable options, and a run record and health diagnostics in every run. Jobs resume safely from restarts, on clusters, Kubernetes or prebuilt containers. Under the hood, everything typed is now one lowercase mallard command, every input error names its key and line, and old inputs are upgraded automatically.

Changes that affect results

  • The default TFLES efficiency is "dynamic" (#310). efficiency = "charlette" restores the 0.8.0 model.
  • TENO results change in troubled cells (#436): stencil selection no longer depends on axis orientation or cell numbering. TENO cache files are recomputed.
  • 2D TENO on straight-sided curved walls no longer settles into an odd-even wall-pressure mode (#382). 2D stencils on anisotropic or faceted-wall meshes change.
  • The stable time step includes the exterior states of boundary faces (#374). Runs whose inflow is faster than the adjacent cells take smaller steps.
  • Bound-preserving TENO keeps density and pressure positive in every cell at CFL ≤ 0.5 in planar single-gas runs (#603). Faces that need no limiting are unchanged.
  • Transmissive boundaries on unstructured meshes take the image cell's polynomial instead of its average (#607). A uniform subsonic inflow on triangles or tetrahedra was unstable with TENO.
  • [source] expressions of t are re-evaluated at every Runge-Kutta stage (#384). Before, they were evaluated once unless time_dependent = true.
  • Analytical initial states on quadrilaterals change by their quadrature error, up to 1e-8 relative (#436). They are integrated on the device.
  • The hybrid convective flux with SIMPLER coupling is rejected (#300). The combination was linearly unstable.

Upgrading inputs

mallard input upgrade old.toml > new.toml rewrites a 0.8.0 input, keeping comments and layout. Renamed keys still run in 0.9.0 with a warning naming the replacement, and are removed in 0.10.0.

  • Boundary types are explicit subtypes, kind first (#564): upt → inlet_state, p_out → outlet_pressure (average = true for p_out_average), extrapolation → outlet_extrapolation, nscbc_inlet / nscbc_outlet → inlet_state / outlet_pressure with an nscbc table.
  • Snapshots and geometry:
    • [[write_data]] geometry → region (#542);
    • [[write_data]] format = "hdf5" → "vtkhdf", now the default;
    • [[mesh.curved]] shape → type, with filled = false for surfaces (#407).
  • [mesh] periodic → [[periodic]] entries, one syntax for generated meshes and mesh files.
  • Unknown or unused keys are errors, with the line and the likely intended key (#362).
  • One lowercase mallard executable with subcommands replaces Mallard, MallardReactor, mallard-mesh-convert and MallardTest (#414). The 2D and 3D solvers install side by side behind it (#483).

Regions everywhere

  • Named regions: [regions] defines a region once; every region key takes its name (#500). The mesh's domain, boundary zones and cell zones are regions by name, without an entry (#580).
  • Every dimension: each region type has one dimension. {type = "boundary", of = ...}, {type = "interior", of = ...} and filled = false move between dimensions, and intersection/difference clip across them (#580, #582). Surfaces of primitives are exact, boundaries of booleans are extracted from the mesh, and surfaces and curves are oriented (#582).
  • Users:
    • sources, sponges (#500) and initial states ([[initialize.regions]], plus [[initialize.data]] from tables and point clouds; #497);
    • snapshots of any region (#578) and curved boundaries projected onto any surface region (#574).
  • mallard check lists every region with its dimension, measure and cells, and writes a ParaView case view (#584).

Probes and measurement

  • Probes of every dimension:
    • points and curves with high-order point values (#358);
    • boundary zones with wall shear, heat flux, C_f, C_p, y+ and fluxes (#570);
    • exact cut planes and resampled region surfaces with fluxes through them (#572, #586);
    • volume and surface region reductions: integral, mean, Favre, min, max, rms (#500, #586).
  • Variables: probes and statistics accept the conserved variables RHOU_*, RHOE and RHOY_* for instantaneous control-volume balances (#616).
  • Consistency: probe output is the same bit for bit on any rank count, tested against the GPU (#598, #605).
  • Statistics: [statistics.spectra] gives in-situ spectra and two-point correlations, and [statistics.phase] phase averages (#284, #311).

Boundary conditions

  • Options compose on each type: nscbc, average, turbulence, wall_model (#564). Unsupported combinations are errors that point to the plan (#565).
  • Values as expressions: temperatures, heat fluxes, pressures, velocities and mass fluxes may be expressions of x, y, z and t, evaluated on the GPU at each Runge-Kutta stage (#569). dirichlet states of t also run on the device inside CUDA graphs (#602).
  • New boundaries: inlet_mass_flow (#313), inlet_stagnation (#312), and isothermal and heat-flux walls for mixtures (#351).
  • Rotational periodicity (#315).

Output, run health and restarts

  • VTKHDF is the default snapshot format (#559).
  • Asynchronous and compressed output (#309).
  • Units and provenance: units and descriptions in every output file (#518), and provenance in every file (#376).
  • The run record mallard.jsonl holds start, progress and end records. A resumed run names its restart file and marks rewinds (#376, #619).
  • Run health:
    • extremes and activity per rank, a conservation ledger that closes at round-off, and an audit of numerical repairs (#571);
    • divergence detection that stops naming the bad cells (#355).
  • Safe restarts:
    • restart files are written under a temporary name and renamed once complete (#608);
    • resumed runs continue snapshot numbering instead of overwriting earlier snapshots (#611);
    • mallard ctl checkpoints, stops and changes a running job (#355).

Running anywhere

  • Prebuilt installs: container images on ghcr.io/matthewbonanni/mallard (CPU, and :cuda for A100 to H100) and a Spack package (#328).
  • Job scripts: for Slurm, PBS, Kubernetes and mpirun. They resume from the newest restart and resubmit themselves (#330, #386). Kubernetes Job and MPIJob manifests (#453).
  • Placement: each rank picks the GPU nearest its cores, and parallel runs report where every rank runs (#330, #386).
  • Mesh readers: CGNS, ANSYS Fluent and OpenFOAM (#354, #357, #361), plus [mesh.transform].
  • Solution remap between meshes (#353).

Setting up a case

  • Templates: mallard new <template> creates a ready-to-run case from 23 examples and tutorials (#368).
  • Pre-flight checks: mallard check finds unused keys, unconditioned zones, mesh quality issues and the stable time step, and estimates memory and cost (#321, #362).
  • Help: every command has a full --help page (#503).
  • Input conveniences: units in input files (p = "1 atm"; #389), numerics presets (#391), and one expression language with loops and conditionals, compiled for the GPU (#317).

Gallery and tutorials

  • New gallery cases: a pulsed jet launching vortex rings (#579), flow past an ellipsoid with flat against curved wall faces (#575), and the Mallard logo as an initial condition (#576). The Mach 8 mallard is rerun on a duck traced from the logo (#547).
  • New tutorials: running on a cluster (#558) and measuring a flow with regions, probes, fluxes and phase averages (#596). The user guide is reorganized by topic.

Known limitations

  • GPU performance: teno5_3d runs about 2–4% slower than early-October builds on A100, and the explicit boundary-type dispatch costs about 1.7% of that (#613). A possible GPU race in the 3D rotational-periodic path is under investigation (#612).
  • [[forces]] viscous forces on several ranks read stale halo values without a wall model, an O(dt) error (4e-7 in C_D in tutorial 8). It is fixed in #623 for the next release. Boundary probes are not affected.

Full list of changes: CHANGELOG.md · v0.8.0...v0.9.0

Mallard 0.8.0

Choose a tag to compare

@MatthewBonanni MatthewBonanni released this 15 Oct 19:55

Mallard 0.8.0 adds curved high-order boundaries, which bring TENO-E orders 3–6 to their design accuracy at curved walls, where straight-sided faces did not converge. It sets up 3D TENO 6–9.5× faster on the GPU and fits about 15% more cells on one (twice as many with opt-in single-precision tables), scales 3D TENO5 to 91% weak efficiency on 16 GPUs, makes reacting mixtures 2× faster per step, and adds SIMPLER splitting, Soret diffusion and optically thin radiation.

Changes that affect results

  • RHLL blends toward HLL across faces that carry a shock (#248, fixes #80): w F_rotated + (1 − w) F_HLL(n) with w = min(p_l/p_r, p_r/p_l)^3, so RHLL results change wherever a face carries a pressure jump; contacts and shear layers keep the rotated flux, and faces with equal pressures are bitwise unchanged. In the examples: shu_osher max |Δρ| 1.2e-5; double_mach at 240×60 shocks shift by a cell locally (mean |Δρ| 0.04; the kink in the RHLL Mach stem is gone); mallard on a coarse mesh mean |Δρ| 7e-3. Other Riemann solvers are unchanged.
  • Multi-jagged partitions are the default (#261): results are bitwise identical on any rank count, but distributed runs get different partitions, so their per-rank output pieces and per-rank TENO cache files (recomputed on first use) differ from 0.7.0's. [parallel] partitioner = "hilbert" restores the 0.7.0 partitions.
  • Gmsh high-order boundary elements now curve their faces (#263): 0.7.0 read elements of order 2 and 3 by their corners; now line3/line4 (2D) and triangle6 … quadrangle16 (3D) boundary elements give curved faces by default. [mesh] curved_geometry = false restores straight faces. Linear meshes, and every example, are bitwise unchanged (10 2D + 2 3D examples on CPU, 9 2D + 2 3D on an A100).
  • 3D TENO tables on macOS (arm64) change at round-off (#247): the TENO setup is now compiled without FMA contraction, so its tables in macOS host builds differ from 0.7.0's at round-off; on x86 and CUDA they are bitwise identical to 0.7.0's.
  • TENO cache files are recomputed (#247, #265): the cache format is now MALLARD-TENO-10 (3D caches hold the cells' moments instead of their smoothness matrices); older caches are detected and rebuilt.
  • Everything else is opt-in or bitwise identical: CUDA graphs and NCCL (#217), the MUSCL and halo-exchange changes (#239, #258), the GPU TENO setup and its smaller memory (#247, #265), mixture performance (#264, except the opt-in chemistry.C_reg), chemistry load balancing (#255, #262), the large-mechanism solves (#246) and the conversion-warning series (#267, #269–#276). SIMPLER coupling (#235), Soret diffusion (#243) and radiation (#226) change nothing unless enabled.

Curved high-order boundaries

Design and results: docs/design/curved_boundaries.md (#263, closes #162).

  • Curved faces from the mesh or from an analytic shape: Gmsh 2.2/4.1 high-order boundary elements of order 2 or 3 (kept by mallard-mesh-convert in HDF5), or [[mesh.curved]] projection onto a circle (2D), sphere or cylinder (3D) on any mesh. Curved faces get mapped quadrature points with per-point normals and Jacobians, closed exactly so gas at rest stays at rest to round-off; the cells touching them get exact volumes, centroids, TENO moments and initial averages. In 3D, faces sharing a curved edge (prism-layer sides, symmetry planes meeting the wall) are blended from their edges. Any rank count gives bitwise identical results. make_cylinder_mesh.py --order 2|3, make_sphere_mesh.py and the new make_annulus_mesh.py write high-order boundaries.
  • TENO at curved walls: one-sided stencils at curved walls and symmetry faces (curved_mirrors = false; mirroring across chords does not converge), and the central fits of cells reaching them capped at degree 3 (curved_wall_degree), since one-sided degree-4/5 fits grow exponentially there.
  • Steady vortex between circular walls (tools/curved_convergence.py, quadrilaterals, n_r = 8–64): subsonic (M = 0.5) curved L1 rates 3.03 / 4.89 / 4.90 / 4.92 for orders 3 / 4 / 5 / 6 (Linf 2.86 / 3.99 / 3.95 / 3.65), L1 at n_r = 64 4.2e-8 / 3.5e-10 / 3.2e-10 / 3.8e-10; with straight walls none converges (L1 2e-2 to 8e-2). Triangles: L1 rates 3.45 / 4.38 / 4.91 / 4.91. Supersonic (M = 2.25): 3.64 / 4.86 / 5.00 / 5.57, against O(0.1–1) errors with straight walls. Gmsh P3 walls match the analytic projection to three digits.
  • Cylinder at Re = 100: curving the walls moves C_D by 0.1–0.3% (128×64 TENO3: St 0.1633, C_D 1.3671 → 1.3643; 384×128 TENO3: 0.1648, 1.3637 → 1.3629). On 384×128, TENO5 with straight walls is unstable (C_D fluctuations ±2.9); with curved walls it runs: St 0.1648, C_D 1.3614.
  • Mach 3 sphere (810k tetrahedra, quarter box): MUSCL standoff Δ/R 0.2211 straight / 0.2212 curved, pressure C_D 0.9716 / 0.9720 (Billig 0.205); TENO5 with curved walls, which straight walls cannot run here: 0.2213, C_D 0.9790.
  • Fix: the 3D TENO host retry wrote compacted cells out of bounds (segfault on the Mach 3 sphere mesh).

Performance and scaling

Details: docs/design/performance.md.

  • GPU-resident 3D TENO setup (#247, #140): the per-cell setup (stencil search, moments, QR pseudo-inverses, Lebesgue bound, smoothness indicators) runs one team per cell on GPUs, with tables bitwise identical to the host's. 3D TENO5 on one A100: 1M hexahedra 43.5 → 4.5 s (9.5×), 998k tetrahedra 123.2 → 20.0 s (6.2×). This makes restarts on another rank count and re-setup after repartitioning cheap; TENO::rebuild_cells recomputes a subset of cells.
  • 3D TENO device memory (#265, fixes #58): cells keep their central moments instead of their smoothness matrices, troubled-cell passes work in bounded rounds (troubled_capacity), and the setup peak is no higher than the steady state; bitwise identical. Per cell on an A100-80GB: hexahedra TENO5 47.3 → 39.8 KB, tetrahedra TENO5 58.7 → 51.2 KB, hexahedra TENO6 129.6 (setup peak) → 77.9 KB. TENO5 fits ~2.1M hexahedra / 1.65M tetrahedra per A100 (before: ~1.8M / 1.45M); a 125³ Taylor–Green vortex that ran out of memory now runs. Opt-in single_precision_tables (3D, single gas) keeps the pseudo-inverses in float: 25.0 / 30.6 KB per cell, ~3.4M / 2.75M cells, another 3–8% faster; differences below 1e-7 of each field's range on Taylor–Green, up to 6.9e-3 in shock cells of a blast.
  • CUDA graph per step (#217, [run] cuda_graphs, default on in CUDA builds): the time-step reduction, halo exchanges and stages are captured once and replayed, with dt kept on the device. Small MUSCL cases: −20% per step at 4k cells (286 → 230 µs), −25% at 8k cells on 2 GPUs, −5% at 262k cells.
  • NCCL halo exchange (#217, Mallard_ENABLE_NCCL, [parallel] halo_exchange = "nccl" | "mpi"): stream-ordered exchanges and dt reduction, with no host synchronization per exchange, so multi-GPU steps can run inside the graph.
  • MUSCL reconstruction fused per cell and overlapped with the halo exchange (#239): from 2^17 cells per rank; −12% per step on 1M cells (4.09 → 3.61 ms), −10% on 2 GPUs.
  • Slimmer halo exchanges (#258): only the halo cells the kept stencils read are exchanged (TENO's halo holds 7 vertex layers for 3D TENO5). 3D TENO5 at 12 GPUs: weak scaling 85% → 88%, strong (96³) 68% → 72%.
  • Multi-jagged partitions (#261): the new default partitioner cuts equal-count slabs along x, y and z, numbered along a Hilbert curve; a box of cuboid cells gets cuboid parts. 3D TENO5 Taylor–Green on A100s: weak (64³ per GPU) 91% at 16 GPUs (Hilbert 86%); strong (96³) 77% at 12 and 73% at 16 GPUs (Hilbert 72%, 72%); 2D within 1–4% of Hilbert. The setup summary reports the partition surface.
  • Reacting mixtures 2× faster (#264, #257): stencil kernels loop over the stencil once with the variables innermost, and thread-per-cell chemistry keeps its work memory on the stack; bitwise unchanged. New performance case flame_3d (221k hexahedra, propane two-step, Sigma + TFLES): 64.9 → 32.7 ms per step on an A100, 7.8× the air case (was 15.5×); h2o2 reactor 6.5 → 9.7 M cells/s at dt = 1e-8. Opt-in chemistry.C_reg = 1e-10 (default 1e-12, as before) cheapens trace fuel burning out in hot gas: 24.6 ms per step, with T within 0.54 K and integrated heat release within 0.08% after 100 steps.
  • Large mechanisms on GPUs (#246, #146): the sparse LU factors in stages of independent pivots and solves by levels, with the dense radical core stored by columns; bitwise as before. A100, dt = 1e-6: n-hexane (1268 species) 576 → 1278 cells/s, n-dodecane 48.0k → 60.2k.
  • Chemistry load balancing across MPI ranks (#255, #141; with SIMPLER #262): [chemistry] load_balance = true moves the integration of cells from ranks above the mean chemistry work to ranks below it; bitwise unchanged. On A100s: detonation_2d 1.24× faster on 4 GPUs and 1.61× on 8 (time waiting for the slowest rank 51.6% → 8.0% of the step), the flame_turbulence DNS 1.10× / 1.16×, detonation_3d 1.04× / 1.06×.

Reacting flow

  • SIMPLER balanced splitting (#235, #233): [chemistry] coupling = "simpler" (Wu, Ma & Ihme 2019), second order and exact at steady state (a burning stirred reactor moves by 2e-16 where Strang moves by 2.3e-3). detonation_1d at CFL 1.0 and 10 cells per induction length: induction length −0.9% from ZND (Strang −25.5%), 3.7× cheaper than Strang at CFL 0.35; autoignition_2d −19% per step with the same result to 5 digits; flame speeds as with Strang. Strang stays the default.
  • Thermal diffusion (Soret) (#243, #158): physics.soret = true with Cantera's mixture-averaged thermal diffusion coefficients (matched to 5.7e-13). Change of the H2/air flame speed at φ = 0.5 / 1.0: −7.23 / −8.26% (Cantera −6.98 / −8.44%); +9% per step on flame_2d. Multicomponent transport was assessed and not implemente...
Read more

Mallard 0.7.0

Choose a tag to compare

@MatthewBonanni MatthewBonanni released this 09 Oct 17:13

Mallard 0.7.0 adds large-eddy simulation with explicit subgrid-scale and combustion models and a kinetic-energy-preserving hybrid flux, which also brings the Re_τ = 180 channel DNS within 2% of Moser, Kim & Mansour. It adds synthetic turbulent inflow, makes 3D TENO 2.2× faster and stable next to walls, and is the first release under Apache-2.0.

Changes that affect results

  • Example CFLs raised toward the measured stability limits (#210): most examples now run at cfl = 1.0 (from 0.25–0.4; detonation_1d 0.35), 1.4–5× fewer steps. Their results are not bitwise equal to 0.6.0 but each stated result was re-checked: e.g. Sod density L1 2.554e-3 → 2.587e-3, cylinder St/Cd identical to 4 digits, premixed flame speeds identical to 5 digits. noh_axisymmetric, detonation_2d, mallard, the Mach 3 and Re 300 spheres and the multi-hour reacting examples keep their values.
  • channel_retau180 uses the hybrid flux (#216): Re_τ 180.8 instead of 172.9 and C_f +2.0% instead of −6.8% against MKM (see below).
  • 3D TENO stencils are bounded by a Lebesgue constant of 4 (#232, fixes #126 and #229; was 10): this changes 3D TENO results on tetrahedra and pyramids at orders 4–6 (all cells), mixed meshes, prisms at orders 5–6, Kuhn tetrahedra at order 3 and hexahedra at order 6 next to walls. Hexahedra at orders 3–5 and prisms at orders 3–4 are unchanged. Existing TENO cache files are rebuilt (format MALLARD-TENO-8). 2D is unchanged.
  • SSPRK3's last stage is computed as U + 2/3 (U2′ − U) (#237), so a steady state stays exact: in single precision the rounded weights summed to 1 + 3e-8 and drifted by O(1e-4) over 1e4 steps. SSPRK3 results change at round-off in double precision too. Unstretched generated 3D Cartesian meshes now end exactly at L.
  • NSCBC transverse terms (#238, #221): in 3D the fit uses faces sharing an edge instead of a node, faces on an edge with a wall or other non-characteristic boundary get no transverse terms, reversed outlet faces get beta = 0, and fits continue across periodic seams, so 3D NSCBC results change; in 2D only faces next to non-characteristic boundaries, with reversed flow or at periodic seams change.
  • NSCBC runs restart exactly (#219): restart files of runs with nscbc_inlet/nscbc_outlet (format version 4) now store the per-face boundary state, so a restarted run is bitwise equal to an uninterrupted one on any rank count. Such files cannot be read by 0.6.0; files without characteristic faces are written in the older formats as before.
  • MallardReactor's ignition criterion (#220): ignition is the peak of dT/dt once dT/dt has fallen below half of it, instead of a 400 K rise. Lean mixtures that used to report "none" now report their delay (H2/air, φ = 0.1: 3.650 ms, as Cantera).
  • LES and the hybrid flux are opt-in: without [les] and with the default convective_flux = "riemann", the LES work changes nothing (#203, #205). The faster 3D TENO kernel (#212) and the TENO setup's streaming (#227) are bitwise identical to what they replace.
  • eps_numerical is redefined (#222): the kinetic-energy budget now takes the numerical dissipation from the scheme's discrete pressure work (new column pressure_work) instead of p div u from cell gradients, which in flames is 1e3–1e4 times the dissipation and measured the gradients' error. Non-reacting values change little (CBC 64³: −0.9% instead of +0.7% numerical share).

Large-eddy simulation

Design and validation: docs/design/les.md (#200).

  • Eddy-viscosity models (#203): [les] model = "sigma" (Nicoud et al. 2011, 3D default), "wale" (2D default), "vreman" and "smagorinsky", with filter width V^(1/d) on every cell type (in 3D corrected for anisotropy by default, below). The SGS stress, heat flux (Pr_t, 0.9) and species fluxes (Sc_t, 0.9, Hirschfelder–Curtiss with correction velocity) are added at faces, none on walls; ν_t enters the time step. New output MU_T. Single gases and mixtures, MPI-bitwise against serial.
  • Hybrid KEP + Ducros flux (#205): [numerics] convective_flux = "hybrid" uses the kinetic-energy- and entropy-preserving central flux of Kuya, Totani & Kawai (2018) and switches to the Riemann solver only where a compression-only Ducros sensor fires ([numerics.hybrid] sensor_threshold, 0.65). Perfect gases, mixtures and double flux.
  • Kinetic-energy budget (#203): [integrals] budget = true writes the convective, viscous and SGS kinetic-energy rates and the numerical dissipation eps_numerical, so each run shows whether the model or the scheme dissipates; explicit LES needs eps_numerical ≤ 0.5 eps_sgs. The budget does not change the solution.
  • Dynamically thickened flame (TFLES) (#215): [les.combustion] model = "tfles", thickening F = max(1, n_res Δ / δ_L) inside the flame only, Charlette efficiency (β = 0.5) with Colin's u′, chemistry scaled per cell through each Strang reactor's time. New outputs TF_F, TF_E, TF_OMEGA. A 1D stoichiometric H2/air flame keeps its consumption speed within 0.2% of Cantera at 2 and at 0.5 cells per δ_L (F = 7.9 and 31.6), displacement speed −1.8% / −3.5%.
  • Scotti filter width, the 3D default (#223): [les] filter_width = "scotti" (Scotti, Meneveau & Lilly 1993) scales V^(1/3) by each cell's anisotropy factor (1 on cubes; "volume" restores V^(1/3), the 2D default). In the channel it brings Re_τ to +0.7% at 395 (64³) and +1.6% at 590 (96³), against +2.6% / +4.1% with V^(1/3), and C_f to +0.9% / +2.9%.
  • Dynamic constant (#223, opt-in): [les] dynamic = true, a global Germano–Lilly procedure (Favre-weighted, rank-count independent; column les_C). It gives the best CBC spectra at 128³ (0.060 / 0.054 decades) but C = 1.6 / 1.4 rather than the 1.8 that fits 64³, and Re_τ +4.0% in the 395 channel, worse than the static constant.
  • Partially stirred reactor (PaSR), experimental (#224): [les.combustion] model = "pasr" scales the filtered reaction rates by κ = τ_c / (τ_c + τ_mix) (output PASR_KAPPA), for non-premixed flames. Against a DNS of an H2/N2–air diffusion flame in decaying turbulence it brings the heat release from +16.6% (quasi-laminar) to −10.7% at Δ = 0.27 mm, but overcorrects at 0.53 mm (−50%, near extinction), so it is opt-in and experimental.

Validation:

  • Decaying isotropic turbulence, Comte-Bellot & Corrsin (examples/cbc_les, tools/cbc_les.py, #213): Sigma + hybrid flux at 128³ matches the measured spectra to 0.063 / 0.085 decades RMS at stations 2 / 3, with numerical dissipation 2% of the total (64³ with C = 1.8: 0.058 / 0.063). Without a model energy piles up (0.18–0.38 decades). With HLLC the scheme dissipates instead: 84% without a model and still 46% with Sigma, so upwind fluxes do not give explicit LES.
  • Channel, Re_τ = 395 (examples/channel_les, #214, #223): 64³ wall-resolved LES (Δx⁺ 39, Δz⁺ 19) with Sigma and the hybrid flux, against MKM: Re_τ +0.7% and C_f +0.9% with the default Scotti filter width (+2.6% and +4.8% with V^(1/3), where U⁺ is within 3% for y⁺ 30–300, u_rms, w_rms and −u′v′ within 10%, and numerical dissipation 3.5% against 12.6% SGS). Without a model Re_τ is +7.1%; with HLLC −19%.
  • Channel, Re_τ = 590 (examples/channel_les/input_590.toml, #230): 96³ (Δx⁺ 38, Δz⁺ 19) with Sigma, Scotti width and the hybrid flux: Re_τ +1.6%, C_f +2.9%, U⁺ within 1.5% at y⁺ 30 and 100, numerical dissipation 0.20 of the SGS dissipation. With V^(1/3) Re_τ is +4.1%, without a model +7.8%.
  • Premixed H2/air flame in decaying turbulence (examples/flame_turbulence, tools/flame_turbulence.py, #222): TFLES + Sigma at Δ = 0.4, 0.8 and 1.6 δ_L (F = 2, 4, 8) against a DNS at δ_L/10 (u′ = 2.1 S_L, Ka = 2). With the default Colin estimate of u′ the turbulent flame speed is within 7% of the DNS for Δ ≤ 0.8 δ_L (S_T/S_L 1.056 and 1.053 against 1.132), with numerical dissipation at most 0.20 of the SGS dissipation; quasi-laminar and no-model runs are erratic in time and 15–21% numerically dissipated at F = 8. New opt-in [les.combustion] subgrid_velocity = "eddy_viscosity" (u′ = C_u ν_t/Δ) for coarser meshes; its C_u = 28 comes from this case and is not validated.

Channel flow at Re_τ = 180 within 2% of DNS

examples/channel_retau180 (#216) now runs with the hybrid flux: Re_τ 180.8 against MKM's 178.1, C_f +2.0%, u_rms, v_rms, w_rms and −u′v′ peaks within 1.8%, numerical dissipation 0.9% of the total. The 7% C_f deficit of 0.6.0 was HLLC's upwind dissipation (6.8% of the total at Δx⁺ 12, Δz⁺ 6), not a bug: halving Δx and Δz recovers it with HLLC too (−0.8%) at about 5× the GPU time.

Numerics

  • 3D TENO stable next to walls and on jittered meshes (#232, #126, #229): large stencils must have a Lebesgue constant of at most 4. Near-degenerate stencils at walls, symmetry seams and on jittered meshes put up to 73% of a face value's weight on the cell across the face, which was anti-dissipative. The largest growth rate of the linearized operator drops from up to 3.5 c/L to ≤ 1e-7 c/L on stretched prisms and Kuhn tetrahedra with walls and on jittered periodic prisms; jittered prisms with walls keep ≤ 1.4e-3 and AR-25 tetrahedra at order 6 with side walls 0.022. Accuracy keeps its order of convergence (errors −66% to +22% depending on cell type and order). Cost: larger stencils on tetrahedra and pyramids (step time +9–29% on CPU) and 5–9× slower order-6 setup on tetrahedra, pyramids and mixed meshes (cached by cache_file).

Boundaries

  • Synthetic turbulent inflow (#221): nscbc_inlet can inject turbulence from a digital filter (Klein, Sadiki & Janicka 2003, Lund et al.'s Cholesky factor) with a prescribed mean profile, Reynolds stresses (reynolds_stress, or a profile CSV from tools/plane_average.py) and length scales, through [boundaries.turbulence]; inlet u accepts expressions. The field depends only on time, so runs are bitwise identic...
Read more

Mallard 0.6.0

Choose a tag to compare

@MatthewBonanni MatthewBonanni released this 05 Oct 05:25

Mallard 0.6.0 adds axisymmetric flows, non-reflecting boundaries, in-situ statistics, parallel HDF5 output and AMD GPUs, and switches cfl to the conventional unstructured normalization.

Changes that affect results

  • Conventional CFL normalization (#175, #173): the CFL = 1 step is now dt = 2V / Σ_f λ_f A_f (Blazek's convention, with the viscous term 4 ν Σ_f A_f² / V), for single gases and mixtures; on a uniform 2D grid dt = cfl h / (|u| + |v| + 2a). The same cfl now gives twice the time step as in 0.5.0: halve cfl in inputs written for 0.5 or earlier to keep their results. The CFL output variable halves accordingly. Every cfl in examples/, benchmarks/, the tests, the tools' defaults and the docs was halved, so the examples' CFL values changed but their results did not (restart files of shortened 2D and 3D example runs are byte-identical). Measured stability limits are 1.26–1.55 for SSPRK3 and 1.39–1.71 for RK4, depending on reconstruction and cell type; the recommended maxima are 1.2 (SSPRK3) and 1.35 (RK4), with 1.0 as a default and about 0.35 for detonations (time step). The examples keep their halved values for now.
  • 3D analytical initial conditions change (#188, #186): each tetrahedron is now integrated with a 14-point degree-5 rule, and the 3D default n_subdivisions is 1. Initialization is much faster (1M hexahedra: 58.8 s → 1.6 s on 64 threads; the 128³ Taylor–Green box: 2m 46s → 4.2 s), but 3D initial states are not bitwise equal to 0.5.0: smooth fields differ at round-off (Taylor–Green ≤ 2.2e-14 relative), while discontinuous initial conditions differ in cells cut by the discontinuity, by up to 2.9% in explosion_3d and 9.2e-4 of the maximum RHOE in sedov_3d, with mass and energy errors comparable to before. 2D initial states are byte-identical. The 128³ Taylor–Green example needs about 27 KB of host memory per cell and fits on 4 GPUs.
  • Double-flux mixtures with HLL, Rusanov and HLLC (#187): wave speeds now use each side's own frozen (gamma, e0) (see below), which changes their results; single gases and conservative mixtures are unchanged.
  • The faster TENO-E setup (#181) changes results at round-off (≤ 6e-13 relative after 2 steps); Shu–Osher amplifies such differences (see Known limitations).

Physics and boundaries

  • Axisymmetric (r–z) flows (#184): [physics] axisymmetric = true in 2D builds, a revolved finite-volume formulation that conserves mass, axial momentum, energy and species discretely and is exactly well balanced at rest; TENO-E fits r-weighted moments and keeps design order up to the axis (manufactured solution: 3.1, 4.5, 5.4 for TENO3–5). Mixtures, chemistry and MPI work unchanged, and planar runs are bitwise unchanged. Validated on Hagen–Poiseuille flow (second order: 1.3e-3, 3.2e-4, 8.0e-5 of U on 16/32/64 cells), Noh (shock radius within 1%), Sedov (shock 0.6% ahead of exact), a sphere at Re 100 / 150 (Cd 1.100 / 0.900 vs 1.087 / 0.889 from Clift, Grace & Weber; separation 126.5° / 120.2° vs 126.5° / 120.4°) and a counterflow diffusion flame (peak T within 0.7% of Cantera).
  • Characteristic (NSCBC) boundaries and sponge layers (#176): nscbc_outlet and nscbc_inlet (Poinsot & Lele with transverse terms, single gases and mixtures, double flux) and [[sponges]]. A normal acoustic pulse reflects 0.1–0.2% with MUSCL (1% with TENO5), against 0.6–34% for extrapolation; partially reflecting outlets match the theoretical reflection coefficient within 3%. A vortex at M = 0.25 leaves with an RMS pressure error of 0.05 Δp_vortex (1.2–1.3 for Poinsot & Lele's original condition), and an H2/air flame blows out through the outlet (examples/flame_outflow) with the pressure within 0.7% of its target. Existing boundary conditions are bitwise unchanged.
  • Gas-mixture Roe and RHLL (#187): Roe (Glaister; Shuen, Liou & van Leer) and RHLL now accept gas mixtures, with exact upwinding at contacts between different gases. The 1D CJ detonation holds D_CJ within +0.11% / +0.01% at 10 / 20 cells per induction length with either, like HLLC. With double flux, each side's own frozen (gamma, e0) now sets the wave speeds of both fluxes of a face; before, HLL-type fluxes perturbed p by 0.1% at contacts. This also changes HLL, Rusanov and HLLC results with double flux (single gases and conservative mixtures are unchanged).
  • Fractional reaction orders below 1 (#177): the mass-action law is regularized below 1e-12 kmol/m³, so mechanisms such as Westbrook–Dryer propane (previously NaN Jacobians before ignition) now integrate; ignition delays match Cantera within 4.5e-5 at the V1 conditions.
  • Inflow compositions as expressions (#167): an upt inlet's X or Y can vary along the boundary (x, y, z expressions), e.g. for stratified fuel/air layers.
  • viscosity_model = "power_law" (#170).

Output and post-processing

  • In-situ statistics and probes (#169): [statistics] keeps time-weighted means and covariances of the flow and mass fractions on the device, and [[probes]] writes point and line probes as CSV. Restarts continue both bitwise on any rank count, and solutions are bitwise identical with statistics on or off; 9 statistics sampled every step cost 2.8% on an A100 (1024² cells). tools/plane_average.py averages over homogeneous directions.
  • Parallel HDF5 output with XDMF (#179): [[write_data]] format = "hdf5" writes the mesh once and two files per snapshot that ParaView opens, byte-identical across rank counts. On 2.1M hexahedra it writes half the bytes of VTU (689 vs 1288 MB) and is 3.2× faster on one rank; on a single local disk VTU pieces are faster beyond about 4 ranks, while HDF5 is what scales on a parallel file system.
  • Q (Q-criterion) and VORTICITY output variables (#139).
  • tools/animate_cartesian.py animates Cartesian runs with millions of cells (#178), such as the 4096² Riemann run.
  • New [integrals] columns (velocity_squared, vorticity_squared, density_squared, temperature, temperature_squared), and monitor files created by a run started from a restart file now get their header (#170).

Performance and platforms

  • AMD GPUs (#180): Mallard builds, passes its 2D, 3D and MPI suites and is tuned on AMD Instinct MI355X (Kokkos HIP, -Werror clean). One MI355X against one A100: 2D TENO5 on 1M quads 3.4× faster (5.36 vs 18.4 ms/step), MUSCL 2.5×, 3D TENO5 on 1M hexahedra 2.0×, the reactive shock tube 2.6×, chemistry 1.3–1.9×. Build notes are in the README.
  • TENO-E setup 5.6× faster (#181, #140): 372 → 66 s on 1M hexahedra (8 threads), bitwise identical stencils except round-off (≤ 6e-13 relative in restart files after 2 steps); setup of the 64³ 3D examples drops from about 115 s to 18–21 s. cache_single_precision = true (#151) halves the cache (19 instead of 36 KB per cell for order-5 hexahedra); it is off by default.
  • GPU performance set (#168): benchmarks/perf/ with A100 (and MI355X) baselines and tools/perf_suite.py to run and compare them (fails when slower than baseline by more than 5% and the run-to-run spread).
  • The float build compiles and passes again (#185, #190).

Examples

  • Decaying isotropic turbulence (examples/isotropic_turbulence, #170): the Johnsen et al. (2010) DNS at Mt = 0.6, Re_λ = 100 on 64³–256³. At 256³, filtered to 64³, kinetic energy is within 2.1% (mean) and enstrophy within 2.4% of Johnsen et al., and within 2.3% and 0.3% of Subramaniam et al. (2019); 128³ and 256³ agree within 0.3% in kinetic energy.
  • Triple flame (examples/triple_flame, #182): H2:N2 against air with inlet mixing layers of 4 and 12 thermal thicknesses (Ruetsch, Vervisch & Liñán 1995); U_F / S_L = 2.31 and 2.46 rises with the layer thickness as there, but higher, because the hydrogen tip burns at 1.24–1.30 S_L. The comparison with the one-step reference is qualitative.
  • Flame–vortex interaction (examples/flame_vortex, #182): 11 vortex pairs on a rich H2/air flame mapped onto Poinsot, Veynante & Candel's (1991) diagram (no effect, wrinkled, pocket); none quenches, consistent with no radiative losses being modeled. 1D flame speeds are within −1.5% to +0.2% of Cantera.
  • Turbulent channel flow at Re_τ = 180 (examples/channel_retau180, #172): DNS at bulk Mach 0.2 against Moser, Kim & Mansour (1999): Re_τ = 172.6 vs 178.1 (Cf 7% low), u_rms⁺ peak 2.88 vs 2.66, v_rms⁺, w_rms⁺ and −u′v′⁺ peaks within 1–4%. It adds [source] mass_flow (constant mass flow by a uniform body force, single gases and mixtures) and [mesh] stretching (tanh clustering of generated meshes).
  • Counterflow diffusion flame (examples/counterflow_diffusion, #171): H2/N2 against air from 3D opposed jets, against Cantera at matched strain rate: peak temperature within 2% up to 90% of Cantera's extinction strain (+0.49% at 2 m/s), and extinction 0–9% above Cantera's strain.
  • Axisymmetric poiseuille_pipe, noh_axisymmetric, sedov_axisymmetric and sphere_axisymmetric, and flame_outflow (above).

Known limitations

  • Runs with NSCBC boundaries do not restart bitwise: the per-face boundary state is re-initialized from the solution (#189). Every other run restarts bitwise.
  • With the default low-Mach correction, extrapolation outlets reflect 15% (MUSCL) to 34% (TENO5) of a normal acoustic pulse; use nscbc_outlet where reflections matter.
  • With gas mixtures, RHLL + TENO5 can leave mass fractions slightly negative (N2 = −2e-10 in three cells of a shock tube), because species bounds are enforced at face points only.
  • TENO orders 4 and 6 can grow modes next to walls on mildly stretched prisms and on strongly stretched tetrahedra (#126); prefer orders 3 or 5, or MUSCL, on such meshes.
  • The Mach 3 sphere develops a carbuncle on the axis where two symmetry planes meet with RHLL and TENO5 (#80).
  • Shu–Osher amplifies round-off through TENO's discrete ...
Read more

Mallard 0.5.0

Choose a tag to compare

@MatthewBonanni MatthewBonanni released this 03 Oct 20:10

Mallard 0.5.0 makes finite-rate chemistry fast on GPUs and makes TENO-E robust on stretched cells.

Chemistry on GPUs

  • Team-per-cell kernels (#119): from 16 species, each cell is integrated by a warp, with a sparse LU from 30 species and cells ordered by cost; small mechanisms keep one thread per cell. The choice is fixed per mechanism and build, so results stay independent of the rank count, the queue order and restarts. On an A100 (reactor benchmark, cells per second at dt = 1e-8 s):
Mechanism 0.4.0 0.5.0
h2o2 6.75M 6.53M
GRI-3.0 (53 species) 121k 525k
n-dodecane 11.7k 308k
n-hexane (1268 species) 3.9 5.13k

With every cell reacting, a GRI-3.0 step of the full solver is 2.6× faster. Large-mechanism ignition delays on the GPU match Cantera within 6e-6.

  • Wide teams for stiff cells (#128): on GPUs, cells that needed 16 or more sub-steps in their last call get 8 warps (16 from 512 species) on a second stream; results are bitwise identical whatever the team width. At dt = 1e-6 s n-dodecane runs at 48.8k cells/s on an A100 (1.7× faster, 2× 16 EPYC cores) and n-hexane at 581 (3.1× faster).
  • Benchmark mode with recorded A100 and CPU baselines (benchmarks/baselines.csv, docs/design/chemistry.md); the CPU baselines are now measured correctly (#124).
  • Fused Strang half steps are available but off by default (they make results depend on when output is written).

Reacting-flow examples

  • 2D cellular detonation (examples/detonation_2d, #130): 2H2–O2–7Ar at 6.67 kPa from the ZND solution with seeded fresh-gas pockets, at 10 cells per induction length (as Oran et al. 1998); the front holds the CJ speed within 0.01% while triple points form and coarsen toward the published 3 cm cell width. A new P_MAX output records each cell's peak pressure, the numerical soot foil, and survives restarts (#118).
  • Lean H2/air flame in 2D (examples/flame_2d, #131): φ = 0.4 at 700 K, mixture-averaged against unity Lewis transport; the planar flame speed matches Cantera within 0.04% (mixture-averaged) and 0.6% (unity Lewis) on the same mesh, and the mixture-averaged front shows superadiabatic burnt gas behind its bulges.

Numerics

  • TENO-E on thin cells (#120): stencil cells are ranked by distance in a metric of the local mesh spacing where the mesh is clearly stretched. This removes growing grid-scale modes on thin tetrahedra and prisms (aspect ratios up to 25, orders 3–6), e.g. on boundary-layer prism meshes; regular hexahedral, tetrahedral, prismatic and pyramid tilings keep exactly their stencils.
  • 2D cell centroids accurate to round-off (#127): the centroid formula lost accuracy far from the origin on fine meshes, which could split TENO's equidistant stencil candidates differently in different cells; a 1D problem on a 2D strip now stays one-dimensional to round-off. 2D results change at round-off level.
  • The CUDA build is warning-clean under -DMallard_WARNINGS_AS_ERRORS=ON (#125).
  • The non-reacting solver's speed is unchanged from 0.4.0 (A100: 2D TENO5 18.3 ms/step on 1M quads, MUSCL 3.88 ms/step, 3D TENO5 398 ms/step on 1M hexahedra).

Known limitations

  • TENO orders 4 and 6 can grow modes next to walls on mildly stretched prisms and on strongly stretched tetrahedra (#126); prefer orders 3 or 5, or MUSCL, on such meshes until it is fixed.
  • With very large mechanisms and only a few igniting cells, the A100 is still underused: n-hexane (1268 species) at dt = 1e-6 s runs at 0.66× the rate of 16 CPU cores.
  • RHLL can amplify round-off transverse velocity where the velocity jump across a face is small (documented in docs/numerics/overview.md).
  • The Mach 3 sphere develops a carbuncle on the axis where two symmetry planes meet with RHLL and TENO5 (#80).

Full list of changes: v0.4.0...v0.5.0

Mallard 0.4.0

Choose a tag to compare

@MatthewBonanni MatthewBonanni released this 03 Oct 17:25

Mallard 0.4.0 completes the distributed-memory solver and adds finite-rate chemistry: reacting Euler and Navier–Stokes flows with arbitrary Cantera mechanisms.

Finite-rate chemistry

Following the design in docs/design/chemistry.md (#52), milestones 1–9 (#94, #96, #102, #107, #113 (with #110), #114–#116):

  • Mechanisms: Cantera YAML read at run time (yaml-cpp): species, NASA-7/9 thermodynamics, elementary, three-body, falloff (Lindemann, Troe, SRI), PLOG and Chebyshev reactions. No recompilation per mechanism.
  • Mixtures: thermally perfect multicomponent gas, species transported with the flow; Riemann solvers on a fixed-width face state (gamma, e0 surrogates); species fluxes by mass-flux upwinding; MUSCL and TENO-E for mixtures with bound-preserving mass fractions; double flux as an input option for oscillation-free material interfaces.
  • Chemistry: analytical Jacobians; an adaptive Rosenbrock (RODAS) integrator per cell, always in double precision; Strang splitting with the flow; a [chemistry] input section; MallardReactor, a 0D constant-volume reactor tool.
  • Transport: species transport fits ported from Cantera (agreeing to 2e-12), mixture-averaged, unity-Lewis and constant-Lewis models, species diffusion with a correction velocity, enthalpy diffusion.
  • Validation against Cantera and the Shock and Detonation Toolbox on the same mechanisms:
    • 0D ignition delay within 1.4e-4 (H2/air) and 3e-6 (CH4/air); equilibrium temperature within 0.1 K.
    • 1D CJ detonation (2H2–O2–7Ar, 6.67 kPa): front speed within 0.01% of D_CJ at 20 cells per induction length; von Neumann pressure reproduced.
    • Reactive shock tube: detonation front position converged to 0.03 mm.
    • Premixed laminar flame speed: H2/air at φ = 0.6–1.4 within 0.85% of Cantera (mixture-averaged and unity Lewis); CH4/air at φ = 1 within 0.93%.
  • Reacting runs are bitwise identical on any number of MPI ranks and across restarts; the single-gas solver is byte-for-byte unchanged.

Distributed memory: complete

  • Periodic boundaries in distributed runs no longer gather any global data (#117): periodic nodes are matched by hashing their coordinates to owner ranks, so no rank holds more than its share of any part of the mesh.
  • Periodic zone pairs for mesh files (#81): [[periodic]] zones = [A, B], translation = [...] for Gmsh and HDF5 meshes, with clear errors for mismatched seams.
  • TENO setup memory and caches (#89): stencils stored at their actual size (GPU memory −15–48%, host setup peak about 3× lower; 3D hexahedra at order 5 need 41 KB per cell), and the stencil cache now works on any number of ranks with one file per rank.
  • Multi-node runs with GPU-aware MPI over InfiniBand (GPUDirect RDMA) verified: 16M cells scale 1.91× from 8 to 16 A100s across 4 nodes, and weak scaling is 93% efficient at 16 GPUs.
  • Dynamic load balancing was prototyped and measured and is not adopted: after the TENO kernel work in 0.3.0 the remaining imbalance is too small to pay for repartitioning (docs/design/mpi.md, §10, #97).

Numerics

  • TENO stencils (#109): ill-conditioned central stencils are rejected using a bound on the reconstruction's Lebesgue constant, and candidate cells near walls are gathered completely. This fixes TENO4 and TENO6 on jittered and mixed-element 3D meshes (maximum error up to 25× lower) and TENO6 next to symmetry walls (#68). Results change in boundary cells of 2D meshes and on tetrahedral, prismatic and mixed 3D meshes; hexahedral meshes are unchanged.
  • MUSCL on tetrahedra (#111) uses vertex-neighbour gradients; face-neighbour gradients amplified small perturbations.
  • Single precision builds compile warning-clean and are tested (#95); several tolerance bugs that only bit in single precision were fixed.

Validation cases

  • Mach 3 flow over a sphere: bow-shock standoff Δ/R = 0.226 against Billig's 0.205, i.e. within half a cell (#77).
  • Sedov–Taylor blast in 3D: shock radius within 1.1% of the exact similarity solution (#76).
  • Flow past a sphere at Re = 300: Strouhal number 0.133, drag 0.666 and lift 0.070, within the published range (Johnson & Patel 1999 and others) (#106).
  • Taylor–Green vortex in the full periodic box at 128³: kinetic energy within 2.6% of the 512³ spectral DNS to t = 20 (#100).

Fixes and engineering

  • Boundary face lists are cleared when the halo is rebuilt at startup; a stale entry could cause an out-of-bounds write in distributed TENO runs with Dirichlet or average-pressure boundaries (#98).
  • Strict compiler warnings are errors in every CI job, with a Clang job (#83, #85–#87, #112); nightly AddressSanitizer and UndefinedBehaviorSanitizer runs (#99).
  • Unused Toro star-pressure estimators removed (#101).

Known limitations

  • TENO-E can amplify perturbations on cells of high aspect ratio (thin prisms and tetrahedra); use MUSCL on boundary-layer meshes for now (a fix is in progress).
  • Chemistry on GPUs is latency-bound in this release; faster kernels are in progress.
  • The Mach 3 sphere develops a carbuncle on the axis where two symmetry planes meet with RHLL and TENO5 (#80).
  • Gmsh files are read whole by every rank; convert large meshes to HDF5 for distributed runs.

Full list of changes: v0.3.0...v0.4.0

Mallard 0.3.0

Choose a tag to compare

@MatthewBonanni MatthewBonanni released this 03 Oct 00:50

Mallard 0.3.0 adds three-dimensional simulation, periodic boundaries and a distributed-memory (MPI) solver that scales across GPU nodes.

3D

  • -DMallard_DIM=3 builds a 3D solver for tetrahedra, hexahedra, prisms and pyramids in any mix (#35–#37, #39). 2D and 3D builds live side by side.
  • Every numerical method carries over: Riemann solvers, MUSCL, TENO-E up to sixth order (#40), boundaries, viscous fluxes, Gmsh 3D meshes, VTU and restart output.
  • New examples: spherical explosion (#45) and the Taylor–Green vortex at Re = 1600 against the 512³ spectral DNS (#60), with an [integrals] monitor for kinetic energy, enstrophy and dilatation that uses the TENO reconstruction gradients (#72).

Periodic boundaries

  • [mesh] periodic = ["x", "y", "z"] on generated box meshes (all cell types), in 2D and 3D, for every scheme including TENO stencils that wrap across the seam, serial and distributed (#70, #75).

Distributed memory (MPI)

  • -DMallard_ENABLE_MPI=ON, then mpirun -n N Mallard -i input.toml (#33, #38, #41).
  • No rank holds the global mesh: a parallel HDF5 mesh format with the mallard-mesh-convert converter, distributed face matching, partitioning, cell migration and halo growth (#53, #54, #62, #66). Setup memory per rank now falls as 1/ranks; 64M quads set up on 16 ranks at 3.3 GB per rank.
  • Partitioning by Hilbert curve or by the dKaMinPar graph partitioner (-DMallard_ENABLE_KAMINPAR=ON, #46); GPU-aware halo exchange (-DMallard_GPU_AWARE_MPI=ON).
  • Output from every rank (.pvtu), and restart files that do not depend on the rank count (#41).
  • Results are bitwise identical on any number of ranks and threads (#49, #63).
  • Multi-GPU performance: TENO's shock-cell pass split into finer parallel work and the halo exchange overlapped with computation (#57, #65, #67). 2D TENO5 on A100s with GPU-aware MPI over InfiniBand: 4M cells run 1.7× faster on 16 GPUs than on 8, and weak scaling to 16 GPUs is 93% efficient.

Numerics

  • Low-Mach correction of the convective flux (Thornber et al. 2008), on by default with a cutoff Mach number [numerics] low_mach_cutoff = 0.1 (#78). It brings the Taylor–Green vortex kinetic energy onto the DNS and leaves the shock validation cases and the cylinder (St, Cd) unchanged.
  • Second-order viscous fluxes on triangles next to transmissive boundaries; quadratic vertex gradients (#55).
  • Deterministic flux accumulation without atomics (#49).

Usability

  • Clear console output: a build and setup summary, an aligned progress table (time step, throughput, ETA, minimum density and pressure, maximum Mach number, TENO troubled cells) and an end-of-run summary; input errors are one line naming the key and value (#88).
  • -DMallard_WARNINGS_AS_ERRORS=ON builds Mallard's own code with a strict warning set (#83).

Documentation

  • References for every method and validation dataset, with DOIs (#73, #90), and a CITATION.cff. This release is archived on Zenodo.
  • Design notes for MPI and for finite-rate chemistry (#32, #52).

Known limitations

  • 3D TENO uses about 50–80 KB of GPU memory per cell, so one A100 holds roughly 1M 3D cells at fifth order (#58).
  • The Mach 3 sphere develops a carbuncle on the axis where two symmetry planes meet with RHLL and TENO5 (#80); HLL is unaffected.
  • Gmsh files are read whole by every rank; convert large meshes to HDF5 for distributed runs.
  • The TENO setup cache works for serial runs only.

Full list of changes: v0.2.0...v0.3.0

Mallard 0.2.0

Choose a tag to compare

@MatthewBonanni MatthewBonanni released this 02 Oct 14:52

Mallard 0.2.0 is the first release built with agentic development. It turns the 0.1 prototype into a validated, high-order 2D solver that runs on CPUs and NVIDIA GPUs.

Documentation and gallery: https://matthewbonanni.github.io/mallard/

Highlights

  • TENO-E (Liang, Shyy & Fu 2025), orders 3 to 6 on triangles, quadrilaterals and mixed meshes: k-exact least squares, sector stencils, characteristic stencil selection, adaptive cutoff, boundary mirror cells, optional bound preservation and a stencil cache. Design order verified on triangles and quads.
  • Navier–Stokes: viscous fluxes with constant or Sutherland viscosity; adiabatic, isothermal, heat-flux and moving walls; wall force monitors. Validated against exact solutions and vortex shedding behind a cylinder at Re = 100 (St = 0.164, mean Cd = 1.36).
  • Boundary conditions: characteristic far field, inflow, pressure outlets (local or area-averaged), symmetry, time-dependent Dirichlet states from expressions in x, y and t, and partial-zone assignment with where.
  • Riemann solvers: Roe and the carbuncle-free rotated-hybrid HLL–Roe (RHLL), in addition to Rusanov, HLL and HLLC.
  • Meshes and I/O: Gmsh 2.2 and 4.1 reader, surface output, exact restarts, .pvd time series.
  • GPUs: builds and runs with CUDA (tested on A100); the TENO kernel is templated on degree and split into smooth and troubled passes.
  • Examples: 2D Riemann problem on triangles and quads, double Mach reflection, the Daru–Tenaud viscous shock tube (matches the grid-converged reference), Shu–Osher, Sod, wedge, cylinder.

Fixes

The 0.1 code had several bugs that affected results, among them:

  • Builds were accidentally single precision.
  • Transmissive boundaries made the boundary shocks of the 2D Riemann problem run almost twice too fast.
  • A sign error in the TRRS wave-speed estimate corrupted HLL/HLLC.
  • First-order reconstruction read out of bounds, and quad centroids were wrong.
  • Integer TOML values were silently ignored.

See the merged pull requests #20–#30 for details.

Testing

The test suite grew from 53 tests (3 failing) to more than 200, all passing, with CI on the Serial and OpenMP backends.

Dependencies

Kokkos 5.2.2, toml11 4.4.0, googletest 1.18, exprtk; Kokkos Kernels is no longer required.