Repository navigation
Mallard 0.8.0
Mallard 0.8.0 adds curved high-order boundaries, which bring TENO-E orders 3–6 to their design accuracy at curved walls, where straight-sided faces did not converge. It sets up 3D TENO 6–9.5× faster on the GPU and fits about 15% more cells on one (twice as many with opt-in single-precision tables), scales 3D TENO5 to 91% weak efficiency on 16 GPUs, makes reacting mixtures 2× faster per step, and adds SIMPLER splitting, Soret diffusion and optically thin radiation.
Changes that affect results
- RHLL blends toward HLL across faces that carry a shock (#248, fixes #80):
w F_rotated + (1 − w) F_HLL(n)withw = min(p_l/p_r, p_r/p_l)^3, so RHLL results change wherever a face carries a pressure jump; contacts and shear layers keep the rotated flux, and faces with equal pressures are bitwise unchanged. In the examples:shu_oshermax |Δρ| 1.2e-5;double_machat 240×60 shocks shift by a cell locally (mean |Δρ| 0.04; the kink in the RHLL Mach stem is gone);mallardon a coarse mesh mean |Δρ| 7e-3. Other Riemann solvers are unchanged. - Multi-jagged partitions are the default (#261): results are bitwise identical on any rank count, but distributed runs get different partitions, so their per-rank output pieces and per-rank TENO cache files (recomputed on first use) differ from 0.7.0's.
[parallel] partitioner = "hilbert"restores the 0.7.0 partitions. - Gmsh high-order boundary elements now curve their faces (#263): 0.7.0 read elements of order 2 and 3 by their corners; now
line3/line4(2D) andtriangle6…quadrangle16(3D) boundary elements give curved faces by default.[mesh] curved_geometry = falserestores straight faces. Linear meshes, and every example, are bitwise unchanged (10 2D + 2 3D examples on CPU, 9 2D + 2 3D on an A100). - 3D TENO tables on macOS (arm64) change at round-off (#247): the TENO setup is now compiled without FMA contraction, so its tables in macOS host builds differ from 0.7.0's at round-off; on x86 and CUDA they are bitwise identical to 0.7.0's.
- TENO cache files are recomputed (#247, #265): the cache format is now
MALLARD-TENO-10(3D caches hold the cells' moments instead of their smoothness matrices); older caches are detected and rebuilt. - Everything else is opt-in or bitwise identical: CUDA graphs and NCCL (#217), the MUSCL and halo-exchange changes (#239, #258), the GPU TENO setup and its smaller memory (#247, #265), mixture performance (#264, except the opt-in
chemistry.C_reg), chemistry load balancing (#255, #262), the large-mechanism solves (#246) and the conversion-warning series (#267, #269–#276). SIMPLER coupling (#235), Soret diffusion (#243) and radiation (#226) change nothing unless enabled.
Curved high-order boundaries
Design and results: docs/design/curved_boundaries.md (#263, closes #162).
- Curved faces from the mesh or from an analytic shape: Gmsh 2.2/4.1 high-order boundary elements of order 2 or 3 (kept by
mallard-mesh-convertin HDF5), or[[mesh.curved]]projection onto a circle (2D), sphere or cylinder (3D) on any mesh. Curved faces get mapped quadrature points with per-point normals and Jacobians, closed exactly so gas at rest stays at rest to round-off; the cells touching them get exact volumes, centroids, TENO moments and initial averages. In 3D, faces sharing a curved edge (prism-layer sides, symmetry planes meeting the wall) are blended from their edges. Any rank count gives bitwise identical results.make_cylinder_mesh.py --order 2|3,make_sphere_mesh.pyand the newmake_annulus_mesh.pywrite high-order boundaries. - TENO at curved walls: one-sided stencils at curved walls and symmetry faces (
curved_mirrors = false; mirroring across chords does not converge), and the central fits of cells reaching them capped at degree 3 (curved_wall_degree), since one-sided degree-4/5 fits grow exponentially there. - Steady vortex between circular walls (
tools/curved_convergence.py, quadrilaterals, n_r = 8–64): subsonic (M = 0.5) curved L1 rates 3.03 / 4.89 / 4.90 / 4.92 for orders 3 / 4 / 5 / 6 (Linf 2.86 / 3.99 / 3.95 / 3.65), L1 at n_r = 64 4.2e-8 / 3.5e-10 / 3.2e-10 / 3.8e-10; with straight walls none converges (L1 2e-2 to 8e-2). Triangles: L1 rates 3.45 / 4.38 / 4.91 / 4.91. Supersonic (M = 2.25): 3.64 / 4.86 / 5.00 / 5.57, against O(0.1–1) errors with straight walls. Gmsh P3 walls match the analytic projection to three digits. - Cylinder at Re = 100: curving the walls moves C_D by 0.1–0.3% (128×64 TENO3: St 0.1633, C_D 1.3671 → 1.3643; 384×128 TENO3: 0.1648, 1.3637 → 1.3629). On 384×128, TENO5 with straight walls is unstable (C_D fluctuations ±2.9); with curved walls it runs: St 0.1648, C_D 1.3614.
- Mach 3 sphere (810k tetrahedra, quarter box): MUSCL standoff Δ/R 0.2211 straight / 0.2212 curved, pressure C_D 0.9716 / 0.9720 (Billig 0.205); TENO5 with curved walls, which straight walls cannot run here: 0.2213, C_D 0.9790.
- Fix: the 3D TENO host retry wrote compacted cells out of bounds (segfault on the Mach 3 sphere mesh).
Performance and scaling
Details: docs/design/performance.md.
- GPU-resident 3D TENO setup (#247, #140): the per-cell setup (stencil search, moments, QR pseudo-inverses, Lebesgue bound, smoothness indicators) runs one team per cell on GPUs, with tables bitwise identical to the host's. 3D TENO5 on one A100: 1M hexahedra 43.5 → 4.5 s (9.5×), 998k tetrahedra 123.2 → 20.0 s (6.2×). This makes restarts on another rank count and re-setup after repartitioning cheap;
TENO::rebuild_cellsrecomputes a subset of cells. - 3D TENO device memory (#265, fixes #58): cells keep their central moments instead of their smoothness matrices, troubled-cell passes work in bounded rounds (
troubled_capacity), and the setup peak is no higher than the steady state; bitwise identical. Per cell on an A100-80GB: hexahedra TENO5 47.3 → 39.8 KB, tetrahedra TENO5 58.7 → 51.2 KB, hexahedra TENO6 129.6 (setup peak) → 77.9 KB. TENO5 fits ~2.1M hexahedra / 1.65M tetrahedra per A100 (before: ~1.8M / 1.45M); a 125³ Taylor–Green vortex that ran out of memory now runs. Opt-insingle_precision_tables(3D, single gas) keeps the pseudo-inverses in float: 25.0 / 30.6 KB per cell, ~3.4M / 2.75M cells, another 3–8% faster; differences below 1e-7 of each field's range on Taylor–Green, up to 6.9e-3 in shock cells of a blast. - CUDA graph per step (#217,
[run] cuda_graphs, default on in CUDA builds): the time-step reduction, halo exchanges and stages are captured once and replayed, with dt kept on the device. Small MUSCL cases: −20% per step at 4k cells (286 → 230 µs), −25% at 8k cells on 2 GPUs, −5% at 262k cells. - NCCL halo exchange (#217,
Mallard_ENABLE_NCCL,[parallel] halo_exchange = "nccl" | "mpi"): stream-ordered exchanges and dt reduction, with no host synchronization per exchange, so multi-GPU steps can run inside the graph. - MUSCL reconstruction fused per cell and overlapped with the halo exchange (#239): from 2^17 cells per rank; −12% per step on 1M cells (4.09 → 3.61 ms), −10% on 2 GPUs.
- Slimmer halo exchanges (#258): only the halo cells the kept stencils read are exchanged (TENO's halo holds 7 vertex layers for 3D TENO5). 3D TENO5 at 12 GPUs: weak scaling 85% → 88%, strong (96³) 68% → 72%.
- Multi-jagged partitions (#261): the new default partitioner cuts equal-count slabs along x, y and z, numbered along a Hilbert curve; a box of cuboid cells gets cuboid parts. 3D TENO5 Taylor–Green on A100s: weak (64³ per GPU) 91% at 16 GPUs (Hilbert 86%); strong (96³) 77% at 12 and 73% at 16 GPUs (Hilbert 72%, 72%); 2D within 1–4% of Hilbert. The setup summary reports the partition surface.
- Reacting mixtures 2× faster (#264, #257): stencil kernels loop over the stencil once with the variables innermost, and thread-per-cell chemistry keeps its work memory on the stack; bitwise unchanged. New performance case
flame_3d(221k hexahedra, propane two-step, Sigma + TFLES): 64.9 → 32.7 ms per step on an A100, 7.8× the air case (was 15.5×); h2o2 reactor 6.5 → 9.7 M cells/s at dt = 1e-8. Opt-inchemistry.C_reg = 1e-10(default 1e-12, as before) cheapens trace fuel burning out in hot gas: 24.6 ms per step, with T within 0.54 K and integrated heat release within 0.08% after 100 steps. - Large mechanisms on GPUs (#246, #146): the sparse LU factors in stages of independent pivots and solves by levels, with the dense radical core stored by columns; bitwise as before. A100, dt = 1e-6: n-hexane (1268 species) 576 → 1278 cells/s, n-dodecane 48.0k → 60.2k.
- Chemistry load balancing across MPI ranks (#255, #141; with SIMPLER #262):
[chemistry] load_balance = truemoves the integration of cells from ranks above the mean chemistry work to ranks below it; bitwise unchanged. On A100s:detonation_2d1.24× faster on 4 GPUs and 1.61× on 8 (time waiting for the slowest rank 51.6% → 8.0% of the step), theflame_turbulenceDNS 1.10× / 1.16×,detonation_3d1.04× / 1.06×.
Reacting flow
- SIMPLER balanced splitting (#235, #233):
[chemistry] coupling = "simpler"(Wu, Ma & Ihme 2019), second order and exact at steady state (a burning stirred reactor moves by 2e-16 where Strang moves by 2.3e-3).detonation_1dat CFL 1.0 and 10 cells per induction length: induction length −0.9% from ZND (Strang −25.5%), 3.7× cheaper than Strang at CFL 0.35;autoignition_2d−19% per step with the same result to 5 digits; flame speeds as with Strang. Strang stays the default. - Thermal diffusion (Soret) (#243, #158):
physics.soret = truewith Cantera's mixture-averaged thermal diffusion coefficients (matched to 5.7e-13). Change of the H2/air flame speed at φ = 0.5 / 1.0: −7.23 / −8.26% (Cantera −6.98 / −8.44%); +9% per step onflame_2d. Multicomponent transport was assessed and not implemented: 59× (h2o2) to 336× (GRI-3.0) the cost of mixture-averaged properties for a 0.1–3.6% change in flame speed. - Optically thin radiation (#226, #160):
[radiation]with the TNF workshop's Planck-mean fits for H2O, CO2, CO and CH4; new outputQRAD. S_rad/S_ad of CH4/air flames against Cantera with the same model: 0.9735 / 0.808 / 0.8507 (Cantera 0.9738 / 0.811 / 0.8509) at φ = 0.6 / 0.5 (GRI-3.0) / 0.44 (two-step). - Flame–vortex quenching (
examples/flame_vortex_quenching, #226): CH4/air at φ = 0.42 with radiation, 12 vortex pairs (r = 1–8 δ_L, u′ = 3–100 S_L), reproducing the shape of Poinsot, Veynante & Candel's (1991) quenching boundary (qualitative). Newmechanisms/ch4_bfer.yaml. - CH4/air flame speeds (
examples/premixed_flame/ch4,tools/flame_sweep.py, #241, #145): GRI-3.0, φ = 0.6–1.4, at 20 cells per thermal thickness within 1.82% (mixture-averaged) and 0.80% (unity Lewis) of Cantera.
Large-eddy simulation
- Volvo bluff-body combustor, non-reacting (
examples/volvo_bluff_body/cold.toml,tools/make_volvo_mesh.py,tools/volvo.py, #252): Sigma, hybrid flux, 2.7M hexahedra (Δ = D/32), against Sjunnesson et al.'s LDA data: recirculation length L_r/D 1.34 (experiment 1.35), centerline minimum −0.59 U_b at x/D 0.61 (−0.64 at 0.75), mean |ΔU| 0.07–0.14 U_b at five stations, St 0.285. Caveat: eps_num/eps_sgs = −0.38: the scheme feeds resolved energy back at 38% of the SGS rate, steady in every 40 ms window. The reacting case (reacting.toml, newmechanisms/c3h8_2step.yaml) is set up, not validated. - Fix:
[integrals] budget = truewithdouble_flux = truethrew "Double flux needs the cells' thermodynamics frozen" (#252).
Numerics
- RHLL across shocks (#248, above): on the Mach 3 sphere with MUSCL (796k tetrahedra), RHLL's bulge to −0.624 / stagnation pressure 10.6 becomes a steady −0.6151 / 11.93, as HLL's −0.6149 / 11.86.
- HLL/RHLL at face-aligned stationary shocks, evaluated (#266, #244): Davis wave-speed bounds, tie-breaks and smoothed wave speeds were tested and none is worth adopting: Davis bounds replace an O(0.2) carbuncle at an exactly stationary face-aligned shock with O(2–7e-2) carbuncles on every slowly drifting one, and double HLL's error at slow strong shocks. Documented in
docs/numerics/overview.md.
Build and code quality
- CUDA 13 (#260, #259): Mallard builds with CUDA 13 (versioned driver typedefs).
- CUDA build fix (#268): the radiation test failed to compile with nvcc under
-Werror. -Wconversion -Wsign-conversion(#267, #269–#276, closes #148): every conversion warning in Mallard's code is fixed (1,125–1,408 per build before, 0 after), bitwise identical, and the flags are now on, as errors underMallard_WARNINGS_AS_ERRORS. They found latent truncations:Solver::get_step()to 32 bits, mesh local indices touint8_t, LU substitution indices touint16_t. Code compiled only for the device (KOKKOS_IF_ON_DEVICE/__CUDA_ARCH__branches) is not covered.
Examples and README
- New examples:
flame_vortex_quenching,premixed_flame/ch4andvolvo_bluff_body(40 example directories); performance casesflame_3dandair_3d. - README hero (#253, #254): a 4096² Riemann problem, the 3D cellular detonation, the Taylor–Green vortex at 256³ beside its dissipation rate (peak within 0.3% of the 512³ spectral DNS) and stratified autoignition; LES has its own feature group (#251).
Known limitations
- Orders 5 and 6 are fourth order at curved walls (degree cap; design order needs a boundary-constrained reconstruction). Axisymmetric runs reject curved boundaries. The 3D TENO setup takes at most 32 mirror planes per stencil search, which faceted curved far fields (the quarter-cylinder domain of
sphere_mach3) and straight-walled spheres at TENO exceed; curved walls take no mirror images. - 3D TENO device memory is not halved bitwise: 30 of the 40 KB per hexahedron are pseudo-inverses that differ in every cell. Larger runs need
single_precision_tables, which changes results at single-precision round-off. Order-6 steps are 0.7–1.1% slower than before #265. - The GPU TENO setup is 6.2× faster on tetrahedra, not 10× (the QR fits are latency-bound under the bitwise constraint), and the 2D setup still runs on the host.
- 3D TENO5 strong scaling at 16 GPUs is 73%: the GPUs stay busy (55 of 57 ms per step at 12 GPUs) with the redundant reconstruction of halo layer 1, which grows with the partition surface.
- CUDA graphs are not used for steps the host takes part in (gas mixtures, characteristic boundaries, expressions of
t, average-pressure outlets, MPI halo exchanges); HIP and OpenMP builds have no graphs. - Reacting mixtures still cost 7.8× air per step on
flame_3d(5.9× withC_reg = 1e-10). Chemistry cost is not part of the partition weights; load balancing moves it per call instead. - SIMPLER: detonation fronts oscillate more from output to output at CFL ≥ 1 (induction-length std 5.5% against Strang's 0.6% at 10 cells); not with
double_fluxorfuse_half_steps. - Volvo LES: eps_num/eps_sgs = −0.38 (above). The reacting case needs F ≤ 5, i.e. 15–28M cells and about 2,700–5,000 A100-hours; it is not validated.
- HLL and RHLL can still grow a carbuncle at a normal shock at rest exactly on cell faces (#245, #266), and RHLL is marginal on slowly drifting ones (transverse velocity up to 1e-2).
- The other LES and 3D TENO limitations of 0.7.0 remain (Sigma constant vs resolution, v_rms peak in channels, TENO-E with the hybrid flux, TFLES at Δ = 1.6 δ_L, pyramid lattices at orders 4 and 6, extrapolation outlets, negative mass fractions with RHLL + TENO5).
Full list of changes: v0.7.0...v0.8.0