Skip to content

Fix LagrangeSpace device captures for CUDA - #594

Open
aaadelmann wants to merge 10 commits into
masterfrom
593-gh200-warnings
Open

Fix LagrangeSpace device captures for CUDA#594
aaadelmann wants to merge 10 commits into
masterfrom
593-gh200-warnings

Conversation

@aaadelmann

@aaadelmann aaadelmann commented Aug 15, 2026

Copy link
Copy Markdown
Member
  • replace class captures with a lightweight device mirror
  • add device-side mesh, DOF, and boundary helpers
  • capture field views directly in Kokkos kernels
  • eliminate GH200 host-only destructor warnings

LUMI Results

Red old master:
image

Justification

Observed behavior Most likely related change Confidence
PCG is approximately 18% faster at 16 GPUs, with the improvement decreasing at larger GPU counts e0e88e188: Poisson and differential expressions capture only the Kokkos field view instead of the complete Field object, producing smaller device-kernel closures High
Additional PCG improvement may come from reduced halo overhead e6e2387d0: halo packing and unpacking use submdspan; PCG performs halo exchanges repeatedly during its iterations Medium
FFT is approximately unchanged at 16–32 GPUs and modestly faster at 64–512 GPUs Shared infrastructure changes in e6e2387d0, including direct reallocation of overwritten field/layout storage and updated particle bookkeeping Low–medium
FEM performance is essentially unchanged, with mixed differences of approximately 1–2% 9d9387c57: Lagrange kernels capture a lightweight device mirror and the required views instead of the complete host-side object Medium; no clear performance effect
No expected performance impact Nedelec mirror, FEL warning fixes, Kokkos::pair, [[maybe_unused]], Archive initialization, and unit-test changes High

The PCG mapping is particularly strong. Before e0e88e188, every PCG Laplace expression stored a complete field:

const E u_m;

Now it stores only the device data handle:

using view_type = typename E::view_type;
const view_type u_m;

This affects meta_poisson, meta_lower_laplace, meta_upper_laplace, and meta_upper_and_lower_laplace, which are repeatedly invoked by PoissonCG. It produces smaller kernel closures and gives the AMD compiler much less host-side ownership state to carry through device kernels.

The scaling signature supports that explanation: the improvement is largest at 16 GPUs and shrinks as communication becomes a larger fraction of runtime. That is typical of a compute-kernel improvement.

For FFT, no FFT algorithm was optimized in PR593. The PrunedCC change only adds [[maybe_unused]], so it cannot explain the improvement. Likely shared contributors are:

  • Kokkos::resizeKokkos::realloc where old field/layout contents are discarded.
  • Persistent host mirror for the particle destination counter.
  • The new halo packing/unpacking path.

Because the FFT differences are small, measurement variability remains plausible.

Several changes can be ruled out:

  • Nedelec device mirror: not used by these runs.
  • FEL initialization/declaration fixes: unrelated.
  • Kokkos::pair: intended to be runtime-equivalent.
  • Archive value initialization and test changes: unrelated.

To establish causality, the cleanest LUMI comparison would be:

  1. 22095a0f8 versus e0e88e188 — isolates the expression change; inspect CG and applyOp.
  2. ee4985bc3 versus e6e2387d0 — isolates the Kokkos 5 compatibility group; inspect updateLayout, halo time, and locateParticles.
  3. Compare identical PCG iteration counts and final residuals, with at least three runs per point.

My current conclusion: the PCG improvement is very likely a real benefit of the field-view expression architecture; the FFT improvement is probably shared infrastructure but needs an ablation run; FEM is essentially performance-neutral.

ALPS

image

- replace class captures with a lightweight device mirror
- add device-side mesh, DOF, and boundary helpers
- capture field views directly in Kokkos kernels
- eliminate GH200 host-only destructor warnings
@aaadelmann aaadelmann linked an issue Aug 15, 2026 that may be closed by this pull request
@aaadelmann aaadelmann self-assigned this Aug 15, 2026
@aaadelmann
aaadelmann marked this pull request as draft August 15, 2026 14:37
@aaadelmann

Copy link
Copy Markdown
Member Author

cscs-ci run cscs-ci-gh200, cscs-ci-mi300, cscs-ci-openmp

Replace std::pair ranges passed to Kokkos::subview with Kokkos::pair in
binning, particle sorting, and particle spatial layout code.

Kokkos::pair is device-compatible, avoiding the NVHPC GH200 host/device
warnings emitted for std::pair without changing the selected ranges.

Validated with OpenMP and CUDA builds and unit tests on 1, 2, and 4 MPI
ranks. The only four-rank failure is the pre-existing NedelecSpace
partitioning limitation. LandauDamping FFT and FEM results remain
consistent with the previous baseline.
@aaadelmann
aaadelmann marked this pull request as ready for review August 16, 2026 10:13
@aaadelmann

Copy link
Copy Markdown
Member Author

cscs-ci run cscs-ci-gh200, cscs-ci-mi300, cscs-ci-openmp

Field differential expressions were storing their input as a complete Field object. When an expression was copied into a Kokkos kernel closure, that also copied host-owned layout, boundary-condition, and lifetime state. NVHPC consequently generated device destruction paths for Field and reported calls to the host-only Field destructor.

Keep the Field template parameter for mesh, dimension, and value-type traits, but store only its Kokkos view in the expression object. Construct gradient, divergence, Laplacian, curl, Hessian, Poisson, and triangular Laplacian expressions from getView(), and size their Expression metadata from the view rather than the complete Field.

This establishes the intended host/device boundary: halo exchange and boundary-condition application remain in the host wrapper, while the deferred device expression retains only the field data handle and its finite-difference coefficients. Include meta_div and meta_poisson even though the CDash warning limit hid those instantiations, since they had the same ownership pattern.

This removes the GH200 host-only Field destructor diagnostics without device-annotating Field lifetime operations or suppressing compiler warnings.

Validation: full GCC/OpenMP and NVHPC CUDA/A100 builds completed successfully. Unit tests were exercised at 1, 2, and 4 MPI ranks on both backends. All affected tests pass; the four-rank runs retain only the established NedelecSpace fixture limitation, which supports three partitions (40/41 tests).
Archive reconstructs each Vector component by copying sizeof(T) bytes from the receive buffer into a local T before assigning it to the destination view. GCC cannot prove that the device-compatible copyBytes loop initializes every byte and emits -Wmaybe-uninitialized for each template instantiation.

Value-initialize the local in both vector deserialize overloads, including the offset variant. copyBytes still overwrites the complete object representation, so serialized data and communication semantics are unchanged; initialization only provides a defined starting state that is visible to compiler data-flow analysis.

This removes the OpenMP CDash warning family without compiler switches, diagnostic suppression, or a host-only memcpy dependency.

Validation: full GCC/OpenMP and NVHPC CUDA/A100 builds completed successfully. Unit tests were exercised at 1, 2, and 4 MPI ranks on both backends. All affected tests pass; the four-rank runs retain only the established NedelecSpace fixture limitation, which supports three partitions (40/41 tests).
@aaadelmann

Copy link
Copy Markdown
Member Author

cscs-ci run cscs-ci-gh200, cscs-ci-mi300, cscs-ci-openmp

aaadelmann and others added 6 commits August 17, 2026 11:27
NedelecSpace is a host-side owner whose FieldLayout member carries MPI decomposition metadata and host-only lifetime operations. Capturing the complete object in KOKKOS_CLASS_LAMBDA forces CUDA to form device closure construction and destruction paths for that host-owned state, which is why the GH200 build diagnoses FieldLayout destruction from device code.

Introduce a documented DeviceStruct snapshot containing only immutable, device-copyable mesh geometry and the reference element. Reproduce the element-index conversion, vertex geometry, edge-DOF mapping, ghosted FEMVector indexing, boundary classification, and basis/curl evaluation needed inside kernels, and expose getDeviceMirror() as the explicit host-to-device architectural boundary.

Change every affected assembly, reconstruction, error, and point-compaction kernel to capture the mirror and required Kokkos views by value with KOKKOS_LAMBDA. The device closure no longer owns or references FieldLayout, MPI state, quadrature ownership, or the parent NedelecSpace object. The detailed header documentation records the mirror lifetime, synchronization contract, exclusions, index ordering, and extension rules.

Validated with warning-free clean OpenMP and CUDA sm_80 builds. The complete unit suite passes 41/41 at one and two ranks on both backends; at four ranks, only the established Nedelec fixture whose three-cell domain cannot be split into four partitions fails.
Resolve the remaining OpenMP and common-host diagnostics by expressing initialization, dimensional specialization, and index-domain conversions directly in the code rather than suppressing compiler warnings.

Value-initialize the FEL Charge aggregate so every member, including the electron charge field reported by GCC, has a defined state. Initialize the bunching factor before the optional modulation branch and scope the derived bunching amplitude and electron count to the branches that consume them, eliminating maybe-uninitialized data flow without changing the generated bunch distribution.

Mark the pruned complex FFT third-dimension extents as maybe_unused because they are intentionally present for the shared two- and three-dimensional setup but are consumed only by the three-dimensional FFTW calls. Convert the particle scatter policy endpoint to the hash-view extent type before comparison so the bounds check uses one unsigned index domain.

Place the full three-dimensional ORB corner-migration test inside the else branch of its if constexpr guard. Non-three-dimensional typed fixtures now compile only the GTEST_SKIP path, preventing unreachable rank-check code from being instantiated while preserving the complete three-dimensional regression.

Validated in warning-free clean OpenMP and CUDA sm_80 builds and in the full unit suite at one, two, and four MPI ranks on both backends. All supported configurations pass; the only four-rank exception remains the unrelated Nedelec fixture with a domain that supports at most three partitions.
Remove the Kokkos 5 legacy View diagnostics that were hidden behind the CDash warning display cap. These warnings came from rank-zero scalar deep copies, View remapping during overwrite-only resize operations, direct access to complex View storage, and complex-valued Kokkos subviews.

Represent particle destination and overlap counters as one-element rank-one views. ParticleSpatialLayout keeps a persistent host mirror for its counter, while the overlap layout mirrors its final send count explicitly. This preserves the atomic counter behavior without instantiating the deprecated rank-zero View conversion machinery.

Use realloc when RegionLayout and BareField replace every element of their allocations, avoiding unnecessary remap kernels and accurately documenting that updateLayout does not preserve field values. Serialize and deserialize scalar values through mdspan element access and local byte-copy temporaries, so complex values no longer require legacy View pointer conversions.

Build halo slices with submdspan and construct pack/unpack range policies from the slice extents, keeping complex halo exchange on the supported Kokkos 5 mdspan path. Capture nested overlap interaction lambdas by value, and give the truncated-Green test particle type a uniquely named namespace so CUDA-generated closure visibility matches the captured functor type.

Validated by warning-free clean OpenMP and CUDA sm_80 builds, including focused NUFFT and truncated-Green interaction recompiles. Full unit tests pass 41/41 at one and two ranks on both backends and 40/41 at four ranks, where only the pre-existing three-partition Nedelec fixture is unsupported.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

GH200 Warnings

1 participant