Fix LagrangeSpace device captures for CUDA - #594
Open
aaadelmann wants to merge 10 commits into
Open
Conversation
- replace class captures with a lightweight device mirror - add device-side mesh, DOF, and boundary helpers - capture field views directly in Kokkos kernels - eliminate GH200 host-only destructor warnings
Member
Author
|
cscs-ci run cscs-ci-gh200, cscs-ci-mi300, cscs-ci-openmp |
Replace std::pair ranges passed to Kokkos::subview with Kokkos::pair in binning, particle sorting, and particle spatial layout code. Kokkos::pair is device-compatible, avoiding the NVHPC GH200 host/device warnings emitted for std::pair without changing the selected ranges. Validated with OpenMP and CUDA builds and unit tests on 1, 2, and 4 MPI ranks. The only four-rank failure is the pre-existing NedelecSpace partitioning limitation. LandauDamping FFT and FEM results remain consistent with the previous baseline.
aaadelmann
marked this pull request as ready for review
August 16, 2026 10:13
Member
Author
|
cscs-ci run cscs-ci-gh200, cscs-ci-mi300, cscs-ci-openmp |
Field differential expressions were storing their input as a complete Field object. When an expression was copied into a Kokkos kernel closure, that also copied host-owned layout, boundary-condition, and lifetime state. NVHPC consequently generated device destruction paths for Field and reported calls to the host-only Field destructor. Keep the Field template parameter for mesh, dimension, and value-type traits, but store only its Kokkos view in the expression object. Construct gradient, divergence, Laplacian, curl, Hessian, Poisson, and triangular Laplacian expressions from getView(), and size their Expression metadata from the view rather than the complete Field. This establishes the intended host/device boundary: halo exchange and boundary-condition application remain in the host wrapper, while the deferred device expression retains only the field data handle and its finite-difference coefficients. Include meta_div and meta_poisson even though the CDash warning limit hid those instantiations, since they had the same ownership pattern. This removes the GH200 host-only Field destructor diagnostics without device-annotating Field lifetime operations or suppressing compiler warnings. Validation: full GCC/OpenMP and NVHPC CUDA/A100 builds completed successfully. Unit tests were exercised at 1, 2, and 4 MPI ranks on both backends. All affected tests pass; the four-rank runs retain only the established NedelecSpace fixture limitation, which supports three partitions (40/41 tests).
Archive reconstructs each Vector component by copying sizeof(T) bytes from the receive buffer into a local T before assigning it to the destination view. GCC cannot prove that the device-compatible copyBytes loop initializes every byte and emits -Wmaybe-uninitialized for each template instantiation. Value-initialize the local in both vector deserialize overloads, including the offset variant. copyBytes still overwrites the complete object representation, so serialized data and communication semantics are unchanged; initialization only provides a defined starting state that is visible to compiler data-flow analysis. This removes the OpenMP CDash warning family without compiler switches, diagnostic suppression, or a host-only memcpy dependency. Validation: full GCC/OpenMP and NVHPC CUDA/A100 builds completed successfully. Unit tests were exercised at 1, 2, and 4 MPI ranks on both backends. All affected tests pass; the four-rank runs retain only the established NedelecSpace fixture limitation, which supports three partitions (40/41 tests).
Member
Author
|
cscs-ci run cscs-ci-gh200, cscs-ci-mi300, cscs-ci-openmp |
NedelecSpace is a host-side owner whose FieldLayout member carries MPI decomposition metadata and host-only lifetime operations. Capturing the complete object in KOKKOS_CLASS_LAMBDA forces CUDA to form device closure construction and destruction paths for that host-owned state, which is why the GH200 build diagnoses FieldLayout destruction from device code. Introduce a documented DeviceStruct snapshot containing only immutable, device-copyable mesh geometry and the reference element. Reproduce the element-index conversion, vertex geometry, edge-DOF mapping, ghosted FEMVector indexing, boundary classification, and basis/curl evaluation needed inside kernels, and expose getDeviceMirror() as the explicit host-to-device architectural boundary. Change every affected assembly, reconstruction, error, and point-compaction kernel to capture the mirror and required Kokkos views by value with KOKKOS_LAMBDA. The device closure no longer owns or references FieldLayout, MPI state, quadrature ownership, or the parent NedelecSpace object. The detailed header documentation records the mirror lifetime, synchronization contract, exclusions, index ordering, and extension rules. Validated with warning-free clean OpenMP and CUDA sm_80 builds. The complete unit suite passes 41/41 at one and two ranks on both backends; at four ranks, only the established Nedelec fixture whose three-cell domain cannot be split into four partitions fails.
Resolve the remaining OpenMP and common-host diagnostics by expressing initialization, dimensional specialization, and index-domain conversions directly in the code rather than suppressing compiler warnings. Value-initialize the FEL Charge aggregate so every member, including the electron charge field reported by GCC, has a defined state. Initialize the bunching factor before the optional modulation branch and scope the derived bunching amplitude and electron count to the branches that consume them, eliminating maybe-uninitialized data flow without changing the generated bunch distribution. Mark the pruned complex FFT third-dimension extents as maybe_unused because they are intentionally present for the shared two- and three-dimensional setup but are consumed only by the three-dimensional FFTW calls. Convert the particle scatter policy endpoint to the hash-view extent type before comparison so the bounds check uses one unsigned index domain. Place the full three-dimensional ORB corner-migration test inside the else branch of its if constexpr guard. Non-three-dimensional typed fixtures now compile only the GTEST_SKIP path, preventing unreachable rank-check code from being instantiated while preserving the complete three-dimensional regression. Validated in warning-free clean OpenMP and CUDA sm_80 builds and in the full unit suite at one, two, and four MPI ranks on both backends. All supported configurations pass; the only four-rank exception remains the unrelated Nedelec fixture with a domain that supports at most three partitions.
Remove the Kokkos 5 legacy View diagnostics that were hidden behind the CDash warning display cap. These warnings came from rank-zero scalar deep copies, View remapping during overwrite-only resize operations, direct access to complex View storage, and complex-valued Kokkos subviews. Represent particle destination and overlap counters as one-element rank-one views. ParticleSpatialLayout keeps a persistent host mirror for its counter, while the overlap layout mirrors its final send count explicitly. This preserves the atomic counter behavior without instantiating the deprecated rank-zero View conversion machinery. Use realloc when RegionLayout and BareField replace every element of their allocations, avoiding unnecessary remap kernels and accurately documenting that updateLayout does not preserve field values. Serialize and deserialize scalar values through mdspan element access and local byte-copy temporaries, so complex values no longer require legacy View pointer conversions. Build halo slices with submdspan and construct pack/unpack range policies from the slice extents, keeping complex halo exchange on the supported Kokkos 5 mdspan path. Capture nested overlap interaction lambdas by value, and give the truncated-Green test particle type a uniquely named namespace so CUDA-generated closure visibility matches the captured functor type. Validated by warning-free clean OpenMP and CUDA sm_80 builds, including focused NUFFT and truncated-Green interaction recompiles. Full unit tests pass 41/41 at one and two ranks on both backends and 40/41 at four ranks, where only the pre-existing three-partition Nedelec fixture is unsupported.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
LUMI Results
Red old master:

Justification
e0e88e188: Poisson and differential expressions capture only the Kokkos field view instead of the completeFieldobject, producing smaller device-kernel closurese6e2387d0: halo packing and unpacking usesubmdspan; PCG performs halo exchanges repeatedly during its iterationse6e2387d0, including direct reallocation of overwritten field/layout storage and updated particle bookkeeping9d9387c57: Lagrange kernels capture a lightweight device mirror and the required views instead of the complete host-side objectKokkos::pair,[[maybe_unused]], Archive initialization, and unit-test changesThe PCG mapping is particularly strong. Before
e0e88e188, every PCG Laplace expression stored a complete field:Now it stores only the device data handle:
This affects
meta_poisson,meta_lower_laplace,meta_upper_laplace, andmeta_upper_and_lower_laplace, which are repeatedly invoked byPoissonCG. It produces smaller kernel closures and gives the AMD compiler much less host-side ownership state to carry through device kernels.The scaling signature supports that explanation: the improvement is largest at 16 GPUs and shrinks as communication becomes a larger fraction of runtime. That is typical of a compute-kernel improvement.
For FFT, no FFT algorithm was optimized in PR593. The
PrunedCCchange only adds[[maybe_unused]], so it cannot explain the improvement. Likely shared contributors are:Kokkos::resize→Kokkos::reallocwhere old field/layout contents are discarded.Because the FFT differences are small, measurement variability remains plausible.
Several changes can be ruled out:
Kokkos::pair: intended to be runtime-equivalent.To establish causality, the cleanest LUMI comparison would be:
22095a0f8versuse0e88e188— isolates the expression change; inspectCGandapplyOp.ee4985bc3versuse6e2387d0— isolates the Kokkos 5 compatibility group; inspectupdateLayout, halo time, andlocateParticles.My current conclusion: the PCG improvement is very likely a real benefit of the field-view expression architecture; the FFT improvement is probably shared infrastructure but needs an ablation run; FEM is essentially performance-neutral.
ALPS