Update Particle Container to Pure SoA #348

ax3l · 2023-04-19T07:35:51Z

Transition particle containers to pure SoA layouts.

test on CUDA with MPI
test on OMP with MPI
wait for 1st and 2nd CPU particle Redistribute fix in AMReX
- Implement StructOfArray::empty() and ParticleTile::empty() for pure SoA AMReX-Codes/amrex#3296
- More constexpr if for pure soa in RedistributeCPU AMReX-Codes/amrex#3299
more AMReX updates:
- Fixing the ParIter numParticles functions for pure SOA AMReX-Codes/amrex#3303
- PureSoA: Disable AoS Access AMReX-Codes/amrex#3290
merge WarpX first Update Particle Container to Pure SoA WarpX#3850
merge pyAMReX first Particle Container: Support Pure SoA AMReX-Codes/pyamrex#124
improve performance of id handling via constants and Add ParticleIDWrapper::make_invalid() AMReX-Codes/amrex#3735
update docs to make sure AoS is not mentioned anymore and particle id handling is guided

Benchmarks on NVIDIA A100-SXM 80GB GPU, SP

./impactx ../../examples/fodo/input_fodo.in beam.npart=600000000 diag.enable=false diag.slice_step_diagnostics=false lattice.elements=drift1 quad1 drift2 quad2 drift3

soa_gpu_sp.txt, dev_gpu_sp.txt

FODO runtime: 2.52x faster 🚀 ✨
impactx::Push::Drift: 2.33x faster
impactx::Push::Quad: 2.85x faster
impactX::collect_lost_particles: 2.77x faster
ImpactX::add_particles: 5.4% faster

Benchmarks on NVIDIA A100-SXM 80GB GPU, DP

./impactx ../../examples/fodo/input_fodo.in beam.npart=600000000 diag.enable=false diag.slice_step_diagnostics=false lattice.elements=drift1 quad1 drift2 quad2 drift3

soa_gpu_dp.txt, dev_gpu_dp.txt

FODO runtime: 1.73x faster 🚀 ✨
impactx::Push::Drift: 1.85x faster
impactx::Push::Quad: 1.42x faster
impactX::collect_lost_particles: 3.68x faster
ImpactX::add_particles: 6.6% faster

Benchmarks on CPU, SP

Laptop: 12th Gen Intel(R) Core(TM) i9-12900H, performance core w/ performance power mode on

export OMP_NUM_THREADS=1
./impactx ../../examples/fodo/input_fodo.in beam.npart=1000000 diag.enable=false diag.slice_step_diagnostics=false lattice.elements=drift1 quad1 drift2 quad2 drift3 &
taskset -cp 6 $!

soa_cpu_sp.txt, dev_cpu_sp.txt

FODO runtime: 1.28x faster 🚀 ✨
impactx::Push::Drift: 2.17x faster
impactx::Push::Quad: 4.8% faster
impactX::collect_lost_particles: 2.48x faster
ImpactX::add_particles: 1.7% faster

Benchmarks on CPU, DP

Laptop: 12th Gen Intel(R) Core(TM) i9-12900H, performance core w/ performance power mode on

export OMP_NUM_THREADS=1
./impactx ../../examples/fodo/input_fodo.in beam.npart=1000000 diag.enable=false diag.slice_step_diagnostics=false lattice.elements=drift1 quad1 drift2 quad2 drift3 &
taskset -cp 6 $!

soa_cpu_dp.txt, dev_cpu_dp.txt

FODO runtime: 1.21x faster 🚀 ✨
impactx::Push::Drift: 1.76x faster
impactx::Push::Quad: same perf.
impactX::collect_lost_particles: 3.24x faster
ImpactX::add_particles: 2.2% faster

src/particles/diagnostics/DiagnosticOutput.cpp

src/particles/elements/mixin/beamoptic.H

src/particles/elements/diagnostics/openPMD.cpp

src/particles/ImpactXParticleContainer.cpp

src/particles/elements/Multipole.H

            pxout = px + dpx;

-            p.pos(RealAoS::y) = y;
+            // yout = y;


src/particles/elements/NonlinearLens.H

            pxout = px + dpx;

-            p.pos(RealAoS::y) = y;
+            // yout = y;


src/particles/elements/ShortRF.H

…oA (#3296) The implementation of `ParticleTile::empty()` was wrong for pure SoA, which led to problems in ImpactX: ECP-WarpX/impactx#348 The proposed changes: - [x] fix a bug or incorrect behavior in AMReX - [x] add new capabilities to AMReX - [ ] changes answers in the test suite to more than roundoff level - [ ] are likely to significantly affect the results of downstream AMReX users - [ ] include documentation in the code and/or rst files, if appropriate

cmake/dependencies/ABLASTR.cmake

ax3l · 2024-01-28T18:44:34Z

@cemitch99 when you have time next week, can you please review the chances I did to individual elements?

cemitch99 · 2024-01-29T17:55:33Z

This pattern of changes to individual elements looks correct, and I like the improved symmetry between the treatment of position and momentum variables. The main risk introduced is possible mixing of the initial values and final values, so I will need to check every element line-by-line before review.

examples/pytorch_surrogate_model/run_ml_surrogate.py

Transition particle containers to pure SoA layouts.

ax3l · 2024-02-09T20:17:27Z

Congrats to all who helped @Thierry992, @atmyers, @AlexanderSinn et al. 🎉 👏

ax3l · 2024-02-16T18:40:26Z

I repeated the benchmarks on GPU for scrutiny on Perlmutter A100 SMX 80 GB GPUs using the new AMReX AMReX-Codes/amrex#3763 tiny_profiler.device_synchronize_around_region = 1 syncs.

There is now a bit more overhead added due to the excessive syncs, but individual elements should be more precisely measured.

Benchmarks on NVIDIA A100-SXM 80GB GPU, SP

./impactx ../../examples/fodo/input_fodo.in beam.npart=600000000 diag.enable=false diag.slice_step_diagnostics=false tiny_profiler.device_synchronize_around_region=1 lattice.elements=drift1 quad1 drift2 quad2 drift3

soa_gpu_sp.txt, old_gpu_sp.txt

FODO runtime: 2.34x faster 🚀 ✨
impactx::Push::Drift: 2.32x faster
impactx::Push::Quad: 2.32x faster
impactX::collect_lost_particles: 2.78x faster
ImpactX::add_particles: 5.2% faster

Benchmarks on NVIDIA A100-SXM 80GB GPU, DP

./impactx ../../examples/fodo/input_fodo.in beam.npart=600000000 diag.enable=false diag.slice_step_diagnostics=false tiny_profiler.device_synchronize_around_region=1 lattice.elements=drift1 quad1 drift2 quad2 drift3

soa_gpu_dp.txt, old_gpu_dp.txt

FODO runtime: 1.65x faster 🚀 ✨
impactx::Push::Drift: 1.80x faster
impactx::Push::Quad: 1.32x faster
impactX::collect_lost_particles: 3.65x faster
ImpactX::add_particles: 1.6% faster

ax3l added component: core Core ImpactX functionality Performance optimization labels Apr 19, 2023

ax3l requested a review from atmyers April 19, 2023 07:35

This was referenced Apr 19, 2023

Use the new SoA Data Structure #245

Closed

Support SoA-Only Particles AMReX-Codes/amrex#2878

Merged

github-advanced-security bot found potential problems Apr 19, 2023

View reviewed changes

src/particles/diagnostics/DiagnosticOutput.cpp Fixed Show fixed Hide fixed

src/particles/elements/mixin/beamoptic.H Fixed Show fixed Hide fixed

ax3l force-pushed the topic-particle-soa branch 2 times, most recently from 38c7f86 to c4268f4 Compare April 19, 2023 08:20

ax3l commented Apr 19, 2023

View reviewed changes

src/particles/elements/diagnostics/openPMD.cpp Outdated Show resolved Hide resolved

ax3l commented Apr 19, 2023

View reviewed changes

src/particles/ImpactXParticleContainer.cpp Outdated Show resolved Hide resolved

ax3l force-pushed the topic-particle-soa branch 8 times, most recently from cf12d2c to 83f96ff Compare April 28, 2023 01:43

github-advanced-security bot found potential problems Apr 28, 2023

View reviewed changes

ax3l force-pushed the topic-particle-soa branch 4 times, most recently from a469076 to ba1d781 Compare April 28, 2023 06:46

ax3l mentioned this pull request May 2, 2023

[WIP] Fix: RedistributeCPU for SoA AMReX-Codes/amrex#3291

Closed

7 tasks

ax3l force-pushed the topic-particle-soa branch from ba1d781 to c04c094 Compare May 4, 2023 16:44

atmyers mentioned this pull request May 4, 2023

Implement StructOfArray::empty() and ParticleTile::empty() for pure SoA AMReX-Codes/amrex#3296

Merged

5 tasks

ax3l force-pushed the topic-particle-soa branch 2 times, most recently from 99fb419 to 03d6b6c Compare May 4, 2023 22:59

ax3l commented May 5, 2023

View reviewed changes

cmake/dependencies/ABLASTR.cmake Outdated Show resolved Hide resolved

cmake/dependencies/ABLASTR.cmake Outdated Show resolved Hide resolved

ax3l force-pushed the topic-particle-soa branch from 7d89ec1 to 51a7f03 Compare January 28, 2024 19:17

This comment was marked as resolved.

Sign in to view

ax3l force-pushed the topic-particle-soa branch 3 times, most recently from fe812bd to 7b85a26 Compare January 28, 2024 20:27

ax3l mentioned this pull request Jan 28, 2024

Python: Clean CUDA Shutdown #510

Merged

ax3l force-pushed the topic-particle-soa branch from 7b85a26 to 4772210 Compare January 29, 2024 18:50

ax3l changed the title ~~[WIP] Update Particle Container to Pure SoA~~ Update Particle Container to Pure SoA Jan 29, 2024

This comment was marked as resolved.

Sign in to view

ax3l force-pushed the topic-particle-soa branch 3 times, most recently from d158624 to 0e8b676 Compare January 31, 2024 20:09

This comment was marked as resolved.

Sign in to view

github-advanced-security bot found potential problems Jan 31, 2024

View reviewed changes

examples/pytorch_surrogate_model/run_ml_surrogate.py Fixed Show fixed Hide fixed

examples/pytorch_surrogate_model/run_ml_surrogate.py Fixed Show fixed Hide fixed

ax3l force-pushed the topic-particle-soa branch 6 times, most recently from 6968ba3 to e954247 Compare February 8, 2024 02:07

Update Particle Container to Pure SoA

1829d1f

Transition particle containers to pure SoA layouts.

ax3l force-pushed the topic-particle-soa branch from e954247 to 1829d1f Compare February 8, 2024 07:43

BenWibking mentioned this pull request Feb 9, 2024

convert to pure SoA particle containers quokka-astro/quokka#515

Open

ax3l merged commit 9876a9e into ECP-WarpX:development Feb 9, 2024
15 checks passed

ax3l deleted the topic-particle-soa branch February 9, 2024 20:16

debog mentioned this pull request Feb 16, 2024

SoA-only Particle Container erf-model/ERF#1444

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update Particle Container to Pure SoA #348

Update Particle Container to Pure SoA #348

ax3l commented Apr 19, 2023 •

edited

Loading

ax3l commented Jan 28, 2024

This comment was marked as resolved.

cemitch99 commented Jan 29, 2024

This comment was marked as resolved.

This comment was marked as resolved.

ax3l commented Feb 9, 2024

ax3l commented Feb 16, 2024 •

edited

Loading

Update Particle Container to Pure SoA #348

Update Particle Container to Pure SoA #348

Conversation

ax3l commented Apr 19, 2023 • edited Loading

Benchmarks on NVIDIA A100-SXM 80GB GPU, SP

Benchmarks on NVIDIA A100-SXM 80GB GPU, DP

Benchmarks on CPU, SP

Benchmarks on CPU, DP

ax3l commented Jan 28, 2024

This comment was marked as resolved.

cemitch99 commented Jan 29, 2024

This comment was marked as resolved.

This comment was marked as resolved.

ax3l commented Feb 9, 2024

ax3l commented Feb 16, 2024 • edited Loading

Benchmarks on NVIDIA A100-SXM 80GB GPU, SP

Benchmarks on NVIDIA A100-SXM 80GB GPU, DP

ax3l commented Apr 19, 2023 •

edited

Loading

ax3l commented Feb 16, 2024 •

edited

Loading