From 736abd64a197b930a1a29bb4301e9ad899058f30 Mon Sep 17 00:00:00 2001 From: Spencer Bryngelson Date: Sat, 19 Jul 2025 09:57:54 -0400 Subject: [PATCH 1/7] update cursor rules. --- .cursor/rules/mfc-agent-rules.mdc | 138 ++++++++++++++++++++++++------ 1 file changed, 113 insertions(+), 25 deletions(-) diff --git a/.cursor/rules/mfc-agent-rules.mdc b/.cursor/rules/mfc-agent-rules.mdc index 6a8baddd64..0784cd2224 100644 --- a/.cursor/rules/mfc-agent-rules.mdc +++ b/.cursor/rules/mfc-agent-rules.mdc @@ -4,29 +4,28 @@ alwaysApply: true --- # 0 Purpose & Scope -Consolidated guidance for the MFC exascale, many-physics solver. -Written primarily for Fortran/Fypp; the OpenACC and style sections matter only when -`.fpp` / `.f90` files are in view. +Consolidated guidance for the MFC exascale, many-physics solver. +Written primarily for Fortran/Fypp; the OpenACC and style sections matter only when `.fpp` / `.f90` files are in view. --- # 1 Global Project Context (always) -- **Project**: *MFC* is modern Fortran 2008+ generated with **Fypp**. - - Sources `src/`, tests `tests/`, examples `examples/`. - - Most sources are `.fpp`; CMake transpiles them to `.f90`. -- **Fypp macros** live in `src//include/` you should scan these first. - `` ∈ {`simulation`,`common`,`pre_process`,`post_process`}. -- Only `simulation` (+ its `common` calls) is GPU-accelerated via **OpenACC** or **OpenMP**. +- **Project**: *MFC* is modern Fortran 2008+ generated with **Fypp**. + - Sources `src/`, tests `tests/`, examples `examples/`. + - Most sources are `.fpp`; CMake transpiles them to `.f90`. +- **Fypp macros** live in `src//include/` you should scan these first. + `` ∈ {`simulation`,`common`,`pre_process`,`post_process`}. +- Only `simulation` (+ its `common` calls) is GPU-accelerated via **OpenACC**. - Assume free-form Fortran 2008+, `implicit none`, explicit `intent`, and modern - intrinsics. + intrinsics. - Prefer `module … contains … subroutine foo()`; avoid `COMMON` blocks and - file-level `include` files. -- **Read the full codebase and docs *before* changing code.** - Docs: and the respository root `README.md`. + file-level `include` files. +- **Read the full codebase and docs *before* changing code.** + Docs: and the repository root `README.md`. ### Incremental-change workflow -1. Draft a step-by-step plan. -2. After each step, build: +1. Draft a step-by-step plan. +2. After each step, build: ```bash ./mfc.sh build -t pre_process simulation -j $(nproc) ``` @@ -51,34 +50,123 @@ Written primarily for Fortran/Fypp; the OpenACC and style sections matter only w * Private helpers stay in the module; avoid nested procedures. * **Size limits**: subroutine ≤ 500 lines, helper ≤ 150, function ≤ 100, module/file ≤ 1000. -* ≤ 6 arguments per routine; otherwise pass a derived-type “params” struct. +* ≤ 6 arguments per routine; otherwise pass a derived-type "params" struct. * No `goto` (except unavoidable legacy); no global state (`COMMON`, `save`). * Every variable: `intent(in|out|inout)` + appropriate `dimension` / `allocatable` / `pointer`. * Use `s_mpi_abort()` for errors, not `stop`. -* Mark GPU-callable helpers that are called from GPU parallel loops immediately after declaration: +* Mark OpenACC-callable helpers that are called from OpenACC parallel loops immediately after declaration: ```fortran subroutine s_flux_update(...) - $:GPU_ROUTINE(function_name='s_flux_update', parallelism='[seq]') + !$acc routine seq ... end subroutine ``` --- -# 3 FYPP Macros for GPU acceleration Pogramming Guidelines (for kernels) - -Do not directly use OpenACC or OpenMP directives directly. Instead, use the FYPP macros contained in src/common/include/parallel_macros.fpp +# 3 OpenACC Programming Guidelines (for kernels) Wrap tight loops with ```fortran -$:GPU_PARALLEL_FOR(private='[...]', copy='[...]') +!$acc parallel loop gang vector default(present) reduction(...) ``` -* Add `collapse=n` to merge nested loops when safe. -* Declare loop-local variables with `private='[...]'`. +* Add `collapse(n)` to merge nested loops when safe. +* Declare loop-local variables with `private(...)`. * Allocate large arrays with `managed` or move them into a persistent - `$:GPU_ENTER_DATA(...)` region at start-up. + `!$acc enter data` region at start-up. * **Do not** place `stop` / `error stop` inside device code. * Must compile with Cray `ftn` and NVIDIA `nvfortran` for GPU offloading; also build CPU-only with GNU `gfortran` and Intel `ifx`/`ifort`. + +--- + +# 4 File & Module Structure + +- **File Naming**: + - `.fpp` files: Fypp preprocessed files that get translated to `.f90` + - Modules are named with `m_` prefix followed by feature name: `m_helper_basic`, `m_viscous` + - Primary program file is named `p_main.fpp` + +- **Module Layout**: + - Start with Fypp include for macros: `#:include 'macros.fpp'` + - Header comments using `!>` style documentation + - `module` declaration with name matching filename + - `use` statements for dependencies + - `implicit none` statement + - `private` declaration followed by explicit `public` exports + - `contains` section + - Implementation of subroutines and functions + +# 5 Fypp Macros and GPU Acceleration + +- **Fypp Directives**: + - Start with `#:` (e.g., `#:include`, `#:def`, `#:enddef`) + - Macros defined in `include/*.fpp` files + - Used for code generation, conditional compilation, and GPU offloading + +- **GPU Macros**: + - `$:GPU_ROUTINE(parallelism='[seq]')` - Marks GPU-callable routines + - `$:GPU_PARALLEL_LOOP(collapse=N)` - Parallelizes loops + - `$:GPU_LOOP(parallelism='[seq]')` - Marks sequential loops + - `$:GPU_UPDATE(device='[var1,var2]')` - Updates device data + - `$:GPU_ENTER_DATA(copyin='[var]')` - Copies data to device + - `$:GPU_EXIT_DATA(delete='[var]')` - Removes data from device + +# 6 Documentation Style + +- **Subroutine/Function Documentation**: + ```fortran + !> This procedure + !! @param param_name Description of the parameter + !! @return Description of the return value (for functions) + ``` + +# 7 Error Handling + +- **Assertions**: + - Use `ASSERT` macro for validating conditions + - Example: `@:ASSERT(predicate, message)` + +- **Error Reporting**: + - Use `s_mpi_abort()` for error termination, not `stop` + - No `stop` / `error stop` inside device code + +# 8 Memory Management + +- **Allocation/Deallocation**: + - Use `@:ALLOCATE(var1, var2)` macro for device-aware allocation + - Use `@:DEALLOCATE(var1, var2)` macro for device-aware deallocation + +# 9. Additional Observed Patterns + +- **Derived Types**: + - Extensive use of derived types for encapsulation + - Use pointers within derived types (e.g., `pointer, dimension(:,:,:) => null()`) + - Clear documentation of derived type components + +- **Pure & Elemental Functions**: + - Use `pure` and `elemental` attributes for side-effect-free functions + - Combine them for operations on arrays (`pure elemental function`) + +- **Precision Handling**: + - Use `wp` (working precision) parameter from `m_precision_select` + - Never hardcode precision with `real*8` or similar + +- **Loop Optimization**: + - Favor array operations over explicit loops when possible + - Use `collapse(N)` directive to optimize nested loops + +# 10. Fortran Practices to Avoid + +- **Fixed Format**: Only free-form Fortran is used + - No column-position dependent code + +- **Older Intrinsics**: Avoid outdated Fortran features like: + - `equivalence` statements + - `data` statements (use initialization expressions) + - Character*N (use `character(len=N)` instead) + +- **Using same variable for multiple purposes**: Maintain single responsibility + - Each variable should have one clear purpose From a35a3a542e69f33e44ec66becd3242deb2ed71aa Mon Sep 17 00:00:00 2001 From: Spencer Bryngelson Date: Sat, 19 Jul 2025 14:51:35 -0400 Subject: [PATCH 2/7] update --- .cursor/rules/mfc-agent-rules.mdc | 72 +++++++++++++++++-------------- 1 file changed, 39 insertions(+), 33 deletions(-) diff --git a/.cursor/rules/mfc-agent-rules.mdc b/.cursor/rules/mfc-agent-rules.mdc index 0784cd2224..a63276f493 100644 --- a/.cursor/rules/mfc-agent-rules.mdc +++ b/.cursor/rules/mfc-agent-rules.mdc @@ -1,11 +1,11 @@ ---- -description: Full MFC project rules – consolidated for Agent Mode -alwaysApply: true ---- +---- +-description: Full MFC project rules – consolidated for Agent Mode +-alwaysApply: true +---- # 0 Purpose & Scope Consolidated guidance for the MFC exascale, many-physics solver. -Written primarily for Fortran/Fypp; the OpenACC and style sections matter only when `.fpp` / `.f90` files are in view. +Written primarily for Fortran/Fypp; the GPU and style sections matter only when `.fpp` / `.f90` files are in view. --- @@ -15,15 +15,14 @@ Written primarily for Fortran/Fypp; the OpenACC and style sections matter only w - Most sources are `.fpp`; CMake transpiles them to `.f90`. - **Fypp macros** live in `src//include/` you should scan these first. `` ∈ {`simulation`,`common`,`pre_process`,`post_process`}. -- Only `simulation` (+ its `common` calls) is GPU-accelerated via **OpenACC**. -- Assume free-form Fortran 2008+, `implicit none`, explicit `intent`, and modern - intrinsics. -- Prefer `module … contains … subroutine foo()`; avoid `COMMON` blocks and - file-level `include` files. +- Only `simulation` (+ its `common` calls) is GPU-accelerated via **OpenACC** or **OpenMP**. +- Assume free-form Fortran 2008+, `implicit none`, explicit `intent`, and modern intrinsics. +- Prefer `module … contains … subroutine foo()`; avoid `COMMON` blocks and file-level `include` files. - **Read the full codebase and docs *before* changing code.** - Docs: and the repository root `README.md`. + - Docs: and the respository root `README.md`. ### Incremental-change workflow + 1. Draft a step-by-step plan. 2. After each step, build: ```bash @@ -48,34 +47,35 @@ Written primarily for Fortran/Fypp; the OpenACC and style sections matter only w * Subroutine → `s__` (e.g. `s_compute_flux`) * Function → `f__` * Private helpers stay in the module; avoid nested procedures. -* **Size limits**: subroutine ≤ 500 lines, helper ≤ 150, function ≤ 100, - module/file ≤ 1000. +* **Size limits**: subroutine ≤ 500 lines, helper ≤ 150, function ≤ 100, module/file ≤ 1000. * ≤ 6 arguments per routine; otherwise pass a derived-type "params" struct. * No `goto` (except unavoidable legacy); no global state (`COMMON`, `save`). -* Every variable: `intent(in|out|inout)` + appropriate `dimension` / `allocatable` - / `pointer`. +* Every variable: `intent(in|out|inout)` + appropriate `dimension` / `allocatable` / `pointer`. * Use `s_mpi_abort()` for errors, not `stop`. -* Mark OpenACC-callable helpers that are called from OpenACC parallel loops immediately after declaration: +* Mark GPU-callable helpers that are called from GPU parallel loops immediately after declaration: ```fortran subroutine s_flux_update(...) - !$acc routine seq + $:GPU_ROUTINE(function_name='s_flux_update', parallelism='[seq]') ... end subroutine ``` --- -# 3 OpenACC Programming Guidelines (for kernels) +# 3 FYPP Macros for GPU acceleration Pogramming Guidelines (for GPU kernels) + +Do not directly use OpenACC or OpenMP directives directly. +Instead, use the FYPP macros contained in src/common/include/parallel_macros.fpp Wrap tight loops with ```fortran -!$acc parallel loop gang vector default(present) reduction(...) +$:GPU_PARALLEL_FOR(private='[...]', copy='[...]') ``` -* Add `collapse(n)` to merge nested loops when safe. -* Declare loop-local variables with `private(...)`. +* Add `collapse=n` to merge nested loops when safe. +* Declare loop-local variables with `private='[...]'`. * Allocate large arrays with `managed` or move them into a persistent - `!$acc enter data` region at start-up. + `$:GPU_ENTER_DATA(...)` region at start-up. * **Do not** place `stop` / `error stop` inside device code. * Must compile with Cray `ftn` and NVIDIA `nvfortran` for GPU offloading; also build CPU-only with GNU `gfortran` and Intel `ifx`/`ifort`. @@ -101,18 +101,23 @@ Wrap tight loops with # 5 Fypp Macros and GPU Acceleration +## Use of Fypp - **Fypp Directives**: - Start with `#:` (e.g., `#:include`, `#:def`, `#:enddef`) - Macros defined in `include/*.fpp` files - Used for code generation, conditional compilation, and GPU offloading -- **GPU Macros**: - - `$:GPU_ROUTINE(parallelism='[seq]')` - Marks GPU-callable routines - - `$:GPU_PARALLEL_LOOP(collapse=N)` - Parallelizes loops - - `$:GPU_LOOP(parallelism='[seq]')` - Marks sequential loops - - `$:GPU_UPDATE(device='[var1,var2]')` - Updates device data - - `$:GPU_ENTER_DATA(copyin='[var]')` - Copies data to device - - `$:GPU_EXIT_DATA(delete='[var]')` - Removes data from device +## Some examples + +Documentation on how to use the Fypp macros for GPU offloading is available at https://mflowcode.github.io/documentation/md_gpuParallelization.html + +Some examples include: +- `$:GPU_ROUTINE(parallelism='[seq]')` - Marks GPU-callable routines +- `$:GPU_PARALLEL_LOOP(collapse=N)` - Parallelizes loops +- `$:GPU_LOOP(parallelism='[seq]')` - Marks sequential loops +- `$:GPU_UPDATE(device='[var1,var2]')` - Updates device data +- `$:GPU_ENTER_DATA(copyin='[var]')` - Copies data to device +- `$:GPU_EXIT_DATA(delete='[var]')` - Removes data from device # 6 Documentation Style @@ -122,11 +127,12 @@ Wrap tight loops with !! @param param_name Description of the parameter !! @return Description of the return value (for functions) ``` +which conforms to the Doxygen Fortran format. # 7 Error Handling - **Assertions**: - - Use `ASSERT` macro for validating conditions + - Use the fypp `ASSERT` macro for validating conditions - Example: `@:ASSERT(predicate, message)` - **Error Reporting**: @@ -136,8 +142,8 @@ Wrap tight loops with # 8 Memory Management - **Allocation/Deallocation**: - - Use `@:ALLOCATE(var1, var2)` macro for device-aware allocation - - Use `@:DEALLOCATE(var1, var2)` macro for device-aware deallocation + - Use fypp macro `@:ALLOCATE(var1, var2)` macro for device-aware allocation + - Use fypp macro `@:DEALLOCATE(var1, var2)` macro for device-aware deallocation # 9. Additional Observed Patterns @@ -156,7 +162,7 @@ Wrap tight loops with - **Loop Optimization**: - Favor array operations over explicit loops when possible - - Use `collapse(N)` directive to optimize nested loops + - Use `collapse=N` directive to optimize nested loops # 10. Fortran Practices to Avoid From ed338165191235697ccca61f315ab6b8940e8d49 Mon Sep 17 00:00:00 2001 From: Spencer Bryngelson Date: Sat, 19 Jul 2025 14:55:19 -0400 Subject: [PATCH 3/7] typos --- .cursor/rules/mfc-agent-rules.mdc | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/.cursor/rules/mfc-agent-rules.mdc b/.cursor/rules/mfc-agent-rules.mdc index a63276f493..5bab4b7634 100644 --- a/.cursor/rules/mfc-agent-rules.mdc +++ b/.cursor/rules/mfc-agent-rules.mdc @@ -19,7 +19,7 @@ Written primarily for Fortran/Fypp; the GPU and style sections matter only when - Assume free-form Fortran 2008+, `implicit none`, explicit `intent`, and modern intrinsics. - Prefer `module … contains … subroutine foo()`; avoid `COMMON` blocks and file-level `include` files. - **Read the full codebase and docs *before* changing code.** - - Docs: and the respository root `README.md`. + - Docs: and the repository root `README.md`. ### Incremental-change workflow @@ -62,7 +62,7 @@ Written primarily for Fortran/Fypp; the GPU and style sections matter only when --- -# 3 FYPP Macros for GPU acceleration Pogramming Guidelines (for GPU kernels) +# 3 FYPP Macros for GPU acceleration Programming Guidelines (for GPU kernels) Do not directly use OpenACC or OpenMP directives directly. Instead, use the FYPP macros contained in src/common/include/parallel_macros.fpp @@ -136,7 +136,7 @@ which conforms to the Doxygen Fortran format. - Example: `@:ASSERT(predicate, message)` - **Error Reporting**: - - Use `s_mpi_abort()` for error termination, not `stop` + - Use `s_mpi_abort(error_message)` for error termination, not `stop` - No `stop` / `error stop` inside device code # 8 Memory Management From 3c10b13330fd3c185b1ae59a999396a1e8db7168 Mon Sep 17 00:00:00 2001 From: Spencer Bryngelson Date: Sat, 19 Jul 2025 14:59:53 -0400 Subject: [PATCH 4/7] Apply suggestion from @Copilot Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --- .cursor/rules/mfc-agent-rules.mdc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.cursor/rules/mfc-agent-rules.mdc b/.cursor/rules/mfc-agent-rules.mdc index 5bab4b7634..c8ab2e1162 100644 --- a/.cursor/rules/mfc-agent-rules.mdc +++ b/.cursor/rules/mfc-agent-rules.mdc @@ -62,7 +62,7 @@ Written primarily for Fortran/Fypp; the GPU and style sections matter only when --- -# 3 FYPP Macros for GPU acceleration Programming Guidelines (for GPU kernels) +# 3 FYPP Macros for GPU acceleration Programming Guidelines (for GPU kernels) Do not directly use OpenACC or OpenMP directives directly. Instead, use the FYPP macros contained in src/common/include/parallel_macros.fpp From 2a84897728a4d15ab52426edbb47056c3230c968 Mon Sep 17 00:00:00 2001 From: Spencer Bryngelson Date: Sat, 19 Jul 2025 15:00:17 -0400 Subject: [PATCH 5/7] Update mfc-agent-rules.mdc --- .cursor/rules/mfc-agent-rules.mdc | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/.cursor/rules/mfc-agent-rules.mdc b/.cursor/rules/mfc-agent-rules.mdc index c8ab2e1162..9d721bf1ef 100644 --- a/.cursor/rules/mfc-agent-rules.mdc +++ b/.cursor/rules/mfc-agent-rules.mdc @@ -1,7 +1,7 @@ ----- --description: Full MFC project rules – consolidated for Agent Mode --alwaysApply: true ----- +--- +description: Full MFC project rules – consolidated for Agent Mode +alwaysApply: true +--- # 0 Purpose & Scope Consolidated guidance for the MFC exascale, many-physics solver. From e8353f78115e1c9ed9b0f3b459c4f73a573dd762 Mon Sep 17 00:00:00 2001 From: Spencer Bryngelson Date: Sat, 19 Jul 2025 15:05:54 -0400 Subject: [PATCH 6/7] Update mfc-agent-rules.mdc --- .cursor/rules/mfc-agent-rules.mdc | 63 +++++++++++++++---------------- 1 file changed, 31 insertions(+), 32 deletions(-) diff --git a/.cursor/rules/mfc-agent-rules.mdc b/.cursor/rules/mfc-agent-rules.mdc index 9d721bf1ef..a4c3d3bb60 100644 --- a/.cursor/rules/mfc-agent-rules.mdc +++ b/.cursor/rules/mfc-agent-rules.mdc @@ -62,27 +62,7 @@ Written primarily for Fortran/Fypp; the GPU and style sections matter only when --- -# 3 FYPP Macros for GPU acceleration Programming Guidelines (for GPU kernels) - -Do not directly use OpenACC or OpenMP directives directly. -Instead, use the FYPP macros contained in src/common/include/parallel_macros.fpp - -Wrap tight loops with - -```fortran -$:GPU_PARALLEL_FOR(private='[...]', copy='[...]') -``` -* Add `collapse=n` to merge nested loops when safe. -* Declare loop-local variables with `private='[...]'`. -* Allocate large arrays with `managed` or move them into a persistent - `$:GPU_ENTER_DATA(...)` region at start-up. -* **Do not** place `stop` / `error stop` inside device code. -* Must compile with Cray `ftn` and NVIDIA `nvfortran` for GPU offloading; also build CPU-only with - GNU `gfortran` and Intel `ifx`/`ifort`. - ---- - -# 4 File & Module Structure +# 3 File & Module Structure - **File Naming**: - `.fpp` files: Fypp preprocessed files that get translated to `.f90` @@ -99,25 +79,44 @@ $:GPU_PARALLEL_FOR(private='[...]', copy='[...]') - `contains` section - Implementation of subroutines and functions -# 5 Fypp Macros and GPU Acceleration +--- + +# 4 Fypp Macros -## Use of Fypp - **Fypp Directives**: - Start with `#:` (e.g., `#:include`, `#:def`, `#:enddef`) - Macros defined in `include/*.fpp` files - Used for code generation, conditional compilation, and GPU offloading -## Some examples +--- + +# 5 FYPP Macros for GPU acceleration Programming Guidelines (for GPU kernels) + +- Do not use OpenACC or OpenMP directives directly. +- Instead, use the FYPP macros contained in `src/common/include/parallel_macros.fpp` +- Documentation on how to use the Fypp macros for GPU offloading is available at https://mflowcode.github.io/documentation/md_gpuParallelization.html + +Wrap tight loops with +```fortran +$:GPU_PARALLEL_FOR(private='[...]', copy='[...]') +``` +* Add `collapse=n` to merge nested loops when safe. +* Declare loop-local variables with `private='[...]'`. +* Allocate large arrays with `managed` or move them into a persistent + `$:GPU_ENTER_DATA(...)` region at start-up. +* **Do not** place `stop` / `error stop` inside device code. +* Must compile with Cray `ftn` or NVIDIA `nvfortran` for GPU offloading; also build CPU-only with + GNU `gfortran` and Intel `ifx`/`ifort`. -Documentation on how to use the Fypp macros for GPU offloading is available at https://mflowcode.github.io/documentation/md_gpuParallelization.html +- Example GPU macros include the below, among others: + - `$:GPU_ROUTINE(parallelism='[seq]')` - Marks GPU-callable routines + - `$:GPU_PARALLEL_LOOP(collapse=N)` - Parallelizes loops + - `$:GPU_LOOP(parallelism='[seq]')` - Marks sequential loops + - `$:GPU_UPDATE(device='[var1,var2]')` - Updates device data + - `$:GPU_ENTER_DATA(copyin='[var]')` - Copies data to device + - `$:GPU_EXIT_DATA(delete='[var]')` - Removes data from device -Some examples include: -- `$:GPU_ROUTINE(parallelism='[seq]')` - Marks GPU-callable routines -- `$:GPU_PARALLEL_LOOP(collapse=N)` - Parallelizes loops -- `$:GPU_LOOP(parallelism='[seq]')` - Marks sequential loops -- `$:GPU_UPDATE(device='[var1,var2]')` - Updates device data -- `$:GPU_ENTER_DATA(copyin='[var]')` - Copies data to device -- `$:GPU_EXIT_DATA(delete='[var]')` - Removes data from device +--- # 6 Documentation Style From 2b8c79693780952a203d775fe3d16dbe720c17e9 Mon Sep 17 00:00:00 2001 From: Spencer Bryngelson Date: Sat, 19 Jul 2025 15:08:46 -0400 Subject: [PATCH 7/7] Apply suggestion from @Copilot Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --- .cursor/rules/mfc-agent-rules.mdc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.cursor/rules/mfc-agent-rules.mdc b/.cursor/rules/mfc-agent-rules.mdc index a4c3d3bb60..ea67445ac4 100644 --- a/.cursor/rules/mfc-agent-rules.mdc +++ b/.cursor/rules/mfc-agent-rules.mdc @@ -90,7 +90,7 @@ Written primarily for Fortran/Fypp; the GPU and style sections matter only when --- -# 5 FYPP Macros for GPU acceleration Programming Guidelines (for GPU kernels) +# 5 FYPP Macros for GPU Acceleration Programming Guidelines (for GPU kernels) - Do not use OpenACC or OpenMP directives directly. - Instead, use the FYPP macros contained in `src/common/include/parallel_macros.fpp`