07_HandsOnMPASA/07_HandsOnMPASA.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "746d1042-9b52-4e9f-bce4-0bcd7eb15d48",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "![NCAR UCAR Logo](../NCAR_CISL_NSF_banner.jpeg)\n",
    "# Hands-On Session Using OpenACC in MPAS-A\n",
    "\n",
    "By: Daniel Howard [dhoward@ucar.edu](mailto:dhoward@ucar.edu), Consulting Services Group, CISL & NCAR \n",
    "\n",
    "Date: April 28th 2022"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0ca1ca32-c6ce-4ac3-8aa4-0fa7771c6560",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "source": [
    "In this notebook, we explore the GPU enabled [MPAS-A](http://mpas-dev.github.io/atmosphere/OpenACC/index.html) (Model Prediction Across Scales-Atmosphere) to apply techniques learned from MiniWeather and implementing OpenACC to develop for GPUs. \n",
    "\n",
    "* Review of exercises from prior OpenACC/MiniWeather sessions Part 1 and Part 2\n",
    "* MPAS-Atmosphere model overview\n",
    "* Managing GPU data in large software projects\n",
    "* Assessing performance of extracted GPU kernels in MPAS-A"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dfdb6b56-f032-42a0-a6c5-84ccece39ad8",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    " Head to the [NCAR JupyterHub portal](https://jupyterhub.hpc.ucar.edu/stable) and __start a JupyterHub session on Casper login__ (or batch nodes using 1 CPU, no GPUs) and open the notebook in `07_HandsOnMPASA/07_HandsOnMPASA.ipynb`. Be sure to clone (if needed) and update/pull the NCAR GPU_workshop directory.\n",
    "\n",
    "```shell\n",
    "# Use the JupyterHub GitHub GUI on the left panel or the below shell commands\n",
    "git clone git@github.com:NCAR/GPU_workshop.git\n",
    "git pull\n",
    "```\n",
    "\n",
    "# Workshop Etiquette\n",
    "* Please mute yourself and turn off video during the session.\n",
    "* Questions may be submitted in the chat and will be answered when appropriate. You may also raise your hand, unmute, and ask questions during Q&A at the end of the presentation.\n",
    "* By participating, you are agreeing to [UCAR’s Code of Conduct](https://www.ucar.edu/who-we-are/ethics-integrity/codes-conduct/participants)\n",
    "* Recordings & other material will be archived & shared publicly.\n",
    "* Feel free to follow up with the GPU workshop team via Slack or submit support requests to [support.ucar.edu](https://support.ucar.edu)\n",
    "    * Office Hours: Asynchronous support via [Slack](https://ncargpuusers.slack.com) or schedule a time with an organizer"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "75e08459-56ad-4ee5-bea7-9a988b0e15c0",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "## Notebook Setup\n",
    "Set the `PROJECT` code to a currently active project, ie `UCIS0004` for the GPU workshop, and `QUEUE` to the appropriate routing queue depending on if during a live workshop session (`gpuworkshop`), during weekday 8am to 5:30pm MT (`gpudev`), or all other times (`casper`). Due to limited shared GPU resources, please use `GPU_TYPE=gp100` during the workshop. Otherwise, set `GPU_TYPE=v100` (required for `gpudev`) for independent work. See [Casper queue documentation](https://arc.ucar.edu/knowledge_base/72581396#StartingCasperjobswithPBS-Concurrentresourcelimits) for more info.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "38ce23ae-c614-4c46-ab85-c80484f0655d",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "export PROJECT=UCIS0004\n",
    "export QUEUE=gpudev\n",
    "export GPU_TYPE=v100"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e25bb660-24a0-4b0c-9abd-e8b72839fa01",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "## Review of MiniWeather Performance Optimization\n",
    "At the end of last session, it was suggested to use `async` and predominantly `collapse` clauses to achieve optimal performance in MiniWeather kernels. Using `NX=1024` and `NZ=512`, the most expensive kernel in terms of compute time was at [__Line 231__](../05_DirectivesOpenACC/fortran/miniWeather_mpi_openacc.F90#L231) in the `semi_discrete_step` subroutine, with `NVCOMPILER_ACC_TIME` statistics highlighted below:\n",
    "\n",
    "```shell\n",
    "/glade/u/home/dhoward/GPU_workshop/05_DirectivesOpenACC/fortran/miniWeather_mpi_exercise2.F90 # Source file with OpenACC kernel code\n",
    "semi_discrete_step  NVIDIA  devicenum=0              # Name of subroutine from which kernel is launched\n",
    "time(us): 62,147\n",
    "257: compute region reached 924 times                # Specific line number for GPU kernel and number times reached/launched\n",
    "257: kernel launched 924 times\n",
    "grid: [16384]  block: [128]                          # Arrangement of gang/worker/vector in terms of grids and blocks\n",
    "device time(us): total=62,147 max=70 min=66 avg=67   # Timing statistics of the GPU kernel\n",
    "elapsed time(us): total=76,527 max=87 min=80 avg=82  # Timing statistics of the CPU call (less accurate with asynchronous execution)\n",
    "257: data region reached 1848 times\n",
    "```\n",
    "\n",
    "The arrangement of __gang/worker/vector__ units is provided by __grid: [NUM_GANGS]__ and __block: [VECTOR_LENGTH x NUM_WORKERS]__. The number of workers was 1 in the previous case so is omitted."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "56673f67-12cb-45d6-a9f6-aea341492f9d",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "source": [
    "Running this version with the NVIDIA NSight Systems Profiler (discussed in later session), we can get a visual representation of the model runtime. You can download and view this profile using the [NVIDIA NSight Systems client](https://developer.nvidia.com/nsight-systems) by downloading (`SHIFT` + `RIGHT-CLICK`) [MW_baseline.nsys-rep](MW_baseline.nsys-rep) in this folder.\n",
    "![Profile of MiniWeather - Baseline](img/Profile_MiniWeather_Baseline.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3bbf3b06-f85d-4417-9ca8-1c1407480789",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "source": [
    "This timeline shows the kernels running on the GPU runtime in the upper __blue__ compute kernels, __pink__ device to host transfers, and __teal__ host to device transfers segments. The lower segments show the CPU runtime in __blue__ compute kernel launches, __red__ data directives/regions, and beige __wait/synchronize__ sections.\n",
    "\n",
    "The bright blue highlights the most expensive GPU kernel in the `semi_discrete_step` subroutine with the associated launch call from the CPU highlighted earlier in the timeline.\n",
    "\n",
    "![Profile of MiniWeather - Baseline timeline only](img/Profile_MiniWeather_Baseline_cropped.png)\n",
    "\n",
    "Since we used `async`, the GPU kernels run right after one another without any kernel launch/exit costs."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "035b6af5-4c1c-4c6a-9b42-eb4eaf7df984",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "source": [
    "If we did not use `async`, the profile would look like this ([MW_noasync.qdrep](MW_noasync.qdrep)) and time would be lost as the CPU waits between every kernel launch/exit.\n",
    "\n",
    "![Profile of MiniWeather - No async](img/Profile_MiniWeather_noasync.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5ae967af-8ba7-4b02-81e5-32f389501137",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "## MiniWeather - Testing different kernel launch configurations and clauses\n",
    "Recall the final exercise of the [prior MiniWeather session](../05_DirectivesOpenACC/05p2_openACC_miniWeather_Tutorial.ipynb) where we experimented with various launch configurations in the [miniWeather_mpi_exercise2.F90](../05_DirectivesOpenACC/fortran/miniWeather_mpi_exercise2.F90) source file for specific kernels.\n",
    "\n",
    "__Were you able to achieve any significant speed-up?__\n",
    " \n",
    "The next panels shows statistical results from some launch configuration experiments using parameters `_NX=1024`, `_NZ=512`, and `_SIM_TIME=10` and different clauses in place of `***` for the `semi_discrete_step` subroutine kernel. Note that `NUM_VARS=4`.\n",
    "\n",
    "```fortran\n",
    "    !$acc parallel loop *** async\n",
    "    do ll = 1 , NUM_VARS \n",
    "      !$acc loop ***\n",
    "      do k = 1 , nz\n",
    "        !$acc loop ***\n",
    "        do i = 1 , nx\n",
    "          state_out(i,k,ll) = state_init(i,k,ll) + dt * tend(i,k,ll)\n",
    "        enddo\n",
    "      enddo\n",
    "    enddo\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6f208ea7-76f9-4d7f-9dce-f89447b87b2f",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "source": [
    "1. __Using `worker/vector/seq` on each loop respectively, the profiler shows `grid: [1]  block: [32x4]`. Why is this arrangement the least performant?__\n",
    "\n",
    "| MiniWeather Kernel L231, `semi_discrete_step` | Total Device Time ($ \\mu s$) |\n",
    "|---|---|\n",
    "| BaseLine (on V100) - `collapse(3)` auto `vector_length(128)`        | 62,936      |\n",
    "| clause - `gang/worker/vector` on each loop resepctively     | 852,859      |\n",
    "| clause - `worker/vector/seq` (Move NUM_VARS innermost, seq)     | 2,271,059      |\n",
    "| clause - `gang/vector/seq` (Move NUM_VARS innermost, seq)     | 72,584      |"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "73ce818e-09c3-4fa1-a069-377866df0dac",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "source": [
    "2. __Did you find any better configurations for this or other kernels in MiniWeather? Explain why it performed better.__\n",
    "3. __Do you trust the compiler to make relatively optimal choices with minimal direction?__\n",
    "\n",
    "| MiniWeather Kernel L231, `semi_discrete_step` | Total Device Time ($ \\mu s$) |\n",
    "|---|---|\n",
    "| BaseLine (on V100) - `collapse(3)` auto `vector_length(128)`        | 62,936      |\n",
    "| clause - `collapse(3) vector_length(32)` | 100,797      | \n",
    "| clause - `collapse(3) vector_length(64)` | 63,010      |\n",
    "| clause - `collapse(3) vector_length(256)` | 62,990      |\n",
    "| clause - `collapse(3) vector_length(512)` | 63,032      |\n",
    "| clause - `collapse(3) vector_length(1024)` | 66,458      |"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1240c613-ba62-4e09-8008-6ac3043b1753",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "source": [
    "4. __For `tile()`, why do you think the `(32,1,NUM_VARS=4)` clause was closest to the most performant?__\n",
    "5. __Can you infer the condition that causes the `tile()` clause to produce incorrect results?__ Hint: What is the max warp size?\n",
    "\n",
    "| MiniWeather Kernel L231, `semi_discrete_step` | Total Device Time ($ \\mu s$) |\n",
    "|---|---|\n",
    "| BaseLine (on V100) - `collapse(3)` auto `vector_length(128)`        | 62,936      |\n",
    "| clause - `tile(32,32,NUM_VARS)` __INCORRECT__     | 26,992      |\n",
    "| clause - `tile(32,32,1)`     | 73,476      |\n",
    "| clause - `tile(32,8,NUM_VARS)`      | 77,124      |\n",
    "| clause - `tile(32,1,NUM_VARS)`      | 65,040      |\n",
    "| clause - `tile(1024,1,1)`      | 67,393      |\n",
    "| clause - `tile(128,1,NUM_VARS)`      | 66,421      |\n",
    "| clause - `tile(128,2,NUM_VARS)`      | 74,295      |\n",
    "| clause - `tile(128,4,NUM_VARS)` __INCORRECT__      | 35,999      |\n",
    "| clause - `tile(*,*,*)` -> 32,4,32   | 150,374      |"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3c156991-f82a-4b05-8f34-bff0ef320f55",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "## MPAS-Atmosphere Overview\n",
    "![Global Voronoi mesh](img/mpas-logo-mesh.jpeg)\n",
    "\n",
    "We will now look at a real world production model __[MPAS (Model Prediction Across Scales)](https://mpas-dev.github.io/)__, specifically the GPU version of the atmosphere core __MPAS-A__ and how this model leveraged OpenACC to refactor towards GPU devices.\n",
    "\n",
    "So far, __only the v6.x Atmosphere core has been ported to GPUs__ and is freely available to review via their [website](https://mpas-dev.github.io/atmosphere/OpenACC/index.html) and the stable [v6.x](https://github.com/MPAS-Dev/MPAS-Model/tree/atmosphere/v6.x-openacc) or v7.x [develop-openacc](https://github.com/MPAS-Dev/MPAS-Model/tree/atmosphere/develop-openacc) branches on GitHub. Some work has also been done on the MPAS-Ocean core given this [presentation](https://www.lanl.gov/org/padwp/adx/computational-physics/parallelcomputing/_assets/docs/2020-student-projects/Ashwath_PCSRI_Final_Presentation.pdf) by PhD student Ashwath Venkataraman.\n",
    "\n",
    "If you'd like a more complete overview of MPAS, how to run the model, and research applications, see the [2021 MPAS Virtual tutorial](https://www.mmm.ucar.edu/mpas-tutorial-agenda) page or the upcoming [2022 joint WRF/MPAS workshop](https://www.mmm.ucar.edu/events/workshops/2022)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "45375814-e803-4c61-9c8b-301c275b5c49",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "source": [
    "* Fully compressible non-hydrostatic equations written in flux form\n",
    "* Split-Explicit timestepping via 3rd Order Runge-Kutta, see [AMS Paper - Klemp, Skamarock, and Dudhia](https://journals.ametsoc.org/view/journals/mwre/135/8/mwr3440.1.xml)\n",
    "\n",
    "<img src=\"img/MPAS-grid_diagram.png\" alt=\"MPAS grid diagram\" style=\"width:300px;\"/>\n",
    "\n",
    "The MPAS-A kernels we will focus on computes coefficients for vertically implicit gravity-wave/acoustic computations needed for each Runge-Kutta timestep. The previosly linked paper, specifically section 2 and the appendix, covers this in depth with a broader overview given in the 2021 tutorial [Time Integration](https://www2.mmm.ucar.edu/projects/mpas/tutorial/Virtual2021/MPAS_dynamics_time_integration.pdf) presentation.\n",
    "\n",
    "__However, understanding the numerical physics at play is not required to port well written code to GPUs.__"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6f0269a4-bff7-41f6-b801-c379317c0b18",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "## Development Process of MPAS-A\n",
    "![Development Process](img/DevelopmentProcess.png)\n",
    "Courtesy of Raghu Raj Kumar, NVIDIA"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "78549c3c-052b-473b-8027-a5174bb48afb",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "source": [
    "Identifying an established iterative process for GPU development ahead of work performed significantly eases development cost and increases success outcomes.\n",
    "\n",
    "1. __Establish a baseline__, ensure working and accurate configuration with target hardware and external software.\n",
    "2. __Port the code__, using incremental addition of OpenACC, perhaps using tools for kernel extraction like [KGen](https://github.com/NCAR/KGen) (Fortran only) to allow separation of concerns.\n",
    "    * See [KGen Guide](https://ncar.github.io/kgendocs/overview.html) if interested\n",
    "3. __Optimize computationally expensive kernels__ individually via an analysis and profiling iterative process.\n",
    "4. __Check portability expectations__ are met and that code satisfies both CPU and GPU unit tests. \n",
    "    * Look for and eliminate any __GPU anti-patterns__ such as linked lists data structures or global memory variables which may cause excessive data movement.\n",
    "    * Repeat Steps 2-4 as needed.\n",
    "5. __Integrate changes into benchmarks and verification suite__, utilizing version control and ideally a continuous integration process."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0519c79f-f179-4a58-9716-434cd48cf275",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "#### MPAS-A Performance Baseline\n",
    "Getting an accurate baseline helps inform where to dedicate development effort. This can be measured using internal timing metrics or your preferred CPU profiler (like [TAU](https://www.cs.uoregon.edu/research/tau/home.php), [Arm Forge Map](https://developer.arm.com/tools-and-software/server-and-hpc/debug-and-profile/arm-forge/arm-map), _gprof_, etc), to __identifiy hotspots__ in the code.\n",
    "![MPAS Execution Time Baseline](img/MPAS_Baseline.png)\n",
    "\n",
    "From KISTI, Kim, Kang, & Joh [GPU Acceleration of MPAS Physics\n",
    "Schemes Using OpenACC](https://www2.cisl.ucar.edu/sites/default/files/2021-10/KISTI%20-%20Joh%2C%20Kang%2C%20%26%20Kim.pdf)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "603ad70a-447e-4a63-be79-02c31176dfe1",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "source": [
    "Specific dynamics/physics schemes were prioritized for GPU while some set for CPU.\n",
    "\n",
    "![CPU GPU Scheme Distribution](img/CPU_GPU_Schemes.png)\n",
    "\n",
    "A [lagged computation of radiation](https://mpas-dev.github.io/atmosphere/OpenACC/lagged_radiation.html) was established in order to utilize idle CPUs. Requires __manual tuning of load balancing__ between number of CPU and GPU tasks.\n",
    "\n",
    "![Lagged Radiation Timeline](img/lagged_radiation.svg)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a6c39c0d-d1a4-4144-b968-7a1642db071e",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "## Managing GPU Data in MPAS-A\n",
    "Recall that using `!$acc kernels ...` and similar directives will generate lists of variables needed to manage data movement for each compute region.\n",
    "```shell\n",
    "153, Generating implicit copyin(rdzu(:)) [if not already present]\n",
    "         Generating implicit copyout(cofwr(:,:)) [if not already present]\n",
    "         Generating implicit copyin(p(:,:)) [if not already present]\n",
    "         Generating implicit copyout(cofwz(:,:)) [if not already present]\n",
    "         Generating implicit copyin(fzp(:),t(:,:),zz(:,:),fzm(:),cqw(:,:)) [if not already present]\n",
    "         Generating implicit copyout(coftz(:,:)) [if not already present]\n",
    "```\n",
    "These lists can be used and leveraged for your own data directives as GPU development progresses. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1f88edab-fd72-4c5a-9317-faf7c7b228f3",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "source": [
    "![Iterative Development from kernels to optimized OpenACC](img/GPU_Data_Iterations.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "102f5152-c5cd-4730-b75d-9e6b6619dfcd",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "source": [
    "Given ported kernels, MPAS-A was designed to create CPU and GPU data copies [at initialization](https://github.com/MPAS-Dev/MPAS-Model/blob/atmosphere/v6.x-openacc/src/core_atmosphere/physics/mpas_atmphys_vars.F) via `!$acc declare create(...)` and copy data at [unstructured data regions](https://github.com/MPAS-Dev/MPAS-Model/blob/atmosphere/v6.x-openacc/src/framework/mpas_pool_routines.F) via `!$acc enter data copyin(...)` prior to each kernel call. Then, each kernel would only require a `present(...)` clause using the prior variable lists. __Reference counters__ would mitigate excessive copies.\n",
    "\n",
    "```fortran\n",
    "!!! From mpas_atmphys_vars.F module\n",
    "real(kind=RKIND),dimension(:,:,:),allocatable:: &\n",
    "!... arrays related to u- and v-velocities interpolated to theta points:\n",
    "    u_p,              &!u-velocity interpolated to theta points            [m/s]\n",
    "    v_p                !v-velocity interpolated to theta points            [m/s]\n",
    "!$acc declare create(u_p, v_p)\n",
    "```\n",
    "\n",
    "Any lingering excessive data copies could be identified by profilers and fixed while other required copies for CPU algorithms & I/O were managed by `!$acc update` directives."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3c988f2d-689b-476d-8cc8-c3b7ba923dda",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "## MPAS-A Kernel Extraction\n",
    "We will focus on the `atm_compute_vert_imp_coefs_work` subroutine and kernels as extracted by [Supreeth Suresh](https://staff.ucar.edu/users/ssuresh), TDD/ASAP in CISL. This is the [link, Line 2641](https://github.com/MPAS-Dev/MPAS-Model/blob/ff0e97f8de30c06adbc751f3808f246b33281dd0/src/core_atmosphere/dynamics/mpas_atm_time_integration.F#L2641) to the source subroutine in the full model codebase and in this workshop directory is the the extracted set of kernels [mpas_atm_compute_vert_imp_coefs_work.F90](mpas_atm_compute_vert_imp_coefs_work.F90).\n",
    "\n",
    "Assuming data locality is resolved, this extracted kernel simply utilizes __randomized input data__ as we will be __focusing on optimizing the performance__ of the subroutine's kernels. The kernel is run in a repeating loop so we can get a relatively consistent average of measured performance. A validation tool has not been included at this time but is typically highly recommended.\n",
    "\n",
    "For large codebases, building and/or using an automated tool like NCAR's [KGen](https://github.com/NCAR/KGen) for Fortran codes or [Kernel Tuner](https://github.com/benvanwerkhoven/kernel_tuner) from NL eScience Center for CUDA/OpenCL codes will likely speed up the development/optimization process."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e0f8f9f1-1e59-4f52-b90a-edd127762330",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "## EXERCISE: MPAS-A Kernel Optimization\n",
    "Open the [mpas_atm_compute_vert_imp_coefs_work.F90](mpas_atm_compute_vert_imp_coefs_work.F90) source file and convert the `!$acc kernels` loops to optimized `!$acc parallel ...` compute constructs. Analyze each set of loops and apply appropriate sets of kernel configuraton clauses to achieve improved performance. Note: `!DIR$ IVDEP` tells compiler to ignore loop dependencies for serial vector SIMD compilations.\n",
    "\n",
    "You are encouraged to reference the initial attempts at optimization done by the `!$acc kernels` directive output during the compilation process. Data management has already been done for you using `-gpu=managed` and `present(var-list)`/`create(var-list)` clauses.\n",
    "\n",
    "__Record results of your optimization experiments on a chosen kernel__ and try to determine optimal configurations for that kernel. Compare your achieved performance with the [original at Line 2641](https://github.com/MPAS-Dev/MPAS-Model/blob/ff0e97f8de30c06adbc751f3808f246b33281dd0/src/core_atmosphere/dynamics/mpas_atm_time_integration.F#L2641). Work on other kernels as time allows. Note that most kernels may benefit from similar clause specifications since they operate on similar domain sizes/variables."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bcc7f1e6-5e95-48f1-b106-081aed2956be",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "module load nvhpc/22.2 &> /dev/null\n",
    "export _OPENACC=true\n",
    "make"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a5e61fd7-a29f-44c5-b81a-c94cc286f9c1",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "qcmd -A $PROJECT -q $QUEUE -l select=1:ncpus=1:ngpus=1 -l gpu_type=$GPU_TYPE -l walltime=20 -v NVCOMPILER_ACC_TIME=1 -- \\\n",
    "`pwd`/vert_implicit_coefs.exe"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a7941d0b-6d08-4b41-b729-405876d99dfe",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    },
    "tags": []
   },
   "source": [
    "| MPAS-A Kernels L### | Device Time ($ \\mu s $) |\n",
    "|---|---|\n",
    "| BaseLine (on V100) - `!$acc kernels` | XX      |\n",
    "| clause - `gang/vector`               | XX      |\n",
    "| clause - `tile(##,##)`               | XX      |\n",
    "| clause - `tile(*,*)`                 | XX      |\n",
    "| clause - vector_length(XX)           | XX      |\n",
    "| clause - num_workers(XX)             | XX      |\n",
    "| ...                                  | XX      |"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a7990a65-f240-45c7-a668-530a54271620",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "## Final Points\n",
    "1. Plan for and __commit to a defined iterative GPU development process__ to remove pain points and manage long term goals of your code project\n",
    "    * __Smaller, validated incremental changes__ are easier to debug\n",
    "2. __Start with descriptive__ `!$acc kernels` __then add prescriptive__ `!$acc parallel ...` kernels as needed for expensive kernels\n",
    "    * `!$acc kernels` can still achieve meaningful performance alone\n",
    "3. Understand that the GPU development process takes time and effort but __specific tools/techniques can drastically speed up development time__."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "80240bea-4ca0-4e35-aee5-3f44b637bbe6",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "## Suggested Resources\n",
    "* [2021 MPAS Virtual tutorial](https://www.mmm.ucar.edu/mpas-tutorial-agenda)\n",
    "* Computers & Geosciences, [GPU acceleration of MPAS microphysics WSM6 using OpenACC directives: Performance and verification](https://www.sciencedirect.com/science/article/pii/S0098300420306051) by J. Kim, J. Kang, and M. Joh (KISTI)\n",
    "* OpenACC.org and NVIDIA managed GitHub, presentations, and learning materials [GPU Bootcamps](https://github.com/gpuhackathons-org/gpubootcamp/tree/master/hpc)\n",
    "    * Lab sequence on [OpenACC](https://github.com/gpuhackathons-org/gpubootcamp/tree/master/hpc/openacc)\n",
    "    * Lab sequence on [Profiling Tools with MiniWeather](https://github.com/gpuhackathons-org/gpubootcamp/tree/master/hpc/miniprofiler) \n",
    "    * Lab sequence on [Various GPU Programming Paradigms (CUDA, OpenACC, stdPar, OpenMP)](https://github.com/gpuhackathons-org/gpubootcamp/tree/master/hpc/nways)\n",
    "    * Lab sequence on [Multi-GPU Programming](https://github.com/gpuhackathons-org/gpubootcamp/tree/master/hpc/multi_gpu_nways)\n",
    "    * Lab sequences on [GPU AI with CFD, PINNs, and Climate models](https://github.com/gpuhackathons-org/gpubootcamp/tree/master/hpc_ai)\n",
    "    \n",
    "After this session, we will have three weeks until the next workshop. Order of upcoming sessions will also be adjusted to accomodate availability of a NVIDIA engineer to present on Multi-GPU programming. Look out for upcoming announcements."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Bash",
   "language": "bash",
   "name": "bash"
  },
  "language_info": {
   "codemirror_mode": "shell",
   "file_extension": ".sh",
   "mimetype": "text/x-sh",
   "name": "bash"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}