14_DaskGPU/14_DaskGPU-completed.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "e6597bb3-15b6-4639-82ed-841386591567",
   "metadata": {},
   "source": [
    "![NCAR UCAR Logo](../NCAR_CISL_NSF_banner.jpeg)\n",
    "# Using Dask with GPUs via CuPy and introducing cuDF\n",
    "**NCAR GPU Workshop  \n",
    "August 11th 2022**\n",
    "\n",
    "**Presenter:**  \n",
    "Brian Vanderwende  \n",
    "[vanderwb@ucar.edu](mailto:vanderwb@ucar.edu)  \n",
    "Consulting Services Group"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b77cb026-61af-4635-b40d-3ed940d9c4c9",
   "metadata": {},
   "source": [
    "## Obtaining this notebook\n",
    "Follow these steps to create a clone of this repo. If you have already done so in a prior session, simply run a pull operation to obtain the materials for this week.\n",
    "\n",
    "| JupyterLab | Terminal |\n",
    "|-|-|\n",
    "| Use the JupyterHub GitHub GUI on the left</br>panel or the below shell commands | `git clone git@github.com:NCAR/GPU_workshop.git`</br>`git pull` |\n",
    "\n",
    "Once cloned or updated, navigate to `GPU_workshop/13_DaskGPU` in the repo and open this notebook.\n",
    "\n",
    "*For this session, we do not have the resources allocated for attendees to follow along during the live presentation, as we will demonstrate scaling and memory effects and thus need to use V100s. Instructions are provided at the end of this notebook for running it independently after the presentation.*"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "300697eb-0d7a-4e09-80fd-869a2d502971",
   "metadata": {},
   "source": [
    "## Workshop Etiquette\n",
    "* Please mute yourself and turn off video during the session.\n",
    "* Questions may be submitted in the chat and will be answered when appropriate. You may also raise your hand, unmute, and ask questions during Q&A at the end of the presentation.\n",
    "* By participating, you are agreeing to [UCAR’s Code of Conduct](https://www.ucar.edu/who-we-are/ethics-integrity/codes-conduct/participants)\n",
    "* Recordings & other material will be archived & shared publicly.\n",
    "* Feel free to follow up with the GPU workshop team via Slack or submit support requests to [support.ucar.edu](https://support.ucar.edu)\n",
    "    * Office Hours: Asynchronous support via [Slack](https://ncargpuusers.slack.com) or schedule a time with an organizer\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d519440c-5da3-4d0a-a930-f56aa6a04d89",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Recall from last session...\n",
    "\n",
    "We explored the **CuPy** package, which provides a GPU drop-in replacement for *most* **Numpy** functionality ([comparison table](https://docs.cupy.dev/en/stable/reference/comparison.html)).\n",
    "\n",
    "### **CuPy**: creating arrays of data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "37561ab0-c0c0-4ec1-b06c-245daea3047d",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import cupy as cp"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "73c1a336-bdc6-40e9-af47-fa1177a9438b",
   "metadata": {},
   "source": [
    "Creating a random Gaussian array on the CPU with Numpy ..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "60bb7e8b-d094-405d-965d-173b132ea6c1",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Mem used: 762.94 MiB\n"
     ]
    }
   ],
   "source": [
    "cpu_arr = np.random.randn(10_000, 10_000)\n",
    "print(f'Mem used: {cpu_arr.nbytes / 1024**2:.2f} MiB')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8fd7b37b-e910-4e33-9baa-5f64f3ad6c11",
   "metadata": {},
   "source": [
    "And creating the same array on the GPU ..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "c413692a-9734-4a36-9019-35a429bb8a63",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Mem used: 762.94 MiB\n"
     ]
    }
   ],
   "source": [
    "gpu_arr = cp.random.randn(10_000, 10_000)\n",
    "print(f'Mem used: {gpu_arr.nbytes / 1024**2:.2f} MiB')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eafaa657-8746-4ebc-85ac-8f9c8cbd897b",
   "metadata": {},
   "source": [
    "#### Comparing the speed of `.mean()`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "b51cad4d-2095-406f-b7cd-594851f7e393",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 65.6 ms, sys: 1.2 ms, total: 66.8 ms\n",
      "Wall time: 65.8 ms\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "0.00010802916697213106"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%time\n",
    "cpu_arr.mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "a221ca21-2934-4926-a90a-2c02c42d9ec2",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 411 µs, sys: 146 µs, total: 557 µs\n",
      "Wall time: 451 µs\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "array(-2.11241951e-05)"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%time\n",
    "gpu_arr.mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a78917b8-e146-4b96-8c94-15b324f1c5ae",
   "metadata": {},
   "source": [
    "#### Checking GPU results for correctness"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "99f19a2e-5a47-43f2-972a-07dbd822236a",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 154 ms, sys: 398 ms, total: 552 ms\n",
      "Wall time: 551 ms\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "gpu_copy = cp.array(cpu_arr)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "29b9c0ae-910c-490c-9ab8-a5b875a4e7e7",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 340 µs, sys: 181 µs, total: 521 µs\n",
      "Wall time: 390 µs\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "array(0.00010803)"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%time\n",
    "gpu_copy.mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5dec6c5f-0914-4add-bdcb-9a220106fe52",
   "metadata": {},
   "source": [
    "#### Analyzing CuPy's memory pool\n",
    "\n",
    "Once GPU memory is allocated to an object, CuPy will expand it's *memory pool* by at least that amount. When an object is deallocated, the memory pool will hold on to that space in memory for faster subsequent operations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "75ac79a8-60f6-41dd-96ee-ba09891ac5d2",
   "metadata": {},
   "outputs": [],
   "source": [
    "mempool = cp.get_default_memory_pool()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "580ac30b-c4d8-458d-8b38-85711e851571",
   "metadata": {},
   "outputs": [],
   "source": [
    "def cupy_mem_stats():\n",
    "    print(f'GPU: used memory = {mempool.used_bytes() / 1024**2:.2f} MiB')\n",
    "    print(f'GPU: total pool  = {mempool.total_bytes() / 1024**2:.2f} MiB')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "0dd07a83-3bba-4f2d-aea3-dc8a6856bdb3",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "GPU: used memory = 1525.88 MiB\n",
      "GPU: total pool  = 1525.88 MiB\n"
     ]
    }
   ],
   "source": [
    "cupy_mem_stats()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4352fca7-72ac-4724-8d86-4f7923bac772",
   "metadata": {},
   "source": [
    "Let's release our data for garbage collection. This will free up space in CuPy's memory pool on the GPU."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "b5585921-d3fc-4627-8082-18861be3881f",
   "metadata": {},
   "outputs": [],
   "source": [
    "del cpu_arr, gpu_arr, gpu_copy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "449b2f11-983a-474f-a4bc-fcb7b6eb5a74",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "GPU: used memory = 0.00 MiB\n",
      "GPU: total pool  = 1525.88 MiB\n"
     ]
    }
   ],
   "source": [
    "cupy_mem_stats()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7b087252-0fe0-49c9-a286-09bf30afef64",
   "metadata": {},
   "source": [
    "We can also explicitly release all GPU memory from CuPy's pool. This step can be useful if utilizing other GPU libraries in your workflow."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "56898727-8da5-4283-9bf2-6e3446812215",
   "metadata": {},
   "outputs": [],
   "source": [
    "mempool.free_all_blocks()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "b5e14be9-08e4-4c88-831b-beb523626abd",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "GPU: used memory = 0.00 MiB\n",
      "GPU: total pool  = 0.00 MiB\n"
     ]
    }
   ],
   "source": [
    "cupy_mem_stats()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6f549db0-c793-4e73-a806-d774687b7bda",
   "metadata": {},
   "source": [
    "#### Let's track performance and memory usage for something bigger ...\n",
    "\n",
    "Rather than load a big data file, we will use the following function to produce a facimile temperature dataset. Feel free to play with the array size but be aware of the memory footprint!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "774e9246-15f6-4a63-8819-597287129007",
   "metadata": {},
   "outputs": [],
   "source": [
    "def aqua_gen(nx, ny, nt):\n",
    "    \"\"\"Takes dimensions nx, ny, nt and returns an ndarray with facimile dataset.\"\"\"\n",
    "    \n",
    "    t = np.random.random((nx, ny, nt))\n",
    "    t = t + np.linspace(287, 289, nt)\n",
    "    t = t + np.sin(np.linspace(0.0, np.pi * nt, nt))\n",
    "    t = t + (np.sin(np.linspace(0.0, np.pi, ny)) * 20 - 10)[None,:,None]\n",
    "    \n",
    "    return t"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0bb35767-ec20-42b4-a825-aa4d8a000b7e",
   "metadata": {},
   "source": [
    "First, we will use the function to create a NumPy array of decent size"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "db53a903-1088-4658-a5d4-fb27639ab02e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Size: 8.34 GiB\n"
     ]
    }
   ],
   "source": [
    "t = aqua_gen(4000, 2000, 140)\n",
    "print(f'Size: {t.nbytes / 1024**3:.2f} GiB')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8a02945f-0bab-450d-ac7a-16ddf2d9cb72",
   "metadata": {},
   "source": [
    "We've produced a dataset that looks something like surface temperatures across an aquaplanet. (*at least one without weather!*)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "5c257cfe-f3ce-4f19-b024-a02217b44142",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.contour.QuadContourSet at 0x2afc3a4d76d0>"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX0AAAD4CAYAAAAAczaOAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8qNh9FAAAACXBIWXMAAAsTAAALEwEAmpwYAAAYnElEQVR4nO3df8xc1X3n8fenBhtKbH7UEFw/JjjUgbWt1MVeyjY/xG6S4qCqkCpRjdRCdiM5pFA1m6600FQJuxJSt1sSLepC5RQUWKVQN5RCS0jj0vzQSqT0MXHABhzs8OvBxm5DEihpH8fm2z/mTLke7p2583vG5/OSRnPnO+ec+71n7nw9vnOfO4oIzMwsDz8x7gTMzGx0XPTNzDLiom9mlhEXfTOzjLjom5ll5LhxJ9DJwp84MU5csHjcaZiZTZWXD//DP0bE6a3xiS/6Jy5YzC8s/dC40zAzmypffvHmZ8viPrxjZpYRF30zs4xM/OGdOGEhh86bGXcaZmbT5cXysD/pm5llxEXfzCwjLvpmZhlx0Tczy4iLvplZRlz0zcwy4qJvZpYRF30zs4y46JuZZWTi/yL3yAnih+csOip28t55fnjOIk7eO39UvDXW7FdsX7xvHa+5XOxbbNO6nrI+rWO3jl/2uCzHqnatqrapXdtOsbq5V81VL23L2nfanqoxmqpe/6r2rf2qxquTQ519odt9pd2c1O1XttxuXtq9b8rGa/Ype1zVvrVNWQ5V29xu/LLHZdtd1afdXFW1KZuzdtvXqW/ZGFX15g05fa102tCk/zD6SaeviH936X8ddxpmZlNl+62/vT0iNrTGfXjHzCwjLvpmZhlx0Tczy4iLvplZRlz0zcwy4qJvZpYRF30zs4y46JuZZaRj0Zd0m6SDknYWYn8qaUe6PSNpR4qfLemfC8/9UaHPekmPSdoj6SZJGsoWmZlZpTqXYfg88IfAHc1ARPxqc1nSjcAPC+33RsS6knFuATYD3wS+BGwEHui08iML4ZW3jOffh8XPxhvWXRYbx/r6yaNu32K7dn0WP9v4q+66+dQdt1O+VeM0l4t5VeU4yHnsdrv6Mezxh5XDMF7vYSruS8X7QY8/aM39vUytyzBIOhv4q4hY2xIX8BzwnyLiqTbtlgFfjYjz0uPLgYsi4qOd1n3C8hXxlqs+0TFHMzN73Xc+9YmhXIbhXcCBiHiqEFsp6VuSvi7pXSm2HJgrtJlLsVKSNkualTR75NVX+0zRzMya+r3K5uXAnYXH+4GzIuJ7ktYDfyFpDVD2/5fK/2JExBZgCzQ+6feZo5mZJT0XfUnHAb8CrG/GImIemE/L2yXtBd5G45P9TKH7DLCv13WbmVlv+jm8817gyYj4t8M2kk6XtCAtvxVYBXw3IvYDr0i6MH0PcAVwbx/rNjOzHtQ5ZfNO4CHgXElzkj6SntrE0Yd2AN4NPCrp28AXgasi4qX03MeAPwb2AHupceaOmZkNVsfDOxFxeUX8wyWxu4G7K9rPAmvLnjMzs9HwX+SamWXERd/MLCMu+mZmGZnqoj9/1qGe+pT162WsUWnNrfl4knPuRdX2dLOdVXPVb9tp12nb6mx73fmZtnns9v00bdvXqtZlGMZp0cqZWHb9b447DTOzsVv03MLa/+g8++Frh3IZBjMzG5FB/C/DRd/MLCMu+mZmGXHRNzPLiIu+mVlGXPTNzDLiom9mlhEXfTOzjLjom5llxEXfzCwjLvpmZhmZiqJ/9sw/HLVc9rgZK3vcet86Rjd5tPYrW1e75U6xTrlXjV82L1Xra52LsnbdrrtKWf9247VT9dqVbU/dfaTT9nfKp06sn/bdvh51xi1r1+71qVpfN69hN/NZt20321313qoas5f6UEc37/uyPu322arxW038BdcWn3tmrL/518adhpnZVPn6e2/s7YJrkm6TdFDSzkLsekkvSNqRbpcUnrtO0h5JuyVdXIivl/RYeu6m9APpZmY2QnUO73we2FgS/2xErEu3LwFIWk3jB9PXpD43S1qQ2t8CbAZWpVvZmGZmNkQdi35EfAN4qeZ4lwJ3RcR8RDwN7AEukLQMWBIRD0XjeNIdwGU95mxmZj3q54vcayQ9mg7/nJpiy4HnC23mUmx5Wm6Nm5nZCPVa9G8BzgHWAfuBG1O87Dh9tImXkrRZ0qyk2R//4Ec9pmhmZq16KvoRcSAijkTEa8DngAvSU3PAikLTGWBfis+UxKvG3xIRGyJiw/Gn/GQvKZqZWYmein46Rt/0AaB5Zs99wCZJiyStpPGF7cMRsR94RdKF6aydK4B7+8jbzMx6cFynBpLuBC4ClkqaAz4NXCRpHY1DNM8AHwWIiF2StgKPA4eBqyPiSBrqYzTOBDoReCDdzMxshDoW/Yi4vCR8a5v2NwA3lMRngbVdZWdmZgM1FZdhMDOzwXDRNzPLiIu+mVlGXPTNzDJyTBf995355NSuv9u+w24/Ks28usmv27ad2o9jbuquc9i5Tep+0Y1jYRuGaeIvrbx8zSnxG1vfOZZ1b3vxvL53oEGMMWrDznka5+RYNczXYtuL5wHTUYSPxX3yd9feX3ppZRd9M7NjUFXRP6YP75iZ2dFc9M3MMuKib2aWERd9M7OMuOibmWXERd/MLCMu+mZmGXHRNzPLiIu+mVlGXPTNzDLiom9mlhEXfTOzjHQs+pJuk3RQ0s5C7H9LelLSo5LukXRKip8t6Z8l7Ui3Pyr0WS/pMUl7JN0kSUPZIjMzq1Tnk/7ngY0tsW3A2oh4O/Ad4LrCc3sjYl26XVWI3wJsBlalW+uYZmY2ZB2LfkR8A3ipJfaViDicHn4TmGk3hqRlwJKIeCga13K+A7isp4zNzKxngzim/1+ABwqPV0r6lqSvS3pXii0H5gpt5lKslKTNkmYlzb76/UMDSNHMzACO66ezpE8Ch4EvpNB+4KyI+J6k9cBfSFoDlB2/r/z1lojYAmyBxo+o9JOjmZm9rueiL+lK4JeA96RDNkTEPDCflrdL2gu8jcYn++IhoBlgX6/rNjOz3vR0eEfSRuC/A78cET8qxE+XtCAtv5XGF7bfjYj9wCuSLkxn7VwB3Nt39mZm1pWOn/Ql3QlcBCyVNAd8msbZOouAbenMy2+mM3XeDfxPSYeBI8BVEdH8EvhjNM4EOpHGdwDF7wHMzGwE6py9c3lELIuI4yNiJiJujYifiYgVradmRsTdEbEmIn42Is6PiL8sjDMbEWsj4pyIuKZ5SKhbH1zySM9tu+k7qUa1DZ3W03z+WJjTbg1q23vpX9Wn7liT9nr1+h7t9zX44JJHSvv2U186PVeMDfN16DS2eqy9I7P27Qvj7vuXHhX74svn1560L758PtDbJJetp06s+LjsuWI+nbalU/5V41WN2y5etZ7WMdvl3G4u6uTRjW5fi3aP677Ww1T2GlTlVye3ftrU3fY6+0QvudV9T3W7v9fZd/upGb2sv3V7murOa9XcnHfW/u0RsaG1/VQWfTMza6+q6PvaO2ZmGXHRNzPLiIu+mVlGXPTNzDLiom9mlhEXfTOzjLjom5llxEXfzCwjLvpmZhlx0Tczy4iLvplZRlz0zcwy4qJvZpYRF30zs4y46JuZZcRF38wsIx2LvqTbJB2UtLMQO03SNklPpftTC89dJ2mPpN2SLi7E10t6LD13U/qBdDMzG6E6n/Q/D2xsiV0LPBgRq4AH02MkrQY2AWtSn5slLUh9bgE2A6vSrXVMMzMbsjo/jP4N4KWW8KXA7Wn5duCyQvyuiJiPiKeBPcAFkpYBSyLiofSD6HcU+piZ2Yj0ekz/zRGxHyDdn5Hiy4HnC+3mUmx5Wm6Nl5K0WdKspNnvv/RajymamVmrQX+RW3acPtrES0XElojYEBEbTj3N3zWbmQ1KrxX1QDpkQ7o/mOJzwIpCuxlgX4rPlMTNzGyEei369wFXpuUrgXsL8U2SFklaSeML24fTIaBXJF2Yztq5otDHzMxG5LhODSTdCVwELJU0B3wa+D1gq6SPAM8BHwKIiF2StgKPA4eBqyPiSBrqYzTOBDoReCDdzMxshNQ4mWZyrX37wrj7/qXjTsPMbKqcd9b+7RGxoTXub0nNzDLiom9mlhEXfTOzjLjom5llxEXfzCwjLvpmZhlx0Tczy4iLvplZRlz0zcwy4qJvZpYRF30zs4y46JuZZcRF38wsIy76ZmYZmfqi/8WXz+/quXbt+113nce9rL/Zp9i33bqGuY39tu9lDrp5HTu9BpNgUPtl2X4xLFX7V2sOZa9vMzbO16bOukbZpt9xu617RVNzPf1B7CAfXPLIvy0PY4f74JJHjhq39XGn9t2so5e+7dZbJ1egq+2rk0frmJ3aD6tQ9DonZX2g/TzV2U/qbGtxX2ius+4ctWvXzbp71fpeHOQ6i+16ndthtOvUb9Dv6d9de3/p9fQnvugvX3NK/MbWd447DTOzqVJV9Kf+8I6ZmdXXc9GXdK6kHYXby5I+Lul6SS8U4pcU+lwnaY+k3ZIuHswmmJlZXR1/GL1KROwG1gFIWgC8ANwD/GfgsxHxB8X2klYDm4A1wE8DfyPpbYUfTjczsyEb1OGd9wB7I+LZNm0uBe6KiPmIeBrYA1wwoPWbmVkNgyr6m4A7C4+vkfSopNsknZpiy4HnC23mUuwNJG2WNCtp9tXvHxpQimZm1nfRl7QQ+GXgz1LoFuAcGod+9gM3NpuWdC89dSgitkTEhojYcNKpC/tN0czMkkF80n8/8EhEHACIiAMRcSQiXgM+x+uHcOaAFYV+M8C+AazfzMxqGkTRv5zCoR1JywrPfQDYmZbvAzZJWiRpJbAKeHgA6zczs5p6PnsHQNJPAu8DPloI/76kdTQO3TzTfC4idknaCjwOHAau9pk7Zmaj1VfRj4gfAT/VEvv1Nu1vAG7oZ51mZtY7/0WumVlGXPTNzDLiom9mlhEXfTOzjLjom5llxEXfzCwjLvpmZhlx0Tczy4iLvplZRlz0zcwy4qJvR9n24nnjTsHMhqiva++MwsuHT+iqEL3vzCeB3ovX+858smPfZpvW+37WUzZWL+OWraeoznjt2nTKqez5OttaZ6xu2g9i7gZpkLl3s4/WiXczV4PIrR+9vOd6aVu3T1ktgHrvs27377ptXnd/aVQRpb9jMjEWn3tmrL/518adhpnZVPn6e2/cHhEbWuM+vGNmlhEXfTOzjLjom5llxEXfzCwjLvpmZhnpq+hLekbSY5J2SJpNsdMkbZP0VLo/tdD+Okl7JO2WdHG/yZuZWXcG8Un/P0bEusKpQdcCD0bEKuDB9BhJq4FNwBpgI3CzpAUDWL+ZmdU0jMM7lwK3p+XbgcsK8bsiYj4ingb2ABcMYf1mZlah36IfwFckbZe0OcXeHBH7AdL9GSm+HHi+0Hcuxd5A0mZJs5Jmf/yDH/WZopmZNfV7GYZ3RMQ+SWcA2yQ92aatSmKlfw4cEVuALdD4i9w+czQzs6SvT/oRsS/dHwTuoXG45oCkZQDp/mBqPgesKHSfAfb1s34zM+tOz0Vf0kmSFjeXgV8EdgL3AVemZlcC96bl+4BNkhZJWgmsAh7udf1mZta9fg7vvBm4R1JznD+JiC9L+ntgq6SPAM8BHwKIiF2StgKPA4eBqyPiSF/Zm5lZV3r+pB8R342In023NRFxQ4p/LyLeExGr0v1LhT43RMQ5EXFuRDzQy3qfmTu9bbx4X9W2dZy6fQaZZ/Fx6/qLt7L8yvKs2p6yNq1ty3KtWkdV26rtq2pTd53t+rdbb9Vr2mlO6sxvVd9O29Yut6oxOq23U7texq/Kud1+WdWv09h1X/Nu+rbbd6vmo2o9depEu9e/m3rVbv1Vy53GLTPxl1ZetHImll3/m+NOw8xsqjz74Wt9aWUzs9y56JuZZcRF38wsIy76ZmYZcdE3M8uIi76ZWUZc9M3MMuKib2aWERd9M7OMuOibmWXERd/MLCP9/ojK0OmQWPTcwqNi82cdOirW+rhO27pjzJ91CKBjDnXz62W8bp6vGr+bseq2ade+mUdrLnXzq2pTfK4sx9a+3cx36xyW5V42x1X7VCe99mvXv9/Xre5z7doW569Vnde92a7qdaiTR9V9Wb7tcigbsyrWOm7VflK2rrLtrLN/dFMLmyb+gmsnLF8Rb7nqE+NOw8xsqnznU5/wBdfMzHLnom9mlhEXfTOzjLjom5llpJ8fRl8h6auSnpC0S9Jvpfj1kl6QtCPdLin0uU7SHkm7JV08iA0wM7P6+jll8zDw2xHxiKTFwHZJ29Jzn42IPyg2lrQa2ASsAX4a+BtJb/OPo5uZjU4/P4y+PyIeScuvAE8Ay9t0uRS4KyLmI+JpYA9wQa/rNzOz7g3kmL6ks4GfA/4uha6R9Kik2ySdmmLLgecL3eZo/4+EmZkNWN9FX9KbgLuBj0fEy8AtwDnAOmA/cGOzaUn30r8Mk7RZ0qyk2SOvvtpvimZmlvRV9CUdT6PgfyEi/hwgIg5ExJGIeA34HK8fwpkDVhS6zwD7ysaNiC0RsSEiNiw46aR+UrRj0OJnJ/uvyCeN58uKev4iV5KAW4EnIuIzhfiyiNifHn4A2JmW7wP+RNJnaHyRuwp4uNN6FhzyTmtv5H2iO54va+rn7J13AL8OPCZpR4r9DnC5pHU0Dt08A3wUICJ2SdoKPE7jzJ+rfeaOmdlo9Vz0I+L/U36c/ktt+twA3NDrOs3MrD/+i1wzs4y46JuZZcRF38wsIy76ZmYZcdE3M8uIi76ZWUZc9M3MMjI1Rf/kvfMdl9vFunm+17xG0W/QTt47XyuXQec7iPFa94NhzWmvY0/KvtHr+2FS8hj2etu1a431834ZZd1p97wiJvvPs5csnol/v+HqcadhZjZV/vZrv7M9Ija0xqfmk76ZmfXPRd/MLCMu+mZmGXHRNzPLiIu+mVlGXPTNzDLiom9mlhEXfTOzjLjom5llpJ/fyB0J/cshFj45N/BxD50309e4rf37HW/Yhp1v2fjASOekmUPZthVjvWz7tL3enVTlP6j3Rb/z3Us+g3hNpvF17TbnkV+GQdJG4P8AC4A/jojfa9f+5OPPiF9Y+qGR5GZmdqz48os3j/8yDJIWAP8XeD+wGrhc0upR5mBmlrNRH9O/ANgTEd+NiEPAXcClI87BzCxboz6mvxx4vvB4Dvj51kaSNgOb08P5L794884R5NavpcA/jjuJDqYhR3Ceg+Y8B2ta8nxLWXDURV8lsTd8qRARW4AtAJJmy45LTZppyHMacgTnOWjOc7CmJc8qoz68MwesKDyeAfaNOAczs2yNuuj/PbBK0kpJC4FNwH0jzsHMLFsjPbwTEYclXQP8NY1TNm+LiF0dum0ZfmYDMQ15TkOO4DwHzXkO1rTkWWrify7RzMwGx5dhMDPLiIu+mVlGJrboS9ooabekPZKunYB8npH0mKQdkmZT7DRJ2yQ9le5PLbS/LuW+W9LFQ8zrNkkHJe0sxLrOS9L6tH17JN0kqez02kHneb2kF9Kc7pB0yTjzlLRC0lclPSFpl6TfSvGJms82eU7afJ4g6WFJ3055/o8Un7T5rMpzouZzYCJi4m40vuTdC7wVWAh8G1g95pyeAZa2xH4fuDYtXwv8r7S8OuW8CFiZtmXBkPJ6N3A+sLOfvICHgf9A428pHgDeP4I8rwf+W0nbseQJLAPOT8uLge+kXCZqPtvkOWnzKeBNafl44O+ACydwPqvynKj5HNRtUj/pT8vlGi4Fbk/LtwOXFeJ3RcR8RDwN7KGxTQMXEd8AXuonL0nLgCUR8VA09tw7Cn2GmWeVseQZEfsj4pG0/ArwBI2/Ip+o+WyTZ5Vx5RkR8U/p4fHpFkzefFblWWVs76NBmNSiX3a5hnY79SgE8BVJ29W4TATAmyNiPzTeiMAZKT7u/LvNa3labo2PwjWSHk2Hf5r/zR97npLOBn6Oxqe+iZ3PljxhwuZT0gJJO4CDwLaImMj5rMgTJmw+B2FSi36tyzWM2Dsi4nwaVwi9WtK727SdxPyhOq9x5XsLcA6wDtgP3JjiY81T0puAu4GPR8TL7ZpW5DOuPCduPiPiSESso/HX9xdIWtum+aTlOXHzOQiTWvQn7nINEbEv3R8E7qFxuOZA+i8d6f5gaj7u/LvNay4tt8aHKiIOpDfba8DneP0Q2NjylHQ8jUL6hYj48xSeuPksy3MS57MpIn4AfA3YyATOZ1mekzyf/ZjUoj9Rl2uQdJKkxc1l4BeBnSmnK1OzK4F70/J9wCZJiyStBFbR+IJnVLrKK/0X+xVJF6azDa4o9Bma5hs/+QCNOR1bnmnMW4EnIuIzhacmaj6r8pzA+Txd0ilp+UTgvcCTTN58luY5afM5MOP+JrnqBlxC46yEvcAnx5zLW2l8W/9tYFczH+CngAeBp9L9aYU+n0y572aI3+ADd9L4r+ePaXzS+EgveQEbaOzUe4E/JP219pDz/H/AY8CjNN5Iy8aZJ/BOGv8dfxTYkW6XTNp8tslz0ubz7cC3Uj47gU/1+r4ZU54TNZ+DuvkyDGZmGZnUwztmZjYELvpmZhlx0Tczy4iLvplZRlz0zcwy4qJvZpYRF30zs4z8K2JOuESg/o+mAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "plt.contourf(t[:,:,0].transpose())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5e758c59-85a4-4213-81ad-4860dfd06d61",
   "metadata": {},
   "source": [
    "Now we can copy it to the GPU and track what happens with the memory usage and pool"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "55ecece0-a0d8-4ae3-b5c7-d09dc951d456",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "GPU: used memory = 0.00 MiB\n",
      "GPU: total pool  = 0.00 MiB\n",
      "GPU: used memory = 8544.92 MiB\n",
      "GPU: total pool  = 8544.92 MiB\n"
     ]
    }
   ],
   "source": [
    "cupy_mem_stats()\n",
    "tg = cp.asarray(t)\n",
    "cupy_mem_stats()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3080b7cb-1a92-4590-b516-2e4f51cf0001",
   "metadata": {},
   "source": [
    "Finally, let's compare the performance of an operation to convert to Celsius and take the time-mean at each grid point"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "a6e8b663-b3cf-4a96-88ae-52639c07d491",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 1.78 s, sys: 1.74 s, total: 3.51 s\n",
      "Wall time: 3.51 s\n"
     ]
    }
   ],
   "source": [
    "%time cpu_tm = np.mean((t - 273.15), axis = 2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "f8ae09f1-b474-4a2d-a97e-b7822f8af06b",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 1.06 ms, sys: 52 µs, total: 1.11 ms\n",
      "Wall time: 710 µs\n",
      "GPU: used memory = 8605.96 MiB\n",
      "GPU: total pool  = 17211.92 MiB\n"
     ]
    }
   ],
   "source": [
    "%time gpu_tm = cp.mean((tg - 273.15), axis = 2)\n",
    "cupy_mem_stats()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "37a4043b-13de-4e2f-9869-2680207868a1",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "GPU: used memory = 0.00 MiB\n",
      "GPU: total pool  = 0.00 MiB\n"
     ]
    }
   ],
   "source": [
    "del cpu_tm, gpu_tm, tg\n",
    "mempool.free_all_blocks()\n",
    "cupy_mem_stats()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "82eb2a0a-1752-4887-a158-385d8c87a3ce",
   "metadata": {},
   "source": [
    "**GPU memory is smaller than node memory - if we need to process very large arrays of data, we will need a tool to parallelize across multiple GPUs ...**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4a1a1aed-3090-46d6-946a-e4ce5a4aaece",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ef2fa115-4a27-4788-9935-c3d9b1f55e35",
   "metadata": {},
   "source": [
    "## Using Dask with GPUs\n",
    "\n",
    "Dask is a popular Python library for parallelizing computations across multiple workers. It provides incredibly easily scaling of code from problem sizes that fit on a dual-core laptop to those that can span an entire analysis cluster like Casper.\n",
    "\n",
    "* Dask is easy to install\n",
    "* Dask leverages high-level collections like NumPy arrays and pandas DataFrames\n",
    "* Dask can schedule workers itself or submit jobs to batch schedulers like PBS and Slurm\n",
    "* Dask workers can use GPUs\n",
    "\n",
    "**Dask allows you to scale up from problem sizes that fit in node *or GPU* memory to ones that fit on disk.**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2f59c010-5446-40c3-af50-e139c7ea38c9",
   "metadata": {},
   "source": [
    "### Dask fundamentals and how the GPU is supported\n",
    "\n",
    "Dask operates on either a high-level collection of data like an array, Series, or DataFrame, or a low-level collection of arbitrary code marked for either `delayed` or asynchronous (`futures`) parallel execution. In this tutorial, we will focus on high-level data structures.\n",
    "\n",
    "| Dask collection | CPU object | GPU object | GPUs |\n",
    "|-|-|-|-|\n",
    "| dask.array | numpy.array | cupy.array | NVIDIA, AMD |\n",
    "| dask.series | pandas.series | cudf.series | NVIDIA |\n",
    "| dask.dataframe | pandas.dataframe | cudf.dataframe | NVIDIA |\n",
    "\n",
    "Most Dask operations use *lazy evaluation*, meaning you tell Dask what you want the workers to do, but then they wait until you start the computation using `.compute()` (or similar methods). These instructions are collectively called a *task graph*, and then map out what Dask will do to get from your data collection to your desired output. Notably, worker memory is not consumed until you begin executing the task graph.  \n",
    "</br>\n",
    "<div style=\"text-align:center;\"><img src=images/dask-overview.svg width=800px alt=\"Dask Flow\"></img></br><i>Image credit: Anaconda, Inc. and contributors</i></div>  \n",
    "</br>  \n",
    "\n",
    "**Fundamentally, nothing about this workflow is different when using GPUs.**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e3a810e4-3e3a-4817-8d86-c4a18942377c",
   "metadata": {
    "jp-MarkdownHeadingCollapsed": true,
    "tags": []
   },
   "source": [
    "#### When might you use Dask with GPUs?\n",
    "\n",
    "While Dask is easy to use, even the simplest of parallel computing approaches can be tricky to get right. Here are some scenarios where scaling GPU resources with Dask can be powerful:\n",
    "\n",
    "1. You are lucky enough to have many GPU resources with low contention and can be bursty\n",
    "2. Your workflow benefits tremendously from GPU acceleration, but is too big for a single GPU\n",
    "3. A small segment of your analysis workflow can benefit from on-demand access of GPUs\n",
    "4. You are running a large application with MPI and wish to do between-cycle data work"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "59906c1d-9d2e-4112-85f1-10d7d9ab9190",
   "metadata": {},
   "source": [
    "#### Notable Dask packages\n",
    "\n",
    "Dask functions are broken into a diverse ecosystem of packages. Here is a list of notables for GPU usage:\n",
    "\n",
    "* **dask** - the core API, including collections and a basic task scheduler\n",
    "* **distributed** - cluster objects (groups of workers) that can span a network and detailed dashboards\n",
    "* **dask-jobqueue** - Dask HPC clusters (e.g., `PBSCluster`, `SLURMCluster`)\n",
    "* **dask-cuda** - tools to set up GPU-backed workers (especially on single nodes)\n",
    "* **dask-mpi** - construct clusters of workers using MPI processes (*can be used with dask-cuda*)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "69ed2699-e18f-494c-ab0a-7e17980e622f",
   "metadata": {},
   "source": [
    "### Using NVIDIA's LocalCUDACluster\n",
    "\n",
    "The `LocalCUDACluster` cluster object comes from the **dask-cuda** package. It isn't required to use **CuPy** and **cuDF** collections with Dask, but it eases pinning local GPUs to specific workers and provides functionality for GPU memory spilling and high-speed worker communication via UCX. It is developed as part of RAPIDS.  \n",
    "\n",
    "*UCX requires [compilation and customization](https://docs.rapids.ai/api/dask-cuda/stable/ucx.html) so we will avoid here for the sake of simplicity.*\n",
    "\n",
    "#### First, we need to start our cluster...\n",
    "\n",
    "The **dask-cuda** [API document](https://docs.rapids.ai/api/dask-cuda/stable/api.html#cluster) is useful here for cluster settings."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "92b7944c-8ea6-4a31-846b-817b31eee5f5",
   "metadata": {},
   "outputs": [],
   "source": [
    "import dask.array as da\n",
    "from dask.distributed import Client\n",
    "from dask_cuda import LocalCUDACluster"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9785645f-4e95-4dde-bf7e-927e39ccae1e",
   "metadata": {},
   "source": [
    "By default, it would spawn `len(CUDA_VISIBLE_DEVICES)` workers.  \n",
    "\n",
    "We will also associate our cluster object with a `Client`. The client is our runtime interface to the cluster and Dask scheduler."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "ae084c2b-b53e-4b77-a05e-623677d42577",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2022-08-11 10:29:39,195 - distributed.preloading - INFO - Import preload module: dask_cuda.initialize\n",
      "2022-08-11 10:29:39,232 - distributed.preloading - INFO - Import preload module: dask_cuda.initialize\n",
      "Unable to start CUDA Context\n",
      "Traceback (most recent call last):\n",
      "  File \"/glade/u/apps/opt/conda/envs/pygpu-dask/lib/python3.9/site-packages/dask_cuda/initialize.py\", line 31, in _create_cuda_context\n",
      "    distributed.comm.ucx.init_once()\n",
      "  File \"/glade/u/apps/opt/conda/envs/pygpu-dask/lib/python3.9/site-packages/distributed/comm/ucx.py\", line 104, in init_once\n",
      "    cuda_visible_device = int(\n",
      "ValueError: invalid literal for int() with base 10: 'GPU-30104623-af2a-4c39-2a28-07ff7b90b70e'\n",
      "Unable to start CUDA Context\n",
      "Traceback (most recent call last):\n",
      "  File \"/glade/u/apps/opt/conda/envs/pygpu-dask/lib/python3.9/site-packages/dask_cuda/initialize.py\", line 31, in _create_cuda_context\n",
      "    distributed.comm.ucx.init_once()\n",
      "  File \"/glade/u/apps/opt/conda/envs/pygpu-dask/lib/python3.9/site-packages/distributed/comm/ucx.py\", line 104, in init_once\n",
      "    cuda_visible_device = int(\n",
      "ValueError: invalid literal for int() with base 10: 'GPU-ff46f738-51a2-cfd7-2163-45524f96ac82'\n"
     ]
    }
   ],
   "source": [
    "cluster = LocalCUDACluster()\n",
    "client = Client(cluster, timeout = '60s')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f4c0e897-5e37-4ecd-b14b-8d30a494ba78",
   "metadata": {},
   "source": [
    "*Did it work?*\n",
    "\n",
    "If you are using multiple GPUs, the `CUDA_VISIBLE_DEVICES` environment variable needs to be set to integer device IDs, not string GPU UUIDs as PBS specifies.\n",
    "\n",
    "First, let's shut down our prior attempt to free the port for a new cluster and client."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "c5b2d111-433f-40d9-bbc6-b03434629ba7",
   "metadata": {},
   "outputs": [],
   "source": [
    "client.shutdown()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9398f23b-7111-4338-a143-a75e92a37457",
   "metadata": {
    "tags": []
   },
   "source": [
    "#### Querying GPU information with **pynvml**\n",
    "\n",
    "The **pynvml** provides Python bindings to the NVIDIA Management Library (NVML), which allows users to query and (often with root) modify GPU characteristics and behavior. We can import and use it to get information about our GPUs from within Python."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "3e340ec3-1da8-4dcd-9150-d0d188b5935f",
   "metadata": {},
   "outputs": [],
   "source": [
    "from pynvml import *\n",
    "nvmlInit()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "a1967cc8-e679-480f-80ab-c8bb98a7d350",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Device 0 UUID: GPU-30104623-af2a-4c39-2a28-07ff7b90b70e; Compute mode: Default\n",
      "Device 1 UUID: GPU-ff46f738-51a2-cfd7-2163-45524f96ac82; Compute mode: Default\n"
     ]
    }
   ],
   "source": [
    "cmode_names = ['Default', 'Exclusive Thread', 'Prohibited', 'Exclusive Process']\n",
    "\n",
    "for i in range(nvmlDeviceGetCount()):\n",
    "    handle = nvmlDeviceGetHandleByIndex(i)\n",
    "    uuid = nvmlDeviceGetUUID(handle).decode(\"utf-8\")\n",
    "    cmode = nvmlDeviceGetComputeMode(handle)\n",
    "    print(f'Device {i} UUID: {uuid}; Compute mode: {cmode_names[cmode]}')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a4ca798b-ca93-4a29-a808-2c209f333160",
   "metadata": {},
   "source": [
    "**Heads up:** You are likely to run into issues in Python if you use a GPU that is not using the Default compute mode. Exclusive process mode, for example, will prevent you from using **CuPy** and a **CUDAWorker** in the same Python process!\n",
    "```\n",
    "Device 0 UUID: GPU-c0bdd08e-994c-f92d-6ad7-7b629156e76d; Compute mode: Exclusive Process\n",
    "Device 1 UUID: GPU-e8571304-3a7b-443e-d688-933be18cfbeb; Compute mode: Default\n",
    "```\n",
    "If you see something like the above, make sure to change (or more likely have your sys admin change) the compute mode to default either in Python or using `nvidia-smi`:\n",
    "```bash\n",
    "sudo nvidia-smi -c 0 -i UUID\n",
    "```\n",
    "#### Let's put CUDA_VISIBLE_DEVICES in a form (device IDs) that `LocalCUDACluster` will accept"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "7f7ceffb-9a31-490e-9313-b0e462d93522",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "os.environ['CUDA_VISIBLE_DEVICES'] = '0,1'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "bd528332-b7dd-461e-ada4-db76c6e4d5e1",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2022-08-11 10:34:21,358 - distributed.preloading - INFO - Import preload module: dask_cuda.initialize\n",
      "2022-08-11 10:34:21,447 - distributed.preloading - INFO - Import preload module: dask_cuda.initialize\n"
     ]
    }
   ],
   "source": [
    "cluster = LocalCUDACluster()\n",
    "client = Client(cluster, timeout = '60s')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "e4c4215b-8cb2-4bd3-b37a-7d86384cec29",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "    <div style=\"width: 24px; height: 24px; background-color: #e1e1e1; border: 3px solid #9D9D9D; border-radius: 5px; position: absolute;\"> </div>\n",
       "    <div style=\"margin-left: 48px;\">\n",
       "        <h3 style=\"margin-bottom: 0px;\">Client</h3>\n",
       "        <p style=\"color: #9D9D9D; margin-bottom: 0px;\">Client-747404b0-1993-11ed-a439-3cecef0c464c</p>\n",
       "        <table style=\"width: 100%; text-align: left;\">\n",
       "\n",
       "        <tr>\n",
       "        \n",
       "            <td style=\"text-align: left;\"><strong>Connection method:</strong> Cluster object</td>\n",
       "            <td style=\"text-align: left;\"><strong>Cluster type:</strong> dask_cuda.LocalCUDACluster</td>\n",
       "        \n",
       "        </tr>\n",
       "\n",
       "        \n",
       "            <tr>\n",
       "                <td style=\"text-align: left;\">\n",
       "                    <strong>Dashboard: </strong> <a href=\"https://jupyterhub.hpc.ucar.edu/stable/user/vanderwb/proxy/8787/status\" target=\"_blank\">https://jupyterhub.hpc.ucar.edu/stable/user/vanderwb/proxy/8787/status</a>\n",
       "                </td>\n",
       "                <td style=\"text-align: left;\"></td>\n",
       "            </tr>\n",
       "        \n",
       "\n",
       "        </table>\n",
       "\n",
       "        \n",
       "            <details>\n",
       "            <summary style=\"margin-bottom: 20px;\"><h3 style=\"display: inline;\">Cluster Info</h3></summary>\n",
       "            <div class=\"jp-RenderedHTMLCommon jp-RenderedHTML jp-mod-trusted jp-OutputArea-output\">\n",
       "    <div style=\"width: 24px; height: 24px; background-color: #e1e1e1; border: 3px solid #9D9D9D; border-radius: 5px; position: absolute;\">\n",
       "    </div>\n",
       "    <div style=\"margin-left: 48px;\">\n",
       "        <h3 style=\"margin-bottom: 0px; margin-top: 0px;\">LocalCUDACluster</h3>\n",
       "        <p style=\"color: #9D9D9D; margin-bottom: 0px;\">b5b22f43</p>\n",
       "        <table style=\"width: 100%; text-align: left;\">\n",
       "            <tr>\n",
       "                <td style=\"text-align: left;\">\n",
       "                    <strong>Dashboard:</strong> <a href=\"https://jupyterhub.hpc.ucar.edu/stable/user/vanderwb/proxy/8787/status\" target=\"_blank\">https://jupyterhub.hpc.ucar.edu/stable/user/vanderwb/proxy/8787/status</a>\n",
       "                </td>\n",
       "                <td style=\"text-align: left;\">\n",
       "                    <strong>Workers:</strong> 2\n",
       "                </td>\n",
       "            </tr>\n",
       "            <tr>\n",
       "                <td style=\"text-align: left;\">\n",
       "                    <strong>Total threads:</strong> 2\n",
       "                </td>\n",
       "                <td style=\"text-align: left;\">\n",
       "                    <strong>Total memory:</strong> 100.00 GiB\n",
       "                </td>\n",
       "            </tr>\n",
       "            \n",
       "            <tr>\n",
       "    <td style=\"text-align: left;\"><strong>Status:</strong> running</td>\n",
       "    <td style=\"text-align: left;\"><strong>Using processes:</strong> True</td>\n",
       "</tr>\n",
       "\n",
       "            \n",
       "        </table>\n",
       "\n",
       "        <details>\n",
       "            <summary style=\"margin-bottom: 20px;\">\n",
       "                <h3 style=\"display: inline;\">Scheduler Info</h3>\n",
       "            </summary>\n",
       "\n",
       "            <div style=\"\">\n",
       "    <div>\n",
       "        <div style=\"width: 24px; height: 24px; background-color: #FFF7E5; border: 3px solid #FF6132; border-radius: 5px; position: absolute;\"> </div>\n",
       "        <div style=\"margin-left: 48px;\">\n",
       "            <h3 style=\"margin-bottom: 0px;\">Scheduler</h3>\n",
       "            <p style=\"color: #9D9D9D; margin-bottom: 0px;\">Scheduler-bf2fff52-8de8-422e-a9f4-b0371f3bc49b</p>\n",
       "            <table style=\"width: 100%; text-align: left;\">\n",
       "                <tr>\n",
       "                    <td style=\"text-align: left;\">\n",
       "                        <strong>Comm:</strong> tcp://127.0.0.1:43624\n",
       "                    </td>\n",
       "                    <td style=\"text-align: left;\">\n",
       "                        <strong>Workers:</strong> 2\n",
       "                    </td>\n",
       "                </tr>\n",
       "                <tr>\n",
       "                    <td style=\"text-align: left;\">\n",
       "                        <strong>Dashboard:</strong> <a href=\"https://jupyterhub.hpc.ucar.edu/stable/user/vanderwb/proxy/8787/status\" target=\"_blank\">https://jupyterhub.hpc.ucar.edu/stable/user/vanderwb/proxy/8787/status</a>\n",
       "                    </td>\n",
       "                    <td style=\"text-align: left;\">\n",
       "                        <strong>Total threads:</strong> 2\n",
       "                    </td>\n",
       "                </tr>\n",
       "                <tr>\n",
       "                    <td style=\"text-align: left;\">\n",
       "                        <strong>Started:</strong> Just now\n",
       "                    </td>\n",
       "                    <td style=\"text-align: left;\">\n",
       "                        <strong>Total memory:</strong> 100.00 GiB\n",
       "                    </td>\n",
       "                </tr>\n",
       "            </table>\n",
       "        </div>\n",
       "    </div>\n",
       "\n",
       "    <details style=\"margin-left: 48px;\">\n",
       "        <summary style=\"margin-bottom: 20px;\">\n",
       "            <h3 style=\"display: inline;\">Workers</h3>\n",
       "        </summary>\n",
       "\n",
       "        \n",
       "        <div style=\"margin-bottom: 20px;\">\n",
       "            <div style=\"width: 24px; height: 24px; background-color: #DBF5FF; border: 3px solid #4CC9FF; border-radius: 5px; position: absolute;\"> </div>\n",
       "            <div style=\"margin-left: 48px;\">\n",
       "            <details>\n",
       "                <summary>\n",
       "                    <h4 style=\"margin-bottom: 0px; display: inline;\">Worker: 0</h4>\n",
       "                </summary>\n",
       "                <table style=\"width: 100%; text-align: left;\">\n",
       "                    <tr>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Comm: </strong> tcp://127.0.0.1:40331\n",
       "                        </td>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Total threads: </strong> 1\n",
       "                        </td>\n",
       "                    </tr>\n",
       "                    <tr>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Dashboard: </strong> <a href=\"https://jupyterhub.hpc.ucar.edu/stable/user/vanderwb/proxy/44079/status\" target=\"_blank\">https://jupyterhub.hpc.ucar.edu/stable/user/vanderwb/proxy/44079/status</a>\n",
       "                        </td>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Memory: </strong> 50.00 GiB\n",
       "                        </td>\n",
       "                    </tr>\n",
       "                    <tr>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Nanny: </strong> tcp://127.0.0.1:38616\n",
       "                        </td>\n",
       "                        <td style=\"text-align: left;\"></td>\n",
       "                    </tr>\n",
       "                    <tr>\n",
       "                        <td colspan=\"2\" style=\"text-align: left;\">\n",
       "                            <strong>Local directory: </strong> /glade/work/vanderwb/tutorials/GPU_workshop/13_DaskGPU/dask-worker-space/worker-ir94v5_f\n",
       "                        </td>\n",
       "                    </tr>\n",
       "\n",
       "                    \n",
       "                    <tr>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>GPU: </strong>Tesla V100-SXM2-32GB\n",
       "                        </td>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>GPU memory: </strong> 32.00 GiB\n",
       "                        </td>\n",
       "                    </tr>\n",
       "                    \n",
       "\n",
       "                    \n",
       "\n",
       "                </table>\n",
       "            </details>\n",
       "            </div>\n",
       "        </div>\n",
       "        \n",
       "        <div style=\"margin-bottom: 20px;\">\n",
       "            <div style=\"width: 24px; height: 24px; background-color: #DBF5FF; border: 3px solid #4CC9FF; border-radius: 5px; position: absolute;\"> </div>\n",
       "            <div style=\"margin-left: 48px;\">\n",
       "            <details>\n",
       "                <summary>\n",
       "                    <h4 style=\"margin-bottom: 0px; display: inline;\">Worker: 1</h4>\n",
       "                </summary>\n",
       "                <table style=\"width: 100%; text-align: left;\">\n",
       "                    <tr>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Comm: </strong> tcp://127.0.0.1:43578\n",
       "                        </td>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Total threads: </strong> 1\n",
       "                        </td>\n",
       "                    </tr>\n",
       "                    <tr>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Dashboard: </strong> <a href=\"https://jupyterhub.hpc.ucar.edu/stable/user/vanderwb/proxy/38170/status\" target=\"_blank\">https://jupyterhub.hpc.ucar.edu/stable/user/vanderwb/proxy/38170/status</a>\n",
       "                        </td>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Memory: </strong> 50.00 GiB\n",
       "                        </td>\n",
       "                    </tr>\n",
       "                    <tr>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>Nanny: </strong> tcp://127.0.0.1:43042\n",
       "                        </td>\n",
       "                        <td style=\"text-align: left;\"></td>\n",
       "                    </tr>\n",
       "                    <tr>\n",
       "                        <td colspan=\"2\" style=\"text-align: left;\">\n",
       "                            <strong>Local directory: </strong> /glade/work/vanderwb/tutorials/GPU_workshop/13_DaskGPU/dask-worker-space/worker-bb6gf_pe\n",
       "                        </td>\n",
       "                    </tr>\n",
       "\n",
       "                    \n",
       "                    <tr>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>GPU: </strong>Tesla V100-SXM2-32GB\n",
       "                        </td>\n",
       "                        <td style=\"text-align: left;\">\n",
       "                            <strong>GPU memory: </strong> 32.00 GiB\n",
       "                        </td>\n",
       "                    </tr>\n",
       "                    \n",
       "\n",
       "                    \n",
       "\n",
       "                </table>\n",
       "            </details>\n",
       "            </div>\n",
       "        </div>\n",
       "        \n",
       "\n",
       "    </details>\n",
       "</div>\n",
       "\n",
       "        </details>\n",
       "    </div>\n",
       "</div>\n",
       "            </details>\n",
       "        \n",
       "\n",
       "    </div>\n",
       "</div>"
      ],
      "text/plain": [
       "<Client: 'tcp://127.0.0.1:43624' processes=2 threads=2, memory=100.00 GiB>"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "client"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "008d41a1-8259-4e02-a038-71703dd49f3b",
   "metadata": {},
   "source": [
    "**Now we are ready to scale up computations with Dask!**  \n",
    "\n",
    "We can use our aquaplanet temperature array we created earlier. Let's imagine that the array is a single output file, and we need to process twelve of them."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8ba561b3-adb1-4043-b728-b33d5b226bbf",
   "metadata": {},
   "source": [
    "#### Construct a Dask Array *on the GPU* using the NumPy array we created ...\n",
    "\n",
    "First we use the NumPy array to define our chunk size - the number of elements for each chunk of work to be done by a worker."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "ebe63a43-dcf6-43db-91bb-562a97c4d38d",
   "metadata": {},
   "outputs": [],
   "source": [
    "dc_arr = da.from_array(t, chunks = (1000, 500, 140))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e2d22524-975f-4f5c-97ac-e29f90b69a7c",
   "metadata": {},
   "source": [
    "To create a Dask Array on the GPU, we simply need to map a conversion from NumPy to CuPy to all of our data chunks."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "cbe9cb35-ee39-4365-b2e8-391d83f803c3",
   "metadata": {},
   "outputs": [],
   "source": [
    "dg_arr = dc_arr.map_blocks(cp.asarray)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "da515cf9-2587-4124-bb79-babf3a90154b",
   "metadata": {},
   "source": [
    "We can now concatenate multiple copies of this dask array together to form our \"multifile\" dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "2ec26955-04d3-445f-812b-046895fbc856",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table>\n",
       "    <tr>\n",
       "        <td>\n",
       "            <table>\n",
       "                <thead>\n",
       "                    <tr>\n",
       "                        <td> </td>\n",
       "                        <th> Array </th>\n",
       "                        <th> Chunk </th>\n",
       "                    </tr>\n",
       "                </thead>\n",
       "                <tbody>\n",
       "                    \n",
       "                    <tr>\n",
       "                        <th> Bytes </th>\n",
       "                        <td> 100.14 GiB </td>\n",
       "                        <td> 534.06 MiB </td>\n",
       "                    </tr>\n",
       "                    \n",
       "                    <tr>\n",
       "                        <th> Shape </th>\n",
       "                        <td> (4000, 2000, 1680) </td>\n",
       "                        <td> (1000, 500, 140) </td>\n",
       "                    </tr>\n",
       "                    <tr>\n",
       "                        <th> Count </th>\n",
       "                        <td> 224 Tasks </td>\n",
       "                        <td> 192 Chunks </td>\n",
       "                    </tr>\n",
       "                    <tr>\n",
       "                    <th> Type </th>\n",
       "                    <td> float64 </td>\n",
       "                    <td> cupy._core.core.ndarray </td>\n",
       "                    </tr>\n",
       "                </tbody>\n",
       "            </table>\n",
       "        </td>\n",
       "        <td>\n",
       "        <svg width=\"180\" height=\"180\" style=\"stroke:rgb(0,0,0);stroke-width:1\" >\n",
       "\n",
       "  <!-- Horizontal lines -->\n",
       "  <line x1=\"10\" y1=\"0\" x2=\"80\" y2=\"70\" style=\"stroke-width:2\" />\n",
       "  <line x1=\"10\" y1=\"15\" x2=\"80\" y2=\"85\" />\n",
       "  <line x1=\"10\" y1=\"30\" x2=\"80\" y2=\"100\" />\n",
       "  <line x1=\"10\" y1=\"45\" x2=\"80\" y2=\"115\" />\n",
       "  <line x1=\"10\" y1=\"60\" x2=\"80\" y2=\"130\" style=\"stroke-width:2\" />\n",
       "\n",
       "  <!-- Vertical lines -->\n",
       "  <line x1=\"10\" y1=\"0\" x2=\"10\" y2=\"60\" style=\"stroke-width:2\" />\n",
       "  <line x1=\"27\" y1=\"17\" x2=\"27\" y2=\"77\" />\n",
       "  <line x1=\"45\" y1=\"35\" x2=\"45\" y2=\"95\" />\n",
       "  <line x1=\"62\" y1=\"52\" x2=\"62\" y2=\"112\" />\n",
       "  <line x1=\"80\" y1=\"70\" x2=\"80\" y2=\"130\" style=\"stroke-width:2\" />\n",
       "\n",
       "  <!-- Colored Rectangle -->\n",
       "  <polygon points=\"10.0,0.0 80.58823529411765,70.58823529411765 80.58823529411765,130.58823529411765 10.0,60.0\" style=\"fill:#ECB172A0;stroke-width:0\"/>\n",
       "\n",
       "  <!-- Horizontal lines -->\n",
       "  <line x1=\"10\" y1=\"0\" x2=\"60\" y2=\"0\" style=\"stroke-width:2\" />\n",
       "  <line x1=\"27\" y1=\"17\" x2=\"78\" y2=\"17\" />\n",
       "  <line x1=\"45\" y1=\"35\" x2=\"95\" y2=\"35\" />\n",
       "  <line x1=\"62\" y1=\"52\" x2=\"113\" y2=\"52\" />\n",
       "  <line x1=\"80\" y1=\"70\" x2=\"130\" y2=\"70\" style=\"stroke-width:2\" />\n",
       "\n",
       "  <!-- Vertical lines -->\n",
       "  <line x1=\"10\" y1=\"0\" x2=\"80\" y2=\"70\" style=\"stroke-width:2\" />\n",
       "  <line x1=\"14\" y1=\"0\" x2=\"84\" y2=\"70\" />\n",
       "  <line x1=\"18\" y1=\"0\" x2=\"88\" y2=\"70\" />\n",
       "  <line x1=\"22\" y1=\"0\" x2=\"93\" y2=\"70\" />\n",
       "  <line x1=\"26\" y1=\"0\" x2=\"97\" y2=\"70\" />\n",
       "  <line x1=\"31\" y1=\"0\" x2=\"101\" y2=\"70\" />\n",
       "  <line x1=\"35\" y1=\"0\" x2=\"105\" y2=\"70\" />\n",
       "  <line x1=\"39\" y1=\"0\" x2=\"109\" y2=\"70\" />\n",
       "  <line x1=\"43\" y1=\"0\" x2=\"114\" y2=\"70\" />\n",
       "  <line x1=\"47\" y1=\"0\" x2=\"118\" y2=\"70\" />\n",
       "  <line x1=\"52\" y1=\"0\" x2=\"122\" y2=\"70\" />\n",
       "  <line x1=\"56\" y1=\"0\" x2=\"126\" y2=\"70\" />\n",
       "  <line x1=\"60\" y1=\"0\" x2=\"130\" y2=\"70\" style=\"stroke-width:2\" />\n",
       "\n",
       "  <!-- Colored Rectangle -->\n",
       "  <polygon points=\"10.0,0.0 60.4,0.0 130.98823529411766,70.58823529411765 80.58823529411765,70.58823529411765\" style=\"fill:#ECB172A0;stroke-width:0\"/>\n",
       "\n",
       "  <!-- Horizontal lines -->\n",
       "  <line x1=\"80\" y1=\"70\" x2=\"130\" y2=\"70\" style=\"stroke-width:2\" />\n",
       "  <line x1=\"80\" y1=\"85\" x2=\"130\" y2=\"85\" />\n",
       "  <line x1=\"80\" y1=\"100\" x2=\"130\" y2=\"100\" />\n",
       "  <line x1=\"80\" y1=\"115\" x2=\"130\" y2=\"115\" />\n",
       "  <line x1=\"80\" y1=\"130\" x2=\"130\" y2=\"130\" style=\"stroke-width:2\" />\n",
       "\n",
       "  <!-- Vertical lines -->\n",
       "  <line x1=\"80\" y1=\"70\" x2=\"80\" y2=\"130\" style=\"stroke-width:2\" />\n",
       "  <line x1=\"84\" y1=\"70\" x2=\"84\" y2=\"130\" />\n",
       "  <line x1=\"88\" y1=\"70\" x2=\"88\" y2=\"130\" />\n",
       "  <line x1=\"93\" y1=\"70\" x2=\"93\" y2=\"130\" />\n",
       "  <line x1=\"97\" y1=\"70\" x2=\"97\" y2=\"130\" />\n",
       "  <line x1=\"101\" y1=\"70\" x2=\"101\" y2=\"130\" />\n",
       "  <line x1=\"105\" y1=\"70\" x2=\"105\" y2=\"130\" />\n",
       "  <line x1=\"109\" y1=\"70\" x2=\"109\" y2=\"130\" />\n",
       "  <line x1=\"114\" y1=\"70\" x2=\"114\" y2=\"130\" />\n",
       "  <line x1=\"118\" y1=\"70\" x2=\"118\" y2=\"130\" />\n",
       "  <line x1=\"122\" y1=\"70\" x2=\"122\" y2=\"130\" />\n",
       "  <line x1=\"126\" y1=\"70\" x2=\"126\" y2=\"130\" />\n",
       "  <line x1=\"130\" y1=\"70\" x2=\"130\" y2=\"130\" style=\"stroke-width:2\" />\n",
       "\n",
       "  <!-- Colored Rectangle -->\n",
       "  <polygon points=\"80.58823529411765,70.58823529411765 130.98823529411766,70.58823529411765 130.98823529411766,130.58823529411765 80.58823529411765,130.58823529411765\" style=\"fill:#ECB172A0;stroke-width:0\"/>\n",
       "\n",
       "  <!-- Text -->\n",
       "  <text x=\"105.788235\" y=\"150.588235\" font-size=\"1.0rem\" font-weight=\"100\" text-anchor=\"middle\" >1680</text>\n",
       "  <text x=\"150.988235\" y=\"100.588235\" font-size=\"1.0rem\" font-weight=\"100\" text-anchor=\"middle\" transform=\"rotate(-90,150.988235,100.588235)\">2000</text>\n",
       "  <text x=\"35.294118\" y=\"115.294118\" font-size=\"1.0rem\" font-weight=\"100\" text-anchor=\"middle\" transform=\"rotate(45,35.294118,115.294118)\">4000</text>\n",
       "</svg>\n",
       "        </td>\n",
       "    </tr>\n",
       "</table>"
      ],
      "text/plain": [
       "dask.array<concatenate, shape=(4000, 2000, 1680), dtype=float64, chunksize=(1000, 500, 140), chunktype=cupy.ndarray>"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dg_mf = da.concatenate([dg_arr] * 12, axis=2)\n",
    "dg_mf"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7ce3f36c-cd6e-43d1-a895-634c6249a7dd",
   "metadata": {},
   "source": [
    "#### Assigning tasks to a Dask collection\n",
    "\n",
    "Let's do some data processing on our GPU Dask Array. We can convert from Kelvin to Celsius and then take the mean across all times at each spatial index."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "id": "a4a5bf05-0d1f-4afa-bb1c-55cd79de3af9",
   "metadata": {},
   "outputs": [],
   "source": [
    "tg = da.mean((dg_mf - 273.15), axis = 2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "id": "79f0e0eb-30f7-43f4-8335-88b5dac35c6e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "    <div>\n",
       "        <div style=\"width: 52px; height: 52px; position: absolute;\">\n",
       "            <svg width=\"76\" height=\"71\" viewBox=\"0 0 76 71\" fill=\"none\" xmlns=\"http://www.w3.org/2000/svg\">\n",
       "                <circle cx=\"61.5\" cy=\"36.5\" r=\"13.5\" style=\"stroke: var(--jp-ui-font-color2, #1D1D1D); fill: var(--jp-layout-color1, #F2F2F2);\" stroke-width=\"2\"/>\n",
       "                <circle cx=\"14.5\" cy=\"14.5\" r=\"13.5\" style=\"stroke: var(--jp-ui-font-color2, #1D1D1D); fill: var(--jp-layout-color1, #F2F2F2);\" stroke-width=\"2\"/>\n",
       "                <circle cx=\"14.5\" cy=\"56.5\" r=\"13.5\" style=\"stroke: var(--jp-ui-font-color2, #1D1D1D); fill: var(--jp-layout-color1, #F2F2F2);\" stroke-width=\"2\"/>\n",
       "                <path d=\"M28 16L30.5 16C33.2614 16 35.5 18.2386 35.5 21L35.5 32.0001C35.5 34.7615 37.7386 37.0001 40.5 37.0001L43 37.0001\" style=\"stroke: var(--jp-ui-font-color2, #1D1D1D);\" stroke-width=\"1.5\"/>\n",
       "                <path d=\"M40.5 37L40.5 37.75L40.5 37.75L40.5 37ZM35.5 42L36.25 42L35.5 42ZM35.5 52L34.75 52L35.5 52ZM30.5 57L30.5 57.75L30.5 57ZM41.5001 36.25L40.5 36.25L40.5 37.75L41.5001 37.75L41.5001 36.25ZM34.75 42L34.75 52L36.25 52L36.25 42L34.75 42ZM30.5 56.25L28.0001 56.25L28.0001 57.75L30.5 57.75L30.5 56.25ZM34.75 52C34.75 54.3472 32.8472 56.25 30.5 56.25L30.5 57.75C33.6756 57.75 36.25 55.1756 36.25 52L34.75 52ZM40.5 36.25C37.3244 36.25 34.75 38.8243 34.75 42L36.25 42C36.25 39.6528 38.1528 37.75 40.5 37.75L40.5 36.25Z\" style=\"fill: var(--jp-ui-font-color2, #1D1D1D);\"/>\n",
       "                <circle cx=\"28\" cy=\"16\" r=\"2.25\" fill=\"#E5E5E5\" style=\"stroke: var(--jp-ui-font-color2, #1D1D1D);\" stroke-width=\"1.5\"/>\n",
       "                <circle cx=\"28\" cy=\"57\" r=\"2.25\" fill=\"#E5E5E5\" style=\"stroke: var(--jp-ui-font-color2, #1D1D1D);\" stroke-width=\"1.5\"/>\n",
       "                <path d=\"M45.25 36.567C45.5833 36.7594 45.5833 37.2406 45.25 37.433L42.25 39.1651C41.9167 39.3575 41.5 39.117 41.5 38.7321V35.2679C41.5 34.883 41.9167 34.6425 42.25 34.8349L45.25 36.567Z\" style=\"fill: var(--jp-ui-font-color2, #1D1D1D);\"/>\n",
       "            </svg>\n",
       "        </div>\n",
       "        <div style=\"margin-left: 64px;\">\n",
       "            <h3 style=\"margin-bottom: 0px;\">HighLevelGraph</h3>\n",
       "            <p style=\"color: var(--jp-ui-font-color2, #5D5851); margin-bottom:0px;\">\n",
       "                HighLevelGraph with 7 layers and 672 keys from all layers.\n",
       "            </p>\n",
       "            \n",
       "            <div style=\"\">\n",
       "    <svg width=\"24\" height=\"24\" viewBox=\"0 0 32 32\" fill=\"none\" xmlns=\"http://www.w3.org/2000/svg\" style=\"position: absolute;\">\n",
       "        \n",
       "        <circle cx=\"16\" cy=\"16\" r=\"14\" fill=\"#8F8F8F\" style=\"stroke: var(--jp-ui-font-color2, #1D1D1D);\" stroke-width=\"2\"/>\n",
       "        \n",
       "    </svg>\n",
       "\n",
       "    <details style=\"margin-left: 32px;\">\n",
       "        <summary style=\"margin-bottom: 10px; margin-top: 10px;\">\n",
       "            <h4 style=\"display: inline;\">Layer1: array</h4>\n",
       "        </summary>\n",
       "        <p style=\"color: var(--jp-ui-font-color2, #5D5851); margin: -0.25em 0px 0px 0px;\">\n",
       "            array-2d6d607525c93a60d3ebad8394f5ac20\n",
       "        </p>\n",
       "\n",
       "        <table>\n",
       "        <tr>\n",
       "            <td>\n",
       "                <table>\n",
       "                \n",
       "                    <tr>\n",
       "                        <th style=\"text-align: left; width: 150px;\">layer_type</th>\n",
       "                        <td style=\"text-align: left;\">MaterializedLayer</td>\n",
       "                    </tr>\n",
       "                \n",
       "                    <tr>\n",
       "                        <th style=\"text-align: left; width: 150px;\">is_materialized</th>\n",
       "                        <td style=\"text-align: left;\">True</td>\n",
       "                    </tr>\n",
       "                \n",
       "                    <tr>\n",
       "                        <th style=\"text-align: left; width: 150px;\">number of outputs</th>\n",
       "                        <td style=\"text-align: left;\">16</td>\n",
       "                    </tr>\n",
       "                \n",
       "                    <tr>\n",
       "                        <th style=\"text-align: left; width: 150px;\">shape</th>\n",
       "                        <td style=\"text-align: left;\">(4000, 2000, 140)</td>\n",
       "                    </tr>\n",
       "                \n",
       "                    <tr>\n",
       "                        <th style=\"text-align: left; width: 150px;\">dtype</th>\n",
       "                        <td style=\"text-align: left;\">float64</td>\n",
       "                    </tr>\n",
       "                \n",
       "                    <tr>\n",
       "                        <th style=\"text-align: left; width: 150px;\">chunksize</th>\n",
       "                        <td style=\"text-align: left;\">(1000, 500, 140)</td>\n",
       "                    </tr>\n",
       "                \n",
       "                    <tr>\n",
       "                        <th style=\"text-align: left; width: 150px;\">type</th>\n",
       "                        <td style=\"text-align: left;\">dask.array.core.Array</td>\n",
       "                    </tr>\n",
       "                \n",
       "                    <tr>\n",
       "                        <th style=\"text-align: left; width: 150px;\">chunk_type</th>\n",
       "                        <td style=\"text-align: left;\">numpy.ndarray</td>\n",
       "                    </tr>\n",
       "                \n",
       "                </table>\n",
       "            </td>\n",
       "            <td>\n",
       "                <svg width=\"231\" height=\"267\" style=\"stroke:rgb(0,0,0);stroke-width:1\" >\n",
       "\n",
       "  <!-- Horizontal lines -->\n",
       "  <line x1=\"10\" y1=\"0\" x2=\"127\" y2=\"117\" style=\"stroke-width:2\" />\n",
       "  <line x1=\"10\" y1=\"25\" x2=\"127\" y2=\"142\" />\n",
       "  <line x1=\"10\" y1=\"50\" x2=\"127\" y2=\"167\" />\n",
       "  <line x1=\"10\" y1=\"75\" x2=\"127\" y2=\"192\" />\n",
       "  <line x1=\"10\" y1=\"100\" x2=\"127\" y2=\"217\" style=\"stroke-width:2\" />\n",
       "\n",
       "  <!-- Vertical lines -->\n",
       "  <line x1=\"10\" y1=\"0\" x2=\"10\" y2=\"100\" style=\"stroke-width:2\" />\n",
       "  <line x1=\"39\" y1=\"29\" x2=\"39\" y2=\"129\" />\n",
       "  <line x1=\"68\" y1=\"58\" x2=\"68\" y2=\"158\" />\n",
       "  <line x1=\"98\" y1=\"88\" x2=\"98\" y2=\"188\" />\n",
       "  <line x1=\"127\" y1=\"117\" x2=\"127\" y2=\"217\" style=\"stroke-width:2\" />\n",
       "\n",
       "  <!-- Colored Rectangle -->\n",
       "  <polygon points=\"10.0,0.0 127.64705882352942,117.64705882352942 127.64705882352942,217.64705882352942 10.0,100.0\" style=\"fill:#ECB172A0;stroke-width:0\"/>\n",
       "\n",
       "  <!-- Horizontal lines -->\n",
       "  <line x1=\"10\" y1=\"0\" x2=\"63\" y2=\"0\" style=\"stroke-width:2\" />\n",
       "  <line x1=\"39\" y1=\"29\" x2=\"93\" y2=\"29\" />\n",
       "  <line x1=\"68\" y1=\"58\" x2=\"122\" y2=\"58\" />\n",
       "  <line x1=\"98\" y1=\"88\" x2=\"152\" y2=\"88\" />\n",
       "  <line x1=\"127\" y1=\"117\" x2=\"181\" y2=\"117\" style=\"stroke-width:2\" />\n",
       "\n",
       "  <!-- Vertical lines -->\n",
       "  <line x1=\"10\" y1=\"0\" x2=\"127\" y2=\"117\" style=\"stroke-width:2\" />\n",
       "  <line x1=\"63\" y1=\"0\" x2=\"181\" y2=\"117\" style=\"stroke-width:2\" />\n",
       "\n",
       "  <!-- Colored Rectangle -->\n",
       "  <polygon points=\"10.0,0.0 63.86661338516172,0.0 181.51367220869113,117.64705882352942 127.64705882352942,117.64705882352942\" style=\"fill:#ECB172A0;stroke-width:0\"/>\n",
       "\n",
       "  <!-- Horizontal lines -->\n",
       "  <line x1=\"127\" y1=\"117\" x2=\"181\" y2=\"117\" style=\"stroke-width:2\" />\n",
       "  <line x1=\"127\" y1=\"142\" x2=\"181\" y2=\"142\" />\n",
       "  <line x1=\"127\" y1=\"167\" x2=\"181\" y2=\"167\" />\n",
       "  <line x1=\"127\" y1=\"192\" x2=\"181\" y2=\"192\" />\n",
       "  <line x1=\"127\" y1=\"217\" x2=\"181\" y2=\"217\" style=\"stroke-width:2\" />\n",
       "\n",
       "  <!-- Vertical lines -->\n",
       "  <line x1=\"127\" y1=\"117\" x2=\"127\" y2=\"217\" style=\"stroke-width:2\" />\n",
       "  <line x1=\"181\" y1=\"117\" x2=\"181\" y2=\"217\" style=\"stroke-width:2\" />\n",
       "\n",
       "  <!-- Colored Rectangle -->\n",
       "  <polygon points=\"127.64705882352942,117.64705882352942 181.51367220869113,117.64705882352942 181.51367220869113,217.64705882352942 127.64705882352942,217.64705882352942\" style=\"fill:#ECB172A0;stroke-width:0\"/>\n",
       "\n",
       "  <!-- Text -->\n",
       "  <text x=\"154.580366\" y=\"237.647059\" font-size=\"1.0rem\" font-weight=\"100\" text-anchor=\"middle\" >140</text>\n",
       "  <text x=\"201.513672\" y=\"167.647059\" font-size=\"1.0rem\" font-weight=\"100\" text-anchor=\"middle\" transform=\"rotate(-90,201.513672,167.647059)\">2000</text>\n",
       "  <text x=\"58.823529\" y=\"178.823529\" font-size=\"1.0rem\" font-weight=\"100\" text-anchor=\"middle\" transform=\"rotate(45,58.823529,178.823529)\">4000</text>\n",
       "</svg>\n",
       "            </td>\n",
       "        </tr>\n",
       "        </table>\n",
       "\n",
       "    </details>\n",
       "</div>\n",
       "            \n",
       "            <div style=\"\">\n",
       "    <svg width=\"24\" height=\"24\" viewBox=\"0 0 32 32\" fill=\"none\" xmlns=\"http://www.w3.org/2000/svg\" style=\"position: absolute;\">\n",
       "        \n",
       "        <circle cx=\"16\" cy=\"16\" r=\"14\" style=\"stroke: var(--jp-ui-font-color2, #1D1D1D); fill: var(--jp-layout-color1, #F2F2F2);\" stroke-width=\"2\" />\n",
       "        \n",
       "    </svg>\n",
       "\n",
       "    <details style=\"margin-left: 32px;\">\n",
       "        <summary style=\"margin-bottom: 10px; margin-top: 10px;\">\n",
       "            <h4 style=\"display: inline;\">Layer2: asarray</h4>\n",
       "        </summary>\n",
       "        <p style=\"color: var(--jp-ui-font-color2, #5D5851); margin: -0.25em 0px 0px 0px;\">\n",
       "            asarray-0bb0329305b0a5ad59d3ae1aa871cf9b\n",
       "        </p>\n",
       "\n",
       "        <table>\n",
       "        <tr>\n",
       "            <td>\n",
       "                <table>\n",
       "                \n",
       "                    <tr>\n",
       "                        <th style=\"text-align: left; width: 150px;\">layer_type</th>\n",
       "                        <td style=\"text-align: left;\">Blockwise</td>\n",
       "                    </tr>\n",
       "                \n",
       "                    <tr>\n",
       "                        <th style=\"text-align: left; width: 150px;\">is_materialized</th>\n",
       "                        <td style=\"text-align: left;\">False</td>\n",
       "                    </tr>\n",
       "                \n",
       "                    <tr>\n",
       "                        <th style=\"text-align: left; width: 150px;\">number of outputs</th>\n",
       "                        <td style=\"text-align: left;\">16</td>\n",
       "                    </tr>\n",
       "                \n",
       "                    <tr>\n",
       "                        <th style=\"text-align: left; width: 150px;\">shape</th>\n",
       "                        <td style=\"text-align: left;\">(4000, 2000, 140)</td>\n",
       "                    </tr>\n",
       "                \n",
       "                    <tr>\n",
       "                        <th style=\"text-align: left; width: 150px;\">dtype</th>\n",
       "                        <td style=\"text-align: left;\">float64</td>\n",
       "                    </tr>\n",
       "                \n",
       "                    <tr>\n",
       "                        <th style=\"text-align: left; width: 150px;\">chunksize</th>\n",
       "                        <td style=\"text-align: left;\">(1000, 500, 140)</td>\n",
       "                    </tr>\n",
       "                \n",
       "                    <tr>\n",
       "                        <th style=\"text-align: left; width: 150px;\">type</th>\n",
       "                        <td style=\"text-align: left;\">dask.array.core.Array</td>\n",
       "                    </tr>\n",
       "                \n",
       "                    <tr>\n",
       "                        <th style=\"text-align: left; width: 150px;\">chunk_type</th>\n",
       "                        <td style=\"text-align: left;\">cupy._core.core.ndarray</td>\n",
       "                    </tr>\n",
       "                \n",
       "                </table>\n",
       "            </td>\n",
       "            <td>\n",
       "                <svg width=\"231\" height=\"267\" style=\"stroke:rgb(0,0,0);stroke-width:1\" >\n",
       "\n",
       "  <!-- Horizontal lines -->\n",
       "  <line x1=\"10\" y1=\"0\" x2=\"127\" y2=\"117\" style=\"stroke-width:2\" />\n",
       "  <line x1=\"10\" y1=\"25\" x2=\"127\" y2=\"142\" />\n",
       "  <line x1=\"10\" y1=\"50\" x2=\"127\" y2=\"167\" />\n",
       "  <line x1=\"10\" y1=\"75\" x2=\"127\" y2=\"192\" />\n",
       "  <line x1=\"10\" y1=\"100\" x2=\"127\" y2=\"217\" style=\"stroke-width:2\" />\n",
       "\n",
       "  <!-- Vertical lines -->\n",
       "  <line x1=\"10\" y1=\"0\" x2=\"10\" y2=\"100\" style=\"stroke-width:2\" />\n",
       "  <line x1=\"39\" y1=\"29\" x2=\"39\" y2=\"129\" />\n",
       "  <line x1=\"68\" y1=\"58\" x2=\"68\" y2=\"158\" />\n",
       "  <line x1=\"98\" y1=\"88\" x2=\"98\" y2=\"188\" />\n",
       "  <line x1=\"127\" y1=\"117\" x2=\"127\" y2=\"217\" style=\"stroke-width:2\" />\n",
       "\n",
       "  <!-- Colored Rectangle -->\n",
       "  <polygon points=\"10.0,0.0 127.64705882352942,117.64705882352942 127.64705882352942,217.64705882352942 10.0,100.0\" style=\"fill:#ECB172A0;stroke-width:0\"/>\n",
       "\n",
       "  <!-- Horizontal lines -->\n",
       "  <line x1=\"10\" y1=\"0\" x2=\"63\" y2=\"0\" style=\"stroke-width:2\" />\n",
       "  <line x1=\"39\" y1=\"29\" x2=\"93\" y2=\"29\" />\n",
       "  <line x1=\"68\" y1=\"58\" x2=\"122\" y2=\"58\" />\n",
       "  <line x1=\"98\" y1=\"88\" x2=\"152\" y2=\"88\" />\n",
       "  <line x1=\"127\" y1=\"117\" x2=\"181\" y2=\"117\" style=\"stroke-width:2\" />\n",
       "\n",
       "  <!-- Vertical lines -->\n",
       "  <line x1=\"10\" y1=\"0\" x2=\"127\" y2=\"117\" style=\"stroke-width:2\" />\n",
       "  <line x1=\"63\" y1=\"0\" x2=\"181\" y2=\"117\" style=\"stroke-width:2\" />\n",
       "\n",
       "  <!-- Colored Rectangle -->\n",
       "  <polygon points=\"10.0,0.0 63.86661338516172,0.0 181.51367220869113,117.64705882352942 127.64705882352942,117.64705882352942\" style=\"fill:#ECB172A0;stroke-width:0\"/>\n",
       "\n",
       "  <!-- Horizontal lines -->\n",
       "  <line x1=\"127\" y1=\"117\" x2=\"181\" y2=\"117\" style=\"stroke-width:2\" />\n",
       "  <line x1=\"127\" y1=\"142\" x2=\"181\" y2=\"142\" />\n",
       "  <line x1=\"127\" y1=\"167\" x2=\"181\" y2=\"167\" />\n",
       "  <line x1=\"127\" y1=\"192\" x2=\"181\" y2=\"192\" />\n",
       "  <line x1=\"127\" y1=\"217\" x2=\"181\" y2=\"217\" style=\"stroke-width:2\" />\n",
       "\n",
       "  <!-- Vertical lines -->\n",
       "  <line x1=\"127\" y1=\"117\" x2=\"127\" y2=\"217\" style=\"stroke-width:2\" />\n",
       "  <line x1=\"181\" y1=\"117\" x2=\"181\" y2=\"217\" style=\"stroke-width:2\" />\n",
       "\n",
       "  <!-- Colored Rectangle -->\n",
       "  <polygon points=\"127.64705882352942,117.64705882352942 181.51367220869113,117.64705882352942 181.51367220869113,217.64705882352942 127.64705882352942,217.64705882352942\" style=\"fill:#ECB172A0;stroke-width:0\"/>\n",
       "\n",
       "  <!-- Text -->\n",
       "  <text x=\"154.580366\" y=\"237.647059\" font-size=\"1.0rem\" font-weight=\"100\" text-anchor=\"middle\" >140</text>\n",
       "  <text x=\"201.513672\" y=\"167.647059\" font-size=\"1.0rem\" font-weight=\"100\" text-anchor=\"middle\" transform=\"rotate(-90,201.513672,167.647059)\">2000</text>\n",
       "  <text x=\"58.823529\" y=\"178.823529\" font-size=\"1.0rem\" font-weight=\"100\" text-anchor=\"middle\" transform=\"rotate(45,58.823529,178.823529)\">4000</text>\n",
       "</svg>\n",
       "            </td>\n",
       "        </tr>\n",
       "        </table>\n",
       "\n",
       "    </details>\n",
       "</div>\n",
       "            \n",
       "            <div style=\"\">\n",
       "    <svg width=\"24\" height=\"24\" viewBox=\"0 0 32 32\" fill=\"none\" xmlns=\"http://www.w3.org/2000/svg\" style=\"position: absolute;\">\n",
       "        \n",
       "        <circle cx=\"16\" cy=\"16\" r=\"14\" fill=\"#8F8F8F\" style=\"stroke: var(--jp-ui-font-color2, #1D1D1D);\" stroke-width=\"2\"/>\n",
       "        \n",
       "    </svg>\n",
       "\n",
       "    <details style=\"margin-left: 32px;\">\n",
       "        <summary style=\"margin-bottom: 10px; margin-top: 10px;\">\n",
       "            <h4 style=\"display: inline;\">Layer3: concatenate</h4>\n",
       "        </summary>\n",
       "        <p style=\"color: var(--jp-ui-font-color2, #5D5851); margin: -0.25em 0px 0px 0px;\">\n",
       "            concatenate-24ceb3de61d0b0ff573788d6cb562fe3\n",
       "        </p>\n",
       "\n",
       "        <table>\n",
       "        <tr>\n",
       "            <td>\n",
       "                <table>\n",
       "                \n",
       "                    <tr>\n",
       "                        <th style=\"text-align: left; width: 150px;\">layer_type</th>\n",
       "                        <td style=\"text-align: left;\">MaterializedLayer</td>\n",
       "                    </tr>\n",
       "                \n",
       "                    <tr>\n",
       "                        <th style=\"text-align: left; width: 150px;\">is_materialized</th>\n",
       "                        <td style=\"text-align: left;\">True</td>\n",
       "                    </tr>\n",
       "                \n",
       "                    <tr>\n",
       "                        <th style=\"text-align: left; width: 150px;\">number of outputs</th>\n",
       "                        <td style=\"text-align: left;\">192</td>\n",
       "                    </tr>\n",
       "                \n",
       "                    <tr>\n",
       "                        <th style=\"text-align: left; width: 150px;\">shape</th>\n",
       "                        <td style=\"text-align: left;\">(4000, 2000, 1680)</td>\n",
       "                    </tr>\n",
       "                \n",
       "                    <tr>\n",
       "                        <th style=\"text-align: left; width: 150px;\">dtype</th>\n",
       "                        <td style=\"text-align: left;\">float64</td>\n",
       "                    </tr>\n",
       "                \n",
       "                    <tr>\n",
       "                        <th style=\"text-align: left; width: 150px;\">chunksize</th>\n",
       "                        <td style=\"text-align: left;\">(1000, 500, 140)</td>\n",
       "                    </tr>\n",
       "                \n",
       "                    <tr>\n",
       "                        <th style=\"text-align: left; width: 150px;\">type</th>\n",
       "                        <td style=\"text-align: left;\">dask.array.core.Array</td>\n",
       "                    </tr>\n",
       "                \n",
       "                    <tr>\n",
       "                        <th style=\"text-align: left; width: 150px;\">chunk_type</th>\n",
       "                        <td style=\"text-align: left;\">cupy._core.core.ndarray</td>\n",
       "                    </tr>\n",
       "                \n",
       "                </table>\n",
       "            </td>\n",
       "            <td>\n",
       "                <svg width=\"261\" height=\"267\" style=\"stroke:rgb(0,0,0);stroke-width:1\" >\n",
       "\n",
       "  <!-- Horizontal lines -->\n",
       "  <line x1=\"10\" y1=\"0\" x2=\"127\" y2=\"117\" style=\"stroke-width:2\" />\n",
       "  <line x1=\"10\" y1=\"25\" x2=\"127\" y2=\"142\" />\n",
       "  <line x1=\"10\" y1=\"50\" x2=\"127\" y2=\"167\" />\n",
       "  <line x1=\"10\" y1=\"75\" x2=\"127\" y2=\"192\" />\n",
       "  <line x1=\"10\" y1=\"100\" x2=\"127\" y2=\"217\" style=\"stroke-width:2\" />\n",
       "\n",
       "  <!-- Vertical lines -->\n",
       "  <line x1=\"10\" y1=\"0\" x2=\"10\" y2=\"100\" style=\"stroke-width:2\" />\n",
       "  <line x1=\"39\" y1=\"29\" x2=\"39\" y2=\"129\" />\n",
       "  <line x1=\"68\" y1=\"58\" x2=\"68\" y2=\"158\" />\n",
       "  <line x1=\"98\" y1=\"88\" x2=\"98\" y2=\"188\" />\n",
       "  <line x1=\"127\" y1=\"117\" x2=\"127\" y2=\"217\" style=\"stroke-width:2\" />\n",
       "\n",
       "  <!-- Colored Rectangle -->\n",
       "  <polygon points=\"10.0,0.0 127.64705882352942,117.64705882352942 127.64705882352942,217.64705882352942 10.0,100.0\" style=\"fill:#ECB172A0;stroke-width:0\"/>\n",
       "\n",
       "  <!-- Horizontal lines -->\n",
       "  <line x1=\"10\" y1=\"0\" x2=\"94\" y2=\"0\" style=\"stroke-width:2\" />\n",
       "  <line x1=\"39\" y1=\"29\" x2=\"123\" y2=\"29\" />\n",
       "  <line x1=\"68\" y1=\"58\" x2=\"152\" y2=\"58\" />\n",
       "  <line x1=\"98\" y1=\"88\" x2=\"182\" y2=\"88\" />\n",
       "  <line x1=\"127\" y1=\"117\" x2=\"211\" y2=\"117\" style=\"stroke-width:2\" />\n",
       "\n",
       "  <!-- Vertical lines -->\n",
       "  <line x1=\"10\" y1=\"0\" x2=\"127\" y2=\"117\" style=\"stroke-width:2\" />\n",
       "  <line x1=\"17\" y1=\"0\" x2=\"134\" y2=\"117\" />\n",
       "  <line x1=\"24\" y1=\"0\" x2=\"141\" y2=\"117\" />\n",
       "  <line x1=\"31\" y1=\"0\" x2=\"148\" y2=\"117\" />\n",
       "  <line x1=\"38\" y1=\"0\" x2=\"155\" y2=\"117\" />\n",
       "  <line x1=\"45\" y1=\"0\" x2=\"162\" y2=\"117\" />\n",
       "  <line x1=\"52\" y1=\"0\" x2=\"169\" y2=\"117\" />\n",
       "  <line x1=\"59\" y1=\"0\" x2=\"176\" y2=\"117\" />\n",
       "  <line x1=\"66\" y1=\"0\" x2=\"183\" y2=\"117\" />\n",
       "  <line x1=\"73\" y1=\"0\" x2=\"190\" y2=\"117\" />\n",
       "  <line x1=\"80\" y1=\"0\" x2=\"197\" y2=\"117\" />\n",
       "  <line x1=\"87\" y1=\"0\" x2=\"204\" y2=\"117\" />\n",
       "  <line x1=\"94\" y1=\"0\" x2=\"211\" y2=\"117\" style=\"stroke-width:2\" />\n",
       "\n",
       "  <!-- Colored Rectangle -->\n",
       "  <polygon points=\"10.0,0.0 94.0,0.0 211.64705882352942,117.64705882352942 127.64705882352942,117.64705882352942\" style=\"fill:#ECB172A0;stroke-width:0\"/>\n",
       "\n",
       "  <!-- Horizontal lines -->\n",
       "  <line x1=\"127\" y1=\"117\" x2=\"211\" y2=\"117\" style=\"stroke-width:2\" />\n",
       "  <line x1=\"127\" y1=\"142\" x2=\"211\" y2=\"142\" />\n",
       "  <line x1=\"127\" y1=\"167\" x2=\"211\" y2=\"167\" />\n",
       "  <line x1=\"127\" y1=\"192\" x2=\"211\" y2=\"192\" />\n",
       "  <line x1=\"127\" y1=\"217\" x2=\"211\" y2=\"217\" style=\"stroke-width:2\" />\n",
       "\n",
       "  <!-- Vertical lines -->\n",
       "  <line x1=\"127\" y1=\"117\" x2=\"127\" y2=\"217\" style=\"stroke-width:2\" />\n",
       "  <line x1=\"134\" y1=\"117\" x2=\"134\" y2=\"217\" />\n",
       "  <line x1=\"141\" y1=\"117\" x2=\"141\" y2=\"217\" />\n",
       "  <line x1=\"148\" y1=\"117\" x2=\"148\" y2=\"217\" />\n",
       "  <line x1=\"155\" y1=\"117\" x2=\"155\" y2=\"217\" />\n",
       "  <line x1=\"162\" y1=\"117\" x2=\"162\" y2=\"217\" />\n",
       "  <line x1=\"169\" y1=\"117\" x2=\"169\" y2=\"217\" />\n",
       "  <line x1=\"176\" y1=\"117\" x2=\"176\" y2=\"217\" />\n",
       "  <line x1=\"183\" y1=\"117\" x2=\"183\" y2=\"217\" />\n",
       "  <line x1=\"190\" y1=\"117\" x2=\"190\" y2=\"217\" />\n",
       "  <line x1=\"197\" y1=\"117\" x2=\"197\" y2=\"217\" />\n",
       "  <line x1=\"204\" y1=\"117\" x2=\"204\" y2=\"217\" />\n",
       "  <line x1=\"211\" y1=\"117\" x2=\"211\" y2=\"217\" style=\"stroke-width:2\" />\n",
       "\n",
       "  <!-- Colored Rectangle -->\n",
       "  <polygon points=\"127.64705882352942,117.64705882352942 211.64705882352942,117.64705882352942 211.64705882352942,217.64705882352942 127.64705882352942,217.64705882352942\" style=\"fill:#ECB172A0;stroke-width:0\"/>\n",
       "\n",
       "  <!-- Text -->\n",
       "  <text x=\"169.647059\" y=\"237.647059\" font-size=\"1.0rem\" font-weight=\"100\" text-anchor=\"middle\" >1680</text>\n",
       "  <text x=\"231.647059\" y=\"167.647059\" font-size=\"1.0rem\" font-weight=\"100\" text-anchor=\"middle\" transform=\"rotate(-90,231.647059,167.647059)\">2000</text>\n",
       "  <text x=\"58.823529\" y=\"178.823529\" font-size=\"1.0rem\" font-weight=\"100\" text-anchor=\"middle\" transform=\"rotate(45,58.823529,178.823529)\">4000</text>\n",
       "</svg>\n",
       "            </td>\n",
       "        </tr>\n",
       "        </table>\n",
       "\n",
       "    </details>\n",
       "</div>\n",
       "            \n",
       "            <div style=\"\">\n",
       "    <svg width=\"24\" height=\"24\" viewBox=\"0 0 32 32\" fill=\"none\" xmlns=\"http://www.w3.org/2000/svg\" style=\"position: absolute;\">\n",
       "        \n",
       "        <circle cx=\"16\" cy=\"16\" r=\"14\" style=\"stroke: var(--jp-ui-font-color2, #1D1D1D); fill: var(--jp-layout-color1, #F2F2F2);\" stroke-width=\"2\" />\n",
       "        \n",
       "    </svg>\n",
       "\n",
       "    <details style=\"margin-left: 32px;\">\n",
       "        <summary style=\"margin-bottom: 10px; margin-top: 10px;\">\n",
       "            <h4 style=\"display: inline;\">Layer4: sub</h4>\n",
       "        </summary>\n",
       "        <p style=\"color: var(--jp-ui-font-color2, #5D5851); margin: -0.25em 0px 0px 0px;\">\n",
       "            sub-093fe9c8396bd9974170f9e9861562cf\n",
       "        </p>\n",
       "\n",
       "        <table>\n",
       "        <tr>\n",
       "            <td>\n",
       "                <table>\n",
       "                \n",
       "                    <tr>\n",
       "                        <th style=\"text-align: left; width: 150px;\">layer_type</th>\n",
       "                        <td style=\"text-align: left;\">Blockwise</td>\n",
       "                    </tr>\n",
       "                \n",
       "                    <tr>\n",
       "                        <th style=\"text-align: left; width: 150px;\">is_materialized</th>\n",
       "                        <td style=\"text-align: left;\">False</td>\n",
       "                    </tr>\n",
       "                \n",
       "                    <tr>\n",
       "                        <th style=\"text-align: left; width: 150px;\">number of outputs</th>\n",
       "                        <td style=\"text-align: left;\">192</td>\n",
       "                    </tr>\n",
       "                \n",
       "                    <tr>\n",
       "                        <th style=\"text-align: left; width: 150px;\">shape</th>\n",
       "                        <td style=\"text-align: left;\">(4000, 2000, 1680)</td>\n",
       "                    </tr>\n",
       "                \n",
       "                    <tr>\n",
       "                        <th style=\"text-align: left; width: 150px;\">dtype</th>\n",
       "                        <td style=\"text-align: left;\">float64</td>\n",
       "                    </tr>\n",
       "                \n",
       "                    <tr>\n",
       "                        <th style=\"text-align: left; width: 150px;\">chunksize</th>\n",
       "                        <td style=\"text-align: left;\">(1000, 500, 140)</td>\n",
       "                    </tr>\n",
       "                \n",
       "                    <tr>\n",
       "                        <th style=\"text-align: left; width: 150px;\">type</th>\n",
       "                        <td style=\"text-align: left;\">dask.array.core.Array</td>\n",
       "                    </tr>\n",
       "                \n",
       "                    <tr>\n",
       "                        <th style=\"text-align: left; width: 150px;\">chunk_type</th>\n",
       "                        <td style=\"text-align: left;\">cupy._core.core.ndarray</td>\n",
       "                    </tr>\n",
       "                \n",
       "                </table>\n",
       "            </td>\n",
       "            <td>\n",
       "                <svg width=\"261\" height=\"267\" style=\"stroke:rgb(0,0,0);stroke-width:1\" >\n",
       "\n",
       "  <!-- Horizontal lines -->\n",
       "  <line x1=\"10\" y1=\"0\" x2=\"127\" y2=\"117\" style=\"stroke-width:2\" />\n",
       "  <line x1=\"10\" y1=\"25\" x2=\"127\" y2=\"142\" />\n",
       "  <line x1=\"10\" y1=\"50\" x2=\"127\" y2=\"167\" />\n",
       "  <line x1=\"10\" y1=\"75\" x2=\"127\" y2=\"192\" />\n",
       "  <line x1=\"10\" y1=\"100\" x2=\"127\" y2=\"217\" style=\"stroke-width:2\" />\n",
       "\n",
       "  <!-- Vertical lines -->\n",
       "  <line x1=\"10\" y1=\"0\" x2=\"10\" y2=\"100\" style=\"stroke-width:2\" />\n",
       "  <line x1=\"39\" y1=\"29\" x2=\"39\" y2=\"129\" />\n",
       "  <line x1=\"68\" y1=\"58\" x2=\"68\" y2=\"158\" />\n",
       "  <line x1=\"98\" y1=\"88\" x2=\"98\" y2=\"188\" />\n",
       "  <line x1=\"127\" y1=\"117\" x2=\"127\" y2=\"217\" style=\"stroke-width:2\" />\n",
       "\n",
       "  <!-- Colored Rectangle -->\n",
       "  <polygon points=\"10.0,0.0 127.64705882352942,117.64705882352942 127.64705882352942,217.64705882352942 10.0,100.0\" style=\"fill:#ECB172A0;stroke-width:0\"/>\n",
       "\n",
       "  <!-- Horizontal lines -->\n",
       "  <line x1=\"10\" y1=\"0\" x2=\"94\" y2=\"0\" style=\"stroke-width:2\" />\n",
       "  <line x1=\"39\" y1=\"29\" x2=\"123\" y2=\"29\" />\n",
       "  <line x1=\"68\" y1=\"58\" x2=\"152\" y2=\"58\" />\n",
       "  <line x1=\"98\" y1=\"88\" x2=\"182\" y2=\"88\" />\n",
       "  <line x1=\"127\" y1=\"117\" x2=\"211\" y2=\"117\" style=\"stroke-width:2\" />\n",
       "\n",
       "  <!-- Vertical lines -->\n",
       "  <line x1=\"10\" y1=\"0\" x2=\"127\" y2=\"117\" style=\"stroke-width:2\" />\n",
       "  <line x1=\"17\" y1=\"0\" x2=\"134\" y2=\"117\" />\n",
       "  <line x1=\"24\" y1=\"0\" x2=\"141\" y2=\"117\" />\n",
       "  <line x1=\"31\" y1=\"0\" x2=\"148\" y2=\"117\" />\n",
       "  <line x1=\"38\" y1=\"0\" x2=\"155\" y2=\"117\" />\n",
       "  <line x1=\"45\" y1=\"0\" x2=\"162\" y2=\"117\" />\n",
       "  <line x1=\"52\" y1=\"0\" x2=\"169\" y2=\"117\" />\n",
       "  <line x1=\"59\" y1=\"0\" x2=\"176\" y2=\"117\" />\n",
       "  <line x1=\"66\" y1=\"0\" x2=\"183\" y2=\"117\" />\n",
       "  <line x1=\"73\" y1=\"0\" x2=\"190\" y2=\"117\" />\n",
       "  <line x1=\"80\" y1=\"0\" x2=\"197\" y2=\"117\" />\n",
       "  <line x1=\"87\" y1=\"0\" x2=\"204\" y2=\"117\" />\n",
       "  <line x1=\"94\" y1=\"0\" x2=\"211\" y2=\"117\" style=\"stroke-width:2\" />\n",
       "\n",
       "  <!-- Colored Rectangle -->\n",
       "  <polygon points=\"10.0,0.0 94.0,0.0 211.64705882352942,117.64705882352942 127.64705882352942,117.64705882352942\" style=\"fill:#ECB172A0;stroke-width:0\"/>\n",
       "\n",
       "  <!-- Horizontal lines -->\n",
       "  <line x1=\"127\" y1=\"117\" x2=\"211\" y2=\"117\" style=\"stroke-width:2\" />\n",
       "  <line x1=\"127\" y1=\"142\" x2=\"211\" y2=\"142\" />\n",
       "  <line x1=\"127\" y1=\"167\" x2=\"211\" y2=\"167\" />\n",
       "  <line x1=\"127\" y1=\"192\" x2=\"211\" y2=\"192\" />\n",
       "  <line x1=\"127\" y1=\"217\" x2=\"211\" y2=\"217\" style=\"stroke-width:2\" />\n",
       "\n",
       "  <!-- Vertical lines -->\n",
       "  <line x1=\"127\" y1=\"117\" x2=\"127\" y2=\"217\" style=\"stroke-width:2\" />\n",
       "  <line x1=\"134\" y1=\"117\" x2=\"134\" y2=\"217\" />\n",
       "  <line x1=\"141\" y1=\"117\" x2=\"141\" y2=\"217\" />\n",
       "  <line x1=\"148\" y1=\"117\" x2=\"148\" y2=\"217\" />\n",
       "  <line x1=\"155\" y1=\"117\" x2=\"155\" y2=\"217\" />\n",
       "  <line x1=\"162\" y1=\"117\" x2=\"162\" y2=\"217\" />\n",
       "  <line x1=\"169\" y1=\"117\" x2=\"169\" y2=\"217\" />\n",
       "  <line x1=\"176\" y1=\"117\" x2=\"176\" y2=\"217\" />\n",
       "  <line x1=\"183\" y1=\"117\" x2=\"183\" y2=\"217\" />\n",
       "  <line x1=\"190\" y1=\"117\" x2=\"190\" y2=\"217\" />\n",
       "  <line x1=\"197\" y1=\"117\" x2=\"197\" y2=\"217\" />\n",
       "  <line x1=\"204\" y1=\"117\" x2=\"204\" y2=\"217\" />\n",
       "  <line x1=\"211\" y1=\"117\" x2=\"211\" y2=\"217\" style=\"stroke-width:2\" />\n",
       "\n",
       "  <!-- Colored Rectangle -->\n",
       "  <polygon points=\"127.64705882352942,117.64705882352942 211.64705882352942,117.64705882352942 211.64705882352942,217.64705882352942 127.64705882352942,217.64705882352942\" style=\"fill:#ECB172A0;stroke-width:0\"/>\n",
       "\n",
       "  <!-- Text -->\n",
       "  <text x=\"169.647059\" y=\"237.647059\" font-size=\"1.0rem\" font-weight=\"100\" text-anchor=\"middle\" >1680</text>\n",
       "  <text x=\"231.647059\" y=\"167.647059\" font-size=\"1.0rem\" font-weight=\"100\" text-anchor=\"middle\" transform=\"rotate(-90,231.647059,167.647059)\">2000</text>\n",
       "  <text x=\"58.823529\" y=\"178.823529\" font-size=\"1.0rem\" font-weight=\"100\" text-anchor=\"middle\" transform=\"rotate(45,58.823529,178.823529)\">4000</text>\n",
       "</svg>\n",
       "            </td>\n",
       "        </tr>\n",
       "        </table>\n",
       "\n",
       "    </details>\n",
       "</div>\n",
       "            \n",
       "            <div style=\"\">\n",
       "    <svg width=\"24\" height=\"24\" viewBox=\"0 0 32 32\" fill=\"none\" xmlns=\"http://www.w3.org/2000/svg\" style=\"position: absolute;\">\n",
       "        \n",
       "        <circle cx=\"16\" cy=\"16\" r=\"14\" style=\"stroke: var(--jp-ui-font-color2, #1D1D1D); fill: var(--jp-layout-color1, #F2F2F2);\" stroke-width=\"2\" />\n",
       "        \n",
       "    </svg>\n",
       "\n",
       "    <details style=\"margin-left: 32px;\">\n",
       "        <summary style=\"margin-bottom: 10px; margin-top: 10px;\">\n",
       "            <h4 style=\"display: inline;\">Layer5: mean_chunk</h4>\n",
       "        </summary>\n",
       "        <p style=\"color: var(--jp-ui-font-color2, #5D5851); margin: -0.25em 0px 0px 0px;\">\n",
       "            mean_chunk-531ead781e91c7fa6a7e1202e6f60623\n",
       "        </p>\n",
       "\n",
       "        <table>\n",
       "        <tr>\n",
       "            <td>\n",
       "                <table>\n",
       "                \n",
       "                    <tr>\n",
       "                        <th style=\"text-align: left; width: 150px;\">layer_type</th>\n",
       "                        <td style=\"text-align: left;\">Blockwise</td>\n",
       "                    </tr>\n",
       "                \n",
       "                    <tr>\n",
       "                        <th style=\"text-align: left; width: 150px;\">is_materialized</th>\n",
       "                        <td style=\"text-align: left;\">False</td>\n",
       "                    </tr>\n",
       "                \n",
       "                    <tr>\n",
       "                        <th style=\"text-align: left; width: 150px;\">number of outputs</th>\n",
       "                        <td style=\"text-align: left;\">192</td>\n",
       "                    </tr>\n",
       "                \n",
       "                    <tr>\n",
       "                        <th style=\"text-align: left; width: 150px;\">shape</th>\n",
       "                        <td style=\"text-align: left;\">(4000, 2000, 1680)</td>\n",
       "                    </tr>\n",
       "                \n",
       "                    <tr>\n",
       "                        <th style=\"text-align: left; width: 150px;\">dtype</th>\n",
       "                        <td style=\"text-align: left;\">float64</td>\n",
       "                    </tr>\n",
       "                \n",
       "                    <tr>\n",
       "                        <th style=\"text-align: left; width: 150px;\">chunksize</th>\n",
       "                        <td style=\"text-align: left;\">(1000, 500, 140)</td>\n",
       "                    </tr>\n",
       "                \n",
       "                    <tr>\n",
       "                        <th style=\"text-align: left; width: 150px;\">type</th>\n",
       "                        <td style=\"text-align: left;\">dask.array.core.Array</td>\n",
       "                    </tr>\n",
       "                \n",
       "                    <tr>\n",
       "                        <th style=\"text-align: left; width: 150px;\">chunk_type</th>\n",
       "                        <td style=\"text-align: left;\">cupy._core.core.ndarray</td>\n",
       "                    </tr>\n",
       "                \n",
       "                </table>\n",
       "            </td>\n",
       "            <td>\n",
       "                <svg width=\"261\" height=\"267\" style=\"stroke:rgb(0,0,0);stroke-width:1\" >\n",
       "\n",
       "  <!-- Horizontal lines -->\n",
       "  <line x1=\"10\" y1=\"0\" x2=\"127\" y2=\"117\" style=\"stroke-width:2\" />\n",
       "  <line x1=\"10\" y1=\"25\" x2=\"127\" y2=\"142\" />\n",
       "  <line x1=\"10\" y1=\"50\" x2=\"127\" y2=\"167\" />\n",
       "  <line x1=\"10\" y1=\"75\" x2=\"127\" y2=\"192\" />\n",
       "  <line x1=\"10\" y1=\"100\" x2=\"127\" y2=\"217\" style=\"stroke-width:2\" />\n",
       "\n",
       "  <!-- Vertical lines -->\n",
       "  <line x1=\"10\" y1=\"0\" x2=\"10\" y2=\"100\" style=\"stroke-width:2\" />\n",
       "  <line x1=\"39\" y1=\"29\" x2=\"39\" y2=\"129\" />\n",
       "  <line x1=\"68\" y1=\"58\" x2=\"68\" y2=\"158\" />\n",
       "  <line x1=\"98\" y1=\"88\" x2=\"98\" y2=\"188\" />\n",
       "  <line x1=\"127\" y1=\"117\" x2=\"127\" y2=\"217\" style=\"stroke-width:2\" />\n",
       "\n",
       "  <!-- Colored Rectangle -->\n",
       "  <polygon points=\"10.0,0.0 127.64705882352942,117.64705882352942 127.64705882352942,217.64705882352942 10.0,100.0\" style=\"fill:#ECB172A0;stroke-width:0\"/>\n",
       "\n",
       "  <!-- Horizontal lines -->\n",
       "  <line x1=\"10\" y1=\"0\" x2=\"94\" y2=\"0\" style=\"stroke-width:2\" />\n",
       "  <line x1=\"39\" y1=\"29\" x2=\"123\" y2=\"29\" />\n",
       "  <line x1=\"68\" y1=\"58\" x2=\"152\" y2=\"58\" />\n",
       "  <line x1=\"98\" y1=\"88\" x2=\"182\" y2=\"88\" />\n",
       "  <line x1=\"127\" y1=\"117\" x2=\"211\" y2=\"117\" style=\"stroke-width:2\" />\n",
       "\n",
       "  <!-- Vertical lines -->\n",
       "  <line x1=\"10\" y1=\"0\" x2=\"127\" y2=\"117\" style=\"stroke-width:2\" />\n",
       "  <line x1=\"17\" y1=\"0\" x2=\"134\" y2=\"117\" />\n",
       "  <line x1=\"24\" y1=\"0\" x2=\"141\" y2=\"117\" />\n",
       "  <line x1=\"31\" y1=\"0\" x2=\"148\" y2=\"117\" />\n",
       "  <line x1=\"38\" y1=\"0\" x2=\"155\" y2=\"117\" />\n",
       "  <line x1=\"45\" y1=\"0\" x2=\"162\" y2=\"117\" />\n",
       "  <line x1=\"52\" y1=\"0\" x2=\"169\" y2=\"117\" />\n",
       "  <line x1=\"59\" y1=\"0\" x2=\"176\" y2=\"117\" />\n",
       "  <line x1=\"66\" y1=\"0\" x2=\"183\" y2=\"117\" />\n",
       "  <line x1=\"73\" y1=\"0\" x2=\"190\" y2=\"117\" />\n",
       "  <line x1=\"80\" y1=\"0\" x2=\"197\" y2=\"117\" />\n",
       "  <line x1=\"87\" y1=\"0\" x2=\"204\" y2=\"117\" />\n",
       "  <line x1=\"94\" y1=\"0\" x2=\"211\" y2=\"117\" style=\"stroke-width:2\" />\n",
       "\n",
       "  <!-- Colored Rectangle -->\n",
       "  <polygon points=\"10.0,0.0 94.0,0.0 211.64705882352942,117.64705882352942 127.64705882352942,117.64705882352942\" style=\"fill:#ECB172A0;stroke-width:0\"/>\n",
       "\n",
       "  <!-- Horizontal lines -->\n",
       "  <line x1=\"127\" y1=\"117\" x2=\"211\" y2=\"117\" style=\"stroke-width:2\" />\n",
       "  <line x1=\"127\" y1=\"142\" x2=\"211\" y2=\"142\" />\n",
       "  <line x1=\"127\" y1=\"167\" x2=\"211\" y2=\"167\" />\n",
       "  <line x1=\"127\" y1=\"192\" x2=\"211\" y2=\"192\" />\n",
       "  <line x1=\"127\" y1=\"217\" x2=\"211\" y2=\"217\" style=\"stroke-width:2\" />\n",
       "\n",
       "  <!-- Vertical lines -->\n",
       "  <line x1=\"127\" y1=\"117\" x2=\"127\" y2=\"217\" style=\"stroke-width:2\" />\n",
       "  <line x1=\"134\" y1=\"117\" x2=\"134\" y2=\"217\" />\n",
       "  <line x1=\"141\" y1=\"117\" x2=\"141\" y2=\"217\" />\n",
       "  <line x1=\"148\" y1=\"117\" x2=\"148\" y2=\"217\" />\n",
       "  <line x1=\"155\" y1=\"117\" x2=\"155\" y2=\"217\" />\n",
       "  <line x1=\"162\" y1=\"117\" x2=\"162\" y2=\"217\" />\n",
       "  <line x1=\"169\" y1=\"117\" x2=\"169\" y2=\"217\" />\n",
       "  <line x1=\"176\" y1=\"117\" x2=\"176\" y2=\"217\" />\n",
       "  <line x1=\"183\" y1=\"117\" x2=\"183\" y2=\"217\" />\n",
       "  <line x1=\"190\" y1=\"117\" x2=\"190\" y2=\"217\" />\n",
       "  <line x1=\"197\" y1=\"117\" x2=\"197\" y2=\"217\" />\n",
       "  <line x1=\"204\" y1=\"117\" x2=\"204\" y2=\"217\" />\n",
       "  <line x1=\"211\" y1=\"117\" x2=\"211\" y2=\"217\" style=\"stroke-width:2\" />\n",
       "\n",
       "  <!-- Colored Rectangle -->\n",
       "  <polygon points=\"127.64705882352942,117.64705882352942 211.64705882352942,117.64705882352942 211.64705882352942,217.64705882352942 127.64705882352942,217.64705882352942\" style=\"fill:#ECB172A0;stroke-width:0\"/>\n",
       "\n",
       "  <!-- Text -->\n",
       "  <text x=\"169.647059\" y=\"237.647059\" font-size=\"1.0rem\" font-weight=\"100\" text-anchor=\"middle\" >1680</text>\n",
       "  <text x=\"231.647059\" y=\"167.647059\" font-size=\"1.0rem\" font-weight=\"100\" text-anchor=\"middle\" transform=\"rotate(-90,231.647059,167.647059)\">2000</text>\n",
       "  <text x=\"58.823529\" y=\"178.823529\" font-size=\"1.0rem\" font-weight=\"100\" text-anchor=\"middle\" transform=\"rotate(45,58.823529,178.823529)\">4000</text>\n",
       "</svg>\n",
       "            </td>\n",
       "        </tr>\n",
       "        </table>\n",
       "\n",
       "    </details>\n",
       "</div>\n",
       "            \n",
       "            <div style=\"\">\n",
       "    <svg width=\"24\" height=\"24\" viewBox=\"0 0 32 32\" fill=\"none\" xmlns=\"http://www.w3.org/2000/svg\" style=\"position: absolute;\">\n",
       "        \n",
       "        <circle cx=\"16\" cy=\"16\" r=\"14\" fill=\"#8F8F8F\" style=\"stroke: var(--jp-ui-font-color2, #1D1D1D);\" stroke-width=\"2\"/>\n",
       "        \n",
       "    </svg>\n",
       "\n",
       "    <details style=\"margin-left: 32px;\">\n",
       "        <summary style=\"margin-bottom: 10px; margin-top: 10px;\">\n",
       "            <h4 style=\"display: inline;\">Layer6: mean_combine-partial</h4>\n",
       "        </summary>\n",
       "        <p style=\"color: var(--jp-ui-font-color2, #5D5851); margin: -0.25em 0px 0px 0px;\">\n",
       "            mean_combine-partial-755fbbc74e3e8418a15be3fa795b0520\n",
       "        </p>\n",
       "\n",
       "        <table>\n",
       "        <tr>\n",
       "            <td>\n",
       "                <table>\n",
       "                \n",
       "                    <tr>\n",
       "                        <th style=\"text-align: left; width: 150px;\">layer_type</th>\n",
       "                        <td style=\"text-align: left;\">MaterializedLayer</td>\n",
       "                    </tr>\n",
       "                \n",
       "                    <tr>\n",
       "                        <th style=\"text-align: left; width: 150px;\">is_materialized</th>\n",
       "                        <td style=\"text-align: left;\">True</td>\n",
       "                    </tr>\n",
       "                \n",
       "                    <tr>\n",
       "                        <th style=\"text-align: left; width: 150px;\">number of outputs</th>\n",
       "                        <td style=\"text-align: left;\">48</td>\n",
       "                    </tr>\n",
       "                \n",
       "                    <tr>\n",
       "                        <th style=\"text-align: left; width: 150px;\">shape</th>\n",
       "                        <td style=\"text-align: left;\">(4000, 2000, 3)</td>\n",
       "                    </tr>\n",
       "                \n",
       "                    <tr>\n",
       "                        <th style=\"text-align: left; width: 150px;\">dtype</th>\n",
       "                        <td style=\"text-align: left;\">float64</td>\n",
       "                    </tr>\n",
       "                \n",
       "                    <tr>\n",
       "                        <th style=\"text-align: left; width: 150px;\">chunksize</th>\n",
       "                        <td style=\"text-align: left;\">(1000, 500, 1)</td>\n",
       "                    </tr>\n",
       "                \n",
       "                    <tr>\n",
       "                        <th style=\"text-align: left; width: 150px;\">type</th>\n",
       "                        <td style=\"text-align: left;\">dask.array.core.Array</td>\n",
       "                    </tr>\n",
       "                \n",
       "                    <tr>\n",
       "                        <th style=\"text-align: left; width: 150px;\">chunk_type</th>\n",
       "                        <td style=\"text-align: left;\">cupy._core.core.ndarray</td>\n",
       "                    </tr>\n",
       "                \n",
       "                </table>\n",
       "            </td>\n",
       "            <td>\n",
       "                <svg width=\"220\" height=\"267\" style=\"stroke:rgb(0,0,0);stroke-width:1\" >\n",
       "\n",
       "  <!-- Horizontal lines -->\n",
       "  <line x1=\"10\" y1=\"0\" x2=\"127\" y2=\"117\" style=\"stroke-width:2\" />\n",
       "  <line x1=\"10\" y1=\"25\" x2=\"127\" y2=\"142\" />\n",
       "  <line x1=\"10\" y1=\"50\" x2=\"127\" y2=\"167\" />\n",
       "  <line x1=\"10\" y1=\"75\" x2=\"127\" y2=\"192\" />\n",
       "  <line x1=\"10\" y1=\"100\" x2=\"127\" y2=\"217\" style=\"stroke-width:2\" />\n",
       "\n",
       "  <!-- Vertical lines -->\n",
       "  <line x1=\"10\" y1=\"0\" x2=\"10\" y2=\"100\" style=\"stroke-width:2\" />\n",
       "  <line x1=\"39\" y1=\"29\" x2=\"39\" y2=\"129\" />\n",
       "  <line x1=\"68\" y1=\"58\" x2=\"68\" y2=\"158\" />\n",
       "  <line x1=\"98\" y1=\"88\" x2=\"98\" y2=\"188\" />\n",
       "  <line x1=\"127\" y1=\"117\" x2=\"127\" y2=\"217\" style=\"stroke-width:2\" />\n",
       "\n",
       "  <!-- Colored Rectangle -->\n",
       "  <polygon points=\"10.0,0.0 127.64705882352942,117.64705882352942 127.64705882352942,217.64705882352942 10.0,100.0\" style=\"fill:#ECB172A0;stroke-width:0\"/>\n",
       "\n",
       "  <!-- Horizontal lines -->\n",
       "  <line x1=\"10\" y1=\"0\" x2=\"52\" y2=\"0\" style=\"stroke-width:2\" />\n",
       "  <line x1=\"39\" y1=\"29\" x2=\"81\" y2=\"29\" />\n",
       "  <line x1=\"68\" y1=\"58\" x2=\"111\" y2=\"58\" />\n",
       "  <line x1=\"98\" y1=\"88\" x2=\"140\" y2=\"88\" />\n",
       "  <line x1=\"127\" y1=\"117\" x2=\"170\" y2=\"117\" style=\"stroke-width:2\" />\n",
       "\n",
       "  <!-- Vertical lines -->\n",
       "  <line x1=\"10\" y1=\"0\" x2=\"127\" y2=\"117\" style=\"stroke-width:2\" />\n",
       "  <line x1=\"24\" y1=\"0\" x2=\"141\" y2=\"117\" />\n",
       "  <line x1=\"38\" y1=\"0\" x2=\"155\" y2=\"117\" />\n",
       "  <line x1=\"52\" y1=\"0\" x2=\"170\" y2=\"117\" style=\"stroke-width:2\" />\n",
       "\n",
       "  <!-- Colored Rectangle -->\n",
       "  <polygon points=\"10.0,0.0 52.354360857637474,0.0 170.0014196811669,117.64705882352942 127.64705882352942,117.64705882352942\" style=\"fill:#ECB172A0;stroke-width:0\"/>\n",
       "\n",
       "  <!-- Horizontal lines -->\n",
       "  <line x1=\"127\" y1=\"117\" x2=\"170\" y2=\"117\" style=\"stroke-width:2\" />\n",
       "  <line x1=\"127\" y1=\"142\" x2=\"170\" y2=\"142\" />\n",
       "  <line x1=\"127\" y1=\"167\" x2=\"170\" y2=\"167\" />\n",
       "  <line x1=\"127\" y1=\"192\" x2=\"170\" y2=\"192\" />\n",
       "  <line x1=\"127\" y1=\"217\" x2=\"170\" y2=\"217\" style=\"stroke-width:2\" />\n",
       "\n",
       "  <!-- Vertical lines -->\n",
       "  <line x1=\"127\" y1=\"117\" x2=\"127\" y2=\"217\" style=\"stroke-width:2\" />\n",
       "  <line x1=\"141\" y1=\"117\" x2=\"141\" y2=\"217\" />\n",
       "  <line x1=\"155\" y1=\"117\" x2=\"155\" y2=\"217\" />\n",
       "  <line x1=\"170\" y1=\"117\" x2=\"170\" y2=\"217\" style=\"stroke-width:2\" />\n",
       "\n",
       "  <!-- Colored Rectangle -->\n",
       "  <polygon points=\"127.64705882352942,117.64705882352942 170.0014196811669,117.64705882352942 170.0014196811669,217.64705882352942 127.64705882352942,217.64705882352942\" style=\"fill:#ECB172A0;stroke-width:0\"/>\n",
       "\n",
       "  <!-- Text -->\n",
       "  <text x=\"148.824239\" y=\"237.647059\" font-size=\"1.0rem\" font-weight=\"100\" text-anchor=\"middle\" >3</text>\n",
       "  <text x=\"190.001420\" y=\"167.647059\" font-size=\"1.0rem\" font-weight=\"100\" text-anchor=\"middle\" transform=\"rotate(-90,190.001420,167.647059)\">2000</text>\n",
       "  <text x=\"58.823529\" y=\"178.823529\" font-size=\"1.0rem\" font-weight=\"100\" text-anchor=\"middle\" transform=\"rotate(45,58.823529,178.823529)\">4000</text>\n",
       "</svg>\n",
       "            </td>\n",
       "        </tr>\n",
       "        </table>\n",
       "\n",
       "    </details>\n",
       "</div>\n",
       "            \n",
       "            <div style=\"\">\n",
       "    <svg width=\"24\" height=\"24\" viewBox=\"0 0 32 32\" fill=\"none\" xmlns=\"http://www.w3.org/2000/svg\" style=\"position: absolute;\">\n",
       "        \n",
       "        <circle cx=\"16\" cy=\"16\" r=\"14\" fill=\"#8F8F8F\" style=\"stroke: var(--jp-ui-font-color2, #1D1D1D);\" stroke-width=\"2\"/>\n",
       "        \n",
       "    </svg>\n",
       "\n",
       "    <details style=\"margin-left: 32px;\">\n",
       "        <summary style=\"margin-bottom: 10px; margin-top: 10px;\">\n",
       "            <h4 style=\"display: inline;\">Layer7: mean_agg-aggregate</h4>\n",
       "        </summary>\n",
       "        <p style=\"color: var(--jp-ui-font-color2, #5D5851); margin: -0.25em 0px 0px 0px;\">\n",
       "            mean_agg-aggregate-6271223e55f4c885b20b0e134ec94962\n",
       "        </p>\n",
       "\n",
       "        <table>\n",
       "        <tr>\n",
       "            <td>\n",
       "                <table>\n",
       "                \n",
       "                    <tr>\n",
       "                        <th style=\"text-align: left; width: 150px;\">layer_type</th>\n",
       "                        <td style=\"text-align: left;\">MaterializedLayer</td>\n",
       "                    </tr>\n",
       "                \n",
       "                    <tr>\n",
       "                        <th style=\"text-align: left; width: 150px;\">is_materialized</th>\n",
       "                        <td style=\"text-align: left;\">True</td>\n",
       "                    </tr>\n",
       "                \n",
       "                    <tr>\n",
       "                        <th style=\"text-align: left; width: 150px;\">number of outputs</th>\n",
       "                        <td style=\"text-align: left;\">16</td>\n",
       "                    </tr>\n",
       "                \n",
       "                    <tr>\n",
       "                        <th style=\"text-align: left; width: 150px;\">shape</th>\n",
       "                        <td style=\"text-align: left;\">(4000, 2000)</td>\n",
       "                    </tr>\n",
       "                \n",
       "                    <tr>\n",
       "                        <th style=\"text-align: left; width: 150px;\">dtype</th>\n",
       "                        <td style=\"text-align: left;\">float64</td>\n",
       "                    </tr>\n",
       "                \n",
       "                    <tr>\n",
       "                        <th style=\"text-align: left; width: 150px;\">chunksize</th>\n",
       "                        <td style=\"text-align: left;\">(1000, 500)</td>\n",
       "                    </tr>\n",
       "                \n",
       "                    <tr>\n",
       "                        <th style=\"text-align: left; width: 150px;\">type</th>\n",
       "                        <td style=\"text-align: left;\">dask.array.core.Array</td>\n",
       "                    </tr>\n",
       "                \n",
       "                    <tr>\n",
       "                        <th style=\"text-align: left; width: 150px;\">chunk_type</th>\n",
       "                        <td style=\"text-align: left;\">cupy._core.core.ndarray</td>\n",
       "                    </tr>\n",
       "                \n",
       "                </table>\n",
       "            </td>\n",
       "            <td>\n",
       "                <svg width=\"150\" height=\"250\" style=\"stroke:rgb(0,0,0);stroke-width:1\" >\n",
       "\n",
       "  <!-- Horizontal lines -->\n",
       "  <line x1=\"0\" y1=\"0\" x2=\"100\" y2=\"0\" style=\"stroke-width:2\" />\n",
       "  <line x1=\"0\" y1=\"50\" x2=\"100\" y2=\"50\" />\n",
       "  <line x1=\"0\" y1=\"100\" x2=\"100\" y2=\"100\" />\n",
       "  <line x1=\"0\" y1=\"150\" x2=\"100\" y2=\"150\" />\n",
       "  <line x1=\"0\" y1=\"200\" x2=\"100\" y2=\"200\" style=\"stroke-width:2\" />\n",
       "\n",
       "  <!-- Vertical lines -->\n",
       "  <line x1=\"0\" y1=\"0\" x2=\"0\" y2=\"200\" style=\"stroke-width:2\" />\n",
       "  <line x1=\"25\" y1=\"0\" x2=\"25\" y2=\"200\" />\n",
       "  <line x1=\"50\" y1=\"0\" x2=\"50\" y2=\"200\" />\n",
       "  <line x1=\"75\" y1=\"0\" x2=\"75\" y2=\"200\" />\n",
       "  <line x1=\"100\" y1=\"0\" x2=\"100\" y2=\"200\" style=\"stroke-width:2\" />\n",
       "\n",
       "  <!-- Colored Rectangle -->\n",
       "  <polygon points=\"0.0,0.0 100.0,0.0 100.0,200.0 0.0,200.0\" style=\"fill:#ECB172A0;stroke-width:0\"/>\n",
       "\n",
       "  <!-- Text -->\n",
       "  <text x=\"50.000000\" y=\"220.000000\" font-size=\"1.0rem\" font-weight=\"100\" text-anchor=\"middle\" >2000</text>\n",
       "  <text x=\"120.000000\" y=\"100.000000\" font-size=\"1.0rem\" font-weight=\"100\" text-anchor=\"middle\" transform=\"rotate(-90,120.000000,100.000000)\">4000</text>\n",
       "</svg>\n",
       "            </td>\n",
       "        </tr>\n",
       "        </table>\n",
       "\n",
       "    </details>\n",
       "</div>\n",
       "            \n",
       "        </div>\n",
       "    </div>\n",
       "</div>"
      ],
      "text/plain": [
       "HighLevelGraph with 7 layers.\n",
       "<dask.highlevelgraph.HighLevelGraph object at 0x2afc9e63e820>\n",
       " 0. array-2d6d607525c93a60d3ebad8394f5ac20\n",
       " 1. asarray-0bb0329305b0a5ad59d3ae1aa871cf9b\n",
       " 2. concatenate-24ceb3de61d0b0ff573788d6cb562fe3\n",
       " 3. sub-093fe9c8396bd9974170f9e9861562cf\n",
       " 4. mean_chunk-531ead781e91c7fa6a7e1202e6f60623\n",
       " 5. mean_combine-partial-755fbbc74e3e8418a15be3fa795b0520\n",
       " 6. mean_agg-aggregate-6271223e55f4c885b20b0e134ec94962"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tg.dask"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "761d3d8e-fd4c-40c0-bdbb-0f9365a44368",
   "metadata": {},
   "source": [
    "Using the above attribute, we can see the tasks that Dask will run, but remember, it does not actually perform any computation (*or even map our CPU array into GPU memory*) until we run `.compute()`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "id": "46796830-8354-424e-8e30-f57cec6f659b",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 2.63 s, sys: 9.96 s, total: 12.6 s\n",
      "Wall time: 18.1 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "dg_means = tg.compute()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "id": "a51f1d0c-c4dd-4371-ab7b-20549d893d2f",
   "metadata": {},
   "outputs": [],
   "source": [
    "del dg_means"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "id": "9e248797-8a69-4a03-9014-15df7e58c2a6",
   "metadata": {},
   "outputs": [],
   "source": [
    "client.shutdown()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "id": "9e9cddd5-ca08-427d-9586-c0cf956e66d0",
   "metadata": {},
   "outputs": [],
   "source": [
    "del client, cluster"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e0e5eb40-46b9-4e6c-bc8b-5d9d7ffc1ecd",
   "metadata": {},
   "source": [
    "### Scheduling GPUs with `dask-jobqueue`\n",
    "\n",
    "In this workflow, we use a scheduler cluster object like `PBSCluster` to submit batch jobs to run single workers. This cluster is often easier to augment with GPUs, as we only need to make a few tweaks to our worker resource specification and GPU pinning to each worker is done implicitly through job GPU assignment."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b9917c09-1f2d-4118-88b2-e2af18331bcc",
   "metadata": {},
   "source": [
    "#### Consider the following distributed CPU code\n",
    "\n",
    "```python\n",
    "from dask_jobqueue import PBSCluster\n",
    "from dask.distributed import Client\n",
    "\n",
    "cluster = PBSCluster(\n",
    "    cores=1,\n",
    "    memory='35GiB',\n",
    "    processes=1,\n",
    "    resource_spec='select=1:ncpus=1:mem=35GB',\n",
    "    queue='casper',\n",
    "    walltime='10:00',\n",
    "    interface='ib0'\n",
    ")\n",
    "```\n",
    "\n",
    "#### Let's modify it to create GPU workers from dask_jobqueue import PBSCluster"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "id": "80916dc5-0d5b-43fa-9e85-689d318aa00a",
   "metadata": {},
   "outputs": [],
   "source": [
    "from dask_jobqueue import PBSCluster\n",
    "from dask.distributed import Client\n",
    "\n",
    "cluster = PBSCluster(\n",
    "    cores=1,\n",
    "    memory='35GiB',\n",
    "    processes=1,\n",
    "    resource_spec='select=1:ncpus=1:mem=35GB:ngpus=1',\n",
    "    job_extra=['-l gpu_type=a100'],\n",
    "    queue='preview',\n",
    "    walltime='10:00',\n",
    "    interface='ib0'\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "id": "67f4256c-f0dc-464c-ad51-aa97f3733b29",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "#!/usr/bin/env bash\n",
      "\n",
      "#PBS -N dask-worker\n",
      "#PBS -q preview\n",
      "#PBS -A SCSG0001\n",
      "#PBS -l select=1:ncpus=1:mem=35GB:ngpus=1\n",
      "#PBS -l walltime=10:00\n",
      "#PBS -e dask-worker-logs/\n",
      "#PBS -o dask-worker-logs/\n",
      "#PBS -l gpu_type=a100\n",
      "\n",
      "/glade/u/apps/opt/conda/envs/pygpu-dask/bin/python -m distributed.cli.dask_worker tcp://10.12.205.44:34094 --nthreads 1 --memory-limit 35.00GiB --name dummy-name --nanny --death-timeout 60 --local-directory /local_scratch/pbs.$PBS_JOBID/dask --interface ib0\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(cluster.job_script())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "id": "7a615d8a-3d22-4280-bc55-436a4f9230ae",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "    <div style=\"width: 24px; height: 24px; background-color: #e1e1e1; border: 3px solid #9D9D9D; border-radius: 5px; position: absolute;\"> </div>\n",
       "    <div style=\"margin-left: 48px;\">\n",
       "        <h3 style=\"margin-bottom: 0px;\">Client</h3>\n",
       "        <p style=\"color: #9D9D9D; margin-bottom: 0px;\">Client-c4f94dc6-1994-11ed-a439-3cecef0c464c</p>\n",
       "        <table style=\"width: 100%; text-align: left;\">\n",
       "\n",
       "        <tr>\n",
       "        \n",
       "            <td style=\"text-align: left;\"><strong>Connection method:</strong> Cluster object</td>\n",
       "            <td style=\"text-align: left;\"><strong>Cluster type:</strong> dask_jobqueue.PBSCluster</td>\n",
       "        \n",
       "        </tr>\n",
       "\n",
       "        \n",
       "            <tr>\n",
       "                <td style=\"text-align: left;\">\n",
       "                    <strong>Dashboard: </strong> <a href=\"https://jupyterhub.hpc.ucar.edu/stable/user/vanderwb/proxy/8787/status\" target=\"_blank\">https://jupyterhub.hpc.ucar.edu/stable/user/vanderwb/proxy/8787/status</a>\n",
       "                </td>\n",
       "                <td style=\"text-align: left;\"></td>\n",
       "            </tr>\n",
       "        \n",
       "\n",
       "        </table>\n",
       "\n",
       "        \n",
       "            <details>\n",
       "            <summary style=\"margin-bottom: 20px;\"><h3 style=\"display: inline;\">Cluster Info</h3></summary>\n",
       "            <div class=\"jp-RenderedHTMLCommon jp-RenderedHTML jp-mod-trusted jp-OutputArea-output\">\n",
       "    <div style=\"width: 24px; height: 24px; background-color: #e1e1e1; border: 3px solid #9D9D9D; border-radius: 5px; position: absolute;\">\n",
       "    </div>\n",
       "    <div style=\"margin-left: 48px;\">\n",
       "        <h3 style=\"margin-bottom: 0px; margin-top: 0px;\">PBSCluster</h3>\n",
       "        <p style=\"color: #9D9D9D; margin-bottom: 0px;\">daf52777</p>\n",
       "        <table style=\"width: 100%; text-align: left;\">\n",
       "            <tr>\n",
       "                <td style=\"text-align: left;\">\n",
       "                    <strong>Dashboard:</strong> <a href=\"https://jupyterhub.hpc.ucar.edu/stable/user/vanderwb/proxy/8787/status\" target=\"_blank\">https://jupyterhub.hpc.ucar.edu/stable/user/vanderwb/proxy/8787/status</a>\n",
       "                </td>\n",
       "                <td style=\"text-align: left;\">\n",
       "                    <strong>Workers:</strong> 0\n",
       "                </td>\n",
       "            </tr>\n",
       "            <tr>\n",
       "                <td style=\"text-align: left;\">\n",
       "                    <strong>Total threads:</strong> 0\n",
       "                </td>\n",
       "                <td style=\"text-align: left;\">\n",
       "                    <strong>Total memory:</strong> 0 B\n",
       "                </td>\n",
       "            </tr>\n",
       "            \n",
       "        </table>\n",
       "\n",
       "        <details>\n",
       "            <summary style=\"margin-bottom: 20px;\">\n",
       "                <h3 style=\"display: inline;\">Scheduler Info</h3>\n",
       "            </summary>\n",
       "\n",
       "            <div style=\"\">\n",
       "    <div>\n",
       "        <div style=\"width: 24px; height: 24px; background-color: #FFF7E5; border: 3px solid #FF6132; border-radius: 5px; position: absolute;\"> </div>\n",
       "        <div style=\"margin-left: 48px;\">\n",
       "            <h3 style=\"margin-bottom: 0px;\">Scheduler</h3>\n",
       "            <p style=\"color: #9D9D9D; margin-bottom: 0px;\">Scheduler-68abd810-d540-4fe5-9ac4-b49c94089f28</p>\n",
       "            <table style=\"width: 100%; text-align: left;\">\n",
       "                <tr>\n",
       "                    <td style=\"text-align: left;\">\n",
       "                        <strong>Comm:</strong> tcp://10.12.205.44:34094\n",
       "                    </td>\n",
       "                    <td style=\"text-align: left;\">\n",
       "                        <strong>Workers:</strong> 0\n",
       "                    </td>\n",
       "                </tr>\n",
       "                <tr>\n",
       "                    <td style=\"text-align: left;\">\n",
       "                        <strong>Dashboard:</strong> <a href=\"https://jupyterhub.hpc.ucar.edu/stable/user/vanderwb/proxy/8787/status\" target=\"_blank\">https://jupyterhub.hpc.ucar.edu/stable/user/vanderwb/proxy/8787/status</a>\n",
       "                    </td>\n",
       "                    <td style=\"text-align: left;\">\n",
       "                        <strong>Total threads:</strong> 0\n",
       "                    </td>\n",
       "                </tr>\n",
       "                <tr>\n",
       "                    <td style=\"text-align: left;\">\n",
       "                        <strong>Started:</strong> Just now\n",
       "                    </td>\n",
       "                    <td style=\"text-align: left;\">\n",
       "                        <strong>Total memory:</strong> 0 B\n",
       "                    </td>\n",
       "                </tr>\n",
       "            </table>\n",
       "        </div>\n",
       "    </div>\n",
       "\n",
       "    <details style=\"margin-left: 48px;\">\n",
       "        <summary style=\"margin-bottom: 20px;\">\n",
       "            <h3 style=\"display: inline;\">Workers</h3>\n",
       "        </summary>\n",
       "\n",
       "        \n",
       "\n",
       "    </details>\n",
       "</div>\n",
       "\n",
       "        </details>\n",
       "    </div>\n",
       "</div>\n",
       "            </details>\n",
       "        \n",
       "\n",
       "    </div>\n",
       "</div>"
      ],
      "text/plain": [
       "<Client: 'tcp://10.12.205.44:34094' processes=0 threads=0, memory=0 B>"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "client = Client(cluster, timeout = '60s')\n",
    "client"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "id": "034479b7-62c4-4918-b56c-67c06930be4d",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "cluster.scale(2)\n",
    "client.wait_for_workers(2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "id": "cad8b3be-270e-4a64-91b2-72b56100ed32",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "                                                            Req'd  Req'd   Elap\n",
      "Job ID          Username Queue    Jobname    SessID NDS TSK Memory Time  S Time\n",
      "--------------- -------- -------- ---------- ------ --- --- ------ ----- - -----\n",
      "3810280.casper* vanderwb R3786268 STDIN        8809   1   2  100gb 01:30 R 00:54\n",
      "3811912.casper* vanderwb asdgpu   dask-work* 283440   1   1   35gb 00:10 R 00:00\n",
      "3811913.casper* vanderwb asdgpu   dask-work* 219531   1   1   35gb 00:10 R 00:00\n",
      "\n",
      "Cached at: Thu Aug 11 10:44:26 MDT 2022\n"
     ]
    }
   ],
   "source": [
    "!qstat -u $USER"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "id": "9e248ee5-5e54-4257-b8cb-cb0d88f7027a",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 2.98 s, sys: 8.63 s, total: 11.6 s\n",
      "Wall time: 23.2 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "dg_means = tg.compute()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "id": "ae51aba0-d3eb-4452-8bf2-8c1e759c76fb",
   "metadata": {},
   "outputs": [],
   "source": [
    "client.shutdown()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "id": "5fa46167-f34a-45b7-a6bb-74ef225bdca2",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "                                                            Req'd  Req'd   Elap\n",
      "Job ID          Username Queue    Jobname    SessID NDS TSK Memory Time  S Time\n",
      "--------------- -------- -------- ---------- ------ --- --- ------ ----- - -----\n",
      "3810280.casper* vanderwb R3786268 STDIN        8809   1   2  100gb 01:30 R 00:54\n",
      "\n",
      "Cached at: Thu Aug 11 10:46:06 MDT 2022\n"
     ]
    }
   ],
   "source": [
    "!qstat -u $USER"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aea9da49-4bd7-4334-ba8f-34bd062226aa",
   "metadata": {},
   "source": [
    "#### dask-jobqueue and dask-cuda compatibility\n",
    "You cannot launch CUDAWorkers via the *dask-jobqueue* Python API at this time, and so the interface for GPU memory spill is not accessible using this method. It may be possible to start the workers via [dask-cuda-worker](https://docs.rapids.ai/api/dask-cuda/stable/api.html#worker) from the shell on each host, and then connect your Client to the IP of a [command-line scheduler](https://docs.dask.org/en/stable/deploying-cli.html)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ff535989-2d76-4147-b8a4-3b7e533d72f7",
   "metadata": {
    "jp-MarkdownHeadingCollapsed": true,
    "tags": []
   },
   "source": [
    "### Some caveats to using Dask (especially with GPUs)\n",
    "\n",
    "Even in CPU computing, use of Dask should be carefully considered. While typically minimal, starting and managing Dask workers does incur overhead that can become large when using very high worker counts. It is also more challenging to debug Dask workflows compared to serial ones!\n",
    "\n",
    "**On GPUs:**\n",
    "* Setting up Dask is more difficult\n",
    "* Typical task sizes may starve the GPU for work\n",
    "* Worker memory management is even more important\n",
    "\n",
    "In some ways, Dask and GPU-offload typically solve orthogonal problems:\n",
    "\n",
    "| CuPy, cuDF, ... | Dask |\n",
    "|-----------------|------|\n",
    "| Lower time to solution | Bigger problem size |\n",
    "\n",
    "Of course Dask can be used to lower the time to solution as well, but often you are better off optimizing your serial code instead. (*and using a GPU may be a great choice if your problem is highly vectorizable!*)\n",
    "\n",
    "A big issue is that limited GPU availability (or higher cost) on many platforms may hamper Dask's power of scalability. On *Casper*, wait times for V100 GPUs are typically longer than for CPU resources. And GPUs feature much lower memory than many data-analysis CPU nodes."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "78babce5-c0bf-43b5-9b7b-7a79bdec591e",
   "metadata": {},
   "source": [
    "## **cuDF**: Series and DataFrames on GPUs\n",
    "\n",
    "A *DataFrame* object is simply an annotated 2D table of columnar data. In Python, the `pandas` library is the go-to package for working with these object representations of spreadsheets, and their simpler single-column variant, the *series*.\n",
    "\n",
    "#### Let's download some data\n",
    "\n",
    "The following dataset contains daily records of Boulder, CO weather from 1897 to present-day."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "id": "a186b3f4-087f-4912-8040-dd975f3c0d73",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current\n",
      "                                 Dload  Upload   Total   Spent    Left  Speed\n",
      "100 2199k  100 2199k    0     0  7487k      0 --:--:-- --:--:-- --:--:-- 7508k\n"
     ]
    }
   ],
   "source": [
    "!curl https://psl.noaa.gov/boulder/data/boulderdaily.complete.txt | grep \"^ [12]\" > data/daily.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "49635d3e-ff41-47c7-9b48-3a87b525a7b0",
   "metadata": {},
   "source": [
    "#### We can quickly read in and inspect the data on the CPU"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "id": "7d258191-9a27-4bec-ab24-a56acfca98a2",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "id": "36375ec1-9ad2-49c2-8d93-67dc2a6679bd",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 27.7 ms, sys: 7.26 ms, total: 35 ms\n",
      "Wall time: 34.2 ms\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "col_names = ['year', 'month', 'day', 'tmax', 'tmin', 'precip', 'snow', 'snowcover']\n",
    "cpu_df = pd.read_csv('data/daily.txt', delim_whitespace=True, names = col_names,\n",
    "                     na_values = [-998, -999])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "id": "658c44ca-bf68-4e62-a221-d5db77c2300b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>year</th>\n",
       "      <th>month</th>\n",
       "      <th>day</th>\n",
       "      <th>tmax</th>\n",
       "      <th>tmin</th>\n",
       "      <th>precip</th>\n",
       "      <th>snow</th>\n",
       "      <th>snowcover</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>45958</th>\n",
       "      <td>2022</td>\n",
       "      <td>7</td>\n",
       "      <td>27</td>\n",
       "      <td>86.0</td>\n",
       "      <td>58.0</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>45959</th>\n",
       "      <td>2022</td>\n",
       "      <td>7</td>\n",
       "      <td>28</td>\n",
       "      <td>82.0</td>\n",
       "      <td>60.0</td>\n",
       "      <td>1.11</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>45960</th>\n",
       "      <td>2022</td>\n",
       "      <td>7</td>\n",
       "      <td>29</td>\n",
       "      <td>81.0</td>\n",
       "      <td>58.0</td>\n",
       "      <td>0.03</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>45961</th>\n",
       "      <td>2022</td>\n",
       "      <td>7</td>\n",
       "      <td>30</td>\n",
       "      <td>91.0</td>\n",
       "      <td>51.0</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>45962</th>\n",
       "      <td>2022</td>\n",
       "      <td>7</td>\n",
       "      <td>31</td>\n",
       "      <td>90.0</td>\n",
       "      <td>61.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       year  month  day  tmax  tmin  precip  snow  snowcover\n",
       "45958  2022      7   27  86.0  58.0    0.00   0.0        0.0\n",
       "45959  2022      7   28  82.0  60.0    1.11   0.0        0.0\n",
       "45960  2022      7   29  81.0  58.0    0.03   0.0        0.0\n",
       "45961  2022      7   30  91.0  51.0    0.00   0.0        0.0\n",
       "45962  2022      7   31  90.0  61.0     NaN   0.0        0.0"
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cpu_df.tail()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "id": "ef3d2227-0bff-4819-8a2d-50249616d534",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Mem used: 2.81 MiB\n"
     ]
    }
   ],
   "source": [
    "print(f'Mem used: {cpu_df.memory_usage(deep = True).sum() / 1024**2:.2f} MiB')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "80eb61e0-cbb1-4592-b7a7-32929b9c80d7",
   "metadata": {},
   "source": [
    "*Note: `memory_usage` returns a pandas series here. We take the sum to get the amount used over all columns and specify `deep = True` to account for mutable data referenced within the DataFrame.*\n",
    "\n",
    "#### Average snowfall in each month\n",
    "\n",
    "**Pandas** can succinctly and easily compute complex statistics on DataFrames and then plot them with **matplotlib**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "id": "9d2e4179-648f-439d-8409-a02eea831ea3",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 10.6 ms, sys: 563 µs, total: 11.2 ms\n",
      "Wall time: 12.9 ms\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "avg_snow = cpu_df[cpu_df.year > 1897].groupby(['year', 'month']).sum().groupby('month').snow.mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "id": "14e23696-10c4-4850-8529-10fa8e8d2ccb",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<AxesSubplot:xlabel='month', ylabel='Average Snowfall [in]'>"
      ]
     },
     "execution_count": 57,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEJCAYAAAB7UTvrAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8qNh9FAAAACXBIWXMAAAsTAAALEwEAmpwYAAAlbUlEQVR4nO3deXyc9XXv8c+ZGWFDwE5aDCEhKVkopKVJyHUTEihJIGlyydamvWnakHJLbp1mg9AsQEqz3bQlmFAIJBQXCATMZgNGYTHYxsYYsEGy5X2RkBdtlmTLlmTJGmlmTv+YkRBGy0iaeZ5Zvu/Xa16jGc3Mc8aIOfP8lnPM3REREQGIhB2AiIgUDiUFEREZoqQgIiJDlBRERGSIkoKIiAxRUhARkSF5SwpmdruZtZnZpmH3/Z6ZLTGz2sz1G/J1fBERmbh8nincAXzyiPuuAJa5+6nAssxtEREpEJbPzWtmdgrwqLufkbm9HfiIu7eY2UnACnc/bbzXOf744/2UU07JW5wiIqWourp6n7vPmshzYvkKZhQnunsLQCYxnDDaA81sDjAH4K1vfStVVVUBhSgiUhrMbPdEn1OwE83uPs/dZ7v77FmzJpToRERkkoJOCq2ZYSMy120BH19ERMYQdFKoBC7K/HwR8EjAxxcRkTHkc0nqvcALwGlm1mhmXwGuBj5uZrXAxzO3RUSkQORtotnd/3aUX52fr2OKiMjUFOxEs4iIBE9JQUREhigpFCB1wxORsCgpFJjO3gH+/D9X8qNHNo3/YBGRHFNSKDD/+sgmatsOcecLu3lpZ0fY4YhImVFSKCCP1DRRub6ZaOa/yjfvXRtuQCJSdpQUCkTTwcNctWgTFREjmUrf19oV5541Ey5dIiIyaUoKBSCZcv75/hoO9ycxe/XvfvroFhKDWUJEJM+UFArArc/Ws2ZnBxj0J1+98qhvIMXPHtsaUmQiUm6UFEK2ubmTuU9uJxoxEsmRl6L+9oVdHOztDzgyESlHSgoh6htI8u37atJDRmPsTUg5fOvedcEFJiJlS0khRFc/sY3atkMYxignCUOerd3H9r1dwQQmImVLSSEkK3e0c8fzu4hGoD/LieSv3a0lqiKSX0oKITjQ0893F6ynImqkJrCwqH5fD4s3teQvMBEpe0oKAXN3rnxoI/sOxYkYTLTK0eULN6g2kojkjZJCwBZWN7J4814iZsQTE/9w7+xLcOOyujxEJiKipBCoPft7+VHlZmIRI5Ga/Lf9Xz5dS19/MoeRiYikKSkEJJFMcdkDNfQnUkRs/MeP+Vop5/sL1+cmMBGRYZQUAnLzipep3n0AG2HX8mRUbmih+WBvDiITEXmFkkIA1jcc5PpltUQjxkAOEsKgf9ISVRHJMSWFPOvtT/Dt+2vSQ0Y5XjW0obGTNfX7c/qaIlLelBTy7N8e28rOfT1Z7VqejEtU/kJEckhJIY+WbW1l/po9RC37XcsT1dod5+7V6rkgIrmhpJAn7d1xvrdwAxXR/JwhDPezx9RzQURyQ0khD9ydKx7cwMHe/tc0zcmHvoEUP310S/4PJCIlT0khD+55cQ/LtrURMaN/EruWJ+Pu1bs50KOeCyIyNUoKOVbffoj//+iWKe9anqh0zwUtURWRqVFSyKGBZIrL7q8hmfJAho2OtKpuP9ta1HNBRCZPSSGHblxWy/rGTozcblKbiK/Nrw7luCJSGpQUcqR6dwc3La8jGrG8LT/Nxs59vTyxUT0XRGRylBRy4FA8wbfvqyEasYLodXD5g+q5ICKTo6SQAz+p3EzjgcMYRoBzy6Pq6ktww7LasMMQkSKkpDBFize1sKC6kUgedy1Pxk1P19Hbnwg7DBEpMkoKU9Da1cflD24MZNfyRCVSzvcXqOeCiExMKEnBzC4zs81mtsnM7jWz6WHEMRWplPPdBes5FE+Esvw0G49u3Etjh3ouiEj2Ak8KZvZm4BJgtrufAUSBLwYdx1T99oVdPFu7D4PAdi1Pxtfma0ObiGQvrOGjGHC0mcWAY4DmkOKYlB2t3fzHE9sC37U8GRubOnmhfl/YYYhIkQg8Kbh7E3AtsAdoATrd/amg45iseCLJt++rIeUOFHZCGHTpvTVhhyAiRSKM4aM3AJ8D3ga8CXidmV04wuPmmFmVmVW1t7cHHeaorluygy0tXRhGonAWG42prTvOXS/sCjsMESkCYQwffQzY6e7t7j4APAR86MgHufs8d5/t7rNnzZoVeJAjWV2/n3nP1Ie+a3ky/u2xreq5ICLjCiMp7AHOMrNjzMyA84GtIcQxIZ2HB/jnB2qIRY1Ugc8jjKQvkeInv1PPBREZWxhzCmuAhcBaYGMmhnlBxzFRP3pkEy2dfUSwIplJeK35a3bToZ4LIjKGUFYfufuP3P10dz/D3b/s7vEw4sjWIzVNLKppJmpGvIiHYNRzQUTGox3N42g6eJirFm2iIlr4y0+z8VzdfrY2q+eCiIxMSWEMqZTznQdq6O1PUqCblidFPRdEZDRKCmO4dVU9q+s7MsXuiv8sYdCu/b08uqGo9guKSECUFEaxpn4/c5/cTiwSXhe1fLryoY3quSAir6GkMIJ71uzhS7euSd8o0Q/O7r4E1y9VzwUReTUlhWH6EymuWrSRHzy8EUjPKRRwrbsp+9Vy9VwQkVdTUsjYdyjOhbet4e7Ve4hG0v0ISnDU6FUSmfLfIiKDlBSATU2dfObGVVTvOpBumFO8WxEm7PGNe2lQzwURyYiN9gsz25DF89vd/fwcxhO4yvXNfH/BepLuxKIQL+XxolF87e5qHr3kz8IOQ0QKwKhJgXTzmwvG+L0BlbkNJzjJlHPtU9u5ecXLVEQNdyeeDDuqcGxq7uL5un186J3Hhx2KiIRsrKTwVXffPdaTzezrOY4nEF19A1x67zqWb28nWqJLTifq0vtreOlfPhZ2GCISslHnFNx91XhPzuYxhebl9kP8xU3P8cyOdmIRI1kCpStyob07zp3P7wo7DBEJ2bgTzWZ2tpktMbMdZlZvZjvNrD6I4HJt+bY2PnfTczQc6CUWiZRELaNc+vfH1XNBpNxls/roNuA64BzgT4HZmeui4e78ekUdF9/xEvFEuo5RsTXJCUI8keJHlZvDDkNEQpRNUuh09yfcvc3d9w9e8h5ZjhzuT/Kte9dxzeLtRCNGIuklVcco1+55cQ898YGwwxApGcVWTmasieZBy81sLum2mUN9D9y94AvzNx7o5at3VbO5uYtYpDRKX+ebO8xfvYc5H35H2KGIFL3K9c1894H1HDstxqwZ03jTzOmcOGM6Jxw3jVkzpnPicdM4IXP7+GOncVQs/K1j2SSFD2SuZw+7z4Hzch9O7qyp388/3V1Nd1+Co6IRDRdNwJKtrUoKIjnwm1U7SbrT1TfAwcP9bN/bjcGo3RtnTI8x67hpnDTz6HTymDGNE46bxgnHTefEGenrE2ZMY3pFNG8xj5sU3P2jeTt6nty1ejc/rtxM1CxT9loJYSK2qAmPyJTVtXWzruHga0YpRkoIsYgRMePwQJKd+3p4ub1nzORxzFFRjj92GifNnM4bZ04fShwnzJjGrGFJZDLG2tF8obvfbWb/PNLv3f26SR0xj/ozE6X3vrgns9w0VfL1i/Khpz/Job4Bjp1eEXYoIkVrQXUjBqSymFNIJ41XP26kZ0UjRtRgIJmi6UAve8YpUWOxoyacGcY6U3hd5vq4ib5oGNq743zt7mqqdh8YKmgnk7dkSyt/+b6Tww5DpCglkikeqm7CLN0bPVeSKSebwgtRAzMjNvONb5/oMUZNCu5+S+b6JxN90aBtbOzkH++qor07TkVUO5Rz4fGNLUoKIpO0srad9kNxpsUixBPBD18nnfSqEZt4J+FRp7rNbM54T87mMfn2SE0Tf/1fz9NxKF6yXdLCUL3nQNghiBStBVWNxCIWSkKYqrGGj64ws31j/N6AS4F5uQ0pO8mUc83ibdyysp6KiOE4CSWEnOnoGSCRTBGLhr9ETqSYdPT0s2RLa1ZzCYVorKTwDPCZcZ6/JIexZK2zd4BL7lvHMzsyBe00f5AXL9Tv589OnRV2GCJFZdG6JhIppyJqpIrwi+pYcwr/EGQg2apr6+b/3VnFno5eYlHT2UEeVa5vVlIQmaAF1Q1FPZSdzea1grF0SyuX3r+O/kSKWEQb0vLt+bqiqWYiUhA2NXWytaWbiuiE53cLRtEkhV8tr+PaJ7cTi5oK2gWkpfMw7o5Z8f6BiwRpYXUjEaOoRzCKYhZxT0cvc59UQbugpRy27e0OOwyRohBPJHl4XRMw+k7kYjDWjuYRdzIPCnJHc+fhAWaooF0oFq1r4l0nzQg7DJGCt2xrG52HB0Lbm5ArY50pHDfOJVBKCOFYsaMt7BBEisKCqoai3Zsw3FirjwpmJ7NGtMOzs70n7BBECl5rVx/P7Ggvic+qsYaPfjnWE939ktyHI4WmP+ns7TrMG2ccHXYoIgXrobVNpBwqIhR9Ec6xVh9VBxaFFLTKmmbmnKv+CiIjcXcWVDWUzEbasYaP7gwyEClcS7a0KimIjGLtnoPU7+vhqJhRCivlx92nYGazgMuBPwKmD97v7gXdeU1yR013REa3sLoh3cwrUfxnCZDdPoX5wFbgbcBPgF3AS1M5qJm93swWmtk2M9tqZh+cyutJfvX0J+mJD4QdhkjBOdyfpLKmOewwciqbpPD77n4bMODuz7j7xcBZUzzuDcBidz8deA/ppCMF7KnNrWGHIFJwFm9uoac/SSxSFPuAs5LNOxn8ithiZp8yszOBSXdfMbMZwLnAbQDu3u/uByf7ehKMxze2hB2CSMEZ7JtQSmV3sql99DMzmwl8B7gRmAFcNoVjvh1oB35jZu8hvcrpUnd/1YL4TAOfOQCxGarUGTY13RF5tYaOXp5/eT+xSCnsTnjFWJ3Xfp758Wh373T3Te7+UXf/X+5eOYVjxoD3ATe7+5lAD3DFkQ9y93nuPtvdZ0ePmTmFw0kuDDbdEZG0B9c2AuklqaVkrOGjC8ysArgyx8dsBBrdfU3m9kLSSUIK3Or6jrBDECkIqZSzsLqRaMSKfrPakcZKCouBfcC7zaxr2KXbzCa9RtHd9wINZnZa5q7zgS2TfT0JTuX60lplITJZq3fup/HAYWKlM788ZNS35O7fc/eZwGPuPmPY5Th3n2rZzG8B881sA/Be4N+n+HoSgOfqxmrZLVI+FlalzxLiJbI3YbhsJporzeyd7l6Xq4O6ew0wO1evJ8FQ0x0R6O4b4PGNLSU3lzAom5OfPwDmmVm9mT1gZt8ys/fmOS4pQCmH7Wq6I2XusQ0t9CVSVJTYqqNB4yYFd/9hpqTFHwOrgO+hYnlla1Gms5RIuVpQnd6bEC+1GeaMcZOCmV1lZk8ATwHvBL7LFDavSXFbrqY7UsZebj9E9e7S3rOTzZzC54EE8BjwDLDa3fvyGpUULDXdkXK2sLoRA5IlUCJ7NNkMH72P9LLRF4GPAxvNbFW+A5PCNNh0R6TcJFPOg9WNmEHppoTsho/OAC4ELgL+hvTms6fzHJcUsMoa1UGS8rOytp227jgV0RLcnDBMNu/u56TrHf0SeFem1MUP8xuWFLIlW/aGHYJI4BZmit/FE6Vd7mXcOQV3/5SZHQX8IXCamW13dxXXL2NquiPl5kBPP09t2UuqRPcmDJdN57UPA78l3VzHgLeY2UXuvjLPsUmBGmy687ppFWGHIhKIR2qaGEg6FVEjVaJLUQdlM3x0HfDn7v5hdz8X+ATwn/kNSwqdmu5IORncmzBQ4gkBsksKFe6+ffCGu+8A9BWxzD2+SZPNUh62NHexubmLcqnuks0+hSozuw24K3P7S2hHc9kr9Q08IoMWVDcQMUiUwVkCZJcUvgZ8A7iE9JzCSuDX+QxKCt9g051YiS/Pk/LWn0gNlXYpj5SQ3eqjOOl5hevyH44Uk9X1HZxz6vFhhyGSN09va+VA7wDTYpGSX4o6KJvNa2eb2RIz25GplFpvZvVBBCeF7XdquiMlbkGZ7E0YLpvho9uAy0jPIyTzG44Uk1VquiMlrK27jxXb2ymfgaO0bJJCp7s/kfdIpOio6Y6UsofXNpF0pyJiJMtg09qgbGYJl5vZXDP7oJm9b/CS98ik4KnpjpQqdx8aOhoo4YqoI8nmTOEDmevh7TMdOC/34UixWbSuiStOmmrLbpHCUtNwkLr2QxwVMyif6QQgu9VHHw0iEClOK3a0c8UF7wo7DJGcWlDdSMSgP1FeZwkwxvCRmX3GzP5g2O0fmtl6M6s0s1MCiU4KXn37obBDEMmpvoEkv6sp35V1Y80p/BvQDmBmnybdU+FioBK4Jf+hSTHoTzqtXWrEJ6Xjyc176Y4nSr5vwmjGetfu7r2Znz8P3Obu1e5+KzAr/6FJsXikjL9VSekpx70Jw42VFMzMjjWzCOl2nMuG/W56fsOSYrJUTXekRDQdPMxzZb7/ZqyJ5uuBGqAL2OruVQBmdiagEpkyZLOa7kiJeLC6Mb1VrYz2JRxp1KTg7reb2ZPACcD6Yb/aC/xDvgOT4qGmO1IKUilnQXUD0YiRKLO9CcONOZPi7k3uvs7dU8Pua3H3PfkPTYqJmu5IsXtxVwcNHYeJlef88pAyf/uSK2q6I8VuQVUj0YgRL8O9CcMpKUhOqOmOFLND8QSPbWzGy3guYVBWScHMzjGzf8j8PMvM3pbfsKTYDDbdESlGj29ooW8gRSyi4o7Z9FP4EXA5cGXmrgrg7nwGJcVpdX1H2CGITMqC6gZiEaO/TFpujiWbM4W/BD4L9AC4ezNwXD6DkuKkpjtSjHbu6+GlXRr+HJRNUuj39ECbA5jZ6/IbkhSr514u700/UpwWVjdgpJekSnZJ4QEzuwV4vZn9I7AU+O/8hiXFqPngYU3USVFJppyF1Y1Y+VXIHtW4ScHdrwUWAg8CpwE/dPcbp3pgM4ua2Toze3SqryWFQU13pNisqttHa1e8bIvfjSSbJju4+xJgSY6PfSmwFVCHlhKipjtSTBZUNZR18buRZLP6qNvMuo64NJjZw2b29skc1MxOBj4F3DqZ50vhWrGjPewQRLJysLefpza3asjzCNmcKVwHNAP3AAZ8EXgjsB24HfjIJI57PfB9xljFZGZzgDkAsRmq1F0s1HRHikXl+mb6kykqokZSS1GHZDOQ9kl3v8Xdu929y93nARe4+/3AGyZ6wEzDnjZ3rx7rce4+z91nu/vs6DEzJ3oYCYma7kixGBw6GlBCeJVskkLKzL5gZpHM5QvDfjeZf82zgc+a2S7gPuA8M9NmuBKipjtS6Lbt7WJjUxemDcyvkU1S+BLwZaANaM38fKGZHQ18c6IHdPcr3f1kdz+F9FDU0+5+4URfRwqXmu5IoVtQlV6GmtBZwmuMO6fg7vXAZ0b59archiOlYHOLmu5I4RpIpnh4XVN6w1rYwRSgcZOCmU0HvgL8McPacLr7xVM9uLuvAFZM9XWksPTE1XRHCtfT29ro6OlnWiyipagjyGb46C7Sq40+ATwDnAxoh5KMackWNd2RwrSgqlF7E8aQTVJ4p7v/K9Dj7neS3l/wJ/kNS4rd4xvVdEcKT3t3nOXb2rQ3YQzZJIWBzPVBMzsDmAmckreIpCRU7T4Ydggir7FoXRNJdyLqmzCqbDavzTOzNwBXAZXAscC/5jUqKXodPf0kkiliqikjBcLdtTchC2MmBTOLAF3ufgBYCUyqrIWUp9X1HZxz6vFhhyECwIbGTna0HeKoqM4SxjLm1zh3TzGJvQgioKY7UlgWVDcQMdRdbRzZnNsvMbPvmtlbzOz3Bi95j0yKnpruSKHoG0hSqZ32WclmTmFwP8I3ht3naChJxjHYdMdUS0BC9tSWVrr6EtqbkIVsdjS/LYhApPSkHHa0dnPaG9VfQcKlvgnZy6afwjFmdpWZzcvcPjVT6VRkXA+vawo7BClz1bs7eLZWQ5nZymZO4TdAP/ChzO1G4Gd5i0hKyortaroj4XF3/uPxbVREjURKE8zZyCYpvMPdryGzic3dD5NutiMyLjXdkTAt3dpG1e4DkyvyX6aySQr9mTLZDmBm7wDieY1KSkZ/0mlT0x0JQSKZ4prF6bOEAZ0lZC2bpPBjYDHwFjObDywj3UpTJCtquiNheGhtE7Vth7T6bYLGTQru/hTweeD/AvcCszMlr0WyskRNdyRgfQNJfrFkOxURo18rjiYkm34KlaSTQaW79+Q/JCk1arojQbvj+V20dsWZFouk10ZL1rIZPvoF8GfAFjNbYGZ/nWm8I5KVwaY7IkE42NvPr5fXEdW+hEnJZvjoGXf/OukdzPOAL5Du1yySNTXdkaD8esXLdPUliKlA76Rk9c+WWX30V8A/AX8K3JnPoKT0qOmOBKHp4GHueG4XUTPiCQ0bTUY2cwr3Ax8gvQLpV8CKTPVUkayp6Y4E4T+X7CCRShE1Q8VQJyebgni/Af7O3ZMAZna2mf2du39jnOeJDFHTHcm3bXu7eLC6kYihfQlTkM2cwmLgT8zs52a2i3SJi235DkxKz5qdHWGHICVs7uLtRCOmzctTNOqZgpn9IfBF4G+B/cD9gLn7RwOKTUpMZU0zZ79Tndgk99bU72fZtjZiEdMK1Ckaa/hoG/As8Bl3rwMws8sCiUpKkpruSD64O1cPlrPQRMKUjTV89FfAXmC5mf23mZ2PCuHJFAw23RHJpSc372XdnoNhh1EyRk0K7v6wu/8NcDqwArgMONHMbjazPw8oPikhg013RHIlkUzx88XbdZaQQ9lMNPe4+3x3/zRwMlADXJHvwKQ0LVJxPMmhB6oa2bmvR0XvcmhC6wPdvcPdb3H38/IVkJS25du0GV5yo7c/wfVLdxBT0buc0qJxCZSa7kiu/Oa5XbR1x4lGdJaQS0oKEig13ZFc6Ojp5+YVL6voXR4oKUjgKtdrXkGm5qan6+iJJ9AG+dzTP6kEThVTZSoaOnr57Qu7iBj0q+hdzikpSOA2NXeGHYIUseuW7CDljhYc5YeSggROTXdksjY3d7JoXRMGaCohPwJPCmb2FjNbbmZbzWyzmV0adAwSPg0hyWRco6J3eRfGmUIC+I67vws4C/iGmf1RCHFIiJ7YuDfsEKTIPF+3j2d2tANqu5xPgScFd29x97WZn7uBrcCbg45DwvXS7gNhhyBFJJVyrn4iXfQuoYyQV6HOKZjZKcCZwJow45DgDTbdEcnG45ta2NCkBQpBCC0pmNmxwIPAt929a4TfzzGzKjOrSvbqj6EUqemOZGMgmeIaFb0LTChJwcwqSCeE+e7+0EiPcfd57j7b3WdHj5kZbIASiN9pE5tk4b4X97Cno5eI1qAGIozVRwbcBmx19+uCPr4UjlV1arojY+uJJ7h+aS0xlbMITBhnCmcDXwbOM7OazOWCEOKQkKnpjozn1md3sr+nX0XvAjRWO868cPdVqIObkF5WWNvazR++cUbYoUgB2ncozi0rXyZq6CwhQNrRLKF6WE13ZBQ3LqvlcH9SZwkBU1KQUKnpjoxk9/4e7l6zh0gkXW5dgqOkIKFS0x0ZybVP7QA0zhwGJQUJlZruyJE2Nnamlyu7q+hdCJQUJHRquiPD/XzxNmIRQwvTwqGkIKFTxVQZ9Gxt+9D+FZ0khENJQUK3WU13BBW9KxRKChK6Q2q6I8DvNjSzublLk8shU1KQgqAhpPIWTySZ++R2KiKmJaghU1KQgqCmO+XtnjV7aDxwmIg2qoVOSUEKQpWa7pSt7r4BfrlMRe8KhZKCFAQ13Slf/72yngO9A0RVGrsgKClIQXDgRTXdKTtt3X3Me7aeaMSI60tBQVBSkILx4NrGsEOQgN2wtJZ4IkVUJwkFQ0lBCsZDa5u4fdVO9VgoE/Xth7jvxQYiqOhdIVFSkILy00e3cPEdL3G4Pxl2KJJn1z61XRXvCpCSghSMwe+Ky7e3c+7c5dS1dYcaj+TPuj0HeDyzDFknCYVFSUEKUnt3nE9c/ywLqhrCDkVyzP2VchYplbMoOEoKUrBSKed7CzfwrXvXEk9oOKlUrNjRzpqdHeCvnB1K4VBSkII1+IHxu/UtfOwXz9B4oDfUeGTqksOK3g3oLKEgKSlIUWg4cJiPXruCJzerHEYxW7Suie17uzW/XMCUFKRoJJLOV++q5gcPb9Tu5yLU25/gF0u2E4uq6F0hU1KQojH4MXLPmj38718+S1u32ngWg76BJHc8t5OPzF1B88E+lbMocLGwAxCZjNrWQ5x7zXJu/fvZnHPqrLDDkRHEE0keeKmBm5bX0doVJxYxpsUiKnpX4JQUpGjFB1JceNuLfPXDb+fyT5yusssFoj+R4oGqBn61vI6Wzr5XJQN1VCt8SgpStJz0hthbnqln9cv7ufPi9/P6Y44KO6yy1Z9IsbC6kZuW19J8UMmgWCkpSFEb/KhZ39jJOT9fzl1feT9nvvUNocZUbgaSKR5a28iNT9fReOCwkkGRU1KQktETT/D5Xz/Pdz9xGl//yDswTWjmVSKZ4qF1Tdz0dB17OnqVDEqEkoKUjMHhpLlPbue5un3M+/vZHDtNf+K5lkimWFTTzC+X1SoZlCD9HyMlZfAj6fmX93PuNU8z/ytn8a43zQg1plKRTDmV65u4YWktu/b3UhE1jopF6FcyKClKClKyOnoG+NSNz/LTz57BhR/8g7DDKVrJlPPohmZuWFpL/b6eVyUDVS8qPUoKUtLc4apHNvFsbTs3/O2ZTK+Ihh1S0UilnMc2tnDD0lrq2g8pGZQJJQUpaYMfXU9uaeUjc1dw35wPcMrxx4YaU6FLpZwnNu3lhmU72NGqZFBulBSkbOzt6uNj163kF194D59775vDDqfgpFLOk5v3cv2yWrbv7U4ng6E6RUoG5UJJQcpKMuVcel8NK3e0c/VfvZuKqMp/uTtPbWnl+qU72NpyZDKQchNKUjCzTwI3AFHgVne/Oow4pPwMfsw9uLaJ6t0HuHfOWZw08+hQYwqLu7N0axvXL93B5uYuJQMBQkgKZhYFfgV8HGgEXjKzSnffEnQsUt527e/lw3NX8IMLTmfWsdOJRiAaiRCNQMSMWCRCJAJRM2JRI2JGNDLsMux2JPOYqBmRiBGLDLu2Vx5fCPWZ3J2nt6WTwcYmJQN5tTDOFN4P1Ll7PYCZ3Qd8Dhg1KehPVfKlP5Hix5XBfh+JWDrpRMwIo9uMuzMwLAEMKBnIMGEkhTcDw7uxNwIfOPJBZjYHmANgsaNSzbd/cyCY8F6ROtwdjRx9XFk1By7L9xzvmRY9ZmbYYQQq2dtJ9JiZeDLRT5l87yrHv+2BA80T/owPIymM9N3oNX+U7j4PmJf/cEZnZlWJ7n2zw4whaGX7njvb9J5LXLn+bU/0OWEsvWgE3jLs9slAcwhxiIjIEcJICi8Bp5rZ28zsKOCLQGUIcYiIyBECHz5y94SZfRN4kvSS1NvdfXPQcWQp1OGrkOg9lwe95/Iw4fds7mUxxyQiIlnQdk4RERmipCAiIkOUFEZgZm8xs+VmttXMNpvZpWHHFBQzi5rZOjN7NOxYgmBmrzezhWa2LfPf+4Nhx5RPZnZZ5m96k5nda2bTw44pH8zsdjNrM7NNw+77PTNbYma1meuSaeY9yvudm/m73mBmD5vZ67N5LSWFkSWA77j7u4CzgG+Y2R+FHFNQLgW2hh1EgG4AFrv76cB7KOH3bmZvBi4BZrv7GaQXenwx3Kjy5g7gk0fcdwWwzN1PBZZlbpeKO3jt+10CnOHu7wZ2AFdm80JKCiNw9xZ3X5v5uZv0B0XJ11o2s5OBTwG3hh1LEMxsBnAucBuAu/e7+8FQg8q/GHC0mcWAYyjRPULuvhLoOOLuzwF3Zn6+E/iLIGPKp5Her7s/5e6JzM3VpPeEjUtJYRxmdgpwJrAm5FCCcD3wfSAVchxBeTvQDvwmM2R2q5m9Luyg8sXdm4BrgT1AC9Dp7k+FG1WgTnT3Fkh/8QNOCDmeIF0MPJHNA5UUxmBmxwIPAt92966w48knM/s00Obu1WHHEqAY8D7gZnc/E+ihtIYUXiUzhv454G3Am4DXmdmF4UYl+WZm/0J6SHx+No9XUhiFmVWQTgjz3f2hsOMJwNnAZ81sF3AfcJ6Z3R1uSHnXCDS6++BZ4ELSSaJUfQzY6e7t7j4APAR8KOSYgtRqZicBZK7bQo4n78zsIuDTwJc8y01pSgojMDMjPc681d2vCzueILj7le5+srufQnry8Wl3L+lvke6+F2gws9Myd53PGCXcS8Ae4CwzOybzN34+JTyxPoJK4KLMzxcBj4QYS95lmpldDnzW3XuzfZ6SwsjOBr5M+ttyTeZyQdhBSV58C5hvZhuA9wL/Hm44+ZM5I1oIrAU2kv7/vyRLP5jZvcALwGlm1mhmXwGuBj5uZrWkm3yVTMfHUd7vTcBxwJLMZ9h/ZfVaKnMhIiKDdKYgIiJDlBRERGSIkoKIiAxRUhARkSFKCiIiMkRJQSQPMtVXvz7s9kfKpfKsFDclBZH8eD3w9fEeJFJolBSk7JnZKZm687dm+gzMN7OPmdlzmdr778/U4l+UqU2/2szenXnujzO17FeYWb2ZXZJ52auBd2Q2Dc3N3HfssN4N8zO7ikUKSizsAEQKxDuB/wPMAV4C/g44B/gs8AOgAVjn7n9hZucBvyW9AxrgdOCjpHePbjezm0kX1jvD3d8L6eEj0tV2/5h0uernSO+cX5X3dyYyATpTEEnb6e4b3T0FbCbdjMVJl4M4hXSCuAvA3Z8Gft/MZmae+5i7x919H+kiayeOcowX3b0xc4yazOuKFBQlBZG0+LCfU8Nup0ifUY801DNYI2b4c5OMfgae7eNEQqOkIJKdlcCXYGgoaN84PTa6SQ8niRQVfVMRyc6PSXdo2wD08koJ5hG5+/7MRPUm0h2vHst/iCJTpyqpIiIyRMNHIiIyRElBRESGKCmIiMgQJQURERmipCAiIkOUFEREZIiSgoiIDPkfv6g6h7bch6UAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "avg_snow.plot.area(ylabel = \"Average Snowfall [in]\", xlim = (1,12))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7b5e47fb-43b1-4a7c-a1cf-f20d16721c86",
   "metadata": {},
   "source": [
    "#### Replacing **pandas** with **cuDF** for GPU use\n",
    "\n",
    "For *series* and *DataFrames*, **cuDF** allows us to leverage GPUs for computation. Note that unlike **CuPy**, the **cuDF** package is part of RAPIDS and developed by NVIDIA. Thus it will NOT work on other hardware at this time.\n",
    "\n",
    "Many **pandas** methods and functions and have been replicated with **cuDF**, allowing for near drop-in replacement."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "id": "b8d5edab-1ba4-4442-94cf-54173e58c773",
   "metadata": {},
   "outputs": [],
   "source": [
    "import cudf as cd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "id": "89e76aa8-b1fc-4b99-8e41-c94567d1f810",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 41.4 ms, sys: 13.9 ms, total: 55.3 ms\n",
      "Wall time: 610 ms\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "gpu_df = cd.read_csv('data/daily.txt', delim_whitespace = True,\n",
    "                     names = ['x'] + col_names).drop(columns = 'x').replace([-998, -999], cd.NA)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bedf05b2-b201-4c80-8072-e32d483c5b49",
   "metadata": {},
   "source": [
    "*This doesn't look the same, however!*\n",
    "\n",
    "It turns out we need to make a few modifications for cuDF to read the data in:\n",
    "1. cuDF does not ignore a leading space on each row, so we handle this by assigning the column a dummy name and then dropping it after it is read.\n",
    "2. cuDF will not take integers for the na values, and so we use `.replace` after read-in to get the same effect."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "id": "d625a68a-d8fd-42f8-b252-a795e672234f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>year</th>\n",
       "      <th>month</th>\n",
       "      <th>day</th>\n",
       "      <th>tmax</th>\n",
       "      <th>tmin</th>\n",
       "      <th>precip</th>\n",
       "      <th>snow</th>\n",
       "      <th>snowcover</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1897</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1897</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1897</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1897</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1897</td>\n",
       "      <td>1</td>\n",
       "      <td>5</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "      <td>&lt;NA&gt;</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   year  month  day  tmax  tmin precip  snow snowcover\n",
       "0  1897      1    1  <NA>  <NA>   <NA>  <NA>      <NA>\n",
       "1  1897      1    2  <NA>  <NA>   <NA>  <NA>      <NA>\n",
       "2  1897      1    3  <NA>  <NA>   <NA>  <NA>      <NA>\n",
       "3  1897      1    4  <NA>  <NA>   <NA>  <NA>      <NA>\n",
       "4  1897      1    5  <NA>  <NA>   <NA>  <NA>      <NA>"
      ]
     },
     "execution_count": 60,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gpu_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "id": "ea07ffa5-e777-418b-b8e3-aa62dadd68e8",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Mem used: 2.83 MiB\n"
     ]
    }
   ],
   "source": [
    "print(f'Mem used: {gpu_df.memory_usage(deep = True).sum() / 1024**2:.2f} MiB')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "de81f593-9c7a-476f-9021-7454aac93d0e",
   "metadata": {},
   "source": [
    "Here again we must make a small modification to our CPU code - cuDF's groupby operation will not operate on a *column-index* unless it is explicitly told to do so with the `level` parameter. pandas will infer that we mean the index by names, but cuDF better adheres to pandas' own specification!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "id": "aca9b613-4e43-4a32-9f7f-f7d2ff288e1c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 11.5 ms, sys: 2.54 ms, total: 14.1 ms\n",
      "Wall time: 49.6 ms\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "gpu_snow = gpu_df[gpu_df.year > 1897].groupby(['year', 'month']).sum().groupby(level = 'month').snow.mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "id": "53a2b1fa-e777-4102-b3b3-21c1a18b0ded",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<AxesSubplot:xlabel='month', ylabel='Average Snowfall [in]'>"
      ]
     },
     "execution_count": 63,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEGCAYAAACKB4k+AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8qNh9FAAAACXBIWXMAAAsTAAALEwEAmpwYAABAk0lEQVR4nO3dd3iU1fLA8e/sbgi9gzQh9CbVIB3pgv3avXZF7L0XwIJXvfZr+4kFFRC7gmKhiygtQOi9I71DIHXn98duYsAkbJJ9tyTzeZ48kGT3PRPKzr7nnDkjqooxxhgD4Ap3AMYYYyKHJQVjjDFZLCkYY4zJYknBGGNMFksKxhhjsnjCHUAgqlatqnFxceEOwxhjosr8+fP3qGq1/DwnKpJCXFwcCQkJ4Q7DGGOiiohsyu9zbPrIGGNMFksKxhhjslhSMMYYk8WSgjHGmCyWFIwxxmSxpGCMMSaLJQVjjDFZLClEIDvO3BgTLpYUIsyOg8n0eeU3hny/NNyhGGOKIUsKEeRISjo3fjyP9XuSGDV7E7+v2R3ukIwxxYwlhQiRnuHljjELWLHjEDFuAeDusQttKskYE1KWFCKAqjJk3DJ+W70bt0tIy/Algv1H03hj8powR2eMKU4sKUSAd39bx9i5m3G7ID3j+DuDN6et5fCxtDBFZowpbiwphNn4Rdv47y+r8LiEDO8/v5/hVe74bEHoAzPGFEuWFMJo7oZ9PPBlIjFuId2b+9rBjDV7WLL1YAgjM8YUV5YUwmTd7iPc/GkCgqB5JIRMt4yyfhLGGOc5lhRE5CMR2SUiS0/4+l0iskpElonIf50aP5LtOZLC9R/N5WhqOqCkB7DBaNvBZEbN2uh0aMaYYs7JO4WPgQHZvyAivYALgNaq2hJ42cHxI9Kx1AwGfZLAtoPJuBBSMwLfcvrsjytITc9wMDpjTHHnWFJQ1RnAvhO+fBvwgqqm+B+zy6nxI1GGV7n3i4UkbjmAW4SUnFaW85Ca4eXBrxY7FJ0xxoR+TaEJ0F1E5ojIbyLSIbcHishgEUkQkYTdu4tGZe9zE1bw67KdeFxCaj4TQqbxi7axcU9SkCMzxhifUCcFD1AJ6AQ8BHwpIpLTA1V1hKrGq2p8tWrVQhmjI0b+sYGP/tiA25X3TqNA3PypLTobY5wR6qSwFfhWfeYCXqBqiGMIuYnLdvDMD8v9tQiFP7Ziza4j/Lh4WxAiM8aY44U6KXwP9AYQkSZACWBPiGMIqcQtB7h77EI87uAkhEyPfL2YjAJOQRljTG6c3JI6FpgFNBWRrSJyE/AR0MC/TfVz4Dotwie+bdl3lJs+nodXFVUlmD9oUmoGz/64IohXNMYY3xy/I1T1yly+dbVTY0aSA0dTuW7kXA4lpyECTuwk/WT2Rm7v1ZDq5UsG/+LGmGLJKpodkJKewS2j5rNxTxKCkBpIdVoBqMIto+Y7cm1jTPFkSSHIvF7l4a8XM2fDPtyF2HoaqIVbDvDH2qKxZdcYE36WFILslUmrGJe4DU+2vghOu+sza8ZjjAkOSwpB9Pnczbw9bV1QahHyY9/RNN6cYs14jDGFZ0khSH5bvZsnvlsatFqE/HpjqjXjMcYUniWFIFi+7RC3jZ6PywXeME3jZHiVu8YuDMvYxpiiw5JCIW0/eIwbPp5Lmn9BOQw3CVmmr97N0r+sGY8xpuAsKRTC4eQ0bhg5j71HUnEJIVtYzottUTXGFIYlhQJKy/By+5gFrN55GJcIKQ7VIuTXXweOMXrWpnCHYYyJUpYUCkBVefK7pfy+Zg+uENQi5NczE5ZbMx5jTIFYUiiAd6av44uELbhdkB4BU0YnSk338vDX1ozHGJN/lhTyaVziX7z06yrcLiHCbhCO833iNjbutWY8xpj8saSQD7PX7+XBrxYRE+RjsJ0y2JrxGGPyyZJCgNbuOszgTxMQfOcbRYPVO4/w05Lt4Q7DGBNFLCkEYPfhFK4fOY9jab7F2whcRsjVQ18vipokZowJP0sKJ3E0NZ1Bn8xj+8FkXAip0ZQRgKSUDJ6dsDzcYRhjooSTndc+EpFd/i5rJ37vQRFREYno/swZXuXusYks3noQtwgpkbyynIeP/9zIniPJ4Q7DGBMFnLxT+BgYcOIXReRUoB+w2cGxC01VefbH5UxesTMkfRGcpAq3fGqVzsaYk3MsKajqDGBfDt96DXgYgtqyOOg++mMjH/+5EU+Ij8F2yvzNB5i1bk+4wzDGRLiQrimIyPnAX6q6KIDHDhaRBBFJ2L07tJ3Fflm6neE/Li8yCSHTHdaMxxhzEiFLCiJSGngCGBrI41V1hKrGq2p8tWrVnA0umwWb93PP54l4oqQWIT/2JaXy9rS14Q7DGBPBQnmn0BCoDywSkY1AHWCBiNQIYQx52rQ3iUGfJKCqvo9wB+SA1yavISklPdxhGGMiVMiSgqouUdXqqhqnqnHAVqC9qu4IVQx52Z+UyvUj53E42de9LD1615XzlOFV7vxsQbjDMMZEKCe3pI4FZgFNRWSriNzk1FiFlZyWweBRCWzeexSJwlqE/Jq2ajfLrBmPMSYHTu4+ulJVa6pqjKrWUdUPT/h+nKqGfTuM16s8+NUi5m3cj8tFVG89zQ9rxmOMyUmxr2h+aeIqfly8HY9LIqJzWqhsPXCMMXOsGY8x5njFOil8Nmcz705fV+S2ngbqmR+sGY8x5njFNilMW7WLId8vLbYJASAl3csj3ywJdxjGmAhSLJPC0r8OcvvoBbhc4C3mxVzfLfyLzdaMxxjjV+ySwrYDx7jx43mke30LysX0JuE4g6wZjzHGr1glhUPJadwwci77klIRoVgtLOdl9c4j/GzNeIwxgCe3b4hIIJ3fd6tqnyDG45i0DC+3j17Aml1H8LhcpBbV6rQCeujrRZzVsgYul4Q7FGNMGOWaFAA3cHYe3xdgfHDDcYaq8ti3S5i5dg8x7ug+BtspR1IyeO6n5Qw5t2W4QzHGhFFeSeEWVc1zI7uI3B7keBzx5tS1fD1/K26XTRnlZeQfG7mtZ0Oqli0Z7lCMMWGS65qCqs482ZMDeUy4fbtgK69OWo3HJdgNQt68CreMsnORjCnOTrrQLCJdRWSSiKwWkfUiskFE1ociuML6c+0eHv56cbGuRciv+Zv2M2vd3nCHYYwJk0B2H30IvAp0AzoA8f5fI9qanYe5ZfR8XCLWWCaf7vxsgf2ZGVNMBZIUDqrqz6q6S1X3Zn44Hlkh7DqczPUj55GclgEotoyQP3uTUnln+rpwh2GMCYNAksI0EXlJRDqLSPvMD8cjK6CklHRu+ngeOw4l4xKK/DHYTnlt0mprxmNMMZTX7qNMHf2/xmf7mgK9gx9O4aRneLl77EKW/nWIGLeLFKtFKLB0r9Jy2K80qFomKNcTB8ofBGdqKhyJtbj//MG/JIgzfwbB/vnrVy3DW/+O2PfR/3DSpKCqvQpyYRH5CDgX2KWqp/m/9hJwHpAKrANuUNUDBbl+DnHy9A/LmbJyFx6X1SIEy/o9SRSnejZH7ivtZrXYUmDZtkPc1vMgLWtVCHc4AcmrovlqVR0tIvfn9H1VffUk1/4YeAv4NNvXJgGPqWq6iLwIPAY8kr+Qc/bB7xsYNXsTbhe20yjI7I/TmIKrV6U0LWqWD3cYActrTSFz3qBcLh95UtUZwL4TvjZRVTMnqmcDdfIbcE5+WrKd535aYbUIxpiI4hJ475rTESfm5ByS652Cqr7n//Vph8a+Efgit2+KyGBgMEDdunVzvcj8Tfu49/NEYtxCui0qG2MiyL/a1aZZjei5S4A87hT8L8p5CuQxuTzvCSAdGJPbY1R1hKrGq2p8tWrVcnzMxj1J3PRJQubjberWGBMxSnpcPHV+eM4SS07L4NFvAjnT9J/yWmh+VET25PF9Ae4BRuRnQBG5Dt8CdB8tRIXUvqRUrhs5l6SUdASwjUbGmEghAs9f3JpyJWNCPraq8vh3S/h2wV94KtSol9/n55UUfsO3Uygvk/IzmIgMwLewfKaqHs3Pc7NLTsvg5k8T2LrvGG7baWSMiTANqpbhwra1wjL2p7M2+RKCS8hI2r8zv8/Pa03hhsIEJiJjgZ5AVRHZCgzDt9soFpjkX3iZraq35ue6Xq/ywJeLmL9pPyXsGGxjTIQJ5+Ly3A37eObH5bgE7u7TmHueT0nO7zUCKV4rEFW9Mocvf1jY6774y0omLNnur0WwVQRjTGS5okNdGlU/6QbNoNt+8Bi3jZ5PhldpekpZ7uzViHsKcJ2oasc5avYm3puxHredemqMiUClYlw8eW7zkI+bkp7BbaMXsDcpFY9LGHnDGQXuohg1SWHKip0MG7fUX4tgCcEYE1lE4JVL21C6hGMTMLl6avwyErccAODFi1tTq2KpAl8rr4rmHCuZMwVQ0Rw0x1IzuOOzBbhdQoZNGRljIlDzGuUY2KpmyMf9bM5mxs7dAkCXBpW5+PTC1QTnldJCPymWi417k6jlVVCwZWVjTKRxC7xzVegXlxds3s/QcUsB39TVu1fHn+QZJ5fX7iOnKpnzLd3rK0yzdQRjTCS6rksccUE6UThQuw4nc9uo+Vmvi+9fG0+F0oWvi8hr+uh/eT1RVe8u9OgBErAjLIwxEalsrJuHBzQL6Zip6V7uGLOAnYdTALioXW26Nc755If8ymv6aH5QRgiC5jXL06BuReZvPhDuUIwxJosIvH5FW0rGuEM67nMTljNv434AqpQpwX8uahW0a+c1ffRJ0EYpJLdL+Ob2rkxYvI37v1xkzXOMMRGhTZ2K9G1eI6Rjfj1/K5/M2pT1+aibOgY1KZ10S6qIVBORl0XkJxGZmvkRtAjy4ZzWtVg0rB99mlUPx/AmDx9f34HzWtekhDt6jgg2pjDcLnjr3+1COuaSrQd5/NslWZ/f3bsRLWoF9xTWQOoUxgArgPrA08BGYF5Qo8iHkjEePry+A2MHdaRcydDvBzY5Gzp+KW/+uz2rhg/krSvb0ah6aBfdjAm1wd0bUKdS6ZCNt/dICoNHJWQd7RNXpTT39G0S9HECSQpVVPVDIE1Vf1PVG4FOQY8knzo3qkri0P5cGh+UPj2mkDbvO8YX8zYjIpzbphaT7+/JoqH9ubZTPcrEhna+1RinVSgZw739gv+CnJv0DC93fraQ7Qd9Rxm5Xb5pI7cDvXIDSQpp/l+3i8g5ItKOIHVMKyy3S3jpkjb8dE93qpeLDXc4xd6wcctIS8/I+rxC6RieufA0lj09gM9u7ki7uhUdajVvTOi4xDdtFOsJ3ZudF35eyaz1e7M+H37haZxa2Zm7lECSwnARqQA8ADwIfADc50g0BdSiZnnmPN6H23s2tBedMEpO9/LYd0tz/F6XhlX57vaurHh2APf0aUyVMiVCHJ0xwdEhrjLdmwRn+2cgxiX+xQczN/w9fr1KXNEh926UhZVX57UX/b8tpaoHVXWpqvZS1dNVdbxjERWQiPDwgGbMfKQXDarZfHa4fDN/K3/tz71VRskYN/f1a8L8If349d7u9GxazZFbYGOc4HEJb1wRusXl5dsO8cjXf3dQKxnjYsS18Y5WTud1p3C2iMTg64EQNWpXKs3UB3oy9NwW9mITBgoM8rdIPZmmNcrz8Q1nsPrZATx7QUtqVyr4IV7GhMJdfRpRo0LJkIx14Ggqt4xKINm/Bd8l8M5V7ank8F12XknhF2AP0FpEDmX7OCwihxyNKghu7FafhCf60PbUCuEOpdhZseMwk5bvCPjxbreLazrH8ccjvZn1WG8ubFuLWE/UHOBrionKZWK4vWejkIyV4VXuGruQrfuPZX3tnNY16d3sFMfHzvV/nqo+pKoVgAmqWj7bRzlVPenGWBH5SER2icjSbF+rLCKTRGSN/9dKQfo5clSpTCzf39GN/13R1l5kQuz+LxdRkBbcNSuU4vUr2rFq+EDevao9TWtEzLmMphhzCbzz7/bEuEPzOvLKxFX8vmYPmf+DKpaK4b8XtwnJ2IH8hONFpCDp8WNgwAlfexSYoqqNgSn+zx13ftvaLBraj15NQ7c4VNwdTk7n+Z9XFOoaA1vV5Nd7e7B4WH9u6BpH2VirSzHh0bVRVTo1rBqSsX5esp13pq/L2jTjEt/201IlQrPbKZCkUA8YISLrReRLEblLRNqe7EmqOgPYd8KXLwAyj8/4BLgwH7EWSskSHkbecAajB51BOXtxCYkPft/AvqTUQl+nfKkYhp3XkqVPn8WXt3SiQ71KhKH9rSmmPC7h1cvahmSs1TsPc/+Xi/C4BMV3GOgtPRrQqk7opsFPmhRUdaiq9gZaAjOBhyj4YXmnqOp2/3W3A7meVyEig0UkQUQSdu/eXcDh/qlbo2okDuvPxe1rB+2aJmdehdtGB/dcxTPqV+Gr27qw6pkBPNC/CVXL2tZW4xwBHj6rKdVCUAd18Fgagz9NIN3rzeouWadSKR7o39TxsbML5OyjJ0XkZ2Ai0AhfrYLjxWuqOkJV41U1vlq14E77uF3CK5e1ZcJd3ahmLyqOmrNhH/M27D35A/OpRIybu3o3JuHJfky6vwd9mlXHY7vNTJBVKxfLTd0bOD6O16vc/0Uim/cdRX39xPC4hNGDOuIJ0TpGpkBGuwioAkwGvgXGZ77bL4CdIlITwP/rrgJeJyha1q7A3Cf6ckuPBlb05qDbxywo0KJzoBpXL8eH13dg9fABPH9RK+o6VOlpiheXwLtXtw/J1vY3pqxhyspdiPiaiQkw7LwW1KsS+pqrQKaP2gN9gLlAP2CJiMws4Hjjgev8v78OGFfA6wSNiPDY2c2Z8XAv4qrYi4kTdh9J5b0Z6xwfx+VyceUZdZnxcC/mPN6HS9rXpmSM7TozBdOnWXVOr1fZ8XEmLd/JG1PW+HrQ+7sCtDm1Ild3quf42DmRk72DE5HTgO7AmUA8sAX4XVWHnuR5Y4GeQFVgJzAM+B74EqgLbAYuVdUTF6P/IT4+XhMSAiuIKqwPfl/P8z+tJMPBd7bFkcclLB7Wn9JhWOSftHwHr01aw/LtEV9eYyJEjFuY83hfKjtcKLZu9xEueOsPUtIzSPN3l4z1uPjj0d5ULVv4dQwRma+q+WrcHEhSmADMAH4H5qlqWp5PcEAokwLAvqQUrv9oLov/sheRYOrfojojru0QtvEPJ6fxxpQ1fDlvC4eS08MWh4lsmVM313et7+g4R1LSufDtP9i0NwkU0ryKS+C9a06nX4vgNO4pSFIIZProHOA14BDQ1H/0RZFWuUws4+/qzmuXt6GEFb0FzcTlu1i943DYxi9XMoYnz2nB4qfO4rvbutCxfmVsbdqcqGaFklzbOc7RMVSVB79cxLrdRxCENP9uo7NanhK0hFBQgew+OhNYA7wNvAOsFpEeTgcWCf7Vrg6LhvSjR+PQFK0UB4M+Dd0dX17a1avEF7d0ZvXwgTwysKkdvW4A/O/U43E5/G7hnenr+GXZDtwiWU1zypf08PKlbR0dNxCBvA1+Feivqmeqag/gLHx3DsVCqVgPn97UkU9vPMOaxQTB5n1H+SphS7jDyOJxu7jtzEbMfaIv0x48k/4tTrGtrcXYwNNqOF4oNn3VLl7+dRVul5Duv0NwCXx8QwfKREBhbSBJIUZVV2V+oqqrgSI/hXSiHk2qkTikH/9qZ0VvhTVk3NLjmvFEivpVyzLi2njWPDeQly5pbbvRipkSHhf/+VdrR8fYvPco93yeiMcteL1/r+de3yWO9iHY6RSIQJJCgoh8KCI9/R/vU/CK5qgW43Hz2uVt+eHOrlSxorcCS07z8uT3y8IdRq5EhEvjT2X6Q71IeLIvl8fXoVSM3SUWZQI8c34LKpR27v3u0dR0Bo9K4GhqOgJZh93VrliSx85u7ti4+RVIUrgNWAbcDdwDLAdudTKoSNeqTkUSnujLoO71reitgL5M2JJnM55IUbVsLC9e0oYVzw7g4xs6cFqt8vZ3XgTVrVKayx3sZqaqPPLNElbuOIwgpPq3n7pdwqibOobs9NVAnHRLaiQI9ZbU/Ni0N4lrPpzL5n2R/wIXaVrULM9P93QPdxj5lpSSxptT1/H53M0cOBbyHdomyFwCP93TnWY1TtoRoMA++H09wyeswJNtHUGAJ89p7ugxGo5sSRWRrv7eB6v9J6WuF5H1BQ+zaKlXpQwzHu7FowOb4bajO/Nl+fZDTF6+M9xh5FuZ2BgeHdiMxGH9GX9nV7o0rGJbW6PYv9rVdjQh/Ll2D//56fiEANCyVnlu7OZsLURBBFK8thK4D986QtbqoKoG/5SzXETynUJ2ew6ncP3IuSzdZkVvgSpf0sOiYf0d7TkbCukZXkb+sZEPZ25gx6HkcIdjAlQyxsW8J/pSrqQzawlb9x/lvDdnkpSSQXqGF/8pFpRwu5j5aC+ql3O2tacjdwrAQVX9WVV3qerezI8CxlikVS0Xy493d+flS1pTIoLmCCPZoeR0XvxlZbjDKDSP28XNPRow+/E+/PZQTwaeVoMYd3QnuqJOBJ6/qLVjCSE5LYNbR83ncHI6oFkJwSXwxuVtHE8IBRXIK9c0EXlJRDqLSPvMD8cji2KXxJ9K4tB+dGtkRW+BGDFjfVCa8USKelXK8O7Vp7N6+EBeu6wNDauF/qRLc3INq5Xlwra1HLm2qvL4d0tYuu0QLvl7YRmgd7PqDGztzLjBEMj00bQcvqz+xjshES3TRzmZtnIXd3y2gKOpkbcvP5J0alCZzwd3DncYjtl3JIWXJ63m+4V/2b+FCOASmHjfmTSqXtaR63/y50aGjV/2j3WEsrFuZj/eN2StZR05EC8SRHNSAEhNz+CBrxbzw6Jt4Q4lon11a2c6xEVGAY+Tfl+zm5d/XcXirQeJ/P99RdNVZ9TluYtaOXLtuRv2ceX7sxE4LiG4BL4Y3IkO9as4Mm5OgrqmICLniUi9bJ8PFZFFIjJeROIKEWexU8Lj5s0r2/H97V0cP4o3mt0+ZkG4QwiJ7o2rMe7Obix/ZgB39W5IJQcLpsw/lYpx88S5zhSL7TiYzG2j5+MW8J7whvuqjvVCmhAKKq81heeA3QAici5wNXAjvkY57zkfWtHTtm4lEp7oyw1d46wAKge7D6cwIgTNeCJFqRJuHujfjIVD+zPh7m50b1QVW5t2lgi8cmkbSpcI/vRNSnoGt46ez0F/7Uq2mwRqlI9lyLktgj6mE/JKCqqqmRVZFwEfqup8Vf0AKFTTZBG5T0SWichSERkrIpG5DO8Al0sYdl5Lpj5wJnUqlQp3OBHnv7+s4lhq8et10LJWBUYN6sjq4QN56rwW1KpYbP5LhFTzmuUZ2MqZo6mfGr+MxC0HcAnHLSy7XTB6UMeoOYY/ryhFRMqKiAtfO84p2b5X4H+xIlIb35EZ8ap6GuAGrijo9aJV/Wpl+f3hXjx0VlMrfMom3avc+0ViuMMIG7fbxfVd6/Pno33445FenNe6pm1vDhK3C969qr0jNTGfzdnM2LlbcLuOTwgCPHRWUxpVLxf0MZ2S17+214FEIAFYoaoJACLSDtheyHE9QCkR8QClgWK5Aisi3NGrEXMe70uLmtHzj8Zpvy7byeqd4WvGEylqVyrNm/9uz6rhA3jrynY0dminTHFxfZc46lUJ/vbgBZv3M3TcUjzZeixnalqjHIO7Nwz6mE7Kc/eR/119dWCRqnr9X6uJ7zjtzQUeVOQefGsWx4CJqnpVDo8ZDAwGqFu37umbNm0q6HBR44t5m3ny+6VZvVqLs3pVSvPbQ73CHUbEOXg0jVcmreKbBVtJSrGtrYEqG+sh4cm+lAzyabe7Didz3v9msv9oKhleJft/3Ri38PvDvalRIXxTgUGvaFbVv1R1YWZC8H9teyETQiXgAqA+UAsoIyJX5zD2CFWNV9X4atUKtYQRNS7vUJeFQ/rTuUHR35Z5Mpv2HuXr+VvDHUbEqVA6hmcuOI1lTw9g7M0daV+3om1aOAmXwBtXtA16QkhN93LHmAXsSUoF4biEIAIvX9omrAmhoMIxWdkX2KCqu1U1DfgW6BKGOCJS2ZIexg7uzAfXxVOqRPE+w//J75eQfuL9uMnSuWFVvr29KyufHcC9fRpTxbY756jNqRXp0/yUoF/3uQnLmbdxv29hOf34u/vujapyQdvobMgVjqSwGegkIqXFt+LTB1gRhjgiWt/mp7BoaD/OcWinRDTwNeNZGu4wIl5sjJt7+zVh/pB+TLy3B72aVsNtuxcA3+Lym1e2C/p1v56/lU9mbcLt4h/TvaVLuHnrqug9CSigpCAi3UTkBv/vq4lIgc97VdU5wNfAAmCJP4YRBb1eUVbC4+btq07nm9s6F9sCpy/mbWHbgWPhDiNqNKlRjpE3nMGa4QMYfuFpxX7b8609GlCnUnDbqi7ZepDHv12S48KyS+DD6+Ip79Ahe6EQyNlHw4B4oKmqNhGRWsBXqto1FAFC9B9zEQxer/LUD8v4dFbRX3A/Ucta5Zlwd/Q144kU2w8e47+/rOSnJTtISS8+03EVSsUw74m+Qa0P2HskhfPemsmewyl4VTnxj/Py+FN58RJn+zznh1NHZ/8LOB9IAlDVbYDtnwwxl0t45oLTmPLAmdSuWLze/S3bdoipK6KvGU+kqFmhFK9d3o5Vwwfy7lXtaVaj6P/3dQm8dWW7oCaE9Awvd41dyM6DyYjwj4RQrWwsz1zYMmjjhUsgf2Kp6rudUAARsXOAw6hhtbLMfKQX9/drXKyK3u79IpFoOLwx0g1sVZNf7u3Bkqf6c2O3OMqF6LTOUOsQV5nuTYK7a/HFX1by57q9uFxCygkLy24XjBp0BrGe6N8cEkhS+FJE3gMqisjNwGTgfWfDMnkREe7u04Q/H+tdLN71ga8Zz39/jf5mPJGiXMkYhp7bkiVPn8VXt3aiQ1wlorz5XRaPS3jjiuAuLo9L/Iv3f9+A2yX/WFgW4J4+jR1t6RlKAR2dLSL9gP74fv5fVXWS04FlZ2sKeftszmaGjlt63DG9RZFLYMGQflQsbVsvnZCalsH7M9cz8o+N7DkSvU2P7u/XhLv7NA7a9ZZvO8RF7/xBhmqOhaWNqpVh4n1n4orAW3frp1CMHU5OY9AnCczZsC/coTiqc4PKjC3CzXgixdpdR3jh5xVMX7U7qt5sVC5TgjmP9yEmSOdFHTiaynlvzmTHoWQ0h4Vlj0uY8XAvakXoOp8jC80iclhEDp3wsUVEvhORBgUP1wRTuZIxfHFLZ9675nRKBblyM5LMWr+P+Zv2hzuMIq9R9bJ8cF0HVg8fwAsXt6Ju5eBu63SCS+Cdf7cLWkLI8Cp3f57IXweO4eKfC8si8MJFrSI2IRRUIH96rwIPAbWBOsCD+NYUPgc+ci40UxBntaxB4tB+DGhZdIvebhs9P9whFBsul4srOtRlxsO9mPt4Hy45vTYlYyLz1NaujarSqWHw+qK/MnEVM1bv9i0s5zBt1Ll+ZS6JPzVo40WKQOoU5qhqxxO+NltVO4nIIlVt42iE2PRRQc3buI+bP0nggL/pR1HyxNnNuLlHdJ0+WZRMXr6TVyetZvn2Q+EOBfBN48x6rA/VysUG5Xo/L9nObWMW4BYhI4fXyFIxLmY/1pcKEV5U6lSdgldELhMRl//jsmzfi57JxmKoQ1xlFgzpx1Ud64Y7lKD776/FsxlPpOjb4hR+uqc7S5/qz+DuDShfMnxbWwV4+KymQUsIa3Ye5oGvFhHjzjkhuATevzY+4hNCQQWSFK4CrgF2ATv9v79aREoBdzoYmwkCl0t47l+tmHRfD2pG4YmNuUnLUO7/clG4wyj2ypaM4fFzmrP4qbP47vYudKpfOeT1M9XKxXJT9+Asbx5KTuPmTxNIy+Mgxgvb1qZb46J7crPtPipGVJXXJ6/hzalriKINJXmaeF8PmpxSPGo1okV6hpcPZq5n5MyN7Dyc4uhYLoGvbu3M6fUKf9y816vc/GkC01btwuNykZpDYqhcugR/PtY76MdwO8WRLan+/sk3AS3J1oZTVW8sSJAFYUkhuLYfOMZ1I+eyeueRcIdSaHFVSzP9QWvGE6k27jnC8z+vZOrKXY40j+rXvDrvX9chKNd6ffJqXp+8Bo9LctyG6xYYf1c3WtaqEJTxQsGpNYVRQA3gLOA3fDuQrFdiFKtZsRQT7zuTZy9oiScCC27yY+Oeo3y3wJrxRKq4qmV575p4Vg8fyMuXtqF+ENthxriFFy8Jzj6Xyct38vrkNbhzSQgC3NazYVQlhIIKJCk0UtUhQJKqfgKcA7RyNiwTCtd0jmPBkH7E16sU7lAK5fHvlloznggnIlxyeh2mPdSThCf7ckWHUwtVTyPAk+c0p3IQGgut332Ee79I9C0s5zKvWq9Kae7r17TQY0WDQJJC5n7GAyJyGlABiHMsIhNS5UvF8PVtXXj3qvYRu//8ZI6lZTB0nDXjiRZVy8bywsWtWfHsAD6+oQOtapfPd0vRWhVLck2nuELHciQlncGj5pOSnpFrDB6XMOqmjsWmcVEgrwIj/H2VnwTGA8uBFx2NyoTcwFY1WTS0P/0caFsYCmPnbmG7NeOJOj2bVueHu7qz7OmzuK1nQyqWOvk2T5fAe9fEF/qsIVXlwS8XsW7XEQQhNYc1DwGeubAlp0ZBRXew5JkURMQFHFLV/ao6Q1UbqGp1VX2vMIOKSEUR+VpEVorIChGxw2wiQGyMm/evi+eLwZ2oUCq6jlRWYPAoq3SOVqVjPTwyoBmJw/rzw13d6NqwSq5bW89pVZPTahd+bv+d6ev4ZdkO3C7JcacRQHy9SlzZoejV+eQlz6Sgql6cqUV4A/hFVZsBbbAezRGlY4MqLBjSnys6RFcJ/5K/DjJt5a5wh2EKqVXtCoy5uRNrhg/kyXOaU+OE+ppmNcqzL6lwp7j+tno3L/+6KtedRgAlY1yMuDYeKSpnigcokC2pQ4BjwBf4u68BqGqBjuMUkfLAIqCBBlgkYVtSw2fVjkNc+9Fcdh5ydr95sFQoFUPi0H7F7j9yUZaclkGn56dw4Ojfx7W4RejVrBrXdo6jW6Oq+ZpK2rz3KOe9NZOjqemkZ2iOxzK4BD64Np7eUTqdmsmpLak3AncAM4D5/o/CvEI3AHYDI0VkoYh8kFM3NxEZLCIJIpKwe/fuQgxnCqNpjfLMfqwPd/ZqFBVNWA4eS+OViavDHYYJonemr+PA0TRKHHf6qTJ5xS6u/WguHZ6bzKsTV/FXAGtKR1PTGTwqgaSUdITcz+k5p1XNqE8IBRXyimYRiQdmA11VdY6IvIFv3WJIbs+xO4XIsO3AMa79cA5rdyed/MFhZM14io6Ne5Lo/9oMMlRz3S7qErIq9DvEVeL6LvXp26L6P1pjqir3fJ7I+EXbKOHOuWIZoGKpGGY91odSJaKjajkvTvVTKC0iT4rICP/njUXk3IIGCWwFtqrqHP/nXwPtC3E9EyK1KpZi8gM9eer8FhG9Pc+rcPuYBeEOwxSSqjJs/DK8qkgeZ29mJgQB5m3czx2fLaD9M5MYNm4pq3f+XWf74cwNjF+0DU8eC8sugU9vOqNIJISCCmT6aCSQCnTxf74VGF7QAVV1B7BFRDIrQfrg2+ZqosT1Xeoz/8m+tK9bMdyh5OrPdXtZuNma8USzX5ft4LfVvqnjExvc5CR72khOy+CTWZvo/9oMznptBq9NWsXzP63Mc2FZgJu7N6B1nYqFjj2aBZIUGqrqf/EXsanqMch3rcmJ7gLGiMhioC3wn0Jez4RYxdIl+Pb2rrz973bEeiKz6O1Wa8YTtY6mpvP0D8uJcef+Ip6XjGx3D6t3HeaNKWsR8R16l5s6lUrx0FnFo2o5L4H8b071H5OtACLSECjUVhRVTVTVeFVtraoXqqq9pYtS57SuxaJh/ejTrHq4Q/mHnYdS+HDmhnCHYQrgf1PWsv1gMlLI958KZC6bpnuV3G44PC5h9KCOeILUyjOaBfIn8BTwC3CqiIwBpgAPOxmUiS4lYzx8eH0Hxg7qSLkwNlvJyQs/ryA5NSPcYZh8WLvrMO//vh635D73H0wCDDmnOfWCeFhfNDtpUlDVicBFwPXAWCBeVac7G5aJRp0bVSVxaH8uja8T7lCypGUo93+VGO4wTIBUlSHfL/PfH4RmZ2SbUytybZe4kIwVDQLZfTQe6A9MV9UfVXWP82GZaOV2CS9d0oaf7ulO9SC1Ryysn5bsYO0uO+09GoxftI1Z6/cCf68LOCnW4+KD64pf1XJeApk+egXoDiwXka9E5BJ/4x1jctWiZnnmPN6HW89sUOhdCcEw+FNbdI50h5PTeG7CigIvLueXS+DNK9tStWxkvHmJFIFMH/2mqrfjq0QeAVyGr1+zMXkSER4d2JyZj/SiQdXwzteu35PE9wutGU8ke23SGnYdTsEVorcR/VucQv+WNUMyVjQJaKndv/voYuBWoAPwiZNBmaKldqXSTH2wJ0PPDW/R22PfWjOeSLVi+yE++XMjbhekhODvqHxJD69c1tbxcaJRIGsKX+A7xbQ38Da+uoW7nA7MFD03dqvPvCf60KZOeFoaHkvLYNj4ZWEZ2+TO61WGfL8Ulws0BDnbJTDyhg6UiY2snXKRItCK5oaqequqTgU6i8jbDsdliqjKZWIZd2c3/ndF27AUvX02ZzM7DloznkjyzYKtJGzylSqF4j7uui5xnF6vcghGik6BrCn8ArQSkRdFZCO+Iy5WOh2YKdrOb1ubxCH96Nm0WkjHtWY8keXg0TSe/9l3/ERaCLYb1a5YksfPbu74ONEs16QgIk1EZKiIrADewnfmkahqL1V9M2QRmiKrVKyHj284g9GDzqBcCG/lF289yPRVtlciErw0cSX7k1JxheCm0e3vtRxjVct5yutPZyW+w+rOU9Vu/kRgpaEm6Lo1qkbisP5c3L52yMa85/NEQn1svDne4q0HGDN7My6B1HRn/y4EeHRgUxpUK+voOEVBXknhYmAHME1E3heRPhT+IDxjcuR2Ca9c1pYJd3Wjalnn+yAcPJbGq9aMJ2wy/IvLHrcQitzcslZ5BnVr4PxARUCuSUFVv1PVy4FmwHTgPuAUEXlXRPqHKD5TzLSsXYF5T/RlcA/ni97emb6Og8cK1+vXFMzn8zazaOtBwPnF5RJuFx/d0MGqlgMUyEJzkqqOUdVzgTpAIvCo04GZ4ktEePzs5vz2UC/iqpR2bJwMVe6wZjwht/dICv/9ZVVIFpddAq9f3obq5ewQhkDla8VFVfep6nuq2tupgIzJVLdKaaY/1IvHz26G26F3eTPX7iXRmvGE1H9/WcWh5DRCUcfYq2k1zm5dy/mBipCwLcOLiFtEForIj+GKwUSHwT0aMu/JPrSqXd6R69862u4WQmX+pv18kbAFF5Dq8F1C2Vg3b1xpnX7zK5x7s+7BVyltzElVLhPLD3d159XL2lAiyFsKdxxKZuQf1ozHaekZXoZ8v5QYt+D0eXcugZHXd6CsVS3nW1iSgojUAc4BPgjH+CZ6XdS+DolD+9GjcdWgXvc/P1kzHqeNnr2J5dsPIYjjnRL+3bEuHepXcXiUoilcdwqv4+veZqeTmXwrHevh05s68umNZ1Am1h2Ua6ZlKA9+tSgo1zL/tOtwMi9PXI3H5Xw3tRrlYxl6bktHxyjKQp4URORcYJeq5nnWgIgMFpEEEUnYvXt3iKIz0aRHk2okDunHhW2Ds5D445LtrN99JCjXMsd7/qeVHE1Nx+ldoW4XjLqpIyXCcK5WURGOP7muwPn+c5Q+B3qLyOgTH6SqI1Q1XlXjq1UL7fk4JnrEeNy8fkU7xt3RlSplCl/0dvOnCUGIymQ3e/1evlv4FwKObkEV4MH+TWl8SjnHxigOQp4UVPUxVa2jqnHAFcBUVb061HGYoqXNqRVJeLIvN3WrX6iit3W7kxiX+FfQ4iru0jK8DB3nW1x2+ry7pjXKckuPhs4OUgzYPZYpMkSEIee2YPpDPalbueBFb499u8Sa8QTJyD82sHrnEcTh+vQYt/DxDR1xhbGJU1ER1qSgqtP9ldLGBE29KmWY8XAvHh3YDFcBJrGPpmbw9A/WjKewth88xmuT1uB2eHFZBF66uA01KljVcjDYnYIpsm49syFzH+9Dy1r5L3obPXszOw8lOxBV8TH8xxWkpntxO/zmvXujqlwYwhN2izpLCqZIq1oulgl3d+flS1oTk49XJwUG26Jzgc1YvZsJS7Yj4mzlcukSbt66yqqWg8mSgikWLok/lUVD+9O1YeAFTYu2HmTGatsOnV8p6RlZi8vpDpYuuwQ+vC6e8iVjHBujOLKkYIqN0rEextzciZHXd6B0icCK3u7+fKE148mn92esZ+PeowVaz8mPS06vQ+eGwa1sN5YUTDHUq1l1Eof247w2NU/62ANH03htsjXjCdSWfUd5a+pa3C5ISXducblq2RI8e+Fpjl2/OLOkYIqlEh43b17Znu9v70Ll0nlPP7w9dR2HrBlPQJ7+YTlpXsXl4BZUt8DoQR2J9QTniBNzPEsKplhrW7cSCU/24/oucbm+jGWocsdnC0MaVzSasmInk1fs9FUuO7SWIMDdfRrTrIYzx6gbSwrG4HIJT53fkqkPnEmdSqVyfMzva/awaIs148lNcloGT41f5vjicsNqZbird2PHrm8sKRiTpX61svz+cC8ePKtJjl3BrBlP7t6ZtpYt+485WrnscQmf3mRVy06zpGBMNiLCnb0aM/uxPjSvefzBatsPJvPJnxvDE1gE27AniXd/W+do5bIIvHBRK2pVzPlOzgSPJQVjclC9fEl+vqcHL17c6riit2d/XG7NeLJRVYaNX4YqjrbO6Vi/MhefXsex65u/WVIwJg+Xd6jLwiH96dygMgDpXmXg/2aEOarI8cvSHVkFfk7tQC0V4+K9q+MRp5sxGMCSgjEnVbakh7GDO/PBdfEAbNhzlLNem8Guw8X7bKSklHSe/nG5o4vLLoER18RT4STbhk3wWFIwJkB9m5/CxHu7A7Bq52G6vTCN0bM3FduK5/9NXcOOg8mOLi5f2LY23ZtYk61QsqRgTD40qVGeazvXAyA1w8uT3y/lvDdnsnnv0TBHFlprdh7mg9834BbnFpcrly7Bfy5q5ci1Te4sKRiTTw8PaEbZWE/W++Ol2w7R8+Vp/G/KmmLRnEdVGTJuqf/nd+YuyS0watAZlIyxquVQC3lSEJFTRWSaiKwQkWUick+oYzCmMMrGenjqvBbHvRyqwquTVtP7ld9Yvu1g2GILhfGLtjF7/T4EHGmxKcBtPRvSslaF4F/cnFQ47hTSgQdUtTnQCbhDRFqEIQ5jCuzi0+vQ5JSyWZ9nvjZu3neUc/43k2Hjl5GcVvS2rh5KTmP4jyuIcYtjR1nUq1Ka+/o1deTa5uRCnhRUdbuqLvD//jCwArC2SSaqiAivXtbW9/scvv/Jnxvp9uJUZq/bG9K4nPb6pDXsPpLi2AuHxyWMuqkjbqtaDpuwrimISBzQDpiTw/cGi0iCiCTs3m2NToq6vw4c42hqerjDyJfTalfggja1/jGrnvn53iOpXPH+bO4eu5DDyWmhDi/oVmw/xCd/bvQdi+3AvJEAz1zYklMrlw76tU3gwpYURKQs8A1wr6oeOvH7qjpCVeNVNb5aNduSVpR5vcrZb/xO22cmcf8XiczftD9qtnkOO78lsR5XjmclZf4E4xdto/PzU/ll6faQxhZMXq8y5PuluFy+9RMnnF6vEld2qOvMxU3AwpIURCQGX0IYo6rfhiMGEzlcLmHouS1Iz/Dy7cK/uPjdP+n6wlT+b/q6iC8Qq1ymBA+d1ZSTTa8npaRz6+gFXPPhHPYcSQlNcEH0zYKtJGzynRLrxFJCrMfFiGutajkSSKjfkYnvb/0TYJ+q3hvIc+Lj4zUhwZqoF3Vrdx7mkvdmceDo31MtItC1YVWu7VyPXs2qE+OOvF3U6RleznxpGtsOJAe0QTPW4+Kp81tyRYdTo+JF8ODRNHq9Mp1Dx9IcqVx2CXxwbTy9m58S9GsXdyIyX1Xj8/OccPwP6wpcA/QWkUT/x9lhiMNEmEanlGPu433p3uj4vrsz1+5h8Kj5tHtmEs/8sIw1Ow+HKcKcedwuXr60TcA79lPTvTz27RLOf+sPtuyL/KK3lyauZH9SKi6HXi3OblXTEkIECfmdQkHYnULxM/KPDTzzw/LjXmjd8ve++CanlOW6LnGc16YW5UtGxrk41380l+mr87cpwu0S7u/XhFvPbBiRO24Wbz3ABW/9gcsFTtTlVSwVw6zH+lCqhBWpOaEgdwqWFEzEWr3zEJf+3ywOHjt+V1LmS6fi28J4VstTuLpTHB3rVw5rA5btB4/R7cVpeFUDXowV8S3cxlUpzXvXxNO0RrmTPylEMrzKv975gxXbD5GRoQQ7J7gEvr+jK63rVAzylU2maJk+MiYgTU4pz9wn+tK1YZXjvq78vbNHUSYs2cGV78+m4/NTeGPyarYdOBbyWAFqVijF7Wc2zNfunMzHbtx7lAGvz2D4j8tJSY+MorfP521m8VZfdXawE4IAN3drYAkhAtmdgokKH/y+nucmrMhz3t4lf++M6RBXieu6xNGvxSnEekI3NZGclkGXF6ayPyk136cCCb5kV61cLO9e1Z74uMoORBiYvUdS6P3KbySlpDuyuHxqpVJMe7AnngjcOFCU2J2CKbIGdW/AT/d0p3xJT66PyXztEmDexv3c+dlC2j0ziSe+W8KyEJ1HVDLGzQsXtSrQMXGZz9lzOIVL/m8W932RyJGU8BT0vfjLSg4lp+VYf1FYHpcwelBHSwgRyv5WTNRoXrM8c5/oQ6cGeb+Dzv6CnJLuZcyczZzzv5n0enk6H/+xgQNHUx2Ns1+LU2h3asUCPz8z/u8W/kXn56cwefmOoMQVqPmb9vFlwlZcQGqQK5cFuLl7fepVKRPU65rgsaRgokrJGA+fD+7M42c3C6i1S0a2qY+Ne5N46oflnP7sZG7+ZB4zVu8+7vvBIiK8fFkbRHI+Fyk/jiSnM+jT+Vw/ci57Q1D0lu7vERHjFkeK1DxuoUrZ2OBf2ASNJQUTlQb3aMiEu7tRLo/ppBP9vXymTFqxi2s/mkv88Em89OvKoDfJaVitLFd1rFvobgOZz5++ajddXpjKl/O2OHoEyKjZm1ix/TCCONQpwUQ6SwomarWoVYF5j/fhjLhK+Xpe9hmRQ8fSeHvaOnq8NI0L3p7Jdwu3ciw1OLt/Hh3YnDKxboJVtJya4eXhbxbzr3f+ZOv+4Be97TqUzCsTV+NxOddNzUQ+SwomqpUs4eHLW7vwyIDAppNOlJkgRGDxloPc98Ui2j4zkYe+WkTilgOFeldeNtbDsHNbBu0AuczrJG45wJkvTefd6WuDOv31n59WcDQ1PWhJzEQnSwqmSLitZ0N+uKsrZWMDn07KTvXvqZr0DOWr+Vu58O0/6PbiNEb8tq7Ah9hdGl+HRtXLnvyB+eRV5cVfVtH31d+CcuzHrHV7+T5xGwKkOdFOzUQNSwqmyDitdkXmPdGH0+vlbzrpRBnZ3tpvP3iM//y8kjOem8y1H85h8vKd+erDLCK8fnlb3+8LFdXxMkPcsCeJ/q/N4LkJy0lNL9iUT1qGl6HjfIvLlg+MJQVTpJQq4eGb27rwYP8mQXkRzj47M2PNHgZ9mkD7Zycx/MflrN11JKBrnFa7Aue1/mcznmB6//cNdP/vVBZs2pfv5340cwNrdh1Bgpq2TLSypGCKpDt7N2bcnV0pExucaubsySEpJZ0PZm6g76u/MfCNGXwxb/NJi8yeOr8FJXJpxlNYmaHtOpTCRe/O4oEvF5EUYNHb9oPHeH3yGty2uGz8LCmYIqt1nYokPNGXdnUrBvW6Gdkqp1duP8wj3yyh3TMTuXvsQuZu2Jfj4nSVsrE8dFYTR/b+Z8q89DcLttL5+SlMXbnzpM959kfftJPbbhKMnyUFU6SVKuHhu9u7cn+/JkG/dvaD+bxeZfyibVz23iw6PT+Ft6auYcfB47vG3dClPrUqlAzJJM3h5HRu/DiBmz6ex76knCu4Z6zezU9LdiAS/MplE70sKZhi4e4+jfn+ji6Ucejc/uyvqbsPp/DyxNV0fn4KV4yYxc9LtpOa7sXjdvHKZYE34ymMzDGmrNxFlxem8PX844veUtIzshaXnTjwzkSvcPVoHiAiq0RkrYg8Go4YTPHT9tRKzHuiL23qVHB0HG+22ofZ6/dx25gFtHtmIkPHLaVSmRL0aFw17wsEWUq6lwe/WszF7/6Zdaz4iN/Ws3HvUVtcNv8Qjh7NbmA10A/YCswDrlTV5bk9x47ONsH22qTVvDFlTcjGc7skq9CsSpkS7E1KzWqwE0oel3DrmQ15//f1pHu9jnRTy0uMW3hkQDMGdW8Q2oGLqYIcnV2wSp/COQNYq6rrAUTkc+ACINekcDg5nc/mbA5ReH/bduAYv+WzvaIpmHSvsmL7oXCH4Zjslcd7/XP84Whlku5V3pq2NvQD+6VlKMMnrGBc4rawxRBqHrdwUbvauJ1qcp0H8cSWzO9zwpEUagNbsn2+Feh44oNEZDAwGEBiSnLDhX1CE102GUcP4i7t7FRDpLGfuXgI98+8PQxjhvNn/iksowIiLfP7lHAkhZwmMf/xnklVRwAjnA8ndyKSkH5wV75uvaKd/czFg/3MxYOI5HvePRwLzVuBU7N9XgcoPveSxhgTwcKRFOYBjUWkvoiUAK4AxochDmOMMScI+fSRqqaLyJ3Ar4Ab+EhVl4U6jgCFdfoqTOxnLh7sZy4e8v0zh3xLqjHGmMhlFc3GGGOyWFIwxhiTxZJCDkTkVBGZJiIrRGSZiNwT7phCRUTcIrJQRH4MdyyhICIVReRrEVnp//vuHO6YnCQi9/n/TS8VkbEiku/ipmggIh+JyC4RWZrta5VFZJKIrPH/WrhuTBEkl5/3Jf+/68Ui8p2IVAzkWpYUcpYOPKCqzYFOwB0i0iLMMYXKPcCKcAcRQm8Av6hqM6ANRfhnF5HawN1AvKqehm+jxxXhjcoxHwMDTvjao8AUVW0MTPF/XlR8zD9/3knAaaraGt/RQo8FciFLCjlQ1e2qusD/+8P4Xihqhzcq54lIHeAc4INwxxIKIlIe6AF8CKCqqap6IKxBOc8DlBIRD1CaIlojpKozgBPb0F0AfOL//SfAhaGMyUk5/byqOlFVM7stzcZXE3ZSlhROQkTigHbAnDCHEgqvAw8DxaUFVwNgNzDSP2X2gYiUCXdQTlHVv4CXgc34Tpo4qKoTwxtVSJ2iqtvB98YPqB7meELpRuDnQB5oSSEPIlIW+Aa4V1WL7mltgIicC+xS1fnhjiWEPEB74F1VbQckUbSmFI7jn0O/AKgP1ALKiMjV4Y3KOE1EnsA3JT4mkMdbUsiFiMTgSwhjVPXbcMcTAl2B80VkI/A50FtERoc3JMdtBbaqauZd4Nf4kkRR1RfYoKq7VTUN+BboEuaYQmmniNQE8P+6K8zxOE5ErgPOBa7SAIvSLCnkQEQE3zzzClV9NdzxhIKqPqaqdVQ1Dt/i41RVLdLvIlV1B7BFRJr6v9SHPI5wLwI2A51EpLT/33gfivDCeg7GA9f5f38dMC6MsThORAYAjwDnq+rRQJ9nSSFnXYFr8L1bTvR/nB3uoIwj7gLGiMhioC3wn/CG4xz/HdHXwAJgCb7//0Xy6AcRGQvMApqKyFYRuQl4AegnImvwNfl6IZwxBlMuP+9bQDlgkv817P8CupYdc2GMMSaT3SkYY4zJYknBGGNMFksKxhhjslhSMMYYk8WSgjHGmCyWFIxxgP/01duzfd6zuJw8a6KbJQVjnFERuP1kDzIm0lhSMMWeiMT5z53/wN9nYIyI9BWRP/xn75/hP4v/e//Z9LNFpLX/uU/5z7KfLiLrReRu/2VfABr6i4Ze8n+tbLbeDWP8VcXGRBRPuAMwJkI0Ai4FBgPzgH8D3YDzgceBLcBCVb1QRHoDn+KrgAZoBvTCVz26SkTexXew3mmq2hZ800f4Ttttie+46j/wVc7PdPwnMyYf7E7BGJ8NqrpEVb3AMnzNWBTfcRBx+BLEKABVnQpUEZEK/udOUNUUVd2D75C1U3IZY66qbvWPkei/rjERxZKCMT4p2X7vzfa5F98ddU5TPZlnxGR/bga534EH+jhjwsaSgjGBmQFcBVlTQXtO0mPjML7pJGOiir1TMSYwT+Hr0LYYOMrfRzDnSFX3+heql+LreDXB+RCNKTw7JdUYY0wWmz4yxhiTxZKCMcaYLJYUjDHGZLGkYIwxJoslBWOMMVksKRhjjMliScEYY0yW/wdeS9gonGwwMQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "gpu_snow.to_pandas().plot.area(ylabel = \"Average Snowfall [in]\", xlim = (1,12))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "71bd2995-94fd-499d-a53e-193d03882732",
   "metadata": {},
   "source": [
    "That's an interesting plot. :-) It brings us to another difference between pandas and cuDF - it's groupby operations do not guarantee order as pandas' do. As such, if we want to get the pandas-equivalent result, we can sort the resulting data series by its index:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "id": "af853bdd-6b22-4a1c-870e-88ee7f0c0475",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<AxesSubplot:xlabel='month', ylabel='Average Snowfall [in]'>"
      ]
     },
     "execution_count": 64,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEGCAYAAACKB4k+AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8qNh9FAAAACXBIWXMAAAsTAAALEwEAmpwYAAAoP0lEQVR4nO3deXycdbn38c81k6qUVRRQKVhUxKO4oPXIET0qqMdHcTmbjwvn4VGPPSoq4u5zVPQc9aAgKuLxsUJZtLRgW2jZWkppKYUCbdJ0SdM2bdI2SbMnbfZlZq7zx0xCKFkmydxzz/J9v17zSjLLfV9Twly5f7/fdf3M3REREQGIhB2AiIjkDiUFEREZoaQgIiIjlBRERGSEkoKIiIwoCTuAdLz4xS/2uXPnhh2GiEheKS0tbXX306bymrxICnPnzmXLli1hhyEiklfM7OBUX6PhIxERGaGkICIiI5QURERkhJKCiIiMUFIQEZERSgoiIjJCSUFEREYoKeQgtTMXkbAoKeSYo71DvP9XG/jxyoqwQxGRIqSkkGN+sGInVc3d3PLEAbbWdoQdjogUGSWFHLKivJ6V2w4TTf1XuXJxeajxiEjxUVLIEfVH+vj+PTuZFTXiieR9h9p7Wb2zIdzARKSoKCnkgHjC+fqd5fQNxrFjHvvu8h2hxCQixUlJIQfc9Fg1T9W0g8Fg/Nkrjzp6h7j18ZqQIhORYqOkELKKw0e5dvUeohEjFh97Keo1q3YTHx5TEhEJkJJCiPqH4nxtSTlmwAS1Cf1DCX6+enf2AhORohVYUjCzhWbWbGY7j7n/K2a2x8wqzOwXQZ0/H1zz4G6qmrsxjHEuEkbcvPEAvYOx7AQmIkUryCuFW4EPjL7DzN4DfBR4g7u/DrguwPPntA17W7j1iQNEIzCYxtBQPOF8d+n2LEQmIsUssKTg7huA9mPu/iJwjbsPpJ7THNT5c1lHzyDf/Ms2ZkWNxBSmCu7d3kBLV39wgYlI0cv2nMKrgXea2VNm9qiZvXW8J5rZfDPbYmZbWlpashhisNyd7y3fQWv3ABGDqXQ5cuAri7cGFZqISNaTQgnwQuBC4FvAXWZ27NJ8ANx9gbvPc/d5p512WjZjDNTS0jpWVTQSMWMgNvXGd09Wt1PV1BVAZCIi2U8KdcByT3oaSAAvznIMoTnU1svVKysoiRixxPQ7oV5xR1kGoxIReUa2k8I9wMUAZvZq4HlAa5ZjCEUsnuCqu8oZjCWIjHltlL69Td08sb8o/tlEJMuCXJK6GNgEnGdmdWb2OWAh8IrUMtUlwOVeJJsH/H79fkoPdmBjVC1Px1V3ls88KBGRY5QEdWB3/+Q4D10W1Dlz1bbaI/x6bRXRiDGUgYQA0NQ5wLLSOv7xLXMycjwREVBFc+B6B2N87c7y5JBRhi+Krl5ZoV3aRCSjlBQC9tP7K6lp7Umranmqugdi3LhuX2YPKiJFTUkhQGsrm1j01CGill7V8nTcsLaKoVg8kGOLSPFRUghIS9cA31q6PblpToAjPENx5+p7dwV3AhEpKkoKAXB3vrtsO0d6Bxm7NC+zljx9iM6+oeBPJCIFT0khAHc8fYi1u5uJmDE4jarlqUq4lqiKSGYoKWRYdUs3/3nfrhlXLU/V2t3N1HX0Zu18IlKYlBQyaCie4Ko7y4knPCvDRse6YpHaX4jIzCgpZNBv11axre4oRuaK1KZiW91Ryms7sn5eESkcSgoZUnqwnRvX7SMascCWn6bjysXloZ1bRPKfkkIGdA/E+NqScqIRC73C+GB7L2t2NYYag4jkLyWFDPjxygrqOvowjCzOLY/r29q2U0SmSUlhhlbtbOAvpXVEAqxanqqO3iFufbwm7DBEJA8pKcxAU2c/31m2I/Cq5em4ZtVu4jmSpEQkfygpTFMi4XzzL9voHoiFsvx0Mv1DCX6xek/YYYhInlFSmKbbNx3gsapWDLJStTwdN22soW9QzfJEJH1B7ry20MyaU7usHfvYN83MzSwv92fe29TFfz24O+tVy1MVTyR7MImIpCvIK4VbgQ8ce6eZnQW8DzgU4LkDMxhL8LUl5STcgdxNCMNWbjtMa9dA2GGISJ4ILCm4+wagfYyHfgV8m3z4RB3D9Wv2squhE8OI5cE8rgNfWbI17DBEJE9kdU7BzD4C1Lv7tjSeO9/MtpjZlpaWlixEN7knq9v4w6P7Q69anqpN+9uoauoKOwwRyQNZSwpmNhv4d+CH6Tzf3Re4+zx3n3faaacFG1wajvYN8fW7ypkVNRI5PI8wni/foWZ5IjK5bF4pvBI4B9hmZgeAOUCZmb0kizFM29UrdtJwtB/D8nLca09TN0/sbw07DBHJcVlLCu6+w91Pd/e57j4XqAPe7O4536hnRXk995QfJmrGQB4NGx3r63dOOmonIkUuyCWpi4FNwHlmVmdmnwvqXEGqP9LH9+/Zyaxobi8/TUdjZz/Ly+rCDkNEclhJUAd2909O8vjcoM6dKYmE8827ttE7GCeag1XL03H1igr+/oIzsVwswxaR0KmieQI3b6xhU3Vbqtldfl8lDOsaiPG7dfvDDkNEcpSSwji2HGjnF6uTVcth7KIWpBvWVjEUU/sLEXkuJYUxrNrZwKdvegrDIORNc4IwGE/wo3t3hR2GiOQgJYVjLNxYwxf/XEbCHfcEOdrrbsYWP32Izr6hsMMQkRyjpJASTzj/ce8u/uO+XUQjRizuDOXv6tNJJRy+fld52GGISI5RUgD6h+JcsaiMhY/XJBNCwvOyQG2qHq5spq6jN+wwRCSHjLsk1czS6bnc4u6XZDCerGvrHuDzt2+h7NCRnG+FHYQv37GVe664KOwwRCRHTFSnEAU+OMHjBqzMbDjZdaC1h8tveZq6jj6eF43kVZO7TCmvPcK22iO88axTwg5FRHLAREnh39z94EQvNrMvZTierCk92MG/3raZ7oEYUcuvrqeZ9tUlW3n0W+8JOwwRyQHjzim4+8bJXpzOc3LRqp2NfOqPT9KT2l+5mBMCwMG2Xtbsago7DBHJAZNONJvZRWa2xsz2mlm1mdWYWXU2ggtCcslpKQl3Eu45u79ytn1nqZrliUh6vY9uBq4CSoG8LYNNJJyf3F/JwsdrkhPK8eJYYZSu9t4hbnviAJe/fW7YoYhIiNJZknrU3R9092Z3bxu+BR5ZBvUPxflSES45naprHtxNvMiH0kSKXTpJYZ2ZXWtmf2Nmbx6+BR5ZhrT3DPKpPz7JqopGSiJGvMiWnE5F31Cca1fvCTsMEQlROsNHb0t9nTfqPgcuznw4maUlp1N308YavvH+85hVorpGkZkqPdjOdav3ctapx/HqM07k3DNO5NzTT+ClJ78gZ9vXT5oU3H1aaxXNbCFwKdDs7uen7rsW+DAwCOwHPuPuR6Zz/MmUHergc7dqyelUxRLOnVtquezCl4cdikje+/36/TxV08amY5bmvGBWhFe8+ARe+7ITk8ni9BN51ekncOYpxxGJhJssJqpovszd/2xmXx/rcXe/fpJj3wrcCNw+6r41wPfcPWZmPwe+B3xnaiFPbtXORq5cshV3Ty45jSkhTMWDOxuUFERmqKVrgHW7Wzj2Iz5qyV5ruxs72dXQ+azHnlcS4ZwXHc9rX3ZSKlmcwLlnnMCcF84mmqVkMdGVwvGprydO58DuvsHM5h5z30OjfnwS+KfpHHsitzxew3/cu4uSqOHuaNuAqdtedzTsEETy3oryeuLulEQNRu3JEneIj7FHS9TA3alq7mJPU9ezHpsVNV7+ouN5XSpZvOr0Ezj39BM4+9TZlEQzO9Q7blJw9z+kvv44o2d8xmeBO8d70MzmA/MBzj777EkPlkg4P32gkps3asnpTHX1x+gfivOCWdGwQxHJW8vK6kY+i9IxXrIYvkCobulmX3P3sx4riRhnnzr7OVcWL3/R8cyaZrKYaPhovrsvmOjF6TxnnNf9OxADFo33nNRxFwDMmzdvwn/V/qE4V91ZzoM7G0eWnMrMrNnVxIff+LKwwxDJSxWHj1LZ0MXzMrC5e8IhMUayMEveDrT1UN3aAzSMPBYxOOvU2cBzRq8mNdHw0XfNrHWCxw24ktQHd7rM7HKSE9CXuM98W7P2nsFkl9ODHUXZ5TQo929vUFIQmaZlpfWYEehWvu5jH9+AaMSobe+l5NQ55071uBMlhUdJrhSayJqpnMzMPkByYvld7j7jRv4H23q4fOHT1Hb0MUtLTjNq84H2sEMQyUtD8QT3lNdjQBifSM4zycIikXTKDp5lojmFz0w/LDCzxcC7gRebWR1wNcnVRs8H1qTW6D7p7l+YzvG3Hurgs1pyGpi2nkFi8UTGJ7FECt36PS209wzy/JIIA3m48nHKWSRd7v7JMe6+ORPHXl3RyJWLt5LQktNAbapu453nnhZ2GCJ5ZVlpcoI5HxMC5OF2nLc+XsMX/lRKXF1OA7eivD7sEETySkfPIA9XNpGY+XRpaAK7Usi0RML52QOV3KQlp1nzxL686nsoErqV2w4TSzizojbmiqF8MNGS1DErmYelUdGcMe5wxR1lWnKaZY2d/amq8Nzs0SKSa5aW1lISsUBXHQVtoiuFaVUyB6G6tZv2nY1acpplCYed9Z28fs7JYYcikvP2NHaxo76TWRmoTQjTRKuPgqpknrLewTinaclpKO7eWqekIJKGZWV1GGNXJeeTiYaPbpjohe7+1cyHMz4lhHA8urcl7BBEcl4snuDusmTBWr4PZkw0fFSatSgmkd8XY/ntYNuMawxFCt5jVa20dA/kbW3CaBMNH92WzUAkN8USzqH2Hs4+9fjJnyxSpJaW5XdtwmiT1imY2Wlmdp2ZPWBmjwzfshGc5IblZapXEBnP0d4hHqpozOvahNHSKV5bBFQC5wA/Bg4AmwOMSXLM2srmsEMQyVn3bj/MUNwpCXnHtExJJym8yN1vBobc/VF3/yxwYcBxSQ6pOmbDDxF5xtJUW4vBPF91NCydpDCU+tpgZh8yswuAOQHGJDmmP5agvWcg7DBEcs6+5m7Ka4+EHUZGpZMUfmJmJwPfAL4J3ARcFWhUknNWlB8OOwSRnDNcm1Ao8wkwQVIws5+nvj3O3Y+6+053f4+7v8XdV2YpPskRqysaww5BJKfEE87ysrqCqE0YbaIrhQ+a2SySeyBIkdtZfzTsEERyyhP7W2nqHJj2Xsi5aqLitVVAK3C8mXWOut8Ad/eTAo1Mckr3QJyegSGOf/6ssEMRyQlLS+uIFkhtwmjjpjh3/5a7nwzc7+4njbqdmE5CMLOFZtZsZjtH3Xeqma0xs6rU1xdm6H1IFjxU0RR2CCI5obN/iFU7G8nANvM5J53rnpVm9qppHPtW4APH3PddYK27nwusTf0seeKBHQ1hhyCSEx7Y3sBALFEwtQmjpZMUXg4sMLNqM7vLzL5iZm+a7EXuvgE4dvf3jwLD7TNuAz42hVglZFsOdoQdgkhOWFZWWLUJo02aFNz9h+5+MfA6YCPwLabfLO8Md29IHbcBOH28J5rZfDPbYmZb4r2a5MwFHb1DDKlbrRS5A609bD5QuH8gpdP76Ptm9iDwEPAqkrUKgRevufsCd5/n7vOis9XPP1c8vq817BBEQrW8AGsTRktn+OgfgBcBDwPLgZXDf+1PQ5OZvRQg9VVNdfLMynI1x5PilUg4S8vqiJgVVG3CaOkMH70ZuAR4GngfsMPMNk7zfCuBy1PfXw6smOZxJCRPVB87TSRSPJ6saePwkX5KomFHEpyJ6hQAMLPzgXcC7wLmAbXAY2m8bjHwbuDFZlYHXA1cA9xlZp8DDgH/PO3IJRRNnf0kEk6kAFddiEzmmdqEAr1MII2kAPwc2ADcAGx296FJng+Au39ynIcuSTM2yUHusL3uCG86WyUmUlx6BmI8uKMwaxNGS2f46EPAr4BO4LxU6wspYvdoXkGK0AM7GugbilMSKay2FsdKZ/joXcDtJDfXMeAsM7s8VYcgRejRvVqBJMXnmdqEwl6Wnc7w0fXA+919D4CZvRpYDLwlyMAkdx1q7w07BJGsqm3v5cnq9oKsYD5WOtdBs4YTAoC77wU0hFTE4gmnuqU77DBEsmZ4n/JCn0+A9JLCFjO72czenbr9kelXNEuBuHtrXdghiGSFu7O0rJZoxCjArhbPkU5S+CJQAXwVuBLYBXwhyKAk9z1S2RJ2CCJZsflAB7XtfRTYtgnjmnROwd0HSM4rXB98OJIvqjR8JEViaWktUTMGC7g2YbR0eh9dlNr7YG+qU2q1mVVnIzjJXYOxBC1dA2GHIRKo3sEY929vwCmOhADpDR/dTPIq4R3AW0fdpMitUL2CFLjVFY30DBZ+bcJo6bzTo+7+oLs3u3vb8C3wyCTnPVTRGHYIIoFaVlpfFLUJo6VTp7DOzK4l2SF1ZLzA3csCi0ryQsXhzsmfJJKnDh/p4/F9rRRBacKzpJMU3pb6Om/UfQ5cnPlwJJ/0DMbp7h/ihBeobEUKz91b63HADIpoSiGt1UfvyUYgkp9WVTTxT28JfM8lkaxy95GOqLFC3ThhHOPOKZjZh83s5aN+/qGZbTOzlWY2NyvRSc57YMd091sSyV1lh45Q09pTNLUJo030ln8KtACY2aXAZcBnSW6U84fgQ5N8UHawcPeqleK1rKyOiFE0tQmjTZQU3N2HO5/9A3Czu5e6+03AaTM5qZldZWYVZrbTzBab2QtmcjwJz5G+IQZjxbMyQwpf/1Cce8sPhx1GaCZKCmZmJ5hZhOTGOGtHPTbtD3EzO5Nky4x57n4+EAU+Md3jSfg2VKnlhRSONbua6BqIMasYx46YOCn8GigHtgCV7r4FwMwuAGY6kFwCHGdmJcBsoHjTcgFYqSI2KSBLS5P7JgwU6RXwuKuP3H2hma0GTge2jXqoEfjMdE/o7vVmdh3JPZr7gIfc/aFjn2dm84H5ACUnzWi0SgL2ZE172CGIZERTZz+PVbVQZKUJzzLh9ZG717v7VndPjLqvwd0PTfeEZvZC4KPAOcDLgOPN7LIxzr3A3ee5+7zo7JOnezrJgpbOARJFtmxPCtPdW+tJeKo2oUiFMWj2XqDG3VvcfYhkpfTbQ4hDMsSBskNahST5zd1Zlho6KtKRIyCcpHAIuNDMZpuZkZzErgwhDskgNceTfLe97ihVzd1F19biWGklBTN7h5l9JvX9aWZ2znRP6O5PAUuBMmBHKoYF0z2e5IYNe1vDDkFkRkZqE4phe7UJTNrmwsyuJtn36DzgFpL7M/8ZuGi6J3X3q4Grp/t6yT11R/pwd6yYB2Mlbw3E4tyjq10gvSuFvwc+AvQAuPth4MQgg5L8E084+7Ubm+SpRyqb6ewr3tqE0dL5Fxh0dyfVJ9DMjg82JMlXy8v0l5bkp2KvTRgtnaRwl5n9ATjFzD4PPAz8MdiwJB89srs57BBEpqyla4D1e1pI/u0r6bTOvs7M3gd0kpxX+KG7rwk8Msk71Ro+kjy0oryeuDuzoka8yCeZIb1NdkglASUCmdBg3Gns7OMlJx0XdigiaRseOhpSQgDSGD4ysy4z6zzmVmtmd5vZK7IRpOSPFVvVxkryR8Xho+xu7CrqCuZjpTOncD3wLeBMYA7wTZJzCkuAhcGFJvnooV2NYYcgkralpcnahJiuEkakkxQ+4O5/cPcud+909wXAB939TuCFAccneaayoSvsEETSMhhLcM/W5Io5pYRnpJMUEmb2cTOLpG4fH/WY/i3lWXoH43T2DYYdhsik1u9ppqN3SLUJx0jnX+PTwL8AzUBT6vvLzOw44MsBxiZ56oGdGkKS3LesTLUJY0lnSWo18OFxHt6Y2XCkEDy4o4FPvPXssMMQGVd7zyBrK5tJqDbhOdLpffQC4HPA6xi1Dae7fzbAuCSPldceCTsEkQmtLK8nlkjWJiQ0yfws6Qwf/Ql4CfB3wKMkVyBpNlHGdbQvxkAsHnYYIuNSbcL40kkKr3L3HwA97n4b8CHg9cGGJfluvVpeSI7a3djJzsOdqk0YRzpJYSj19YiZnQ+cDMwNLCIpCPdubwg7BJExLSutw1SbMK502lwsSO2r/H1gJXAC8INAo5K891RNe9ghiDxHLJ7g7q31GKA1R2ObMCmYWQTodPcOYAOQkbYWZnYKcBNwPslah8+6+6ZMHFtyQ2vXAPGEEy32vQ0lp2yoaqG1e5Dnl0S0FHUcEw4fuXuCYGoRfgOscvfXAG9EezQXHAe2HNDVguSWZaX1qk2YRDpzCmvM7JtmdpaZnTp8m+4Jzewk4G+BmwHcfdDdj0z3eJK7VpSrOZ7kjiO9gzy0q1G1CZNIZ05huB7hilH3OdMfSnoF0ALcYmZvBEqBK929Z/STzGw+MB+g5KTTpnkqCdNjVS1hhyAy4t7tDQzFneeVGIMxJYbxTHql4O7njHGbydxCCfBm4PfufgHJvZ+/O8Z5F7j7PHefF5198gxOJ2GpP9Kn3awkZyzdUktJRAlhMunspzDbzL5vZgtSP59rZpfO4Jx1QJ27P5X6eSnJJCEFJuGwt0l1jhK+fc1dbKs7qtqENKQzp3ALMAi8PfVzHfCT6Z7Q3RuBWjM7L3XXJcCu6R5PctvysvqwQxBhaWlyGWo8oauEyaSTFF7p7r8gVcTm7n3ATPPtV4BFZrYdeBPwsxkeT3LU+j2qbJZwdQ/E+MuWWsySV68ysXQmmgdTbbIdwMxeCQzM5KTuXg7Mm8kxJD9Ut/ZM/iSRAP3h0f209Qzy/GiEgbiWok4mnSuFHwGrgLPMbBGwFvh2kEFJ4RiKO4eP9oYdhhSpw0f6WLChmmjElBDSlM7qo4eAfwD+L7AYmOfu64MNSwrJPWWqV5BwXLd6D0PxBCqsT186q49WAu8H1rv7fe7eGnxYUkjW7GoKOwQpQtvrjrA81edILbLTl87w0S+BdwK7zOwvZvZPqY13RNKyu1HLUiW73J2f3FfJrKihUpmpSWf46FF3/xLJSuQFwMdJ7tcskpa+oThH+wbDDkOKyOqKJp4+0A6ubqhTlc6VAqnVR/8IfAF4K3BbkEFJ4blP+ytIlgzGEvzsgeRVwpDWoE5ZOnMKd5LsYnox8DuSdQtfCTowKSyrdjaGHYIUids3HeBQey8RlS9PSzp1CrcAn3L3OICZXWRmn3L3KyZ5nciIbbVHwg5BikBHzyA3rK1Se+wZSGdOYRXwejP7uZkdINniYnfQgUlh6eyP0T8UDzsMKXA3PFJFV3+MaFoD4zKWcf/pzOzVZvZDM6sEbiTZ88jc/T3u/tusRSgF45FKrU+Q4FS3dHP7poNEIjCgTqjTNlE+3U2yWd2H3f0dqUSgP/Vk2u7boSI2Cc5/PZgcwDDlgxmZKCn8I9AIrDOzP5rZJcy8EZ4UsadrtD2nBGPT/jbW7GrCAF0kzMy4ScHd73b3/w28BlgPXAWcYWa/N7P3Zyk+KSBt3YNqXSwZl0g4P7l/F7OiRky/XzOWzkRzj7svcvdLgTlAOWPslCYyGQeeqm4LOwwpMMu31lNxuFPDGBkypTl6d2939z+4+8VBBSSFbUW55hUkc3oHY1y7ajezIsag+htlRGgLt8wsamZbzey+sGKQ7Nu4T/0UJXP+uKGGpq4BImqDmjFhrua9kmSltBSRhqN9uDqUSQY0dfbz+0f3JfdKUKFaxoSSFMxsDvAh4KYwzi/hSTjsaugMOwwpANet3sNgLEFUFwkZFdaVwq9J7t6m9F6E7tlaH3YIkucqDh9laWkdBppLyLCsJwUzuxRodvfSSZ4338y2mNmWeO/RLEUn2bBuT0vYIUgec3d+en8l0YihdJB5YVwpXAR8JNVHaQlwsZn9+dgnufsCd5/n7vOis0/OdowSoANtPWGHIHlsbWUzT+xvw0gOR0pmZT0puPv33H2Ou88FPgE84u6XZTsOCU8s7tR19IYdhuShoXiCn2qvhECpl6CEYnmZ5hVk6hY9eZCa1h7tlRCgUJOCu69PVUpLkVlb2RR2CJJnjvYO8WvtlRA4XSlIKPY0dYUdguSZG9dVcaR3iIg+tQKlf14JRf9Qgo7ewbDDkDxxsK2HWx4/QNSMQbVBDZSSgoTm3m3qgyTp+fmq3Thg2iwhcEoKEppVOxvDDkHywOYD7TywozG5V4KmEgKnpCCh2V6nokSZWCLh/Od92ishm5QUJDTdAzH6BmNhhyE5bOW2w/rjIcuUFCRUa3ZpaaqMrX8ozs9X7U4Wqqm/UdYoKUio7t/eEHYIkqNu3lhDw9F+FaplmZKChGrLwY6wQ5Ac1NzVz+/Waa+EMCgpSKjaegaJxfU/vTzbr9bspW8ojjZUyz4lBQndE/u1Rac8Y3djJ3duriViaC4hBEoKErqV5Spik6ThvRIiZmjX1nAoKUjontjfFnYIkiPW723hsapW7ZUQIiUFCV1jZz+uPwuLXiye4Gf3a6+EsCkpSOgSDjvqVaBU7JZsrqWquRvTEtRQKSlITrh7qzbdKWad/UNcv2YvJRFjUEtQQ5X1pGBmZ5nZOjOrNLMKM7sy2zFI7tmwpyXsECRE/71uP+09g0S1BjV0JSGcMwZ8w93LzOxEoNTM1rj7rhBikRxxsF17Nher2vZeFm6sIWoqVMsFWb9ScPcGdy9Lfd8FVAJnZjsOyS2xhHOgtSfsMCQEv1i9h3jCiWivhJwQ6pyCmc0FLgCeGuOx+Wa2xcy2xHs1CVkM7inXvEKxKT3YkdpsyRnSRUJOCC0pmNkJwDLga+7eeezj7r7A3ee5+7zo7JOzH6Bk3cKNNRxo7Q47DMkSd+cn9yf3StAK1NwRSlIws1kkE8Iid18eRgySezr7Y7z3+g3cvbUu7FAkC+7f0cDWQ0cAUE7IHWGsPjLgZqDS3a/P9vklt8UTzlV3buPKJVsZiMXDDkcC0j8U55oHtVdCLgrjSuEi4F+Ai82sPHX7YAhxSA4a/nhYUX6YS375KIfatCqpEN36xAHqOvq0V0IOCmP10UZ3N3d/g7u/KXV7INtxSO6r6+jj4l+uZ6UmoAtKW/cANz6ivRJylSqaJafFE85Xl5Rz1Z3lqnQtEL9+uIqegRhRffrkJP1nkZw2PJx099Z6LvnlempV5JbXqpq6uOOpQ0QMBmOaS8hFSgqSN2pTw0n3bdP+C/nqZw9UommE3KakIHklFne+vHgr37hLw0n5pLa9l+8s3c66VI8rLTjKXWH0PhKZtuHPkmVl9Ww+0MEdn38bc144O9SYZHwH23r43bp9LCurx92JWrKlieQuJQXJW4fae3nPdeu54ZMX8L/Of2nY4cgo1S3d/G7dfu7ZWo/jGBAx0MVd7lNSkLwWiztf/HMZH583h5/+/euZpSUtodrX3M2Nj1SxctthDAOcCBBzVLacJ5QUJK8Nf87ctaWOp2vauePzF/KyU44LNaZitLepi98+so/7th0mEkmmA3DNHeQhJQUpGAfaenn3tev47afezN+97iVhh1MUdjd28tu1+3hgRwORiBGxZKM7TRvkLyUFKShDceff/lTKJ956Fv/5sfM1nBSQisNH+e3afayqaCSqZFBQlBSkoAx/Ji3ZXMvTNe0s+vzbeOnJGk7KlB11R/nN2ioermyiJGJElQwKjpKCFKzq1h7ede16fvepC3jfazWcNBPltUe4YW0Vj+xuTiaDiJFIOFpMVHiUFKSgDcUSfP72Uj79trP58UdeR4mGk6ak9GAHN6yt4tG9LZREjJKIEU+4FhIVMCUFKWjDH16LnjrEk9VtLPrXC3nJyS8INaZ8sPlAO795uIqN+1qVDIqMkoIUjf0tPbzr2nX896ffzCV/dUbY4eSkJ6vb+M3DVWyqbqMkmkwGqkAuLkoKUlQGYwk+d9sW/s/fvJwfXvpaDSeRnCjetL+NX6+t4umadmYNJwMVGRSlUJKCmX0A+A0QBW5y92vCiEOKz/DH3O2bDrJpfxuL/vVtnH5ScQ4nuTuPVbVyw9oqthzsYFbUmBXR9pjFLutJwcyiwO+A9wF1wGYzW+nuu7IdixS3quZu3vmLdXz/Q3/Fi054PhFLrqopiRiRiBFN/Zy8QTQSIWpGJAIlkQjRCETMKIlEiERIPm/UayLDxxp1XMuBvtHuzvq9Ldywtoqth44oGcizhHGl8NfAPnevBjCzJcBHgXGTgn5VJSgDsQQ/WFGR1XNGLJlMImYQQo5w92clACUDGS2MpHAmUDvq5zrgbcc+yczmA/MBrOR5icMLvzyUnfCekejrikaOOzGe7fOGqSjf80DP86OzTw47jKyK9x4lOvtkPB4bpEj+7irG3+2hjsNT/owPIymM9bfRc34p3X0BsCD4cMZnZltiXa3zwowh24r2PR9t1nsucMX6uz3V14Sx9KIOOGvUz3MA7a8oIpIDwkgKm4FzzewcM3se8AlgZQhxiIjIMbI+fOTuMTP7MrCa5JLUhe6e3Zm+9IU6fBUSvefioPdcHKb8ns29KOaYREQkDSrnFBGREUoKIiIyQklhDGZ2lpmtM7NKM6swsyvDjilbzCxqZlvN7L6wY8kGMzvFzJaa2e7Uf++/CTumIJnZVanf6Z1mttjMCrLHh5ktNLNmM9s56r5TzWyNmVWlvr4wzBgzaZz3e23q93q7md1tZqekcywlhbHFgG+4+18BFwJXmNlrQ44pW64EKsMOIot+A6xy99cAb6SA37uZnQl8FZjn7ueTXOjxiXCjCsytwAeOue+7wFp3PxdYm/q5UNzKc9/vGuB8d38DsBf4XjoHUlIYg7s3uHtZ6vsukh8UZ4YbVfDMbA7wIeCmsGPJBjM7Cfhb4GYAdx909yOhBhW8EuA4MysBZlOgNULuvgFoP+bujwK3pb6/DfhYNmMK0ljv190fcvdY6scnSdaETUpJYRJmNhe4AHgq5FCy4dfAt6Fodll8BdAC3JIaMrvJzI4PO6iguHs9cB1wCGgAjrr7Q+FGlVVnuHsDJP/wA04POZ5s+izwYDpPVFKYgJmdACwDvubunWHHEyQzuxRodvfSsGPJohLgzcDv3f0CoIfCGlJ4ltQY+keBc4CXAceb2WXhRiVBM7N/Jzkkviid5yspjMPMZpFMCIvcfXnY8WTBRcBHzOwAsAS42Mz+HG5IgasD6tx9+CpwKckkUajeC9S4e4u7DwHLgbeHHFM2NZnZSwFSX5tDjidwZnY5cCnwaU+zKE1JYQyWbHp/M1Dp7teHHU82uPv33H2Ou88lOfn4iLsX9F+R7t4I1JrZeam7LmGCFu4F4BBwoZnNTv2OX0IBT6yPYSVweer7y4EVIcYSuNRmZt8BPuLuvem+TklhbBcB/0Lyr+Xy1O2DYQclgfgKsMjMtgNvAn4WbjjBSV0RLQXKgB0k//8vyNYPZrYY2AScZ2Z1ZvY54BrgfWZWRXKTr4LZ8XGc93sjcCKwJvUZ9v/TOpbaXIiIyDBdKYiIyAglBRERGaGkICIiI5QURERkhJKCiIiMUFIQCUCq++qXRv387mLpPCv5TUlBJBinAF+a7EkiuUZJQYqemc1N9Z2/KbXPwCIze6+ZPZ7qvf/XqV7896R60z9pZm9IvfZHqV72682s2sy+mjrsNcArU0VD16buO2HU3g2LUlXFIjmlJOwARHLEq4B/BuYDm4FPAe8APgL8P6AW2OruHzOzi4HbSVZAA7wGeA/J6tE9ZvZ7ko31znf3N0Fy+Ihkt93XkWxX/TjJyvmNgb8zkSnQlYJIUo2773D3BFBBcjMWJ9kOYi7JBPEnAHd/BHiRmZ2ceu397j7g7q0km6ydMc45nnb3utQ5ylPHFckpSgoiSQOjvk+M+jlB8op6rKGe4R4xo18bZ/wr8HSfJxIaJQWR9GwAPg0jQ0Gtk+yx0UVyOEkkr+gvFZH0/IjkDm3bgV6eacE8JndvS01U7yS549X9wYcoMnPqkioiIiM0fCQiIiOUFEREZISSgoiIjFBSEBGREUoKIiIyQklBRERGKCmIiMiI/wEPDg2/Of5fmQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "gpu_snow.to_pandas().sort_index().plot.area(ylabel = \"Average Snowfall [in]\", xlim = (1,12))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "39b4eefc-4a4e-4e3b-9cdc-4c8da18eb3a9",
   "metadata": {},
   "outputs": [],
   "source": [
    "del cpu_df, gpu_df, avg_snow, gpu_snow"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bdf4e65e-606c-4e8d-b157-2512f6a0a6c4",
   "metadata": {},
   "source": [
    "#### **cuDF** benefits appear at scale\n",
    "\n",
    "Last time, we saw that **CuPy** only sped up calculations compared to **Numpy** at a certain problem size. We can show the same phenomenon with *DataFrames*.\n",
    "\n",
    "Here, we use a big [Earth Surface Temperature dataset](https://www.kaggle.com/datasets/berkeleyearth/climate-change-earth-surface-temperature-data) provided at Kaggle from Berkeley Earth."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "id": "3bc18cbc-2fb6-4507-9916-0a87438b1f61",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "-rw-rw-r-- 1 vanderwb csgteam 509M Sep 19  2019 data/GlobalLandTemperaturesByCity.csv\n"
     ]
    }
   ],
   "source": [
    "!ls -lh data/GlobalLandTemperaturesByCity.csv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "id": "0e43eef8-4daa-459f-aadb-49ad8cec2892",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Mem used: 2.25 GiB\n"
     ]
    }
   ],
   "source": [
    "cpu_big = pd.read_csv('data/GlobalLandTemperaturesByCity.csv',\n",
    "                      infer_datetime_format = True, parse_dates = [0])\n",
    "print(f'Mem used: {cpu_big.memory_usage(deep = True).sum() / 1024**3:.2f} GiB')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "id": "0f0115fa-ef96-411c-b2bb-5270cae72ad2",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 944 ms, sys: 49.3 ms, total: 994 ms\n",
      "Wall time: 988 ms\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "gpu_big = cd.DataFrame.from_pandas(cpu_big)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2deb65a4-08cd-495c-92ea-eadec027796a",
   "metadata": {},
   "source": [
    "**Compute global mean August temperature timeseries using the CPU and GPU ...**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "id": "2d42309b-465c-44a0-ad83-062501664fbb",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 1.25 s, sys: 46.9 ms, total: 1.29 s\n",
      "Wall time: 1.26 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "global_mean = cpu_big[(cpu_big.dt.dt.month == 8) &\n",
    "                      (cpu_big.dt.dt.year >= 1900)].groupby('dt')['AverageTemperature'].mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "id": "82d63952-b310-44dc-8d9d-5e1118fdfa17",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 8.34 ms, sys: 4.66 ms, total: 13 ms\n",
      "Wall time: 12 ms\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "global_mean = gpu_big[(gpu_big.dt.dt.month == 8) &\n",
    "                      (gpu_big.dt.dt.year >= 1900)].groupby('dt')['AverageTemperature'].mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "id": "b8024f7b-6726-49fa-9f4e-613a8ce5e33a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<AxesSubplot:xlabel='dt'>"
      ]
     },
     "execution_count": 71,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYAAAAEKCAYAAAAb7IIBAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8qNh9FAAAACXBIWXMAAAsTAAALEwEAmpwYAABVt0lEQVR4nO29eXhcZ33o//nOppnRLlmytXiJlzh2YsdJTBZCAgkNhQBhCS1tuYGWe5vbNmmhFy6lpaX0cukPSgu3t6X0hq20DVBokrIFSKCBJCxJHMdxvCWOHW+SrF0azb69vz/OMmdGM5ozsmR5pPfzPH48c855Z945mnm/73cXpRQajUajWXl4lnoCGo1Go1katADQaDSaFYoWABqNRrNC0QJAo9FoVihaAGg0Gs0KRQsAjUajWaFUFQAislZEHhGRwyJyUETeU3L+/SKiRGRVLWNF5CMiMiAi+8x/ty7MR9JoNBqNG3wurskC71NK7RWRZuBpEXlYKXVIRNYCtwCnah1rnv+0Uuqvz/lTaDQajaZmqmoASqkhpdRe8/EMcBjoM09/GvgAUDabrMpYjUaj0SwhbjQAGxHZAFwBPCEitwEDSqlnRaSmsY7Dd4vIO4E9GJrC5FyvsWrVKrVhw4ZapqzRaDQrnqeffnpMKdVVelzcloIQkSbgJ8DHgO8DjwCvUUpNi8gJYLdSaqzaWKXU/eax1cAYhvbwUaBHKfXuMmPvBO4EWLdu3VUnT550NV+NRqPRGIjI00qp3aXHXUUBiYgfuA+411zANwEXAc+ai38/sFdE1rgYC4BSalgplVNK5YHPAVeXe2+l1D1Kqd1Kqd1dXbMEmEaj0WjmSVUTkBj2nS8Ah5VSnwJQSj0HdDuuOUEZDaDcWMe5HqXUkPn0LcCBc/gcGo1Go6kRNxrA9cAdwM1uQjZFpFdEHnQx9q9E5DkR2Q/cBPzhOXwOjUaj0dRIVQ1AKfU4MKeXVym1wfF4ELi12lil1B21TFSj0Wg0C4vOBNZoNJoVihYAGo1Gs0LRAkCj0WhWKFoAaDQaTZ3y2NFRjo9GZx1//OgYw5Fk1fFaAGg0Gk2d8p6v7ePvH3mx6Fg+r3j3l5/inkePVx2vBYBGo9HUIbFUlolYmrFouuh4JJkhnc0zMJmo+hpaAGg0Gk0dMjRtLPATsVTR8fGYIRCGtAlIo9FolicDU8YCP1GiAUxYAmBKawAajUazLLFMPOOxNM6inuOmQBiNpkhn83O+hhYAGo1GU4cMmjv8VDZPPJ2zj1sagFIwMjO3GUgLAI1Go6lDBh0mHmvRNx4XfAJD01oAaDQazbJjwCEAxh0CwBkVpAWARqPRLEMGpxOs7wwDxbv+iViazsYAUN0RrAWARqPR1Bm5vOLsdJIdfa1AwfELhgBY1xmmucGnNQCNRqNZboxFU2Ryip39hgBw+gDGTQ1gTWvQzhWohBYAGo1GU2ecMUNAN3c30eDzFPkAJmIpOhoD9LSFqmoAVRvCaDQajebCwooA6msL09kYsE1ASikmYmk6GhsQhMNDkTlfRwsAjUajqTMsAdDbFqSjKWA7gWdSWTI5RWdjgKDfw1iVZDAtADQajabOGJxK0Bz00Rz009HYYPsArLIQHY0BWkI+lGLOstBVfQAislZEHhGRwyJyUETeU3L+/SKiRGRVhfGvFZHnReRFEfmg43iHiDwsIkfN/9tdfXKNRqNZ4QxMJelrCwEYJiBTAFj/dzQF6Gk1zs/lB3DjBM4C71NKbQOuBe4Ske1gCAfgFuBUuYEi4gU+A7wO2A78ujUW+CDwI6XUFuBH5nONRqPRVGFwKkGvKQA6GgO2BjAeNUxBnY0BelqDAHNGAlUVAEqpIaXUXvPxDHAY6DNPfxr4AKAqDL8aeFEpdVwplQa+BrzJPPcm4Mvm4y8Db642F41Go6lX/voHz/P2//fzBXmtwemErQF0NAaIp3MkMzlbEHQ2NdDTtjAagI2IbACuAJ4QkduAAaXUs3MM6QNOO56foSA8ViulhsAQMkB3hfe8U0T2iMie0dHRWqar0Wg0Fwx7Tk5wqEpUjhtiqSxT8YytAVhZv+OxtG0C6mwM0NTgoznomzMb2LUAEJEm4D7gvRhmoQ8BH642rMyxStpCWZRS9yildiuldnd1ddUyVKPRaBaUfF4VJV3VwumJBDPJLNnc3CWaq+GMAAJDAwDDATwRSxMOeAn6vQD0tAbPXQMQET/G4n+vUup+YBNwEfCsiJwA+oG9IrKmZOgZYK3jeT8waD4eFpEe8/V7gBE3c9FoNJql4mtPneaGT/wnCUf5ZTeks3kGTVv8VCJzTnMYsHMATA2gydIAUmYOQMC+tqd17mQwN1FAAnwBOKyU+hSAUuo5pVS3UmqDUmoDxkJ/pVLqbMnwp4AtInKRiASAXwO+ZZ77FvAu8/G7gG9Wm4tGo9EsJU+8NE4snWO8pA1jNQamElg9W6bi7jSIJ1+aYN/pqVnHB81OYAUTUANglIMYdxSCg4XRAK4H7gBuFpF95r9bK10sIr0i8iCAUioL3A38AMN5/HWl1EHz0o8Dt4jIUYxIoo+7mItGo9EsGQcGpgGYite2iz89EbcfT7oc+ycPPMdffvfwrOODUwm8HqG72Vj4O0wNYCKWtstAWPS0hhiLVhZWVRPBlFKPU96W77xmg+PxIHCr4/mDwINlxowDr672/hqNRnMhEEtlOT4WA2B6DjPOC8MzPHdmmtuv6rePnXIKABc+hGQmx0tjMRIts01Ng1MJ1rQE8XmN/Xtzgw+/VxiPpZmIptm6usW+1goFrYQuBqfRaDQuODwUcZhxKguAzz16nP/578+SzBQWb6cG4EZ7eHEkapR8jiRnOY3PTBVCQAFEhI7GAOPRlGECanJoAG1aAGg0Gs05Y5l/AKYSlXfxx0aj5BU8f3bGPnZqIs6aFmMxnnThA7DGWkLAycBkwo4AsuhobOD0RIJUNj/LBzAXWgBoNBqNCw4MRmgN+YHKu3ilFMdGDTORM+b/1EScrWua8XvFlQ/gheGC8LCcvmBEEw1NJ1jXES66vrMxwNGRKECRD2BNa4i50AJAo9FoXHBgYJor1rUR9Hsq+gAmYmn7nFWKWSnFqfE46zvDtIUDrqKAjpydoanBcNEOTBXMR2cm4+QVrOtsLLq+ozFgO3udJqCmBh+/cc26iu+jBYBGo9FgLNRHHTtvJ8lMjqMjUXb0tdIWqryIW05in0c4NGgIgOlEhplUlnUdYdrDftcmoFdsNuprOjWAk6YvweoFbOHc9XeYYaEWf/mWHRXfRwsAjUajAf7zyAi3fPrRIoetxZGzM+Tyikt7W2kN+SuagI6ZZpgbtqzi8FCEfF5xesJI3FrbYWgA1UxA0/EMZyNJdq1ro6MxYHf/Ajg1bgqAMiagco+roQWARqPRACfNxbVc3LzlAL6sr4XWsL9iNu/xsRgBn4dbtq8hls5xaiJuh4BaGkA1E9CRs4bmsHVNM31tIbv0gzXHkN9LV3PxLr+jyakBaAGg0Wg0NWEt/PEyZR4ODk7TFvbT1xaiLeRnusIu/vholIs6G7msz4jFPzwUsQXA2o4w7S40gOdNM9Qla5rpbQvapR8ATk3EWN8ZxijQUMDa9Tf4PIQDXjcfF9ACQKPRaAAYnaksAA4MRListxURoS3sr+gEPjYaY1N3IxevbsbrEQ6ZAsCqzmk5gZWqXBPz+bMztAR9rGkJ0tcWZnAqYV9/cjw+KwIICnb/zsbALOEwF1oAaDQaDU4NIFt0PJ3N8/zZGS41d/Vt4UDZPIB0Ns+piTgbVzUR9HvZ1NXIocEIpyfirDUX7fawn0xOEU1lZ423eP7sDFvXNCMi9LYFiadzTMUz5POKUxPxWQ5gKET+OE1BbtACQKPRaIAxs59uqQZwdGSGdC7PZb2tALSG/CQz+aJMXzDMM7m8YlO3EaK5rafF1gDW2QLAWKDnyiN4ftgQAAD97UYc/8BUgpGZFKlsflYIKBRMQJ0lEUDV0AJAo9FoKJiAYiW7cyuc89JeSwMwksFKzUBWAtjGVU0AbO9pYWg6yZnJOGs7jIW83VyoK4WCDk4nmUlm2brGeC+r4ufAVIKT48brl0YAAbQE/Xg9UlMEEGgBoNFoNOTzyi7xXFrr31qsrcibtlD5XfyxUSMEdGOXsUPfbgqMvMKhAfjN1yyvATxvRgBdYmoAVs2fgclExRwAAI9HuGlrN9du7HTxaQtUrQaq0Wg0y53pRIZMznC0xkoEQCxlPG8MGMulpQGUhnMeH43R3dxAc9A4v62nUJXT8gG02Sag8hrAEbMG0MWrDQHQ0Rgg6PcwOJUg6Pfi9YitFZTy+XftdvNRi9ACQKPRrHicsf+JEidwLJUlHPDi8RjRNXY9oBIT0PHRKJu6muznq5oa6G5uYGQmNVsDqFAS+sjQDD2tQfs9DEdwiIGpBD6vh762EH7vwhlutAlIo9GseEYdAmCWBpDOEg4U9sq2D8BhxrGKwFnmH4vtvS34PEKPWZTNWtjLmYCeOzPN9w4Mcb1ZAsLCSgY7NR4ra/45F7QGoNFoVjyWA9jrkVlhoNFUjqaGQnKVbcZxhIJaReCcGgDAr71sLVu6m/Ca2oPP66El6JtlAoqmsvz+V/eyqqmBP339tqJzfW0hDg9FyOYVr9/Rc46ftBgtADQazYrHCgHtbw/NCgONp7I0NhSWysaAYYt3OoHtCKASDeC1l/Xw2suKF+32xtnZwH/+zYOcmojzld++1hYwFn1tIXt+C60BuGkKv1ZEHhGRwyJyUETeYx7/qIjsN3sEPyQivWXGbnX0Ed4nIhERea957iMiMuCmz7BGo9EsJqMzKfxeYU1LkHiqWABESwSAiNAWKq4HdNyMACrVAMphFIQraADfenaQ+/ae4e6bt5SN4nE6fdd1zM4BOBfc+ACywPuUUtuAa4G7RGQ78Eml1E6l1C7gO8CHSwcqpZ5XSu0yr7kKiAMPOC75tHXe7B2s0Wg0552xaIpVTQ00NviIZ0qcwOksjSX1dVrDxfWAjo1GafB5ilo1VsIoCFcY+6WfvsQla5r5g5s3l72+r73wmuddA1BKDSml9pqPZ4DDQJ9SKuK4rBGoXNzC4NXAMaXUyflOVqPRaBaD0RlDAIQD3lkaQCyVK9IAAKMgnEMDeGksxkWrGu1Ioblod2gAmVyeg4MRXrF5ld3kvZS+Ig3gPAsAJyKyAbgCeMJ8/jEROQ28gzIaQAm/Bny15NjdphnpiyLSXstcNBqNZqEYi6boajYFwKw8gKzdncuitB7Q8TIRQJVoc2gAz5+dIZ3Ns3NtW8Xr17QGEcHWUBYS1wJARJqA+4D3Wrt/pdSHlFJrgXuBu+cYGwBuA77hOPxZYBOwCxgC/qbC2DtFZI+I7BkdHXU7XY1Go3GNYQIKEA74iJXJAyinAViLeCZnFIG7aJU7AdAeDhBNZUln8zx7ZgqAXf1tFa/3ez2sbg4uuPkHXAoAEfFjLP73KqXuL3PJV4Db53iJ1wF7lVLD1gGl1LBSKqeUygOfA64uN1ApdY9SardSandXV5eb6Wo0Go1r8nnFWDRtawCJdM4uv5zPK2Lp3Jw+gFMTcbJ5ZdcAqoaVDDaVSLP/9DTtYb9dK6gSd1y3nl/d3V/rR6tKVX1CjOLSXwAOK6U+5Ti+RSl11Hx6G3Bkjpf5dUrMPyLSo5QaMp++BThQy8Q1Go1mIZhKZMjlFauaGoinc2TzinQuT4PPS9ys+DlbAwgwk8qSyeV5qUIIaCXaHBVBnz0zxY7+tqo1/O+6qbyD+FxxY1C6HrgDeE5E9pnH/gT4ryKyFcgDJ4HfATDDQT+vlLrVfB4GbgH+e8nr/pWI7MJwHp8oc16j0WgWHSsJbFVTg/04kc4ZAsCsDDpLAJi7+Egiw/Exswicaw3AEACDUwmOjkS5Zfvqc/8Q86SqAFBKPQ6UE09lwzaVUoPArY7ncWBWcKtS6g7309RoNJrFwaoD1NXcYGcBx9I52sLYjVtmO4EL9YCOj8bobAzQah6rhjX2py+Okcsrds5h/19sdC0gjUazorEEgBEGaiz0VkE4qxJoaZ9duyBcPFNTBBAUegL85AUjqOXy/tZzmP25oQWARqNZ0VhmH8sJDIWFv7IGYCzi04k0x8eirs0/UHACvzAcpac1SHdL8Nw+wDmgBYBGo1nRjEZTBMwibZYGYIWCWiahUh+ApQGcGo8zFk3XpAGE/F4CPmPp3bmEu3/QAkCj0axwjCzgACJiawBWV7BoJSewKQD2nZ4CYKOLGkAWImJrAUtp/wctADQazQrHygEAaDTLPls9AexuYA3FPoAWUwDsPTUF4DoJzMKKBNo1Rwbw+UALAI1Gs6x44vg4mVze9fVjZh0ggNAsJ3B5DcDrEVqCPk5NxPF6pOYaPVYk0GV92gSk0Wg0C8KJsRhvv+cXPPjcUPWLTUbNOkCAnfFr7fwtX0BjYHbEvOUIXtcRtm36blnf0cj2nhbbl7BU6IYwGo1m2XBmMgEY1TndkMsrJmJphwZg+gAylgkoS8hsxl5KW9jPqQnYWKP5B+DDb9xOOuteS1kstADQaDTLhuFIEoABUxBUYzKeNstAGLv5gNeDzyO26Seays2y/1tYu/da7f9gmJQaG2oetuBoE5BGo1k2jJgx/WdcCoBCFrARiy8ihBwloctVArWwTEC1RABdaGgBoNFolg22BjDlUgDMGDX9LQ0ADHt/3JEHUM7+D4VQ0FpyAC40tADQaDTLhpEZQwAMTiXI5as1KcRu6mKVZwAINxQ0AKMfcHkTkBXJU88CQPsANBrNsmE4Yph0snnFyEySnta56+zPJI2dfnOwsBSGi0xAOTod2oGTN+zsxesRupouAGP+PNECQKPRLBuGI0k6GgNMxNIMTCZcCACjqUtzsBCOGXaYgGKpLOsqdOLauqaZrWuaF2jmS4M2AWk0mmWBUoqRSIor17UB7hzBM8ksIhR1/CrSANJZmir4AJYDWgBoFpULIdZZszKYimdI5/Jcsa4dcOcInkkaDd+dHbkMJ3DBBBSu4ANYDmgBoFk0zk4nuezPf8CTL00s9VQ0K4Bh0wG8obORzsYAZybjVcdEkhlagsXZuKGA0QlMKWVoABXCQJcDWgBoFo2BqTjpXJ6nTmgBoFl8LAfw6pYG+ttDrk1ATgcwGOagWDpHPJ1Dqdl1gJYTVQWAiKwVkUdE5LCIHBSR95jHPyoi+0Vkn4g8ZPYCLjf+hIg8Z163x3G8Q0QeFpGj5v/tC/exNBcCVoTFiyPRJZ6JZiVg5QCsbgnS1x5yaQLKzBIAoYCPRDpXqAO0kgUAkAXep5TaBlwL3CUi24FPKqV2KqV2Ad8BPjzHa9yklNqllNrtOPZB4EdKqS3Aj8znmmWEVVBLCwDN+WDEFABdzQ30tYUYmEyg1Ny5AIYGUGwCagx4SefyTMcz9vPlSlUBoJQaUkrtNR/PAIeBPqVUxHFZI1A966KYNwFfNh9/GXhzjeM1FzjRlPEDOjYaJe8iKUejORdGZlK0hvwE/V7628OksnnGouk5x5QzAYXNHb/VKnKlawA2IrIBuAJ4wnz+MRE5DbyDyhqAAh4SkadF5E7H8dVKqSEwhAzQXePcNRc4UVMDiKdzDE67S83XaObLcCTJ6hYjKauvzYj/r+YILmcCsrqCjZp1grQTGBCRJuA+4L3W7l8p9SGl1FrgXuDuCkOvV0pdCbwOw3x0Yy0TFJE7RWSPiOwZHR2tZahmiYmaPgDQZiDN4jMcSbHabLDe32EIgLn8AEqpsiYgWwBoDcBARPwYi/+9Sqn7y1zyFeD2cmOVUoPm/yPAA8DV5qlhEekxX78HGKkw/h6l1G6l1O6uri4309VcIERTGazwai0ANOX44/v38/WnTi/Ia41EknSbVT0tDWCustDJTJ5sXpXRAEpMQCvZByBGhsQXgMNKqU85jm9xXHYbcKTM2EYRabYeA68BDpinvwW8y3z8LuCb8/kAmguXaCpHZ2OAjsbAeRUAPzw0zB/fv/+8vZ9mfuTzivv2DvDDw8Ozzr3tsz+rSTDk84qRmZRtAmoO+mkJ+uYMBS1XBgIKC/6I1gAAuB64A7jZDOXcJyK3Ah8XkQMish9jYbfCQ3tF5EFz7GrgcRF5FngS+K5S6vvmuY8Dt4jIUeAW87lmGRFNGUk0m7uazqsA+I99A3ztqdPa8XyBMxZLkc7mbVu7RTydZc/JSX5+fHzWmC/99CV+cPDsrL/tRDxNNq9sExBAf3t4ThNQxDRRtswKA105JqCqn0wp9Tgwux8aPFjmmGXyudV8fBy4vMJ148CrXc9UU3fEUlmagj42dTfxvQNDKKWKUu4XixdHoihlRHi0hqv3XM3k8uTyiqB/+ar6FyKWeWYkUiwArOelDtzRmRR/8e1DAGxd3cxdN2/mDTt68HjEkQNQqMzZ1x7i5Hjl1pAFDaAkEaw0Cmglm4A0mvkSTRrNNLZ0NzEVzzAemzskbyHI5vIcHzV+9Fat92p86IHn+K0vPbWY09KUwTLPjM6kiuL1rcW81HxzasIQCHdcu56cUvzBV5/hX35xEigIje4iDWDuXIBoyioFXVIKwl+IAmrwefB5l+8yuXw/mWbJiaaMGOvN3UbLvKPDi28GOjlhlJ8AoziYG06MxXnyxITdB1ZzfrDMM+lcnkiicO8t2/vZSLKomOBpUwC86+Xreei9N3J5fytfM/0Ezixgi762ELF0julE+e9BuV4AUNAAJmLpZR0CCloAaBaRqNlP1RIAL44uvgBwCplKP/xSIskMubzi2TNTizQrTTmcETpWJy8oLOZKwZAjf8QSAP3tYTwe4far+jk8FOHwUMSuA+RsztLfbtTxr+QIruQEDjtMPsvZ/g9aAGgWEcsJ3NMapDHg5ZjpCE6kc3z1yVOLUir66PCM/XjKrQAwr9t7cnLB56OpjNPGb+36oWB7N64pLN6nJuKsbmmwfTVv2NmLzyM88MwAwzNGI5iAr7CkrTVzAY5V2HhU0gAafB48pqsqvIzt/6AFgGYRiZpOYBFhU3cTR0dmUErxgfv288f3P8fT57jgjkdT7D1V/BpHR6K202467s4HYGkK5zofTW0MTCVs7bBUAwiYdnenkDg1EWddR6E7V0djgJsu6eaBZwYYmkrQ3VzcmvGSNS20hf08+sJY2fePmM1gShu+iIjdCF6bgDSaeZDO5kln8/aPa3O3EQr6hcdf4tvPDgIFJ9x8ePzoGL/8fx7jbZ/9mW0yAHhheMZuCOLGB5DN5YmZzT+ePjmpQ0fPE0opBiYTXLG2DSje9Y/MpNjW04zXI0UawOmJOGs7itsz3n5lH6MzKX764niR/R/A6xFu3NLFT14YKft3nUlmaAr48HhmR6ZZoaDaBKTRzAPLodoULAiA4UiK/+97R7isrwXA7rtaC9lcnk/+4Ah3fPEJGnwe8gp+8sKofe74WIxLe1sIB7yuTECWGeDS3hYiyWxFc4FmYZmKZ4ilc2xd00zQ7ykKBR2ZSdHbFmJNS9AWAKlsjqFIskgDALjpkm5aQ37SuXxRCKjFzZd0MxZNs39geta5coXgLKyFX2sAGs08sHb31g9pc5eh6m/oDPPpX90FGL6AWrlv7xk+88gxfuWqfh7+HzfS3dxgC4DTkwnS2Tybu5toC/ldaQAR0xF401ajFqE2A50frAig/vYQ3c3BIh/AcCRJd3MDfe0h2wRkhHPC2vZiAdDg8/KGnT0AszQAgFde3IUIPHJkdqUZoxBc+TwRKxRU+wA0mnlgx1ibAuCajZ28fmcP97xzN12mrTY+DwFwdDhK0O/hE7fvJBzw8cqLu3jshVGyuTwvmA7gi1c30xoOMO0iD8AKP7x8bRvtYb8WAOcJa2ff3x6mu7nB9gEk0jlmklm6W4JFXb2sHIB1neFZr/XWK/sB6GkNzTrX3hjgirVtPPJ8OQEwlwagTUAazbyJlpiAWkN+PvMbV7Kpq8m2ryYytQuAwekEvW0hO6P4VVu7iSSzPHtmyi43sakGDcByALeG/Fy1vl0LgPOEtbPvawvR1dxg+wAsQdDd3EB/e5hhMxfACgEtNQEBXLW+nS/91st4066yTQm5+ZJu9p+ZLnI0w9wCIKSdwBrN/LFKQZfbQQW8HrwemZcPYGAqaVd6BHjF5lV4BH78/CgvDM/Q1xaiqcFHW9jvKg/AMgG1hHxcub6d42MxJs5DxvJKZ2AqQTjgpS3sNzUASwBYfX0NDSCv4Ox0ktOTCRp8nqI4fyc3be2uuFu/6RLDvPeT54vLyc9lAmrUTmCNZv6UmoCciAhhv3deJqChqQQ9rQVbb2vYz5Xr2vnJC6McHY7aYYVtYb8rJ7CVA9AS9LN7fQdg5APMJDPc8+ixsrbjC4m9pya58qMPFyVMXYg8e3qKF0cKORoDkwn62w1NrrslyEwySzKTsyO6us3G7mBoC6fGjQigchE71dje08LqloZZZqC5NQBLACxvH8DyFm+aJaPUCVxKKOCt2QmcyubsCBEnr9raxV8/9AJ+r3D95k4AWkJ+puOZqgXoLA2gNeSnozGAzyP83SMv8v5/jzEVz3D52jZ7BzlfxqJGq0L/ItSU+aefnmAilub4aKysDfxC4QP/vp+Az8O3f/8VgKEBWJqc5RMaiaTsaKDVzUE7Fv/MZGJWDkAtiAg3be3mu/uHyOTy9t+hXDMYC+u9GwPLe4nUGoBmUSgNAy0lHKhdAxieNhaHUgHwyouNBTqTU2zpbgagLRQgnctX9TNEElm8HiEc8BL0e9nZ38qzp6fYtbaNG7as4szE3C0Fq5HJ5Xn13/yEzz/20jm9Tjmm4xm+f/As4L7u0VJxNpLkuYFpuzrnmckEfe3FAmA0mmR4xkgCawv7WdMaxCNwejLO6XMQAGCYgWZSWZ45NQVAMpMjnctX1ADC2gSk0cwfK76+0g4qFPDVLACsvsK9JTvdS3tbWNUUAGDz6oIJCKovjNOJDC1mtjLA/3n7FXzn91/BP/3W1bx80yrGY+lzKhJ3eiLOdCKzKM7l/9g3YJfTcFv3qFaeOD7OFx8/N+GVyhYKsn1n/xDRVJbpRIa+NmNB73ZoAKORFF3NDYgIfq+HNS1BDgxMM5PKzkoCq4XL+lqBQme6mQq9ACzC2gms0cyfaCpLOODFW8FmGw54SWRqW1gHzdjx3rbieG+PmfEJsMXyAYTcCYBIMkNLqGAGWNcZthcLq5bM6SqNxefCKk39/HBk3q9RiX976rTt83Bb+rpWvvLkKf72R0fP6TXGooW5fXf/kF0EzrLxW20cR2ZSDM8k6W4pLuj25EsTQPkIILf0tAQJ+DycMDWQSoXgLCzbf3iZ+wC0ANAsCjGzEFwl5mMCKgiA2bbuP3j1Fj75tp32D9pqBFNtYYwkMrRUWASsBefU+PwFwEtjxoJzeiJhLzoLwYGBaQ4NRXjndesJ+DyLpgEMTiXmFa7rxArxvH5zJ4eGIjx21IjGsUxAnY0BvB5hZCbJSCTF6ubimv5WqQ5LIM8Hj0dY3xHmxJglAEwT5Rw+qrnOLxe0ANAsCjMuBECtTuDBaaPiY7nOXRtWNfIru9faz9tChkkoUmVhjCSztIbKCwAr6/T0HH1lq3F8rFBa4gVHpdJz5d+eOk3A5+FNl/fRGvJX/ZzzZXDKiMPPnUONJEsAvPO6DQC2SanfFOQej7CqKcDoTMrIAi7SAAqLfmkWcK1sWNXo0ADKVwK12LW2jV1r24refzmiBYBmUbDaQVYiPB8fwFRilvmnEm59AJFEhpZQ+Xm2hf00NfjsJKT5cHw0Rq8Ztnp4aGEEQDKT4z/2DfC6y9bQGva7TnqrlVxe2WGZyXPQAiwBsLO/lZdtaGdw2nD0rnLE9Hc1N3B6IkEkmZ3V1xdgVVPgnB2yGzrDnByPk8+rqiagS3tb+Y+7rrd9AcuVqgJARNaKyCMiclhEDoqI1fz9oyKy32wS/5CIzErDqzTWPPcRERkoaTSvWSZEk3NrAKF5moBKHcCVsAVAlZ3x9BwmIBFhbUf43ATAWIyXb15Fc4OP588ujADYe2qSmWTWznxtDblLequVsWiKrLnzn0/OhoUlADobG3j9DqNuT197qCimv7s5yMFBo2BbV/NsDeBcHMAW6zsbSWXznI0kq2oAKwU3GkAWeJ9SahtwLXCXiGwHPqmU2qmU2gV8B/hwDWMtPq2U2mX+K9tkXlOfWN3AKmEkgtXmBB6aSpa1/5cj5PcS8HpqdgKXsrY9ZNehqZWZZIbRmRSbupq4pKeZI2cXxhFsfSbrXrSFF0cDsAq2wTlqANEk7WE/AZ+H1+3oQYSibG4wIoEi5qJcTgM4FwewxUWrGgE4MR4rZIBXEP4rhaoCQCk1pJTaaz6eAQ4DfUop57e5EZhlJKw0diEmrrmwiaayZbOALYwooFzFht2lRJIZZlJZ1yYgETGSweZwAqeyOZKZfMVQQDAWntOTcdfzdGI5gC9a1cgla1o4MjQzr9cpxVm/CMykt3PUAPaemuSvvn+k6NjQVKF2jltH8OmJuB3rbzE6k7J39atbgvz2DRu5raRuj7OZi/PxGrOb3MWrm919kDlYbxaSOzEWn1WraqVS06cXkQ3AFcAT5vOPAe8EpoGbahlrcreIvBPYg6EpzAqWFpE7gTsB1q1bV8t0NUtItIoPIBTwoRQkM3k74mIu5ooAqkS1nbFlBqjkBAbD9JDM5BmNpuxwRbdYIaCbuhoZi6aYSWUZmErYu9r54ixfAQtjAvrqE6f4xtNnuPPGjbSFDQe6s7yEWxPQB+/fTzKT577ffbl9zCkAAP7k1m2zxjnPOzWAgM/D995zY5FjeL70tIYIeD2cHI+RzSsa5whTXim4dgKLSBNwH/Bea/evlPqQUmotcC9wdy1jgc8Cm4BdwBDwN+XGKqXuUUrtVkrt7urqcjtdzRKilCJWzQRkLvpuzUDWbrSWcgfVnKP2QjqHALBMD6cnao8EOj4WwyNGbsG2HmMHuxB+gOlEBp+ZvQxGxFM0lSWTm3+P5cOmeerYaGH3PujUAFwKgBNj8VlNddwIzy7zvN8rtIeL/x7rOsNlI79qxesR1nWGOTEem7MQ3ErClQAQET/GAn6vUur+Mpd8Bbi9lrFKqWGlVE4plQc+B1xd6+Q154efHxtnPJqqfqFJKpsnk1NVncDgfmdp2aNLbcdzUa0g3HSiuh3YTgabhx/g+GiU/vYwDb6CCeOIQwDMpxoqGPNuDfnt7OVWM4ppvqGgRi8FY9F2Lt5ODcCND8CKGpqKZ5g2Ba9SapYGUA5rh9/dHJyzdtO5sqEzzImx+JyF4FYSbqKABPgCcFgp9SnH8S2Oy24Djrgda57rcTx9C3CgtqlrzgenJ+L8xud/wWd/fMz1GLsOkAsNwK1teXAqgc8jVRcSJ62hwJyLouV0rBQGCgUn5HwEwEtjMdvx2Bz0s7YjxOEhY6d95GyEqz76Qz7zyIs1v24kmS3SWiyTzXzNQMfHYnZJCacAGJxK2G0W3fydRmcKUUMnJwxNIprKkszkK5ZxtrDs/rX8fefDhs5GTk4YTmAtANxpANcDdwA3l4RsflxEDojIfuA1gBUe2isiD1YZC/BXIvKcOf4m4A8X8oNpFoav7zmNUrD/zOyeqpWI1iAA3GoAQ9NJVrcEa7LZGj6Ayk7gUlt6OYJ+L93NDTVHAimleGksxsauRvvY1tUtPH92hmwuz//8xn4SmRx/+8OjtrMYjHaIv/3Pe4pKJ5cynSiOXLJ8GPMVAJZQCvo9HBtxmICmk2wyW3m6+TsNOjSGk2b2tBUCWm1ht3ICyvX1XUjWr2okmcnz4khUm4Bw4QRWSj0OlPvVlQ3bVEoNArdWGYtS6g7309QsBdlcnm/sOQPAwcFp8nnlqh67nWY/lxPYb5xzawZxlg92S1vITyydI53NE/DN3us4S0HPhRUJVAvDkRTxdI6NqwoCYFtPM488P8Lf/eeLPDcwzUfeuJ2/eegFPvzNA/zzu68mkcnxX7/8FAcGIrx8Uyebu8tHvlgF7Cxaq+Q8TCcyHBmKcM3GzrLnDw1FCHg9vGJzF8dNDSCdzTMWTfHLl67mZ8fGXWkAg46wUUtguhUAQb+XvrYQGxz3azHYYEYCDUdSvGyD1gB0JrDG5vREITwO4CcvjHI2kuSW7auJpXMcH4vNMbpATSYglxpALVnAFlYyWKWdsdUPeC4nMGAmg9XmBLYW0o3mDhrgkjUt5PKKv/3RUV532Rp+8/qLeN9rLuaxo2N869lB3vO1fRwajODziN0Ltxwzpg/Awnpcztw1OJXg9s/+jLff84uKZqzDQzNs7m7ikjXNnJyIk87mGY4kUQo2rjLmn3Txd7Ic9U0NPrvmzmjUnQAAuP/3Xs57Xr2l6nXnwobOgoDRGoAWABqTRDrH6//vY9z+Dz+zd8ZfffI0q5oa7B/lgQF3ZqCFNgFZzsWeGjWAVts2Xt4MFElmCHg9NJTRDpys7QgzNJ2w7eRusISl0wR0iRkJ1Bb28xdvuhSAO67bwGV9LfyPrz/Lw4eG+fM3XspFqxrtnrnlmK4gAEojnl4YnuGt//AzezGuVIvo8FCEbT0tbOpuJJdXnJqI2U53a/5uNICBqQRNDT5bkIBDA6jiAwAj/HOxSy/0thmhoFC5FPRKQgsADQAPHx4mkszy/PAMv/uvT3NmMs4jz4/wK7v7uWRNM0G/h+dqFABzhYGGatAAxqIpMjlVUw4AVF4YLabNOkDVok7Wmr1pnSaOctz5z3v4yLcOks7mOT4aI+T3FlW23NDZyM2XdPOJ23faYZFej/CxN+/AI/CbL9/Au16+gb72UFEWrhOllCsfwEtjMd722Z+RU4p//W/XAHB0pDg8E4x7OzqTYltPs23vf3EkakcAre0IE/B5XAnqoWlDS1vXGbYrqI7OpPB7paqZ7Xzh9Ygd2aWdwLolpMbkP54ZoKc1yB/ecjEf+Pf9vO2zPyeXV7x991p8Xg/belpqFgBz/cCsnZ4bH0AhBLRGE1AVATBXKWgndi7AZHxOG/VjR8dIZHIcGJgmpxQXrWos8pl4PcIXf/Nls8ZdvraNpz70S/Yi2d8eYt/pqbLvkcjkyOZV0YLq93poDHiLPucjR0aIJLP88PdezubuZrqaG+xmKE4sB/D2nhbbXOXMBehtDRHye12FgQ5OJelpDbGhs5H79w6QzOQYnUmxqqlhXr18F4sNnY0cG41pExBaA9AA49EUP3lhlDft6uNXd6/lPa/ewtlIkus2dtoL3o6+Vg4OGI7gatTiA4g7FpYzk3HuunfvrMgda+dda8/bqj6AknDKSliFyOaKBMrlFYlMjpdtaOfgYIRnTk1xUZd7h2ZbOGBrIv3tYabimbL9A0rLQDjHOz/nwFSCoN9j7+q3dDeV1QCOmBVKL+lpoanBx5qWIMdMDaAt7CcU8BLyuyvdbWgAIbvkwqmJOKPR6jkA55v1nVZort7/agGg4Tv7h8jlFW+5wijT9N5f2sJH33wZ/8u0U4PRUs+tIziazCJSWOTL0eDzIFJsAnri+ATffW6If/3FyaJrHz40TFODr8iB5warJ0Cl6JhIYu5CcBarW4IEvB4ePzrGh795gKs/9kM+9fALRdfETE3mNdvXcN/vvpwdfa28ep7N5K0KmOXMQJWS10rrHg1MGlFTllDZ0t3EsZHorFpEh4cirG5poKPRuFebuhs5Nhq1d/NgNu+pogEkMznGoml6W4O2xnRyPG4kgbmw/59PLlplzE9rAFoAaIAHnhlgW08LW9cYTkoR4Y5r17PFUYBrh9km0Y0jeCaVpSkwt21dRMyKoIWFZdLc+f/zz0/aDtfBqQTf2T/Er71srauaQU6agz5EYLpCLkAkmXHlCPR6hP72EN87cJZ/e+o0M8ksB0vuQ8zh99je28K3f/8VvPXK/prma2Eln50pE3lkRS7N0gBK6gGdmYoX1Rza3N1ENJXlbCRZNO6Q6QC22NTVxLHRGINTCdvkFnShAQxNG6/b2xayBfXJ8ZirLODzzSXm513TUptJcTmiBcAK56WxGPtOT/GWK2a1cyhiS3cTDT53juBqdYAsShvDWzbskZkU39k/CMCXfmp0j/qtV1xU9fVK8XgM5+O5agAAH3vLDj799svZ86e/xI7+1qJwWXAKgHOvWWPlO8ylAZQKgNKCcAOTCbvlImDnFBwdLpiB0tk8x0ajswRANJXlxZGorQGEAtV9AEOWma4tSFvYT3PQx0tjMcZj6QtOALxsQwcP/eGN7OhvXeqpLDlaAKxwvrlvABG47fK5q3TX4giuVgnUwmgLWVhIJ+Np2sN+Nnc38YXHXyKSzPDVJ0/z+h09NSeBWVQqCKeUIpLIuq4Hf92mTt5yRT/NQT/NDb4yAsBYIBeih+yqpgANPk/ZUFDbBFRSvsJZ+TSezjIZzxTdsy2rDV+A0w/w4kiUTE7NEgAA2byix9QAQn5v1TBQZ60mEWF9Z5h9p6fI5dUFJwCABSkvvRzQAmCF8+BzQ1y3sZM1rdXV4R19rRwajFR1BEdTOVcLYWlj+Kl4hvbGAO++/iIODkZ4/9efJZrK8ts3bKz+QSrQGg6U1QBS2TzpXH5e4YmNDT57x28RcxH66hYRw+RULhks4kIDGDDHOfvZdjYGaAv7iyKBChFAhcVwU3fBz9Lr0ACqhYFaJiDre7S+o9F+/QvNB6ApoAXACiaXN+rV7Frb5ur6HX2G6eOl8bkdwdFkxpUACAWKd5aGBhDgrVf20R7289ChYa7d2HFOqnpbyF/WBxCpsJN2Q1NwtgbgJvmtFvrbw2UFgLXIlzowW0J+Utk8yUzOHucUACLClu6mohpDjx0dpTXkL3Kur2kJ2s57K+/CTRjo4FSCVU0NNPiMses7w1j7hAtRA9AYaAGwghmZSZLJqSJb8VxcZjqCv3/g7JxaQGyeGsBkPEN72E/Q7+Ud16wHOKfdPxg747FommxJrfxzaQnY1OCz6x1ZWFFAc0U+1YKhAZQ3ATU3+GYVxXOGvJ6xzTHFjWc2dzdz1IwEiqezPHRomFt39ODzFpYBEbHNQD2tDhNQFQ1gcDpZVKrDCgUFLQAuZOpKAMxVH0VTO4WdorsOVVtWN7F1dTOf/MHzvOqvf8znHj1OKjt7YajWD9gi5C91Aqft0sZ33bSZz77jSm6eZyilxfbeFgamErzykz/m848dt2PrC7b0+QkAo+dBQahEF9AHAEbT9Ml4ZpapqVIPY2fW88BkAr9XilorghEJNBXPMB5L8/ChYeLpnN1Y3smmrkZECuYcwwQ0d8Le4FTCNhkBrOsoaBWrtAnogqWuBEAkkTmnrkeaYqwdZr9LDcDv9fDt338Ff/frV7CmJcjHHjzMX//g+VnXzbistV7JCQzGomM0ED+3DNLfvmEj/++Oq+hrC/G/v3uYN3/mp6Sz+UIhuHkkA1mLvHNxXkgfABSEcmkkUKSkDpCFlfMwnchwZjJOb1toVvbtlm7TETwc5Zv7BulpDXL1ho5Zr/X2l63j7ps24zc1AyMKqPLvTinF0FTCdhoDbDBj7RsD3gW7J5qFp64EQE4pfn5sfKmnsWyw4sxribAJ+Dy88fJevv471/HWK/r4l1+cZMzRLUwpRSxduwkomTEatFsawELh9Qi/fOkavv471/H3v3EFx0ZjfH3PadeloMthfbZoiQColvxWC5ZQLjUDWfWLSiloAOmKpbOtSKCnTkzw6Auj3HZ5b9kSDddt6uR9r9lqPw/5vaRz+VlmNItIIkssnSt6z9XNQQI+jzb/XODUlQDwiPC9A2eXehrLhoGpBF3NDfPut3rXzZtJZ/N87rHj9rFkJk8ur1zmARRsy1YSWPsCCwAnr9/Rw1Xr2/m7/zzKSMQQWvMyAQVnC4BoKktjleS3WigIgFINIFteA3D4AKws4FLWtARpavDxhcdfIptXvGnX3KG/FiHz+5GsUA3VagTjLNXh8QjrOsLa/HOBU1cCoDno46GDZyvuRDS1cabCQuGWTV1NvPHyXv7l5yeZiBkLuB0N49IEFM/kUEoxGTN25KUNwRcSEeH9r9nKcCRlJ5jNpx5MYwUT0EIkgVl0NTWYuQDFAqC0FLSFJchGoylGZlJl/ToiwqbuJqYTGbZ0N9mN6qsRtEt3l/cDWLWaSvs1/NFrL+EPFrm+v+bcqCsB0BryMx5L8+SJiaWeyrLgzGTctf2/EnfftJlEJsfnHzvOcCTJB/79WcBd5c5wwEcur0jn8nYBuIU2AZVy3aZOXrF5FYPTSYJ+jx22WAuWCcgZCRRL5xbU1i0i9LWF7Jh+i+kKFUybG4yyF4fN4m6VIrssP8Cbr+hzra2ELQ0gXUkDKJSBcHLL9tXceHGXq/fQLA1umsKvFZFHROSwiBwUEav370dFZL/Z5/chESlbS0BEXisiz4vIiyLyQcfxDhF5WESOmv+3V5tLc9BPyO/le89pM1AtJNI5PvH9I3xz34B9LJ9XDEwlXEcAVWLL6mZu3dHDP/3sBLd86if8/Pg4H37Ddl51cfXoHcu0kEjnmDSzWNsbF79A1/t/2bBvzycEFCr7ABYqAsiiryQUNJ3Nk8jkymoAVtmLg4NGpnYlze7S3ha8HuG2y+cu/eHE7t1QIRdgcMqIOtIJX/WHGw0gC7xPKbUNuBa4S0S2A59USu1USu0CvgN8uHSgiHiBzwCvA7YDv26OBfgg8COl1BbgR+bzuScr8KqtXXz/4Nxx6CuZrzxxiv/x9X38/Ng4SikODUZ4498/zmd/fIz/95OCrX5kxmiycq4aAMAf3LyFTC7P5u4mHvyDG3j3Ky5yVf/d2RXsfPgALHatbeP1O3vm3X/WMm/NMgEtcDer0mQw23FdwUzWGvLbDeYr/V1/45p1/OC9N9olrt1gCWqnCWjf6Sk+8q2DvP8bz/Ld/UOsbgleUDX/Ne5w0xR+CBgyH8+IyGGgTyl1yHFZI1BuRb4aeFEpdRxARL4GvAk4ZP7/KvO6LwM/Bv6o2nxet6OH7x04y9OnJnlZmRC2xSSVzc3LZHA++cqTJzkwEOH+vQNs6AwzOJWkLeznpq1dPHp0jGQmR9DvZWCqthDQudi6ppmfffDVdDQGZiUozUXIIQAKJqDzU6L3b9++q+wX1g3lTEDRVI6+toUVXv3tIcZjaeLpLOGAr2IpaIu2kJ+TytgoVSrt0eDzsrm7qey5SlhBAk4N4B8eeZEfHRlhdXMDTUEfr9/hXqPQXDjUtGURkQ3AFcAT5vOPAe8EpoGbygzpA047np8BrjEfrzaFC0qpIRFxlfFz8yXdBHwevvfc2fMqAE6Ox/ilT/2EB37vejsj9kJkYDLB7Vf2c/3mTr6x5ww7+9v4yG2X8tSJCR55fpRDQxGuXNdecxJYNeYT7md1BbNMQOGA97wJWGf2a600moLLKgBnPF5YJzA4+gJMJtiyurliHSALyxG8piVox/AvBJamliwp2/GyDe187c7rFux9NOcf198SEWkC7gPeq5SKACilPqSUWgvcC9xdbliZYzVtvETkThHZIyJ7RkdHaWrwcfHqJk5UqUez0FiVE+fqCrXUWFUgN3U38tYr+/nqndfyf3/9CjoaA1ze3wbAc2cMG7ElAM4lCuhcCTuiSybjabuF44WOz+sh5PcSTRWKzLktgV0LlgA4bfoBqmUvWw70hRLqFoX+zQUn8FQ8YyefaeoXVwJARPwYi/+9Sqn7y1zyFeD2MsfPAGsdz/uBQfPxsIj0mK/fA4yUe2+l1D1Kqd1Kqd1dXUZEQVsoMKtt4GIzOmPEjbtpjbdYPH50jK8+eari+YE5FvXVLQ10NTew3xYAcVY1BWpusrKQONtCTsUzix4BtJA0Nvjs8g9glsBeYAFg1fA/NGhU1Sz0Aij/PtZxt7Wd3FLOBzAZz5wXh71mcXETBSTAF4DDSqlPOY47A3xvA46UGf4UsEVELhKRAPBrwLfMc98C3mU+fhfwTbeTbg37K/Z5XSwsAVCtNd5icWgwwm//8x7+93cOzWrrZ3FmqrIAEBF29rWy/8yUce1kgr4F3inWSrEJKF1XC0qzoyJoNpcnlc0vuBO4NeRnY1cjz5pCO2L6HCppAJZpaKG1OssHYJmAlFJFdZs09YsbDeB64A7gZjPkc5+I3Ap8XEQOiMh+4DWAFR7aKyIPAiilshimoR8Ah4GvK6UOmq/7ceAWETkK3GI+d0Vp96PzwWjU0gDmLoq1GEzE0vz2P+8hkckRS+cYjqTKXmcl5FTaAe7ob+XF0SixVJaBycSCOIDPBWcU0HTdaQBeOwooZmqFC+0DALi8v41nT08BjhLWFZ3AlgloYf+u4ZIw0Fg6Rzav6sZkp6mMmyigxylvy3+wwvWDwK2O5w+Wu1YpNQ682vVMHVhdnpRSC5Z6Xw2r3k21xhgLTSaX56579zIaTfFHr72ET3z/CMdGo2WjPAYmE/g8Qndz+QiQnf2tKAXPDUxzZirBLZeuXuzpz0nBtpwtKgRXDzQ1+IiaO/LYAvcCcHJ5fysPPDPA2ekk04kMDT5PxdIdtgawwALAjgIyfQCTsfMXsqtZXOoqE9iiLewnm1f2zut8UKsP4JlTk7zubx+zY7fny+cfe4mfHx/n42/dwVuvNGq3HB+Nlr12wKzIWCkUc0dfGwCPHBkhnc3Tv4QOYCjsLGdSWaYTmbpaUJoa/MykigXAYlS9vNxs1rPv9FTFSqAWl/a1sLYjVNTicSHweoSAz0M8Y3xOS/s+XyG7msWjPgWAo/Tt+cIWAC59AA88M8DhoQhHh2eqXzwHT5+c5OLVTbz1yn66mxtoDHg5Nlo+AqpSETCLruYGeluDPHhgCFj4aJFaCZohnyORFHm1+GUgFpImhwloobuBOdnW04LPIzx7ZsqsBDqHAOht5bEP3LwoBdhCfi/JksJ99fT30pSnLgWAlQl5PiOBbCewSw3g8aNjAAxOJc/pfU+Mx+yWfVYxr2NzaAClXaBK2dHfyumJ2S0DlwKPRwj5vXbN+7oyATmcwFY+wGJoAEG/l209Lew3BcB8ylcvBGFH+067bEcd/b005alPAWD+CKbj50cDiKWytrnJjQno9ESc42ZK/tD0/LuY5fKKU+NxLnKULNjU1cSxkdkCIJPLMxxJVi3CttPMB4CFtxXPh3DAazuv68kEZISBFmsAi+EEBrh8bSv7T08zFV86ARDyF3o3WD2WK5Wk0NQPdSkALNvj1CKZgL67f6hooXc2PKnWGg/g8ReN3b/IuWkAZyNJ0rk86zudAqCRwenkrHmcnU6SV9UX9R1mFnNnY8AOw1xKQg4BUE825eYGH+lsnnQ2X/ABLNL93Nnfxkwqy9GRmXl1MFsIgo7G8JYGoBPB6p/6FADmF29qETSAl8Zi3PWVvTzwTKFyZrEAqK4BPHZ0lJ7WIFu6m+zFbT6cMLWIDY4G21bD7uMlfoCBCo3AS7EEwIWw+wdDAyiYFOpnQXH2BLAawi9W68NdpiM4k1NLpwE4TEBT8QxNDT4CvrpcPjQO6vIv6Ox+tNBYi72z1IRl/+9ubiiqh1KOXF7x+NExbtiyip7WkN0taT5Yc3BWrdxoCoBSP4CdBVxlYW9vDLC5u4nNXbUVBFssQo5dcz0JAGdJ6NgCN4QvZVNXkx0xtaQ+gLQlANJLNg/NwlKXAiDo9xLweZhKLLwT2OpsZe2+oSAA1neGq2oA+89MEUlmuWFLF71tIYbOwQR0cjxOwOdhTUvBrr++M4xHmBUJZGkAPRWqQDr51/96DX/+xkvnPa+FxGo24pH5dedaKpwVQWOpLB6BoH9xfk5ej9ia23xaWC4EQYcPoN6ytjWVqUsBAEYy2GI4ga0kl5PjhaJvozMpPGKk2FcTAI8dHUMErt+8it7WIOOxdFWtoRIvjcVY3xEuqrMe9HtZ2xGelQswMOm+v++a1uAF48CzdrZt4UBd1ZO3ewKks0Y/4IaF6wdcDssMtFQCIOTwAUzVWc6GpjL1KwDC/kXxAUyYEQ4nJ2J205nRaIqORqPuebU8gMeOjrKjr5WOxoDdIm9ouroWoJSa1ev45HisbNOSTV1NZTWApazsOV9CtgC4MASSW2wTkKkBLJb5x8KK3lrKKCCnD0CbgJYH9SsAQoFFMQFZGkAyk2fENP2MzqToam4gHPDNir7J5PI88MwZfnDwLE+dmGDvqSlu2LIKgB4zJHPIhSP4X35xkld84hHSWUMI5POKk+PxIgewxcZVjRwfjRZ1RRuYSlwwjt1asDSAettRFvkA0gtfCrqUV27t4jeuWcc1F53fJkgWoUDBBDQVT9fd30tTnvoxupbQEvLbdu+FZCJW0CpOjMdY0xpkNJq2zSvJTJ58Xtnmip8dG+cP/+3Zote4YYtRtrq31Wzo4WKejx8d42wkyd5Tk1y7sZPhmSSpbHEIqMWm7iZS2TwDUwnWdoRRyujve8v2pa3tMx+sUNR6SyqyTEDRVJZoamEbwpd9vwYff/mWHYv6HnMRChgmoHxeMZ3I1J3GpilP/WoAYb+dkOKWfF7xv759iAMD0xWvmXREOJw0o3DGZlJ0NTUUOiNlC2YgKxv5H95xJffccRX/8I4r7V2aVbDNjQno0JBR893KILZ6u15UwQQE2MlmY9E06Wy+zk1A9bWjLAoDTWVpWqQksAuFkN9LJqeYiKfrrmyHpjL1KwBC/poTwX5+fJwv/vQlvvvcUMVrJmJpLu016q+cHI+jlHKYgArliy2svrC717fzmkvXcOuOHtsZGPR7WdUUqJoLMB3P2B26HjOTyCwn9PoyJqBNXYZQsDKCB+boA3ChY0UB1ZsGYCV9WVFAi5UEdqFgNYU5a25m6u3vpSlP/QqAsJ94OmfbzN3w9T1Ge+LhSOUd+WTcMPes7QhzcjxOJJElncvT1dxg/wicWcJ2IbAKIYy9bSEGq2gA1u7/inVt7D8zxVQ8zYnxGAGvh57W2Yt6R2OAtrDfzgVwmwNwIVKvGoDXI4QDRkG4xegGdqFh/Z3qMWtbU5m6FQCt4doqgk7HM3zvwFnAqD5ZiYmY4eBa3xnmxHiM0aixeDvbJzo1gGgyi9csalaOntZgVSfwwUHDJPXfb9yEUoZf4cRYjHWd4bKlnUWEjasaefbMFDPJDANThrZQjwKg4AOoLwEAZk+AVJZ4Okd4BZiAoGDOrDeBrSlP/QoAqyCcy0igb+0fJJ3Ns64jXFEDyOTyzCSzdDQG2NDZyMnxuB0J5DQBOUNBrd1fpRjw3rYQg1OJim0cwWj32N3cwKu3ddPU4OOxo2MVI4AsXnPpGg4MRHj5x/+Trz11muagr2KnqAuZQhRQ/c29KehjJlXIA1jO2BqAmdmuu4EtD+pWAFhfQLe5AN/Yc5pL1jRz09auigLAqnPe3mhoANFUlufPGvX8u5sbCPmNH7kzFHQmObf639saIpbO2f1cy3FoKMKlvS34vR6u3djJY0dHOTEeKxsBZPE7r9zEt+6+nhsv7uLEWMwuEVFv1KsJCAwNYDqeIZ3N07RCfABWZns9amya2bhpCr9WRB4RkcMiclBErN6/nxSRIyKyX0QeEJG2MmO3OvoI7xORiIi81zz3EREZKOkz7Bq7IqgLAXB4KML+M9P86u61rG4NEklmy5Z1njRDQDvCAbsG/54TkwB0NQULGkCREzgzZwkDKxfAsp3OJDP8+PkR+3wyk+PoSJTtvUYXpxu2rOLMZIJkJl82CczJzv42PvMbV/LoB27iH//LlXPfhAuUl23o4K1X9rGzv3Wpp1IzTQ0+ezOxUjSAoekEIkuXkaxZWNxoAFngfUqpbcC1wF0ish14GLhMKbUTeAH449KBSqnnlVK7lFK7gKuAOPCA45JPW+fN3sGusSuCuvABfGPPGfxe4c1X9LHa7JdbTguw6gC1N/rt6JunTkwQ8HpoCfnKRgFVcwAWsoENAfCph1/gN7/0lN3o+4XhGXJ5xaW9xgL4CjOJDJjTBOSkvz1c1llcD3Q0BvjUr+6qywW0scHHWfN7tOydwH7LCZykJeiv2HZUU19UFQBKqSGl1F7z8QxwGOhTSj2klLLsGr8A+qu81KuBY0qpk+cyYYuCD2BuAZDLK/5j3wC/tG01HY0BVrdUFgCWCaijMUB/u1F0bcQMARURR3PsEgEwhwZgJYMNTiVJpHPc9/QZAL78sxOAYf8HuNTUADauaqTXzB/YMIcJSLP0NDf47DDgehRgtWBpAMORZF36azTlqckHICIbgCuAJ0pOvRv4XpXhvwZ8teTY3aYJ6Ysi0l7LXJqDPkSomgx2YGCaiVia1162BoDVLUa/1OGZ2ZFA46YG0BEOEPB57N37qmZjTFkncDJL8xzO167mBnweYXAqwbf3DxJJZrl8bRvf3j/I6EyKg4MRmhp8rDX784oIN17cRdBfeH/NhYlz0V+sbmAXCpYGkM2ruvTXaMrjWgCISBNwH/BepVTEcfxDGGaie+cYGwBuA77hOPxZYBOwCxgC/qbC2DtFZI+I7BkdHS1M3CO0ukgGs7pzXb/ZMK10mxrASDkNIFbc7NragXc1Gc+tkMWiRLAqJiCvR1jdEmRoOsm9T5xic3cTn/rVy8nkFF998hSHhiJs72kpqoT5P395K/f+t2u1mn2B49T8lr0JKFAQcDoHYPngSgCIiB9j8b9XKXW/4/i7gDcA71BzxTnC64C9Sqlh64BSalgplVNK5YHPAVeXG6iUukcptVsptburq6voXFuoekXQx4+Osa2nhVVNxi6+Jegj6PdU9AE0OzodWX6ALlMDCPo9iEDCEQVkaABz//h724I8dnSMZ09P8Y5r1rGpq4kbL+7iX39xksNDEdsBbNHZ1MBV62tSiDRLQFORBrDMBYAjz0VHAC0f3EQBCfAF4LBS6lOO468F/gi4TSkVrzTe5NcpMf+ISI/j6VuAA24nbdEaDszpA0ikczx9cpJXbO50vi+rW4IMl0kGMxpdFL7cBQ2gwR7rbI6dyeVJZHJVd3+9bSHGoimCfg9vvdJwlfzWyzcwMpMins7NEgCa+sD5d1/uGoCzz4QuBb18cKMBXA/cAdxcErL590Az8LB57B8BRKRXROyIHhEJA7cA95e87l+JyHMish+4CfjDWidfzQT01IkJ0rm8bf6xWN0crKgBOAVAqQYAxk4obvoArGbg1X78VoTObZf32j+eV17cZUf5XKoFQF3i/LuHA8vbB+D1iK0Zaw1g+VB126KUehwoZ4wuG7aplBoEbnU8jwOdZa67w/00y9MW8nNqPFbx/E9fHMPvFa4uqaHe3dLAwcHIrOsn42l7tw+wracFv1fYsrrZPhYKeEmaGoAVATJXFBAUwjnfcc16+5jHI9x98xb+4ZEX2dLdXGmo5gKmcQWZgMAQculsXreDXEbU9be2LTy3BvD4i2Ncua7ddt5arGkJ8p9HRlBKFZVwmIxluNix2K/tCPP0n91SVGIh7GiMYRWCa67y43/zFX1sXdPM5WZbP4u3XdXP266qFj2ruVCxfD8+j9Dgq9uketeE/F6m0N3AlhN1/a1tC/mZTmTszlj/9NOX+M8jhp95Ipbm4GDE7s7lZHVLkHg6Zy/gFhOxNB0l6m1pfZ1QwGebgGwBUKUGT9Dv5Yp12qm73LB2/YvdD/hCIWSX7tYmoOVCXWsALSE/ShmhmLFUlo98+xAegY/fvtO2yZba/8EwAQEMR1L24p1I50hkckU+gHKE/V47Cijq0gSkWZ5YPoDl7gC2qNf+zZrK1PU314rXn45n+P5Bo8nLrrVtfODf93PRqkaagz529M2uMbPakQuwudsooubMAp6LUMDLcMQwO0WSxv8rZQHQFNNkawDL2wFsoTWA5Ufdm4AAphJpvvXsIJf3t/LVO6/lNdtX89JYjOs2duLzzv6IlgA464gEsuoAuREAiVkmIC0AViKW5rcSHMCgNYDlSF1/c60v4jOnpjgwEOFPX7+NBp+Xz7zjSu559DivvLir7Lju5oIJyMKtBmCYgEwBkHQXBqpZnoT9XkRWzt8/6Pfi88iK+bwrgbr+S1oC4F9+cRIReMPOXgD8Xg933bS54rjGBh/NjlK+4KgEWkW9LY0CEln+MeCa8ng8QmPAt+z7AVuEA17awv4V4fBeKdT1N9eqSf7iSJRrLupgjVlF0w3dLQ2MzBQEwKRrE5DP1gCsZjD6B7Fy6WgMrJi4+LfvXsvLNnRUv1BTN9S1AHDGI9+2q7emsaXlICbiGUSqp7mH/F7SuTzZXJ5oKluXbRg1C8c//perqm4algsv37yKl5eJqtPUL3XtBG7weQkHDLvkrZf1VB/gYE1LcTmIyViatlD1RhfOktDRKu0gNcuf7b0tNWmeGs2FRN2vXt3NDWzsaqoavz9rXEuQkUjKzgaeKCkEV4mQoy3kTCqjcwA0Gk3dUver1+fftXteDSpWtzSQzuWZimdobwwwWSYLuBzOtpDRZFY3x9BoNHVLXZuAADZ3N9u1/mvBbg1pOoJLK4FWwkqGiadzRjMYrQFoNJo6pe4FwHxZ3VKcCzAZd6cB2CagTNZoBqN9ABqNpk5ZsQKgu9nQAB4+dJaJWJrJWMaVBmBVFk2kjSggnQWs0WjqlRW7evW0BrliXRv/+otTfO3J02Tzig4X8dyWDyCayhBP52hq0GGgGo2mPlmxAsDn9fDA713P4aEIDzwzwKMvjHLV+upJLpYJaHTGMB1pH4BGo6lXVvzqta2nhW09LfzJrdtcXW85gS3fgfYBaDSaemXF+gDmi2UCsspIaA1Ao9HUK1UFgIisFZFHROSwiBwUkfeYxz8pIkdEZL+IPCAibRXGnzCbv+8TkT2O4x0i8rCIHDX/r4uWWbNMQFoD0Gg0dYobDSALvE8ptQ24FrhLRLYDDwOXKaV2Ai8AfzzHa9yklNqllNrtOPZB4EdKqS3Aj8znFzwBrwevRxjRPgCNRlPnVBUASqkhpdRe8/EMcBjoU0o9pJSymur+Aqi1u/mbgC+bj78MvLnG8UuCiBD2e20B0KIFgEajqVNq8gGIyAbgCuCJklPvBr5XYZgCHhKRp0XkTsfx1UqpITCEDNBdy1yWkmDAy3jUMgHpMFCNRlOfuN6+ikgTcB/wXqVUxHH8QxhmonsrDL1eKTUoIt3AwyJyRCn1aA3veydwJ8C6devcDltUwgEvo8p4rE1AGo2mXnGlAYiIH2Pxv1cpdb/j+LuANwDvUEqpcmOVUoPm/yPAA8DV5qlhEekxX6cHGKkw/h6l1G6l1O6urvItHs83ViioiNEWUKPRaOoRN1FAAnwBOKyU+pTj+GuBPwJuU0rFK4xtFJFm6zHwGuCAefpbwLvMx+8CvjnfD3G+sUJBmwI+PFX6B2g0Gs2FihsN4HrgDuBmM5Rzn4jcCvw90Ixh1tknIv8IICK9IvKgOXY18LiIPAs8CXxXKfV989zHgVtE5Chwi/m8LrDqAWnzj0ajqWeqrmBKqceBctvcB8scs0w+t5qPjwOXV7huHHi165leQARNs4/OAdBoNPWMzgSeB5YJSFcC1Wg09YwWAPPA9gHohvAajaaO0QJgHljlIHQhOI1GU89oATAPbA1ACwCNRlPHaAEwD6w8AB0FpNFo6hktAOZByAoD1RqARqOpY7QAmAc6Ckij0SwHtACYB1oAaDSa5YAWAPOgkAimw0A1Gk39ogXAPLA0gMYGXQhOo9HUL1oAzIPd6zu488aNXH1Rx1JPRaPRaOaNNmLPg1DAy5/cum2pp6HRaDTnhNYANBqNZoWiBYBGo9GsULQA0Gg0mhWKFgAajUazQtECQKPRaFYoWgBoNBrNCkULAI1Go1mhaAGg0Wg0KxRRSi31HFwjItPA0XkObwWmz/PYc3nPVcDYPMcuxWc9l7H6Pi3+WH2f3LFc79N6pVTXrKNKqbr5B9xTT2PP8T331NNn1ffpwh6r75O+T+X+1ZsJ6Nt1NvZc3vNc0Pdp8d+3HscuxXvW49ileM8lGVtXJqCVhIjsUUrtXup5XOjo++QOfZ/csdLuU71pACuJe5Z6AnWCvk/u0PfJHSvqPmkNQKPRaFYoWgPQaDSaFYoWAOcREfmiiIyIyAHHsctF5Oci8pyIfFtEWszjt4jI0+bxp0XkZseYq8zjL4rI/xURWYrPs1jUcp8c59eJSFRE3u849mMReV5E9pn/us/n51hsavw+vcNxH/aJSF5Edpnn9PepcJ8CIvIl8/izIvIqx5jl932ab/iQ/jevcK0bgSuBA45jTwGvNB+/G/io+fgKoNd8fBkw4BjzJHAdIMD3gNct9WdbqvvkOH8f8A3g/Y5jPwZ2L/XnuZDuk3l8B3Bcf5/K/u7uAr5kPu4GngY8y/X7pDWA84hS6lFgouTwVuBR8/HDwO3mtc8opQbN4weBoIg0iEgP0KKU+rkyvpX/DLx50Sd/HqnlPgGIyJuB4xj3acVQ631y8OvAVwH09wkovk/bgR+Z40aAKWDZRgVpAbD0HABuMx//CrC2zDW3A88opVJAH3DGce6MeWy5U/Y+iUgj8EfAX1QY9yVTXf+z5WbaqICb79PbMQUA+vsExffpWeBNIuITkYuAqyi+h8vq+6QFwNLzbuAuEXkaaAbSzpMicinwCeC/W4fKvMZKCOWqdJ/+Avi0UipaZsw7lFI7gBvMf3ecl5kuLdW+T9cAcaWUZQ/X36fi+/RFDCG4B/g/wM+ArHlu2X2fdFP4JUYpdQR4DYCIXAy83jonIv3AA8A7lVLHzMNngH7HS/QDgyxz5rhP1wBvE5G/AtqAvIgklVJ/r5QaMMfOiMhXgKsxTBzLlrm+Tya/RmH3D/r7VHSflFJZ4A+t60TkZ5j1x5bj90lrAEuMFUkgIh7gT4F/NJ+3Ad8F/lgp9VPreqXUEDAjIteaKug7gW+e73mfbyrdJ6XUDUqpDUqpDRg7tr9USv29qcKvMsf4gTdgqP3Lmkr3yXHsV4CvWcf092nW7y5smhURkVuArFLq0HL9PmkN4DwiIl8FXgWsEpEzwJ8DTSJyl3nJ/cCXzMd3A5uBPxORPzOPvcZ0TP0u8E9ACCNq43vn5QOcJ2q8T5VoAH5g/li9wA+Bzy3OjJeGedynG4EzSqnjJS+lv0+F+9SN8b3JAwMUzDzL8vukM4E1Go1mhaJNQBqNRrNC0QJAo9FoVihaAGg0Gs0KRQsAjUajWaFoAaDRaDQrFB0GqtEsACLyESCK0VD8IUcdJ43mgkVrABrNwvKbQO9ST0KjcYMWABrNPBGRD5n14X+IUV0SjMqR95oFw0JLOD2NpiraBKTRzAMRuQqjrs4VGL+jvRi14/dg9CTYs4TT02hcoQWARjM/bgAeUErFAUTkW0s8H42mZrQJSKOZP7qOiqau0QJAo5kfjwJvEZGQiDQDbzSPz2DUl9doLnh0MTiNZp6IyIcwyiefxKirfwh4CfhLIAFcp5RKLN0MNZq50QJAo9FoVijaBKTRaDQrFC0ANBqNZoWiBYBGo9GsULQA0Gg0mhWKFgAajUazQtECQKPRaFYoWgBoNBrNCkULAI1Go1mh/P+YnHad7xGEXgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "global_mean.to_pandas().sort_index().plot()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "id": "aec77cb1-6872-423c-a42c-264c022770ec",
   "metadata": {},
   "outputs": [],
   "source": [
    "del cpu_big, gpu_big, global_mean"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "30311f1b-d792-4915-b011-6e67d301d2b7",
   "metadata": {},
   "source": [
    "#### **cuDF-pandas** compatibility\n",
    "\n",
    "Some notable differences ([more here](https://docs.rapids.ai/api/cudf/stable/user_guide/pandas-comparison.html)) between cuDF and Pandas:\n",
    "* `.join` and `.groupby` operations do not guarantee the row order of the output with **cuDF**\n",
    "* Iteration over **cuDF** objects is not allowed, as GPUs are not efficient for sequential tasks\n",
    "* Storing arbitrary Python objects (e.g., a column with strings and lists) is not allowed in **cuDF**\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b55b358b-6e25-44fb-93ee-480a2c8d40e5",
   "metadata": {},
   "source": [
    "### Xarray - N-dimensional labeled arrays\n",
    "\n",
    "Xarray can organize layered annotated sets of array data into data structures that enable complex operations on these N-dimensional datasets. It couples tightly with netCDF/Zarr data and Dask and is very popular at NCAR. As demonstrated last week, it is possible to use CuPy arrays as the underlying data structure in Xarray, enabling offload to GPUs. Efforts to improve compatibility and usability are ongoing - reports of any issues or needed functionalities are desired by the maintainers! (see [this GitHub repo](https://github.com/xarray-contrib/cupy-xarray))\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "29c9fa25-d281-4b3a-a591-bf27b776545b",
   "metadata": {},
   "source": [
    "## Dask Resources\n",
    "\n",
    "[Dask Online Tutorial](https://tutorial.dask.org/)\n",
    "\n",
    "[Dask Distributed Documentation](https://distributed.dask.org/en/stable/)\n",
    "\n",
    "[dask-jobqueue Documentation](https://jobqueue.dask.org/en/latest/)\n",
    "\n",
    "[dask-cuda Documentation](https://docs.rapids.ai/api/dask-cuda/stable/)\n",
    "\n",
    "[How to: Use Dask with GPUs](https://docs.dask.org/en/stable/gpu.html)\n",
    "\n",
    "[dask-mpi and GPUs](http://mpi.dask.org/en/latest/gpu.html)\n",
    "\n",
    "[Dask on GPUs Speedups](https://blog.dask.org/2019/01/03/dask-array-gpus-first-steps)\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ba6c30ad-ae31-4178-895b-feefc090ea7c",
   "metadata": {},
   "source": [
    "\n",
    "## High-level Collections Resources\n",
    "\n",
    "[CuPy Documentation](https://docs.cupy.dev/en/stable/index.html)\n",
    "\n",
    "[cuDF Documentation](https://docs.rapids.ai/api/cudf/stable/)\n",
    "\n",
    "[cupy-xarray Quickstart](https://cupy-xarray.readthedocs.io/quickstart.html)\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "823b837c-eee1-49ba-86c3-e5743ed4edb5",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Running this notebook yourself\n",
    "### Notebook Setup\n",
    "\n",
    "This notebook will require running within a JupyterHub PBS Batch (interactive) session.  Much of the Python code will fail if a GPU is not detected on the node.  Select the PBS Batch option when launching from JupyterHub and set the `PROJECT` code to a currently active project, ie `UCIS0004` for the GPU workshop, and `QUEUE` to the appropriate routing queue depending on if during weekday 8am to 5:30pm MT (`gpudev`), or all other times (`casper`). Request `30` GB of memory for this session. Due to limited shared GPU resources, please use `GPU_TYPE=gp100` during the workshop. Otherwise, set `GPU_TYPE=v100` (required for `gpudev`) for independent work. See [Casper queue documentation](https://arc.ucar.edu/knowledge_base/72581396#StartingCasperjobswithPBS-Concurrentresourcelimits) for more info.\n",
    "\n",
    "If running this notebook outside of the NCAR computing environment, at least one compatible NVIDIA GPU is required to run most of the local cuDF and Dask examples - and access to a batch cluster with GPUs is required for the Dask Distributed examples.\n",
    "\n",
    "### Changing Notebook Kernel\n",
    "\n",
    "You'll need to use a JupyterHub kernel that features the RAPIDS suite of packages (at least `CuPy`, `cuDF`, and various `Dask` packages).  There are several ways to do this but you can navigate to the `Kernel` dropdown and select `Change Kernel`.  Select **GPU Workshop** kernel from the dropdown to run the CuPy examples in this notebook."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e639a5f1-9551-41af-b4ec-29836da8ee9f",
   "metadata": {
    "tags": []
   },
   "source": [
    "### Python Virtual Environment Setup\n",
    "\n",
    "You can set up your own virtual environment for running the Python code outside of this notebook.  This will also be useful if you would like to create your own virtual environment for GPU programming experimentation with Python.\n",
    "\n",
    "See [Python Conda environment documentation](https://kb.ucar.edu/display/RC/Using+conda+environments+for+Python+access) for background on using Conda on NCAR clusters. Use conda (or mamba for faster package solving) to create your own GPU-enabled environment. Here are the steps for doing so on Casper:"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fcd9c63f-414a-4511-943f-975b8b6aa977",
   "metadata": {
    "tags": []
   },
   "source": [
    "```bash\n",
    "# Conda used for virtual environment in NCAR clusters\n",
    "module load conda/latest\n",
    "\n",
    "# Create the environment and the populate it with packages\n",
    "# Note that we need channel priority flexible for RAPIDS to install\n",
    "conda create -n pygpu-dask\n",
    "conda activate pygpu-dask\n",
    "conda config --env --set channel_priority flexible\n",
    "conda env update --file envs/environment.yml\n",
    "\n",
    "# Don't forget to deactivate with \"conda deactivate\"\n",
    "\n",
    "```\n",
    "\n",
    "If running on other systems, you should install Jupyter as well so that you may use notebooks."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "GPU Workshop",
   "language": "python",
   "name": "pygpu"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}