energy/03_energy_efficient_computing.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hp8vD1uduApN"
      },
      "source": [
        "# Energy Efficient GPU Computing - Tutorial\n",
        "\n",
        "## Hands-on: Optimizing GPU Core Clock Frequency\n",
        "\n",
        "\n",
        "We again start by installing Kernel Tuner into the notebook. We also download the data that we will use in this notebook."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "vscode": {
          "languageId": "shellscript"
        },
        "id": "IeLZfTWruApP"
      },
      "outputs": [],
      "source": [
        "%%capture\n",
        "%pip install kernel-tuner[tutorial]==1.0.0b5\n",
        "%pip install seaborn~=0.13.0\n",
        "\n",
        "!wget -O GEMM_A100_cache.json.bz2 \"https://github.com/KernelTuner/kernel_tuner_tutorial/blob/master/energy/data/GEMM_NVML_NVIDIA_A100-PCIE-40GB_freq_cache_fake_timings.json.bz2?raw=true\"\n",
        "!bunzip2 GEMM_A100_cache.json.bz2"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cy9cJip4uApP"
      },
      "source": [
        "Next, we import the required packages and set our defaults for plotting with Seaborn and Matplotlib:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "_3TSusZUuApQ"
      },
      "outputs": [],
      "source": [
        "import kernel_tuner as kt\n",
        "import pandas as pd\n",
        "from matplotlib import pyplot as plt\n",
        "import seaborn as sns\n",
        "\n",
        "plt.rcParams['figure.figsize'] = [15, 8]\n",
        "sns.set_theme(style=\"whitegrid\", palette=\"muted\")\n",
        "sns.set_context(\"paper\", rc={\"font.size\":10,\"axes.titlesize\":9,\"axes.labelsize\":12})\n",
        "sns.set(font_scale = 1.6)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "b6j7IA2VuApQ"
      },
      "source": [
        "### Tuning Setup\n",
        "\n",
        "In this notebook, we will simulate the auto-tuning process of a highly-tunable matrix multiplication kernel. We start the tuning setup by defining the metrics we will use to measure performance."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "S4H5sNsZuApR"
      },
      "outputs": [],
      "source": [
        "# Compute the number of operations that the matrix multiply performs\n",
        "def ops(m, n, k):\n",
        "    return (m * n * k * 2 + 2 * m * k) / 1e9\n",
        "\n",
        "# Size of the matrices\n",
        "m = n = k = 4096\n",
        "problem_size = (512, 512)\n",
        "total_flops = ops(m, n, k)\n",
        "\n",
        "metrics = dict()\n",
        "# Throughput\n",
        "metrics[\"GFLOP/s\"] = lambda p: total_flops / (p[\"time\"] / 1000.0)\n",
        "# Energy efficiency\n",
        "metrics[\"GFLOPS/W\"] = lambda p: total_flops / p[\"nvml_energy\"]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BnUxr9qUuApR"
      },
      "source": [
        "Next, we define the parameters we would like to tune, their possible values, and restrictions that apply, creating the search space of possible configurations.\n",
        "\n",
        "This includes the GPU core clock frequencies that we would like to test for each kernel configuration in the search space.\n",
        "These are specified to Kernel Tuner with the special parameter name \"nvml_gr_clock\". Setting the GPU clock frequencies requires root privileges however. *That is why we will be simulating the tuning process for this hands-on.*\n",
        "\n",
        "Kernel Tuner can simulate the optimization process for search spaces that have already been fully explored. The performance data collected by Kernel Tuner is stored in the data file that we have downloaded by running the first cell in this notebook."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "AAkJKwcruApT"
      },
      "outputs": [],
      "source": [
        "# Tunable parameters\n",
        "tune_params = dict()\n",
        "# The nvml_gr_clock is the tunable parameter affecting the GPU frequency in MHz, 690 is closest to the baseclock of 765\n",
        "tune_params[\"nvml_gr_clock\"] = [330, 510, 690, 870, 1050, 1230, 1410]\n",
        "\n",
        "# Parameters needed for the code\n",
        "tune_params[\"MWG\"] = [16, 32, 64, 128]\n",
        "tune_params[\"NWG\"] = [16, 32, 64, 128]\n",
        "tune_params[\"KWG\"] = [32]\n",
        "tune_params[\"MDIMC\"] = [8, 16, 32]\n",
        "tune_params[\"NDIMC\"] = [8, 16, 32]\n",
        "tune_params[\"MDIMA\"] = [8, 16, 32]\n",
        "tune_params[\"NDIMB\"] = [8, 16, 32]\n",
        "tune_params[\"KWI\"] = [2]\n",
        "tune_params[\"VWM\"] = [1, 2, 4, 8]\n",
        "tune_params[\"VWN\"] = [1, 2, 4, 8]\n",
        "tune_params[\"STRM\"] = [0]\n",
        "tune_params[\"STRN\"] = [0]\n",
        "tune_params[\"SA\"] = [0, 1]\n",
        "tune_params[\"SB\"] = [0, 1]\n",
        "tune_params[\"PRECISION\"] = [32]\n",
        "\n",
        "# Grid size\n",
        "grid_div_x = [\"MWG\"]\n",
        "grid_div_y = [\"NWG\"]\n",
        "block_size_names = [\"MDIMC\", \"NDIMC\"]\n",
        "\n",
        "# Search space restriction\n",
        "restrict = []\n",
        "restrict += [\"KWG % KWI == 0\"]\n",
        "restrict += [\"MWG % (MDIMC * VWM) == 0\"]\n",
        "restrict += [\"NWG % (NDIMC * VWN) == 0\"]\n",
        "restrict += [\"MWG % (MDIMA * VWM) == 0\"]\n",
        "restrict += [\"NWG % (NDIMB * VWN) == 0\"]\n",
        "restrict += [\"KWG % ((MDIMC * NDIMC)/MDIMA) == 0\"]\n",
        "restrict += [\"KWG % ((MDIMC * NDIMC)/NDIMB) == 0\"]\n",
        "restrict += [\"not (MWG == 128 and NWG == 128 and MDIMC == 8 and NDIMC == 8)\"]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "abeXlhVRuApU"
      },
      "source": [
        "Next, we define a function that wraps the call to tune_kernel, because we will be calling it multiple times in this notebook. The function returns the best configuration, according to the current optimization objective, found during the tuning run."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "wUaA1JdjuApU"
      },
      "outputs": [],
      "source": [
        "def get_optimal_config(objective: str, tune_parameters: dict, higher_is_better=True, strategy='genetic_algorithm', fevals=200) -> tuple[dict, list]:\n",
        "    res_opt, env_opt = kt.tune_kernel(\n",
        "        \"Xgemm\",\n",
        "        \"\",\n",
        "        problem_size,\n",
        "        [],\n",
        "        tune_parameters,\n",
        "        block_size_names=block_size_names.copy(),\n",
        "        simulation_mode=True,\n",
        "        restrictions=restrict,\n",
        "        grid_div_x=grid_div_x,\n",
        "        grid_div_y=grid_div_y,\n",
        "        strategy=strategy,\n",
        "        strategy_options=dict(max_fevals=fevals),\n",
        "        metrics=metrics,\n",
        "        objective=objective,\n",
        "        objective_higher_is_better=higher_is_better,\n",
        "        cache=\"GEMM_A100_cache.json\",\n",
        "        quiet=True\n",
        "    )\n",
        "    return env_opt['best_config'], res_opt"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HU19-v4huApV"
      },
      "source": [
        "Now that we have our function to easily call tune_kernel, let's have a look at tuning our matrix multiply kernel for different optimization objectives.\n",
        "\n",
        "\n",
        "### Tuning for Time\n",
        "\n",
        "We start by simply optimizing for the lowest possible kernel execution time."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "yfrR0NcnuApV"
      },
      "outputs": [],
      "source": [
        "config_race_to_idle, res_race_to_idle = get_optimal_config(\"time\", tune_params, higher_is_better=False)\n",
        "config_race_to_idle['name'] = \"race-to-idle (global)\""
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "z0Xe8pjcuApV"
      },
      "source": [
        "### Tuning for Time and Energy\n",
        "\n",
        "The next step is to use our previous time-optimal configuration, and re-tune only the clock frequencies for energy efficiency. The idea behind this step is that we would like to use the best performing kernel configuration, but run it at the most energy-friendly clock frequency."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "yhintaD2uApW"
      },
      "outputs": [],
      "source": [
        "# create a new dictionary of tunable parameters with only the clock frequencies\n",
        "tune_params_only_clocks = tune_params.copy()\n",
        "for key, value in config_race_to_idle.items():\n",
        "    if key != \"nvml_gr_clock\" and key in tune_params_only_clocks:\n",
        "        tune_params_only_clocks[key] = [value]\n",
        "\n",
        "# tune the clock frequencies for energy efficiency\n",
        "config_race_to_idle_plus_clocks, res_race_to_idle_plus_clocks = get_optimal_config('GFLOPS/W', tune_params_only_clocks, strategy='brute_force')\n",
        "config_race_to_idle_plus_clocks['name'] = \"race-to-idle + clocks\""
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "uNTLasHxuApW"
      },
      "source": [
        "### Tuning for Energy\n",
        "\n",
        "The final step is to tune for energy efficiency. This optimization step attempts to find the configuration with the highest energy efficiency in the whole search space."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "EljuTln_uApW"
      },
      "outputs": [],
      "source": [
        "config_energy_to_solution, res_energy_to_solution = get_optimal_config(\"GFLOPS/W\", tune_params)\n",
        "config_energy_to_solution['name'] = \"energy-to-solution (global)\""
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kVW1pBFduApX"
      },
      "source": [
        "### Plotting the Results\n",
        "\n",
        "We can now look at the results in terms of energy efficiency per configuration in the bar chart below:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "9qyoDJTDuApX"
      },
      "outputs": [],
      "source": [
        "df = pd.DataFrame([config_race_to_idle, config_race_to_idle_plus_clocks, config_energy_to_solution])\n",
        "sns.barplot(x=df.nvml_energy, y=df.name, orient='h', hue=df.name, legend=False)\n",
        "plt.xlabel('Energy (J), lower is better')\n",
        "plt.ylabel('')\n",
        "plt.title('Lowest energy configuration')\n",
        "plt.tight_layout()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MprG7Y11uApX"
      },
      "source": [
        "Finally, we can also make a scatterplot to show the relation between energy and time of our three different optimization efforts.\n",
        "\n",
        "In addition, we can see the difference in how Kernel Tuner optimizes for performance and energy efficiency, and how the optimization algorithm improves over time, by plotting the configurations that have been explored during the optimization process."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "YWX-LJSZuApX"
      },
      "outputs": [],
      "source": [
        "# plot our three configurations\n",
        "sns.scatterplot(x=df['GFLOPS/W'], y=df['GFLOP/s'], hue=df.name, s=250)\n",
        "\n",
        "# plot the configurations tried when optimizing for time\n",
        "df_time = pd.DataFrame(res_race_to_idle)\n",
        "sns.scatterplot(x=df_time['GFLOPS/W'], y=df_time['GFLOP/s'], hue=df_time.index, alpha=0.6, palette=sns.light_palette(\"midnightblue\", as_cmap=True), legend=False)\n",
        "\n",
        "df_time_clocks = pd.DataFrame(res_race_to_idle_plus_clocks)\n",
        "sns.scatterplot(x=df_time_clocks['GFLOPS/W'], y=df_time_clocks['GFLOP/s'], alpha=0.9, color=\"orange\", legend=False)\n",
        "\n",
        "# plot the configurations tried when optimizing for energy\n",
        "df_energy = pd.DataFrame(res_energy_to_solution)\n",
        "sns.scatterplot(x=df_energy['GFLOPS/W'], y=df_energy['GFLOP/s'], hue=df_energy.index, alpha=0.6, palette=sns.light_palette(\"forestgreen\", as_cmap=True), legend=False)\n",
        "\n",
        "# finishing touches to the plot\n",
        "plt.xlabel('GFLOPs per Watt, higher is better')\n",
        "plt.ylabel('GFLOPs per second, higher is better')\n",
        "plt.title('Energy versus Time')\n",
        "plt.legend(title='', loc=\"upper left\")\n",
        "plt.tight_layout()"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "**Optional TODO:** Feel free to experiment, for example by running `get_optimal_config` again with more function evaluations or with different optimization strategies by passing different strategies in the `strategy=` option.\n",
        "\n",
        "You can also try to define your own metrics and objectives. Why not optimize for time and energy in one go by creating an energy-delay-product metric?\n",
        "\n",
        "**That's it! You've successfully completed the fourth hands-on!**"
      ],
      "metadata": {
        "id": "sdCHw31gy3YH"
      }
    },
    {
      "cell_type": "code",
      "source": [],
      "metadata": {
        "id": "trX-kI1Ry7Q4"
      },
      "execution_count": null,
      "outputs": []
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "kerneltuner_bayesopt",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.9.9"
    },
    "orig_nbformat": 4,
    "colab": {
      "provenance": []
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}