examples/cupti_flops_analysis.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "25f114ee",
   "metadata": {},
   "source": [
    "## CUPTI Counter / FLOPs Analysis\n",
    "\n",
    "### About\n",
    "\n",
    "In this demo we leverage the PyTorch Profiler to capture performance characteristics of CUDA kernels. See the section below on how to collect counters using PyTorch Profiler.\n",
    "\n",
    "### Motivation and context\n",
    "\n",
    "Performance counter measurements can provide insights on how to speed up GPU kernels, conduct roofline analysis and other low level optimizations. The PyTorch Profiler includes a lightweight API to program and measure detailed performance counters from the GPU. This mode leverages [CUPTI Range Profiler API](https://docs.nvidia.com/cupti/r_main.html#r_profiler) and supports an extensive list of performance metrics.\n",
    "\n",
    "The annotated trace contains:\n",
    "* Performance measurement events, which are logged under the `cuda_profiler_range` category.\n",
    "* Counter values, which are logged in the args section of the above events.\n",
    "\n",
    "\n",
    "### Instructions\n",
    "\n",
    "#### Collecting the trace with CUPTI Profiler Counters\n",
    "One can collect performance metrics by adding the list of metrics using the experimental config option in PyTorch Profiler.\n",
    "\n",
    "```\n",
    "with torch.profiler.profile(\n",
    "    activities=[torch.profiler.ProfilerActivity.CUDA, torch.profiler.ProfilerActivity.CPU],\n",
    "    record_shapes=True,\n",
    "    on_trace_ready=trace_handler,\n",
    "    experimental_config=torch.profiler._ExperimentalConfig(\n",
    "        profiler_metrics=[\n",
    "            \"kineto__tensor_core_insts\",\n",
    "            \"dram__bytes_read.sum\",\n",
    "            \"dram__bytes_write.sum\"],\n",
    "    profiler_measure_per_kernel=True),\n",
    ") as prof:\n",
    "    res = train_batch(modeldef)\n",
    "    prof.step()```\n",
    "```\n",
    "\n",
    "To collect the trace used in the example we ran [PARAM Benchmarks](https://github.com/facebookresearch/param/tree/main/train/compute/python). PARAM provides a repository of communication and computation micro-benchmarks for AI training and inference. For this example, we ran a simple convolutional neural network model - AlexNet - as a benchmark and collected the trace. Instructions for the same are shown below-\n",
    "\n",
    "Run using the following commands:\n",
    "\n",
    "```\n",
    "# Inside dir \"param/train/compute\"\n",
    "$ python -m python.pytorch.run_benchmark -c python/examples/pytorch/configs/alex_net.json -p -i 1 -d cuda --cupti-profiler --cupti-profiler-measure-per-kernel\n",
    "```\n",
    "\n",
    "#### Trace Analysis\n",
    "\n",
    "To run this demo notebook on your laptop\n",
    "1. Clone the repo `git clone https://github.com/facebookresearch/HolisticTraceAnalysis.git`\n",
    "1. [Optional and recommended] Setup a venv or conda environment. See README for details.\n",
    "1. Set the `trace_dir` parameter in the next cell to the location of the folder containing your collected PyTorch Profiler trace.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "c8e32f5d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2023-09-07 00:43:28,711 - hta - trace.py:L404 - INFO - /Users/bcoutinho/Work/hta/debug_cupti_yang\n",
      "2023-09-07 00:43:28,749 - hta - trace_file.py:L94 - INFO - Rank to trace file map:\n",
      "{0: '/Users/bcoutinho/Work/hta/debug_cupti_yang/libkineto_activities_3660455.json'}\n",
      "2023-09-07 00:43:28,751 - hta - trace.py:L550 - INFO - ranks=[0]\n",
      "2023-09-07 00:43:28,764 - hta - trace.py:L132 - INFO - Parsed /Users/bcoutinho/Work/hta/debug_cupti_yang/libkineto_activities_3660455.json time = 0.01 seconds \n",
      "2023-09-07 00:43:28,821 - hta - trace.py:L690 - WARNING - There is only one iteration in the trace. The analysis result may not be accurate.\n"
     ]
    }
   ],
   "source": [
    "from hta.trace_analysis import TraceAnalysis\n",
    "from hta.analyzers.cupti_counter_analysis import CUDA_SASS_INSTRUCTION_COUNTER_FLOPS\n",
    "\n",
    "#trace_prefix = # ENTER PATH TO HTA HERE\n",
    "trace_dir = \"~/Work/hta/debug_cupti_yang/\"\n",
    "analyzer = TraceAnalysis(trace_dir=trace_dir)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "bcf09da3",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\u001b[0;31mSignature:\u001b[0m\n",
       "\u001b[0manalyzer\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mget_cupti_counter_data_with_operators\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m    \u001b[0mranks\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mOptional\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mList\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mint\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
       "\u001b[0;34m\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m->\u001b[0m \u001b[0mList\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mpandas\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcore\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mframe\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mDataFrame\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
       "\u001b[0;31mDocstring:\u001b[0m\n",
       "Performance counters provide insights on how to speed up GPU\n",
       "kernels. The PyTorch Profiler has a lightweight API [CUPTI Range\n",
       "Profiler API](https://docs.nvidia.com/cupti/r_main.html#r_profiler)\n",
       "that enables users to monitor performance counters from the device.\n",
       "\n",
       "When the CUPTI Profiler mode is enabled then PyTorch will emit the\n",
       "performance counters and annotates them in the trace.\n",
       "    * The events are logged under the `cuda_profiler_range` category.\n",
       "    * Counter values are logged in the `args` section of the trace.\n",
       "\n",
       "This API can investigate performance measurements per kernel and\n",
       "associate them to operators that the kernel belongs to. A single kernel\n",
       "can map to multiple levels of operators (as operators can be nested).\n",
       "To represent this we basically provide a list column called `op_stack`.\n",
       "For further convenience we add the top and bottom level operator column\n",
       "as well.\n",
       "\n",
       "Args:\n",
       "    ranks (List[int]): List of ranks on which to run the analysis. Default = [0].\n",
       "Returns:\n",
       "    List[pd.DataFrame]\n",
       "        A list of dataframes, one per rank, containing kernel name,\n",
       "        op_stack (operator stack), top and bottom level op, and columns\n",
       "        for individual performance counters.\n",
       "\u001b[0;31mFile:\u001b[0m      ~/Work/hta/HolisticTraceAnalysis2/hta/trace_analysis.py\n",
       "\u001b[0;31mType:\u001b[0m      method"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "analyzer.get_cupti_counter_data_with_operators?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "49759168",
   "metadata": {},
   "outputs": [],
   "source": [
    "gpu_kernels = analyzer.get_cupti_counter_data_with_operators(ranks=[0])[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "a9f64c11-c6b0-467b-87d8-1a2859b73fd2",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "index                                                                                               635\n",
       "cat                                                                                 cuda_profiler_range\n",
       "name                                                                   ampere_fp16_sgemm_fp16_32x128_tn\n",
       "pid                                                                                                   7\n",
       "tid                                                                                                   0\n",
       "ts                                                                                                38473\n",
       "dur                                                                                             3619088\n",
       "Trace iteration                                                                                      -1\n",
       "dram__bytes_read.sum                                                                                  0\n",
       "memory_bw_gbps                                                                                       -1\n",
       "lts__throughput.avg.pct_of_peak_sustained_elapsed                                             21.189254\n",
       "l1tex__m_xbar2l1tex_read_bytes.sum                                                              7347264\n",
       "dram__throughput.avg.pct_of_peak_sustained_elapsed                                             0.002168\n",
       "sm__warps_active.avg.pct_of_peak_sustained_active                                             46.889965\n",
       "sm__inst_executed_pipe_tensor.sum                                                                     0\n",
       "l1tex__m_l1tex2xbar_write_bytes.sum                                                            18846560\n",
       "dram__bytes_write.sum                                                                              1792\n",
       "correlation                                                                                          -1\n",
       "sm__throughput.avg.pct_of_peak_sustained_elapsed                                              67.645904\n",
       "External id                                                                                          -1\n",
       "stream                                                                                               -1\n",
       "index_correlation                                                                                    -1\n",
       "iteration                                                                                             0\n",
       "depth                                                                                                 0\n",
       "index_runtime                                                                                       547\n",
       "op_stack                                              [cudaLaunchKernel, aten::addmm, aten::linear, ...\n",
       "top_level_op                                                                               aten::linear\n",
       "bottom_level_op                                                                             aten::addmm\n",
       "Name: 0, dtype: object"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gpu_kernels.loc[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "05225cbe",
   "metadata": {},
   "outputs": [
    {
     "ename": "KeyError",
     "evalue": "\"['smsp__sass_thread_inst_executed_op_ffma_pred_on.sum', 'smsp__sass_thread_inst_executed_op_fmul_pred_on.sum', 'smsp__sass_thread_inst_executed_op_fadd_pred_on.sum', 'smsp__sass_thread_inst_executed_op_hfma_pred_on.sum', 'smsp__sass_thread_inst_executed_op_hmul_pred_on.sum', 'smsp__sass_thread_inst_executed_op_hadd_pred_on.sum', 'smsp__sass_thread_inst_executed_op_dfma_pred_on.sum', 'smsp__sass_thread_inst_executed_op_dmul_pred_on.sum', 'smsp__sass_thread_inst_executed_op_dadd_pred_on.sum'] not in index\"",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mKeyError\u001b[0m                                  Traceback (most recent call last)",
      "Cell \u001b[0;32mIn[6], line 1\u001b[0m\n\u001b[0;32m----> 1\u001b[0m \u001b[43mgpu_kernels\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mhead\u001b[49m\u001b[43m(\u001b[49m\u001b[43m)\u001b[49m\u001b[43m[\u001b[49m\u001b[43m[\u001b[49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[38;5;124;43mname\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[38;5;124;43mop_stack\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[38;5;124;43mtop_level_op\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[43m]\u001b[49m\u001b[43m\\\u001b[49m\n\u001b[1;32m      2\u001b[0m \u001b[43m                   \u001b[49m\u001b[38;5;241;43m+\u001b[39;49m\u001b[43m \u001b[49m\u001b[38;5;28;43mlist\u001b[39;49m\u001b[43m(\u001b[49m\u001b[43mCUDA_SASS_INSTRUCTION_COUNTER_FLOPS\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mkeys\u001b[49m\u001b[43m(\u001b[49m\u001b[43m)\u001b[49m\u001b[43m)\u001b[49m\u001b[43m]\u001b[49m\n",
      "File \u001b[0;32m/opt/miniconda3/envs/hta/lib/python3.9/site-packages/pandas/core/frame.py:3767\u001b[0m, in \u001b[0;36mDataFrame.__getitem__\u001b[0;34m(self, key)\u001b[0m\n\u001b[1;32m   3765\u001b[0m     \u001b[38;5;28;01mif\u001b[39;00m is_iterator(key):\n\u001b[1;32m   3766\u001b[0m         key \u001b[38;5;241m=\u001b[39m \u001b[38;5;28mlist\u001b[39m(key)\n\u001b[0;32m-> 3767\u001b[0m     indexer \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mcolumns\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43m_get_indexer_strict\u001b[49m\u001b[43m(\u001b[49m\u001b[43mkey\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[38;5;124;43mcolumns\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[43m)\u001b[49m[\u001b[38;5;241m1\u001b[39m]\n\u001b[1;32m   3769\u001b[0m \u001b[38;5;66;03m# take() does not accept boolean indexers\u001b[39;00m\n\u001b[1;32m   3770\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;28mgetattr\u001b[39m(indexer, \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mdtype\u001b[39m\u001b[38;5;124m\"\u001b[39m, \u001b[38;5;28;01mNone\u001b[39;00m) \u001b[38;5;241m==\u001b[39m \u001b[38;5;28mbool\u001b[39m:\n",
      "File \u001b[0;32m/opt/miniconda3/envs/hta/lib/python3.9/site-packages/pandas/core/indexes/base.py:5877\u001b[0m, in \u001b[0;36mIndex._get_indexer_strict\u001b[0;34m(self, key, axis_name)\u001b[0m\n\u001b[1;32m   5874\u001b[0m \u001b[38;5;28;01melse\u001b[39;00m:\n\u001b[1;32m   5875\u001b[0m     keyarr, indexer, new_indexer \u001b[38;5;241m=\u001b[39m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_reindex_non_unique(keyarr)\n\u001b[0;32m-> 5877\u001b[0m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43m_raise_if_missing\u001b[49m\u001b[43m(\u001b[49m\u001b[43mkeyarr\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mindexer\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43maxis_name\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m   5879\u001b[0m keyarr \u001b[38;5;241m=\u001b[39m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mtake(indexer)\n\u001b[1;32m   5880\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;28misinstance\u001b[39m(key, Index):\n\u001b[1;32m   5881\u001b[0m     \u001b[38;5;66;03m# GH 42790 - Preserve name from an Index\u001b[39;00m\n",
      "File \u001b[0;32m/opt/miniconda3/envs/hta/lib/python3.9/site-packages/pandas/core/indexes/base.py:5941\u001b[0m, in \u001b[0;36mIndex._raise_if_missing\u001b[0;34m(self, key, indexer, axis_name)\u001b[0m\n\u001b[1;32m   5938\u001b[0m     \u001b[38;5;28;01mraise\u001b[39;00m \u001b[38;5;167;01mKeyError\u001b[39;00m(\u001b[38;5;124mf\u001b[39m\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mNone of [\u001b[39m\u001b[38;5;132;01m{\u001b[39;00mkey\u001b[38;5;132;01m}\u001b[39;00m\u001b[38;5;124m] are in the [\u001b[39m\u001b[38;5;132;01m{\u001b[39;00maxis_name\u001b[38;5;132;01m}\u001b[39;00m\u001b[38;5;124m]\u001b[39m\u001b[38;5;124m\"\u001b[39m)\n\u001b[1;32m   5940\u001b[0m not_found \u001b[38;5;241m=\u001b[39m \u001b[38;5;28mlist\u001b[39m(ensure_index(key)[missing_mask\u001b[38;5;241m.\u001b[39mnonzero()[\u001b[38;5;241m0\u001b[39m]]\u001b[38;5;241m.\u001b[39munique())\n\u001b[0;32m-> 5941\u001b[0m \u001b[38;5;28;01mraise\u001b[39;00m \u001b[38;5;167;01mKeyError\u001b[39;00m(\u001b[38;5;124mf\u001b[39m\u001b[38;5;124m\"\u001b[39m\u001b[38;5;132;01m{\u001b[39;00mnot_found\u001b[38;5;132;01m}\u001b[39;00m\u001b[38;5;124m not in index\u001b[39m\u001b[38;5;124m\"\u001b[39m)\n",
      "\u001b[0;31mKeyError\u001b[0m: \"['smsp__sass_thread_inst_executed_op_ffma_pred_on.sum', 'smsp__sass_thread_inst_executed_op_fmul_pred_on.sum', 'smsp__sass_thread_inst_executed_op_fadd_pred_on.sum', 'smsp__sass_thread_inst_executed_op_hfma_pred_on.sum', 'smsp__sass_thread_inst_executed_op_hmul_pred_on.sum', 'smsp__sass_thread_inst_executed_op_hadd_pred_on.sum', 'smsp__sass_thread_inst_executed_op_dfma_pred_on.sum', 'smsp__sass_thread_inst_executed_op_dmul_pred_on.sum', 'smsp__sass_thread_inst_executed_op_dadd_pred_on.sum'] not in index\""
     ]
    }
   ],
   "source": [
    "gpu_kernels.head()[[\"name\", \"op_stack\", \"top_level_op\"]\\\n",
    "                   + list(CUDA_SASS_INSTRUCTION_COUNTER_FLOPS.keys())]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "5489f0c6-83f0-4a00-9218-f1076b3f6b59",
   "metadata": {},
   "outputs": [],
   "source": [
    "gpu_kernels[\"flops\"] = 0\n",
    "for counter, flops in CUDA_SASS_INSTRUCTION_COUNTER_FLOPS.items():\n",
    "    gpu_kernels[\"flops\"] += gpu_kernels[counter] * flops"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "b002062f-72cc-474a-8d10-46dddb217915",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>name</th>\n",
       "      <th>bottom_level_op</th>\n",
       "      <th>top_level_op</th>\n",
       "      <th>flops</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>void at::native::(anonymous namespace)::distri...</td>\n",
       "      <td>aten::uniform_</td>\n",
       "      <td>aten::rand</td>\n",
       "      <td>87195648</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>__missing__</td>\n",
       "      <td>aten::convolution</td>\n",
       "      <td>aten::conv2d</td>\n",
       "      <td>18263449600</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>void at::native::elementwise_kernel&lt;128, 2, at...</td>\n",
       "      <td>aten::add_</td>\n",
       "      <td>aten::conv2d</td>\n",
       "      <td>148684800</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>void at::native::vectorized_elementwise_kernel...</td>\n",
       "      <td>aten::clamp_min_</td>\n",
       "      <td>aten::relu_</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>void at::native::(anonymous namespace)::max_po...</td>\n",
       "      <td>aten::max_pool2d_with_indices</td>\n",
       "      <td>aten::max_pool2d</td>\n",
       "      <td>11943936</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                name  \\\n",
       "0  void at::native::(anonymous namespace)::distri...   \n",
       "1                                        __missing__   \n",
       "2  void at::native::elementwise_kernel<128, 2, at...   \n",
       "3  void at::native::vectorized_elementwise_kernel...   \n",
       "4  void at::native::(anonymous namespace)::max_po...   \n",
       "\n",
       "                 bottom_level_op      top_level_op        flops  \n",
       "0                 aten::uniform_        aten::rand     87195648  \n",
       "1              aten::convolution      aten::conv2d  18263449600  \n",
       "2                     aten::add_      aten::conv2d    148684800  \n",
       "3               aten::clamp_min_       aten::relu_            0  \n",
       "4  aten::max_pool2d_with_indices  aten::max_pool2d     11943936  "
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gpu_kernels[[\"name\", \"bottom_level_op\", \"top_level_op\", \"flops\"]].head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6e5b10e5",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.17"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}