transformers.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "48c6600b",
   "metadata": {},
   "source": [
    "# Transformers Benchmarks\n",
    "\n",
    "This notebook evaluates BERT and GPT training performance on single and multiple GPUs. \n",
    "\n",
    "## Setup\n",
    "\n",
    "First check environment and all available GPUs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "5c9cd3b8",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Pytorch version\t: 1.13.0a0+08820cb\n",
      "CUDA version\t: 11.7\n",
      "GPU0\t\t: NVIDIA GeForce RTX 3090 Ti\n",
      "GPU1\t\t: NVIDIA GeForce RTX 3090 Ti\n"
     ]
    }
   ],
   "source": [
    "import torch\n",
    "\n",
    "print('Pytorch version\\t:', torch.__version__)\n",
    "print('CUDA version\\t:', torch.version.cuda)\n",
    "\n",
    "for i in range(torch.cuda.device_count()):\n",
    "    print(f'GPU{i}\\t\\t:',torch.cuda.get_device_name(i))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "53bfbb2e",
   "metadata": {},
   "source": [
    "Next install packages we need beyond pytorch. Note that both deepspeed and megatron-lm need `nvcc` to build custom operators. Make sure you have a complete CUDA installation rather than just runtime. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "2c9a07eb",
   "metadata": {},
   "outputs": [],
   "source": [
    "from IPython.display import clear_output\n",
    "\n",
    "!git clone https://github.com/huggingface/transformers\n",
    "!git clone https://github.com/NVIDIA/Megatron-LM\n",
    "!cd transformers; pip install .\n",
    "!pip install datasets evaluate accelerate deepspeed psutil\n",
    "\n",
    "clear_output()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2db7fb26",
   "metadata": {},
   "source": [
    "## Experiment \n",
    "\n",
    "The `Exp` class stores both hyperparameters and performance results for one experiment. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "fe4aa622",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import re\n",
    "import json\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "from dataclasses import dataclass, asdict\n",
    "from transformers import AutoConfig, PretrainedConfig\n",
    "\n",
    "@dataclass\n",
    "class Exp:\n",
    "    name: str           # Experiment name\n",
    "    model: str          # huggingface model name\n",
    "    batch_size: int     # batch size per GPU\n",
    "    seq_len: int = None # input sequence length\n",
    "        \n",
    "    ## Improve speed / reduce memory  \n",
    "    bf16: bool = False  # Faster, less memory. Recommend if GPU supports\n",
    "    fp16: bool = False  # Faster, less memory, but need to scale loos. \n",
    "                        # Recommend if BF16 is not available.\n",
    "    optim: str = 'adamw_hf'  # Optimization method\n",
    "    grad_ckpt: bool = False  # save memory with an extra forward\n",
    "    grad_accum: int = 1      # accumulate gradients for better performance\n",
    "    steps: int = 20          # number of parameter updates\n",
    "        \n",
    "    ## Multi-GPUs\n",
    "    gpus: str = '0'          # GPUs to use. \"0,1\" means use GPU 0 and 1\n",
    "    tensor_para: int = 1     # Tensor parallelism\n",
    "    deepspeed: bool = False  # if or not use deepspeed\n",
    "    ds_config: str = ''      # deepspeed config \n",
    "        \n",
    "    def __post_init__(self):         \n",
    "        model_conf = AutoConfig.from_pretrained(self.model)\n",
    "        get = lambda *keys: max([getattr(model_conf, k) if hasattr(model_conf, k) else 0 for k in keys])\n",
    "        self.num_layers = get('num_hidden_layers', 'n_layer')\n",
    "        self.num_gpus = len(self.gpus.split(','))                      \n",
    "        self.hidden_size = get('hidden_size', 'n_embd', 'd_model')\n",
    "        self.vocab_size = get('vocab_size')\n",
    "        self.num_heads = get('num_attention_heads', 'n_head')\n",
    "        if self.seq_len is None:\n",
    "            self.seq_len = get('max_position_embeddings', 'n_ctx')\n",
    "        n, h, s, v = self.num_layers, self.hidden_size, self.seq_len, self.vocab_size\n",
    "        att, ffn, embed = 4*h*s**2 + 8*s*h**2, 16*s*h**2, 2*s*h*v\n",
    "        forward = n*(att+ffn) + embed\n",
    "        # TFLOPs to train one example\n",
    "        self.tflops = (4 * forward if self.grad_ckpt else 3 * forward) / 1e12\n",
    "        if self.deepspeed:            \n",
    "            self.launcher = 'deepspeed'            \n",
    "        else:\n",
    "            self.launcher = f'torchrun --nproc_per_node {self.num_gpus}' \n",
    "            \n",
    "    def print_results(self):\n",
    "        print('Total samples / second\\t: %.1f' % self.samples_per_sec)\n",
    "        print('Per GPU memory (GB)\\t: %.1f'% self.gpu_mem)\n",
    "        print('Per GPU TFLOPs\\t\\t: %.1f' % (self.samples_per_sec * self.tflops / self.num_gpus))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f81b07aa",
   "metadata": {},
   "source": [
    "The following function visualize results among different experiments. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "de194e1c",
   "metadata": {},
   "outputs": [],
   "source": [
    "%config InlineBackend.figure_formats = ['svg']\n",
    "\n",
    "def compare(exps):\n",
    "    fig, ax = plt.subplots(ncols=3, figsize=(9,len(exps)/2))\n",
    "    x = list(range(len(exps)))\n",
    "    for i, (y, l) in enumerate((\n",
    "        ([e.samples_per_sec for e in exps], 'Samples / sec'), \n",
    "        ([e.samples_per_sec * e.tflops / e.num_gpus for e in exps], 'per GPU TFLOPS'),\n",
    "        ([e.gpu_mem for e in exps], 'per GPU memory (GB)'))):\n",
    "        ax[i].barh(x, y, align='center', height=0.6, color=plt.get_cmap('Set1')(x))\n",
    "        ax[i].invert_yaxis()\n",
    "        ax[i].set_xlabel(l)\n",
    "        if i == 0:\n",
    "            ax[i].set_yticks(x, labels=[e.name for e in exps])\n",
    "        else:\n",
    "            ax[i].set_yticklabels([])\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9ac25574",
   "metadata": {},
   "source": [
    "## BERT on a Single GPU with HuggingFace\n",
    "\n",
    "We use the [masked langunage modeling](https://github.com/huggingface/transformers/tree/main/examples/pytorch/language-modeling) task from Huggingface to evaluate BERT training. `hf_bert` runs the experiment and saves the log into `log.txt`. `hf_log` parses results from the log. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "59a16466",
   "metadata": {},
   "outputs": [],
   "source": [
    "def hf_bert(exp):\n",
    "    cmd = f'''export CUDA_VISIBLE_DEVICES={exp.gpus}; \\\n",
    "{exp.launcher} transformers/examples/pytorch/language-modeling/run_mlm.py \\\n",
    "--config_name {exp.model} --tokenizer_name {exp.model} \\\n",
    "--dataset_name wikitext --dataset_config_name wikitext-2-raw-v1 \\\n",
    "--do_train --max_seq_length {exp.seq_len} \\\n",
    "--per_device_train_batch_size {exp.batch_size} \\\n",
    "--fp16 {exp.fp16} --bf16 {exp.bf16} \\\n",
    "--optim {exp.optim} --max_steps {exp.steps} \\\n",
    "--gradient_accumulation_steps {exp.grad_accum} \\\n",
    "--gradient_checkpointing {exp.grad_ckpt} \\\n",
    "--output_dir /tmp/bert/ --overwrite_output_dir yes --skip_memory_metrics False'''\n",
    "    if exp.deepspeed:\n",
    "        cmd += f' --deepspeed {exp.ds_config}'\n",
    "    cmd += ' > log.txt 2>&1'\n",
    "    os.system(cmd)\n",
    "    return hf_log(exp, 'log.txt')\n",
    "    \n",
    "def hf_log(exp, log_filename):\n",
    "    with open(log_filename) as f:\n",
    "        lines = f.readlines()\n",
    "    for l in lines:\n",
    "        if 'CUDA out of memory' in l:\n",
    "            print('Out of GPU memory, try a smaller batch size')\n",
    "            return None\n",
    "        if '{\\'train_runtime' in l:\n",
    "            metrics = json.loads(l.replace('\\'', '\\\"'))\n",
    "            exp.gpu_mem = (metrics['init_mem_cpu_peaked_delta'] + \\\n",
    "                    metrics['train_mem_gpu_alloc_delta'] + metrics['train_mem_gpu_peaked_delta']) / 1e9\n",
    "            exp.samples_per_sec = metrics['train_samples_per_second']\n",
    "            return exp\n",
    "    print(f'Failed. Check \"{log_filename}\" to find error')    \n",
    "    return None"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a69d8688",
   "metadata": {},
   "source": [
    "First, let's train BERT large using its phase-2 sequence length 512. We choose the largest batch size that can fit into GPU memory for a good performance. In default, it uses `fp32` (or `tf32` if your GPU supports)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "4d6894f8",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Downloading: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 571/571 [00:00<00:00, 625kB/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total samples / second\t: 15.6\n",
      "Per GPU memory (GB)\t: 21.8\n",
      "Per GPU TFLOPs\t\t: 17.2\n"
     ]
    }
   ],
   "source": [
    "bert_single = hf_bert(Exp('HF 32-bit', 'bert-large-uncased', 8))\n",
    "bert_single.print_results()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "daf7b183",
   "metadata": {},
   "source": [
    "Now switch to `bf16` that offers a better performance. It also allows us to use a larger batch size, which further improves performance."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "19ec9a99",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       "  \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<svg xmlns:xlink=\"http://www.w3.org/1999/xlink\" width=\"569.429688pt\" height=\"99.11625pt\" viewBox=\"0 0 569.429688 99.11625\" xmlns=\"http://www.w3.org/2000/svg\" version=\"1.1\">\n",
       " <metadata>\n",
       "  <rdf:RDF xmlns:dc=\"http://purl.org/dc/elements/1.1/\" xmlns:cc=\"http://creativecommons.org/ns#\" xmlns:rdf=\"http://www.w3.org/1999/02/22-rdf-syntax-ns#\">\n",
       "   <cc:Work>\n",
       "    <dc:type rdf:resource=\"http://purl.org/dc/dcmitype/StillImage\"/>\n",
       "    <dc:date>2022-09-02T19:40:14.049688</dc:date>\n",
       "    <dc:format>image/svg+xml</dc:format>\n",
       "    <dc:creator>\n",
       "     <cc:Agent>\n",
       "      <dc:title>Matplotlib v3.5.2, https://matplotlib.org/</dc:title>\n",
       "     </cc:Agent>\n",
       "    </dc:creator>\n",
       "   </cc:Work>\n",
       "  </rdf:RDF>\n",
       " </metadata>\n",
       " <defs>\n",
       "  <style type=\"text/css\">*{stroke-linejoin: round; stroke-linecap: butt}</style>\n",
       " </defs>\n",
       " <g id=\"figure_1\">\n",
       "  <g id=\"patch_1\">\n",
       "   <path d=\"M 0 99.11625 \n",
       "L 569.429688 99.11625 \n",
       "L 569.429688 0 \n",
       "L 0 0 \n",
       "L 0 99.11625 \n",
       "z\n",
       "\" style=\"fill: none\"/>\n",
       "  </g>\n",
       "  <g id=\"axes_1\">\n",
       "   <g id=\"patch_2\">\n",
       "    <path d=\"M 60.029688 61.56 \n",
       "L 207.73557 61.56 \n",
       "L 207.73557 7.2 \n",
       "L 60.029688 7.2 \n",
       "z\n",
       "\" style=\"fill: #ffffff\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_3\">\n",
       "    <path d=\"M 60.029688 9.670909 \n",
       "L 151.957749 9.670909 \n",
       "L 151.957749 28.202727 \n",
       "L 60.029688 28.202727 \n",
       "z\n",
       "\" clip-path=\"url(#p8cab51f4bc)\" style=\"fill: #e41a1c\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_4\">\n",
       "    <path d=\"M 60.029688 40.557273 \n",
       "L 200.701956 40.557273 \n",
       "L 200.701956 59.089091 \n",
       "L 60.029688 59.089091 \n",
       "z\n",
       "\" clip-path=\"url(#p8cab51f4bc)\" style=\"fill: #377eb8\"/>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_1\">\n",
       "    <g id=\"xtick_1\">\n",
       "     <g id=\"line2d_1\">\n",
       "      <defs>\n",
       "       <path id=\"ma26dfc4f5b\" d=\"M 0 0 \n",
       "L 0 3.5 \n",
       "\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </defs>\n",
       "      <g>\n",
       "       <use xlink:href=\"#ma26dfc4f5b\" x=\"60.029688\" y=\"61.56\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_1\">\n",
       "      <!-- 0 -->\n",
       "      <g transform=\"translate(56.848438 76.158437)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-30\" d=\"M 2034 4250 \n",
       "Q 1547 4250 1301 3770 \n",
       "Q 1056 3291 1056 2328 \n",
       "Q 1056 1369 1301 889 \n",
       "Q 1547 409 2034 409 \n",
       "Q 2525 409 2770 889 \n",
       "Q 3016 1369 3016 2328 \n",
       "Q 3016 3291 2770 3770 \n",
       "Q 2525 4250 2034 4250 \n",
       "z\n",
       "M 2034 4750 \n",
       "Q 2819 4750 3233 4129 \n",
       "Q 3647 3509 3647 2328 \n",
       "Q 3647 1150 3233 529 \n",
       "Q 2819 -91 2034 -91 \n",
       "Q 1250 -91 836 529 \n",
       "Q 422 1150 422 2328 \n",
       "Q 422 3509 836 4129 \n",
       "Q 1250 4750 2034 4750 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_2\">\n",
       "     <g id=\"line2d_2\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#ma26dfc4f5b\" x=\"118.80722\" y=\"61.56\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_2\">\n",
       "      <!-- 10 -->\n",
       "      <g transform=\"translate(112.44472 76.158437)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-31\" d=\"M 794 531 \n",
       "L 1825 531 \n",
       "L 1825 4091 \n",
       "L 703 3866 \n",
       "L 703 4441 \n",
       "L 1819 4666 \n",
       "L 2450 4666 \n",
       "L 2450 531 \n",
       "L 3481 531 \n",
       "L 3481 0 \n",
       "L 794 0 \n",
       "L 794 531 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-31\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_3\">\n",
       "     <g id=\"line2d_3\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#ma26dfc4f5b\" x=\"177.584753\" y=\"61.56\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_3\">\n",
       "      <!-- 20 -->\n",
       "      <g transform=\"translate(171.222253 76.158437)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-32\" d=\"M 1228 531 \n",
       "L 3431 531 \n",
       "L 3431 0 \n",
       "L 469 0 \n",
       "L 469 531 \n",
       "Q 828 903 1448 1529 \n",
       "Q 2069 2156 2228 2338 \n",
       "Q 2531 2678 2651 2914 \n",
       "Q 2772 3150 2772 3378 \n",
       "Q 2772 3750 2511 3984 \n",
       "Q 2250 4219 1831 4219 \n",
       "Q 1534 4219 1204 4116 \n",
       "Q 875 4013 500 3803 \n",
       "L 500 4441 \n",
       "Q 881 4594 1212 4672 \n",
       "Q 1544 4750 1819 4750 \n",
       "Q 2544 4750 2975 4387 \n",
       "Q 3406 4025 3406 3419 \n",
       "Q 3406 3131 3298 2873 \n",
       "Q 3191 2616 2906 2266 \n",
       "Q 2828 2175 2409 1742 \n",
       "Q 1991 1309 1228 531 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-32\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"text_4\">\n",
       "     <!-- Samples / sec -->\n",
       "     <g transform=\"translate(99.236535 89.836562)scale(0.1 -0.1)\">\n",
       "      <defs>\n",
       "       <path id=\"DejaVuSans-53\" d=\"M 3425 4513 \n",
       "L 3425 3897 \n",
       "Q 3066 4069 2747 4153 \n",
       "Q 2428 4238 2131 4238 \n",
       "Q 1616 4238 1336 4038 \n",
       "Q 1056 3838 1056 3469 \n",
       "Q 1056 3159 1242 3001 \n",
       "Q 1428 2844 1947 2747 \n",
       "L 2328 2669 \n",
       "Q 3034 2534 3370 2195 \n",
       "Q 3706 1856 3706 1288 \n",
       "Q 3706 609 3251 259 \n",
       "Q 2797 -91 1919 -91 \n",
       "Q 1588 -91 1214 -16 \n",
       "Q 841 59 441 206 \n",
       "L 441 856 \n",
       "Q 825 641 1194 531 \n",
       "Q 1563 422 1919 422 \n",
       "Q 2459 422 2753 634 \n",
       "Q 3047 847 3047 1241 \n",
       "Q 3047 1584 2836 1778 \n",
       "Q 2625 1972 2144 2069 \n",
       "L 1759 2144 \n",
       "Q 1053 2284 737 2584 \n",
       "Q 422 2884 422 3419 \n",
       "Q 422 4038 858 4394 \n",
       "Q 1294 4750 2059 4750 \n",
       "Q 2388 4750 2728 4690 \n",
       "Q 3069 4631 3425 4513 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-61\" d=\"M 2194 1759 \n",
       "Q 1497 1759 1228 1600 \n",
       "Q 959 1441 959 1056 \n",
       "Q 959 750 1161 570 \n",
       "Q 1363 391 1709 391 \n",
       "Q 2188 391 2477 730 \n",
       "Q 2766 1069 2766 1631 \n",
       "L 2766 1759 \n",
       "L 2194 1759 \n",
       "z\n",
       "M 3341 1997 \n",
       "L 3341 0 \n",
       "L 2766 0 \n",
       "L 2766 531 \n",
       "Q 2569 213 2275 61 \n",
       "Q 1981 -91 1556 -91 \n",
       "Q 1019 -91 701 211 \n",
       "Q 384 513 384 1019 \n",
       "Q 384 1609 779 1909 \n",
       "Q 1175 2209 1959 2209 \n",
       "L 2766 2209 \n",
       "L 2766 2266 \n",
       "Q 2766 2663 2505 2880 \n",
       "Q 2244 3097 1772 3097 \n",
       "Q 1472 3097 1187 3025 \n",
       "Q 903 2953 641 2809 \n",
       "L 641 3341 \n",
       "Q 956 3463 1253 3523 \n",
       "Q 1550 3584 1831 3584 \n",
       "Q 2591 3584 2966 3190 \n",
       "Q 3341 2797 3341 1997 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-6d\" d=\"M 3328 2828 \n",
       "Q 3544 3216 3844 3400 \n",
       "Q 4144 3584 4550 3584 \n",
       "Q 5097 3584 5394 3201 \n",
       "Q 5691 2819 5691 2113 \n",
       "L 5691 0 \n",
       "L 5113 0 \n",
       "L 5113 2094 \n",
       "Q 5113 2597 4934 2840 \n",
       "Q 4756 3084 4391 3084 \n",
       "Q 3944 3084 3684 2787 \n",
       "Q 3425 2491 3425 1978 \n",
       "L 3425 0 \n",
       "L 2847 0 \n",
       "L 2847 2094 \n",
       "Q 2847 2600 2669 2842 \n",
       "Q 2491 3084 2119 3084 \n",
       "Q 1678 3084 1418 2786 \n",
       "Q 1159 2488 1159 1978 \n",
       "L 1159 0 \n",
       "L 581 0 \n",
       "L 581 3500 \n",
       "L 1159 3500 \n",
       "L 1159 2956 \n",
       "Q 1356 3278 1631 3431 \n",
       "Q 1906 3584 2284 3584 \n",
       "Q 2666 3584 2933 3390 \n",
       "Q 3200 3197 3328 2828 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-70\" d=\"M 1159 525 \n",
       "L 1159 -1331 \n",
       "L 581 -1331 \n",
       "L 581 3500 \n",
       "L 1159 3500 \n",
       "L 1159 2969 \n",
       "Q 1341 3281 1617 3432 \n",
       "Q 1894 3584 2278 3584 \n",
       "Q 2916 3584 3314 3078 \n",
       "Q 3713 2572 3713 1747 \n",
       "Q 3713 922 3314 415 \n",
       "Q 2916 -91 2278 -91 \n",
       "Q 1894 -91 1617 61 \n",
       "Q 1341 213 1159 525 \n",
       "z\n",
       "M 3116 1747 \n",
       "Q 3116 2381 2855 2742 \n",
       "Q 2594 3103 2138 3103 \n",
       "Q 1681 3103 1420 2742 \n",
       "Q 1159 2381 1159 1747 \n",
       "Q 1159 1113 1420 752 \n",
       "Q 1681 391 2138 391 \n",
       "Q 2594 391 2855 752 \n",
       "Q 3116 1113 3116 1747 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-6c\" d=\"M 603 4863 \n",
       "L 1178 4863 \n",
       "L 1178 0 \n",
       "L 603 0 \n",
       "L 603 4863 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-65\" d=\"M 3597 1894 \n",
       "L 3597 1613 \n",
       "L 953 1613 \n",
       "Q 991 1019 1311 708 \n",
       "Q 1631 397 2203 397 \n",
       "Q 2534 397 2845 478 \n",
       "Q 3156 559 3463 722 \n",
       "L 3463 178 \n",
       "Q 3153 47 2828 -22 \n",
       "Q 2503 -91 2169 -91 \n",
       "Q 1331 -91 842 396 \n",
       "Q 353 884 353 1716 \n",
       "Q 353 2575 817 3079 \n",
       "Q 1281 3584 2069 3584 \n",
       "Q 2775 3584 3186 3129 \n",
       "Q 3597 2675 3597 1894 \n",
       "z\n",
       "M 3022 2063 \n",
       "Q 3016 2534 2758 2815 \n",
       "Q 2500 3097 2075 3097 \n",
       "Q 1594 3097 1305 2825 \n",
       "Q 1016 2553 972 2059 \n",
       "L 3022 2063 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-73\" d=\"M 2834 3397 \n",
       "L 2834 2853 \n",
       "Q 2591 2978 2328 3040 \n",
       "Q 2066 3103 1784 3103 \n",
       "Q 1356 3103 1142 2972 \n",
       "Q 928 2841 928 2578 \n",
       "Q 928 2378 1081 2264 \n",
       "Q 1234 2150 1697 2047 \n",
       "L 1894 2003 \n",
       "Q 2506 1872 2764 1633 \n",
       "Q 3022 1394 3022 966 \n",
       "Q 3022 478 2636 193 \n",
       "Q 2250 -91 1575 -91 \n",
       "Q 1294 -91 989 -36 \n",
       "Q 684 19 347 128 \n",
       "L 347 722 \n",
       "Q 666 556 975 473 \n",
       "Q 1284 391 1588 391 \n",
       "Q 1994 391 2212 530 \n",
       "Q 2431 669 2431 922 \n",
       "Q 2431 1156 2273 1281 \n",
       "Q 2116 1406 1581 1522 \n",
       "L 1381 1569 \n",
       "Q 847 1681 609 1914 \n",
       "Q 372 2147 372 2553 \n",
       "Q 372 3047 722 3315 \n",
       "Q 1072 3584 1716 3584 \n",
       "Q 2034 3584 2315 3537 \n",
       "Q 2597 3491 2834 3397 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-20\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-2f\" d=\"M 1625 4666 \n",
       "L 2156 4666 \n",
       "L 531 -594 \n",
       "L 0 -594 \n",
       "L 1625 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-63\" d=\"M 3122 3366 \n",
       "L 3122 2828 \n",
       "Q 2878 2963 2633 3030 \n",
       "Q 2388 3097 2138 3097 \n",
       "Q 1578 3097 1268 2742 \n",
       "Q 959 2388 959 1747 \n",
       "Q 959 1106 1268 751 \n",
       "Q 1578 397 2138 397 \n",
       "Q 2388 397 2633 464 \n",
       "Q 2878 531 3122 666 \n",
       "L 3122 134 \n",
       "Q 2881 22 2623 -34 \n",
       "Q 2366 -91 2075 -91 \n",
       "Q 1284 -91 818 406 \n",
       "Q 353 903 353 1747 \n",
       "Q 353 2603 823 3093 \n",
       "Q 1294 3584 2113 3584 \n",
       "Q 2378 3584 2631 3529 \n",
       "Q 2884 3475 3122 3366 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "      </defs>\n",
       "      <use xlink:href=\"#DejaVuSans-53\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-61\" x=\"63.476562\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-6d\" x=\"124.755859\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-70\" x=\"222.167969\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-6c\" x=\"285.644531\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-65\" x=\"313.427734\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-73\" x=\"374.951172\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-20\" x=\"427.050781\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-2f\" x=\"458.837891\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-20\" x=\"492.529297\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-73\" x=\"524.316406\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-65\" x=\"576.416016\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-63\" x=\"637.939453\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_2\">\n",
       "    <g id=\"ytick_1\">\n",
       "     <g id=\"line2d_4\">\n",
       "      <defs>\n",
       "       <path id=\"m48fd1b1e59\" d=\"M 0 0 \n",
       "L -3.5 0 \n",
       "\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </defs>\n",
       "      <g>\n",
       "       <use xlink:href=\"#m48fd1b1e59\" x=\"60.029688\" y=\"18.936818\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_5\">\n",
       "      <!-- HF 32-bit -->\n",
       "      <g transform=\"translate(7.2 22.736037)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-48\" d=\"M 628 4666 \n",
       "L 1259 4666 \n",
       "L 1259 2753 \n",
       "L 3553 2753 \n",
       "L 3553 4666 \n",
       "L 4184 4666 \n",
       "L 4184 0 \n",
       "L 3553 0 \n",
       "L 3553 2222 \n",
       "L 1259 2222 \n",
       "L 1259 0 \n",
       "L 628 0 \n",
       "L 628 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-46\" d=\"M 628 4666 \n",
       "L 3309 4666 \n",
       "L 3309 4134 \n",
       "L 1259 4134 \n",
       "L 1259 2759 \n",
       "L 3109 2759 \n",
       "L 3109 2228 \n",
       "L 1259 2228 \n",
       "L 1259 0 \n",
       "L 628 0 \n",
       "L 628 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-33\" d=\"M 2597 2516 \n",
       "Q 3050 2419 3304 2112 \n",
       "Q 3559 1806 3559 1356 \n",
       "Q 3559 666 3084 287 \n",
       "Q 2609 -91 1734 -91 \n",
       "Q 1441 -91 1130 -33 \n",
       "Q 819 25 488 141 \n",
       "L 488 750 \n",
       "Q 750 597 1062 519 \n",
       "Q 1375 441 1716 441 \n",
       "Q 2309 441 2620 675 \n",
       "Q 2931 909 2931 1356 \n",
       "Q 2931 1769 2642 2001 \n",
       "Q 2353 2234 1838 2234 \n",
       "L 1294 2234 \n",
       "L 1294 2753 \n",
       "L 1863 2753 \n",
       "Q 2328 2753 2575 2939 \n",
       "Q 2822 3125 2822 3475 \n",
       "Q 2822 3834 2567 4026 \n",
       "Q 2313 4219 1838 4219 \n",
       "Q 1578 4219 1281 4162 \n",
       "Q 984 4106 628 3988 \n",
       "L 628 4550 \n",
       "Q 988 4650 1302 4700 \n",
       "Q 1616 4750 1894 4750 \n",
       "Q 2613 4750 3031 4423 \n",
       "Q 3450 4097 3450 3541 \n",
       "Q 3450 3153 3228 2886 \n",
       "Q 3006 2619 2597 2516 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-2d\" d=\"M 313 2009 \n",
       "L 1997 2009 \n",
       "L 1997 1497 \n",
       "L 313 1497 \n",
       "L 313 2009 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-62\" d=\"M 3116 1747 \n",
       "Q 3116 2381 2855 2742 \n",
       "Q 2594 3103 2138 3103 \n",
       "Q 1681 3103 1420 2742 \n",
       "Q 1159 2381 1159 1747 \n",
       "Q 1159 1113 1420 752 \n",
       "Q 1681 391 2138 391 \n",
       "Q 2594 391 2855 752 \n",
       "Q 3116 1113 3116 1747 \n",
       "z\n",
       "M 1159 2969 \n",
       "Q 1341 3281 1617 3432 \n",
       "Q 1894 3584 2278 3584 \n",
       "Q 2916 3584 3314 3078 \n",
       "Q 3713 2572 3713 1747 \n",
       "Q 3713 922 3314 415 \n",
       "Q 2916 -91 2278 -91 \n",
       "Q 1894 -91 1617 61 \n",
       "Q 1341 213 1159 525 \n",
       "L 1159 0 \n",
       "L 581 0 \n",
       "L 581 4863 \n",
       "L 1159 4863 \n",
       "L 1159 2969 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-69\" d=\"M 603 3500 \n",
       "L 1178 3500 \n",
       "L 1178 0 \n",
       "L 603 0 \n",
       "L 603 3500 \n",
       "z\n",
       "M 603 4863 \n",
       "L 1178 4863 \n",
       "L 1178 4134 \n",
       "L 603 4134 \n",
       "L 603 4863 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-74\" d=\"M 1172 4494 \n",
       "L 1172 3500 \n",
       "L 2356 3500 \n",
       "L 2356 3053 \n",
       "L 1172 3053 \n",
       "L 1172 1153 \n",
       "Q 1172 725 1289 603 \n",
       "Q 1406 481 1766 481 \n",
       "L 2356 481 \n",
       "L 2356 0 \n",
       "L 1766 0 \n",
       "Q 1100 0 847 248 \n",
       "Q 594 497 594 1153 \n",
       "L 594 3053 \n",
       "L 172 3053 \n",
       "L 172 3500 \n",
       "L 594 3500 \n",
       "L 594 4494 \n",
       "L 1172 4494 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-48\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-46\" x=\"75.195312\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-20\" x=\"132.714844\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-33\" x=\"164.501953\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-32\" x=\"228.125\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-2d\" x=\"291.748047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-62\" x=\"327.832031\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-69\" x=\"391.308594\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-74\" x=\"419.091797\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_2\">\n",
       "     <g id=\"line2d_5\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m48fd1b1e59\" x=\"60.029688\" y=\"49.823182\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_6\">\n",
       "      <!-- HF 16-bit -->\n",
       "      <g transform=\"translate(7.2 53.622401)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-36\" d=\"M 2113 2584 \n",
       "Q 1688 2584 1439 2293 \n",
       "Q 1191 2003 1191 1497 \n",
       "Q 1191 994 1439 701 \n",
       "Q 1688 409 2113 409 \n",
       "Q 2538 409 2786 701 \n",
       "Q 3034 994 3034 1497 \n",
       "Q 3034 2003 2786 2293 \n",
       "Q 2538 2584 2113 2584 \n",
       "z\n",
       "M 3366 4563 \n",
       "L 3366 3988 \n",
       "Q 3128 4100 2886 4159 \n",
       "Q 2644 4219 2406 4219 \n",
       "Q 1781 4219 1451 3797 \n",
       "Q 1122 3375 1075 2522 \n",
       "Q 1259 2794 1537 2939 \n",
       "Q 1816 3084 2150 3084 \n",
       "Q 2853 3084 3261 2657 \n",
       "Q 3669 2231 3669 1497 \n",
       "Q 3669 778 3244 343 \n",
       "Q 2819 -91 2113 -91 \n",
       "Q 1303 -91 875 529 \n",
       "Q 447 1150 447 2328 \n",
       "Q 447 3434 972 4092 \n",
       "Q 1497 4750 2381 4750 \n",
       "Q 2619 4750 2861 4703 \n",
       "Q 3103 4656 3366 4563 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-48\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-46\" x=\"75.195312\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-20\" x=\"132.714844\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-31\" x=\"164.501953\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-36\" x=\"228.125\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-2d\" x=\"291.748047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-62\" x=\"327.832031\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-69\" x=\"391.308594\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-74\" x=\"419.091797\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"patch_5\">\n",
       "    <path d=\"M 60.029688 61.56 \n",
       "L 60.029688 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_6\">\n",
       "    <path d=\"M 207.73557 61.56 \n",
       "L 207.73557 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_7\">\n",
       "    <path d=\"M 60.029688 61.56 \n",
       "L 207.73557 61.56 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_8\">\n",
       "    <path d=\"M 60.029688 7.2 \n",
       "L 207.73557 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "  </g>\n",
       "  <g id=\"axes_2\">\n",
       "   <g id=\"patch_9\">\n",
       "    <path d=\"M 237.276746 61.56 \n",
       "L 384.982629 61.56 \n",
       "L 384.982629 7.2 \n",
       "L 237.276746 7.2 \n",
       "z\n",
       "\" style=\"fill: #ffffff\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_10\">\n",
       "    <path d=\"M 237.276746 9.670909 \n",
       "L 329.204807 9.670909 \n",
       "L 329.204807 28.202727 \n",
       "L 237.276746 28.202727 \n",
       "z\n",
       "\" clip-path=\"url(#p1c1e83ffe1)\" style=\"fill: #e41a1c\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_11\">\n",
       "    <path d=\"M 237.276746 40.557273 \n",
       "L 377.949015 40.557273 \n",
       "L 377.949015 59.089091 \n",
       "L 237.276746 59.089091 \n",
       "z\n",
       "\" clip-path=\"url(#p1c1e83ffe1)\" style=\"fill: #377eb8\"/>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_3\">\n",
       "    <g id=\"xtick_4\">\n",
       "     <g id=\"line2d_6\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#ma26dfc4f5b\" x=\"237.276746\" y=\"61.56\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_7\">\n",
       "      <!-- 0 -->\n",
       "      <g transform=\"translate(234.095496 76.158437)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_5\">\n",
       "     <g id=\"line2d_7\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#ma26dfc4f5b\" x=\"290.660577\" y=\"61.56\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_8\">\n",
       "      <!-- 10 -->\n",
       "      <g transform=\"translate(284.298077 76.158437)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-31\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_6\">\n",
       "     <g id=\"line2d_8\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#ma26dfc4f5b\" x=\"344.044407\" y=\"61.56\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_9\">\n",
       "      <!-- 20 -->\n",
       "      <g transform=\"translate(337.681907 76.158437)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-32\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"text_10\">\n",
       "     <!-- per GPU TFLOPS -->\n",
       "     <g transform=\"translate(270.4375 89.836562)scale(0.1 -0.1)\">\n",
       "      <defs>\n",
       "       <path id=\"DejaVuSans-72\" d=\"M 2631 2963 \n",
       "Q 2534 3019 2420 3045 \n",
       "Q 2306 3072 2169 3072 \n",
       "Q 1681 3072 1420 2755 \n",
       "Q 1159 2438 1159 1844 \n",
       "L 1159 0 \n",
       "L 581 0 \n",
       "L 581 3500 \n",
       "L 1159 3500 \n",
       "L 1159 2956 \n",
       "Q 1341 3275 1631 3429 \n",
       "Q 1922 3584 2338 3584 \n",
       "Q 2397 3584 2469 3576 \n",
       "Q 2541 3569 2628 3553 \n",
       "L 2631 2963 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-47\" d=\"M 3809 666 \n",
       "L 3809 1919 \n",
       "L 2778 1919 \n",
       "L 2778 2438 \n",
       "L 4434 2438 \n",
       "L 4434 434 \n",
       "Q 4069 175 3628 42 \n",
       "Q 3188 -91 2688 -91 \n",
       "Q 1594 -91 976 548 \n",
       "Q 359 1188 359 2328 \n",
       "Q 359 3472 976 4111 \n",
       "Q 1594 4750 2688 4750 \n",
       "Q 3144 4750 3555 4637 \n",
       "Q 3966 4525 4313 4306 \n",
       "L 4313 3634 \n",
       "Q 3963 3931 3569 4081 \n",
       "Q 3175 4231 2741 4231 \n",
       "Q 1884 4231 1454 3753 \n",
       "Q 1025 3275 1025 2328 \n",
       "Q 1025 1384 1454 906 \n",
       "Q 1884 428 2741 428 \n",
       "Q 3075 428 3337 486 \n",
       "Q 3600 544 3809 666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-50\" d=\"M 1259 4147 \n",
       "L 1259 2394 \n",
       "L 2053 2394 \n",
       "Q 2494 2394 2734 2622 \n",
       "Q 2975 2850 2975 3272 \n",
       "Q 2975 3691 2734 3919 \n",
       "Q 2494 4147 2053 4147 \n",
       "L 1259 4147 \n",
       "z\n",
       "M 628 4666 \n",
       "L 2053 4666 \n",
       "Q 2838 4666 3239 4311 \n",
       "Q 3641 3956 3641 3272 \n",
       "Q 3641 2581 3239 2228 \n",
       "Q 2838 1875 2053 1875 \n",
       "L 1259 1875 \n",
       "L 1259 0 \n",
       "L 628 0 \n",
       "L 628 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-55\" d=\"M 556 4666 \n",
       "L 1191 4666 \n",
       "L 1191 1831 \n",
       "Q 1191 1081 1462 751 \n",
       "Q 1734 422 2344 422 \n",
       "Q 2950 422 3222 751 \n",
       "Q 3494 1081 3494 1831 \n",
       "L 3494 4666 \n",
       "L 4128 4666 \n",
       "L 4128 1753 \n",
       "Q 4128 841 3676 375 \n",
       "Q 3225 -91 2344 -91 \n",
       "Q 1459 -91 1007 375 \n",
       "Q 556 841 556 1753 \n",
       "L 556 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-54\" d=\"M -19 4666 \n",
       "L 3928 4666 \n",
       "L 3928 4134 \n",
       "L 2272 4134 \n",
       "L 2272 0 \n",
       "L 1638 0 \n",
       "L 1638 4134 \n",
       "L -19 4134 \n",
       "L -19 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-4c\" d=\"M 628 4666 \n",
       "L 1259 4666 \n",
       "L 1259 531 \n",
       "L 3531 531 \n",
       "L 3531 0 \n",
       "L 628 0 \n",
       "L 628 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-4f\" d=\"M 2522 4238 \n",
       "Q 1834 4238 1429 3725 \n",
       "Q 1025 3213 1025 2328 \n",
       "Q 1025 1447 1429 934 \n",
       "Q 1834 422 2522 422 \n",
       "Q 3209 422 3611 934 \n",
       "Q 4013 1447 4013 2328 \n",
       "Q 4013 3213 3611 3725 \n",
       "Q 3209 4238 2522 4238 \n",
       "z\n",
       "M 2522 4750 \n",
       "Q 3503 4750 4090 4092 \n",
       "Q 4678 3434 4678 2328 \n",
       "Q 4678 1225 4090 567 \n",
       "Q 3503 -91 2522 -91 \n",
       "Q 1538 -91 948 565 \n",
       "Q 359 1222 359 2328 \n",
       "Q 359 3434 948 4092 \n",
       "Q 1538 4750 2522 4750 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "      </defs>\n",
       "      <use xlink:href=\"#DejaVuSans-70\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-65\" x=\"63.476562\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-72\" x=\"125\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-20\" x=\"166.113281\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-47\" x=\"197.900391\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-50\" x=\"275.390625\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-55\" x=\"335.693359\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-20\" x=\"408.886719\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-54\" x=\"440.673828\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-46\" x=\"501.757812\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-4c\" x=\"559.277344\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-4f\" x=\"611.365234\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-50\" x=\"690.076172\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-53\" x=\"750.378906\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_4\">\n",
       "    <g id=\"ytick_3\">\n",
       "     <g id=\"line2d_9\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m48fd1b1e59\" x=\"237.276746\" y=\"18.936818\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_4\">\n",
       "     <g id=\"line2d_10\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m48fd1b1e59\" x=\"237.276746\" y=\"49.823182\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"patch_12\">\n",
       "    <path d=\"M 237.276746 61.56 \n",
       "L 237.276746 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_13\">\n",
       "    <path d=\"M 384.982629 61.56 \n",
       "L 384.982629 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_14\">\n",
       "    <path d=\"M 237.276746 61.56 \n",
       "L 384.982629 61.56 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_15\">\n",
       "    <path d=\"M 237.276746 7.2 \n",
       "L 384.982629 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "  </g>\n",
       "  <g id=\"axes_3\">\n",
       "   <g id=\"patch_16\">\n",
       "    <path d=\"M 414.523805 61.56 \n",
       "L 562.229688 61.56 \n",
       "L 562.229688 7.2 \n",
       "L 414.523805 7.2 \n",
       "z\n",
       "\" style=\"fill: #ffffff\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_17\">\n",
       "    <path d=\"M 414.523805 9.670909 \n",
       "L 548.822405 9.670909 \n",
       "L 548.822405 28.202727 \n",
       "L 414.523805 28.202727 \n",
       "z\n",
       "\" clip-path=\"url(#pf8dffee81c)\" style=\"fill: #e41a1c\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_18\">\n",
       "    <path d=\"M 414.523805 40.557273 \n",
       "L 555.196074 40.557273 \n",
       "L 555.196074 59.089091 \n",
       "L 414.523805 59.089091 \n",
       "z\n",
       "\" clip-path=\"url(#pf8dffee81c)\" style=\"fill: #377eb8\"/>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_5\">\n",
       "    <g id=\"xtick_7\">\n",
       "     <g id=\"line2d_11\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#ma26dfc4f5b\" x=\"414.523805\" y=\"61.56\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_11\">\n",
       "      <!-- 0 -->\n",
       "      <g transform=\"translate(411.342555 76.158437)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_8\">\n",
       "     <g id=\"line2d_12\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#ma26dfc4f5b\" x=\"476.191399\" y=\"61.56\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_12\">\n",
       "      <!-- 10 -->\n",
       "      <g transform=\"translate(469.828899 76.158437)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-31\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_9\">\n",
       "     <g id=\"line2d_13\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#ma26dfc4f5b\" x=\"537.858994\" y=\"61.56\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_13\">\n",
       "      <!-- 20 -->\n",
       "      <g transform=\"translate(531.496494 76.158437)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-32\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"text_14\">\n",
       "     <!-- per GPU memory (GB) -->\n",
       "     <g transform=\"translate(432.657215 89.836562)scale(0.1 -0.1)\">\n",
       "      <defs>\n",
       "       <path id=\"DejaVuSans-6f\" d=\"M 1959 3097 \n",
       "Q 1497 3097 1228 2736 \n",
       "Q 959 2375 959 1747 \n",
       "Q 959 1119 1226 758 \n",
       "Q 1494 397 1959 397 \n",
       "Q 2419 397 2687 759 \n",
       "Q 2956 1122 2956 1747 \n",
       "Q 2956 2369 2687 2733 \n",
       "Q 2419 3097 1959 3097 \n",
       "z\n",
       "M 1959 3584 \n",
       "Q 2709 3584 3137 3096 \n",
       "Q 3566 2609 3566 1747 \n",
       "Q 3566 888 3137 398 \n",
       "Q 2709 -91 1959 -91 \n",
       "Q 1206 -91 779 398 \n",
       "Q 353 888 353 1747 \n",
       "Q 353 2609 779 3096 \n",
       "Q 1206 3584 1959 3584 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-79\" d=\"M 2059 -325 \n",
       "Q 1816 -950 1584 -1140 \n",
       "Q 1353 -1331 966 -1331 \n",
       "L 506 -1331 \n",
       "L 506 -850 \n",
       "L 844 -850 \n",
       "Q 1081 -850 1212 -737 \n",
       "Q 1344 -625 1503 -206 \n",
       "L 1606 56 \n",
       "L 191 3500 \n",
       "L 800 3500 \n",
       "L 1894 763 \n",
       "L 2988 3500 \n",
       "L 3597 3500 \n",
       "L 2059 -325 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-28\" d=\"M 1984 4856 \n",
       "Q 1566 4138 1362 3434 \n",
       "Q 1159 2731 1159 2009 \n",
       "Q 1159 1288 1364 580 \n",
       "Q 1569 -128 1984 -844 \n",
       "L 1484 -844 \n",
       "Q 1016 -109 783 600 \n",
       "Q 550 1309 550 2009 \n",
       "Q 550 2706 781 3412 \n",
       "Q 1013 4119 1484 4856 \n",
       "L 1984 4856 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-42\" d=\"M 1259 2228 \n",
       "L 1259 519 \n",
       "L 2272 519 \n",
       "Q 2781 519 3026 730 \n",
       "Q 3272 941 3272 1375 \n",
       "Q 3272 1813 3026 2020 \n",
       "Q 2781 2228 2272 2228 \n",
       "L 1259 2228 \n",
       "z\n",
       "M 1259 4147 \n",
       "L 1259 2741 \n",
       "L 2194 2741 \n",
       "Q 2656 2741 2882 2914 \n",
       "Q 3109 3088 3109 3444 \n",
       "Q 3109 3797 2882 3972 \n",
       "Q 2656 4147 2194 4147 \n",
       "L 1259 4147 \n",
       "z\n",
       "M 628 4666 \n",
       "L 2241 4666 \n",
       "Q 2963 4666 3353 4366 \n",
       "Q 3744 4066 3744 3513 \n",
       "Q 3744 3084 3544 2831 \n",
       "Q 3344 2578 2956 2516 \n",
       "Q 3422 2416 3680 2098 \n",
       "Q 3938 1781 3938 1306 \n",
       "Q 3938 681 3513 340 \n",
       "Q 3088 0 2303 0 \n",
       "L 628 0 \n",
       "L 628 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-29\" d=\"M 513 4856 \n",
       "L 1013 4856 \n",
       "Q 1481 4119 1714 3412 \n",
       "Q 1947 2706 1947 2009 \n",
       "Q 1947 1309 1714 600 \n",
       "Q 1481 -109 1013 -844 \n",
       "L 513 -844 \n",
       "Q 928 -128 1133 580 \n",
       "Q 1338 1288 1338 2009 \n",
       "Q 1338 2731 1133 3434 \n",
       "Q 928 4138 513 4856 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "      </defs>\n",
       "      <use xlink:href=\"#DejaVuSans-70\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-65\" x=\"63.476562\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-72\" x=\"125\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-20\" x=\"166.113281\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-47\" x=\"197.900391\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-50\" x=\"275.390625\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-55\" x=\"335.693359\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-20\" x=\"408.886719\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-6d\" x=\"440.673828\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-65\" x=\"538.085938\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-6d\" x=\"599.609375\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-6f\" x=\"697.021484\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-72\" x=\"758.203125\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-79\" x=\"799.316406\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-20\" x=\"858.496094\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-28\" x=\"890.283203\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-47\" x=\"929.296875\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-42\" x=\"1006.787109\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-29\" x=\"1075.390625\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_6\">\n",
       "    <g id=\"ytick_5\">\n",
       "     <g id=\"line2d_14\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m48fd1b1e59\" x=\"414.523805\" y=\"18.936818\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_6\">\n",
       "     <g id=\"line2d_15\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m48fd1b1e59\" x=\"414.523805\" y=\"49.823182\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"patch_19\">\n",
       "    <path d=\"M 414.523805 61.56 \n",
       "L 414.523805 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_20\">\n",
       "    <path d=\"M 562.229688 61.56 \n",
       "L 562.229688 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_21\">\n",
       "    <path d=\"M 414.523805 61.56 \n",
       "L 562.229688 61.56 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_22\">\n",
       "    <path d=\"M 414.523805 7.2 \n",
       "L 562.229688 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "  </g>\n",
       " </g>\n",
       " <defs>\n",
       "  <clipPath id=\"p8cab51f4bc\">\n",
       "   <rect x=\"60.029688\" y=\"7.2\" width=\"147.705882\" height=\"54.36\"/>\n",
       "  </clipPath>\n",
       "  <clipPath id=\"p1c1e83ffe1\">\n",
       "   <rect x=\"237.276746\" y=\"7.2\" width=\"147.705882\" height=\"54.36\"/>\n",
       "  </clipPath>\n",
       "  <clipPath id=\"pf8dffee81c\">\n",
       "   <rect x=\"414.523805\" y=\"7.2\" width=\"147.705882\" height=\"54.36\"/>\n",
       "  </clipPath>\n",
       " </defs>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<Figure size 648x72 with 3 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "bert_half = hf_bert(Exp('HF 16-bit', 'bert-large-uncased', 11, bf16=True))\n",
    "compare([bert_single, bert_half])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fa998555",
   "metadata": {},
   "source": [
    "You may be surprised that using 16-bit floating points doesn't reduce memory size by half under the same hyperparameters. That's because \n",
    "the memory usage is mainly due to three parts: model parameters, layer outputs in the forward path (activations) and workspace memory used by backend libraries. 16-bit floats do not save memory related to model parameters because model updating is running with 32-bit. For one model parameter: \n",
    "\n",
    "\n",
    "- with 32-bit, we use 4 bytes for the 32-bit weight, 4 bytes for the 32-bit gradient, 8 bytes for the two momentums in Adam, a total of **16 bytes**\n",
    "- with 32-bit, we use 2 bytes for the 16-bit weight, 2 bytes for the 16-bit gradient (some implementation uses 32-bit gradient), 4 bytes for the master 32-bit weight, and 8 bytes for the two momentums in adam, with a total of **16 bytes** \n",
    "\n",
    "The memory saving is due to all activations are stored in 16-bit. As the activation size is linear to the batch size and sequence length, using 16-bit could allow you to double batch size or sequence length. \n",
    "\n",
    "GPUs using old architectures before Ampere do not support `bf16`, you could try to use `fp16` via changing the above code to `fp16=True`. It often offers same performance as `bf16`, but may require you to tune the [loss scaling](https://moocaholic.medium.com/fp64-fp32-fp16-bfloat16-tf32-and-other-members-of-the-zoo-a1ca7897d407)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1fd12e96",
   "metadata": {},
   "source": [
    "As we shown in the micro-benchmarks, the model updating that involving multiple vector operators could be expensive. If you have [apex](https://github.com/NVIDIA/apex) installed, we can use an faster implementation. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "3d972b05",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       "  \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<svg xmlns:xlink=\"http://www.w3.org/1999/xlink\" width=\"635.307812pt\" height=\"126.29625pt\" viewBox=\"0 0 635.307812 126.29625\" xmlns=\"http://www.w3.org/2000/svg\" version=\"1.1\">\n",
       " <metadata>\n",
       "  <rdf:RDF xmlns:dc=\"http://purl.org/dc/elements/1.1/\" xmlns:cc=\"http://creativecommons.org/ns#\" xmlns:rdf=\"http://www.w3.org/1999/02/22-rdf-syntax-ns#\">\n",
       "   <cc:Work>\n",
       "    <dc:type rdf:resource=\"http://purl.org/dc/dcmitype/StillImage\"/>\n",
       "    <dc:date>2022-09-02T19:40:40.378070</dc:date>\n",
       "    <dc:format>image/svg+xml</dc:format>\n",
       "    <dc:creator>\n",
       "     <cc:Agent>\n",
       "      <dc:title>Matplotlib v3.5.2, https://matplotlib.org/</dc:title>\n",
       "     </cc:Agent>\n",
       "    </dc:creator>\n",
       "   </cc:Work>\n",
       "  </rdf:RDF>\n",
       " </metadata>\n",
       " <defs>\n",
       "  <style type=\"text/css\">*{stroke-linejoin: round; stroke-linecap: butt}</style>\n",
       " </defs>\n",
       " <g id=\"figure_1\">\n",
       "  <g id=\"patch_1\">\n",
       "   <path d=\"M 0 126.29625 \n",
       "L 635.307812 126.29625 \n",
       "L 635.307812 0 \n",
       "L 0 0 \n",
       "L 0 126.29625 \n",
       "z\n",
       "\" style=\"fill: none\"/>\n",
       "  </g>\n",
       "  <g id=\"axes_1\">\n",
       "   <g id=\"patch_2\">\n",
       "    <path d=\"M 125.907813 88.74 \n",
       "L 273.613695 88.74 \n",
       "L 273.613695 7.2 \n",
       "L 125.907813 7.2 \n",
       "z\n",
       "\" style=\"fill: #ffffff\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_3\">\n",
       "    <path d=\"M 125.907813 10.906364 \n",
       "L 212.724729 10.906364 \n",
       "L 212.724729 28.012657 \n",
       "L 125.907813 28.012657 \n",
       "z\n",
       "\" clip-path=\"url(#p32864c3797)\" style=\"fill: #e41a1c\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_4\">\n",
       "    <path d=\"M 125.907813 39.416853 \n",
       "L 258.758788 39.416853 \n",
       "L 258.758788 56.523147 \n",
       "L 125.907813 56.523147 \n",
       "z\n",
       "\" clip-path=\"url(#p32864c3797)\" style=\"fill: #377eb8\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_5\">\n",
       "    <path d=\"M 125.907813 67.927343 \n",
       "L 266.580081 67.927343 \n",
       "L 266.580081 85.033636 \n",
       "L 125.907813 85.033636 \n",
       "z\n",
       "\" clip-path=\"url(#p32864c3797)\" style=\"fill: #4daf4a\"/>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_1\">\n",
       "    <g id=\"xtick_1\">\n",
       "     <g id=\"line2d_1\">\n",
       "      <defs>\n",
       "       <path id=\"mf2a8f50faa\" d=\"M 0 0 \n",
       "L 0 3.5 \n",
       "\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </defs>\n",
       "      <g>\n",
       "       <use xlink:href=\"#mf2a8f50faa\" x=\"125.907813\" y=\"88.74\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_1\">\n",
       "      <!-- 0 -->\n",
       "      <g transform=\"translate(122.726562 103.338438)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-30\" d=\"M 2034 4250 \n",
       "Q 1547 4250 1301 3770 \n",
       "Q 1056 3291 1056 2328 \n",
       "Q 1056 1369 1301 889 \n",
       "Q 1547 409 2034 409 \n",
       "Q 2525 409 2770 889 \n",
       "Q 3016 1369 3016 2328 \n",
       "Q 3016 3291 2770 3770 \n",
       "Q 2525 4250 2034 4250 \n",
       "z\n",
       "M 2034 4750 \n",
       "Q 2819 4750 3233 4129 \n",
       "Q 3647 3509 3647 2328 \n",
       "Q 3647 1150 3233 529 \n",
       "Q 2819 -91 2034 -91 \n",
       "Q 1250 -91 836 529 \n",
       "Q 422 1150 422 2328 \n",
       "Q 422 3509 836 4129 \n",
       "Q 1250 4750 2034 4750 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_2\">\n",
       "     <g id=\"line2d_2\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#mf2a8f50faa\" x=\"181.41735\" y=\"88.74\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_2\">\n",
       "      <!-- 10 -->\n",
       "      <g transform=\"translate(175.05485 103.338438)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-31\" d=\"M 794 531 \n",
       "L 1825 531 \n",
       "L 1825 4091 \n",
       "L 703 3866 \n",
       "L 703 4441 \n",
       "L 1819 4666 \n",
       "L 2450 4666 \n",
       "L 2450 531 \n",
       "L 3481 531 \n",
       "L 3481 0 \n",
       "L 794 0 \n",
       "L 794 531 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-31\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_3\">\n",
       "     <g id=\"line2d_3\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#mf2a8f50faa\" x=\"236.926887\" y=\"88.74\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_3\">\n",
       "      <!-- 20 -->\n",
       "      <g transform=\"translate(230.564387 103.338438)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-32\" d=\"M 1228 531 \n",
       "L 3431 531 \n",
       "L 3431 0 \n",
       "L 469 0 \n",
       "L 469 531 \n",
       "Q 828 903 1448 1529 \n",
       "Q 2069 2156 2228 2338 \n",
       "Q 2531 2678 2651 2914 \n",
       "Q 2772 3150 2772 3378 \n",
       "Q 2772 3750 2511 3984 \n",
       "Q 2250 4219 1831 4219 \n",
       "Q 1534 4219 1204 4116 \n",
       "Q 875 4013 500 3803 \n",
       "L 500 4441 \n",
       "Q 881 4594 1212 4672 \n",
       "Q 1544 4750 1819 4750 \n",
       "Q 2544 4750 2975 4387 \n",
       "Q 3406 4025 3406 3419 \n",
       "Q 3406 3131 3298 2873 \n",
       "Q 3191 2616 2906 2266 \n",
       "Q 2828 2175 2409 1742 \n",
       "Q 1991 1309 1228 531 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-32\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"text_4\">\n",
       "     <!-- Samples / sec -->\n",
       "     <g transform=\"translate(165.11466 117.016563)scale(0.1 -0.1)\">\n",
       "      <defs>\n",
       "       <path id=\"DejaVuSans-53\" d=\"M 3425 4513 \n",
       "L 3425 3897 \n",
       "Q 3066 4069 2747 4153 \n",
       "Q 2428 4238 2131 4238 \n",
       "Q 1616 4238 1336 4038 \n",
       "Q 1056 3838 1056 3469 \n",
       "Q 1056 3159 1242 3001 \n",
       "Q 1428 2844 1947 2747 \n",
       "L 2328 2669 \n",
       "Q 3034 2534 3370 2195 \n",
       "Q 3706 1856 3706 1288 \n",
       "Q 3706 609 3251 259 \n",
       "Q 2797 -91 1919 -91 \n",
       "Q 1588 -91 1214 -16 \n",
       "Q 841 59 441 206 \n",
       "L 441 856 \n",
       "Q 825 641 1194 531 \n",
       "Q 1563 422 1919 422 \n",
       "Q 2459 422 2753 634 \n",
       "Q 3047 847 3047 1241 \n",
       "Q 3047 1584 2836 1778 \n",
       "Q 2625 1972 2144 2069 \n",
       "L 1759 2144 \n",
       "Q 1053 2284 737 2584 \n",
       "Q 422 2884 422 3419 \n",
       "Q 422 4038 858 4394 \n",
       "Q 1294 4750 2059 4750 \n",
       "Q 2388 4750 2728 4690 \n",
       "Q 3069 4631 3425 4513 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-61\" d=\"M 2194 1759 \n",
       "Q 1497 1759 1228 1600 \n",
       "Q 959 1441 959 1056 \n",
       "Q 959 750 1161 570 \n",
       "Q 1363 391 1709 391 \n",
       "Q 2188 391 2477 730 \n",
       "Q 2766 1069 2766 1631 \n",
       "L 2766 1759 \n",
       "L 2194 1759 \n",
       "z\n",
       "M 3341 1997 \n",
       "L 3341 0 \n",
       "L 2766 0 \n",
       "L 2766 531 \n",
       "Q 2569 213 2275 61 \n",
       "Q 1981 -91 1556 -91 \n",
       "Q 1019 -91 701 211 \n",
       "Q 384 513 384 1019 \n",
       "Q 384 1609 779 1909 \n",
       "Q 1175 2209 1959 2209 \n",
       "L 2766 2209 \n",
       "L 2766 2266 \n",
       "Q 2766 2663 2505 2880 \n",
       "Q 2244 3097 1772 3097 \n",
       "Q 1472 3097 1187 3025 \n",
       "Q 903 2953 641 2809 \n",
       "L 641 3341 \n",
       "Q 956 3463 1253 3523 \n",
       "Q 1550 3584 1831 3584 \n",
       "Q 2591 3584 2966 3190 \n",
       "Q 3341 2797 3341 1997 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-6d\" d=\"M 3328 2828 \n",
       "Q 3544 3216 3844 3400 \n",
       "Q 4144 3584 4550 3584 \n",
       "Q 5097 3584 5394 3201 \n",
       "Q 5691 2819 5691 2113 \n",
       "L 5691 0 \n",
       "L 5113 0 \n",
       "L 5113 2094 \n",
       "Q 5113 2597 4934 2840 \n",
       "Q 4756 3084 4391 3084 \n",
       "Q 3944 3084 3684 2787 \n",
       "Q 3425 2491 3425 1978 \n",
       "L 3425 0 \n",
       "L 2847 0 \n",
       "L 2847 2094 \n",
       "Q 2847 2600 2669 2842 \n",
       "Q 2491 3084 2119 3084 \n",
       "Q 1678 3084 1418 2786 \n",
       "Q 1159 2488 1159 1978 \n",
       "L 1159 0 \n",
       "L 581 0 \n",
       "L 581 3500 \n",
       "L 1159 3500 \n",
       "L 1159 2956 \n",
       "Q 1356 3278 1631 3431 \n",
       "Q 1906 3584 2284 3584 \n",
       "Q 2666 3584 2933 3390 \n",
       "Q 3200 3197 3328 2828 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-70\" d=\"M 1159 525 \n",
       "L 1159 -1331 \n",
       "L 581 -1331 \n",
       "L 581 3500 \n",
       "L 1159 3500 \n",
       "L 1159 2969 \n",
       "Q 1341 3281 1617 3432 \n",
       "Q 1894 3584 2278 3584 \n",
       "Q 2916 3584 3314 3078 \n",
       "Q 3713 2572 3713 1747 \n",
       "Q 3713 922 3314 415 \n",
       "Q 2916 -91 2278 -91 \n",
       "Q 1894 -91 1617 61 \n",
       "Q 1341 213 1159 525 \n",
       "z\n",
       "M 3116 1747 \n",
       "Q 3116 2381 2855 2742 \n",
       "Q 2594 3103 2138 3103 \n",
       "Q 1681 3103 1420 2742 \n",
       "Q 1159 2381 1159 1747 \n",
       "Q 1159 1113 1420 752 \n",
       "Q 1681 391 2138 391 \n",
       "Q 2594 391 2855 752 \n",
       "Q 3116 1113 3116 1747 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-6c\" d=\"M 603 4863 \n",
       "L 1178 4863 \n",
       "L 1178 0 \n",
       "L 603 0 \n",
       "L 603 4863 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-65\" d=\"M 3597 1894 \n",
       "L 3597 1613 \n",
       "L 953 1613 \n",
       "Q 991 1019 1311 708 \n",
       "Q 1631 397 2203 397 \n",
       "Q 2534 397 2845 478 \n",
       "Q 3156 559 3463 722 \n",
       "L 3463 178 \n",
       "Q 3153 47 2828 -22 \n",
       "Q 2503 -91 2169 -91 \n",
       "Q 1331 -91 842 396 \n",
       "Q 353 884 353 1716 \n",
       "Q 353 2575 817 3079 \n",
       "Q 1281 3584 2069 3584 \n",
       "Q 2775 3584 3186 3129 \n",
       "Q 3597 2675 3597 1894 \n",
       "z\n",
       "M 3022 2063 \n",
       "Q 3016 2534 2758 2815 \n",
       "Q 2500 3097 2075 3097 \n",
       "Q 1594 3097 1305 2825 \n",
       "Q 1016 2553 972 2059 \n",
       "L 3022 2063 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-73\" d=\"M 2834 3397 \n",
       "L 2834 2853 \n",
       "Q 2591 2978 2328 3040 \n",
       "Q 2066 3103 1784 3103 \n",
       "Q 1356 3103 1142 2972 \n",
       "Q 928 2841 928 2578 \n",
       "Q 928 2378 1081 2264 \n",
       "Q 1234 2150 1697 2047 \n",
       "L 1894 2003 \n",
       "Q 2506 1872 2764 1633 \n",
       "Q 3022 1394 3022 966 \n",
       "Q 3022 478 2636 193 \n",
       "Q 2250 -91 1575 -91 \n",
       "Q 1294 -91 989 -36 \n",
       "Q 684 19 347 128 \n",
       "L 347 722 \n",
       "Q 666 556 975 473 \n",
       "Q 1284 391 1588 391 \n",
       "Q 1994 391 2212 530 \n",
       "Q 2431 669 2431 922 \n",
       "Q 2431 1156 2273 1281 \n",
       "Q 2116 1406 1581 1522 \n",
       "L 1381 1569 \n",
       "Q 847 1681 609 1914 \n",
       "Q 372 2147 372 2553 \n",
       "Q 372 3047 722 3315 \n",
       "Q 1072 3584 1716 3584 \n",
       "Q 2034 3584 2315 3537 \n",
       "Q 2597 3491 2834 3397 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-20\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-2f\" d=\"M 1625 4666 \n",
       "L 2156 4666 \n",
       "L 531 -594 \n",
       "L 0 -594 \n",
       "L 1625 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-63\" d=\"M 3122 3366 \n",
       "L 3122 2828 \n",
       "Q 2878 2963 2633 3030 \n",
       "Q 2388 3097 2138 3097 \n",
       "Q 1578 3097 1268 2742 \n",
       "Q 959 2388 959 1747 \n",
       "Q 959 1106 1268 751 \n",
       "Q 1578 397 2138 397 \n",
       "Q 2388 397 2633 464 \n",
       "Q 2878 531 3122 666 \n",
       "L 3122 134 \n",
       "Q 2881 22 2623 -34 \n",
       "Q 2366 -91 2075 -91 \n",
       "Q 1284 -91 818 406 \n",
       "Q 353 903 353 1747 \n",
       "Q 353 2603 823 3093 \n",
       "Q 1294 3584 2113 3584 \n",
       "Q 2378 3584 2631 3529 \n",
       "Q 2884 3475 3122 3366 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "      </defs>\n",
       "      <use xlink:href=\"#DejaVuSans-53\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-61\" x=\"63.476562\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-6d\" x=\"124.755859\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-70\" x=\"222.167969\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-6c\" x=\"285.644531\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-65\" x=\"313.427734\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-73\" x=\"374.951172\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-20\" x=\"427.050781\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-2f\" x=\"458.837891\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-20\" x=\"492.529297\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-73\" x=\"524.316406\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-65\" x=\"576.416016\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-63\" x=\"637.939453\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_2\">\n",
       "    <g id=\"ytick_1\">\n",
       "     <g id=\"line2d_4\">\n",
       "      <defs>\n",
       "       <path id=\"m5514f5ae47\" d=\"M 0 0 \n",
       "L -3.5 0 \n",
       "\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </defs>\n",
       "      <g>\n",
       "       <use xlink:href=\"#m5514f5ae47\" x=\"125.907813\" y=\"19.45951\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_5\">\n",
       "      <!-- HF 32-bit -->\n",
       "      <g transform=\"translate(73.078125 23.258729)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-48\" d=\"M 628 4666 \n",
       "L 1259 4666 \n",
       "L 1259 2753 \n",
       "L 3553 2753 \n",
       "L 3553 4666 \n",
       "L 4184 4666 \n",
       "L 4184 0 \n",
       "L 3553 0 \n",
       "L 3553 2222 \n",
       "L 1259 2222 \n",
       "L 1259 0 \n",
       "L 628 0 \n",
       "L 628 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-46\" d=\"M 628 4666 \n",
       "L 3309 4666 \n",
       "L 3309 4134 \n",
       "L 1259 4134 \n",
       "L 1259 2759 \n",
       "L 3109 2759 \n",
       "L 3109 2228 \n",
       "L 1259 2228 \n",
       "L 1259 0 \n",
       "L 628 0 \n",
       "L 628 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-33\" d=\"M 2597 2516 \n",
       "Q 3050 2419 3304 2112 \n",
       "Q 3559 1806 3559 1356 \n",
       "Q 3559 666 3084 287 \n",
       "Q 2609 -91 1734 -91 \n",
       "Q 1441 -91 1130 -33 \n",
       "Q 819 25 488 141 \n",
       "L 488 750 \n",
       "Q 750 597 1062 519 \n",
       "Q 1375 441 1716 441 \n",
       "Q 2309 441 2620 675 \n",
       "Q 2931 909 2931 1356 \n",
       "Q 2931 1769 2642 2001 \n",
       "Q 2353 2234 1838 2234 \n",
       "L 1294 2234 \n",
       "L 1294 2753 \n",
       "L 1863 2753 \n",
       "Q 2328 2753 2575 2939 \n",
       "Q 2822 3125 2822 3475 \n",
       "Q 2822 3834 2567 4026 \n",
       "Q 2313 4219 1838 4219 \n",
       "Q 1578 4219 1281 4162 \n",
       "Q 984 4106 628 3988 \n",
       "L 628 4550 \n",
       "Q 988 4650 1302 4700 \n",
       "Q 1616 4750 1894 4750 \n",
       "Q 2613 4750 3031 4423 \n",
       "Q 3450 4097 3450 3541 \n",
       "Q 3450 3153 3228 2886 \n",
       "Q 3006 2619 2597 2516 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-2d\" d=\"M 313 2009 \n",
       "L 1997 2009 \n",
       "L 1997 1497 \n",
       "L 313 1497 \n",
       "L 313 2009 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-62\" d=\"M 3116 1747 \n",
       "Q 3116 2381 2855 2742 \n",
       "Q 2594 3103 2138 3103 \n",
       "Q 1681 3103 1420 2742 \n",
       "Q 1159 2381 1159 1747 \n",
       "Q 1159 1113 1420 752 \n",
       "Q 1681 391 2138 391 \n",
       "Q 2594 391 2855 752 \n",
       "Q 3116 1113 3116 1747 \n",
       "z\n",
       "M 1159 2969 \n",
       "Q 1341 3281 1617 3432 \n",
       "Q 1894 3584 2278 3584 \n",
       "Q 2916 3584 3314 3078 \n",
       "Q 3713 2572 3713 1747 \n",
       "Q 3713 922 3314 415 \n",
       "Q 2916 -91 2278 -91 \n",
       "Q 1894 -91 1617 61 \n",
       "Q 1341 213 1159 525 \n",
       "L 1159 0 \n",
       "L 581 0 \n",
       "L 581 4863 \n",
       "L 1159 4863 \n",
       "L 1159 2969 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-69\" d=\"M 603 3500 \n",
       "L 1178 3500 \n",
       "L 1178 0 \n",
       "L 603 0 \n",
       "L 603 3500 \n",
       "z\n",
       "M 603 4863 \n",
       "L 1178 4863 \n",
       "L 1178 4134 \n",
       "L 603 4134 \n",
       "L 603 4863 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-74\" d=\"M 1172 4494 \n",
       "L 1172 3500 \n",
       "L 2356 3500 \n",
       "L 2356 3053 \n",
       "L 1172 3053 \n",
       "L 1172 1153 \n",
       "Q 1172 725 1289 603 \n",
       "Q 1406 481 1766 481 \n",
       "L 2356 481 \n",
       "L 2356 0 \n",
       "L 1766 0 \n",
       "Q 1100 0 847 248 \n",
       "Q 594 497 594 1153 \n",
       "L 594 3053 \n",
       "L 172 3053 \n",
       "L 172 3500 \n",
       "L 594 3500 \n",
       "L 594 4494 \n",
       "L 1172 4494 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-48\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-46\" x=\"75.195312\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-20\" x=\"132.714844\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-33\" x=\"164.501953\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-32\" x=\"228.125\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-2d\" x=\"291.748047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-62\" x=\"327.832031\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-69\" x=\"391.308594\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-74\" x=\"419.091797\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_2\">\n",
       "     <g id=\"line2d_5\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m5514f5ae47\" x=\"125.907813\" y=\"47.97\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_6\">\n",
       "      <!-- HF 16-bit -->\n",
       "      <g transform=\"translate(73.078125 51.769219)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-36\" d=\"M 2113 2584 \n",
       "Q 1688 2584 1439 2293 \n",
       "Q 1191 2003 1191 1497 \n",
       "Q 1191 994 1439 701 \n",
       "Q 1688 409 2113 409 \n",
       "Q 2538 409 2786 701 \n",
       "Q 3034 994 3034 1497 \n",
       "Q 3034 2003 2786 2293 \n",
       "Q 2538 2584 2113 2584 \n",
       "z\n",
       "M 3366 4563 \n",
       "L 3366 3988 \n",
       "Q 3128 4100 2886 4159 \n",
       "Q 2644 4219 2406 4219 \n",
       "Q 1781 4219 1451 3797 \n",
       "Q 1122 3375 1075 2522 \n",
       "Q 1259 2794 1537 2939 \n",
       "Q 1816 3084 2150 3084 \n",
       "Q 2853 3084 3261 2657 \n",
       "Q 3669 2231 3669 1497 \n",
       "Q 3669 778 3244 343 \n",
       "Q 2819 -91 2113 -91 \n",
       "Q 1303 -91 875 529 \n",
       "Q 447 1150 447 2328 \n",
       "Q 447 3434 972 4092 \n",
       "Q 1497 4750 2381 4750 \n",
       "Q 2619 4750 2861 4703 \n",
       "Q 3103 4656 3366 4563 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-48\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-46\" x=\"75.195312\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-20\" x=\"132.714844\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-31\" x=\"164.501953\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-36\" x=\"228.125\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-2d\" x=\"291.748047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-62\" x=\"327.832031\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-69\" x=\"391.308594\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-74\" x=\"419.091797\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_3\">\n",
       "     <g id=\"line2d_6\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m5514f5ae47\" x=\"125.907813\" y=\"76.48049\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_7\">\n",
       "      <!-- HF 16-bit, fused-adam -->\n",
       "      <g transform=\"translate(7.2 80.279708)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-2c\" d=\"M 750 794 \n",
       "L 1409 794 \n",
       "L 1409 256 \n",
       "L 897 -744 \n",
       "L 494 -744 \n",
       "L 750 256 \n",
       "L 750 794 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-66\" d=\"M 2375 4863 \n",
       "L 2375 4384 \n",
       "L 1825 4384 \n",
       "Q 1516 4384 1395 4259 \n",
       "Q 1275 4134 1275 3809 \n",
       "L 1275 3500 \n",
       "L 2222 3500 \n",
       "L 2222 3053 \n",
       "L 1275 3053 \n",
       "L 1275 0 \n",
       "L 697 0 \n",
       "L 697 3053 \n",
       "L 147 3053 \n",
       "L 147 3500 \n",
       "L 697 3500 \n",
       "L 697 3744 \n",
       "Q 697 4328 969 4595 \n",
       "Q 1241 4863 1831 4863 \n",
       "L 2375 4863 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-75\" d=\"M 544 1381 \n",
       "L 544 3500 \n",
       "L 1119 3500 \n",
       "L 1119 1403 \n",
       "Q 1119 906 1312 657 \n",
       "Q 1506 409 1894 409 \n",
       "Q 2359 409 2629 706 \n",
       "Q 2900 1003 2900 1516 \n",
       "L 2900 3500 \n",
       "L 3475 3500 \n",
       "L 3475 0 \n",
       "L 2900 0 \n",
       "L 2900 538 \n",
       "Q 2691 219 2414 64 \n",
       "Q 2138 -91 1772 -91 \n",
       "Q 1169 -91 856 284 \n",
       "Q 544 659 544 1381 \n",
       "z\n",
       "M 1991 3584 \n",
       "L 1991 3584 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-64\" d=\"M 2906 2969 \n",
       "L 2906 4863 \n",
       "L 3481 4863 \n",
       "L 3481 0 \n",
       "L 2906 0 \n",
       "L 2906 525 \n",
       "Q 2725 213 2448 61 \n",
       "Q 2172 -91 1784 -91 \n",
       "Q 1150 -91 751 415 \n",
       "Q 353 922 353 1747 \n",
       "Q 353 2572 751 3078 \n",
       "Q 1150 3584 1784 3584 \n",
       "Q 2172 3584 2448 3432 \n",
       "Q 2725 3281 2906 2969 \n",
       "z\n",
       "M 947 1747 \n",
       "Q 947 1113 1208 752 \n",
       "Q 1469 391 1925 391 \n",
       "Q 2381 391 2643 752 \n",
       "Q 2906 1113 2906 1747 \n",
       "Q 2906 2381 2643 2742 \n",
       "Q 2381 3103 1925 3103 \n",
       "Q 1469 3103 1208 2742 \n",
       "Q 947 2381 947 1747 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-48\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-46\" x=\"75.195312\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-20\" x=\"132.714844\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-31\" x=\"164.501953\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-36\" x=\"228.125\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-2d\" x=\"291.748047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-62\" x=\"327.832031\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-69\" x=\"391.308594\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-74\" x=\"419.091797\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-2c\" x=\"458.300781\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-20\" x=\"490.087891\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-66\" x=\"521.875\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-75\" x=\"557.080078\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-73\" x=\"620.458984\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-65\" x=\"672.558594\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-64\" x=\"734.082031\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-2d\" x=\"797.558594\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-61\" x=\"833.642578\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-64\" x=\"894.921875\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-61\" x=\"958.398438\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-6d\" x=\"1019.677734\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"patch_6\">\n",
       "    <path d=\"M 125.907813 88.74 \n",
       "L 125.907813 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_7\">\n",
       "    <path d=\"M 273.613695 88.74 \n",
       "L 273.613695 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_8\">\n",
       "    <path d=\"M 125.907813 88.74 \n",
       "L 273.613695 88.74 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_9\">\n",
       "    <path d=\"M 125.907813 7.2 \n",
       "L 273.613695 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "  </g>\n",
       "  <g id=\"axes_2\">\n",
       "   <g id=\"patch_10\">\n",
       "    <path d=\"M 303.154871 88.74 \n",
       "L 450.860754 88.74 \n",
       "L 450.860754 7.2 \n",
       "L 303.154871 7.2 \n",
       "z\n",
       "\" style=\"fill: #ffffff\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_11\">\n",
       "    <path d=\"M 303.154871 10.906364 \n",
       "L 389.971787 10.906364 \n",
       "L 389.971787 28.012657 \n",
       "L 303.154871 28.012657 \n",
       "z\n",
       "\" clip-path=\"url(#pedb0ac1f31)\" style=\"fill: #e41a1c\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_12\">\n",
       "    <path d=\"M 303.154871 39.416853 \n",
       "L 436.005846 39.416853 \n",
       "L 436.005846 56.523147 \n",
       "L 303.154871 56.523147 \n",
       "z\n",
       "\" clip-path=\"url(#pedb0ac1f31)\" style=\"fill: #377eb8\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_13\">\n",
       "    <path d=\"M 303.154871 67.927343 \n",
       "L 443.82714 67.927343 \n",
       "L 443.82714 85.033636 \n",
       "L 303.154871 85.033636 \n",
       "z\n",
       "\" clip-path=\"url(#pedb0ac1f31)\" style=\"fill: #4daf4a\"/>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_3\">\n",
       "    <g id=\"xtick_4\">\n",
       "     <g id=\"line2d_7\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#mf2a8f50faa\" x=\"303.154871\" y=\"88.74\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_8\">\n",
       "      <!-- 0 -->\n",
       "      <g transform=\"translate(299.973621 103.338438)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_5\">\n",
       "     <g id=\"line2d_8\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#mf2a8f50faa\" x=\"353.570593\" y=\"88.74\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_9\">\n",
       "      <!-- 10 -->\n",
       "      <g transform=\"translate(347.208093 103.338438)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-31\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_6\">\n",
       "     <g id=\"line2d_9\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#mf2a8f50faa\" x=\"403.986314\" y=\"88.74\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_10\">\n",
       "      <!-- 20 -->\n",
       "      <g transform=\"translate(397.623814 103.338438)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-32\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"text_11\">\n",
       "     <!-- per GPU TFLOPS -->\n",
       "     <g transform=\"translate(336.315625 117.016563)scale(0.1 -0.1)\">\n",
       "      <defs>\n",
       "       <path id=\"DejaVuSans-72\" d=\"M 2631 2963 \n",
       "Q 2534 3019 2420 3045 \n",
       "Q 2306 3072 2169 3072 \n",
       "Q 1681 3072 1420 2755 \n",
       "Q 1159 2438 1159 1844 \n",
       "L 1159 0 \n",
       "L 581 0 \n",
       "L 581 3500 \n",
       "L 1159 3500 \n",
       "L 1159 2956 \n",
       "Q 1341 3275 1631 3429 \n",
       "Q 1922 3584 2338 3584 \n",
       "Q 2397 3584 2469 3576 \n",
       "Q 2541 3569 2628 3553 \n",
       "L 2631 2963 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-47\" d=\"M 3809 666 \n",
       "L 3809 1919 \n",
       "L 2778 1919 \n",
       "L 2778 2438 \n",
       "L 4434 2438 \n",
       "L 4434 434 \n",
       "Q 4069 175 3628 42 \n",
       "Q 3188 -91 2688 -91 \n",
       "Q 1594 -91 976 548 \n",
       "Q 359 1188 359 2328 \n",
       "Q 359 3472 976 4111 \n",
       "Q 1594 4750 2688 4750 \n",
       "Q 3144 4750 3555 4637 \n",
       "Q 3966 4525 4313 4306 \n",
       "L 4313 3634 \n",
       "Q 3963 3931 3569 4081 \n",
       "Q 3175 4231 2741 4231 \n",
       "Q 1884 4231 1454 3753 \n",
       "Q 1025 3275 1025 2328 \n",
       "Q 1025 1384 1454 906 \n",
       "Q 1884 428 2741 428 \n",
       "Q 3075 428 3337 486 \n",
       "Q 3600 544 3809 666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-50\" d=\"M 1259 4147 \n",
       "L 1259 2394 \n",
       "L 2053 2394 \n",
       "Q 2494 2394 2734 2622 \n",
       "Q 2975 2850 2975 3272 \n",
       "Q 2975 3691 2734 3919 \n",
       "Q 2494 4147 2053 4147 \n",
       "L 1259 4147 \n",
       "z\n",
       "M 628 4666 \n",
       "L 2053 4666 \n",
       "Q 2838 4666 3239 4311 \n",
       "Q 3641 3956 3641 3272 \n",
       "Q 3641 2581 3239 2228 \n",
       "Q 2838 1875 2053 1875 \n",
       "L 1259 1875 \n",
       "L 1259 0 \n",
       "L 628 0 \n",
       "L 628 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-55\" d=\"M 556 4666 \n",
       "L 1191 4666 \n",
       "L 1191 1831 \n",
       "Q 1191 1081 1462 751 \n",
       "Q 1734 422 2344 422 \n",
       "Q 2950 422 3222 751 \n",
       "Q 3494 1081 3494 1831 \n",
       "L 3494 4666 \n",
       "L 4128 4666 \n",
       "L 4128 1753 \n",
       "Q 4128 841 3676 375 \n",
       "Q 3225 -91 2344 -91 \n",
       "Q 1459 -91 1007 375 \n",
       "Q 556 841 556 1753 \n",
       "L 556 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-54\" d=\"M -19 4666 \n",
       "L 3928 4666 \n",
       "L 3928 4134 \n",
       "L 2272 4134 \n",
       "L 2272 0 \n",
       "L 1638 0 \n",
       "L 1638 4134 \n",
       "L -19 4134 \n",
       "L -19 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-4c\" d=\"M 628 4666 \n",
       "L 1259 4666 \n",
       "L 1259 531 \n",
       "L 3531 531 \n",
       "L 3531 0 \n",
       "L 628 0 \n",
       "L 628 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-4f\" d=\"M 2522 4238 \n",
       "Q 1834 4238 1429 3725 \n",
       "Q 1025 3213 1025 2328 \n",
       "Q 1025 1447 1429 934 \n",
       "Q 1834 422 2522 422 \n",
       "Q 3209 422 3611 934 \n",
       "Q 4013 1447 4013 2328 \n",
       "Q 4013 3213 3611 3725 \n",
       "Q 3209 4238 2522 4238 \n",
       "z\n",
       "M 2522 4750 \n",
       "Q 3503 4750 4090 4092 \n",
       "Q 4678 3434 4678 2328 \n",
       "Q 4678 1225 4090 567 \n",
       "Q 3503 -91 2522 -91 \n",
       "Q 1538 -91 948 565 \n",
       "Q 359 1222 359 2328 \n",
       "Q 359 3434 948 4092 \n",
       "Q 1538 4750 2522 4750 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "      </defs>\n",
       "      <use xlink:href=\"#DejaVuSans-70\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-65\" x=\"63.476562\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-72\" x=\"125\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-20\" x=\"166.113281\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-47\" x=\"197.900391\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-50\" x=\"275.390625\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-55\" x=\"335.693359\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-20\" x=\"408.886719\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-54\" x=\"440.673828\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-46\" x=\"501.757812\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-4c\" x=\"559.277344\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-4f\" x=\"611.365234\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-50\" x=\"690.076172\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-53\" x=\"750.378906\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_4\">\n",
       "    <g id=\"ytick_4\">\n",
       "     <g id=\"line2d_10\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m5514f5ae47\" x=\"303.154871\" y=\"19.45951\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_5\">\n",
       "     <g id=\"line2d_11\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m5514f5ae47\" x=\"303.154871\" y=\"47.97\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_6\">\n",
       "     <g id=\"line2d_12\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m5514f5ae47\" x=\"303.154871\" y=\"76.48049\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"patch_14\">\n",
       "    <path d=\"M 303.154871 88.74 \n",
       "L 303.154871 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_15\">\n",
       "    <path d=\"M 450.860754 88.74 \n",
       "L 450.860754 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_16\">\n",
       "    <path d=\"M 303.154871 88.74 \n",
       "L 450.860754 88.74 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_17\">\n",
       "    <path d=\"M 303.154871 7.2 \n",
       "L 450.860754 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "  </g>\n",
       "  <g id=\"axes_3\">\n",
       "   <g id=\"patch_18\">\n",
       "    <path d=\"M 480.40193 88.74 \n",
       "L 628.107813 88.74 \n",
       "L 628.107813 7.2 \n",
       "L 480.40193 7.2 \n",
       "z\n",
       "\" style=\"fill: #ffffff\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_19\">\n",
       "    <path d=\"M 480.40193 10.906364 \n",
       "L 614.70053 10.906364 \n",
       "L 614.70053 28.012657 \n",
       "L 480.40193 28.012657 \n",
       "z\n",
       "\" clip-path=\"url(#p717a9bc195)\" style=\"fill: #e41a1c\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_20\">\n",
       "    <path d=\"M 480.40193 39.416853 \n",
       "L 621.074199 39.416853 \n",
       "L 621.074199 56.523147 \n",
       "L 480.40193 56.523147 \n",
       "z\n",
       "\" clip-path=\"url(#p717a9bc195)\" style=\"fill: #377eb8\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_21\">\n",
       "    <path d=\"M 480.40193 67.927343 \n",
       "L 621.040848 67.927343 \n",
       "L 621.040848 85.033636 \n",
       "L 480.40193 85.033636 \n",
       "z\n",
       "\" clip-path=\"url(#p717a9bc195)\" style=\"fill: #4daf4a\"/>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_5\">\n",
       "    <g id=\"xtick_7\">\n",
       "     <g id=\"line2d_13\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#mf2a8f50faa\" x=\"480.40193\" y=\"88.74\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_12\">\n",
       "      <!-- 0 -->\n",
       "      <g transform=\"translate(477.22068 103.338438)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_8\">\n",
       "     <g id=\"line2d_14\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#mf2a8f50faa\" x=\"542.069524\" y=\"88.74\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_13\">\n",
       "      <!-- 10 -->\n",
       "      <g transform=\"translate(535.707024 103.338438)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-31\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_9\">\n",
       "     <g id=\"line2d_15\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#mf2a8f50faa\" x=\"603.737119\" y=\"88.74\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_14\">\n",
       "      <!-- 20 -->\n",
       "      <g transform=\"translate(597.374619 103.338438)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-32\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"text_15\">\n",
       "     <!-- per GPU memory (GB) -->\n",
       "     <g transform=\"translate(498.53534 117.016563)scale(0.1 -0.1)\">\n",
       "      <defs>\n",
       "       <path id=\"DejaVuSans-6f\" d=\"M 1959 3097 \n",
       "Q 1497 3097 1228 2736 \n",
       "Q 959 2375 959 1747 \n",
       "Q 959 1119 1226 758 \n",
       "Q 1494 397 1959 397 \n",
       "Q 2419 397 2687 759 \n",
       "Q 2956 1122 2956 1747 \n",
       "Q 2956 2369 2687 2733 \n",
       "Q 2419 3097 1959 3097 \n",
       "z\n",
       "M 1959 3584 \n",
       "Q 2709 3584 3137 3096 \n",
       "Q 3566 2609 3566 1747 \n",
       "Q 3566 888 3137 398 \n",
       "Q 2709 -91 1959 -91 \n",
       "Q 1206 -91 779 398 \n",
       "Q 353 888 353 1747 \n",
       "Q 353 2609 779 3096 \n",
       "Q 1206 3584 1959 3584 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-79\" d=\"M 2059 -325 \n",
       "Q 1816 -950 1584 -1140 \n",
       "Q 1353 -1331 966 -1331 \n",
       "L 506 -1331 \n",
       "L 506 -850 \n",
       "L 844 -850 \n",
       "Q 1081 -850 1212 -737 \n",
       "Q 1344 -625 1503 -206 \n",
       "L 1606 56 \n",
       "L 191 3500 \n",
       "L 800 3500 \n",
       "L 1894 763 \n",
       "L 2988 3500 \n",
       "L 3597 3500 \n",
       "L 2059 -325 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-28\" d=\"M 1984 4856 \n",
       "Q 1566 4138 1362 3434 \n",
       "Q 1159 2731 1159 2009 \n",
       "Q 1159 1288 1364 580 \n",
       "Q 1569 -128 1984 -844 \n",
       "L 1484 -844 \n",
       "Q 1016 -109 783 600 \n",
       "Q 550 1309 550 2009 \n",
       "Q 550 2706 781 3412 \n",
       "Q 1013 4119 1484 4856 \n",
       "L 1984 4856 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-42\" d=\"M 1259 2228 \n",
       "L 1259 519 \n",
       "L 2272 519 \n",
       "Q 2781 519 3026 730 \n",
       "Q 3272 941 3272 1375 \n",
       "Q 3272 1813 3026 2020 \n",
       "Q 2781 2228 2272 2228 \n",
       "L 1259 2228 \n",
       "z\n",
       "M 1259 4147 \n",
       "L 1259 2741 \n",
       "L 2194 2741 \n",
       "Q 2656 2741 2882 2914 \n",
       "Q 3109 3088 3109 3444 \n",
       "Q 3109 3797 2882 3972 \n",
       "Q 2656 4147 2194 4147 \n",
       "L 1259 4147 \n",
       "z\n",
       "M 628 4666 \n",
       "L 2241 4666 \n",
       "Q 2963 4666 3353 4366 \n",
       "Q 3744 4066 3744 3513 \n",
       "Q 3744 3084 3544 2831 \n",
       "Q 3344 2578 2956 2516 \n",
       "Q 3422 2416 3680 2098 \n",
       "Q 3938 1781 3938 1306 \n",
       "Q 3938 681 3513 340 \n",
       "Q 3088 0 2303 0 \n",
       "L 628 0 \n",
       "L 628 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-29\" d=\"M 513 4856 \n",
       "L 1013 4856 \n",
       "Q 1481 4119 1714 3412 \n",
       "Q 1947 2706 1947 2009 \n",
       "Q 1947 1309 1714 600 \n",
       "Q 1481 -109 1013 -844 \n",
       "L 513 -844 \n",
       "Q 928 -128 1133 580 \n",
       "Q 1338 1288 1338 2009 \n",
       "Q 1338 2731 1133 3434 \n",
       "Q 928 4138 513 4856 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "      </defs>\n",
       "      <use xlink:href=\"#DejaVuSans-70\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-65\" x=\"63.476562\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-72\" x=\"125\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-20\" x=\"166.113281\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-47\" x=\"197.900391\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-50\" x=\"275.390625\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-55\" x=\"335.693359\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-20\" x=\"408.886719\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-6d\" x=\"440.673828\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-65\" x=\"538.085938\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-6d\" x=\"599.609375\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-6f\" x=\"697.021484\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-72\" x=\"758.203125\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-79\" x=\"799.316406\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-20\" x=\"858.496094\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-28\" x=\"890.283203\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-47\" x=\"929.296875\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-42\" x=\"1006.787109\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-29\" x=\"1075.390625\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_6\">\n",
       "    <g id=\"ytick_7\">\n",
       "     <g id=\"line2d_16\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m5514f5ae47\" x=\"480.40193\" y=\"19.45951\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_8\">\n",
       "     <g id=\"line2d_17\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m5514f5ae47\" x=\"480.40193\" y=\"47.97\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_9\">\n",
       "     <g id=\"line2d_18\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m5514f5ae47\" x=\"480.40193\" y=\"76.48049\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"patch_22\">\n",
       "    <path d=\"M 480.40193 88.74 \n",
       "L 480.40193 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_23\">\n",
       "    <path d=\"M 628.107813 88.74 \n",
       "L 628.107813 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_24\">\n",
       "    <path d=\"M 480.40193 88.74 \n",
       "L 628.107813 88.74 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_25\">\n",
       "    <path d=\"M 480.40193 7.2 \n",
       "L 628.107813 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "  </g>\n",
       " </g>\n",
       " <defs>\n",
       "  <clipPath id=\"p32864c3797\">\n",
       "   <rect x=\"125.907813\" y=\"7.2\" width=\"147.705882\" height=\"81.54\"/>\n",
       "  </clipPath>\n",
       "  <clipPath id=\"pedb0ac1f31\">\n",
       "   <rect x=\"303.154871\" y=\"7.2\" width=\"147.705882\" height=\"81.54\"/>\n",
       "  </clipPath>\n",
       "  <clipPath id=\"p717a9bc195\">\n",
       "   <rect x=\"480.40193\" y=\"7.2\" width=\"147.705882\" height=\"81.54\"/>\n",
       "  </clipPath>\n",
       " </defs>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<Figure size 648x108 with 3 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "bert_half_fused = hf_bert(Exp(\n",
    "    'HF 16-bit, fused-adam', 'bert-large-uncased', 11, bf16=True, optim='adamw_apex_fused'))\n",
    "compare([bert_single, bert_half, bert_half_fused])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a7b12ca2",
   "metadata": {},
   "source": [
    "To further reduce the optimization overhead, we can accumulate the gradients multiple times before updating weight. If we accumulate 4 times, then it leads to an 4x larger effective batch size. It may be too big for the fine tuning task, but often not a problem for pre-training."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "a0bc0213",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       "  \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<svg xmlns:xlink=\"http://www.w3.org/1999/xlink\" width=\"635.307812pt\" height=\"153.47625pt\" viewBox=\"0 0 635.307812 153.47625\" xmlns=\"http://www.w3.org/2000/svg\" version=\"1.1\">\n",
       " <metadata>\n",
       "  <rdf:RDF xmlns:dc=\"http://purl.org/dc/elements/1.1/\" xmlns:cc=\"http://creativecommons.org/ns#\" xmlns:rdf=\"http://www.w3.org/1999/02/22-rdf-syntax-ns#\">\n",
       "   <cc:Work>\n",
       "    <dc:type rdf:resource=\"http://purl.org/dc/dcmitype/StillImage\"/>\n",
       "    <dc:date>2022-09-02T19:41:06.757282</dc:date>\n",
       "    <dc:format>image/svg+xml</dc:format>\n",
       "    <dc:creator>\n",
       "     <cc:Agent>\n",
       "      <dc:title>Matplotlib v3.5.2, https://matplotlib.org/</dc:title>\n",
       "     </cc:Agent>\n",
       "    </dc:creator>\n",
       "   </cc:Work>\n",
       "  </rdf:RDF>\n",
       " </metadata>\n",
       " <defs>\n",
       "  <style type=\"text/css\">*{stroke-linejoin: round; stroke-linecap: butt}</style>\n",
       " </defs>\n",
       " <g id=\"figure_1\">\n",
       "  <g id=\"patch_1\">\n",
       "   <path d=\"M 0 153.47625 \n",
       "L 635.307812 153.47625 \n",
       "L 635.307812 0 \n",
       "L 0 0 \n",
       "L 0 153.47625 \n",
       "z\n",
       "\" style=\"fill: none\"/>\n",
       "  </g>\n",
       "  <g id=\"axes_1\">\n",
       "   <g id=\"patch_2\">\n",
       "    <path d=\"M 125.907813 115.92 \n",
       "L 273.613695 115.92 \n",
       "L 273.613695 7.2 \n",
       "L 125.907813 7.2 \n",
       "z\n",
       "\" style=\"fill: #ffffff\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_3\">\n",
       "    <path d=\"M 125.907813 12.141818 \n",
       "L 205.929428 12.141818 \n",
       "L 205.929428 28.614545 \n",
       "L 125.907813 28.614545 \n",
       "z\n",
       "\" clip-path=\"url(#pd70a7b60cd)\" style=\"fill: #e41a1c\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_4\">\n",
       "    <path d=\"M 125.907813 39.596364 \n",
       "L 248.360326 39.596364 \n",
       "L 248.360326 56.069091 \n",
       "L 125.907813 56.069091 \n",
       "z\n",
       "\" clip-path=\"url(#pd70a7b60cd)\" style=\"fill: #377eb8\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_5\">\n",
       "    <path d=\"M 125.907813 67.050909 \n",
       "L 255.569435 67.050909 \n",
       "L 255.569435 83.523636 \n",
       "L 125.907813 83.523636 \n",
       "z\n",
       "\" clip-path=\"url(#pd70a7b60cd)\" style=\"fill: #4daf4a\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_6\">\n",
       "    <path d=\"M 125.907813 94.505455 \n",
       "L 266.580081 94.505455 \n",
       "L 266.580081 110.978182 \n",
       "L 125.907813 110.978182 \n",
       "z\n",
       "\" clip-path=\"url(#pd70a7b60cd)\" style=\"fill: #984ea3\"/>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_1\">\n",
       "    <g id=\"xtick_1\">\n",
       "     <g id=\"line2d_1\">\n",
       "      <defs>\n",
       "       <path id=\"m1991842d50\" d=\"M 0 0 \n",
       "L 0 3.5 \n",
       "\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </defs>\n",
       "      <g>\n",
       "       <use xlink:href=\"#m1991842d50\" x=\"125.907813\" y=\"115.92\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_1\">\n",
       "      <!-- 0 -->\n",
       "      <g transform=\"translate(122.726562 130.518438)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-30\" d=\"M 2034 4250 \n",
       "Q 1547 4250 1301 3770 \n",
       "Q 1056 3291 1056 2328 \n",
       "Q 1056 1369 1301 889 \n",
       "Q 1547 409 2034 409 \n",
       "Q 2525 409 2770 889 \n",
       "Q 3016 1369 3016 2328 \n",
       "Q 3016 3291 2770 3770 \n",
       "Q 2525 4250 2034 4250 \n",
       "z\n",
       "M 2034 4750 \n",
       "Q 2819 4750 3233 4129 \n",
       "Q 3647 3509 3647 2328 \n",
       "Q 3647 1150 3233 529 \n",
       "Q 2819 -91 2034 -91 \n",
       "Q 1250 -91 836 529 \n",
       "Q 422 1150 422 2328 \n",
       "Q 422 3509 836 4129 \n",
       "Q 1250 4750 2034 4750 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_2\">\n",
       "     <g id=\"line2d_2\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m1991842d50\" x=\"177.072528\" y=\"115.92\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_2\">\n",
       "      <!-- 10 -->\n",
       "      <g transform=\"translate(170.710028 130.518438)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-31\" d=\"M 794 531 \n",
       "L 1825 531 \n",
       "L 1825 4091 \n",
       "L 703 3866 \n",
       "L 703 4441 \n",
       "L 1819 4666 \n",
       "L 2450 4666 \n",
       "L 2450 531 \n",
       "L 3481 531 \n",
       "L 3481 0 \n",
       "L 794 0 \n",
       "L 794 531 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-31\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_3\">\n",
       "     <g id=\"line2d_3\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m1991842d50\" x=\"228.237244\" y=\"115.92\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_3\">\n",
       "      <!-- 20 -->\n",
       "      <g transform=\"translate(221.874744 130.518438)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-32\" d=\"M 1228 531 \n",
       "L 3431 531 \n",
       "L 3431 0 \n",
       "L 469 0 \n",
       "L 469 531 \n",
       "Q 828 903 1448 1529 \n",
       "Q 2069 2156 2228 2338 \n",
       "Q 2531 2678 2651 2914 \n",
       "Q 2772 3150 2772 3378 \n",
       "Q 2772 3750 2511 3984 \n",
       "Q 2250 4219 1831 4219 \n",
       "Q 1534 4219 1204 4116 \n",
       "Q 875 4013 500 3803 \n",
       "L 500 4441 \n",
       "Q 881 4594 1212 4672 \n",
       "Q 1544 4750 1819 4750 \n",
       "Q 2544 4750 2975 4387 \n",
       "Q 3406 4025 3406 3419 \n",
       "Q 3406 3131 3298 2873 \n",
       "Q 3191 2616 2906 2266 \n",
       "Q 2828 2175 2409 1742 \n",
       "Q 1991 1309 1228 531 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-32\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"text_4\">\n",
       "     <!-- Samples / sec -->\n",
       "     <g transform=\"translate(165.11466 144.196562)scale(0.1 -0.1)\">\n",
       "      <defs>\n",
       "       <path id=\"DejaVuSans-53\" d=\"M 3425 4513 \n",
       "L 3425 3897 \n",
       "Q 3066 4069 2747 4153 \n",
       "Q 2428 4238 2131 4238 \n",
       "Q 1616 4238 1336 4038 \n",
       "Q 1056 3838 1056 3469 \n",
       "Q 1056 3159 1242 3001 \n",
       "Q 1428 2844 1947 2747 \n",
       "L 2328 2669 \n",
       "Q 3034 2534 3370 2195 \n",
       "Q 3706 1856 3706 1288 \n",
       "Q 3706 609 3251 259 \n",
       "Q 2797 -91 1919 -91 \n",
       "Q 1588 -91 1214 -16 \n",
       "Q 841 59 441 206 \n",
       "L 441 856 \n",
       "Q 825 641 1194 531 \n",
       "Q 1563 422 1919 422 \n",
       "Q 2459 422 2753 634 \n",
       "Q 3047 847 3047 1241 \n",
       "Q 3047 1584 2836 1778 \n",
       "Q 2625 1972 2144 2069 \n",
       "L 1759 2144 \n",
       "Q 1053 2284 737 2584 \n",
       "Q 422 2884 422 3419 \n",
       "Q 422 4038 858 4394 \n",
       "Q 1294 4750 2059 4750 \n",
       "Q 2388 4750 2728 4690 \n",
       "Q 3069 4631 3425 4513 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-61\" d=\"M 2194 1759 \n",
       "Q 1497 1759 1228 1600 \n",
       "Q 959 1441 959 1056 \n",
       "Q 959 750 1161 570 \n",
       "Q 1363 391 1709 391 \n",
       "Q 2188 391 2477 730 \n",
       "Q 2766 1069 2766 1631 \n",
       "L 2766 1759 \n",
       "L 2194 1759 \n",
       "z\n",
       "M 3341 1997 \n",
       "L 3341 0 \n",
       "L 2766 0 \n",
       "L 2766 531 \n",
       "Q 2569 213 2275 61 \n",
       "Q 1981 -91 1556 -91 \n",
       "Q 1019 -91 701 211 \n",
       "Q 384 513 384 1019 \n",
       "Q 384 1609 779 1909 \n",
       "Q 1175 2209 1959 2209 \n",
       "L 2766 2209 \n",
       "L 2766 2266 \n",
       "Q 2766 2663 2505 2880 \n",
       "Q 2244 3097 1772 3097 \n",
       "Q 1472 3097 1187 3025 \n",
       "Q 903 2953 641 2809 \n",
       "L 641 3341 \n",
       "Q 956 3463 1253 3523 \n",
       "Q 1550 3584 1831 3584 \n",
       "Q 2591 3584 2966 3190 \n",
       "Q 3341 2797 3341 1997 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-6d\" d=\"M 3328 2828 \n",
       "Q 3544 3216 3844 3400 \n",
       "Q 4144 3584 4550 3584 \n",
       "Q 5097 3584 5394 3201 \n",
       "Q 5691 2819 5691 2113 \n",
       "L 5691 0 \n",
       "L 5113 0 \n",
       "L 5113 2094 \n",
       "Q 5113 2597 4934 2840 \n",
       "Q 4756 3084 4391 3084 \n",
       "Q 3944 3084 3684 2787 \n",
       "Q 3425 2491 3425 1978 \n",
       "L 3425 0 \n",
       "L 2847 0 \n",
       "L 2847 2094 \n",
       "Q 2847 2600 2669 2842 \n",
       "Q 2491 3084 2119 3084 \n",
       "Q 1678 3084 1418 2786 \n",
       "Q 1159 2488 1159 1978 \n",
       "L 1159 0 \n",
       "L 581 0 \n",
       "L 581 3500 \n",
       "L 1159 3500 \n",
       "L 1159 2956 \n",
       "Q 1356 3278 1631 3431 \n",
       "Q 1906 3584 2284 3584 \n",
       "Q 2666 3584 2933 3390 \n",
       "Q 3200 3197 3328 2828 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-70\" d=\"M 1159 525 \n",
       "L 1159 -1331 \n",
       "L 581 -1331 \n",
       "L 581 3500 \n",
       "L 1159 3500 \n",
       "L 1159 2969 \n",
       "Q 1341 3281 1617 3432 \n",
       "Q 1894 3584 2278 3584 \n",
       "Q 2916 3584 3314 3078 \n",
       "Q 3713 2572 3713 1747 \n",
       "Q 3713 922 3314 415 \n",
       "Q 2916 -91 2278 -91 \n",
       "Q 1894 -91 1617 61 \n",
       "Q 1341 213 1159 525 \n",
       "z\n",
       "M 3116 1747 \n",
       "Q 3116 2381 2855 2742 \n",
       "Q 2594 3103 2138 3103 \n",
       "Q 1681 3103 1420 2742 \n",
       "Q 1159 2381 1159 1747 \n",
       "Q 1159 1113 1420 752 \n",
       "Q 1681 391 2138 391 \n",
       "Q 2594 391 2855 752 \n",
       "Q 3116 1113 3116 1747 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-6c\" d=\"M 603 4863 \n",
       "L 1178 4863 \n",
       "L 1178 0 \n",
       "L 603 0 \n",
       "L 603 4863 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-65\" d=\"M 3597 1894 \n",
       "L 3597 1613 \n",
       "L 953 1613 \n",
       "Q 991 1019 1311 708 \n",
       "Q 1631 397 2203 397 \n",
       "Q 2534 397 2845 478 \n",
       "Q 3156 559 3463 722 \n",
       "L 3463 178 \n",
       "Q 3153 47 2828 -22 \n",
       "Q 2503 -91 2169 -91 \n",
       "Q 1331 -91 842 396 \n",
       "Q 353 884 353 1716 \n",
       "Q 353 2575 817 3079 \n",
       "Q 1281 3584 2069 3584 \n",
       "Q 2775 3584 3186 3129 \n",
       "Q 3597 2675 3597 1894 \n",
       "z\n",
       "M 3022 2063 \n",
       "Q 3016 2534 2758 2815 \n",
       "Q 2500 3097 2075 3097 \n",
       "Q 1594 3097 1305 2825 \n",
       "Q 1016 2553 972 2059 \n",
       "L 3022 2063 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-73\" d=\"M 2834 3397 \n",
       "L 2834 2853 \n",
       "Q 2591 2978 2328 3040 \n",
       "Q 2066 3103 1784 3103 \n",
       "Q 1356 3103 1142 2972 \n",
       "Q 928 2841 928 2578 \n",
       "Q 928 2378 1081 2264 \n",
       "Q 1234 2150 1697 2047 \n",
       "L 1894 2003 \n",
       "Q 2506 1872 2764 1633 \n",
       "Q 3022 1394 3022 966 \n",
       "Q 3022 478 2636 193 \n",
       "Q 2250 -91 1575 -91 \n",
       "Q 1294 -91 989 -36 \n",
       "Q 684 19 347 128 \n",
       "L 347 722 \n",
       "Q 666 556 975 473 \n",
       "Q 1284 391 1588 391 \n",
       "Q 1994 391 2212 530 \n",
       "Q 2431 669 2431 922 \n",
       "Q 2431 1156 2273 1281 \n",
       "Q 2116 1406 1581 1522 \n",
       "L 1381 1569 \n",
       "Q 847 1681 609 1914 \n",
       "Q 372 2147 372 2553 \n",
       "Q 372 3047 722 3315 \n",
       "Q 1072 3584 1716 3584 \n",
       "Q 2034 3584 2315 3537 \n",
       "Q 2597 3491 2834 3397 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-20\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-2f\" d=\"M 1625 4666 \n",
       "L 2156 4666 \n",
       "L 531 -594 \n",
       "L 0 -594 \n",
       "L 1625 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-63\" d=\"M 3122 3366 \n",
       "L 3122 2828 \n",
       "Q 2878 2963 2633 3030 \n",
       "Q 2388 3097 2138 3097 \n",
       "Q 1578 3097 1268 2742 \n",
       "Q 959 2388 959 1747 \n",
       "Q 959 1106 1268 751 \n",
       "Q 1578 397 2138 397 \n",
       "Q 2388 397 2633 464 \n",
       "Q 2878 531 3122 666 \n",
       "L 3122 134 \n",
       "Q 2881 22 2623 -34 \n",
       "Q 2366 -91 2075 -91 \n",
       "Q 1284 -91 818 406 \n",
       "Q 353 903 353 1747 \n",
       "Q 353 2603 823 3093 \n",
       "Q 1294 3584 2113 3584 \n",
       "Q 2378 3584 2631 3529 \n",
       "Q 2884 3475 3122 3366 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "      </defs>\n",
       "      <use xlink:href=\"#DejaVuSans-53\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-61\" x=\"63.476562\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-6d\" x=\"124.755859\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-70\" x=\"222.167969\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-6c\" x=\"285.644531\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-65\" x=\"313.427734\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-73\" x=\"374.951172\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-20\" x=\"427.050781\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-2f\" x=\"458.837891\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-20\" x=\"492.529297\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-73\" x=\"524.316406\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-65\" x=\"576.416016\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-63\" x=\"637.939453\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_2\">\n",
       "    <g id=\"ytick_1\">\n",
       "     <g id=\"line2d_4\">\n",
       "      <defs>\n",
       "       <path id=\"mddcd2f4e8f\" d=\"M 0 0 \n",
       "L -3.5 0 \n",
       "\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </defs>\n",
       "      <g>\n",
       "       <use xlink:href=\"#mddcd2f4e8f\" x=\"125.907813\" y=\"20.378182\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_5\">\n",
       "      <!-- HF 32-bit -->\n",
       "      <g transform=\"translate(73.078125 24.177401)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-48\" d=\"M 628 4666 \n",
       "L 1259 4666 \n",
       "L 1259 2753 \n",
       "L 3553 2753 \n",
       "L 3553 4666 \n",
       "L 4184 4666 \n",
       "L 4184 0 \n",
       "L 3553 0 \n",
       "L 3553 2222 \n",
       "L 1259 2222 \n",
       "L 1259 0 \n",
       "L 628 0 \n",
       "L 628 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-46\" d=\"M 628 4666 \n",
       "L 3309 4666 \n",
       "L 3309 4134 \n",
       "L 1259 4134 \n",
       "L 1259 2759 \n",
       "L 3109 2759 \n",
       "L 3109 2228 \n",
       "L 1259 2228 \n",
       "L 1259 0 \n",
       "L 628 0 \n",
       "L 628 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-33\" d=\"M 2597 2516 \n",
       "Q 3050 2419 3304 2112 \n",
       "Q 3559 1806 3559 1356 \n",
       "Q 3559 666 3084 287 \n",
       "Q 2609 -91 1734 -91 \n",
       "Q 1441 -91 1130 -33 \n",
       "Q 819 25 488 141 \n",
       "L 488 750 \n",
       "Q 750 597 1062 519 \n",
       "Q 1375 441 1716 441 \n",
       "Q 2309 441 2620 675 \n",
       "Q 2931 909 2931 1356 \n",
       "Q 2931 1769 2642 2001 \n",
       "Q 2353 2234 1838 2234 \n",
       "L 1294 2234 \n",
       "L 1294 2753 \n",
       "L 1863 2753 \n",
       "Q 2328 2753 2575 2939 \n",
       "Q 2822 3125 2822 3475 \n",
       "Q 2822 3834 2567 4026 \n",
       "Q 2313 4219 1838 4219 \n",
       "Q 1578 4219 1281 4162 \n",
       "Q 984 4106 628 3988 \n",
       "L 628 4550 \n",
       "Q 988 4650 1302 4700 \n",
       "Q 1616 4750 1894 4750 \n",
       "Q 2613 4750 3031 4423 \n",
       "Q 3450 4097 3450 3541 \n",
       "Q 3450 3153 3228 2886 \n",
       "Q 3006 2619 2597 2516 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-2d\" d=\"M 313 2009 \n",
       "L 1997 2009 \n",
       "L 1997 1497 \n",
       "L 313 1497 \n",
       "L 313 2009 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-62\" d=\"M 3116 1747 \n",
       "Q 3116 2381 2855 2742 \n",
       "Q 2594 3103 2138 3103 \n",
       "Q 1681 3103 1420 2742 \n",
       "Q 1159 2381 1159 1747 \n",
       "Q 1159 1113 1420 752 \n",
       "Q 1681 391 2138 391 \n",
       "Q 2594 391 2855 752 \n",
       "Q 3116 1113 3116 1747 \n",
       "z\n",
       "M 1159 2969 \n",
       "Q 1341 3281 1617 3432 \n",
       "Q 1894 3584 2278 3584 \n",
       "Q 2916 3584 3314 3078 \n",
       "Q 3713 2572 3713 1747 \n",
       "Q 3713 922 3314 415 \n",
       "Q 2916 -91 2278 -91 \n",
       "Q 1894 -91 1617 61 \n",
       "Q 1341 213 1159 525 \n",
       "L 1159 0 \n",
       "L 581 0 \n",
       "L 581 4863 \n",
       "L 1159 4863 \n",
       "L 1159 2969 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-69\" d=\"M 603 3500 \n",
       "L 1178 3500 \n",
       "L 1178 0 \n",
       "L 603 0 \n",
       "L 603 3500 \n",
       "z\n",
       "M 603 4863 \n",
       "L 1178 4863 \n",
       "L 1178 4134 \n",
       "L 603 4134 \n",
       "L 603 4863 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-74\" d=\"M 1172 4494 \n",
       "L 1172 3500 \n",
       "L 2356 3500 \n",
       "L 2356 3053 \n",
       "L 1172 3053 \n",
       "L 1172 1153 \n",
       "Q 1172 725 1289 603 \n",
       "Q 1406 481 1766 481 \n",
       "L 2356 481 \n",
       "L 2356 0 \n",
       "L 1766 0 \n",
       "Q 1100 0 847 248 \n",
       "Q 594 497 594 1153 \n",
       "L 594 3053 \n",
       "L 172 3053 \n",
       "L 172 3500 \n",
       "L 594 3500 \n",
       "L 594 4494 \n",
       "L 1172 4494 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-48\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-46\" x=\"75.195312\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-20\" x=\"132.714844\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-33\" x=\"164.501953\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-32\" x=\"228.125\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-2d\" x=\"291.748047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-62\" x=\"327.832031\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-69\" x=\"391.308594\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-74\" x=\"419.091797\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_2\">\n",
       "     <g id=\"line2d_5\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#mddcd2f4e8f\" x=\"125.907813\" y=\"47.832727\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_6\">\n",
       "      <!-- HF 16-bit -->\n",
       "      <g transform=\"translate(73.078125 51.631946)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-36\" d=\"M 2113 2584 \n",
       "Q 1688 2584 1439 2293 \n",
       "Q 1191 2003 1191 1497 \n",
       "Q 1191 994 1439 701 \n",
       "Q 1688 409 2113 409 \n",
       "Q 2538 409 2786 701 \n",
       "Q 3034 994 3034 1497 \n",
       "Q 3034 2003 2786 2293 \n",
       "Q 2538 2584 2113 2584 \n",
       "z\n",
       "M 3366 4563 \n",
       "L 3366 3988 \n",
       "Q 3128 4100 2886 4159 \n",
       "Q 2644 4219 2406 4219 \n",
       "Q 1781 4219 1451 3797 \n",
       "Q 1122 3375 1075 2522 \n",
       "Q 1259 2794 1537 2939 \n",
       "Q 1816 3084 2150 3084 \n",
       "Q 2853 3084 3261 2657 \n",
       "Q 3669 2231 3669 1497 \n",
       "Q 3669 778 3244 343 \n",
       "Q 2819 -91 2113 -91 \n",
       "Q 1303 -91 875 529 \n",
       "Q 447 1150 447 2328 \n",
       "Q 447 3434 972 4092 \n",
       "Q 1497 4750 2381 4750 \n",
       "Q 2619 4750 2861 4703 \n",
       "Q 3103 4656 3366 4563 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-48\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-46\" x=\"75.195312\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-20\" x=\"132.714844\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-31\" x=\"164.501953\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-36\" x=\"228.125\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-2d\" x=\"291.748047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-62\" x=\"327.832031\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-69\" x=\"391.308594\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-74\" x=\"419.091797\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_3\">\n",
       "     <g id=\"line2d_6\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#mddcd2f4e8f\" x=\"125.907813\" y=\"75.287273\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_7\">\n",
       "      <!-- HF 16-bit, fused-adam -->\n",
       "      <g transform=\"translate(7.2 79.086491)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-2c\" d=\"M 750 794 \n",
       "L 1409 794 \n",
       "L 1409 256 \n",
       "L 897 -744 \n",
       "L 494 -744 \n",
       "L 750 256 \n",
       "L 750 794 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-66\" d=\"M 2375 4863 \n",
       "L 2375 4384 \n",
       "L 1825 4384 \n",
       "Q 1516 4384 1395 4259 \n",
       "Q 1275 4134 1275 3809 \n",
       "L 1275 3500 \n",
       "L 2222 3500 \n",
       "L 2222 3053 \n",
       "L 1275 3053 \n",
       "L 1275 0 \n",
       "L 697 0 \n",
       "L 697 3053 \n",
       "L 147 3053 \n",
       "L 147 3500 \n",
       "L 697 3500 \n",
       "L 697 3744 \n",
       "Q 697 4328 969 4595 \n",
       "Q 1241 4863 1831 4863 \n",
       "L 2375 4863 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-75\" d=\"M 544 1381 \n",
       "L 544 3500 \n",
       "L 1119 3500 \n",
       "L 1119 1403 \n",
       "Q 1119 906 1312 657 \n",
       "Q 1506 409 1894 409 \n",
       "Q 2359 409 2629 706 \n",
       "Q 2900 1003 2900 1516 \n",
       "L 2900 3500 \n",
       "L 3475 3500 \n",
       "L 3475 0 \n",
       "L 2900 0 \n",
       "L 2900 538 \n",
       "Q 2691 219 2414 64 \n",
       "Q 2138 -91 1772 -91 \n",
       "Q 1169 -91 856 284 \n",
       "Q 544 659 544 1381 \n",
       "z\n",
       "M 1991 3584 \n",
       "L 1991 3584 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-64\" d=\"M 2906 2969 \n",
       "L 2906 4863 \n",
       "L 3481 4863 \n",
       "L 3481 0 \n",
       "L 2906 0 \n",
       "L 2906 525 \n",
       "Q 2725 213 2448 61 \n",
       "Q 2172 -91 1784 -91 \n",
       "Q 1150 -91 751 415 \n",
       "Q 353 922 353 1747 \n",
       "Q 353 2572 751 3078 \n",
       "Q 1150 3584 1784 3584 \n",
       "Q 2172 3584 2448 3432 \n",
       "Q 2725 3281 2906 2969 \n",
       "z\n",
       "M 947 1747 \n",
       "Q 947 1113 1208 752 \n",
       "Q 1469 391 1925 391 \n",
       "Q 2381 391 2643 752 \n",
       "Q 2906 1113 2906 1747 \n",
       "Q 2906 2381 2643 2742 \n",
       "Q 2381 3103 1925 3103 \n",
       "Q 1469 3103 1208 2742 \n",
       "Q 947 2381 947 1747 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-48\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-46\" x=\"75.195312\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-20\" x=\"132.714844\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-31\" x=\"164.501953\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-36\" x=\"228.125\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-2d\" x=\"291.748047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-62\" x=\"327.832031\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-69\" x=\"391.308594\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-74\" x=\"419.091797\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-2c\" x=\"458.300781\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-20\" x=\"490.087891\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-66\" x=\"521.875\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-75\" x=\"557.080078\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-73\" x=\"620.458984\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-65\" x=\"672.558594\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-64\" x=\"734.082031\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-2d\" x=\"797.558594\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-61\" x=\"833.642578\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-64\" x=\"894.921875\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-61\" x=\"958.398438\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-6d\" x=\"1019.677734\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_4\">\n",
       "     <g id=\"line2d_7\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#mddcd2f4e8f\" x=\"125.907813\" y=\"102.741818\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_8\">\n",
       "      <!-- HF 16-bit, fused-adam -->\n",
       "      <g transform=\"translate(7.2 101.701974)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-48\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-46\" x=\"75.195312\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-20\" x=\"132.714844\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-31\" x=\"164.501953\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-36\" x=\"228.125\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-2d\" x=\"291.748047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-62\" x=\"327.832031\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-69\" x=\"391.308594\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-74\" x=\"419.091797\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-2c\" x=\"458.300781\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-20\" x=\"490.087891\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-66\" x=\"521.875\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-75\" x=\"557.080078\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-73\" x=\"620.458984\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-65\" x=\"672.558594\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-64\" x=\"734.082031\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-2d\" x=\"797.558594\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-61\" x=\"833.642578\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-64\" x=\"894.921875\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-61\" x=\"958.398438\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-6d\" x=\"1019.677734\"/>\n",
       "      </g>\n",
       "      <!-- grad_accum=4 -->\n",
       "      <g transform=\"translate(43.026562 112.899787)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-67\" d=\"M 2906 1791 \n",
       "Q 2906 2416 2648 2759 \n",
       "Q 2391 3103 1925 3103 \n",
       "Q 1463 3103 1205 2759 \n",
       "Q 947 2416 947 1791 \n",
       "Q 947 1169 1205 825 \n",
       "Q 1463 481 1925 481 \n",
       "Q 2391 481 2648 825 \n",
       "Q 2906 1169 2906 1791 \n",
       "z\n",
       "M 3481 434 \n",
       "Q 3481 -459 3084 -895 \n",
       "Q 2688 -1331 1869 -1331 \n",
       "Q 1566 -1331 1297 -1286 \n",
       "Q 1028 -1241 775 -1147 \n",
       "L 775 -588 \n",
       "Q 1028 -725 1275 -790 \n",
       "Q 1522 -856 1778 -856 \n",
       "Q 2344 -856 2625 -561 \n",
       "Q 2906 -266 2906 331 \n",
       "L 2906 616 \n",
       "Q 2728 306 2450 153 \n",
       "Q 2172 0 1784 0 \n",
       "Q 1141 0 747 490 \n",
       "Q 353 981 353 1791 \n",
       "Q 353 2603 747 3093 \n",
       "Q 1141 3584 1784 3584 \n",
       "Q 2172 3584 2450 3431 \n",
       "Q 2728 3278 2906 2969 \n",
       "L 2906 3500 \n",
       "L 3481 3500 \n",
       "L 3481 434 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-72\" d=\"M 2631 2963 \n",
       "Q 2534 3019 2420 3045 \n",
       "Q 2306 3072 2169 3072 \n",
       "Q 1681 3072 1420 2755 \n",
       "Q 1159 2438 1159 1844 \n",
       "L 1159 0 \n",
       "L 581 0 \n",
       "L 581 3500 \n",
       "L 1159 3500 \n",
       "L 1159 2956 \n",
       "Q 1341 3275 1631 3429 \n",
       "Q 1922 3584 2338 3584 \n",
       "Q 2397 3584 2469 3576 \n",
       "Q 2541 3569 2628 3553 \n",
       "L 2631 2963 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-5f\" d=\"M 3263 -1063 \n",
       "L 3263 -1509 \n",
       "L -63 -1509 \n",
       "L -63 -1063 \n",
       "L 3263 -1063 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-3d\" d=\"M 678 2906 \n",
       "L 4684 2906 \n",
       "L 4684 2381 \n",
       "L 678 2381 \n",
       "L 678 2906 \n",
       "z\n",
       "M 678 1631 \n",
       "L 4684 1631 \n",
       "L 4684 1100 \n",
       "L 678 1100 \n",
       "L 678 1631 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-34\" d=\"M 2419 4116 \n",
       "L 825 1625 \n",
       "L 2419 1625 \n",
       "L 2419 4116 \n",
       "z\n",
       "M 2253 4666 \n",
       "L 3047 4666 \n",
       "L 3047 1625 \n",
       "L 3713 1625 \n",
       "L 3713 1100 \n",
       "L 3047 1100 \n",
       "L 3047 0 \n",
       "L 2419 0 \n",
       "L 2419 1100 \n",
       "L 313 1100 \n",
       "L 313 1709 \n",
       "L 2253 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-67\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-72\" x=\"63.476562\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-61\" x=\"104.589844\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-64\" x=\"165.869141\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-5f\" x=\"229.345703\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-61\" x=\"279.345703\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-63\" x=\"340.625\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-63\" x=\"395.605469\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-75\" x=\"450.585938\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-6d\" x=\"513.964844\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-3d\" x=\"611.376953\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-34\" x=\"695.166016\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"patch_7\">\n",
       "    <path d=\"M 125.907813 115.92 \n",
       "L 125.907813 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_8\">\n",
       "    <path d=\"M 273.613695 115.92 \n",
       "L 273.613695 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_9\">\n",
       "    <path d=\"M 125.907813 115.92 \n",
       "L 273.613695 115.92 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_10\">\n",
       "    <path d=\"M 125.907813 7.2 \n",
       "L 273.613695 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "  </g>\n",
       "  <g id=\"axes_2\">\n",
       "   <g id=\"patch_11\">\n",
       "    <path d=\"M 303.154871 115.92 \n",
       "L 450.860754 115.92 \n",
       "L 450.860754 7.2 \n",
       "L 303.154871 7.2 \n",
       "z\n",
       "\" style=\"fill: #ffffff\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_12\">\n",
       "    <path d=\"M 303.154871 12.141818 \n",
       "L 383.176486 12.141818 \n",
       "L 383.176486 28.614545 \n",
       "L 303.154871 28.614545 \n",
       "z\n",
       "\" clip-path=\"url(#p56e5bd4f95)\" style=\"fill: #e41a1c\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_13\">\n",
       "    <path d=\"M 303.154871 39.596364 \n",
       "L 425.607385 39.596364 \n",
       "L 425.607385 56.069091 \n",
       "L 303.154871 56.069091 \n",
       "z\n",
       "\" clip-path=\"url(#p56e5bd4f95)\" style=\"fill: #377eb8\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_14\">\n",
       "    <path d=\"M 303.154871 67.050909 \n",
       "L 432.816493 67.050909 \n",
       "L 432.816493 83.523636 \n",
       "L 303.154871 83.523636 \n",
       "z\n",
       "\" clip-path=\"url(#p56e5bd4f95)\" style=\"fill: #4daf4a\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_15\">\n",
       "    <path d=\"M 303.154871 94.505455 \n",
       "L 443.82714 94.505455 \n",
       "L 443.82714 110.978182 \n",
       "L 303.154871 110.978182 \n",
       "z\n",
       "\" clip-path=\"url(#p56e5bd4f95)\" style=\"fill: #984ea3\"/>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_3\">\n",
       "    <g id=\"xtick_4\">\n",
       "     <g id=\"line2d_8\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m1991842d50\" x=\"303.154871\" y=\"115.92\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_9\">\n",
       "      <!-- 0 -->\n",
       "      <g transform=\"translate(299.973621 130.518438)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_5\">\n",
       "     <g id=\"line2d_9\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m1991842d50\" x=\"349.624472\" y=\"115.92\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_10\">\n",
       "      <!-- 10 -->\n",
       "      <g transform=\"translate(343.261972 130.518438)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-31\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_6\">\n",
       "     <g id=\"line2d_10\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m1991842d50\" x=\"396.094073\" y=\"115.92\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_11\">\n",
       "      <!-- 20 -->\n",
       "      <g transform=\"translate(389.731573 130.518438)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-32\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_7\">\n",
       "     <g id=\"line2d_11\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m1991842d50\" x=\"442.563674\" y=\"115.92\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_12\">\n",
       "      <!-- 30 -->\n",
       "      <g transform=\"translate(436.201174 130.518438)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-33\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"text_13\">\n",
       "     <!-- per GPU TFLOPS -->\n",
       "     <g transform=\"translate(336.315625 144.196562)scale(0.1 -0.1)\">\n",
       "      <defs>\n",
       "       <path id=\"DejaVuSans-47\" d=\"M 3809 666 \n",
       "L 3809 1919 \n",
       "L 2778 1919 \n",
       "L 2778 2438 \n",
       "L 4434 2438 \n",
       "L 4434 434 \n",
       "Q 4069 175 3628 42 \n",
       "Q 3188 -91 2688 -91 \n",
       "Q 1594 -91 976 548 \n",
       "Q 359 1188 359 2328 \n",
       "Q 359 3472 976 4111 \n",
       "Q 1594 4750 2688 4750 \n",
       "Q 3144 4750 3555 4637 \n",
       "Q 3966 4525 4313 4306 \n",
       "L 4313 3634 \n",
       "Q 3963 3931 3569 4081 \n",
       "Q 3175 4231 2741 4231 \n",
       "Q 1884 4231 1454 3753 \n",
       "Q 1025 3275 1025 2328 \n",
       "Q 1025 1384 1454 906 \n",
       "Q 1884 428 2741 428 \n",
       "Q 3075 428 3337 486 \n",
       "Q 3600 544 3809 666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-50\" d=\"M 1259 4147 \n",
       "L 1259 2394 \n",
       "L 2053 2394 \n",
       "Q 2494 2394 2734 2622 \n",
       "Q 2975 2850 2975 3272 \n",
       "Q 2975 3691 2734 3919 \n",
       "Q 2494 4147 2053 4147 \n",
       "L 1259 4147 \n",
       "z\n",
       "M 628 4666 \n",
       "L 2053 4666 \n",
       "Q 2838 4666 3239 4311 \n",
       "Q 3641 3956 3641 3272 \n",
       "Q 3641 2581 3239 2228 \n",
       "Q 2838 1875 2053 1875 \n",
       "L 1259 1875 \n",
       "L 1259 0 \n",
       "L 628 0 \n",
       "L 628 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-55\" d=\"M 556 4666 \n",
       "L 1191 4666 \n",
       "L 1191 1831 \n",
       "Q 1191 1081 1462 751 \n",
       "Q 1734 422 2344 422 \n",
       "Q 2950 422 3222 751 \n",
       "Q 3494 1081 3494 1831 \n",
       "L 3494 4666 \n",
       "L 4128 4666 \n",
       "L 4128 1753 \n",
       "Q 4128 841 3676 375 \n",
       "Q 3225 -91 2344 -91 \n",
       "Q 1459 -91 1007 375 \n",
       "Q 556 841 556 1753 \n",
       "L 556 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-54\" d=\"M -19 4666 \n",
       "L 3928 4666 \n",
       "L 3928 4134 \n",
       "L 2272 4134 \n",
       "L 2272 0 \n",
       "L 1638 0 \n",
       "L 1638 4134 \n",
       "L -19 4134 \n",
       "L -19 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-4c\" d=\"M 628 4666 \n",
       "L 1259 4666 \n",
       "L 1259 531 \n",
       "L 3531 531 \n",
       "L 3531 0 \n",
       "L 628 0 \n",
       "L 628 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-4f\" d=\"M 2522 4238 \n",
       "Q 1834 4238 1429 3725 \n",
       "Q 1025 3213 1025 2328 \n",
       "Q 1025 1447 1429 934 \n",
       "Q 1834 422 2522 422 \n",
       "Q 3209 422 3611 934 \n",
       "Q 4013 1447 4013 2328 \n",
       "Q 4013 3213 3611 3725 \n",
       "Q 3209 4238 2522 4238 \n",
       "z\n",
       "M 2522 4750 \n",
       "Q 3503 4750 4090 4092 \n",
       "Q 4678 3434 4678 2328 \n",
       "Q 4678 1225 4090 567 \n",
       "Q 3503 -91 2522 -91 \n",
       "Q 1538 -91 948 565 \n",
       "Q 359 1222 359 2328 \n",
       "Q 359 3434 948 4092 \n",
       "Q 1538 4750 2522 4750 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "      </defs>\n",
       "      <use xlink:href=\"#DejaVuSans-70\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-65\" x=\"63.476562\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-72\" x=\"125\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-20\" x=\"166.113281\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-47\" x=\"197.900391\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-50\" x=\"275.390625\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-55\" x=\"335.693359\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-20\" x=\"408.886719\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-54\" x=\"440.673828\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-46\" x=\"501.757812\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-4c\" x=\"559.277344\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-4f\" x=\"611.365234\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-50\" x=\"690.076172\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-53\" x=\"750.378906\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_4\">\n",
       "    <g id=\"ytick_5\">\n",
       "     <g id=\"line2d_12\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#mddcd2f4e8f\" x=\"303.154871\" y=\"20.378182\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_6\">\n",
       "     <g id=\"line2d_13\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#mddcd2f4e8f\" x=\"303.154871\" y=\"47.832727\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_7\">\n",
       "     <g id=\"line2d_14\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#mddcd2f4e8f\" x=\"303.154871\" y=\"75.287273\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_8\">\n",
       "     <g id=\"line2d_15\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#mddcd2f4e8f\" x=\"303.154871\" y=\"102.741818\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"patch_16\">\n",
       "    <path d=\"M 303.154871 115.92 \n",
       "L 303.154871 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_17\">\n",
       "    <path d=\"M 450.860754 115.92 \n",
       "L 450.860754 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_18\">\n",
       "    <path d=\"M 303.154871 115.92 \n",
       "L 450.860754 115.92 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_19\">\n",
       "    <path d=\"M 303.154871 7.2 \n",
       "L 450.860754 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "  </g>\n",
       "  <g id=\"axes_3\">\n",
       "   <g id=\"patch_20\">\n",
       "    <path d=\"M 480.40193 115.92 \n",
       "L 628.107813 115.92 \n",
       "L 628.107813 7.2 \n",
       "L 480.40193 7.2 \n",
       "z\n",
       "\" style=\"fill: #ffffff\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_21\">\n",
       "    <path d=\"M 480.40193 12.141818 \n",
       "L 614.70053 12.141818 \n",
       "L 614.70053 28.614545 \n",
       "L 480.40193 28.614545 \n",
       "z\n",
       "\" clip-path=\"url(#p0033aee5ea)\" style=\"fill: #e41a1c\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_22\">\n",
       "    <path d=\"M 480.40193 39.596364 \n",
       "L 621.074199 39.596364 \n",
       "L 621.074199 56.069091 \n",
       "L 480.40193 56.069091 \n",
       "z\n",
       "\" clip-path=\"url(#p0033aee5ea)\" style=\"fill: #377eb8\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_23\">\n",
       "    <path d=\"M 480.40193 67.050909 \n",
       "L 621.040848 67.050909 \n",
       "L 621.040848 83.523636 \n",
       "L 480.40193 83.523636 \n",
       "z\n",
       "\" clip-path=\"url(#p0033aee5ea)\" style=\"fill: #4daf4a\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_24\">\n",
       "    <path d=\"M 480.40193 94.505455 \n",
       "L 621.057885 94.505455 \n",
       "L 621.057885 110.978182 \n",
       "L 480.40193 110.978182 \n",
       "z\n",
       "\" clip-path=\"url(#p0033aee5ea)\" style=\"fill: #984ea3\"/>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_5\">\n",
       "    <g id=\"xtick_8\">\n",
       "     <g id=\"line2d_16\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m1991842d50\" x=\"480.40193\" y=\"115.92\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_14\">\n",
       "      <!-- 0 -->\n",
       "      <g transform=\"translate(477.22068 130.518438)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_9\">\n",
       "     <g id=\"line2d_17\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m1991842d50\" x=\"542.069524\" y=\"115.92\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_15\">\n",
       "      <!-- 10 -->\n",
       "      <g transform=\"translate(535.707024 130.518438)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-31\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_10\">\n",
       "     <g id=\"line2d_18\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m1991842d50\" x=\"603.737119\" y=\"115.92\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_16\">\n",
       "      <!-- 20 -->\n",
       "      <g transform=\"translate(597.374619 130.518438)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-32\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"text_17\">\n",
       "     <!-- per GPU memory (GB) -->\n",
       "     <g transform=\"translate(498.53534 144.196562)scale(0.1 -0.1)\">\n",
       "      <defs>\n",
       "       <path id=\"DejaVuSans-6f\" d=\"M 1959 3097 \n",
       "Q 1497 3097 1228 2736 \n",
       "Q 959 2375 959 1747 \n",
       "Q 959 1119 1226 758 \n",
       "Q 1494 397 1959 397 \n",
       "Q 2419 397 2687 759 \n",
       "Q 2956 1122 2956 1747 \n",
       "Q 2956 2369 2687 2733 \n",
       "Q 2419 3097 1959 3097 \n",
       "z\n",
       "M 1959 3584 \n",
       "Q 2709 3584 3137 3096 \n",
       "Q 3566 2609 3566 1747 \n",
       "Q 3566 888 3137 398 \n",
       "Q 2709 -91 1959 -91 \n",
       "Q 1206 -91 779 398 \n",
       "Q 353 888 353 1747 \n",
       "Q 353 2609 779 3096 \n",
       "Q 1206 3584 1959 3584 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-79\" d=\"M 2059 -325 \n",
       "Q 1816 -950 1584 -1140 \n",
       "Q 1353 -1331 966 -1331 \n",
       "L 506 -1331 \n",
       "L 506 -850 \n",
       "L 844 -850 \n",
       "Q 1081 -850 1212 -737 \n",
       "Q 1344 -625 1503 -206 \n",
       "L 1606 56 \n",
       "L 191 3500 \n",
       "L 800 3500 \n",
       "L 1894 763 \n",
       "L 2988 3500 \n",
       "L 3597 3500 \n",
       "L 2059 -325 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-28\" d=\"M 1984 4856 \n",
       "Q 1566 4138 1362 3434 \n",
       "Q 1159 2731 1159 2009 \n",
       "Q 1159 1288 1364 580 \n",
       "Q 1569 -128 1984 -844 \n",
       "L 1484 -844 \n",
       "Q 1016 -109 783 600 \n",
       "Q 550 1309 550 2009 \n",
       "Q 550 2706 781 3412 \n",
       "Q 1013 4119 1484 4856 \n",
       "L 1984 4856 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-42\" d=\"M 1259 2228 \n",
       "L 1259 519 \n",
       "L 2272 519 \n",
       "Q 2781 519 3026 730 \n",
       "Q 3272 941 3272 1375 \n",
       "Q 3272 1813 3026 2020 \n",
       "Q 2781 2228 2272 2228 \n",
       "L 1259 2228 \n",
       "z\n",
       "M 1259 4147 \n",
       "L 1259 2741 \n",
       "L 2194 2741 \n",
       "Q 2656 2741 2882 2914 \n",
       "Q 3109 3088 3109 3444 \n",
       "Q 3109 3797 2882 3972 \n",
       "Q 2656 4147 2194 4147 \n",
       "L 1259 4147 \n",
       "z\n",
       "M 628 4666 \n",
       "L 2241 4666 \n",
       "Q 2963 4666 3353 4366 \n",
       "Q 3744 4066 3744 3513 \n",
       "Q 3744 3084 3544 2831 \n",
       "Q 3344 2578 2956 2516 \n",
       "Q 3422 2416 3680 2098 \n",
       "Q 3938 1781 3938 1306 \n",
       "Q 3938 681 3513 340 \n",
       "Q 3088 0 2303 0 \n",
       "L 628 0 \n",
       "L 628 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-29\" d=\"M 513 4856 \n",
       "L 1013 4856 \n",
       "Q 1481 4119 1714 3412 \n",
       "Q 1947 2706 1947 2009 \n",
       "Q 1947 1309 1714 600 \n",
       "Q 1481 -109 1013 -844 \n",
       "L 513 -844 \n",
       "Q 928 -128 1133 580 \n",
       "Q 1338 1288 1338 2009 \n",
       "Q 1338 2731 1133 3434 \n",
       "Q 928 4138 513 4856 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "      </defs>\n",
       "      <use xlink:href=\"#DejaVuSans-70\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-65\" x=\"63.476562\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-72\" x=\"125\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-20\" x=\"166.113281\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-47\" x=\"197.900391\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-50\" x=\"275.390625\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-55\" x=\"335.693359\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-20\" x=\"408.886719\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-6d\" x=\"440.673828\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-65\" x=\"538.085938\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-6d\" x=\"599.609375\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-6f\" x=\"697.021484\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-72\" x=\"758.203125\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-79\" x=\"799.316406\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-20\" x=\"858.496094\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-28\" x=\"890.283203\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-47\" x=\"929.296875\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-42\" x=\"1006.787109\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-29\" x=\"1075.390625\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_6\">\n",
       "    <g id=\"ytick_9\">\n",
       "     <g id=\"line2d_19\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#mddcd2f4e8f\" x=\"480.40193\" y=\"20.378182\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_10\">\n",
       "     <g id=\"line2d_20\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#mddcd2f4e8f\" x=\"480.40193\" y=\"47.832727\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_11\">\n",
       "     <g id=\"line2d_21\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#mddcd2f4e8f\" x=\"480.40193\" y=\"75.287273\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_12\">\n",
       "     <g id=\"line2d_22\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#mddcd2f4e8f\" x=\"480.40193\" y=\"102.741818\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"patch_25\">\n",
       "    <path d=\"M 480.40193 115.92 \n",
       "L 480.40193 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_26\">\n",
       "    <path d=\"M 628.107813 115.92 \n",
       "L 628.107813 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_27\">\n",
       "    <path d=\"M 480.40193 115.92 \n",
       "L 628.107813 115.92 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_28\">\n",
       "    <path d=\"M 480.40193 7.2 \n",
       "L 628.107813 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "  </g>\n",
       " </g>\n",
       " <defs>\n",
       "  <clipPath id=\"pd70a7b60cd\">\n",
       "   <rect x=\"125.907813\" y=\"7.2\" width=\"147.705882\" height=\"108.72\"/>\n",
       "  </clipPath>\n",
       "  <clipPath id=\"p56e5bd4f95\">\n",
       "   <rect x=\"303.154871\" y=\"7.2\" width=\"147.705882\" height=\"108.72\"/>\n",
       "  </clipPath>\n",
       "  <clipPath id=\"p0033aee5ea\">\n",
       "   <rect x=\"480.40193\" y=\"7.2\" width=\"147.705882\" height=\"108.72\"/>\n",
       "  </clipPath>\n",
       " </defs>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<Figure size 648x144 with 3 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "bert_half_fused_accum = hf_bert(Exp(\n",
    "    'HF 16-bit, fused-adam\\ngrad_accum=4', 'bert-large-uncased', 11, bf16=True, \n",
    "    optim='adamw_apex_fused', grad_accum=4, steps=5))\n",
    "compare([bert_single, bert_half, bert_half_fused, bert_half_fused_accum])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8e7110f1",
   "metadata": {},
   "source": [
    "If your model is too big so not sufficient memory is left for activations, we can throw away them and then re-compute when needed. It can be also used to increase the micro batch size. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "93b1415f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       "  \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<svg xmlns:xlink=\"http://www.w3.org/1999/xlink\" width=\"655.33125pt\" height=\"180.65625pt\" viewBox=\"0 0 655.33125 180.65625\" xmlns=\"http://www.w3.org/2000/svg\" version=\"1.1\">\n",
       " <metadata>\n",
       "  <rdf:RDF xmlns:dc=\"http://purl.org/dc/elements/1.1/\" xmlns:cc=\"http://creativecommons.org/ns#\" xmlns:rdf=\"http://www.w3.org/1999/02/22-rdf-syntax-ns#\">\n",
       "   <cc:Work>\n",
       "    <dc:type rdf:resource=\"http://purl.org/dc/dcmitype/StillImage\"/>\n",
       "    <dc:date>2022-09-02T19:42:18.147195</dc:date>\n",
       "    <dc:format>image/svg+xml</dc:format>\n",
       "    <dc:creator>\n",
       "     <cc:Agent>\n",
       "      <dc:title>Matplotlib v3.5.2, https://matplotlib.org/</dc:title>\n",
       "     </cc:Agent>\n",
       "    </dc:creator>\n",
       "   </cc:Work>\n",
       "  </rdf:RDF>\n",
       " </metadata>\n",
       " <defs>\n",
       "  <style type=\"text/css\">*{stroke-linejoin: round; stroke-linecap: butt}</style>\n",
       " </defs>\n",
       " <g id=\"figure_1\">\n",
       "  <g id=\"patch_1\">\n",
       "   <path d=\"M 0 180.65625 \n",
       "L 655.33125 180.65625 \n",
       "L 655.33125 0 \n",
       "L 0 0 \n",
       "L 0 180.65625 \n",
       "z\n",
       "\" style=\"fill: none\"/>\n",
       "  </g>\n",
       "  <g id=\"axes_1\">\n",
       "   <g id=\"patch_2\">\n",
       "    <path d=\"M 145.93125 143.1 \n",
       "L 293.637132 143.1 \n",
       "L 293.637132 7.2 \n",
       "L 145.93125 7.2 \n",
       "z\n",
       "\" style=\"fill: #ffffff\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_3\">\n",
       "    <path d=\"M 145.93125 13.377273 \n",
       "L 225.952865 13.377273 \n",
       "L 225.952865 29.491897 \n",
       "L 145.93125 29.491897 \n",
       "z\n",
       "\" clip-path=\"url(#p671d8933a7)\" style=\"fill: #e41a1c\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_4\">\n",
       "    <path d=\"M 145.93125 40.23498 \n",
       "L 268.383764 40.23498 \n",
       "L 268.383764 56.349605 \n",
       "L 145.93125 56.349605 \n",
       "z\n",
       "\" clip-path=\"url(#p671d8933a7)\" style=\"fill: #377eb8\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_5\">\n",
       "    <path d=\"M 145.93125 67.092688 \n",
       "L 275.592872 67.092688 \n",
       "L 275.592872 83.207312 \n",
       "L 145.93125 83.207312 \n",
       "z\n",
       "\" clip-path=\"url(#p671d8933a7)\" style=\"fill: #4daf4a\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_6\">\n",
       "    <path d=\"M 145.93125 93.950395 \n",
       "L 286.603519 93.950395 \n",
       "L 286.603519 110.06502 \n",
       "L 145.93125 110.06502 \n",
       "z\n",
       "\" clip-path=\"url(#p671d8933a7)\" style=\"fill: #984ea3\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_7\">\n",
       "    <path d=\"M 145.93125 120.808103 \n",
       "L 274.07328 120.808103 \n",
       "L 274.07328 136.922727 \n",
       "L 145.93125 136.922727 \n",
       "z\n",
       "\" clip-path=\"url(#p671d8933a7)\" style=\"fill: #ff7f00\"/>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_1\">\n",
       "    <g id=\"xtick_1\">\n",
       "     <g id=\"line2d_1\">\n",
       "      <defs>\n",
       "       <path id=\"m97260a81bd\" d=\"M 0 0 \n",
       "L 0 3.5 \n",
       "\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </defs>\n",
       "      <g>\n",
       "       <use xlink:href=\"#m97260a81bd\" x=\"145.93125\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_1\">\n",
       "      <!-- 0 -->\n",
       "      <g transform=\"translate(142.75 157.698438)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-30\" d=\"M 2034 4250 \n",
       "Q 1547 4250 1301 3770 \n",
       "Q 1056 3291 1056 2328 \n",
       "Q 1056 1369 1301 889 \n",
       "Q 1547 409 2034 409 \n",
       "Q 2525 409 2770 889 \n",
       "Q 3016 1369 3016 2328 \n",
       "Q 3016 3291 2770 3770 \n",
       "Q 2525 4250 2034 4250 \n",
       "z\n",
       "M 2034 4750 \n",
       "Q 2819 4750 3233 4129 \n",
       "Q 3647 3509 3647 2328 \n",
       "Q 3647 1150 3233 529 \n",
       "Q 2819 -91 2034 -91 \n",
       "Q 1250 -91 836 529 \n",
       "Q 422 1150 422 2328 \n",
       "Q 422 3509 836 4129 \n",
       "Q 1250 4750 2034 4750 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_2\">\n",
       "     <g id=\"line2d_2\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m97260a81bd\" x=\"197.095966\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_2\">\n",
       "      <!-- 10 -->\n",
       "      <g transform=\"translate(190.733466 157.698438)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-31\" d=\"M 794 531 \n",
       "L 1825 531 \n",
       "L 1825 4091 \n",
       "L 703 3866 \n",
       "L 703 4441 \n",
       "L 1819 4666 \n",
       "L 2450 4666 \n",
       "L 2450 531 \n",
       "L 3481 531 \n",
       "L 3481 0 \n",
       "L 794 0 \n",
       "L 794 531 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-31\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_3\">\n",
       "     <g id=\"line2d_3\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m97260a81bd\" x=\"248.260681\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_3\">\n",
       "      <!-- 20 -->\n",
       "      <g transform=\"translate(241.898181 157.698438)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-32\" d=\"M 1228 531 \n",
       "L 3431 531 \n",
       "L 3431 0 \n",
       "L 469 0 \n",
       "L 469 531 \n",
       "Q 828 903 1448 1529 \n",
       "Q 2069 2156 2228 2338 \n",
       "Q 2531 2678 2651 2914 \n",
       "Q 2772 3150 2772 3378 \n",
       "Q 2772 3750 2511 3984 \n",
       "Q 2250 4219 1831 4219 \n",
       "Q 1534 4219 1204 4116 \n",
       "Q 875 4013 500 3803 \n",
       "L 500 4441 \n",
       "Q 881 4594 1212 4672 \n",
       "Q 1544 4750 1819 4750 \n",
       "Q 2544 4750 2975 4387 \n",
       "Q 3406 4025 3406 3419 \n",
       "Q 3406 3131 3298 2873 \n",
       "Q 3191 2616 2906 2266 \n",
       "Q 2828 2175 2409 1742 \n",
       "Q 1991 1309 1228 531 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-32\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"text_4\">\n",
       "     <!-- Samples / sec -->\n",
       "     <g transform=\"translate(185.138097 171.376563)scale(0.1 -0.1)\">\n",
       "      <defs>\n",
       "       <path id=\"DejaVuSans-53\" d=\"M 3425 4513 \n",
       "L 3425 3897 \n",
       "Q 3066 4069 2747 4153 \n",
       "Q 2428 4238 2131 4238 \n",
       "Q 1616 4238 1336 4038 \n",
       "Q 1056 3838 1056 3469 \n",
       "Q 1056 3159 1242 3001 \n",
       "Q 1428 2844 1947 2747 \n",
       "L 2328 2669 \n",
       "Q 3034 2534 3370 2195 \n",
       "Q 3706 1856 3706 1288 \n",
       "Q 3706 609 3251 259 \n",
       "Q 2797 -91 1919 -91 \n",
       "Q 1588 -91 1214 -16 \n",
       "Q 841 59 441 206 \n",
       "L 441 856 \n",
       "Q 825 641 1194 531 \n",
       "Q 1563 422 1919 422 \n",
       "Q 2459 422 2753 634 \n",
       "Q 3047 847 3047 1241 \n",
       "Q 3047 1584 2836 1778 \n",
       "Q 2625 1972 2144 2069 \n",
       "L 1759 2144 \n",
       "Q 1053 2284 737 2584 \n",
       "Q 422 2884 422 3419 \n",
       "Q 422 4038 858 4394 \n",
       "Q 1294 4750 2059 4750 \n",
       "Q 2388 4750 2728 4690 \n",
       "Q 3069 4631 3425 4513 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-61\" d=\"M 2194 1759 \n",
       "Q 1497 1759 1228 1600 \n",
       "Q 959 1441 959 1056 \n",
       "Q 959 750 1161 570 \n",
       "Q 1363 391 1709 391 \n",
       "Q 2188 391 2477 730 \n",
       "Q 2766 1069 2766 1631 \n",
       "L 2766 1759 \n",
       "L 2194 1759 \n",
       "z\n",
       "M 3341 1997 \n",
       "L 3341 0 \n",
       "L 2766 0 \n",
       "L 2766 531 \n",
       "Q 2569 213 2275 61 \n",
       "Q 1981 -91 1556 -91 \n",
       "Q 1019 -91 701 211 \n",
       "Q 384 513 384 1019 \n",
       "Q 384 1609 779 1909 \n",
       "Q 1175 2209 1959 2209 \n",
       "L 2766 2209 \n",
       "L 2766 2266 \n",
       "Q 2766 2663 2505 2880 \n",
       "Q 2244 3097 1772 3097 \n",
       "Q 1472 3097 1187 3025 \n",
       "Q 903 2953 641 2809 \n",
       "L 641 3341 \n",
       "Q 956 3463 1253 3523 \n",
       "Q 1550 3584 1831 3584 \n",
       "Q 2591 3584 2966 3190 \n",
       "Q 3341 2797 3341 1997 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-6d\" d=\"M 3328 2828 \n",
       "Q 3544 3216 3844 3400 \n",
       "Q 4144 3584 4550 3584 \n",
       "Q 5097 3584 5394 3201 \n",
       "Q 5691 2819 5691 2113 \n",
       "L 5691 0 \n",
       "L 5113 0 \n",
       "L 5113 2094 \n",
       "Q 5113 2597 4934 2840 \n",
       "Q 4756 3084 4391 3084 \n",
       "Q 3944 3084 3684 2787 \n",
       "Q 3425 2491 3425 1978 \n",
       "L 3425 0 \n",
       "L 2847 0 \n",
       "L 2847 2094 \n",
       "Q 2847 2600 2669 2842 \n",
       "Q 2491 3084 2119 3084 \n",
       "Q 1678 3084 1418 2786 \n",
       "Q 1159 2488 1159 1978 \n",
       "L 1159 0 \n",
       "L 581 0 \n",
       "L 581 3500 \n",
       "L 1159 3500 \n",
       "L 1159 2956 \n",
       "Q 1356 3278 1631 3431 \n",
       "Q 1906 3584 2284 3584 \n",
       "Q 2666 3584 2933 3390 \n",
       "Q 3200 3197 3328 2828 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-70\" d=\"M 1159 525 \n",
       "L 1159 -1331 \n",
       "L 581 -1331 \n",
       "L 581 3500 \n",
       "L 1159 3500 \n",
       "L 1159 2969 \n",
       "Q 1341 3281 1617 3432 \n",
       "Q 1894 3584 2278 3584 \n",
       "Q 2916 3584 3314 3078 \n",
       "Q 3713 2572 3713 1747 \n",
       "Q 3713 922 3314 415 \n",
       "Q 2916 -91 2278 -91 \n",
       "Q 1894 -91 1617 61 \n",
       "Q 1341 213 1159 525 \n",
       "z\n",
       "M 3116 1747 \n",
       "Q 3116 2381 2855 2742 \n",
       "Q 2594 3103 2138 3103 \n",
       "Q 1681 3103 1420 2742 \n",
       "Q 1159 2381 1159 1747 \n",
       "Q 1159 1113 1420 752 \n",
       "Q 1681 391 2138 391 \n",
       "Q 2594 391 2855 752 \n",
       "Q 3116 1113 3116 1747 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-6c\" d=\"M 603 4863 \n",
       "L 1178 4863 \n",
       "L 1178 0 \n",
       "L 603 0 \n",
       "L 603 4863 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-65\" d=\"M 3597 1894 \n",
       "L 3597 1613 \n",
       "L 953 1613 \n",
       "Q 991 1019 1311 708 \n",
       "Q 1631 397 2203 397 \n",
       "Q 2534 397 2845 478 \n",
       "Q 3156 559 3463 722 \n",
       "L 3463 178 \n",
       "Q 3153 47 2828 -22 \n",
       "Q 2503 -91 2169 -91 \n",
       "Q 1331 -91 842 396 \n",
       "Q 353 884 353 1716 \n",
       "Q 353 2575 817 3079 \n",
       "Q 1281 3584 2069 3584 \n",
       "Q 2775 3584 3186 3129 \n",
       "Q 3597 2675 3597 1894 \n",
       "z\n",
       "M 3022 2063 \n",
       "Q 3016 2534 2758 2815 \n",
       "Q 2500 3097 2075 3097 \n",
       "Q 1594 3097 1305 2825 \n",
       "Q 1016 2553 972 2059 \n",
       "L 3022 2063 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-73\" d=\"M 2834 3397 \n",
       "L 2834 2853 \n",
       "Q 2591 2978 2328 3040 \n",
       "Q 2066 3103 1784 3103 \n",
       "Q 1356 3103 1142 2972 \n",
       "Q 928 2841 928 2578 \n",
       "Q 928 2378 1081 2264 \n",
       "Q 1234 2150 1697 2047 \n",
       "L 1894 2003 \n",
       "Q 2506 1872 2764 1633 \n",
       "Q 3022 1394 3022 966 \n",
       "Q 3022 478 2636 193 \n",
       "Q 2250 -91 1575 -91 \n",
       "Q 1294 -91 989 -36 \n",
       "Q 684 19 347 128 \n",
       "L 347 722 \n",
       "Q 666 556 975 473 \n",
       "Q 1284 391 1588 391 \n",
       "Q 1994 391 2212 530 \n",
       "Q 2431 669 2431 922 \n",
       "Q 2431 1156 2273 1281 \n",
       "Q 2116 1406 1581 1522 \n",
       "L 1381 1569 \n",
       "Q 847 1681 609 1914 \n",
       "Q 372 2147 372 2553 \n",
       "Q 372 3047 722 3315 \n",
       "Q 1072 3584 1716 3584 \n",
       "Q 2034 3584 2315 3537 \n",
       "Q 2597 3491 2834 3397 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-20\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-2f\" d=\"M 1625 4666 \n",
       "L 2156 4666 \n",
       "L 531 -594 \n",
       "L 0 -594 \n",
       "L 1625 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-63\" d=\"M 3122 3366 \n",
       "L 3122 2828 \n",
       "Q 2878 2963 2633 3030 \n",
       "Q 2388 3097 2138 3097 \n",
       "Q 1578 3097 1268 2742 \n",
       "Q 959 2388 959 1747 \n",
       "Q 959 1106 1268 751 \n",
       "Q 1578 397 2138 397 \n",
       "Q 2388 397 2633 464 \n",
       "Q 2878 531 3122 666 \n",
       "L 3122 134 \n",
       "Q 2881 22 2623 -34 \n",
       "Q 2366 -91 2075 -91 \n",
       "Q 1284 -91 818 406 \n",
       "Q 353 903 353 1747 \n",
       "Q 353 2603 823 3093 \n",
       "Q 1294 3584 2113 3584 \n",
       "Q 2378 3584 2631 3529 \n",
       "Q 2884 3475 3122 3366 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "      </defs>\n",
       "      <use xlink:href=\"#DejaVuSans-53\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-61\" x=\"63.476562\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-6d\" x=\"124.755859\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-70\" x=\"222.167969\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-6c\" x=\"285.644531\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-65\" x=\"313.427734\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-73\" x=\"374.951172\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-20\" x=\"427.050781\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-2f\" x=\"458.837891\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-20\" x=\"492.529297\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-73\" x=\"524.316406\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-65\" x=\"576.416016\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-63\" x=\"637.939453\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_2\">\n",
       "    <g id=\"ytick_1\">\n",
       "     <g id=\"line2d_4\">\n",
       "      <defs>\n",
       "       <path id=\"m8b8866e6b2\" d=\"M 0 0 \n",
       "L -3.5 0 \n",
       "\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </defs>\n",
       "      <g>\n",
       "       <use xlink:href=\"#m8b8866e6b2\" x=\"145.93125\" y=\"21.434585\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_5\">\n",
       "      <!-- HF 32-bit -->\n",
       "      <g transform=\"translate(93.101562 25.233804)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-48\" d=\"M 628 4666 \n",
       "L 1259 4666 \n",
       "L 1259 2753 \n",
       "L 3553 2753 \n",
       "L 3553 4666 \n",
       "L 4184 4666 \n",
       "L 4184 0 \n",
       "L 3553 0 \n",
       "L 3553 2222 \n",
       "L 1259 2222 \n",
       "L 1259 0 \n",
       "L 628 0 \n",
       "L 628 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-46\" d=\"M 628 4666 \n",
       "L 3309 4666 \n",
       "L 3309 4134 \n",
       "L 1259 4134 \n",
       "L 1259 2759 \n",
       "L 3109 2759 \n",
       "L 3109 2228 \n",
       "L 1259 2228 \n",
       "L 1259 0 \n",
       "L 628 0 \n",
       "L 628 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-33\" d=\"M 2597 2516 \n",
       "Q 3050 2419 3304 2112 \n",
       "Q 3559 1806 3559 1356 \n",
       "Q 3559 666 3084 287 \n",
       "Q 2609 -91 1734 -91 \n",
       "Q 1441 -91 1130 -33 \n",
       "Q 819 25 488 141 \n",
       "L 488 750 \n",
       "Q 750 597 1062 519 \n",
       "Q 1375 441 1716 441 \n",
       "Q 2309 441 2620 675 \n",
       "Q 2931 909 2931 1356 \n",
       "Q 2931 1769 2642 2001 \n",
       "Q 2353 2234 1838 2234 \n",
       "L 1294 2234 \n",
       "L 1294 2753 \n",
       "L 1863 2753 \n",
       "Q 2328 2753 2575 2939 \n",
       "Q 2822 3125 2822 3475 \n",
       "Q 2822 3834 2567 4026 \n",
       "Q 2313 4219 1838 4219 \n",
       "Q 1578 4219 1281 4162 \n",
       "Q 984 4106 628 3988 \n",
       "L 628 4550 \n",
       "Q 988 4650 1302 4700 \n",
       "Q 1616 4750 1894 4750 \n",
       "Q 2613 4750 3031 4423 \n",
       "Q 3450 4097 3450 3541 \n",
       "Q 3450 3153 3228 2886 \n",
       "Q 3006 2619 2597 2516 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-2d\" d=\"M 313 2009 \n",
       "L 1997 2009 \n",
       "L 1997 1497 \n",
       "L 313 1497 \n",
       "L 313 2009 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-62\" d=\"M 3116 1747 \n",
       "Q 3116 2381 2855 2742 \n",
       "Q 2594 3103 2138 3103 \n",
       "Q 1681 3103 1420 2742 \n",
       "Q 1159 2381 1159 1747 \n",
       "Q 1159 1113 1420 752 \n",
       "Q 1681 391 2138 391 \n",
       "Q 2594 391 2855 752 \n",
       "Q 3116 1113 3116 1747 \n",
       "z\n",
       "M 1159 2969 \n",
       "Q 1341 3281 1617 3432 \n",
       "Q 1894 3584 2278 3584 \n",
       "Q 2916 3584 3314 3078 \n",
       "Q 3713 2572 3713 1747 \n",
       "Q 3713 922 3314 415 \n",
       "Q 2916 -91 2278 -91 \n",
       "Q 1894 -91 1617 61 \n",
       "Q 1341 213 1159 525 \n",
       "L 1159 0 \n",
       "L 581 0 \n",
       "L 581 4863 \n",
       "L 1159 4863 \n",
       "L 1159 2969 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-69\" d=\"M 603 3500 \n",
       "L 1178 3500 \n",
       "L 1178 0 \n",
       "L 603 0 \n",
       "L 603 3500 \n",
       "z\n",
       "M 603 4863 \n",
       "L 1178 4863 \n",
       "L 1178 4134 \n",
       "L 603 4134 \n",
       "L 603 4863 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-74\" d=\"M 1172 4494 \n",
       "L 1172 3500 \n",
       "L 2356 3500 \n",
       "L 2356 3053 \n",
       "L 1172 3053 \n",
       "L 1172 1153 \n",
       "Q 1172 725 1289 603 \n",
       "Q 1406 481 1766 481 \n",
       "L 2356 481 \n",
       "L 2356 0 \n",
       "L 1766 0 \n",
       "Q 1100 0 847 248 \n",
       "Q 594 497 594 1153 \n",
       "L 594 3053 \n",
       "L 172 3053 \n",
       "L 172 3500 \n",
       "L 594 3500 \n",
       "L 594 4494 \n",
       "L 1172 4494 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-48\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-46\" x=\"75.195312\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-20\" x=\"132.714844\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-33\" x=\"164.501953\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-32\" x=\"228.125\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-2d\" x=\"291.748047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-62\" x=\"327.832031\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-69\" x=\"391.308594\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-74\" x=\"419.091797\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_2\">\n",
       "     <g id=\"line2d_5\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m8b8866e6b2\" x=\"145.93125\" y=\"48.292292\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_6\">\n",
       "      <!-- HF 16-bit -->\n",
       "      <g transform=\"translate(93.101562 52.091511)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-36\" d=\"M 2113 2584 \n",
       "Q 1688 2584 1439 2293 \n",
       "Q 1191 2003 1191 1497 \n",
       "Q 1191 994 1439 701 \n",
       "Q 1688 409 2113 409 \n",
       "Q 2538 409 2786 701 \n",
       "Q 3034 994 3034 1497 \n",
       "Q 3034 2003 2786 2293 \n",
       "Q 2538 2584 2113 2584 \n",
       "z\n",
       "M 3366 4563 \n",
       "L 3366 3988 \n",
       "Q 3128 4100 2886 4159 \n",
       "Q 2644 4219 2406 4219 \n",
       "Q 1781 4219 1451 3797 \n",
       "Q 1122 3375 1075 2522 \n",
       "Q 1259 2794 1537 2939 \n",
       "Q 1816 3084 2150 3084 \n",
       "Q 2853 3084 3261 2657 \n",
       "Q 3669 2231 3669 1497 \n",
       "Q 3669 778 3244 343 \n",
       "Q 2819 -91 2113 -91 \n",
       "Q 1303 -91 875 529 \n",
       "Q 447 1150 447 2328 \n",
       "Q 447 3434 972 4092 \n",
       "Q 1497 4750 2381 4750 \n",
       "Q 2619 4750 2861 4703 \n",
       "Q 3103 4656 3366 4563 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-48\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-46\" x=\"75.195312\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-20\" x=\"132.714844\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-31\" x=\"164.501953\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-36\" x=\"228.125\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-2d\" x=\"291.748047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-62\" x=\"327.832031\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-69\" x=\"391.308594\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-74\" x=\"419.091797\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_3\">\n",
       "     <g id=\"line2d_6\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m8b8866e6b2\" x=\"145.93125\" y=\"75.15\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_7\">\n",
       "      <!-- HF 16-bit, fused-adam -->\n",
       "      <g transform=\"translate(27.223438 78.949219)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-2c\" d=\"M 750 794 \n",
       "L 1409 794 \n",
       "L 1409 256 \n",
       "L 897 -744 \n",
       "L 494 -744 \n",
       "L 750 256 \n",
       "L 750 794 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-66\" d=\"M 2375 4863 \n",
       "L 2375 4384 \n",
       "L 1825 4384 \n",
       "Q 1516 4384 1395 4259 \n",
       "Q 1275 4134 1275 3809 \n",
       "L 1275 3500 \n",
       "L 2222 3500 \n",
       "L 2222 3053 \n",
       "L 1275 3053 \n",
       "L 1275 0 \n",
       "L 697 0 \n",
       "L 697 3053 \n",
       "L 147 3053 \n",
       "L 147 3500 \n",
       "L 697 3500 \n",
       "L 697 3744 \n",
       "Q 697 4328 969 4595 \n",
       "Q 1241 4863 1831 4863 \n",
       "L 2375 4863 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-75\" d=\"M 544 1381 \n",
       "L 544 3500 \n",
       "L 1119 3500 \n",
       "L 1119 1403 \n",
       "Q 1119 906 1312 657 \n",
       "Q 1506 409 1894 409 \n",
       "Q 2359 409 2629 706 \n",
       "Q 2900 1003 2900 1516 \n",
       "L 2900 3500 \n",
       "L 3475 3500 \n",
       "L 3475 0 \n",
       "L 2900 0 \n",
       "L 2900 538 \n",
       "Q 2691 219 2414 64 \n",
       "Q 2138 -91 1772 -91 \n",
       "Q 1169 -91 856 284 \n",
       "Q 544 659 544 1381 \n",
       "z\n",
       "M 1991 3584 \n",
       "L 1991 3584 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-64\" d=\"M 2906 2969 \n",
       "L 2906 4863 \n",
       "L 3481 4863 \n",
       "L 3481 0 \n",
       "L 2906 0 \n",
       "L 2906 525 \n",
       "Q 2725 213 2448 61 \n",
       "Q 2172 -91 1784 -91 \n",
       "Q 1150 -91 751 415 \n",
       "Q 353 922 353 1747 \n",
       "Q 353 2572 751 3078 \n",
       "Q 1150 3584 1784 3584 \n",
       "Q 2172 3584 2448 3432 \n",
       "Q 2725 3281 2906 2969 \n",
       "z\n",
       "M 947 1747 \n",
       "Q 947 1113 1208 752 \n",
       "Q 1469 391 1925 391 \n",
       "Q 2381 391 2643 752 \n",
       "Q 2906 1113 2906 1747 \n",
       "Q 2906 2381 2643 2742 \n",
       "Q 2381 3103 1925 3103 \n",
       "Q 1469 3103 1208 2742 \n",
       "Q 947 2381 947 1747 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-48\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-46\" x=\"75.195312\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-20\" x=\"132.714844\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-31\" x=\"164.501953\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-36\" x=\"228.125\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-2d\" x=\"291.748047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-62\" x=\"327.832031\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-69\" x=\"391.308594\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-74\" x=\"419.091797\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-2c\" x=\"458.300781\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-20\" x=\"490.087891\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-66\" x=\"521.875\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-75\" x=\"557.080078\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-73\" x=\"620.458984\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-65\" x=\"672.558594\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-64\" x=\"734.082031\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-2d\" x=\"797.558594\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-61\" x=\"833.642578\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-64\" x=\"894.921875\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-61\" x=\"958.398438\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-6d\" x=\"1019.677734\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_4\">\n",
       "     <g id=\"line2d_7\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m8b8866e6b2\" x=\"145.93125\" y=\"102.007708\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_8\">\n",
       "      <!-- HF 16-bit, fused-adam -->\n",
       "      <g transform=\"translate(27.223438 100.967864)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-48\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-46\" x=\"75.195312\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-20\" x=\"132.714844\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-31\" x=\"164.501953\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-36\" x=\"228.125\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-2d\" x=\"291.748047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-62\" x=\"327.832031\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-69\" x=\"391.308594\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-74\" x=\"419.091797\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-2c\" x=\"458.300781\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-20\" x=\"490.087891\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-66\" x=\"521.875\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-75\" x=\"557.080078\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-73\" x=\"620.458984\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-65\" x=\"672.558594\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-64\" x=\"734.082031\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-2d\" x=\"797.558594\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-61\" x=\"833.642578\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-64\" x=\"894.921875\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-61\" x=\"958.398438\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-6d\" x=\"1019.677734\"/>\n",
       "      </g>\n",
       "      <!-- grad_accum=4 -->\n",
       "      <g transform=\"translate(63.05 112.165676)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-67\" d=\"M 2906 1791 \n",
       "Q 2906 2416 2648 2759 \n",
       "Q 2391 3103 1925 3103 \n",
       "Q 1463 3103 1205 2759 \n",
       "Q 947 2416 947 1791 \n",
       "Q 947 1169 1205 825 \n",
       "Q 1463 481 1925 481 \n",
       "Q 2391 481 2648 825 \n",
       "Q 2906 1169 2906 1791 \n",
       "z\n",
       "M 3481 434 \n",
       "Q 3481 -459 3084 -895 \n",
       "Q 2688 -1331 1869 -1331 \n",
       "Q 1566 -1331 1297 -1286 \n",
       "Q 1028 -1241 775 -1147 \n",
       "L 775 -588 \n",
       "Q 1028 -725 1275 -790 \n",
       "Q 1522 -856 1778 -856 \n",
       "Q 2344 -856 2625 -561 \n",
       "Q 2906 -266 2906 331 \n",
       "L 2906 616 \n",
       "Q 2728 306 2450 153 \n",
       "Q 2172 0 1784 0 \n",
       "Q 1141 0 747 490 \n",
       "Q 353 981 353 1791 \n",
       "Q 353 2603 747 3093 \n",
       "Q 1141 3584 1784 3584 \n",
       "Q 2172 3584 2450 3431 \n",
       "Q 2728 3278 2906 2969 \n",
       "L 2906 3500 \n",
       "L 3481 3500 \n",
       "L 3481 434 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-72\" d=\"M 2631 2963 \n",
       "Q 2534 3019 2420 3045 \n",
       "Q 2306 3072 2169 3072 \n",
       "Q 1681 3072 1420 2755 \n",
       "Q 1159 2438 1159 1844 \n",
       "L 1159 0 \n",
       "L 581 0 \n",
       "L 581 3500 \n",
       "L 1159 3500 \n",
       "L 1159 2956 \n",
       "Q 1341 3275 1631 3429 \n",
       "Q 1922 3584 2338 3584 \n",
       "Q 2397 3584 2469 3576 \n",
       "Q 2541 3569 2628 3553 \n",
       "L 2631 2963 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-5f\" d=\"M 3263 -1063 \n",
       "L 3263 -1509 \n",
       "L -63 -1509 \n",
       "L -63 -1063 \n",
       "L 3263 -1063 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-3d\" d=\"M 678 2906 \n",
       "L 4684 2906 \n",
       "L 4684 2381 \n",
       "L 678 2381 \n",
       "L 678 2906 \n",
       "z\n",
       "M 678 1631 \n",
       "L 4684 1631 \n",
       "L 4684 1100 \n",
       "L 678 1100 \n",
       "L 678 1631 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-34\" d=\"M 2419 4116 \n",
       "L 825 1625 \n",
       "L 2419 1625 \n",
       "L 2419 4116 \n",
       "z\n",
       "M 2253 4666 \n",
       "L 3047 4666 \n",
       "L 3047 1625 \n",
       "L 3713 1625 \n",
       "L 3713 1100 \n",
       "L 3047 1100 \n",
       "L 3047 0 \n",
       "L 2419 0 \n",
       "L 2419 1100 \n",
       "L 313 1100 \n",
       "L 313 1709 \n",
       "L 2253 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-67\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-72\" x=\"63.476562\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-61\" x=\"104.589844\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-64\" x=\"165.869141\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-5f\" x=\"229.345703\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-61\" x=\"279.345703\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-63\" x=\"340.625\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-63\" x=\"395.605469\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-75\" x=\"450.585938\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-6d\" x=\"513.964844\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-3d\" x=\"611.376953\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-34\" x=\"695.166016\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_5\">\n",
       "     <g id=\"line2d_8\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m8b8866e6b2\" x=\"145.93125\" y=\"128.865415\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_9\">\n",
       "      <!-- HF 16-bit, fused-adam -->\n",
       "      <g transform=\"translate(27.223438 127.825571)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-48\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-46\" x=\"75.195312\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-20\" x=\"132.714844\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-31\" x=\"164.501953\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-36\" x=\"228.125\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-2d\" x=\"291.748047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-62\" x=\"327.832031\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-69\" x=\"391.308594\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-74\" x=\"419.091797\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-2c\" x=\"458.300781\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-20\" x=\"490.087891\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-66\" x=\"521.875\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-75\" x=\"557.080078\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-73\" x=\"620.458984\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-65\" x=\"672.558594\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-64\" x=\"734.082031\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-2d\" x=\"797.558594\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-61\" x=\"833.642578\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-64\" x=\"894.921875\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-61\" x=\"958.398438\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-6d\" x=\"1019.677734\"/>\n",
       "      </g>\n",
       "      <!-- grad_accum=4, grad_ckpt -->\n",
       "      <g transform=\"translate(7.2 139.023384)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-6b\" d=\"M 581 4863 \n",
       "L 1159 4863 \n",
       "L 1159 1991 \n",
       "L 2875 3500 \n",
       "L 3609 3500 \n",
       "L 1753 1863 \n",
       "L 3688 0 \n",
       "L 2938 0 \n",
       "L 1159 1709 \n",
       "L 1159 0 \n",
       "L 581 0 \n",
       "L 581 4863 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-67\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-72\" x=\"63.476562\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-61\" x=\"104.589844\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-64\" x=\"165.869141\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-5f\" x=\"229.345703\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-61\" x=\"279.345703\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-63\" x=\"340.625\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-63\" x=\"395.605469\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-75\" x=\"450.585938\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-6d\" x=\"513.964844\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-3d\" x=\"611.376953\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-34\" x=\"695.166016\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-2c\" x=\"758.789062\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-20\" x=\"790.576172\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-67\" x=\"822.363281\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-72\" x=\"885.839844\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-61\" x=\"926.953125\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-64\" x=\"988.232422\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-5f\" x=\"1051.708984\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-63\" x=\"1101.708984\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-6b\" x=\"1156.689453\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-70\" x=\"1214.599609\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-74\" x=\"1278.076172\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"patch_8\">\n",
       "    <path d=\"M 145.93125 143.1 \n",
       "L 145.93125 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_9\">\n",
       "    <path d=\"M 293.637132 143.1 \n",
       "L 293.637132 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_10\">\n",
       "    <path d=\"M 145.93125 143.1 \n",
       "L 293.637132 143.1 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_11\">\n",
       "    <path d=\"M 145.93125 7.2 \n",
       "L 293.637132 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "  </g>\n",
       "  <g id=\"axes_2\">\n",
       "   <g id=\"patch_12\">\n",
       "    <path d=\"M 323.178309 143.1 \n",
       "L 470.884191 143.1 \n",
       "L 470.884191 7.2 \n",
       "L 323.178309 7.2 \n",
       "z\n",
       "\" style=\"fill: #ffffff\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_13\">\n",
       "    <path d=\"M 323.178309 13.377273 \n",
       "L 389.063145 13.377273 \n",
       "L 389.063145 29.491897 \n",
       "L 323.178309 29.491897 \n",
       "z\n",
       "\" clip-path=\"url(#p3782fd4330)\" style=\"fill: #e41a1c\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_14\">\n",
       "    <path d=\"M 323.178309 40.23498 \n",
       "L 423.998116 40.23498 \n",
       "L 423.998116 56.349605 \n",
       "L 323.178309 56.349605 \n",
       "z\n",
       "\" clip-path=\"url(#p3782fd4330)\" style=\"fill: #377eb8\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_15\">\n",
       "    <path d=\"M 323.178309 67.092688 \n",
       "L 429.933648 67.092688 \n",
       "L 429.933648 83.207312 \n",
       "L 323.178309 83.207312 \n",
       "z\n",
       "\" clip-path=\"url(#p3782fd4330)\" style=\"fill: #4daf4a\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_16\">\n",
       "    <path d=\"M 323.178309 93.950395 \n",
       "L 438.999132 93.950395 \n",
       "L 438.999132 110.06502 \n",
       "L 323.178309 110.06502 \n",
       "z\n",
       "\" clip-path=\"url(#p3782fd4330)\" style=\"fill: #984ea3\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_17\">\n",
       "    <path d=\"M 323.178309 120.808103 \n",
       "L 463.850578 120.808103 \n",
       "L 463.850578 136.922727 \n",
       "L 323.178309 136.922727 \n",
       "z\n",
       "\" clip-path=\"url(#p3782fd4330)\" style=\"fill: #ff7f00\"/>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_3\">\n",
       "    <g id=\"xtick_4\">\n",
       "     <g id=\"line2d_9\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m97260a81bd\" x=\"323.178309\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_10\">\n",
       "      <!-- 0 -->\n",
       "      <g transform=\"translate(319.997059 157.698438)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_5\">\n",
       "     <g id=\"line2d_10\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m97260a81bd\" x=\"361.438497\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_11\">\n",
       "      <!-- 10 -->\n",
       "      <g transform=\"translate(355.075997 157.698438)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-31\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_6\">\n",
       "     <g id=\"line2d_11\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m97260a81bd\" x=\"399.698685\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_12\">\n",
       "      <!-- 20 -->\n",
       "      <g transform=\"translate(393.336185 157.698438)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-32\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_7\">\n",
       "     <g id=\"line2d_12\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m97260a81bd\" x=\"437.958873\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_13\">\n",
       "      <!-- 30 -->\n",
       "      <g transform=\"translate(431.596373 157.698438)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-33\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"text_14\">\n",
       "     <!-- per GPU TFLOPS -->\n",
       "     <g transform=\"translate(356.339063 171.376563)scale(0.1 -0.1)\">\n",
       "      <defs>\n",
       "       <path id=\"DejaVuSans-47\" d=\"M 3809 666 \n",
       "L 3809 1919 \n",
       "L 2778 1919 \n",
       "L 2778 2438 \n",
       "L 4434 2438 \n",
       "L 4434 434 \n",
       "Q 4069 175 3628 42 \n",
       "Q 3188 -91 2688 -91 \n",
       "Q 1594 -91 976 548 \n",
       "Q 359 1188 359 2328 \n",
       "Q 359 3472 976 4111 \n",
       "Q 1594 4750 2688 4750 \n",
       "Q 3144 4750 3555 4637 \n",
       "Q 3966 4525 4313 4306 \n",
       "L 4313 3634 \n",
       "Q 3963 3931 3569 4081 \n",
       "Q 3175 4231 2741 4231 \n",
       "Q 1884 4231 1454 3753 \n",
       "Q 1025 3275 1025 2328 \n",
       "Q 1025 1384 1454 906 \n",
       "Q 1884 428 2741 428 \n",
       "Q 3075 428 3337 486 \n",
       "Q 3600 544 3809 666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-50\" d=\"M 1259 4147 \n",
       "L 1259 2394 \n",
       "L 2053 2394 \n",
       "Q 2494 2394 2734 2622 \n",
       "Q 2975 2850 2975 3272 \n",
       "Q 2975 3691 2734 3919 \n",
       "Q 2494 4147 2053 4147 \n",
       "L 1259 4147 \n",
       "z\n",
       "M 628 4666 \n",
       "L 2053 4666 \n",
       "Q 2838 4666 3239 4311 \n",
       "Q 3641 3956 3641 3272 \n",
       "Q 3641 2581 3239 2228 \n",
       "Q 2838 1875 2053 1875 \n",
       "L 1259 1875 \n",
       "L 1259 0 \n",
       "L 628 0 \n",
       "L 628 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-55\" d=\"M 556 4666 \n",
       "L 1191 4666 \n",
       "L 1191 1831 \n",
       "Q 1191 1081 1462 751 \n",
       "Q 1734 422 2344 422 \n",
       "Q 2950 422 3222 751 \n",
       "Q 3494 1081 3494 1831 \n",
       "L 3494 4666 \n",
       "L 4128 4666 \n",
       "L 4128 1753 \n",
       "Q 4128 841 3676 375 \n",
       "Q 3225 -91 2344 -91 \n",
       "Q 1459 -91 1007 375 \n",
       "Q 556 841 556 1753 \n",
       "L 556 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-54\" d=\"M -19 4666 \n",
       "L 3928 4666 \n",
       "L 3928 4134 \n",
       "L 2272 4134 \n",
       "L 2272 0 \n",
       "L 1638 0 \n",
       "L 1638 4134 \n",
       "L -19 4134 \n",
       "L -19 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-4c\" d=\"M 628 4666 \n",
       "L 1259 4666 \n",
       "L 1259 531 \n",
       "L 3531 531 \n",
       "L 3531 0 \n",
       "L 628 0 \n",
       "L 628 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-4f\" d=\"M 2522 4238 \n",
       "Q 1834 4238 1429 3725 \n",
       "Q 1025 3213 1025 2328 \n",
       "Q 1025 1447 1429 934 \n",
       "Q 1834 422 2522 422 \n",
       "Q 3209 422 3611 934 \n",
       "Q 4013 1447 4013 2328 \n",
       "Q 4013 3213 3611 3725 \n",
       "Q 3209 4238 2522 4238 \n",
       "z\n",
       "M 2522 4750 \n",
       "Q 3503 4750 4090 4092 \n",
       "Q 4678 3434 4678 2328 \n",
       "Q 4678 1225 4090 567 \n",
       "Q 3503 -91 2522 -91 \n",
       "Q 1538 -91 948 565 \n",
       "Q 359 1222 359 2328 \n",
       "Q 359 3434 948 4092 \n",
       "Q 1538 4750 2522 4750 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "      </defs>\n",
       "      <use xlink:href=\"#DejaVuSans-70\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-65\" x=\"63.476562\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-72\" x=\"125\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-20\" x=\"166.113281\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-47\" x=\"197.900391\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-50\" x=\"275.390625\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-55\" x=\"335.693359\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-20\" x=\"408.886719\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-54\" x=\"440.673828\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-46\" x=\"501.757812\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-4c\" x=\"559.277344\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-4f\" x=\"611.365234\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-50\" x=\"690.076172\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-53\" x=\"750.378906\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_4\">\n",
       "    <g id=\"ytick_6\">\n",
       "     <g id=\"line2d_13\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m8b8866e6b2\" x=\"323.178309\" y=\"21.434585\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_7\">\n",
       "     <g id=\"line2d_14\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m8b8866e6b2\" x=\"323.178309\" y=\"48.292292\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_8\">\n",
       "     <g id=\"line2d_15\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m8b8866e6b2\" x=\"323.178309\" y=\"75.15\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_9\">\n",
       "     <g id=\"line2d_16\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m8b8866e6b2\" x=\"323.178309\" y=\"102.007708\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_10\">\n",
       "     <g id=\"line2d_17\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m8b8866e6b2\" x=\"323.178309\" y=\"128.865415\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"patch_18\">\n",
       "    <path d=\"M 323.178309 143.1 \n",
       "L 323.178309 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_19\">\n",
       "    <path d=\"M 470.884191 143.1 \n",
       "L 470.884191 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_20\">\n",
       "    <path d=\"M 323.178309 143.1 \n",
       "L 470.884191 143.1 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_21\">\n",
       "    <path d=\"M 323.178309 7.2 \n",
       "L 470.884191 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "  </g>\n",
       "  <g id=\"axes_3\">\n",
       "   <g id=\"patch_22\">\n",
       "    <path d=\"M 500.425368 143.1 \n",
       "L 648.13125 143.1 \n",
       "L 648.13125 7.2 \n",
       "L 500.425368 7.2 \n",
       "z\n",
       "\" style=\"fill: #ffffff\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_23\">\n",
       "    <path d=\"M 500.425368 13.377273 \n",
       "L 634.723968 13.377273 \n",
       "L 634.723968 29.491897 \n",
       "L 500.425368 29.491897 \n",
       "z\n",
       "\" clip-path=\"url(#p385a8d73f7)\" style=\"fill: #e41a1c\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_24\">\n",
       "    <path d=\"M 500.425368 40.23498 \n",
       "L 641.097637 40.23498 \n",
       "L 641.097637 56.349605 \n",
       "L 500.425368 56.349605 \n",
       "z\n",
       "\" clip-path=\"url(#p385a8d73f7)\" style=\"fill: #377eb8\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_25\">\n",
       "    <path d=\"M 500.425368 67.092688 \n",
       "L 641.064285 67.092688 \n",
       "L 641.064285 83.207312 \n",
       "L 500.425368 83.207312 \n",
       "z\n",
       "\" clip-path=\"url(#p385a8d73f7)\" style=\"fill: #4daf4a\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_26\">\n",
       "    <path d=\"M 500.425368 93.950395 \n",
       "L 641.081322 93.950395 \n",
       "L 641.081322 110.06502 \n",
       "L 500.425368 110.06502 \n",
       "z\n",
       "\" clip-path=\"url(#p385a8d73f7)\" style=\"fill: #984ea3\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_27\">\n",
       "    <path d=\"M 500.425368 120.808103 \n",
       "L 629.502954 120.808103 \n",
       "L 629.502954 136.922727 \n",
       "L 500.425368 136.922727 \n",
       "z\n",
       "\" clip-path=\"url(#p385a8d73f7)\" style=\"fill: #ff7f00\"/>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_5\">\n",
       "    <g id=\"xtick_8\">\n",
       "     <g id=\"line2d_18\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m97260a81bd\" x=\"500.425368\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_15\">\n",
       "      <!-- 0 -->\n",
       "      <g transform=\"translate(497.244118 157.698438)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_9\">\n",
       "     <g id=\"line2d_19\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m97260a81bd\" x=\"562.092962\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_16\">\n",
       "      <!-- 10 -->\n",
       "      <g transform=\"translate(555.730462 157.698438)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-31\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_10\">\n",
       "     <g id=\"line2d_20\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m97260a81bd\" x=\"623.760556\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_17\">\n",
       "      <!-- 20 -->\n",
       "      <g transform=\"translate(617.398056 157.698438)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-32\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"text_18\">\n",
       "     <!-- per GPU memory (GB) -->\n",
       "     <g transform=\"translate(518.558778 171.376563)scale(0.1 -0.1)\">\n",
       "      <defs>\n",
       "       <path id=\"DejaVuSans-6f\" d=\"M 1959 3097 \n",
       "Q 1497 3097 1228 2736 \n",
       "Q 959 2375 959 1747 \n",
       "Q 959 1119 1226 758 \n",
       "Q 1494 397 1959 397 \n",
       "Q 2419 397 2687 759 \n",
       "Q 2956 1122 2956 1747 \n",
       "Q 2956 2369 2687 2733 \n",
       "Q 2419 3097 1959 3097 \n",
       "z\n",
       "M 1959 3584 \n",
       "Q 2709 3584 3137 3096 \n",
       "Q 3566 2609 3566 1747 \n",
       "Q 3566 888 3137 398 \n",
       "Q 2709 -91 1959 -91 \n",
       "Q 1206 -91 779 398 \n",
       "Q 353 888 353 1747 \n",
       "Q 353 2609 779 3096 \n",
       "Q 1206 3584 1959 3584 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-79\" d=\"M 2059 -325 \n",
       "Q 1816 -950 1584 -1140 \n",
       "Q 1353 -1331 966 -1331 \n",
       "L 506 -1331 \n",
       "L 506 -850 \n",
       "L 844 -850 \n",
       "Q 1081 -850 1212 -737 \n",
       "Q 1344 -625 1503 -206 \n",
       "L 1606 56 \n",
       "L 191 3500 \n",
       "L 800 3500 \n",
       "L 1894 763 \n",
       "L 2988 3500 \n",
       "L 3597 3500 \n",
       "L 2059 -325 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-28\" d=\"M 1984 4856 \n",
       "Q 1566 4138 1362 3434 \n",
       "Q 1159 2731 1159 2009 \n",
       "Q 1159 1288 1364 580 \n",
       "Q 1569 -128 1984 -844 \n",
       "L 1484 -844 \n",
       "Q 1016 -109 783 600 \n",
       "Q 550 1309 550 2009 \n",
       "Q 550 2706 781 3412 \n",
       "Q 1013 4119 1484 4856 \n",
       "L 1984 4856 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-42\" d=\"M 1259 2228 \n",
       "L 1259 519 \n",
       "L 2272 519 \n",
       "Q 2781 519 3026 730 \n",
       "Q 3272 941 3272 1375 \n",
       "Q 3272 1813 3026 2020 \n",
       "Q 2781 2228 2272 2228 \n",
       "L 1259 2228 \n",
       "z\n",
       "M 1259 4147 \n",
       "L 1259 2741 \n",
       "L 2194 2741 \n",
       "Q 2656 2741 2882 2914 \n",
       "Q 3109 3088 3109 3444 \n",
       "Q 3109 3797 2882 3972 \n",
       "Q 2656 4147 2194 4147 \n",
       "L 1259 4147 \n",
       "z\n",
       "M 628 4666 \n",
       "L 2241 4666 \n",
       "Q 2963 4666 3353 4366 \n",
       "Q 3744 4066 3744 3513 \n",
       "Q 3744 3084 3544 2831 \n",
       "Q 3344 2578 2956 2516 \n",
       "Q 3422 2416 3680 2098 \n",
       "Q 3938 1781 3938 1306 \n",
       "Q 3938 681 3513 340 \n",
       "Q 3088 0 2303 0 \n",
       "L 628 0 \n",
       "L 628 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-29\" d=\"M 513 4856 \n",
       "L 1013 4856 \n",
       "Q 1481 4119 1714 3412 \n",
       "Q 1947 2706 1947 2009 \n",
       "Q 1947 1309 1714 600 \n",
       "Q 1481 -109 1013 -844 \n",
       "L 513 -844 \n",
       "Q 928 -128 1133 580 \n",
       "Q 1338 1288 1338 2009 \n",
       "Q 1338 2731 1133 3434 \n",
       "Q 928 4138 513 4856 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "      </defs>\n",
       "      <use xlink:href=\"#DejaVuSans-70\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-65\" x=\"63.476562\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-72\" x=\"125\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-20\" x=\"166.113281\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-47\" x=\"197.900391\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-50\" x=\"275.390625\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-55\" x=\"335.693359\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-20\" x=\"408.886719\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-6d\" x=\"440.673828\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-65\" x=\"538.085938\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-6d\" x=\"599.609375\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-6f\" x=\"697.021484\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-72\" x=\"758.203125\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-79\" x=\"799.316406\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-20\" x=\"858.496094\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-28\" x=\"890.283203\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-47\" x=\"929.296875\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-42\" x=\"1006.787109\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-29\" x=\"1075.390625\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_6\">\n",
       "    <g id=\"ytick_11\">\n",
       "     <g id=\"line2d_21\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m8b8866e6b2\" x=\"500.425368\" y=\"21.434585\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_12\">\n",
       "     <g id=\"line2d_22\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m8b8866e6b2\" x=\"500.425368\" y=\"48.292292\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_13\">\n",
       "     <g id=\"line2d_23\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m8b8866e6b2\" x=\"500.425368\" y=\"75.15\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_14\">\n",
       "     <g id=\"line2d_24\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m8b8866e6b2\" x=\"500.425368\" y=\"102.007708\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_15\">\n",
       "     <g id=\"line2d_25\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m8b8866e6b2\" x=\"500.425368\" y=\"128.865415\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"patch_28\">\n",
       "    <path d=\"M 500.425368 143.1 \n",
       "L 500.425368 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_29\">\n",
       "    <path d=\"M 648.13125 143.1 \n",
       "L 648.13125 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_30\">\n",
       "    <path d=\"M 500.425368 143.1 \n",
       "L 648.13125 143.1 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_31\">\n",
       "    <path d=\"M 500.425368 7.2 \n",
       "L 648.13125 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "  </g>\n",
       " </g>\n",
       " <defs>\n",
       "  <clipPath id=\"p671d8933a7\">\n",
       "   <rect x=\"145.93125\" y=\"7.2\" width=\"147.705882\" height=\"135.9\"/>\n",
       "  </clipPath>\n",
       "  <clipPath id=\"p3782fd4330\">\n",
       "   <rect x=\"323.178309\" y=\"7.2\" width=\"147.705882\" height=\"135.9\"/>\n",
       "  </clipPath>\n",
       "  <clipPath id=\"p385a8d73f7\">\n",
       "   <rect x=\"500.425368\" y=\"7.2\" width=\"147.705882\" height=\"135.9\"/>\n",
       "  </clipPath>\n",
       " </defs>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<Figure size 648x180 with 3 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "bert_half_fused_accum_ckpt = hf_bert(Exp(\n",
    "    'HF 16-bit, fused-adam\\ngrad_accum=4, grad_ckpt', 'bert-large-uncased', 66, bf16=True, \n",
    "    optim='adamw_apex_fused', grad_accum=4, grad_ckpt=True, steps=5))\n",
    "compare([bert_single, bert_half, bert_half_fused, bert_half_fused_accum, bert_half_fused_accum_ckpt])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9deff3e8",
   "metadata": {},
   "source": [
    "Though it furthers improve TFLOPS, but decreases the number of samples per second because of the extra forward. So use it only when the model is very big you cannot use an effective batch size. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7fd47f7a",
   "metadata": {},
   "source": [
    "## BERT on a Single GPU with Megatron-LM\n",
    "\n",
    "Though `HuggingFace` is the most popular package for transformers, it's not the fastest one. Here let's use Megatron-LM from Nvidia. First download vocab and a sample dataset. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "7d079ed6",
   "metadata": {},
   "outputs": [],
   "source": [
    "!wget -nc https://s3.amazonaws.com/models.huggingface.co/bert/bert-large-uncased-vocab.txt\n",
    "!wget -nc https://github.com/mli/transformers-benchmarks/raw/main/data/bert-sample_text_sentence.bin\n",
    "!wget -nc https://github.com/mli/transformers-benchmarks/raw/main/data/bert-sample_text_sentence.idx\n",
    "    \n",
    "clear_output()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "186bcdbe",
   "metadata": {},
   "source": [
    "Define the function to run BERT and parse its log."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "ee13c1ff",
   "metadata": {},
   "outputs": [],
   "source": [
    "def megatron_bert(exp):\n",
    "    cmd = f'''{exp.launcher} Megatron-LM/pretrain_bert.py \\\n",
    "--num-layers {exp.num_layers} --hidden-size {exp.hidden_size} \\\n",
    "--num-attention-heads {exp.num_heads} \\\n",
    "--tensor-model-parallel-size {exp.tensor_para} \\\n",
    "--micro-batch-size {exp.batch_size} \\\n",
    "--seq-length {exp.seq_len} --max-position-embeddings {exp.seq_len} \\\n",
    "--train-iters {exp.steps} \\\n",
    "--data-path bert-sample_text_sentence \\\n",
    "--vocab-file bert-large-uncased-vocab.txt \\\n",
    "--data-impl mmap --lr 0.00015 --log-interval 5'''\n",
    "    if exp.bf16: cmd += ' --bf16'\n",
    "    if exp.fp16: cmd += ' --fp16'\n",
    "    cmd += ' > log.txt 2>&1'\n",
    "    os.system(cmd)\n",
    "    return megatron_log(exp, 'log.txt') \n",
    "    \n",
    "def megatron_log(exp, log_filename):\n",
    "    with open(log_filename) as f:\n",
    "        text = f.read()\n",
    "    # Find the last number after the key, returns 0 if not exists\n",
    "    query = lambda key: float(next(iter(        \n",
    "        reversed(re.findall(key+': +([\\d\\.]+)', text))), 0))\n",
    "    if 'CUDA out of memory' in text:\n",
    "        print('Out of GPU memory, try a smaller batch size')\n",
    "        return\n",
    "    iter_time = query('elapsed time per iteration \\(ms\\)') \n",
    "    if iter_time == 0:\n",
    "        print(f'Failed. Check \"{log_filename}\" to find error')\n",
    "        return\n",
    "    exp.samples_per_sec = query('global batch size') / iter_time * 1e3\n",
    "    exp.gpu_mem = query('max allocated')/1e3\n",
    "    print('Time breakdown\\t\\t: forward+backward %.2f, communication %.2f, optimizer %.2f' %(\n",
    "        (query('forward-compute')+query('backward-compute')) / iter_time, \n",
    "        query('backward-params-all-reduce') / iter_time, query('optimizer') / iter_time))        \n",
    "    return exp"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f106218d",
   "metadata": {},
   "source": [
    "Run BERT large again."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "eb6cd6bc",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Time breakdown\t\t: forward+backward 0.93, communication 0.01, optimizer 0.06\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       "  \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<svg xmlns:xlink=\"http://www.w3.org/1999/xlink\" width=\"635.307812pt\" height=\"99.11625pt\" viewBox=\"0 0 635.307812 99.11625\" xmlns=\"http://www.w3.org/2000/svg\" version=\"1.1\">\n",
       " <metadata>\n",
       "  <rdf:RDF xmlns:dc=\"http://purl.org/dc/elements/1.1/\" xmlns:cc=\"http://creativecommons.org/ns#\" xmlns:rdf=\"http://www.w3.org/1999/02/22-rdf-syntax-ns#\">\n",
       "   <cc:Work>\n",
       "    <dc:type rdf:resource=\"http://purl.org/dc/dcmitype/StillImage\"/>\n",
       "    <dc:date>2022-09-02T19:43:01.182648</dc:date>\n",
       "    <dc:format>image/svg+xml</dc:format>\n",
       "    <dc:creator>\n",
       "     <cc:Agent>\n",
       "      <dc:title>Matplotlib v3.5.2, https://matplotlib.org/</dc:title>\n",
       "     </cc:Agent>\n",
       "    </dc:creator>\n",
       "   </cc:Work>\n",
       "  </rdf:RDF>\n",
       " </metadata>\n",
       " <defs>\n",
       "  <style type=\"text/css\">*{stroke-linejoin: round; stroke-linecap: butt}</style>\n",
       " </defs>\n",
       " <g id=\"figure_1\">\n",
       "  <g id=\"patch_1\">\n",
       "   <path d=\"M 0 99.11625 \n",
       "L 635.307812 99.11625 \n",
       "L 635.307812 0 \n",
       "L 0 0 \n",
       "L 0 99.11625 \n",
       "z\n",
       "\" style=\"fill: none\"/>\n",
       "  </g>\n",
       "  <g id=\"axes_1\">\n",
       "   <g id=\"patch_2\">\n",
       "    <path d=\"M 125.907813 61.56 \n",
       "L 273.613695 61.56 \n",
       "L 273.613695 7.2 \n",
       "L 125.907813 7.2 \n",
       "z\n",
       "\" style=\"fill: #ffffff\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_3\">\n",
       "    <path d=\"M 125.907813 9.670909 \n",
       "L 221.414431 9.670909 \n",
       "L 221.414431 28.202727 \n",
       "L 125.907813 28.202727 \n",
       "z\n",
       "\" clip-path=\"url(#p0ad30ebcfb)\" style=\"fill: #e41a1c\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_4\">\n",
       "    <path d=\"M 125.907813 40.557273 \n",
       "L 266.580081 40.557273 \n",
       "L 266.580081 59.089091 \n",
       "L 125.907813 59.089091 \n",
       "z\n",
       "\" clip-path=\"url(#p0ad30ebcfb)\" style=\"fill: #377eb8\"/>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_1\">\n",
       "    <g id=\"xtick_1\">\n",
       "     <g id=\"line2d_1\">\n",
       "      <defs>\n",
       "       <path id=\"mb620112c5a\" d=\"M 0 0 \n",
       "L 0 3.5 \n",
       "\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </defs>\n",
       "      <g>\n",
       "       <use xlink:href=\"#mb620112c5a\" x=\"125.907813\" y=\"61.56\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_1\">\n",
       "      <!-- 0 -->\n",
       "      <g transform=\"translate(122.726562 76.158437)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-30\" d=\"M 2034 4250 \n",
       "Q 1547 4250 1301 3770 \n",
       "Q 1056 3291 1056 2328 \n",
       "Q 1056 1369 1301 889 \n",
       "Q 1547 409 2034 409 \n",
       "Q 2525 409 2770 889 \n",
       "Q 3016 1369 3016 2328 \n",
       "Q 3016 3291 2770 3770 \n",
       "Q 2525 4250 2034 4250 \n",
       "z\n",
       "M 2034 4750 \n",
       "Q 2819 4750 3233 4129 \n",
       "Q 3647 3509 3647 2328 \n",
       "Q 3647 1150 3233 529 \n",
       "Q 2819 -91 2034 -91 \n",
       "Q 1250 -91 836 529 \n",
       "Q 422 1150 422 2328 \n",
       "Q 422 3509 836 4129 \n",
       "Q 1250 4750 2034 4750 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_2\">\n",
       "     <g id=\"line2d_2\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#mb620112c5a\" x=\"195.382329\" y=\"61.56\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_2\">\n",
       "      <!-- 20 -->\n",
       "      <g transform=\"translate(189.019829 76.158437)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-32\" d=\"M 1228 531 \n",
       "L 3431 531 \n",
       "L 3431 0 \n",
       "L 469 0 \n",
       "L 469 531 \n",
       "Q 828 903 1448 1529 \n",
       "Q 2069 2156 2228 2338 \n",
       "Q 2531 2678 2651 2914 \n",
       "Q 2772 3150 2772 3378 \n",
       "Q 2772 3750 2511 3984 \n",
       "Q 2250 4219 1831 4219 \n",
       "Q 1534 4219 1204 4116 \n",
       "Q 875 4013 500 3803 \n",
       "L 500 4441 \n",
       "Q 881 4594 1212 4672 \n",
       "Q 1544 4750 1819 4750 \n",
       "Q 2544 4750 2975 4387 \n",
       "Q 3406 4025 3406 3419 \n",
       "Q 3406 3131 3298 2873 \n",
       "Q 3191 2616 2906 2266 \n",
       "Q 2828 2175 2409 1742 \n",
       "Q 1991 1309 1228 531 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-32\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_3\">\n",
       "     <g id=\"line2d_3\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#mb620112c5a\" x=\"264.856846\" y=\"61.56\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_3\">\n",
       "      <!-- 40 -->\n",
       "      <g transform=\"translate(258.494346 76.158437)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-34\" d=\"M 2419 4116 \n",
       "L 825 1625 \n",
       "L 2419 1625 \n",
       "L 2419 4116 \n",
       "z\n",
       "M 2253 4666 \n",
       "L 3047 4666 \n",
       "L 3047 1625 \n",
       "L 3713 1625 \n",
       "L 3713 1100 \n",
       "L 3047 1100 \n",
       "L 3047 0 \n",
       "L 2419 0 \n",
       "L 2419 1100 \n",
       "L 313 1100 \n",
       "L 313 1709 \n",
       "L 2253 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-34\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"text_4\">\n",
       "     <!-- Samples / sec -->\n",
       "     <g transform=\"translate(165.11466 89.836562)scale(0.1 -0.1)\">\n",
       "      <defs>\n",
       "       <path id=\"DejaVuSans-53\" d=\"M 3425 4513 \n",
       "L 3425 3897 \n",
       "Q 3066 4069 2747 4153 \n",
       "Q 2428 4238 2131 4238 \n",
       "Q 1616 4238 1336 4038 \n",
       "Q 1056 3838 1056 3469 \n",
       "Q 1056 3159 1242 3001 \n",
       "Q 1428 2844 1947 2747 \n",
       "L 2328 2669 \n",
       "Q 3034 2534 3370 2195 \n",
       "Q 3706 1856 3706 1288 \n",
       "Q 3706 609 3251 259 \n",
       "Q 2797 -91 1919 -91 \n",
       "Q 1588 -91 1214 -16 \n",
       "Q 841 59 441 206 \n",
       "L 441 856 \n",
       "Q 825 641 1194 531 \n",
       "Q 1563 422 1919 422 \n",
       "Q 2459 422 2753 634 \n",
       "Q 3047 847 3047 1241 \n",
       "Q 3047 1584 2836 1778 \n",
       "Q 2625 1972 2144 2069 \n",
       "L 1759 2144 \n",
       "Q 1053 2284 737 2584 \n",
       "Q 422 2884 422 3419 \n",
       "Q 422 4038 858 4394 \n",
       "Q 1294 4750 2059 4750 \n",
       "Q 2388 4750 2728 4690 \n",
       "Q 3069 4631 3425 4513 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-61\" d=\"M 2194 1759 \n",
       "Q 1497 1759 1228 1600 \n",
       "Q 959 1441 959 1056 \n",
       "Q 959 750 1161 570 \n",
       "Q 1363 391 1709 391 \n",
       "Q 2188 391 2477 730 \n",
       "Q 2766 1069 2766 1631 \n",
       "L 2766 1759 \n",
       "L 2194 1759 \n",
       "z\n",
       "M 3341 1997 \n",
       "L 3341 0 \n",
       "L 2766 0 \n",
       "L 2766 531 \n",
       "Q 2569 213 2275 61 \n",
       "Q 1981 -91 1556 -91 \n",
       "Q 1019 -91 701 211 \n",
       "Q 384 513 384 1019 \n",
       "Q 384 1609 779 1909 \n",
       "Q 1175 2209 1959 2209 \n",
       "L 2766 2209 \n",
       "L 2766 2266 \n",
       "Q 2766 2663 2505 2880 \n",
       "Q 2244 3097 1772 3097 \n",
       "Q 1472 3097 1187 3025 \n",
       "Q 903 2953 641 2809 \n",
       "L 641 3341 \n",
       "Q 956 3463 1253 3523 \n",
       "Q 1550 3584 1831 3584 \n",
       "Q 2591 3584 2966 3190 \n",
       "Q 3341 2797 3341 1997 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-6d\" d=\"M 3328 2828 \n",
       "Q 3544 3216 3844 3400 \n",
       "Q 4144 3584 4550 3584 \n",
       "Q 5097 3584 5394 3201 \n",
       "Q 5691 2819 5691 2113 \n",
       "L 5691 0 \n",
       "L 5113 0 \n",
       "L 5113 2094 \n",
       "Q 5113 2597 4934 2840 \n",
       "Q 4756 3084 4391 3084 \n",
       "Q 3944 3084 3684 2787 \n",
       "Q 3425 2491 3425 1978 \n",
       "L 3425 0 \n",
       "L 2847 0 \n",
       "L 2847 2094 \n",
       "Q 2847 2600 2669 2842 \n",
       "Q 2491 3084 2119 3084 \n",
       "Q 1678 3084 1418 2786 \n",
       "Q 1159 2488 1159 1978 \n",
       "L 1159 0 \n",
       "L 581 0 \n",
       "L 581 3500 \n",
       "L 1159 3500 \n",
       "L 1159 2956 \n",
       "Q 1356 3278 1631 3431 \n",
       "Q 1906 3584 2284 3584 \n",
       "Q 2666 3584 2933 3390 \n",
       "Q 3200 3197 3328 2828 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-70\" d=\"M 1159 525 \n",
       "L 1159 -1331 \n",
       "L 581 -1331 \n",
       "L 581 3500 \n",
       "L 1159 3500 \n",
       "L 1159 2969 \n",
       "Q 1341 3281 1617 3432 \n",
       "Q 1894 3584 2278 3584 \n",
       "Q 2916 3584 3314 3078 \n",
       "Q 3713 2572 3713 1747 \n",
       "Q 3713 922 3314 415 \n",
       "Q 2916 -91 2278 -91 \n",
       "Q 1894 -91 1617 61 \n",
       "Q 1341 213 1159 525 \n",
       "z\n",
       "M 3116 1747 \n",
       "Q 3116 2381 2855 2742 \n",
       "Q 2594 3103 2138 3103 \n",
       "Q 1681 3103 1420 2742 \n",
       "Q 1159 2381 1159 1747 \n",
       "Q 1159 1113 1420 752 \n",
       "Q 1681 391 2138 391 \n",
       "Q 2594 391 2855 752 \n",
       "Q 3116 1113 3116 1747 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-6c\" d=\"M 603 4863 \n",
       "L 1178 4863 \n",
       "L 1178 0 \n",
       "L 603 0 \n",
       "L 603 4863 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-65\" d=\"M 3597 1894 \n",
       "L 3597 1613 \n",
       "L 953 1613 \n",
       "Q 991 1019 1311 708 \n",
       "Q 1631 397 2203 397 \n",
       "Q 2534 397 2845 478 \n",
       "Q 3156 559 3463 722 \n",
       "L 3463 178 \n",
       "Q 3153 47 2828 -22 \n",
       "Q 2503 -91 2169 -91 \n",
       "Q 1331 -91 842 396 \n",
       "Q 353 884 353 1716 \n",
       "Q 353 2575 817 3079 \n",
       "Q 1281 3584 2069 3584 \n",
       "Q 2775 3584 3186 3129 \n",
       "Q 3597 2675 3597 1894 \n",
       "z\n",
       "M 3022 2063 \n",
       "Q 3016 2534 2758 2815 \n",
       "Q 2500 3097 2075 3097 \n",
       "Q 1594 3097 1305 2825 \n",
       "Q 1016 2553 972 2059 \n",
       "L 3022 2063 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-73\" d=\"M 2834 3397 \n",
       "L 2834 2853 \n",
       "Q 2591 2978 2328 3040 \n",
       "Q 2066 3103 1784 3103 \n",
       "Q 1356 3103 1142 2972 \n",
       "Q 928 2841 928 2578 \n",
       "Q 928 2378 1081 2264 \n",
       "Q 1234 2150 1697 2047 \n",
       "L 1894 2003 \n",
       "Q 2506 1872 2764 1633 \n",
       "Q 3022 1394 3022 966 \n",
       "Q 3022 478 2636 193 \n",
       "Q 2250 -91 1575 -91 \n",
       "Q 1294 -91 989 -36 \n",
       "Q 684 19 347 128 \n",
       "L 347 722 \n",
       "Q 666 556 975 473 \n",
       "Q 1284 391 1588 391 \n",
       "Q 1994 391 2212 530 \n",
       "Q 2431 669 2431 922 \n",
       "Q 2431 1156 2273 1281 \n",
       "Q 2116 1406 1581 1522 \n",
       "L 1381 1569 \n",
       "Q 847 1681 609 1914 \n",
       "Q 372 2147 372 2553 \n",
       "Q 372 3047 722 3315 \n",
       "Q 1072 3584 1716 3584 \n",
       "Q 2034 3584 2315 3537 \n",
       "Q 2597 3491 2834 3397 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-20\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-2f\" d=\"M 1625 4666 \n",
       "L 2156 4666 \n",
       "L 531 -594 \n",
       "L 0 -594 \n",
       "L 1625 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-63\" d=\"M 3122 3366 \n",
       "L 3122 2828 \n",
       "Q 2878 2963 2633 3030 \n",
       "Q 2388 3097 2138 3097 \n",
       "Q 1578 3097 1268 2742 \n",
       "Q 959 2388 959 1747 \n",
       "Q 959 1106 1268 751 \n",
       "Q 1578 397 2138 397 \n",
       "Q 2388 397 2633 464 \n",
       "Q 2878 531 3122 666 \n",
       "L 3122 134 \n",
       "Q 2881 22 2623 -34 \n",
       "Q 2366 -91 2075 -91 \n",
       "Q 1284 -91 818 406 \n",
       "Q 353 903 353 1747 \n",
       "Q 353 2603 823 3093 \n",
       "Q 1294 3584 2113 3584 \n",
       "Q 2378 3584 2631 3529 \n",
       "Q 2884 3475 3122 3366 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "      </defs>\n",
       "      <use xlink:href=\"#DejaVuSans-53\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-61\" x=\"63.476562\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-6d\" x=\"124.755859\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-70\" x=\"222.167969\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-6c\" x=\"285.644531\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-65\" x=\"313.427734\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-73\" x=\"374.951172\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-20\" x=\"427.050781\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-2f\" x=\"458.837891\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-20\" x=\"492.529297\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-73\" x=\"524.316406\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-65\" x=\"576.416016\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-63\" x=\"637.939453\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_2\">\n",
       "    <g id=\"ytick_1\">\n",
       "     <g id=\"line2d_4\">\n",
       "      <defs>\n",
       "       <path id=\"mbb1503d8c4\" d=\"M 0 0 \n",
       "L -3.5 0 \n",
       "\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </defs>\n",
       "      <g>\n",
       "       <use xlink:href=\"#mbb1503d8c4\" x=\"125.907813\" y=\"18.936818\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_5\">\n",
       "      <!-- HF 16-bit, fused-adam -->\n",
       "      <g transform=\"translate(7.2 17.896974)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-48\" d=\"M 628 4666 \n",
       "L 1259 4666 \n",
       "L 1259 2753 \n",
       "L 3553 2753 \n",
       "L 3553 4666 \n",
       "L 4184 4666 \n",
       "L 4184 0 \n",
       "L 3553 0 \n",
       "L 3553 2222 \n",
       "L 1259 2222 \n",
       "L 1259 0 \n",
       "L 628 0 \n",
       "L 628 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-46\" d=\"M 628 4666 \n",
       "L 3309 4666 \n",
       "L 3309 4134 \n",
       "L 1259 4134 \n",
       "L 1259 2759 \n",
       "L 3109 2759 \n",
       "L 3109 2228 \n",
       "L 1259 2228 \n",
       "L 1259 0 \n",
       "L 628 0 \n",
       "L 628 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-31\" d=\"M 794 531 \n",
       "L 1825 531 \n",
       "L 1825 4091 \n",
       "L 703 3866 \n",
       "L 703 4441 \n",
       "L 1819 4666 \n",
       "L 2450 4666 \n",
       "L 2450 531 \n",
       "L 3481 531 \n",
       "L 3481 0 \n",
       "L 794 0 \n",
       "L 794 531 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-36\" d=\"M 2113 2584 \n",
       "Q 1688 2584 1439 2293 \n",
       "Q 1191 2003 1191 1497 \n",
       "Q 1191 994 1439 701 \n",
       "Q 1688 409 2113 409 \n",
       "Q 2538 409 2786 701 \n",
       "Q 3034 994 3034 1497 \n",
       "Q 3034 2003 2786 2293 \n",
       "Q 2538 2584 2113 2584 \n",
       "z\n",
       "M 3366 4563 \n",
       "L 3366 3988 \n",
       "Q 3128 4100 2886 4159 \n",
       "Q 2644 4219 2406 4219 \n",
       "Q 1781 4219 1451 3797 \n",
       "Q 1122 3375 1075 2522 \n",
       "Q 1259 2794 1537 2939 \n",
       "Q 1816 3084 2150 3084 \n",
       "Q 2853 3084 3261 2657 \n",
       "Q 3669 2231 3669 1497 \n",
       "Q 3669 778 3244 343 \n",
       "Q 2819 -91 2113 -91 \n",
       "Q 1303 -91 875 529 \n",
       "Q 447 1150 447 2328 \n",
       "Q 447 3434 972 4092 \n",
       "Q 1497 4750 2381 4750 \n",
       "Q 2619 4750 2861 4703 \n",
       "Q 3103 4656 3366 4563 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-2d\" d=\"M 313 2009 \n",
       "L 1997 2009 \n",
       "L 1997 1497 \n",
       "L 313 1497 \n",
       "L 313 2009 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-62\" d=\"M 3116 1747 \n",
       "Q 3116 2381 2855 2742 \n",
       "Q 2594 3103 2138 3103 \n",
       "Q 1681 3103 1420 2742 \n",
       "Q 1159 2381 1159 1747 \n",
       "Q 1159 1113 1420 752 \n",
       "Q 1681 391 2138 391 \n",
       "Q 2594 391 2855 752 \n",
       "Q 3116 1113 3116 1747 \n",
       "z\n",
       "M 1159 2969 \n",
       "Q 1341 3281 1617 3432 \n",
       "Q 1894 3584 2278 3584 \n",
       "Q 2916 3584 3314 3078 \n",
       "Q 3713 2572 3713 1747 \n",
       "Q 3713 922 3314 415 \n",
       "Q 2916 -91 2278 -91 \n",
       "Q 1894 -91 1617 61 \n",
       "Q 1341 213 1159 525 \n",
       "L 1159 0 \n",
       "L 581 0 \n",
       "L 581 4863 \n",
       "L 1159 4863 \n",
       "L 1159 2969 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-69\" d=\"M 603 3500 \n",
       "L 1178 3500 \n",
       "L 1178 0 \n",
       "L 603 0 \n",
       "L 603 3500 \n",
       "z\n",
       "M 603 4863 \n",
       "L 1178 4863 \n",
       "L 1178 4134 \n",
       "L 603 4134 \n",
       "L 603 4863 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-74\" d=\"M 1172 4494 \n",
       "L 1172 3500 \n",
       "L 2356 3500 \n",
       "L 2356 3053 \n",
       "L 1172 3053 \n",
       "L 1172 1153 \n",
       "Q 1172 725 1289 603 \n",
       "Q 1406 481 1766 481 \n",
       "L 2356 481 \n",
       "L 2356 0 \n",
       "L 1766 0 \n",
       "Q 1100 0 847 248 \n",
       "Q 594 497 594 1153 \n",
       "L 594 3053 \n",
       "L 172 3053 \n",
       "L 172 3500 \n",
       "L 594 3500 \n",
       "L 594 4494 \n",
       "L 1172 4494 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-2c\" d=\"M 750 794 \n",
       "L 1409 794 \n",
       "L 1409 256 \n",
       "L 897 -744 \n",
       "L 494 -744 \n",
       "L 750 256 \n",
       "L 750 794 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-66\" d=\"M 2375 4863 \n",
       "L 2375 4384 \n",
       "L 1825 4384 \n",
       "Q 1516 4384 1395 4259 \n",
       "Q 1275 4134 1275 3809 \n",
       "L 1275 3500 \n",
       "L 2222 3500 \n",
       "L 2222 3053 \n",
       "L 1275 3053 \n",
       "L 1275 0 \n",
       "L 697 0 \n",
       "L 697 3053 \n",
       "L 147 3053 \n",
       "L 147 3500 \n",
       "L 697 3500 \n",
       "L 697 3744 \n",
       "Q 697 4328 969 4595 \n",
       "Q 1241 4863 1831 4863 \n",
       "L 2375 4863 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-75\" d=\"M 544 1381 \n",
       "L 544 3500 \n",
       "L 1119 3500 \n",
       "L 1119 1403 \n",
       "Q 1119 906 1312 657 \n",
       "Q 1506 409 1894 409 \n",
       "Q 2359 409 2629 706 \n",
       "Q 2900 1003 2900 1516 \n",
       "L 2900 3500 \n",
       "L 3475 3500 \n",
       "L 3475 0 \n",
       "L 2900 0 \n",
       "L 2900 538 \n",
       "Q 2691 219 2414 64 \n",
       "Q 2138 -91 1772 -91 \n",
       "Q 1169 -91 856 284 \n",
       "Q 544 659 544 1381 \n",
       "z\n",
       "M 1991 3584 \n",
       "L 1991 3584 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-64\" d=\"M 2906 2969 \n",
       "L 2906 4863 \n",
       "L 3481 4863 \n",
       "L 3481 0 \n",
       "L 2906 0 \n",
       "L 2906 525 \n",
       "Q 2725 213 2448 61 \n",
       "Q 2172 -91 1784 -91 \n",
       "Q 1150 -91 751 415 \n",
       "Q 353 922 353 1747 \n",
       "Q 353 2572 751 3078 \n",
       "Q 1150 3584 1784 3584 \n",
       "Q 2172 3584 2448 3432 \n",
       "Q 2725 3281 2906 2969 \n",
       "z\n",
       "M 947 1747 \n",
       "Q 947 1113 1208 752 \n",
       "Q 1469 391 1925 391 \n",
       "Q 2381 391 2643 752 \n",
       "Q 2906 1113 2906 1747 \n",
       "Q 2906 2381 2643 2742 \n",
       "Q 2381 3103 1925 3103 \n",
       "Q 1469 3103 1208 2742 \n",
       "Q 947 2381 947 1747 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-48\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-46\" x=\"75.195312\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-20\" x=\"132.714844\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-31\" x=\"164.501953\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-36\" x=\"228.125\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-2d\" x=\"291.748047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-62\" x=\"327.832031\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-69\" x=\"391.308594\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-74\" x=\"419.091797\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-2c\" x=\"458.300781\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-20\" x=\"490.087891\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-66\" x=\"521.875\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-75\" x=\"557.080078\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-73\" x=\"620.458984\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-65\" x=\"672.558594\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-64\" x=\"734.082031\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-2d\" x=\"797.558594\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-61\" x=\"833.642578\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-64\" x=\"894.921875\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-61\" x=\"958.398438\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-6d\" x=\"1019.677734\"/>\n",
       "      </g>\n",
       "      <!-- grad_accum=4 -->\n",
       "      <g transform=\"translate(43.026562 29.094787)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-67\" d=\"M 2906 1791 \n",
       "Q 2906 2416 2648 2759 \n",
       "Q 2391 3103 1925 3103 \n",
       "Q 1463 3103 1205 2759 \n",
       "Q 947 2416 947 1791 \n",
       "Q 947 1169 1205 825 \n",
       "Q 1463 481 1925 481 \n",
       "Q 2391 481 2648 825 \n",
       "Q 2906 1169 2906 1791 \n",
       "z\n",
       "M 3481 434 \n",
       "Q 3481 -459 3084 -895 \n",
       "Q 2688 -1331 1869 -1331 \n",
       "Q 1566 -1331 1297 -1286 \n",
       "Q 1028 -1241 775 -1147 \n",
       "L 775 -588 \n",
       "Q 1028 -725 1275 -790 \n",
       "Q 1522 -856 1778 -856 \n",
       "Q 2344 -856 2625 -561 \n",
       "Q 2906 -266 2906 331 \n",
       "L 2906 616 \n",
       "Q 2728 306 2450 153 \n",
       "Q 2172 0 1784 0 \n",
       "Q 1141 0 747 490 \n",
       "Q 353 981 353 1791 \n",
       "Q 353 2603 747 3093 \n",
       "Q 1141 3584 1784 3584 \n",
       "Q 2172 3584 2450 3431 \n",
       "Q 2728 3278 2906 2969 \n",
       "L 2906 3500 \n",
       "L 3481 3500 \n",
       "L 3481 434 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-72\" d=\"M 2631 2963 \n",
       "Q 2534 3019 2420 3045 \n",
       "Q 2306 3072 2169 3072 \n",
       "Q 1681 3072 1420 2755 \n",
       "Q 1159 2438 1159 1844 \n",
       "L 1159 0 \n",
       "L 581 0 \n",
       "L 581 3500 \n",
       "L 1159 3500 \n",
       "L 1159 2956 \n",
       "Q 1341 3275 1631 3429 \n",
       "Q 1922 3584 2338 3584 \n",
       "Q 2397 3584 2469 3576 \n",
       "Q 2541 3569 2628 3553 \n",
       "L 2631 2963 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-5f\" d=\"M 3263 -1063 \n",
       "L 3263 -1509 \n",
       "L -63 -1509 \n",
       "L -63 -1063 \n",
       "L 3263 -1063 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-3d\" d=\"M 678 2906 \n",
       "L 4684 2906 \n",
       "L 4684 2381 \n",
       "L 678 2381 \n",
       "L 678 2906 \n",
       "z\n",
       "M 678 1631 \n",
       "L 4684 1631 \n",
       "L 4684 1100 \n",
       "L 678 1100 \n",
       "L 678 1631 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-67\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-72\" x=\"63.476562\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-61\" x=\"104.589844\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-64\" x=\"165.869141\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-5f\" x=\"229.345703\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-61\" x=\"279.345703\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-63\" x=\"340.625\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-63\" x=\"395.605469\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-75\" x=\"450.585938\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-6d\" x=\"513.964844\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-3d\" x=\"611.376953\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-34\" x=\"695.166016\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_2\">\n",
       "     <g id=\"line2d_5\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#mbb1503d8c4\" x=\"125.907813\" y=\"49.823182\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_6\">\n",
       "      <!-- Megatron BERT -->\n",
       "      <g transform=\"translate(42.698437 53.622401)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-4d\" d=\"M 628 4666 \n",
       "L 1569 4666 \n",
       "L 2759 1491 \n",
       "L 3956 4666 \n",
       "L 4897 4666 \n",
       "L 4897 0 \n",
       "L 4281 0 \n",
       "L 4281 4097 \n",
       "L 3078 897 \n",
       "L 2444 897 \n",
       "L 1241 4097 \n",
       "L 1241 0 \n",
       "L 628 0 \n",
       "L 628 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-6f\" d=\"M 1959 3097 \n",
       "Q 1497 3097 1228 2736 \n",
       "Q 959 2375 959 1747 \n",
       "Q 959 1119 1226 758 \n",
       "Q 1494 397 1959 397 \n",
       "Q 2419 397 2687 759 \n",
       "Q 2956 1122 2956 1747 \n",
       "Q 2956 2369 2687 2733 \n",
       "Q 2419 3097 1959 3097 \n",
       "z\n",
       "M 1959 3584 \n",
       "Q 2709 3584 3137 3096 \n",
       "Q 3566 2609 3566 1747 \n",
       "Q 3566 888 3137 398 \n",
       "Q 2709 -91 1959 -91 \n",
       "Q 1206 -91 779 398 \n",
       "Q 353 888 353 1747 \n",
       "Q 353 2609 779 3096 \n",
       "Q 1206 3584 1959 3584 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-6e\" d=\"M 3513 2113 \n",
       "L 3513 0 \n",
       "L 2938 0 \n",
       "L 2938 2094 \n",
       "Q 2938 2591 2744 2837 \n",
       "Q 2550 3084 2163 3084 \n",
       "Q 1697 3084 1428 2787 \n",
       "Q 1159 2491 1159 1978 \n",
       "L 1159 0 \n",
       "L 581 0 \n",
       "L 581 3500 \n",
       "L 1159 3500 \n",
       "L 1159 2956 \n",
       "Q 1366 3272 1645 3428 \n",
       "Q 1925 3584 2291 3584 \n",
       "Q 2894 3584 3203 3211 \n",
       "Q 3513 2838 3513 2113 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-42\" d=\"M 1259 2228 \n",
       "L 1259 519 \n",
       "L 2272 519 \n",
       "Q 2781 519 3026 730 \n",
       "Q 3272 941 3272 1375 \n",
       "Q 3272 1813 3026 2020 \n",
       "Q 2781 2228 2272 2228 \n",
       "L 1259 2228 \n",
       "z\n",
       "M 1259 4147 \n",
       "L 1259 2741 \n",
       "L 2194 2741 \n",
       "Q 2656 2741 2882 2914 \n",
       "Q 3109 3088 3109 3444 \n",
       "Q 3109 3797 2882 3972 \n",
       "Q 2656 4147 2194 4147 \n",
       "L 1259 4147 \n",
       "z\n",
       "M 628 4666 \n",
       "L 2241 4666 \n",
       "Q 2963 4666 3353 4366 \n",
       "Q 3744 4066 3744 3513 \n",
       "Q 3744 3084 3544 2831 \n",
       "Q 3344 2578 2956 2516 \n",
       "Q 3422 2416 3680 2098 \n",
       "Q 3938 1781 3938 1306 \n",
       "Q 3938 681 3513 340 \n",
       "Q 3088 0 2303 0 \n",
       "L 628 0 \n",
       "L 628 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-45\" d=\"M 628 4666 \n",
       "L 3578 4666 \n",
       "L 3578 4134 \n",
       "L 1259 4134 \n",
       "L 1259 2753 \n",
       "L 3481 2753 \n",
       "L 3481 2222 \n",
       "L 1259 2222 \n",
       "L 1259 531 \n",
       "L 3634 531 \n",
       "L 3634 0 \n",
       "L 628 0 \n",
       "L 628 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-52\" d=\"M 2841 2188 \n",
       "Q 3044 2119 3236 1894 \n",
       "Q 3428 1669 3622 1275 \n",
       "L 4263 0 \n",
       "L 3584 0 \n",
       "L 2988 1197 \n",
       "Q 2756 1666 2539 1819 \n",
       "Q 2322 1972 1947 1972 \n",
       "L 1259 1972 \n",
       "L 1259 0 \n",
       "L 628 0 \n",
       "L 628 4666 \n",
       "L 2053 4666 \n",
       "Q 2853 4666 3247 4331 \n",
       "Q 3641 3997 3641 3322 \n",
       "Q 3641 2881 3436 2590 \n",
       "Q 3231 2300 2841 2188 \n",
       "z\n",
       "M 1259 4147 \n",
       "L 1259 2491 \n",
       "L 2053 2491 \n",
       "Q 2509 2491 2742 2702 \n",
       "Q 2975 2913 2975 3322 \n",
       "Q 2975 3731 2742 3939 \n",
       "Q 2509 4147 2053 4147 \n",
       "L 1259 4147 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-54\" d=\"M -19 4666 \n",
       "L 3928 4666 \n",
       "L 3928 4134 \n",
       "L 2272 4134 \n",
       "L 2272 0 \n",
       "L 1638 0 \n",
       "L 1638 4134 \n",
       "L -19 4134 \n",
       "L -19 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-4d\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-65\" x=\"86.279297\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-67\" x=\"147.802734\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-61\" x=\"211.279297\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-74\" x=\"272.558594\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-72\" x=\"311.767578\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-6f\" x=\"350.630859\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-6e\" x=\"411.8125\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-20\" x=\"475.191406\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-42\" x=\"506.978516\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-45\" x=\"575.582031\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-52\" x=\"638.765625\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-54\" x=\"700.998047\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"patch_5\">\n",
       "    <path d=\"M 125.907813 61.56 \n",
       "L 125.907813 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_6\">\n",
       "    <path d=\"M 273.613695 61.56 \n",
       "L 273.613695 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_7\">\n",
       "    <path d=\"M 125.907813 61.56 \n",
       "L 273.613695 61.56 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_8\">\n",
       "    <path d=\"M 125.907813 7.2 \n",
       "L 273.613695 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "  </g>\n",
       "  <g id=\"axes_2\">\n",
       "   <g id=\"patch_9\">\n",
       "    <path d=\"M 303.154871 61.56 \n",
       "L 450.860754 61.56 \n",
       "L 450.860754 7.2 \n",
       "L 303.154871 7.2 \n",
       "z\n",
       "\" style=\"fill: #ffffff\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_10\">\n",
       "    <path d=\"M 303.154871 9.670909 \n",
       "L 398.66149 9.670909 \n",
       "L 398.66149 28.202727 \n",
       "L 303.154871 28.202727 \n",
       "z\n",
       "\" clip-path=\"url(#p01353317e8)\" style=\"fill: #e41a1c\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_11\">\n",
       "    <path d=\"M 303.154871 40.557273 \n",
       "L 443.82714 40.557273 \n",
       "L 443.82714 59.089091 \n",
       "L 303.154871 59.089091 \n",
       "z\n",
       "\" clip-path=\"url(#p01353317e8)\" style=\"fill: #377eb8\"/>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_3\">\n",
       "    <g id=\"xtick_4\">\n",
       "     <g id=\"line2d_6\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#mb620112c5a\" x=\"303.154871\" y=\"61.56\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_7\">\n",
       "      <!-- 0 -->\n",
       "      <g transform=\"translate(299.973621 76.158437)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_5\">\n",
       "     <g id=\"line2d_7\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#mb620112c5a\" x=\"366.25408\" y=\"61.56\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_8\">\n",
       "      <!-- 20 -->\n",
       "      <g transform=\"translate(359.89158 76.158437)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-32\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_6\">\n",
       "     <g id=\"line2d_8\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#mb620112c5a\" x=\"429.353289\" y=\"61.56\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_9\">\n",
       "      <!-- 40 -->\n",
       "      <g transform=\"translate(422.990789 76.158437)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-34\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"text_10\">\n",
       "     <!-- per GPU TFLOPS -->\n",
       "     <g transform=\"translate(336.315625 89.836562)scale(0.1 -0.1)\">\n",
       "      <defs>\n",
       "       <path id=\"DejaVuSans-47\" d=\"M 3809 666 \n",
       "L 3809 1919 \n",
       "L 2778 1919 \n",
       "L 2778 2438 \n",
       "L 4434 2438 \n",
       "L 4434 434 \n",
       "Q 4069 175 3628 42 \n",
       "Q 3188 -91 2688 -91 \n",
       "Q 1594 -91 976 548 \n",
       "Q 359 1188 359 2328 \n",
       "Q 359 3472 976 4111 \n",
       "Q 1594 4750 2688 4750 \n",
       "Q 3144 4750 3555 4637 \n",
       "Q 3966 4525 4313 4306 \n",
       "L 4313 3634 \n",
       "Q 3963 3931 3569 4081 \n",
       "Q 3175 4231 2741 4231 \n",
       "Q 1884 4231 1454 3753 \n",
       "Q 1025 3275 1025 2328 \n",
       "Q 1025 1384 1454 906 \n",
       "Q 1884 428 2741 428 \n",
       "Q 3075 428 3337 486 \n",
       "Q 3600 544 3809 666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-50\" d=\"M 1259 4147 \n",
       "L 1259 2394 \n",
       "L 2053 2394 \n",
       "Q 2494 2394 2734 2622 \n",
       "Q 2975 2850 2975 3272 \n",
       "Q 2975 3691 2734 3919 \n",
       "Q 2494 4147 2053 4147 \n",
       "L 1259 4147 \n",
       "z\n",
       "M 628 4666 \n",
       "L 2053 4666 \n",
       "Q 2838 4666 3239 4311 \n",
       "Q 3641 3956 3641 3272 \n",
       "Q 3641 2581 3239 2228 \n",
       "Q 2838 1875 2053 1875 \n",
       "L 1259 1875 \n",
       "L 1259 0 \n",
       "L 628 0 \n",
       "L 628 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-55\" d=\"M 556 4666 \n",
       "L 1191 4666 \n",
       "L 1191 1831 \n",
       "Q 1191 1081 1462 751 \n",
       "Q 1734 422 2344 422 \n",
       "Q 2950 422 3222 751 \n",
       "Q 3494 1081 3494 1831 \n",
       "L 3494 4666 \n",
       "L 4128 4666 \n",
       "L 4128 1753 \n",
       "Q 4128 841 3676 375 \n",
       "Q 3225 -91 2344 -91 \n",
       "Q 1459 -91 1007 375 \n",
       "Q 556 841 556 1753 \n",
       "L 556 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-4c\" d=\"M 628 4666 \n",
       "L 1259 4666 \n",
       "L 1259 531 \n",
       "L 3531 531 \n",
       "L 3531 0 \n",
       "L 628 0 \n",
       "L 628 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-4f\" d=\"M 2522 4238 \n",
       "Q 1834 4238 1429 3725 \n",
       "Q 1025 3213 1025 2328 \n",
       "Q 1025 1447 1429 934 \n",
       "Q 1834 422 2522 422 \n",
       "Q 3209 422 3611 934 \n",
       "Q 4013 1447 4013 2328 \n",
       "Q 4013 3213 3611 3725 \n",
       "Q 3209 4238 2522 4238 \n",
       "z\n",
       "M 2522 4750 \n",
       "Q 3503 4750 4090 4092 \n",
       "Q 4678 3434 4678 2328 \n",
       "Q 4678 1225 4090 567 \n",
       "Q 3503 -91 2522 -91 \n",
       "Q 1538 -91 948 565 \n",
       "Q 359 1222 359 2328 \n",
       "Q 359 3434 948 4092 \n",
       "Q 1538 4750 2522 4750 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "      </defs>\n",
       "      <use xlink:href=\"#DejaVuSans-70\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-65\" x=\"63.476562\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-72\" x=\"125\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-20\" x=\"166.113281\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-47\" x=\"197.900391\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-50\" x=\"275.390625\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-55\" x=\"335.693359\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-20\" x=\"408.886719\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-54\" x=\"440.673828\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-46\" x=\"501.757812\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-4c\" x=\"559.277344\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-4f\" x=\"611.365234\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-50\" x=\"690.076172\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-53\" x=\"750.378906\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_4\">\n",
       "    <g id=\"ytick_3\">\n",
       "     <g id=\"line2d_9\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#mbb1503d8c4\" x=\"303.154871\" y=\"18.936818\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_4\">\n",
       "     <g id=\"line2d_10\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#mbb1503d8c4\" x=\"303.154871\" y=\"49.823182\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"patch_12\">\n",
       "    <path d=\"M 303.154871 61.56 \n",
       "L 303.154871 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_13\">\n",
       "    <path d=\"M 450.860754 61.56 \n",
       "L 450.860754 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_14\">\n",
       "    <path d=\"M 303.154871 61.56 \n",
       "L 450.860754 61.56 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_15\">\n",
       "    <path d=\"M 303.154871 7.2 \n",
       "L 450.860754 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "  </g>\n",
       "  <g id=\"axes_3\">\n",
       "   <g id=\"patch_16\">\n",
       "    <path d=\"M 480.40193 61.56 \n",
       "L 628.107813 61.56 \n",
       "L 628.107813 7.2 \n",
       "L 480.40193 7.2 \n",
       "z\n",
       "\" style=\"fill: #ffffff\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_17\">\n",
       "    <path d=\"M 480.40193 9.670909 \n",
       "L 621.074199 9.670909 \n",
       "L 621.074199 28.202727 \n",
       "L 480.40193 28.202727 \n",
       "z\n",
       "\" clip-path=\"url(#pc99471c55f)\" style=\"fill: #e41a1c\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_18\">\n",
       "    <path d=\"M 480.40193 40.557273 \n",
       "L 613.869461 40.557273 \n",
       "L 613.869461 59.089091 \n",
       "L 480.40193 59.089091 \n",
       "z\n",
       "\" clip-path=\"url(#pc99471c55f)\" style=\"fill: #377eb8\"/>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_5\">\n",
       "    <g id=\"xtick_7\">\n",
       "     <g id=\"line2d_11\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#mb620112c5a\" x=\"480.40193\" y=\"61.56\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_11\">\n",
       "      <!-- 0 -->\n",
       "      <g transform=\"translate(477.22068 76.158437)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_8\">\n",
       "     <g id=\"line2d_12\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#mb620112c5a\" x=\"542.076677\" y=\"61.56\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_12\">\n",
       "      <!-- 10 -->\n",
       "      <g transform=\"translate(535.714177 76.158437)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-31\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_9\">\n",
       "     <g id=\"line2d_13\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#mb620112c5a\" x=\"603.751424\" y=\"61.56\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_13\">\n",
       "      <!-- 20 -->\n",
       "      <g transform=\"translate(597.388924 76.158437)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-32\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"text_14\">\n",
       "     <!-- per GPU memory (GB) -->\n",
       "     <g transform=\"translate(498.53534 89.836562)scale(0.1 -0.1)\">\n",
       "      <defs>\n",
       "       <path id=\"DejaVuSans-79\" d=\"M 2059 -325 \n",
       "Q 1816 -950 1584 -1140 \n",
       "Q 1353 -1331 966 -1331 \n",
       "L 506 -1331 \n",
       "L 506 -850 \n",
       "L 844 -850 \n",
       "Q 1081 -850 1212 -737 \n",
       "Q 1344 -625 1503 -206 \n",
       "L 1606 56 \n",
       "L 191 3500 \n",
       "L 800 3500 \n",
       "L 1894 763 \n",
       "L 2988 3500 \n",
       "L 3597 3500 \n",
       "L 2059 -325 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-28\" d=\"M 1984 4856 \n",
       "Q 1566 4138 1362 3434 \n",
       "Q 1159 2731 1159 2009 \n",
       "Q 1159 1288 1364 580 \n",
       "Q 1569 -128 1984 -844 \n",
       "L 1484 -844 \n",
       "Q 1016 -109 783 600 \n",
       "Q 550 1309 550 2009 \n",
       "Q 550 2706 781 3412 \n",
       "Q 1013 4119 1484 4856 \n",
       "L 1984 4856 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-29\" d=\"M 513 4856 \n",
       "L 1013 4856 \n",
       "Q 1481 4119 1714 3412 \n",
       "Q 1947 2706 1947 2009 \n",
       "Q 1947 1309 1714 600 \n",
       "Q 1481 -109 1013 -844 \n",
       "L 513 -844 \n",
       "Q 928 -128 1133 580 \n",
       "Q 1338 1288 1338 2009 \n",
       "Q 1338 2731 1133 3434 \n",
       "Q 928 4138 513 4856 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "      </defs>\n",
       "      <use xlink:href=\"#DejaVuSans-70\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-65\" x=\"63.476562\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-72\" x=\"125\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-20\" x=\"166.113281\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-47\" x=\"197.900391\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-50\" x=\"275.390625\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-55\" x=\"335.693359\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-20\" x=\"408.886719\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-6d\" x=\"440.673828\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-65\" x=\"538.085938\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-6d\" x=\"599.609375\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-6f\" x=\"697.021484\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-72\" x=\"758.203125\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-79\" x=\"799.316406\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-20\" x=\"858.496094\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-28\" x=\"890.283203\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-47\" x=\"929.296875\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-42\" x=\"1006.787109\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-29\" x=\"1075.390625\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_6\">\n",
       "    <g id=\"ytick_5\">\n",
       "     <g id=\"line2d_14\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#mbb1503d8c4\" x=\"480.40193\" y=\"18.936818\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_6\">\n",
       "     <g id=\"line2d_15\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#mbb1503d8c4\" x=\"480.40193\" y=\"49.823182\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"patch_19\">\n",
       "    <path d=\"M 480.40193 61.56 \n",
       "L 480.40193 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_20\">\n",
       "    <path d=\"M 628.107813 61.56 \n",
       "L 628.107813 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_21\">\n",
       "    <path d=\"M 480.40193 61.56 \n",
       "L 628.107813 61.56 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_22\">\n",
       "    <path d=\"M 480.40193 7.2 \n",
       "L 628.107813 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "  </g>\n",
       " </g>\n",
       " <defs>\n",
       "  <clipPath id=\"p0ad30ebcfb\">\n",
       "   <rect x=\"125.907813\" y=\"7.2\" width=\"147.705882\" height=\"54.36\"/>\n",
       "  </clipPath>\n",
       "  <clipPath id=\"p01353317e8\">\n",
       "   <rect x=\"303.154871\" y=\"7.2\" width=\"147.705882\" height=\"54.36\"/>\n",
       "  </clipPath>\n",
       "  <clipPath id=\"pc99471c55f\">\n",
       "   <rect x=\"480.40193\" y=\"7.2\" width=\"147.705882\" height=\"54.36\"/>\n",
       "  </clipPath>\n",
       " </defs>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<Figure size 648x72 with 3 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "mega_bert = megatron_bert(Exp('Megatron BERT', 'bert-large-uncased', 16, bf16=True))\n",
    "compare([bert_half_fused_accum, mega_bert])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7b0787d7",
   "metadata": {},
   "source": [
    "Note that Megatron allows to use a larger batch size and outperforms Huggingface even without gradient accumulation. One reason is the highly efficient [custom kernels](https://github.com/NVIDIA/Megatron-LM/tree/main/megatron/fused_kernels) that not only improve performance but also reduce memory usage. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e7f05b7e",
   "metadata": {},
   "source": [
    "## GPT-2 on a Single GPU\n",
    "\n",
    "Next we train language model with GPT-2. First define the function to use HuggingFace."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "8c6e62cd",
   "metadata": {},
   "outputs": [],
   "source": [
    "def hf_gpt(exp):\n",
    "    cmd = f'''export CUDA_VISIBLE_DEVICES={exp.gpus}; \\\n",
    "{exp.launcher} transformers/examples/pytorch/language-modeling/run_clm.py \\\n",
    "--config_name {exp.model} --tokenizer_name {exp.model} \\\n",
    "--dataset_name wikitext --dataset_config_name wikitext-2-raw-v1 \\\n",
    "--do_train --per_device_train_batch_size {exp.batch_size} \\\n",
    "--block_size {exp.seq_len} --learning_rate 2e-5 \\\n",
    "--max_steps {exp.steps} --optim {exp.optim} \\\n",
    "--fp16 {exp.fp16} --bf16 {exp.bf16} \\\n",
    "--gradient_accumulation_steps {exp.grad_accum} \\\n",
    "--gradient_checkpointing {exp.grad_ckpt} \\\n",
    "--output_dir /tmp/gpt/ --overwrite_output_dir yes --skip_memory_metrics False'''\n",
    "    if exp.deepspeed:\n",
    "        cmd += f' --deepspeed {exp.ds_config}'\n",
    "    cmd += ' > log.txt 2>&1'\n",
    "    os.system(cmd)\n",
    "    return hf_log(exp, 'log.txt')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c6212ad0",
   "metadata": {},
   "source": [
    "We use `gpt2-medium` whose architecture is similar to `bert-large`. GPT-2 models uses a larger sequence length 1024."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "1b4a163c",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Downloading: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 718/718 [00:00<00:00, 631kB/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total samples / second\t: 7.7\n",
      "Per GPU memory (GB)\t: 19.0\n",
      "Per GPU TFLOPs\t\t: 19.1\n"
     ]
    }
   ],
   "source": [
    "hf_gpt2 = hf_gpt(Exp(\n",
    "    \"HF GPT2\", \"gpt2-medium\", 2, bf16=True, optim='adamw_apex_fused', grad_accum=4))\n",
    "hf_gpt2.print_results()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8b1b1180",
   "metadata": {},
   "source": [
    "Let's try Megatron's implementation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "725ac609",
   "metadata": {},
   "outputs": [],
   "source": [
    "def megatron_gpt(exp):\n",
    "    global_batch_size = exp.batch_size * exp.num_gpus * exp.grad_accum / exp.tensor_para\n",
    "    cmd = f'''{exp.launcher} Megatron-LM/pretrain_gpt.py \\\n",
    "--num-layers {exp.num_layers} --hidden-size {exp.hidden_size} \\\n",
    "--num-attention-heads {exp.num_heads} \\\n",
    "--tensor-model-parallel-size {exp.tensor_para} \\\n",
    "--micro-batch-size {exp.batch_size} --global-batch-size {int(global_batch_size)} \\\n",
    "--seq-length {exp.seq_len} --max-position-embeddings {exp.seq_len} \\\n",
    "--train-iters {exp.steps} --data-path gpt2-sample_text_document \\\n",
    "--vocab-file gpt2-vocab.json --merge-file gpt2-merges.txt \\\n",
    "--data-impl mmap --lr 0.00015 --log-interval 5 '''\n",
    "    cmd += '--bf16 ' if exp.bf16 else ''\n",
    "    cmd += '--fp16 ' if exp.fp16 else ''\n",
    "    cmd += ' > log.txt 2>&1'\n",
    "    os.system(cmd)\n",
    "    return megatron_log(exp, 'log.txt') "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b650868c",
   "metadata": {},
   "source": [
    "Downloads data for Megatron"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "6d4679cb",
   "metadata": {},
   "outputs": [],
   "source": [
    "!wget -nc https://s3.amazonaws.com/models.huggingface.co/bert/gpt2-vocab.json\n",
    "!wget -nc https://s3.amazonaws.com/models.huggingface.co/bert/gpt2-merges.txt    \n",
    "!wget -nc https://github.com/mli/transformers-benchmarks/raw/main/data/gpt2-sample_text_document.bin\n",
    "!wget -nc https://github.com/mli/transformers-benchmarks/raw/main/data/gpt2-sample_text_document.idx\n",
    "    \n",
    "clear_output()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a4b5c6ce",
   "metadata": {},
   "source": [
    "Again, Megatron allows a larger batch size and outperforms Huggingface."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "1369b10d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Time breakdown\t\t: forward+backward 0.91, communication 0.01, optimizer 0.07\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       "  \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<svg xmlns:xlink=\"http://www.w3.org/1999/xlink\" width=\"600.546875pt\" height=\"126.29625pt\" viewBox=\"0 0 600.546875 126.29625\" xmlns=\"http://www.w3.org/2000/svg\" version=\"1.1\">\n",
       " <metadata>\n",
       "  <rdf:RDF xmlns:dc=\"http://purl.org/dc/elements/1.1/\" xmlns:cc=\"http://creativecommons.org/ns#\" xmlns:rdf=\"http://www.w3.org/1999/02/22-rdf-syntax-ns#\">\n",
       "   <cc:Work>\n",
       "    <dc:type rdf:resource=\"http://purl.org/dc/dcmitype/StillImage\"/>\n",
       "    <dc:date>2022-09-02T19:44:21.076842</dc:date>\n",
       "    <dc:format>image/svg+xml</dc:format>\n",
       "    <dc:creator>\n",
       "     <cc:Agent>\n",
       "      <dc:title>Matplotlib v3.5.2, https://matplotlib.org/</dc:title>\n",
       "     </cc:Agent>\n",
       "    </dc:creator>\n",
       "   </cc:Work>\n",
       "  </rdf:RDF>\n",
       " </metadata>\n",
       " <defs>\n",
       "  <style type=\"text/css\">*{stroke-linejoin: round; stroke-linecap: butt}</style>\n",
       " </defs>\n",
       " <g id=\"figure_1\">\n",
       "  <g id=\"patch_1\">\n",
       "   <path d=\"M 0 126.29625 \n",
       "L 600.546875 126.29625 \n",
       "L 600.546875 0 \n",
       "L 0 0 \n",
       "L 0 126.29625 \n",
       "z\n",
       "\" style=\"fill: none\"/>\n",
       "  </g>\n",
       "  <g id=\"axes_1\">\n",
       "   <g id=\"patch_2\">\n",
       "    <path d=\"M 91.146875 88.74 \n",
       "L 238.852757 88.74 \n",
       "L 238.852757 7.2 \n",
       "L 91.146875 7.2 \n",
       "z\n",
       "\" style=\"fill: #ffffff\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_3\">\n",
       "    <path d=\"M 91.146875 10.906364 \n",
       "L 231.819144 10.906364 \n",
       "L 231.819144 28.012657 \n",
       "L 91.146875 28.012657 \n",
       "z\n",
       "\" clip-path=\"url(#pb188ee7e5e)\" style=\"fill: #e41a1c\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_4\">\n",
       "    <path d=\"M 91.146875 39.416853 \n",
       "L 117.880669 39.416853 \n",
       "L 117.880669 56.523147 \n",
       "L 91.146875 56.523147 \n",
       "z\n",
       "\" clip-path=\"url(#pb188ee7e5e)\" style=\"fill: #377eb8\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_5\">\n",
       "    <path d=\"M 91.146875 67.927343 \n",
       "L 150.183684 67.927343 \n",
       "L 150.183684 85.033636 \n",
       "L 91.146875 85.033636 \n",
       "z\n",
       "\" clip-path=\"url(#pb188ee7e5e)\" style=\"fill: #4daf4a\"/>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_1\">\n",
       "    <g id=\"xtick_1\">\n",
       "     <g id=\"line2d_1\">\n",
       "      <defs>\n",
       "       <path id=\"m4c59257877\" d=\"M 0 0 \n",
       "L 0 3.5 \n",
       "\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </defs>\n",
       "      <g>\n",
       "       <use xlink:href=\"#m4c59257877\" x=\"91.146875\" y=\"88.74\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_1\">\n",
       "      <!-- 0 -->\n",
       "      <g transform=\"translate(87.965625 103.338438)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-30\" d=\"M 2034 4250 \n",
       "Q 1547 4250 1301 3770 \n",
       "Q 1056 3291 1056 2328 \n",
       "Q 1056 1369 1301 889 \n",
       "Q 1547 409 2034 409 \n",
       "Q 2525 409 2770 889 \n",
       "Q 3016 1369 3016 2328 \n",
       "Q 3016 3291 2770 3770 \n",
       "Q 2525 4250 2034 4250 \n",
       "z\n",
       "M 2034 4750 \n",
       "Q 2819 4750 3233 4129 \n",
       "Q 3647 3509 3647 2328 \n",
       "Q 3647 1150 3233 529 \n",
       "Q 2819 -91 2034 -91 \n",
       "Q 1250 -91 836 529 \n",
       "Q 422 1150 422 2328 \n",
       "Q 422 3509 836 4129 \n",
       "Q 1250 4750 2034 4750 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_2\">\n",
       "     <g id=\"line2d_2\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m4c59257877\" x=\"160.621392\" y=\"88.74\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_2\">\n",
       "      <!-- 20 -->\n",
       "      <g transform=\"translate(154.258892 103.338438)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-32\" d=\"M 1228 531 \n",
       "L 3431 531 \n",
       "L 3431 0 \n",
       "L 469 0 \n",
       "L 469 531 \n",
       "Q 828 903 1448 1529 \n",
       "Q 2069 2156 2228 2338 \n",
       "Q 2531 2678 2651 2914 \n",
       "Q 2772 3150 2772 3378 \n",
       "Q 2772 3750 2511 3984 \n",
       "Q 2250 4219 1831 4219 \n",
       "Q 1534 4219 1204 4116 \n",
       "Q 875 4013 500 3803 \n",
       "L 500 4441 \n",
       "Q 881 4594 1212 4672 \n",
       "Q 1544 4750 1819 4750 \n",
       "Q 2544 4750 2975 4387 \n",
       "Q 3406 4025 3406 3419 \n",
       "Q 3406 3131 3298 2873 \n",
       "Q 3191 2616 2906 2266 \n",
       "Q 2828 2175 2409 1742 \n",
       "Q 1991 1309 1228 531 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-32\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_3\">\n",
       "     <g id=\"line2d_3\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m4c59257877\" x=\"230.095909\" y=\"88.74\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_3\">\n",
       "      <!-- 40 -->\n",
       "      <g transform=\"translate(223.733409 103.338438)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-34\" d=\"M 2419 4116 \n",
       "L 825 1625 \n",
       "L 2419 1625 \n",
       "L 2419 4116 \n",
       "z\n",
       "M 2253 4666 \n",
       "L 3047 4666 \n",
       "L 3047 1625 \n",
       "L 3713 1625 \n",
       "L 3713 1100 \n",
       "L 3047 1100 \n",
       "L 3047 0 \n",
       "L 2419 0 \n",
       "L 2419 1100 \n",
       "L 313 1100 \n",
       "L 313 1709 \n",
       "L 2253 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-34\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"text_4\">\n",
       "     <!-- Samples / sec -->\n",
       "     <g transform=\"translate(130.353722 117.016563)scale(0.1 -0.1)\">\n",
       "      <defs>\n",
       "       <path id=\"DejaVuSans-53\" d=\"M 3425 4513 \n",
       "L 3425 3897 \n",
       "Q 3066 4069 2747 4153 \n",
       "Q 2428 4238 2131 4238 \n",
       "Q 1616 4238 1336 4038 \n",
       "Q 1056 3838 1056 3469 \n",
       "Q 1056 3159 1242 3001 \n",
       "Q 1428 2844 1947 2747 \n",
       "L 2328 2669 \n",
       "Q 3034 2534 3370 2195 \n",
       "Q 3706 1856 3706 1288 \n",
       "Q 3706 609 3251 259 \n",
       "Q 2797 -91 1919 -91 \n",
       "Q 1588 -91 1214 -16 \n",
       "Q 841 59 441 206 \n",
       "L 441 856 \n",
       "Q 825 641 1194 531 \n",
       "Q 1563 422 1919 422 \n",
       "Q 2459 422 2753 634 \n",
       "Q 3047 847 3047 1241 \n",
       "Q 3047 1584 2836 1778 \n",
       "Q 2625 1972 2144 2069 \n",
       "L 1759 2144 \n",
       "Q 1053 2284 737 2584 \n",
       "Q 422 2884 422 3419 \n",
       "Q 422 4038 858 4394 \n",
       "Q 1294 4750 2059 4750 \n",
       "Q 2388 4750 2728 4690 \n",
       "Q 3069 4631 3425 4513 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-61\" d=\"M 2194 1759 \n",
       "Q 1497 1759 1228 1600 \n",
       "Q 959 1441 959 1056 \n",
       "Q 959 750 1161 570 \n",
       "Q 1363 391 1709 391 \n",
       "Q 2188 391 2477 730 \n",
       "Q 2766 1069 2766 1631 \n",
       "L 2766 1759 \n",
       "L 2194 1759 \n",
       "z\n",
       "M 3341 1997 \n",
       "L 3341 0 \n",
       "L 2766 0 \n",
       "L 2766 531 \n",
       "Q 2569 213 2275 61 \n",
       "Q 1981 -91 1556 -91 \n",
       "Q 1019 -91 701 211 \n",
       "Q 384 513 384 1019 \n",
       "Q 384 1609 779 1909 \n",
       "Q 1175 2209 1959 2209 \n",
       "L 2766 2209 \n",
       "L 2766 2266 \n",
       "Q 2766 2663 2505 2880 \n",
       "Q 2244 3097 1772 3097 \n",
       "Q 1472 3097 1187 3025 \n",
       "Q 903 2953 641 2809 \n",
       "L 641 3341 \n",
       "Q 956 3463 1253 3523 \n",
       "Q 1550 3584 1831 3584 \n",
       "Q 2591 3584 2966 3190 \n",
       "Q 3341 2797 3341 1997 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-6d\" d=\"M 3328 2828 \n",
       "Q 3544 3216 3844 3400 \n",
       "Q 4144 3584 4550 3584 \n",
       "Q 5097 3584 5394 3201 \n",
       "Q 5691 2819 5691 2113 \n",
       "L 5691 0 \n",
       "L 5113 0 \n",
       "L 5113 2094 \n",
       "Q 5113 2597 4934 2840 \n",
       "Q 4756 3084 4391 3084 \n",
       "Q 3944 3084 3684 2787 \n",
       "Q 3425 2491 3425 1978 \n",
       "L 3425 0 \n",
       "L 2847 0 \n",
       "L 2847 2094 \n",
       "Q 2847 2600 2669 2842 \n",
       "Q 2491 3084 2119 3084 \n",
       "Q 1678 3084 1418 2786 \n",
       "Q 1159 2488 1159 1978 \n",
       "L 1159 0 \n",
       "L 581 0 \n",
       "L 581 3500 \n",
       "L 1159 3500 \n",
       "L 1159 2956 \n",
       "Q 1356 3278 1631 3431 \n",
       "Q 1906 3584 2284 3584 \n",
       "Q 2666 3584 2933 3390 \n",
       "Q 3200 3197 3328 2828 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-70\" d=\"M 1159 525 \n",
       "L 1159 -1331 \n",
       "L 581 -1331 \n",
       "L 581 3500 \n",
       "L 1159 3500 \n",
       "L 1159 2969 \n",
       "Q 1341 3281 1617 3432 \n",
       "Q 1894 3584 2278 3584 \n",
       "Q 2916 3584 3314 3078 \n",
       "Q 3713 2572 3713 1747 \n",
       "Q 3713 922 3314 415 \n",
       "Q 2916 -91 2278 -91 \n",
       "Q 1894 -91 1617 61 \n",
       "Q 1341 213 1159 525 \n",
       "z\n",
       "M 3116 1747 \n",
       "Q 3116 2381 2855 2742 \n",
       "Q 2594 3103 2138 3103 \n",
       "Q 1681 3103 1420 2742 \n",
       "Q 1159 2381 1159 1747 \n",
       "Q 1159 1113 1420 752 \n",
       "Q 1681 391 2138 391 \n",
       "Q 2594 391 2855 752 \n",
       "Q 3116 1113 3116 1747 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-6c\" d=\"M 603 4863 \n",
       "L 1178 4863 \n",
       "L 1178 0 \n",
       "L 603 0 \n",
       "L 603 4863 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-65\" d=\"M 3597 1894 \n",
       "L 3597 1613 \n",
       "L 953 1613 \n",
       "Q 991 1019 1311 708 \n",
       "Q 1631 397 2203 397 \n",
       "Q 2534 397 2845 478 \n",
       "Q 3156 559 3463 722 \n",
       "L 3463 178 \n",
       "Q 3153 47 2828 -22 \n",
       "Q 2503 -91 2169 -91 \n",
       "Q 1331 -91 842 396 \n",
       "Q 353 884 353 1716 \n",
       "Q 353 2575 817 3079 \n",
       "Q 1281 3584 2069 3584 \n",
       "Q 2775 3584 3186 3129 \n",
       "Q 3597 2675 3597 1894 \n",
       "z\n",
       "M 3022 2063 \n",
       "Q 3016 2534 2758 2815 \n",
       "Q 2500 3097 2075 3097 \n",
       "Q 1594 3097 1305 2825 \n",
       "Q 1016 2553 972 2059 \n",
       "L 3022 2063 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-73\" d=\"M 2834 3397 \n",
       "L 2834 2853 \n",
       "Q 2591 2978 2328 3040 \n",
       "Q 2066 3103 1784 3103 \n",
       "Q 1356 3103 1142 2972 \n",
       "Q 928 2841 928 2578 \n",
       "Q 928 2378 1081 2264 \n",
       "Q 1234 2150 1697 2047 \n",
       "L 1894 2003 \n",
       "Q 2506 1872 2764 1633 \n",
       "Q 3022 1394 3022 966 \n",
       "Q 3022 478 2636 193 \n",
       "Q 2250 -91 1575 -91 \n",
       "Q 1294 -91 989 -36 \n",
       "Q 684 19 347 128 \n",
       "L 347 722 \n",
       "Q 666 556 975 473 \n",
       "Q 1284 391 1588 391 \n",
       "Q 1994 391 2212 530 \n",
       "Q 2431 669 2431 922 \n",
       "Q 2431 1156 2273 1281 \n",
       "Q 2116 1406 1581 1522 \n",
       "L 1381 1569 \n",
       "Q 847 1681 609 1914 \n",
       "Q 372 2147 372 2553 \n",
       "Q 372 3047 722 3315 \n",
       "Q 1072 3584 1716 3584 \n",
       "Q 2034 3584 2315 3537 \n",
       "Q 2597 3491 2834 3397 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-20\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-2f\" d=\"M 1625 4666 \n",
       "L 2156 4666 \n",
       "L 531 -594 \n",
       "L 0 -594 \n",
       "L 1625 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-63\" d=\"M 3122 3366 \n",
       "L 3122 2828 \n",
       "Q 2878 2963 2633 3030 \n",
       "Q 2388 3097 2138 3097 \n",
       "Q 1578 3097 1268 2742 \n",
       "Q 959 2388 959 1747 \n",
       "Q 959 1106 1268 751 \n",
       "Q 1578 397 2138 397 \n",
       "Q 2388 397 2633 464 \n",
       "Q 2878 531 3122 666 \n",
       "L 3122 134 \n",
       "Q 2881 22 2623 -34 \n",
       "Q 2366 -91 2075 -91 \n",
       "Q 1284 -91 818 406 \n",
       "Q 353 903 353 1747 \n",
       "Q 353 2603 823 3093 \n",
       "Q 1294 3584 2113 3584 \n",
       "Q 2378 3584 2631 3529 \n",
       "Q 2884 3475 3122 3366 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "      </defs>\n",
       "      <use xlink:href=\"#DejaVuSans-53\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-61\" x=\"63.476562\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-6d\" x=\"124.755859\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-70\" x=\"222.167969\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-6c\" x=\"285.644531\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-65\" x=\"313.427734\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-73\" x=\"374.951172\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-20\" x=\"427.050781\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-2f\" x=\"458.837891\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-20\" x=\"492.529297\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-73\" x=\"524.316406\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-65\" x=\"576.416016\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-63\" x=\"637.939453\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_2\">\n",
       "    <g id=\"ytick_1\">\n",
       "     <g id=\"line2d_4\">\n",
       "      <defs>\n",
       "       <path id=\"m51ec6adc06\" d=\"M 0 0 \n",
       "L -3.5 0 \n",
       "\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </defs>\n",
       "      <g>\n",
       "       <use xlink:href=\"#m51ec6adc06\" x=\"91.146875\" y=\"19.45951\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_5\">\n",
       "      <!-- Megatron BERT -->\n",
       "      <g transform=\"translate(7.9375 23.258729)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-4d\" d=\"M 628 4666 \n",
       "L 1569 4666 \n",
       "L 2759 1491 \n",
       "L 3956 4666 \n",
       "L 4897 4666 \n",
       "L 4897 0 \n",
       "L 4281 0 \n",
       "L 4281 4097 \n",
       "L 3078 897 \n",
       "L 2444 897 \n",
       "L 1241 4097 \n",
       "L 1241 0 \n",
       "L 628 0 \n",
       "L 628 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-67\" d=\"M 2906 1791 \n",
       "Q 2906 2416 2648 2759 \n",
       "Q 2391 3103 1925 3103 \n",
       "Q 1463 3103 1205 2759 \n",
       "Q 947 2416 947 1791 \n",
       "Q 947 1169 1205 825 \n",
       "Q 1463 481 1925 481 \n",
       "Q 2391 481 2648 825 \n",
       "Q 2906 1169 2906 1791 \n",
       "z\n",
       "M 3481 434 \n",
       "Q 3481 -459 3084 -895 \n",
       "Q 2688 -1331 1869 -1331 \n",
       "Q 1566 -1331 1297 -1286 \n",
       "Q 1028 -1241 775 -1147 \n",
       "L 775 -588 \n",
       "Q 1028 -725 1275 -790 \n",
       "Q 1522 -856 1778 -856 \n",
       "Q 2344 -856 2625 -561 \n",
       "Q 2906 -266 2906 331 \n",
       "L 2906 616 \n",
       "Q 2728 306 2450 153 \n",
       "Q 2172 0 1784 0 \n",
       "Q 1141 0 747 490 \n",
       "Q 353 981 353 1791 \n",
       "Q 353 2603 747 3093 \n",
       "Q 1141 3584 1784 3584 \n",
       "Q 2172 3584 2450 3431 \n",
       "Q 2728 3278 2906 2969 \n",
       "L 2906 3500 \n",
       "L 3481 3500 \n",
       "L 3481 434 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-74\" d=\"M 1172 4494 \n",
       "L 1172 3500 \n",
       "L 2356 3500 \n",
       "L 2356 3053 \n",
       "L 1172 3053 \n",
       "L 1172 1153 \n",
       "Q 1172 725 1289 603 \n",
       "Q 1406 481 1766 481 \n",
       "L 2356 481 \n",
       "L 2356 0 \n",
       "L 1766 0 \n",
       "Q 1100 0 847 248 \n",
       "Q 594 497 594 1153 \n",
       "L 594 3053 \n",
       "L 172 3053 \n",
       "L 172 3500 \n",
       "L 594 3500 \n",
       "L 594 4494 \n",
       "L 1172 4494 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-72\" d=\"M 2631 2963 \n",
       "Q 2534 3019 2420 3045 \n",
       "Q 2306 3072 2169 3072 \n",
       "Q 1681 3072 1420 2755 \n",
       "Q 1159 2438 1159 1844 \n",
       "L 1159 0 \n",
       "L 581 0 \n",
       "L 581 3500 \n",
       "L 1159 3500 \n",
       "L 1159 2956 \n",
       "Q 1341 3275 1631 3429 \n",
       "Q 1922 3584 2338 3584 \n",
       "Q 2397 3584 2469 3576 \n",
       "Q 2541 3569 2628 3553 \n",
       "L 2631 2963 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-6f\" d=\"M 1959 3097 \n",
       "Q 1497 3097 1228 2736 \n",
       "Q 959 2375 959 1747 \n",
       "Q 959 1119 1226 758 \n",
       "Q 1494 397 1959 397 \n",
       "Q 2419 397 2687 759 \n",
       "Q 2956 1122 2956 1747 \n",
       "Q 2956 2369 2687 2733 \n",
       "Q 2419 3097 1959 3097 \n",
       "z\n",
       "M 1959 3584 \n",
       "Q 2709 3584 3137 3096 \n",
       "Q 3566 2609 3566 1747 \n",
       "Q 3566 888 3137 398 \n",
       "Q 2709 -91 1959 -91 \n",
       "Q 1206 -91 779 398 \n",
       "Q 353 888 353 1747 \n",
       "Q 353 2609 779 3096 \n",
       "Q 1206 3584 1959 3584 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-6e\" d=\"M 3513 2113 \n",
       "L 3513 0 \n",
       "L 2938 0 \n",
       "L 2938 2094 \n",
       "Q 2938 2591 2744 2837 \n",
       "Q 2550 3084 2163 3084 \n",
       "Q 1697 3084 1428 2787 \n",
       "Q 1159 2491 1159 1978 \n",
       "L 1159 0 \n",
       "L 581 0 \n",
       "L 581 3500 \n",
       "L 1159 3500 \n",
       "L 1159 2956 \n",
       "Q 1366 3272 1645 3428 \n",
       "Q 1925 3584 2291 3584 \n",
       "Q 2894 3584 3203 3211 \n",
       "Q 3513 2838 3513 2113 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-42\" d=\"M 1259 2228 \n",
       "L 1259 519 \n",
       "L 2272 519 \n",
       "Q 2781 519 3026 730 \n",
       "Q 3272 941 3272 1375 \n",
       "Q 3272 1813 3026 2020 \n",
       "Q 2781 2228 2272 2228 \n",
       "L 1259 2228 \n",
       "z\n",
       "M 1259 4147 \n",
       "L 1259 2741 \n",
       "L 2194 2741 \n",
       "Q 2656 2741 2882 2914 \n",
       "Q 3109 3088 3109 3444 \n",
       "Q 3109 3797 2882 3972 \n",
       "Q 2656 4147 2194 4147 \n",
       "L 1259 4147 \n",
       "z\n",
       "M 628 4666 \n",
       "L 2241 4666 \n",
       "Q 2963 4666 3353 4366 \n",
       "Q 3744 4066 3744 3513 \n",
       "Q 3744 3084 3544 2831 \n",
       "Q 3344 2578 2956 2516 \n",
       "Q 3422 2416 3680 2098 \n",
       "Q 3938 1781 3938 1306 \n",
       "Q 3938 681 3513 340 \n",
       "Q 3088 0 2303 0 \n",
       "L 628 0 \n",
       "L 628 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-45\" d=\"M 628 4666 \n",
       "L 3578 4666 \n",
       "L 3578 4134 \n",
       "L 1259 4134 \n",
       "L 1259 2753 \n",
       "L 3481 2753 \n",
       "L 3481 2222 \n",
       "L 1259 2222 \n",
       "L 1259 531 \n",
       "L 3634 531 \n",
       "L 3634 0 \n",
       "L 628 0 \n",
       "L 628 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-52\" d=\"M 2841 2188 \n",
       "Q 3044 2119 3236 1894 \n",
       "Q 3428 1669 3622 1275 \n",
       "L 4263 0 \n",
       "L 3584 0 \n",
       "L 2988 1197 \n",
       "Q 2756 1666 2539 1819 \n",
       "Q 2322 1972 1947 1972 \n",
       "L 1259 1972 \n",
       "L 1259 0 \n",
       "L 628 0 \n",
       "L 628 4666 \n",
       "L 2053 4666 \n",
       "Q 2853 4666 3247 4331 \n",
       "Q 3641 3997 3641 3322 \n",
       "Q 3641 2881 3436 2590 \n",
       "Q 3231 2300 2841 2188 \n",
       "z\n",
       "M 1259 4147 \n",
       "L 1259 2491 \n",
       "L 2053 2491 \n",
       "Q 2509 2491 2742 2702 \n",
       "Q 2975 2913 2975 3322 \n",
       "Q 2975 3731 2742 3939 \n",
       "Q 2509 4147 2053 4147 \n",
       "L 1259 4147 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-54\" d=\"M -19 4666 \n",
       "L 3928 4666 \n",
       "L 3928 4134 \n",
       "L 2272 4134 \n",
       "L 2272 0 \n",
       "L 1638 0 \n",
       "L 1638 4134 \n",
       "L -19 4134 \n",
       "L -19 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-4d\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-65\" x=\"86.279297\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-67\" x=\"147.802734\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-61\" x=\"211.279297\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-74\" x=\"272.558594\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-72\" x=\"311.767578\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-6f\" x=\"350.630859\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-6e\" x=\"411.8125\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-20\" x=\"475.191406\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-42\" x=\"506.978516\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-45\" x=\"575.582031\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-52\" x=\"638.765625\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-54\" x=\"700.998047\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_2\">\n",
       "     <g id=\"line2d_5\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m51ec6adc06\" x=\"91.146875\" y=\"47.97\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_6\">\n",
       "      <!-- HF GPT2 -->\n",
       "      <g transform=\"translate(41.448438 51.769219)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-48\" d=\"M 628 4666 \n",
       "L 1259 4666 \n",
       "L 1259 2753 \n",
       "L 3553 2753 \n",
       "L 3553 4666 \n",
       "L 4184 4666 \n",
       "L 4184 0 \n",
       "L 3553 0 \n",
       "L 3553 2222 \n",
       "L 1259 2222 \n",
       "L 1259 0 \n",
       "L 628 0 \n",
       "L 628 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-46\" d=\"M 628 4666 \n",
       "L 3309 4666 \n",
       "L 3309 4134 \n",
       "L 1259 4134 \n",
       "L 1259 2759 \n",
       "L 3109 2759 \n",
       "L 3109 2228 \n",
       "L 1259 2228 \n",
       "L 1259 0 \n",
       "L 628 0 \n",
       "L 628 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-47\" d=\"M 3809 666 \n",
       "L 3809 1919 \n",
       "L 2778 1919 \n",
       "L 2778 2438 \n",
       "L 4434 2438 \n",
       "L 4434 434 \n",
       "Q 4069 175 3628 42 \n",
       "Q 3188 -91 2688 -91 \n",
       "Q 1594 -91 976 548 \n",
       "Q 359 1188 359 2328 \n",
       "Q 359 3472 976 4111 \n",
       "Q 1594 4750 2688 4750 \n",
       "Q 3144 4750 3555 4637 \n",
       "Q 3966 4525 4313 4306 \n",
       "L 4313 3634 \n",
       "Q 3963 3931 3569 4081 \n",
       "Q 3175 4231 2741 4231 \n",
       "Q 1884 4231 1454 3753 \n",
       "Q 1025 3275 1025 2328 \n",
       "Q 1025 1384 1454 906 \n",
       "Q 1884 428 2741 428 \n",
       "Q 3075 428 3337 486 \n",
       "Q 3600 544 3809 666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-50\" d=\"M 1259 4147 \n",
       "L 1259 2394 \n",
       "L 2053 2394 \n",
       "Q 2494 2394 2734 2622 \n",
       "Q 2975 2850 2975 3272 \n",
       "Q 2975 3691 2734 3919 \n",
       "Q 2494 4147 2053 4147 \n",
       "L 1259 4147 \n",
       "z\n",
       "M 628 4666 \n",
       "L 2053 4666 \n",
       "Q 2838 4666 3239 4311 \n",
       "Q 3641 3956 3641 3272 \n",
       "Q 3641 2581 3239 2228 \n",
       "Q 2838 1875 2053 1875 \n",
       "L 1259 1875 \n",
       "L 1259 0 \n",
       "L 628 0 \n",
       "L 628 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-48\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-46\" x=\"75.195312\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-20\" x=\"132.714844\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-47\" x=\"164.501953\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-50\" x=\"241.992188\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-54\" x=\"302.294922\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-32\" x=\"363.378906\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_3\">\n",
       "     <g id=\"line2d_6\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m51ec6adc06\" x=\"91.146875\" y=\"76.48049\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_7\">\n",
       "      <!-- Megatron GPT2 -->\n",
       "      <g transform=\"translate(7.2 80.279708)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-4d\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-65\" x=\"86.279297\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-67\" x=\"147.802734\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-61\" x=\"211.279297\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-74\" x=\"272.558594\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-72\" x=\"311.767578\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-6f\" x=\"350.630859\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-6e\" x=\"411.8125\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-20\" x=\"475.191406\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-47\" x=\"506.978516\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-50\" x=\"584.46875\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-54\" x=\"644.771484\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-32\" x=\"705.855469\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"patch_6\">\n",
       "    <path d=\"M 91.146875 88.74 \n",
       "L 91.146875 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_7\">\n",
       "    <path d=\"M 238.852757 88.74 \n",
       "L 238.852757 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_8\">\n",
       "    <path d=\"M 91.146875 88.74 \n",
       "L 238.852757 88.74 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_9\">\n",
       "    <path d=\"M 91.146875 7.2 \n",
       "L 238.852757 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "  </g>\n",
       "  <g id=\"axes_2\">\n",
       "   <g id=\"patch_10\">\n",
       "    <path d=\"M 268.393934 88.74 \n",
       "L 416.099816 88.74 \n",
       "L 416.099816 7.2 \n",
       "L 268.393934 7.2 \n",
       "z\n",
       "\" style=\"fill: #ffffff\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_11\">\n",
       "    <path d=\"M 268.393934 10.906364 \n",
       "L 409.066203 10.906364 \n",
       "L 409.066203 28.012657 \n",
       "L 268.393934 28.012657 \n",
       "z\n",
       "\" clip-path=\"url(#p6d4e346104)\" style=\"fill: #e41a1c\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_12\">\n",
       "    <path d=\"M 268.393934 39.416853 \n",
       "L 328.630478 39.416853 \n",
       "L 328.630478 56.523147 \n",
       "L 268.393934 56.523147 \n",
       "z\n",
       "\" clip-path=\"url(#p6d4e346104)\" style=\"fill: #377eb8\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_13\">\n",
       "    <path d=\"M 268.393934 67.927343 \n",
       "L 401.415582 67.927343 \n",
       "L 401.415582 85.033636 \n",
       "L 268.393934 85.033636 \n",
       "z\n",
       "\" clip-path=\"url(#p6d4e346104)\" style=\"fill: #4daf4a\"/>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_3\">\n",
       "    <g id=\"xtick_4\">\n",
       "     <g id=\"line2d_7\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m4c59257877\" x=\"268.393934\" y=\"88.74\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_8\">\n",
       "      <!-- 0 -->\n",
       "      <g transform=\"translate(265.212684 103.338438)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_5\">\n",
       "     <g id=\"line2d_8\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m4c59257877\" x=\"331.493143\" y=\"88.74\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_9\">\n",
       "      <!-- 20 -->\n",
       "      <g transform=\"translate(325.130643 103.338438)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-32\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_6\">\n",
       "     <g id=\"line2d_9\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m4c59257877\" x=\"394.592352\" y=\"88.74\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_10\">\n",
       "      <!-- 40 -->\n",
       "      <g transform=\"translate(388.229852 103.338438)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-34\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"text_11\">\n",
       "     <!-- per GPU TFLOPS -->\n",
       "     <g transform=\"translate(301.554688 117.016563)scale(0.1 -0.1)\">\n",
       "      <defs>\n",
       "       <path id=\"DejaVuSans-55\" d=\"M 556 4666 \n",
       "L 1191 4666 \n",
       "L 1191 1831 \n",
       "Q 1191 1081 1462 751 \n",
       "Q 1734 422 2344 422 \n",
       "Q 2950 422 3222 751 \n",
       "Q 3494 1081 3494 1831 \n",
       "L 3494 4666 \n",
       "L 4128 4666 \n",
       "L 4128 1753 \n",
       "Q 4128 841 3676 375 \n",
       "Q 3225 -91 2344 -91 \n",
       "Q 1459 -91 1007 375 \n",
       "Q 556 841 556 1753 \n",
       "L 556 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-4c\" d=\"M 628 4666 \n",
       "L 1259 4666 \n",
       "L 1259 531 \n",
       "L 3531 531 \n",
       "L 3531 0 \n",
       "L 628 0 \n",
       "L 628 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-4f\" d=\"M 2522 4238 \n",
       "Q 1834 4238 1429 3725 \n",
       "Q 1025 3213 1025 2328 \n",
       "Q 1025 1447 1429 934 \n",
       "Q 1834 422 2522 422 \n",
       "Q 3209 422 3611 934 \n",
       "Q 4013 1447 4013 2328 \n",
       "Q 4013 3213 3611 3725 \n",
       "Q 3209 4238 2522 4238 \n",
       "z\n",
       "M 2522 4750 \n",
       "Q 3503 4750 4090 4092 \n",
       "Q 4678 3434 4678 2328 \n",
       "Q 4678 1225 4090 567 \n",
       "Q 3503 -91 2522 -91 \n",
       "Q 1538 -91 948 565 \n",
       "Q 359 1222 359 2328 \n",
       "Q 359 3434 948 4092 \n",
       "Q 1538 4750 2522 4750 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "      </defs>\n",
       "      <use xlink:href=\"#DejaVuSans-70\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-65\" x=\"63.476562\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-72\" x=\"125\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-20\" x=\"166.113281\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-47\" x=\"197.900391\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-50\" x=\"275.390625\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-55\" x=\"335.693359\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-20\" x=\"408.886719\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-54\" x=\"440.673828\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-46\" x=\"501.757812\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-4c\" x=\"559.277344\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-4f\" x=\"611.365234\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-50\" x=\"690.076172\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-53\" x=\"750.378906\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_4\">\n",
       "    <g id=\"ytick_4\">\n",
       "     <g id=\"line2d_10\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m51ec6adc06\" x=\"268.393934\" y=\"19.45951\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_5\">\n",
       "     <g id=\"line2d_11\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m51ec6adc06\" x=\"268.393934\" y=\"47.97\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_6\">\n",
       "     <g id=\"line2d_12\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m51ec6adc06\" x=\"268.393934\" y=\"76.48049\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"patch_14\">\n",
       "    <path d=\"M 268.393934 88.74 \n",
       "L 268.393934 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_15\">\n",
       "    <path d=\"M 416.099816 88.74 \n",
       "L 416.099816 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_16\">\n",
       "    <path d=\"M 268.393934 88.74 \n",
       "L 416.099816 88.74 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_17\">\n",
       "    <path d=\"M 268.393934 7.2 \n",
       "L 416.099816 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "  </g>\n",
       "  <g id=\"axes_3\">\n",
       "   <g id=\"patch_18\">\n",
       "    <path d=\"M 445.640993 88.74 \n",
       "L 593.346875 88.74 \n",
       "L 593.346875 7.2 \n",
       "L 445.640993 7.2 \n",
       "z\n",
       "\" style=\"fill: #ffffff\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_19\">\n",
       "    <path d=\"M 445.640993 10.906364 \n",
       "L 586.313262 10.906364 \n",
       "L 586.313262 28.012657 \n",
       "L 445.640993 28.012657 \n",
       "z\n",
       "\" clip-path=\"url(#p022585bbc1)\" style=\"fill: #e41a1c\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_20\">\n",
       "    <path d=\"M 445.640993 39.416853 \n",
       "L 569.425538 39.416853 \n",
       "L 569.425538 56.523147 \n",
       "L 445.640993 56.523147 \n",
       "z\n",
       "\" clip-path=\"url(#p022585bbc1)\" style=\"fill: #377eb8\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_21\">\n",
       "    <path d=\"M 445.640993 67.927343 \n",
       "L 584.975718 67.927343 \n",
       "L 584.975718 85.033636 \n",
       "L 445.640993 85.033636 \n",
       "z\n",
       "\" clip-path=\"url(#p022585bbc1)\" style=\"fill: #4daf4a\"/>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_5\">\n",
       "    <g id=\"xtick_7\">\n",
       "     <g id=\"line2d_13\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m4c59257877\" x=\"445.640993\" y=\"88.74\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_12\">\n",
       "      <!-- 0 -->\n",
       "      <g transform=\"translate(442.459743 103.338438)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_8\">\n",
       "     <g id=\"line2d_14\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m4c59257877\" x=\"510.645017\" y=\"88.74\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_13\">\n",
       "      <!-- 10 -->\n",
       "      <g transform=\"translate(504.282517 103.338438)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-31\" d=\"M 794 531 \n",
       "L 1825 531 \n",
       "L 1825 4091 \n",
       "L 703 3866 \n",
       "L 703 4441 \n",
       "L 1819 4666 \n",
       "L 2450 4666 \n",
       "L 2450 531 \n",
       "L 3481 531 \n",
       "L 3481 0 \n",
       "L 794 0 \n",
       "L 794 531 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-31\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_9\">\n",
       "     <g id=\"line2d_15\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m4c59257877\" x=\"575.64904\" y=\"88.74\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_14\">\n",
       "      <!-- 20 -->\n",
       "      <g transform=\"translate(569.28654 103.338438)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-32\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"text_15\">\n",
       "     <!-- per GPU memory (GB) -->\n",
       "     <g transform=\"translate(463.774403 117.016563)scale(0.1 -0.1)\">\n",
       "      <defs>\n",
       "       <path id=\"DejaVuSans-79\" d=\"M 2059 -325 \n",
       "Q 1816 -950 1584 -1140 \n",
       "Q 1353 -1331 966 -1331 \n",
       "L 506 -1331 \n",
       "L 506 -850 \n",
       "L 844 -850 \n",
       "Q 1081 -850 1212 -737 \n",
       "Q 1344 -625 1503 -206 \n",
       "L 1606 56 \n",
       "L 191 3500 \n",
       "L 800 3500 \n",
       "L 1894 763 \n",
       "L 2988 3500 \n",
       "L 3597 3500 \n",
       "L 2059 -325 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-28\" d=\"M 1984 4856 \n",
       "Q 1566 4138 1362 3434 \n",
       "Q 1159 2731 1159 2009 \n",
       "Q 1159 1288 1364 580 \n",
       "Q 1569 -128 1984 -844 \n",
       "L 1484 -844 \n",
       "Q 1016 -109 783 600 \n",
       "Q 550 1309 550 2009 \n",
       "Q 550 2706 781 3412 \n",
       "Q 1013 4119 1484 4856 \n",
       "L 1984 4856 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-29\" d=\"M 513 4856 \n",
       "L 1013 4856 \n",
       "Q 1481 4119 1714 3412 \n",
       "Q 1947 2706 1947 2009 \n",
       "Q 1947 1309 1714 600 \n",
       "Q 1481 -109 1013 -844 \n",
       "L 513 -844 \n",
       "Q 928 -128 1133 580 \n",
       "Q 1338 1288 1338 2009 \n",
       "Q 1338 2731 1133 3434 \n",
       "Q 928 4138 513 4856 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "      </defs>\n",
       "      <use xlink:href=\"#DejaVuSans-70\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-65\" x=\"63.476562\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-72\" x=\"125\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-20\" x=\"166.113281\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-47\" x=\"197.900391\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-50\" x=\"275.390625\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-55\" x=\"335.693359\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-20\" x=\"408.886719\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-6d\" x=\"440.673828\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-65\" x=\"538.085938\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-6d\" x=\"599.609375\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-6f\" x=\"697.021484\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-72\" x=\"758.203125\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-79\" x=\"799.316406\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-20\" x=\"858.496094\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-28\" x=\"890.283203\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-47\" x=\"929.296875\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-42\" x=\"1006.787109\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-29\" x=\"1075.390625\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_6\">\n",
       "    <g id=\"ytick_7\">\n",
       "     <g id=\"line2d_16\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m51ec6adc06\" x=\"445.640993\" y=\"19.45951\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_8\">\n",
       "     <g id=\"line2d_17\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m51ec6adc06\" x=\"445.640993\" y=\"47.97\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_9\">\n",
       "     <g id=\"line2d_18\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m51ec6adc06\" x=\"445.640993\" y=\"76.48049\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"patch_22\">\n",
       "    <path d=\"M 445.640993 88.74 \n",
       "L 445.640993 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_23\">\n",
       "    <path d=\"M 593.346875 88.74 \n",
       "L 593.346875 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_24\">\n",
       "    <path d=\"M 445.640993 88.74 \n",
       "L 593.346875 88.74 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_25\">\n",
       "    <path d=\"M 445.640993 7.2 \n",
       "L 593.346875 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "  </g>\n",
       " </g>\n",
       " <defs>\n",
       "  <clipPath id=\"pb188ee7e5e\">\n",
       "   <rect x=\"91.146875\" y=\"7.2\" width=\"147.705882\" height=\"81.54\"/>\n",
       "  </clipPath>\n",
       "  <clipPath id=\"p6d4e346104\">\n",
       "   <rect x=\"268.393934\" y=\"7.2\" width=\"147.705882\" height=\"81.54\"/>\n",
       "  </clipPath>\n",
       "  <clipPath id=\"p022585bbc1\">\n",
       "   <rect x=\"445.640993\" y=\"7.2\" width=\"147.705882\" height=\"81.54\"/>\n",
       "  </clipPath>\n",
       " </defs>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<Figure size 648x108 with 3 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "mega_gpt2 = megatron_gpt(Exp(\"Megatron GPT2\", \"gpt2-medium\", 5, bf16=True))\n",
    "compare([mega_bert, hf_gpt2, mega_gpt2])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c8eef1b8",
   "metadata": {},
   "source": [
    "## Multiple GPUs with Data Parallelism\n",
    "\n",
    "Let's first check how GPUs are connected. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "c2d459c0",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\t\u001b[4mGPU0\tGPU1\tCPU Affinity\tNUMA Affinity\u001b[0m\n",
      "GPU0\t X \tNV4\t0-23\t\tN/A\n",
      "GPU1\tNV4\t X \t0-23\t\tN/A\n",
      "\n",
      "Legend:\n",
      "\n",
      "  X    = Self\n",
      "  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)\n",
      "  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node\n",
      "  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)\n",
      "  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)\n",
      "  PIX  = Connection traversing at most a single PCIe bridge\n",
      "  NV#  = Connection traversing a bonded set of # NVLinks\n"
     ]
    }
   ],
   "source": [
    "!nvidia-smi topo -m"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5fdce621",
   "metadata": {},
   "source": [
    "You can use we have two GPUs connected by NVLinks. Besides, they are also connected through PCIe 4.0 x8.\n",
    "\n",
    "You can use the [p2pBandwidthLatencyTest](https://github.com/NVIDIA/cuda-samples/tree/master/Samples/5_Domain_Specific/p2pBandwidthLatencyTest) tool to get a rough estimation of the bandwidth. Here are our results:\n",
    "\n",
    "```bash\n",
    "Unidirectional P2P=Disabled Bandwidth Matrix (GB/s)\n",
    "   D\\D     0      1\n",
    "     0 891.84   6.23\n",
    "     1   6.23 893.88\n",
    "Unidirectional P2P=Enabled Bandwidth (P2P Writes) Matrix (GB/s)\n",
    "   D\\D     0      1\n",
    "     0 883.27  52.77\n",
    "     1  52.89 894.39\n",
    "Bidirectional P2P=Disabled Bandwidth Matrix (GB/s)\n",
    "   D\\D     0      1\n",
    "     0 884.77   9.20\n",
    "     1   9.24 900.06\n",
    "Bidirectional P2P=Enabled Bandwidth Matrix (GB/s)\n",
    "   D\\D     0      1\n",
    "     0 885.52 101.36\n",
    "     1 101.52 900.84\n",
    "```\n",
    "\n",
    "Now let's run GPT-2 with Megatron on two GPUs, which use data parallelism in default. (You can replace with `hf_gpt` as well.) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "9e133a6c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Time breakdown\t\t: forward+backward 0.82, communication 0.10, optimizer 0.07\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       "  \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<svg xmlns:xlink=\"http://www.w3.org/1999/xlink\" width=\"642.75pt\" height=\"99.11625pt\" viewBox=\"0 0 642.75 99.11625\" xmlns=\"http://www.w3.org/2000/svg\" version=\"1.1\">\n",
       " <metadata>\n",
       "  <rdf:RDF xmlns:dc=\"http://purl.org/dc/elements/1.1/\" xmlns:cc=\"http://creativecommons.org/ns#\" xmlns:rdf=\"http://www.w3.org/1999/02/22-rdf-syntax-ns#\">\n",
       "   <cc:Work>\n",
       "    <dc:type rdf:resource=\"http://purl.org/dc/dcmitype/StillImage\"/>\n",
       "    <dc:date>2022-09-02T19:44:57.796402</dc:date>\n",
       "    <dc:format>image/svg+xml</dc:format>\n",
       "    <dc:creator>\n",
       "     <cc:Agent>\n",
       "      <dc:title>Matplotlib v3.5.2, https://matplotlib.org/</dc:title>\n",
       "     </cc:Agent>\n",
       "    </dc:creator>\n",
       "   </cc:Work>\n",
       "  </rdf:RDF>\n",
       " </metadata>\n",
       " <defs>\n",
       "  <style type=\"text/css\">*{stroke-linejoin: round; stroke-linecap: butt}</style>\n",
       " </defs>\n",
       " <g id=\"figure_1\">\n",
       "  <g id=\"patch_1\">\n",
       "   <path d=\"M 0 99.11625 \n",
       "L 642.75 99.11625 \n",
       "L 642.75 0 \n",
       "L 0 0 \n",
       "L 0 99.11625 \n",
       "z\n",
       "\" style=\"fill: none\"/>\n",
       "  </g>\n",
       "  <g id=\"axes_1\">\n",
       "   <g id=\"patch_2\">\n",
       "    <path d=\"M 133.35 61.56 \n",
       "L 281.055882 61.56 \n",
       "L 281.055882 7.2 \n",
       "L 133.35 7.2 \n",
       "z\n",
       "\" style=\"fill: #ffffff\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_3\">\n",
       "    <path d=\"M 133.35 9.670909 \n",
       "L 211.264841 9.670909 \n",
       "L 211.264841 28.202727 \n",
       "L 133.35 28.202727 \n",
       "z\n",
       "\" clip-path=\"url(#p88919dde73)\" style=\"fill: #e41a1c\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_4\">\n",
       "    <path d=\"M 133.35 40.557273 \n",
       "L 274.022269 40.557273 \n",
       "L 274.022269 59.089091 \n",
       "L 133.35 59.089091 \n",
       "z\n",
       "\" clip-path=\"url(#p88919dde73)\" style=\"fill: #377eb8\"/>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_1\">\n",
       "    <g id=\"xtick_1\">\n",
       "     <g id=\"line2d_1\">\n",
       "      <defs>\n",
       "       <path id=\"mc3a958a983\" d=\"M 0 0 \n",
       "L 0 3.5 \n",
       "\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </defs>\n",
       "      <g>\n",
       "       <use xlink:href=\"#mc3a958a983\" x=\"133.35\" y=\"61.56\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_1\">\n",
       "      <!-- 0 -->\n",
       "      <g transform=\"translate(130.16875 76.158437)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-30\" d=\"M 2034 4250 \n",
       "Q 1547 4250 1301 3770 \n",
       "Q 1056 3291 1056 2328 \n",
       "Q 1056 1369 1301 889 \n",
       "Q 1547 409 2034 409 \n",
       "Q 2525 409 2770 889 \n",
       "Q 3016 1369 3016 2328 \n",
       "Q 3016 3291 2770 3770 \n",
       "Q 2525 4250 2034 4250 \n",
       "z\n",
       "M 2034 4750 \n",
       "Q 2819 4750 3233 4129 \n",
       "Q 3647 3509 3647 2328 \n",
       "Q 3647 1150 3233 529 \n",
       "Q 2819 -91 2034 -91 \n",
       "Q 1250 -91 836 529 \n",
       "Q 422 1150 422 2328 \n",
       "Q 422 3509 836 4129 \n",
       "Q 1250 4750 2034 4750 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_2\">\n",
       "     <g id=\"line2d_2\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#mc3a958a983\" x=\"179.195092\" y=\"61.56\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_2\">\n",
       "      <!-- 10 -->\n",
       "      <g transform=\"translate(172.832592 76.158437)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-31\" d=\"M 794 531 \n",
       "L 1825 531 \n",
       "L 1825 4091 \n",
       "L 703 3866 \n",
       "L 703 4441 \n",
       "L 1819 4666 \n",
       "L 2450 4666 \n",
       "L 2450 531 \n",
       "L 3481 531 \n",
       "L 3481 0 \n",
       "L 794 0 \n",
       "L 794 531 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-31\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_3\">\n",
       "     <g id=\"line2d_3\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#mc3a958a983\" x=\"225.040185\" y=\"61.56\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_3\">\n",
       "      <!-- 20 -->\n",
       "      <g transform=\"translate(218.677685 76.158437)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-32\" d=\"M 1228 531 \n",
       "L 3431 531 \n",
       "L 3431 0 \n",
       "L 469 0 \n",
       "L 469 531 \n",
       "Q 828 903 1448 1529 \n",
       "Q 2069 2156 2228 2338 \n",
       "Q 2531 2678 2651 2914 \n",
       "Q 2772 3150 2772 3378 \n",
       "Q 2772 3750 2511 3984 \n",
       "Q 2250 4219 1831 4219 \n",
       "Q 1534 4219 1204 4116 \n",
       "Q 875 4013 500 3803 \n",
       "L 500 4441 \n",
       "Q 881 4594 1212 4672 \n",
       "Q 1544 4750 1819 4750 \n",
       "Q 2544 4750 2975 4387 \n",
       "Q 3406 4025 3406 3419 \n",
       "Q 3406 3131 3298 2873 \n",
       "Q 3191 2616 2906 2266 \n",
       "Q 2828 2175 2409 1742 \n",
       "Q 1991 1309 1228 531 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-32\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_4\">\n",
       "     <g id=\"line2d_4\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#mc3a958a983\" x=\"270.885277\" y=\"61.56\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_4\">\n",
       "      <!-- 30 -->\n",
       "      <g transform=\"translate(264.522777 76.158437)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-33\" d=\"M 2597 2516 \n",
       "Q 3050 2419 3304 2112 \n",
       "Q 3559 1806 3559 1356 \n",
       "Q 3559 666 3084 287 \n",
       "Q 2609 -91 1734 -91 \n",
       "Q 1441 -91 1130 -33 \n",
       "Q 819 25 488 141 \n",
       "L 488 750 \n",
       "Q 750 597 1062 519 \n",
       "Q 1375 441 1716 441 \n",
       "Q 2309 441 2620 675 \n",
       "Q 2931 909 2931 1356 \n",
       "Q 2931 1769 2642 2001 \n",
       "Q 2353 2234 1838 2234 \n",
       "L 1294 2234 \n",
       "L 1294 2753 \n",
       "L 1863 2753 \n",
       "Q 2328 2753 2575 2939 \n",
       "Q 2822 3125 2822 3475 \n",
       "Q 2822 3834 2567 4026 \n",
       "Q 2313 4219 1838 4219 \n",
       "Q 1578 4219 1281 4162 \n",
       "Q 984 4106 628 3988 \n",
       "L 628 4550 \n",
       "Q 988 4650 1302 4700 \n",
       "Q 1616 4750 1894 4750 \n",
       "Q 2613 4750 3031 4423 \n",
       "Q 3450 4097 3450 3541 \n",
       "Q 3450 3153 3228 2886 \n",
       "Q 3006 2619 2597 2516 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-33\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"text_5\">\n",
       "     <!-- Samples / sec -->\n",
       "     <g transform=\"translate(172.556847 89.836562)scale(0.1 -0.1)\">\n",
       "      <defs>\n",
       "       <path id=\"DejaVuSans-53\" d=\"M 3425 4513 \n",
       "L 3425 3897 \n",
       "Q 3066 4069 2747 4153 \n",
       "Q 2428 4238 2131 4238 \n",
       "Q 1616 4238 1336 4038 \n",
       "Q 1056 3838 1056 3469 \n",
       "Q 1056 3159 1242 3001 \n",
       "Q 1428 2844 1947 2747 \n",
       "L 2328 2669 \n",
       "Q 3034 2534 3370 2195 \n",
       "Q 3706 1856 3706 1288 \n",
       "Q 3706 609 3251 259 \n",
       "Q 2797 -91 1919 -91 \n",
       "Q 1588 -91 1214 -16 \n",
       "Q 841 59 441 206 \n",
       "L 441 856 \n",
       "Q 825 641 1194 531 \n",
       "Q 1563 422 1919 422 \n",
       "Q 2459 422 2753 634 \n",
       "Q 3047 847 3047 1241 \n",
       "Q 3047 1584 2836 1778 \n",
       "Q 2625 1972 2144 2069 \n",
       "L 1759 2144 \n",
       "Q 1053 2284 737 2584 \n",
       "Q 422 2884 422 3419 \n",
       "Q 422 4038 858 4394 \n",
       "Q 1294 4750 2059 4750 \n",
       "Q 2388 4750 2728 4690 \n",
       "Q 3069 4631 3425 4513 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-61\" d=\"M 2194 1759 \n",
       "Q 1497 1759 1228 1600 \n",
       "Q 959 1441 959 1056 \n",
       "Q 959 750 1161 570 \n",
       "Q 1363 391 1709 391 \n",
       "Q 2188 391 2477 730 \n",
       "Q 2766 1069 2766 1631 \n",
       "L 2766 1759 \n",
       "L 2194 1759 \n",
       "z\n",
       "M 3341 1997 \n",
       "L 3341 0 \n",
       "L 2766 0 \n",
       "L 2766 531 \n",
       "Q 2569 213 2275 61 \n",
       "Q 1981 -91 1556 -91 \n",
       "Q 1019 -91 701 211 \n",
       "Q 384 513 384 1019 \n",
       "Q 384 1609 779 1909 \n",
       "Q 1175 2209 1959 2209 \n",
       "L 2766 2209 \n",
       "L 2766 2266 \n",
       "Q 2766 2663 2505 2880 \n",
       "Q 2244 3097 1772 3097 \n",
       "Q 1472 3097 1187 3025 \n",
       "Q 903 2953 641 2809 \n",
       "L 641 3341 \n",
       "Q 956 3463 1253 3523 \n",
       "Q 1550 3584 1831 3584 \n",
       "Q 2591 3584 2966 3190 \n",
       "Q 3341 2797 3341 1997 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-6d\" d=\"M 3328 2828 \n",
       "Q 3544 3216 3844 3400 \n",
       "Q 4144 3584 4550 3584 \n",
       "Q 5097 3584 5394 3201 \n",
       "Q 5691 2819 5691 2113 \n",
       "L 5691 0 \n",
       "L 5113 0 \n",
       "L 5113 2094 \n",
       "Q 5113 2597 4934 2840 \n",
       "Q 4756 3084 4391 3084 \n",
       "Q 3944 3084 3684 2787 \n",
       "Q 3425 2491 3425 1978 \n",
       "L 3425 0 \n",
       "L 2847 0 \n",
       "L 2847 2094 \n",
       "Q 2847 2600 2669 2842 \n",
       "Q 2491 3084 2119 3084 \n",
       "Q 1678 3084 1418 2786 \n",
       "Q 1159 2488 1159 1978 \n",
       "L 1159 0 \n",
       "L 581 0 \n",
       "L 581 3500 \n",
       "L 1159 3500 \n",
       "L 1159 2956 \n",
       "Q 1356 3278 1631 3431 \n",
       "Q 1906 3584 2284 3584 \n",
       "Q 2666 3584 2933 3390 \n",
       "Q 3200 3197 3328 2828 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-70\" d=\"M 1159 525 \n",
       "L 1159 -1331 \n",
       "L 581 -1331 \n",
       "L 581 3500 \n",
       "L 1159 3500 \n",
       "L 1159 2969 \n",
       "Q 1341 3281 1617 3432 \n",
       "Q 1894 3584 2278 3584 \n",
       "Q 2916 3584 3314 3078 \n",
       "Q 3713 2572 3713 1747 \n",
       "Q 3713 922 3314 415 \n",
       "Q 2916 -91 2278 -91 \n",
       "Q 1894 -91 1617 61 \n",
       "Q 1341 213 1159 525 \n",
       "z\n",
       "M 3116 1747 \n",
       "Q 3116 2381 2855 2742 \n",
       "Q 2594 3103 2138 3103 \n",
       "Q 1681 3103 1420 2742 \n",
       "Q 1159 2381 1159 1747 \n",
       "Q 1159 1113 1420 752 \n",
       "Q 1681 391 2138 391 \n",
       "Q 2594 391 2855 752 \n",
       "Q 3116 1113 3116 1747 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-6c\" d=\"M 603 4863 \n",
       "L 1178 4863 \n",
       "L 1178 0 \n",
       "L 603 0 \n",
       "L 603 4863 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-65\" d=\"M 3597 1894 \n",
       "L 3597 1613 \n",
       "L 953 1613 \n",
       "Q 991 1019 1311 708 \n",
       "Q 1631 397 2203 397 \n",
       "Q 2534 397 2845 478 \n",
       "Q 3156 559 3463 722 \n",
       "L 3463 178 \n",
       "Q 3153 47 2828 -22 \n",
       "Q 2503 -91 2169 -91 \n",
       "Q 1331 -91 842 396 \n",
       "Q 353 884 353 1716 \n",
       "Q 353 2575 817 3079 \n",
       "Q 1281 3584 2069 3584 \n",
       "Q 2775 3584 3186 3129 \n",
       "Q 3597 2675 3597 1894 \n",
       "z\n",
       "M 3022 2063 \n",
       "Q 3016 2534 2758 2815 \n",
       "Q 2500 3097 2075 3097 \n",
       "Q 1594 3097 1305 2825 \n",
       "Q 1016 2553 972 2059 \n",
       "L 3022 2063 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-73\" d=\"M 2834 3397 \n",
       "L 2834 2853 \n",
       "Q 2591 2978 2328 3040 \n",
       "Q 2066 3103 1784 3103 \n",
       "Q 1356 3103 1142 2972 \n",
       "Q 928 2841 928 2578 \n",
       "Q 928 2378 1081 2264 \n",
       "Q 1234 2150 1697 2047 \n",
       "L 1894 2003 \n",
       "Q 2506 1872 2764 1633 \n",
       "Q 3022 1394 3022 966 \n",
       "Q 3022 478 2636 193 \n",
       "Q 2250 -91 1575 -91 \n",
       "Q 1294 -91 989 -36 \n",
       "Q 684 19 347 128 \n",
       "L 347 722 \n",
       "Q 666 556 975 473 \n",
       "Q 1284 391 1588 391 \n",
       "Q 1994 391 2212 530 \n",
       "Q 2431 669 2431 922 \n",
       "Q 2431 1156 2273 1281 \n",
       "Q 2116 1406 1581 1522 \n",
       "L 1381 1569 \n",
       "Q 847 1681 609 1914 \n",
       "Q 372 2147 372 2553 \n",
       "Q 372 3047 722 3315 \n",
       "Q 1072 3584 1716 3584 \n",
       "Q 2034 3584 2315 3537 \n",
       "Q 2597 3491 2834 3397 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-20\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-2f\" d=\"M 1625 4666 \n",
       "L 2156 4666 \n",
       "L 531 -594 \n",
       "L 0 -594 \n",
       "L 1625 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-63\" d=\"M 3122 3366 \n",
       "L 3122 2828 \n",
       "Q 2878 2963 2633 3030 \n",
       "Q 2388 3097 2138 3097 \n",
       "Q 1578 3097 1268 2742 \n",
       "Q 959 2388 959 1747 \n",
       "Q 959 1106 1268 751 \n",
       "Q 1578 397 2138 397 \n",
       "Q 2388 397 2633 464 \n",
       "Q 2878 531 3122 666 \n",
       "L 3122 134 \n",
       "Q 2881 22 2623 -34 \n",
       "Q 2366 -91 2075 -91 \n",
       "Q 1284 -91 818 406 \n",
       "Q 353 903 353 1747 \n",
       "Q 353 2603 823 3093 \n",
       "Q 1294 3584 2113 3584 \n",
       "Q 2378 3584 2631 3529 \n",
       "Q 2884 3475 3122 3366 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "      </defs>\n",
       "      <use xlink:href=\"#DejaVuSans-53\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-61\" x=\"63.476562\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-6d\" x=\"124.755859\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-70\" x=\"222.167969\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-6c\" x=\"285.644531\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-65\" x=\"313.427734\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-73\" x=\"374.951172\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-20\" x=\"427.050781\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-2f\" x=\"458.837891\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-20\" x=\"492.529297\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-73\" x=\"524.316406\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-65\" x=\"576.416016\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-63\" x=\"637.939453\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_2\">\n",
       "    <g id=\"ytick_1\">\n",
       "     <g id=\"line2d_5\">\n",
       "      <defs>\n",
       "       <path id=\"m5d3f33379c\" d=\"M 0 0 \n",
       "L -3.5 0 \n",
       "\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </defs>\n",
       "      <g>\n",
       "       <use xlink:href=\"#m5d3f33379c\" x=\"133.35\" y=\"18.936818\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_6\">\n",
       "      <!-- Megatron GPT2 -->\n",
       "      <g transform=\"translate(49.403125 22.736037)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-4d\" d=\"M 628 4666 \n",
       "L 1569 4666 \n",
       "L 2759 1491 \n",
       "L 3956 4666 \n",
       "L 4897 4666 \n",
       "L 4897 0 \n",
       "L 4281 0 \n",
       "L 4281 4097 \n",
       "L 3078 897 \n",
       "L 2444 897 \n",
       "L 1241 4097 \n",
       "L 1241 0 \n",
       "L 628 0 \n",
       "L 628 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-67\" d=\"M 2906 1791 \n",
       "Q 2906 2416 2648 2759 \n",
       "Q 2391 3103 1925 3103 \n",
       "Q 1463 3103 1205 2759 \n",
       "Q 947 2416 947 1791 \n",
       "Q 947 1169 1205 825 \n",
       "Q 1463 481 1925 481 \n",
       "Q 2391 481 2648 825 \n",
       "Q 2906 1169 2906 1791 \n",
       "z\n",
       "M 3481 434 \n",
       "Q 3481 -459 3084 -895 \n",
       "Q 2688 -1331 1869 -1331 \n",
       "Q 1566 -1331 1297 -1286 \n",
       "Q 1028 -1241 775 -1147 \n",
       "L 775 -588 \n",
       "Q 1028 -725 1275 -790 \n",
       "Q 1522 -856 1778 -856 \n",
       "Q 2344 -856 2625 -561 \n",
       "Q 2906 -266 2906 331 \n",
       "L 2906 616 \n",
       "Q 2728 306 2450 153 \n",
       "Q 2172 0 1784 0 \n",
       "Q 1141 0 747 490 \n",
       "Q 353 981 353 1791 \n",
       "Q 353 2603 747 3093 \n",
       "Q 1141 3584 1784 3584 \n",
       "Q 2172 3584 2450 3431 \n",
       "Q 2728 3278 2906 2969 \n",
       "L 2906 3500 \n",
       "L 3481 3500 \n",
       "L 3481 434 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-74\" d=\"M 1172 4494 \n",
       "L 1172 3500 \n",
       "L 2356 3500 \n",
       "L 2356 3053 \n",
       "L 1172 3053 \n",
       "L 1172 1153 \n",
       "Q 1172 725 1289 603 \n",
       "Q 1406 481 1766 481 \n",
       "L 2356 481 \n",
       "L 2356 0 \n",
       "L 1766 0 \n",
       "Q 1100 0 847 248 \n",
       "Q 594 497 594 1153 \n",
       "L 594 3053 \n",
       "L 172 3053 \n",
       "L 172 3500 \n",
       "L 594 3500 \n",
       "L 594 4494 \n",
       "L 1172 4494 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-72\" d=\"M 2631 2963 \n",
       "Q 2534 3019 2420 3045 \n",
       "Q 2306 3072 2169 3072 \n",
       "Q 1681 3072 1420 2755 \n",
       "Q 1159 2438 1159 1844 \n",
       "L 1159 0 \n",
       "L 581 0 \n",
       "L 581 3500 \n",
       "L 1159 3500 \n",
       "L 1159 2956 \n",
       "Q 1341 3275 1631 3429 \n",
       "Q 1922 3584 2338 3584 \n",
       "Q 2397 3584 2469 3576 \n",
       "Q 2541 3569 2628 3553 \n",
       "L 2631 2963 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-6f\" d=\"M 1959 3097 \n",
       "Q 1497 3097 1228 2736 \n",
       "Q 959 2375 959 1747 \n",
       "Q 959 1119 1226 758 \n",
       "Q 1494 397 1959 397 \n",
       "Q 2419 397 2687 759 \n",
       "Q 2956 1122 2956 1747 \n",
       "Q 2956 2369 2687 2733 \n",
       "Q 2419 3097 1959 3097 \n",
       "z\n",
       "M 1959 3584 \n",
       "Q 2709 3584 3137 3096 \n",
       "Q 3566 2609 3566 1747 \n",
       "Q 3566 888 3137 398 \n",
       "Q 2709 -91 1959 -91 \n",
       "Q 1206 -91 779 398 \n",
       "Q 353 888 353 1747 \n",
       "Q 353 2609 779 3096 \n",
       "Q 1206 3584 1959 3584 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-6e\" d=\"M 3513 2113 \n",
       "L 3513 0 \n",
       "L 2938 0 \n",
       "L 2938 2094 \n",
       "Q 2938 2591 2744 2837 \n",
       "Q 2550 3084 2163 3084 \n",
       "Q 1697 3084 1428 2787 \n",
       "Q 1159 2491 1159 1978 \n",
       "L 1159 0 \n",
       "L 581 0 \n",
       "L 581 3500 \n",
       "L 1159 3500 \n",
       "L 1159 2956 \n",
       "Q 1366 3272 1645 3428 \n",
       "Q 1925 3584 2291 3584 \n",
       "Q 2894 3584 3203 3211 \n",
       "Q 3513 2838 3513 2113 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-47\" d=\"M 3809 666 \n",
       "L 3809 1919 \n",
       "L 2778 1919 \n",
       "L 2778 2438 \n",
       "L 4434 2438 \n",
       "L 4434 434 \n",
       "Q 4069 175 3628 42 \n",
       "Q 3188 -91 2688 -91 \n",
       "Q 1594 -91 976 548 \n",
       "Q 359 1188 359 2328 \n",
       "Q 359 3472 976 4111 \n",
       "Q 1594 4750 2688 4750 \n",
       "Q 3144 4750 3555 4637 \n",
       "Q 3966 4525 4313 4306 \n",
       "L 4313 3634 \n",
       "Q 3963 3931 3569 4081 \n",
       "Q 3175 4231 2741 4231 \n",
       "Q 1884 4231 1454 3753 \n",
       "Q 1025 3275 1025 2328 \n",
       "Q 1025 1384 1454 906 \n",
       "Q 1884 428 2741 428 \n",
       "Q 3075 428 3337 486 \n",
       "Q 3600 544 3809 666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-50\" d=\"M 1259 4147 \n",
       "L 1259 2394 \n",
       "L 2053 2394 \n",
       "Q 2494 2394 2734 2622 \n",
       "Q 2975 2850 2975 3272 \n",
       "Q 2975 3691 2734 3919 \n",
       "Q 2494 4147 2053 4147 \n",
       "L 1259 4147 \n",
       "z\n",
       "M 628 4666 \n",
       "L 2053 4666 \n",
       "Q 2838 4666 3239 4311 \n",
       "Q 3641 3956 3641 3272 \n",
       "Q 3641 2581 3239 2228 \n",
       "Q 2838 1875 2053 1875 \n",
       "L 1259 1875 \n",
       "L 1259 0 \n",
       "L 628 0 \n",
       "L 628 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-54\" d=\"M -19 4666 \n",
       "L 3928 4666 \n",
       "L 3928 4134 \n",
       "L 2272 4134 \n",
       "L 2272 0 \n",
       "L 1638 0 \n",
       "L 1638 4134 \n",
       "L -19 4134 \n",
       "L -19 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-4d\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-65\" x=\"86.279297\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-67\" x=\"147.802734\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-61\" x=\"211.279297\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-74\" x=\"272.558594\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-72\" x=\"311.767578\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-6f\" x=\"350.630859\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-6e\" x=\"411.8125\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-20\" x=\"475.191406\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-47\" x=\"506.978516\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-50\" x=\"584.46875\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-54\" x=\"644.771484\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-32\" x=\"705.855469\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_2\">\n",
       "     <g id=\"line2d_6\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m5d3f33379c\" x=\"133.35\" y=\"49.823182\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_7\">\n",
       "      <!-- Megatron GPT2, 2 GPUs -->\n",
       "      <g transform=\"translate(7.2 53.622401)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-2c\" d=\"M 750 794 \n",
       "L 1409 794 \n",
       "L 1409 256 \n",
       "L 897 -744 \n",
       "L 494 -744 \n",
       "L 750 256 \n",
       "L 750 794 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-55\" d=\"M 556 4666 \n",
       "L 1191 4666 \n",
       "L 1191 1831 \n",
       "Q 1191 1081 1462 751 \n",
       "Q 1734 422 2344 422 \n",
       "Q 2950 422 3222 751 \n",
       "Q 3494 1081 3494 1831 \n",
       "L 3494 4666 \n",
       "L 4128 4666 \n",
       "L 4128 1753 \n",
       "Q 4128 841 3676 375 \n",
       "Q 3225 -91 2344 -91 \n",
       "Q 1459 -91 1007 375 \n",
       "Q 556 841 556 1753 \n",
       "L 556 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-4d\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-65\" x=\"86.279297\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-67\" x=\"147.802734\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-61\" x=\"211.279297\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-74\" x=\"272.558594\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-72\" x=\"311.767578\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-6f\" x=\"350.630859\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-6e\" x=\"411.8125\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-20\" x=\"475.191406\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-47\" x=\"506.978516\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-50\" x=\"584.46875\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-54\" x=\"644.771484\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-32\" x=\"705.855469\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-2c\" x=\"769.478516\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-20\" x=\"801.265625\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-32\" x=\"833.052734\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-20\" x=\"896.675781\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-47\" x=\"928.462891\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-50\" x=\"1005.953125\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-55\" x=\"1066.255859\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-73\" x=\"1139.449219\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"patch_5\">\n",
       "    <path d=\"M 133.35 61.56 \n",
       "L 133.35 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_6\">\n",
       "    <path d=\"M 281.055882 61.56 \n",
       "L 281.055882 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_7\">\n",
       "    <path d=\"M 133.35 61.56 \n",
       "L 281.055882 61.56 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_8\">\n",
       "    <path d=\"M 133.35 7.2 \n",
       "L 281.055882 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "  </g>\n",
       "  <g id=\"axes_2\">\n",
       "   <g id=\"patch_9\">\n",
       "    <path d=\"M 310.597059 61.56 \n",
       "L 458.302941 61.56 \n",
       "L 458.302941 7.2 \n",
       "L 310.597059 7.2 \n",
       "z\n",
       "\" style=\"fill: #ffffff\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_10\">\n",
       "    <path d=\"M 310.597059 9.670909 \n",
       "L 451.269328 9.670909 \n",
       "L 451.269328 28.202727 \n",
       "L 310.597059 28.202727 \n",
       "z\n",
       "\" clip-path=\"url(#p12ec4bdf5b)\" style=\"fill: #e41a1c\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_11\">\n",
       "    <path d=\"M 310.597059 40.557273 \n",
       "L 437.586263 40.557273 \n",
       "L 437.586263 59.089091 \n",
       "L 310.597059 59.089091 \n",
       "z\n",
       "\" clip-path=\"url(#p12ec4bdf5b)\" style=\"fill: #377eb8\"/>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_3\">\n",
       "    <g id=\"xtick_5\">\n",
       "     <g id=\"line2d_7\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#mc3a958a983\" x=\"310.597059\" y=\"61.56\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_8\">\n",
       "      <!-- 0 -->\n",
       "      <g transform=\"translate(307.415809 76.158437)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_6\">\n",
       "     <g id=\"line2d_8\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#mc3a958a983\" x=\"377.325362\" y=\"61.56\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_9\">\n",
       "      <!-- 20 -->\n",
       "      <g transform=\"translate(370.962862 76.158437)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-32\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_7\">\n",
       "     <g id=\"line2d_9\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#mc3a958a983\" x=\"444.053665\" y=\"61.56\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_10\">\n",
       "      <!-- 40 -->\n",
       "      <g transform=\"translate(437.691165 76.158437)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-34\" d=\"M 2419 4116 \n",
       "L 825 1625 \n",
       "L 2419 1625 \n",
       "L 2419 4116 \n",
       "z\n",
       "M 2253 4666 \n",
       "L 3047 4666 \n",
       "L 3047 1625 \n",
       "L 3713 1625 \n",
       "L 3713 1100 \n",
       "L 3047 1100 \n",
       "L 3047 0 \n",
       "L 2419 0 \n",
       "L 2419 1100 \n",
       "L 313 1100 \n",
       "L 313 1709 \n",
       "L 2253 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-34\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"text_11\">\n",
       "     <!-- per GPU TFLOPS -->\n",
       "     <g transform=\"translate(343.757812 89.836562)scale(0.1 -0.1)\">\n",
       "      <defs>\n",
       "       <path id=\"DejaVuSans-46\" d=\"M 628 4666 \n",
       "L 3309 4666 \n",
       "L 3309 4134 \n",
       "L 1259 4134 \n",
       "L 1259 2759 \n",
       "L 3109 2759 \n",
       "L 3109 2228 \n",
       "L 1259 2228 \n",
       "L 1259 0 \n",
       "L 628 0 \n",
       "L 628 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-4c\" d=\"M 628 4666 \n",
       "L 1259 4666 \n",
       "L 1259 531 \n",
       "L 3531 531 \n",
       "L 3531 0 \n",
       "L 628 0 \n",
       "L 628 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-4f\" d=\"M 2522 4238 \n",
       "Q 1834 4238 1429 3725 \n",
       "Q 1025 3213 1025 2328 \n",
       "Q 1025 1447 1429 934 \n",
       "Q 1834 422 2522 422 \n",
       "Q 3209 422 3611 934 \n",
       "Q 4013 1447 4013 2328 \n",
       "Q 4013 3213 3611 3725 \n",
       "Q 3209 4238 2522 4238 \n",
       "z\n",
       "M 2522 4750 \n",
       "Q 3503 4750 4090 4092 \n",
       "Q 4678 3434 4678 2328 \n",
       "Q 4678 1225 4090 567 \n",
       "Q 3503 -91 2522 -91 \n",
       "Q 1538 -91 948 565 \n",
       "Q 359 1222 359 2328 \n",
       "Q 359 3434 948 4092 \n",
       "Q 1538 4750 2522 4750 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "      </defs>\n",
       "      <use xlink:href=\"#DejaVuSans-70\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-65\" x=\"63.476562\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-72\" x=\"125\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-20\" x=\"166.113281\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-47\" x=\"197.900391\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-50\" x=\"275.390625\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-55\" x=\"335.693359\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-20\" x=\"408.886719\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-54\" x=\"440.673828\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-46\" x=\"501.757812\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-4c\" x=\"559.277344\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-4f\" x=\"611.365234\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-50\" x=\"690.076172\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-53\" x=\"750.378906\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_4\">\n",
       "    <g id=\"ytick_3\">\n",
       "     <g id=\"line2d_10\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m5d3f33379c\" x=\"310.597059\" y=\"18.936818\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_4\">\n",
       "     <g id=\"line2d_11\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m5d3f33379c\" x=\"310.597059\" y=\"49.823182\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"patch_12\">\n",
       "    <path d=\"M 310.597059 61.56 \n",
       "L 310.597059 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_13\">\n",
       "    <path d=\"M 458.302941 61.56 \n",
       "L 458.302941 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_14\">\n",
       "    <path d=\"M 310.597059 61.56 \n",
       "L 458.302941 61.56 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_15\">\n",
       "    <path d=\"M 310.597059 7.2 \n",
       "L 458.302941 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "  </g>\n",
       "  <g id=\"axes_3\">\n",
       "   <g id=\"patch_16\">\n",
       "    <path d=\"M 487.844118 61.56 \n",
       "L 635.55 61.56 \n",
       "L 635.55 7.2 \n",
       "L 487.844118 7.2 \n",
       "z\n",
       "\" style=\"fill: #ffffff\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_17\">\n",
       "    <path d=\"M 487.844118 9.670909 \n",
       "L 628.516387 9.670909 \n",
       "L 628.516387 28.202727 \n",
       "L 487.844118 28.202727 \n",
       "z\n",
       "\" clip-path=\"url(#p788e422828)\" style=\"fill: #e41a1c\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_18\">\n",
       "    <path d=\"M 487.844118 40.557273 \n",
       "L 628.516387 40.557273 \n",
       "L 628.516387 59.089091 \n",
       "L 487.844118 59.089091 \n",
       "z\n",
       "\" clip-path=\"url(#p788e422828)\" style=\"fill: #377eb8\"/>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_5\">\n",
       "    <g id=\"xtick_8\">\n",
       "     <g id=\"line2d_12\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#mc3a958a983\" x=\"487.844118\" y=\"61.56\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_12\">\n",
       "      <!-- 0 -->\n",
       "      <g transform=\"translate(484.662868 76.158437)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_9\">\n",
       "     <g id=\"line2d_13\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#mc3a958a983\" x=\"553.472148\" y=\"61.56\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_13\">\n",
       "      <!-- 10 -->\n",
       "      <g transform=\"translate(547.109648 76.158437)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-31\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_10\">\n",
       "     <g id=\"line2d_14\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#mc3a958a983\" x=\"619.100177\" y=\"61.56\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_14\">\n",
       "      <!-- 20 -->\n",
       "      <g transform=\"translate(612.737677 76.158437)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-32\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"text_15\">\n",
       "     <!-- per GPU memory (GB) -->\n",
       "     <g transform=\"translate(505.977528 89.836562)scale(0.1 -0.1)\">\n",
       "      <defs>\n",
       "       <path id=\"DejaVuSans-79\" d=\"M 2059 -325 \n",
       "Q 1816 -950 1584 -1140 \n",
       "Q 1353 -1331 966 -1331 \n",
       "L 506 -1331 \n",
       "L 506 -850 \n",
       "L 844 -850 \n",
       "Q 1081 -850 1212 -737 \n",
       "Q 1344 -625 1503 -206 \n",
       "L 1606 56 \n",
       "L 191 3500 \n",
       "L 800 3500 \n",
       "L 1894 763 \n",
       "L 2988 3500 \n",
       "L 3597 3500 \n",
       "L 2059 -325 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-28\" d=\"M 1984 4856 \n",
       "Q 1566 4138 1362 3434 \n",
       "Q 1159 2731 1159 2009 \n",
       "Q 1159 1288 1364 580 \n",
       "Q 1569 -128 1984 -844 \n",
       "L 1484 -844 \n",
       "Q 1016 -109 783 600 \n",
       "Q 550 1309 550 2009 \n",
       "Q 550 2706 781 3412 \n",
       "Q 1013 4119 1484 4856 \n",
       "L 1984 4856 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-42\" d=\"M 1259 2228 \n",
       "L 1259 519 \n",
       "L 2272 519 \n",
       "Q 2781 519 3026 730 \n",
       "Q 3272 941 3272 1375 \n",
       "Q 3272 1813 3026 2020 \n",
       "Q 2781 2228 2272 2228 \n",
       "L 1259 2228 \n",
       "z\n",
       "M 1259 4147 \n",
       "L 1259 2741 \n",
       "L 2194 2741 \n",
       "Q 2656 2741 2882 2914 \n",
       "Q 3109 3088 3109 3444 \n",
       "Q 3109 3797 2882 3972 \n",
       "Q 2656 4147 2194 4147 \n",
       "L 1259 4147 \n",
       "z\n",
       "M 628 4666 \n",
       "L 2241 4666 \n",
       "Q 2963 4666 3353 4366 \n",
       "Q 3744 4066 3744 3513 \n",
       "Q 3744 3084 3544 2831 \n",
       "Q 3344 2578 2956 2516 \n",
       "Q 3422 2416 3680 2098 \n",
       "Q 3938 1781 3938 1306 \n",
       "Q 3938 681 3513 340 \n",
       "Q 3088 0 2303 0 \n",
       "L 628 0 \n",
       "L 628 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-29\" d=\"M 513 4856 \n",
       "L 1013 4856 \n",
       "Q 1481 4119 1714 3412 \n",
       "Q 1947 2706 1947 2009 \n",
       "Q 1947 1309 1714 600 \n",
       "Q 1481 -109 1013 -844 \n",
       "L 513 -844 \n",
       "Q 928 -128 1133 580 \n",
       "Q 1338 1288 1338 2009 \n",
       "Q 1338 2731 1133 3434 \n",
       "Q 928 4138 513 4856 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "      </defs>\n",
       "      <use xlink:href=\"#DejaVuSans-70\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-65\" x=\"63.476562\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-72\" x=\"125\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-20\" x=\"166.113281\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-47\" x=\"197.900391\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-50\" x=\"275.390625\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-55\" x=\"335.693359\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-20\" x=\"408.886719\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-6d\" x=\"440.673828\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-65\" x=\"538.085938\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-6d\" x=\"599.609375\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-6f\" x=\"697.021484\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-72\" x=\"758.203125\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-79\" x=\"799.316406\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-20\" x=\"858.496094\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-28\" x=\"890.283203\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-47\" x=\"929.296875\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-42\" x=\"1006.787109\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-29\" x=\"1075.390625\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_6\">\n",
       "    <g id=\"ytick_5\">\n",
       "     <g id=\"line2d_15\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m5d3f33379c\" x=\"487.844118\" y=\"18.936818\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_6\">\n",
       "     <g id=\"line2d_16\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m5d3f33379c\" x=\"487.844118\" y=\"49.823182\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"patch_19\">\n",
       "    <path d=\"M 487.844118 61.56 \n",
       "L 487.844118 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_20\">\n",
       "    <path d=\"M 635.55 61.56 \n",
       "L 635.55 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_21\">\n",
       "    <path d=\"M 487.844118 61.56 \n",
       "L 635.55 61.56 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_22\">\n",
       "    <path d=\"M 487.844118 7.2 \n",
       "L 635.55 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "  </g>\n",
       " </g>\n",
       " <defs>\n",
       "  <clipPath id=\"p88919dde73\">\n",
       "   <rect x=\"133.35\" y=\"7.2\" width=\"147.705882\" height=\"54.36\"/>\n",
       "  </clipPath>\n",
       "  <clipPath id=\"p12ec4bdf5b\">\n",
       "   <rect x=\"310.597059\" y=\"7.2\" width=\"147.705882\" height=\"54.36\"/>\n",
       "  </clipPath>\n",
       "  <clipPath id=\"p788e422828\">\n",
       "   <rect x=\"487.844118\" y=\"7.2\" width=\"147.705882\" height=\"54.36\"/>\n",
       "  </clipPath>\n",
       " </defs>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<Figure size 648x72 with 3 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "dp_gpt2 = megatron_gpt(Exp(\"Megatron GPT2, 2 GPUs\", \"gpt2-medium\", 5, bf16=True, gpus='0,1'))\n",
    "compare([mega_gpt2, dp_gpt2])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a999486d",
   "metadata": {},
   "source": [
    "From the time breakdown, you can see the communication takes 10\\%, which is almost 0 on a single GPU. It leads to a reduced per GPU TFLOPS. \n",
    "\n",
    "If we disable NVLink to use PCIe instead, the performance decreases."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "0766f936",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Time breakdown\t\t: forward+backward 0.57, communication 0.38, optimizer 0.05\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       "  \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<svg xmlns:xlink=\"http://www.w3.org/1999/xlink\" width=\"642.75pt\" height=\"126.29625pt\" viewBox=\"0 0 642.75 126.29625\" xmlns=\"http://www.w3.org/2000/svg\" version=\"1.1\">\n",
       " <metadata>\n",
       "  <rdf:RDF xmlns:dc=\"http://purl.org/dc/elements/1.1/\" xmlns:cc=\"http://creativecommons.org/ns#\" xmlns:rdf=\"http://www.w3.org/1999/02/22-rdf-syntax-ns#\">\n",
       "   <cc:Work>\n",
       "    <dc:type rdf:resource=\"http://purl.org/dc/dcmitype/StillImage\"/>\n",
       "    <dc:date>2022-09-02T19:45:34.129876</dc:date>\n",
       "    <dc:format>image/svg+xml</dc:format>\n",
       "    <dc:creator>\n",
       "     <cc:Agent>\n",
       "      <dc:title>Matplotlib v3.5.2, https://matplotlib.org/</dc:title>\n",
       "     </cc:Agent>\n",
       "    </dc:creator>\n",
       "   </cc:Work>\n",
       "  </rdf:RDF>\n",
       " </metadata>\n",
       " <defs>\n",
       "  <style type=\"text/css\">*{stroke-linejoin: round; stroke-linecap: butt}</style>\n",
       " </defs>\n",
       " <g id=\"figure_1\">\n",
       "  <g id=\"patch_1\">\n",
       "   <path d=\"M 0 126.29625 \n",
       "L 642.75 126.29625 \n",
       "L 642.75 0 \n",
       "L 0 0 \n",
       "L 0 126.29625 \n",
       "z\n",
       "\" style=\"fill: none\"/>\n",
       "  </g>\n",
       "  <g id=\"axes_1\">\n",
       "   <g id=\"patch_2\">\n",
       "    <path d=\"M 133.35 88.74 \n",
       "L 281.055882 88.74 \n",
       "L 281.055882 7.2 \n",
       "L 133.35 7.2 \n",
       "z\n",
       "\" style=\"fill: #ffffff\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_3\">\n",
       "    <path d=\"M 133.35 10.906364 \n",
       "L 211.264841 10.906364 \n",
       "L 211.264841 28.012657 \n",
       "L 133.35 28.012657 \n",
       "z\n",
       "\" clip-path=\"url(#pe9fdbfc80d)\" style=\"fill: #e41a1c\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_4\">\n",
       "    <path d=\"M 133.35 39.416853 \n",
       "L 274.022269 39.416853 \n",
       "L 274.022269 56.523147 \n",
       "L 133.35 56.523147 \n",
       "z\n",
       "\" clip-path=\"url(#pe9fdbfc80d)\" style=\"fill: #377eb8\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_5\">\n",
       "    <path d=\"M 133.35 67.927343 \n",
       "L 231.372434 67.927343 \n",
       "L 231.372434 85.033636 \n",
       "L 133.35 85.033636 \n",
       "z\n",
       "\" clip-path=\"url(#pe9fdbfc80d)\" style=\"fill: #4daf4a\"/>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_1\">\n",
       "    <g id=\"xtick_1\">\n",
       "     <g id=\"line2d_1\">\n",
       "      <defs>\n",
       "       <path id=\"me3c65d8a53\" d=\"M 0 0 \n",
       "L 0 3.5 \n",
       "\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </defs>\n",
       "      <g>\n",
       "       <use xlink:href=\"#me3c65d8a53\" x=\"133.35\" y=\"88.74\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_1\">\n",
       "      <!-- 0 -->\n",
       "      <g transform=\"translate(130.16875 103.338438)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-30\" d=\"M 2034 4250 \n",
       "Q 1547 4250 1301 3770 \n",
       "Q 1056 3291 1056 2328 \n",
       "Q 1056 1369 1301 889 \n",
       "Q 1547 409 2034 409 \n",
       "Q 2525 409 2770 889 \n",
       "Q 3016 1369 3016 2328 \n",
       "Q 3016 3291 2770 3770 \n",
       "Q 2525 4250 2034 4250 \n",
       "z\n",
       "M 2034 4750 \n",
       "Q 2819 4750 3233 4129 \n",
       "Q 3647 3509 3647 2328 \n",
       "Q 3647 1150 3233 529 \n",
       "Q 2819 -91 2034 -91 \n",
       "Q 1250 -91 836 529 \n",
       "Q 422 1150 422 2328 \n",
       "Q 422 3509 836 4129 \n",
       "Q 1250 4750 2034 4750 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_2\">\n",
       "     <g id=\"line2d_2\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#me3c65d8a53\" x=\"179.195092\" y=\"88.74\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_2\">\n",
       "      <!-- 10 -->\n",
       "      <g transform=\"translate(172.832592 103.338438)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-31\" d=\"M 794 531 \n",
       "L 1825 531 \n",
       "L 1825 4091 \n",
       "L 703 3866 \n",
       "L 703 4441 \n",
       "L 1819 4666 \n",
       "L 2450 4666 \n",
       "L 2450 531 \n",
       "L 3481 531 \n",
       "L 3481 0 \n",
       "L 794 0 \n",
       "L 794 531 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-31\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_3\">\n",
       "     <g id=\"line2d_3\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#me3c65d8a53\" x=\"225.040185\" y=\"88.74\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_3\">\n",
       "      <!-- 20 -->\n",
       "      <g transform=\"translate(218.677685 103.338438)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-32\" d=\"M 1228 531 \n",
       "L 3431 531 \n",
       "L 3431 0 \n",
       "L 469 0 \n",
       "L 469 531 \n",
       "Q 828 903 1448 1529 \n",
       "Q 2069 2156 2228 2338 \n",
       "Q 2531 2678 2651 2914 \n",
       "Q 2772 3150 2772 3378 \n",
       "Q 2772 3750 2511 3984 \n",
       "Q 2250 4219 1831 4219 \n",
       "Q 1534 4219 1204 4116 \n",
       "Q 875 4013 500 3803 \n",
       "L 500 4441 \n",
       "Q 881 4594 1212 4672 \n",
       "Q 1544 4750 1819 4750 \n",
       "Q 2544 4750 2975 4387 \n",
       "Q 3406 4025 3406 3419 \n",
       "Q 3406 3131 3298 2873 \n",
       "Q 3191 2616 2906 2266 \n",
       "Q 2828 2175 2409 1742 \n",
       "Q 1991 1309 1228 531 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-32\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_4\">\n",
       "     <g id=\"line2d_4\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#me3c65d8a53\" x=\"270.885277\" y=\"88.74\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_4\">\n",
       "      <!-- 30 -->\n",
       "      <g transform=\"translate(264.522777 103.338438)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-33\" d=\"M 2597 2516 \n",
       "Q 3050 2419 3304 2112 \n",
       "Q 3559 1806 3559 1356 \n",
       "Q 3559 666 3084 287 \n",
       "Q 2609 -91 1734 -91 \n",
       "Q 1441 -91 1130 -33 \n",
       "Q 819 25 488 141 \n",
       "L 488 750 \n",
       "Q 750 597 1062 519 \n",
       "Q 1375 441 1716 441 \n",
       "Q 2309 441 2620 675 \n",
       "Q 2931 909 2931 1356 \n",
       "Q 2931 1769 2642 2001 \n",
       "Q 2353 2234 1838 2234 \n",
       "L 1294 2234 \n",
       "L 1294 2753 \n",
       "L 1863 2753 \n",
       "Q 2328 2753 2575 2939 \n",
       "Q 2822 3125 2822 3475 \n",
       "Q 2822 3834 2567 4026 \n",
       "Q 2313 4219 1838 4219 \n",
       "Q 1578 4219 1281 4162 \n",
       "Q 984 4106 628 3988 \n",
       "L 628 4550 \n",
       "Q 988 4650 1302 4700 \n",
       "Q 1616 4750 1894 4750 \n",
       "Q 2613 4750 3031 4423 \n",
       "Q 3450 4097 3450 3541 \n",
       "Q 3450 3153 3228 2886 \n",
       "Q 3006 2619 2597 2516 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-33\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"text_5\">\n",
       "     <!-- Samples / sec -->\n",
       "     <g transform=\"translate(172.556847 117.016563)scale(0.1 -0.1)\">\n",
       "      <defs>\n",
       "       <path id=\"DejaVuSans-53\" d=\"M 3425 4513 \n",
       "L 3425 3897 \n",
       "Q 3066 4069 2747 4153 \n",
       "Q 2428 4238 2131 4238 \n",
       "Q 1616 4238 1336 4038 \n",
       "Q 1056 3838 1056 3469 \n",
       "Q 1056 3159 1242 3001 \n",
       "Q 1428 2844 1947 2747 \n",
       "L 2328 2669 \n",
       "Q 3034 2534 3370 2195 \n",
       "Q 3706 1856 3706 1288 \n",
       "Q 3706 609 3251 259 \n",
       "Q 2797 -91 1919 -91 \n",
       "Q 1588 -91 1214 -16 \n",
       "Q 841 59 441 206 \n",
       "L 441 856 \n",
       "Q 825 641 1194 531 \n",
       "Q 1563 422 1919 422 \n",
       "Q 2459 422 2753 634 \n",
       "Q 3047 847 3047 1241 \n",
       "Q 3047 1584 2836 1778 \n",
       "Q 2625 1972 2144 2069 \n",
       "L 1759 2144 \n",
       "Q 1053 2284 737 2584 \n",
       "Q 422 2884 422 3419 \n",
       "Q 422 4038 858 4394 \n",
       "Q 1294 4750 2059 4750 \n",
       "Q 2388 4750 2728 4690 \n",
       "Q 3069 4631 3425 4513 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-61\" d=\"M 2194 1759 \n",
       "Q 1497 1759 1228 1600 \n",
       "Q 959 1441 959 1056 \n",
       "Q 959 750 1161 570 \n",
       "Q 1363 391 1709 391 \n",
       "Q 2188 391 2477 730 \n",
       "Q 2766 1069 2766 1631 \n",
       "L 2766 1759 \n",
       "L 2194 1759 \n",
       "z\n",
       "M 3341 1997 \n",
       "L 3341 0 \n",
       "L 2766 0 \n",
       "L 2766 531 \n",
       "Q 2569 213 2275 61 \n",
       "Q 1981 -91 1556 -91 \n",
       "Q 1019 -91 701 211 \n",
       "Q 384 513 384 1019 \n",
       "Q 384 1609 779 1909 \n",
       "Q 1175 2209 1959 2209 \n",
       "L 2766 2209 \n",
       "L 2766 2266 \n",
       "Q 2766 2663 2505 2880 \n",
       "Q 2244 3097 1772 3097 \n",
       "Q 1472 3097 1187 3025 \n",
       "Q 903 2953 641 2809 \n",
       "L 641 3341 \n",
       "Q 956 3463 1253 3523 \n",
       "Q 1550 3584 1831 3584 \n",
       "Q 2591 3584 2966 3190 \n",
       "Q 3341 2797 3341 1997 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-6d\" d=\"M 3328 2828 \n",
       "Q 3544 3216 3844 3400 \n",
       "Q 4144 3584 4550 3584 \n",
       "Q 5097 3584 5394 3201 \n",
       "Q 5691 2819 5691 2113 \n",
       "L 5691 0 \n",
       "L 5113 0 \n",
       "L 5113 2094 \n",
       "Q 5113 2597 4934 2840 \n",
       "Q 4756 3084 4391 3084 \n",
       "Q 3944 3084 3684 2787 \n",
       "Q 3425 2491 3425 1978 \n",
       "L 3425 0 \n",
       "L 2847 0 \n",
       "L 2847 2094 \n",
       "Q 2847 2600 2669 2842 \n",
       "Q 2491 3084 2119 3084 \n",
       "Q 1678 3084 1418 2786 \n",
       "Q 1159 2488 1159 1978 \n",
       "L 1159 0 \n",
       "L 581 0 \n",
       "L 581 3500 \n",
       "L 1159 3500 \n",
       "L 1159 2956 \n",
       "Q 1356 3278 1631 3431 \n",
       "Q 1906 3584 2284 3584 \n",
       "Q 2666 3584 2933 3390 \n",
       "Q 3200 3197 3328 2828 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-70\" d=\"M 1159 525 \n",
       "L 1159 -1331 \n",
       "L 581 -1331 \n",
       "L 581 3500 \n",
       "L 1159 3500 \n",
       "L 1159 2969 \n",
       "Q 1341 3281 1617 3432 \n",
       "Q 1894 3584 2278 3584 \n",
       "Q 2916 3584 3314 3078 \n",
       "Q 3713 2572 3713 1747 \n",
       "Q 3713 922 3314 415 \n",
       "Q 2916 -91 2278 -91 \n",
       "Q 1894 -91 1617 61 \n",
       "Q 1341 213 1159 525 \n",
       "z\n",
       "M 3116 1747 \n",
       "Q 3116 2381 2855 2742 \n",
       "Q 2594 3103 2138 3103 \n",
       "Q 1681 3103 1420 2742 \n",
       "Q 1159 2381 1159 1747 \n",
       "Q 1159 1113 1420 752 \n",
       "Q 1681 391 2138 391 \n",
       "Q 2594 391 2855 752 \n",
       "Q 3116 1113 3116 1747 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-6c\" d=\"M 603 4863 \n",
       "L 1178 4863 \n",
       "L 1178 0 \n",
       "L 603 0 \n",
       "L 603 4863 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-65\" d=\"M 3597 1894 \n",
       "L 3597 1613 \n",
       "L 953 1613 \n",
       "Q 991 1019 1311 708 \n",
       "Q 1631 397 2203 397 \n",
       "Q 2534 397 2845 478 \n",
       "Q 3156 559 3463 722 \n",
       "L 3463 178 \n",
       "Q 3153 47 2828 -22 \n",
       "Q 2503 -91 2169 -91 \n",
       "Q 1331 -91 842 396 \n",
       "Q 353 884 353 1716 \n",
       "Q 353 2575 817 3079 \n",
       "Q 1281 3584 2069 3584 \n",
       "Q 2775 3584 3186 3129 \n",
       "Q 3597 2675 3597 1894 \n",
       "z\n",
       "M 3022 2063 \n",
       "Q 3016 2534 2758 2815 \n",
       "Q 2500 3097 2075 3097 \n",
       "Q 1594 3097 1305 2825 \n",
       "Q 1016 2553 972 2059 \n",
       "L 3022 2063 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-73\" d=\"M 2834 3397 \n",
       "L 2834 2853 \n",
       "Q 2591 2978 2328 3040 \n",
       "Q 2066 3103 1784 3103 \n",
       "Q 1356 3103 1142 2972 \n",
       "Q 928 2841 928 2578 \n",
       "Q 928 2378 1081 2264 \n",
       "Q 1234 2150 1697 2047 \n",
       "L 1894 2003 \n",
       "Q 2506 1872 2764 1633 \n",
       "Q 3022 1394 3022 966 \n",
       "Q 3022 478 2636 193 \n",
       "Q 2250 -91 1575 -91 \n",
       "Q 1294 -91 989 -36 \n",
       "Q 684 19 347 128 \n",
       "L 347 722 \n",
       "Q 666 556 975 473 \n",
       "Q 1284 391 1588 391 \n",
       "Q 1994 391 2212 530 \n",
       "Q 2431 669 2431 922 \n",
       "Q 2431 1156 2273 1281 \n",
       "Q 2116 1406 1581 1522 \n",
       "L 1381 1569 \n",
       "Q 847 1681 609 1914 \n",
       "Q 372 2147 372 2553 \n",
       "Q 372 3047 722 3315 \n",
       "Q 1072 3584 1716 3584 \n",
       "Q 2034 3584 2315 3537 \n",
       "Q 2597 3491 2834 3397 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-20\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-2f\" d=\"M 1625 4666 \n",
       "L 2156 4666 \n",
       "L 531 -594 \n",
       "L 0 -594 \n",
       "L 1625 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-63\" d=\"M 3122 3366 \n",
       "L 3122 2828 \n",
       "Q 2878 2963 2633 3030 \n",
       "Q 2388 3097 2138 3097 \n",
       "Q 1578 3097 1268 2742 \n",
       "Q 959 2388 959 1747 \n",
       "Q 959 1106 1268 751 \n",
       "Q 1578 397 2138 397 \n",
       "Q 2388 397 2633 464 \n",
       "Q 2878 531 3122 666 \n",
       "L 3122 134 \n",
       "Q 2881 22 2623 -34 \n",
       "Q 2366 -91 2075 -91 \n",
       "Q 1284 -91 818 406 \n",
       "Q 353 903 353 1747 \n",
       "Q 353 2603 823 3093 \n",
       "Q 1294 3584 2113 3584 \n",
       "Q 2378 3584 2631 3529 \n",
       "Q 2884 3475 3122 3366 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "      </defs>\n",
       "      <use xlink:href=\"#DejaVuSans-53\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-61\" x=\"63.476562\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-6d\" x=\"124.755859\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-70\" x=\"222.167969\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-6c\" x=\"285.644531\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-65\" x=\"313.427734\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-73\" x=\"374.951172\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-20\" x=\"427.050781\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-2f\" x=\"458.837891\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-20\" x=\"492.529297\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-73\" x=\"524.316406\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-65\" x=\"576.416016\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-63\" x=\"637.939453\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_2\">\n",
       "    <g id=\"ytick_1\">\n",
       "     <g id=\"line2d_5\">\n",
       "      <defs>\n",
       "       <path id=\"m5946a68768\" d=\"M 0 0 \n",
       "L -3.5 0 \n",
       "\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </defs>\n",
       "      <g>\n",
       "       <use xlink:href=\"#m5946a68768\" x=\"133.35\" y=\"19.45951\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_6\">\n",
       "      <!-- Megatron GPT2 -->\n",
       "      <g transform=\"translate(49.403125 23.258729)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-4d\" d=\"M 628 4666 \n",
       "L 1569 4666 \n",
       "L 2759 1491 \n",
       "L 3956 4666 \n",
       "L 4897 4666 \n",
       "L 4897 0 \n",
       "L 4281 0 \n",
       "L 4281 4097 \n",
       "L 3078 897 \n",
       "L 2444 897 \n",
       "L 1241 4097 \n",
       "L 1241 0 \n",
       "L 628 0 \n",
       "L 628 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-67\" d=\"M 2906 1791 \n",
       "Q 2906 2416 2648 2759 \n",
       "Q 2391 3103 1925 3103 \n",
       "Q 1463 3103 1205 2759 \n",
       "Q 947 2416 947 1791 \n",
       "Q 947 1169 1205 825 \n",
       "Q 1463 481 1925 481 \n",
       "Q 2391 481 2648 825 \n",
       "Q 2906 1169 2906 1791 \n",
       "z\n",
       "M 3481 434 \n",
       "Q 3481 -459 3084 -895 \n",
       "Q 2688 -1331 1869 -1331 \n",
       "Q 1566 -1331 1297 -1286 \n",
       "Q 1028 -1241 775 -1147 \n",
       "L 775 -588 \n",
       "Q 1028 -725 1275 -790 \n",
       "Q 1522 -856 1778 -856 \n",
       "Q 2344 -856 2625 -561 \n",
       "Q 2906 -266 2906 331 \n",
       "L 2906 616 \n",
       "Q 2728 306 2450 153 \n",
       "Q 2172 0 1784 0 \n",
       "Q 1141 0 747 490 \n",
       "Q 353 981 353 1791 \n",
       "Q 353 2603 747 3093 \n",
       "Q 1141 3584 1784 3584 \n",
       "Q 2172 3584 2450 3431 \n",
       "Q 2728 3278 2906 2969 \n",
       "L 2906 3500 \n",
       "L 3481 3500 \n",
       "L 3481 434 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-74\" d=\"M 1172 4494 \n",
       "L 1172 3500 \n",
       "L 2356 3500 \n",
       "L 2356 3053 \n",
       "L 1172 3053 \n",
       "L 1172 1153 \n",
       "Q 1172 725 1289 603 \n",
       "Q 1406 481 1766 481 \n",
       "L 2356 481 \n",
       "L 2356 0 \n",
       "L 1766 0 \n",
       "Q 1100 0 847 248 \n",
       "Q 594 497 594 1153 \n",
       "L 594 3053 \n",
       "L 172 3053 \n",
       "L 172 3500 \n",
       "L 594 3500 \n",
       "L 594 4494 \n",
       "L 1172 4494 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-72\" d=\"M 2631 2963 \n",
       "Q 2534 3019 2420 3045 \n",
       "Q 2306 3072 2169 3072 \n",
       "Q 1681 3072 1420 2755 \n",
       "Q 1159 2438 1159 1844 \n",
       "L 1159 0 \n",
       "L 581 0 \n",
       "L 581 3500 \n",
       "L 1159 3500 \n",
       "L 1159 2956 \n",
       "Q 1341 3275 1631 3429 \n",
       "Q 1922 3584 2338 3584 \n",
       "Q 2397 3584 2469 3576 \n",
       "Q 2541 3569 2628 3553 \n",
       "L 2631 2963 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-6f\" d=\"M 1959 3097 \n",
       "Q 1497 3097 1228 2736 \n",
       "Q 959 2375 959 1747 \n",
       "Q 959 1119 1226 758 \n",
       "Q 1494 397 1959 397 \n",
       "Q 2419 397 2687 759 \n",
       "Q 2956 1122 2956 1747 \n",
       "Q 2956 2369 2687 2733 \n",
       "Q 2419 3097 1959 3097 \n",
       "z\n",
       "M 1959 3584 \n",
       "Q 2709 3584 3137 3096 \n",
       "Q 3566 2609 3566 1747 \n",
       "Q 3566 888 3137 398 \n",
       "Q 2709 -91 1959 -91 \n",
       "Q 1206 -91 779 398 \n",
       "Q 353 888 353 1747 \n",
       "Q 353 2609 779 3096 \n",
       "Q 1206 3584 1959 3584 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-6e\" d=\"M 3513 2113 \n",
       "L 3513 0 \n",
       "L 2938 0 \n",
       "L 2938 2094 \n",
       "Q 2938 2591 2744 2837 \n",
       "Q 2550 3084 2163 3084 \n",
       "Q 1697 3084 1428 2787 \n",
       "Q 1159 2491 1159 1978 \n",
       "L 1159 0 \n",
       "L 581 0 \n",
       "L 581 3500 \n",
       "L 1159 3500 \n",
       "L 1159 2956 \n",
       "Q 1366 3272 1645 3428 \n",
       "Q 1925 3584 2291 3584 \n",
       "Q 2894 3584 3203 3211 \n",
       "Q 3513 2838 3513 2113 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-47\" d=\"M 3809 666 \n",
       "L 3809 1919 \n",
       "L 2778 1919 \n",
       "L 2778 2438 \n",
       "L 4434 2438 \n",
       "L 4434 434 \n",
       "Q 4069 175 3628 42 \n",
       "Q 3188 -91 2688 -91 \n",
       "Q 1594 -91 976 548 \n",
       "Q 359 1188 359 2328 \n",
       "Q 359 3472 976 4111 \n",
       "Q 1594 4750 2688 4750 \n",
       "Q 3144 4750 3555 4637 \n",
       "Q 3966 4525 4313 4306 \n",
       "L 4313 3634 \n",
       "Q 3963 3931 3569 4081 \n",
       "Q 3175 4231 2741 4231 \n",
       "Q 1884 4231 1454 3753 \n",
       "Q 1025 3275 1025 2328 \n",
       "Q 1025 1384 1454 906 \n",
       "Q 1884 428 2741 428 \n",
       "Q 3075 428 3337 486 \n",
       "Q 3600 544 3809 666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-50\" d=\"M 1259 4147 \n",
       "L 1259 2394 \n",
       "L 2053 2394 \n",
       "Q 2494 2394 2734 2622 \n",
       "Q 2975 2850 2975 3272 \n",
       "Q 2975 3691 2734 3919 \n",
       "Q 2494 4147 2053 4147 \n",
       "L 1259 4147 \n",
       "z\n",
       "M 628 4666 \n",
       "L 2053 4666 \n",
       "Q 2838 4666 3239 4311 \n",
       "Q 3641 3956 3641 3272 \n",
       "Q 3641 2581 3239 2228 \n",
       "Q 2838 1875 2053 1875 \n",
       "L 1259 1875 \n",
       "L 1259 0 \n",
       "L 628 0 \n",
       "L 628 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-54\" d=\"M -19 4666 \n",
       "L 3928 4666 \n",
       "L 3928 4134 \n",
       "L 2272 4134 \n",
       "L 2272 0 \n",
       "L 1638 0 \n",
       "L 1638 4134 \n",
       "L -19 4134 \n",
       "L -19 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-4d\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-65\" x=\"86.279297\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-67\" x=\"147.802734\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-61\" x=\"211.279297\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-74\" x=\"272.558594\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-72\" x=\"311.767578\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-6f\" x=\"350.630859\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-6e\" x=\"411.8125\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-20\" x=\"475.191406\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-47\" x=\"506.978516\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-50\" x=\"584.46875\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-54\" x=\"644.771484\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-32\" x=\"705.855469\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_2\">\n",
       "     <g id=\"line2d_6\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m5946a68768\" x=\"133.35\" y=\"47.97\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_7\">\n",
       "      <!-- Megatron GPT2, 2 GPUs -->\n",
       "      <g transform=\"translate(7.2 51.769219)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-2c\" d=\"M 750 794 \n",
       "L 1409 794 \n",
       "L 1409 256 \n",
       "L 897 -744 \n",
       "L 494 -744 \n",
       "L 750 256 \n",
       "L 750 794 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-55\" d=\"M 556 4666 \n",
       "L 1191 4666 \n",
       "L 1191 1831 \n",
       "Q 1191 1081 1462 751 \n",
       "Q 1734 422 2344 422 \n",
       "Q 2950 422 3222 751 \n",
       "Q 3494 1081 3494 1831 \n",
       "L 3494 4666 \n",
       "L 4128 4666 \n",
       "L 4128 1753 \n",
       "Q 4128 841 3676 375 \n",
       "Q 3225 -91 2344 -91 \n",
       "Q 1459 -91 1007 375 \n",
       "Q 556 841 556 1753 \n",
       "L 556 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-4d\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-65\" x=\"86.279297\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-67\" x=\"147.802734\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-61\" x=\"211.279297\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-74\" x=\"272.558594\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-72\" x=\"311.767578\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-6f\" x=\"350.630859\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-6e\" x=\"411.8125\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-20\" x=\"475.191406\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-47\" x=\"506.978516\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-50\" x=\"584.46875\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-54\" x=\"644.771484\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-32\" x=\"705.855469\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-2c\" x=\"769.478516\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-20\" x=\"801.265625\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-32\" x=\"833.052734\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-20\" x=\"896.675781\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-47\" x=\"928.462891\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-50\" x=\"1005.953125\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-55\" x=\"1066.255859\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-73\" x=\"1139.449219\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_3\">\n",
       "     <g id=\"line2d_7\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m5946a68768\" x=\"133.35\" y=\"76.48049\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_8\">\n",
       "      <!-- Megatron GPT2, 2 GPUs -->\n",
       "      <g transform=\"translate(7.2 75.440646)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-4d\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-65\" x=\"86.279297\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-67\" x=\"147.802734\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-61\" x=\"211.279297\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-74\" x=\"272.558594\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-72\" x=\"311.767578\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-6f\" x=\"350.630859\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-6e\" x=\"411.8125\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-20\" x=\"475.191406\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-47\" x=\"506.978516\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-50\" x=\"584.46875\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-54\" x=\"644.771484\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-32\" x=\"705.855469\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-2c\" x=\"769.478516\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-20\" x=\"801.265625\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-32\" x=\"833.052734\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-20\" x=\"896.675781\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-47\" x=\"928.462891\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-50\" x=\"1005.953125\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-55\" x=\"1066.255859\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-73\" x=\"1139.449219\"/>\n",
       "      </g>\n",
       "      <!-- no nvlink -->\n",
       "      <g transform=\"translate(80.775 86.638458)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-76\" d=\"M 191 3500 \n",
       "L 800 3500 \n",
       "L 1894 563 \n",
       "L 2988 3500 \n",
       "L 3597 3500 \n",
       "L 2284 0 \n",
       "L 1503 0 \n",
       "L 191 3500 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-69\" d=\"M 603 3500 \n",
       "L 1178 3500 \n",
       "L 1178 0 \n",
       "L 603 0 \n",
       "L 603 3500 \n",
       "z\n",
       "M 603 4863 \n",
       "L 1178 4863 \n",
       "L 1178 4134 \n",
       "L 603 4134 \n",
       "L 603 4863 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-6b\" d=\"M 581 4863 \n",
       "L 1159 4863 \n",
       "L 1159 1991 \n",
       "L 2875 3500 \n",
       "L 3609 3500 \n",
       "L 1753 1863 \n",
       "L 3688 0 \n",
       "L 2938 0 \n",
       "L 1159 1709 \n",
       "L 1159 0 \n",
       "L 581 0 \n",
       "L 581 4863 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-6e\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-6f\" x=\"63.378906\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-20\" x=\"124.560547\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-6e\" x=\"156.347656\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-76\" x=\"219.726562\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-6c\" x=\"278.90625\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-69\" x=\"306.689453\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-6e\" x=\"334.472656\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-6b\" x=\"397.851562\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"patch_6\">\n",
       "    <path d=\"M 133.35 88.74 \n",
       "L 133.35 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_7\">\n",
       "    <path d=\"M 281.055882 88.74 \n",
       "L 281.055882 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_8\">\n",
       "    <path d=\"M 133.35 88.74 \n",
       "L 281.055882 88.74 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_9\">\n",
       "    <path d=\"M 133.35 7.2 \n",
       "L 281.055882 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "  </g>\n",
       "  <g id=\"axes_2\">\n",
       "   <g id=\"patch_10\">\n",
       "    <path d=\"M 310.597059 88.74 \n",
       "L 458.302941 88.74 \n",
       "L 458.302941 7.2 \n",
       "L 310.597059 7.2 \n",
       "z\n",
       "\" style=\"fill: #ffffff\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_11\">\n",
       "    <path d=\"M 310.597059 10.906364 \n",
       "L 451.269328 10.906364 \n",
       "L 451.269328 28.012657 \n",
       "L 310.597059 28.012657 \n",
       "z\n",
       "\" clip-path=\"url(#p6033b76dfc)\" style=\"fill: #e41a1c\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_12\">\n",
       "    <path d=\"M 310.597059 39.416853 \n",
       "L 437.586263 39.416853 \n",
       "L 437.586263 56.523147 \n",
       "L 310.597059 56.523147 \n",
       "z\n",
       "\" clip-path=\"url(#p6033b76dfc)\" style=\"fill: #377eb8\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_13\">\n",
       "    <path d=\"M 310.597059 67.927343 \n",
       "L 399.084939 67.927343 \n",
       "L 399.084939 85.033636 \n",
       "L 310.597059 85.033636 \n",
       "z\n",
       "\" clip-path=\"url(#p6033b76dfc)\" style=\"fill: #4daf4a\"/>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_3\">\n",
       "    <g id=\"xtick_5\">\n",
       "     <g id=\"line2d_8\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#me3c65d8a53\" x=\"310.597059\" y=\"88.74\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_9\">\n",
       "      <!-- 0 -->\n",
       "      <g transform=\"translate(307.415809 103.338438)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_6\">\n",
       "     <g id=\"line2d_9\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#me3c65d8a53\" x=\"377.325362\" y=\"88.74\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_10\">\n",
       "      <!-- 20 -->\n",
       "      <g transform=\"translate(370.962862 103.338438)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-32\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_7\">\n",
       "     <g id=\"line2d_10\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#me3c65d8a53\" x=\"444.053665\" y=\"88.74\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_11\">\n",
       "      <!-- 40 -->\n",
       "      <g transform=\"translate(437.691165 103.338438)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-34\" d=\"M 2419 4116 \n",
       "L 825 1625 \n",
       "L 2419 1625 \n",
       "L 2419 4116 \n",
       "z\n",
       "M 2253 4666 \n",
       "L 3047 4666 \n",
       "L 3047 1625 \n",
       "L 3713 1625 \n",
       "L 3713 1100 \n",
       "L 3047 1100 \n",
       "L 3047 0 \n",
       "L 2419 0 \n",
       "L 2419 1100 \n",
       "L 313 1100 \n",
       "L 313 1709 \n",
       "L 2253 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-34\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"text_12\">\n",
       "     <!-- per GPU TFLOPS -->\n",
       "     <g transform=\"translate(343.757812 117.016563)scale(0.1 -0.1)\">\n",
       "      <defs>\n",
       "       <path id=\"DejaVuSans-46\" d=\"M 628 4666 \n",
       "L 3309 4666 \n",
       "L 3309 4134 \n",
       "L 1259 4134 \n",
       "L 1259 2759 \n",
       "L 3109 2759 \n",
       "L 3109 2228 \n",
       "L 1259 2228 \n",
       "L 1259 0 \n",
       "L 628 0 \n",
       "L 628 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-4c\" d=\"M 628 4666 \n",
       "L 1259 4666 \n",
       "L 1259 531 \n",
       "L 3531 531 \n",
       "L 3531 0 \n",
       "L 628 0 \n",
       "L 628 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-4f\" d=\"M 2522 4238 \n",
       "Q 1834 4238 1429 3725 \n",
       "Q 1025 3213 1025 2328 \n",
       "Q 1025 1447 1429 934 \n",
       "Q 1834 422 2522 422 \n",
       "Q 3209 422 3611 934 \n",
       "Q 4013 1447 4013 2328 \n",
       "Q 4013 3213 3611 3725 \n",
       "Q 3209 4238 2522 4238 \n",
       "z\n",
       "M 2522 4750 \n",
       "Q 3503 4750 4090 4092 \n",
       "Q 4678 3434 4678 2328 \n",
       "Q 4678 1225 4090 567 \n",
       "Q 3503 -91 2522 -91 \n",
       "Q 1538 -91 948 565 \n",
       "Q 359 1222 359 2328 \n",
       "Q 359 3434 948 4092 \n",
       "Q 1538 4750 2522 4750 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "      </defs>\n",
       "      <use xlink:href=\"#DejaVuSans-70\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-65\" x=\"63.476562\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-72\" x=\"125\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-20\" x=\"166.113281\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-47\" x=\"197.900391\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-50\" x=\"275.390625\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-55\" x=\"335.693359\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-20\" x=\"408.886719\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-54\" x=\"440.673828\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-46\" x=\"501.757812\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-4c\" x=\"559.277344\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-4f\" x=\"611.365234\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-50\" x=\"690.076172\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-53\" x=\"750.378906\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_4\">\n",
       "    <g id=\"ytick_4\">\n",
       "     <g id=\"line2d_11\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m5946a68768\" x=\"310.597059\" y=\"19.45951\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_5\">\n",
       "     <g id=\"line2d_12\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m5946a68768\" x=\"310.597059\" y=\"47.97\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_6\">\n",
       "     <g id=\"line2d_13\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m5946a68768\" x=\"310.597059\" y=\"76.48049\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"patch_14\">\n",
       "    <path d=\"M 310.597059 88.74 \n",
       "L 310.597059 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_15\">\n",
       "    <path d=\"M 458.302941 88.74 \n",
       "L 458.302941 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_16\">\n",
       "    <path d=\"M 310.597059 88.74 \n",
       "L 458.302941 88.74 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_17\">\n",
       "    <path d=\"M 310.597059 7.2 \n",
       "L 458.302941 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "  </g>\n",
       "  <g id=\"axes_3\">\n",
       "   <g id=\"patch_18\">\n",
       "    <path d=\"M 487.844118 88.74 \n",
       "L 635.55 88.74 \n",
       "L 635.55 7.2 \n",
       "L 487.844118 7.2 \n",
       "z\n",
       "\" style=\"fill: #ffffff\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_19\">\n",
       "    <path d=\"M 487.844118 10.906364 \n",
       "L 628.516387 10.906364 \n",
       "L 628.516387 28.012657 \n",
       "L 487.844118 28.012657 \n",
       "z\n",
       "\" clip-path=\"url(#p4760f57490)\" style=\"fill: #e41a1c\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_20\">\n",
       "    <path d=\"M 487.844118 39.416853 \n",
       "L 628.516387 39.416853 \n",
       "L 628.516387 56.523147 \n",
       "L 487.844118 56.523147 \n",
       "z\n",
       "\" clip-path=\"url(#p4760f57490)\" style=\"fill: #377eb8\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_21\">\n",
       "    <path d=\"M 487.844118 67.927343 \n",
       "L 628.516387 67.927343 \n",
       "L 628.516387 85.033636 \n",
       "L 487.844118 85.033636 \n",
       "z\n",
       "\" clip-path=\"url(#p4760f57490)\" style=\"fill: #4daf4a\"/>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_5\">\n",
       "    <g id=\"xtick_8\">\n",
       "     <g id=\"line2d_14\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#me3c65d8a53\" x=\"487.844118\" y=\"88.74\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_13\">\n",
       "      <!-- 0 -->\n",
       "      <g transform=\"translate(484.662868 103.338438)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_9\">\n",
       "     <g id=\"line2d_15\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#me3c65d8a53\" x=\"553.472148\" y=\"88.74\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_14\">\n",
       "      <!-- 10 -->\n",
       "      <g transform=\"translate(547.109648 103.338438)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-31\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_10\">\n",
       "     <g id=\"line2d_16\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#me3c65d8a53\" x=\"619.100177\" y=\"88.74\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_15\">\n",
       "      <!-- 20 -->\n",
       "      <g transform=\"translate(612.737677 103.338438)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-32\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"text_16\">\n",
       "     <!-- per GPU memory (GB) -->\n",
       "     <g transform=\"translate(505.977528 117.016563)scale(0.1 -0.1)\">\n",
       "      <defs>\n",
       "       <path id=\"DejaVuSans-79\" d=\"M 2059 -325 \n",
       "Q 1816 -950 1584 -1140 \n",
       "Q 1353 -1331 966 -1331 \n",
       "L 506 -1331 \n",
       "L 506 -850 \n",
       "L 844 -850 \n",
       "Q 1081 -850 1212 -737 \n",
       "Q 1344 -625 1503 -206 \n",
       "L 1606 56 \n",
       "L 191 3500 \n",
       "L 800 3500 \n",
       "L 1894 763 \n",
       "L 2988 3500 \n",
       "L 3597 3500 \n",
       "L 2059 -325 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-28\" d=\"M 1984 4856 \n",
       "Q 1566 4138 1362 3434 \n",
       "Q 1159 2731 1159 2009 \n",
       "Q 1159 1288 1364 580 \n",
       "Q 1569 -128 1984 -844 \n",
       "L 1484 -844 \n",
       "Q 1016 -109 783 600 \n",
       "Q 550 1309 550 2009 \n",
       "Q 550 2706 781 3412 \n",
       "Q 1013 4119 1484 4856 \n",
       "L 1984 4856 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-42\" d=\"M 1259 2228 \n",
       "L 1259 519 \n",
       "L 2272 519 \n",
       "Q 2781 519 3026 730 \n",
       "Q 3272 941 3272 1375 \n",
       "Q 3272 1813 3026 2020 \n",
       "Q 2781 2228 2272 2228 \n",
       "L 1259 2228 \n",
       "z\n",
       "M 1259 4147 \n",
       "L 1259 2741 \n",
       "L 2194 2741 \n",
       "Q 2656 2741 2882 2914 \n",
       "Q 3109 3088 3109 3444 \n",
       "Q 3109 3797 2882 3972 \n",
       "Q 2656 4147 2194 4147 \n",
       "L 1259 4147 \n",
       "z\n",
       "M 628 4666 \n",
       "L 2241 4666 \n",
       "Q 2963 4666 3353 4366 \n",
       "Q 3744 4066 3744 3513 \n",
       "Q 3744 3084 3544 2831 \n",
       "Q 3344 2578 2956 2516 \n",
       "Q 3422 2416 3680 2098 \n",
       "Q 3938 1781 3938 1306 \n",
       "Q 3938 681 3513 340 \n",
       "Q 3088 0 2303 0 \n",
       "L 628 0 \n",
       "L 628 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-29\" d=\"M 513 4856 \n",
       "L 1013 4856 \n",
       "Q 1481 4119 1714 3412 \n",
       "Q 1947 2706 1947 2009 \n",
       "Q 1947 1309 1714 600 \n",
       "Q 1481 -109 1013 -844 \n",
       "L 513 -844 \n",
       "Q 928 -128 1133 580 \n",
       "Q 1338 1288 1338 2009 \n",
       "Q 1338 2731 1133 3434 \n",
       "Q 928 4138 513 4856 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "      </defs>\n",
       "      <use xlink:href=\"#DejaVuSans-70\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-65\" x=\"63.476562\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-72\" x=\"125\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-20\" x=\"166.113281\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-47\" x=\"197.900391\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-50\" x=\"275.390625\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-55\" x=\"335.693359\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-20\" x=\"408.886719\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-6d\" x=\"440.673828\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-65\" x=\"538.085938\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-6d\" x=\"599.609375\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-6f\" x=\"697.021484\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-72\" x=\"758.203125\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-79\" x=\"799.316406\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-20\" x=\"858.496094\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-28\" x=\"890.283203\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-47\" x=\"929.296875\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-42\" x=\"1006.787109\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-29\" x=\"1075.390625\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_6\">\n",
       "    <g id=\"ytick_7\">\n",
       "     <g id=\"line2d_17\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m5946a68768\" x=\"487.844118\" y=\"19.45951\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_8\">\n",
       "     <g id=\"line2d_18\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m5946a68768\" x=\"487.844118\" y=\"47.97\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_9\">\n",
       "     <g id=\"line2d_19\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m5946a68768\" x=\"487.844118\" y=\"76.48049\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"patch_22\">\n",
       "    <path d=\"M 487.844118 88.74 \n",
       "L 487.844118 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_23\">\n",
       "    <path d=\"M 635.55 88.74 \n",
       "L 635.55 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_24\">\n",
       "    <path d=\"M 487.844118 88.74 \n",
       "L 635.55 88.74 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_25\">\n",
       "    <path d=\"M 487.844118 7.2 \n",
       "L 635.55 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "  </g>\n",
       " </g>\n",
       " <defs>\n",
       "  <clipPath id=\"pe9fdbfc80d\">\n",
       "   <rect x=\"133.35\" y=\"7.2\" width=\"147.705882\" height=\"81.54\"/>\n",
       "  </clipPath>\n",
       "  <clipPath id=\"p6033b76dfc\">\n",
       "   <rect x=\"310.597059\" y=\"7.2\" width=\"147.705882\" height=\"81.54\"/>\n",
       "  </clipPath>\n",
       "  <clipPath id=\"p4760f57490\">\n",
       "   <rect x=\"487.844118\" y=\"7.2\" width=\"147.705882\" height=\"81.54\"/>\n",
       "  </clipPath>\n",
       " </defs>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<Figure size 648x108 with 3 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "os.environ[\"NCCL_P2P_DISABLE\"] = \"1\"\n",
    "dp_gpt2_nonvlink = megatron_gpt(Exp(\n",
    "    \"Megatron GPT2, 2 GPUs\\nno nvlink\", \"gpt2-medium\", 5, bf16=True, gpus='0,1'))\n",
    "os.environ[\"NCCL_P2P_DISABLE\"] = \"0\"\n",
    "compare([mega_gpt2, dp_gpt2, dp_gpt2_nonvlink])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b345513b",
   "metadata": {},
   "source": [
    "One improvement idea is using gradient accumulation to reduce communication frequency."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "01f8f956",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Time breakdown\t\t: forward+backward 0.95, communication 0.03, optimizer 0.02\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       "  \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<svg xmlns:xlink=\"http://www.w3.org/1999/xlink\" width=\"642.75pt\" height=\"153.47625pt\" viewBox=\"0 0 642.75 153.47625\" xmlns=\"http://www.w3.org/2000/svg\" version=\"1.1\">\n",
       " <metadata>\n",
       "  <rdf:RDF xmlns:dc=\"http://purl.org/dc/elements/1.1/\" xmlns:cc=\"http://creativecommons.org/ns#\" xmlns:rdf=\"http://www.w3.org/1999/02/22-rdf-syntax-ns#\">\n",
       "   <cc:Work>\n",
       "    <dc:type rdf:resource=\"http://purl.org/dc/dcmitype/StillImage\"/>\n",
       "    <dc:date>2022-09-02T19:47:21.441926</dc:date>\n",
       "    <dc:format>image/svg+xml</dc:format>\n",
       "    <dc:creator>\n",
       "     <cc:Agent>\n",
       "      <dc:title>Matplotlib v3.5.2, https://matplotlib.org/</dc:title>\n",
       "     </cc:Agent>\n",
       "    </dc:creator>\n",
       "   </cc:Work>\n",
       "  </rdf:RDF>\n",
       " </metadata>\n",
       " <defs>\n",
       "  <style type=\"text/css\">*{stroke-linejoin: round; stroke-linecap: butt}</style>\n",
       " </defs>\n",
       " <g id=\"figure_1\">\n",
       "  <g id=\"patch_1\">\n",
       "   <path d=\"M 0 153.47625 \n",
       "L 642.75 153.47625 \n",
       "L 642.75 0 \n",
       "L 0 0 \n",
       "L 0 153.47625 \n",
       "z\n",
       "\" style=\"fill: none\"/>\n",
       "  </g>\n",
       "  <g id=\"axes_1\">\n",
       "   <g id=\"patch_2\">\n",
       "    <path d=\"M 133.35 115.92 \n",
       "L 281.055882 115.92 \n",
       "L 281.055882 7.2 \n",
       "L 133.35 7.2 \n",
       "z\n",
       "\" style=\"fill: #ffffff\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_3\">\n",
       "    <path d=\"M 133.35 12.141818 \n",
       "L 200.960669 12.141818 \n",
       "L 200.960669 28.614545 \n",
       "L 133.35 28.614545 \n",
       "z\n",
       "\" clip-path=\"url(#p5f50d89725)\" style=\"fill: #e41a1c\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_4\">\n",
       "    <path d=\"M 133.35 39.596364 \n",
       "L 255.41848 39.596364 \n",
       "L 255.41848 56.069091 \n",
       "L 133.35 56.069091 \n",
       "z\n",
       "\" clip-path=\"url(#p5f50d89725)\" style=\"fill: #377eb8\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_5\">\n",
       "    <path d=\"M 133.35 67.050909 \n",
       "L 218.40905 67.050909 \n",
       "L 218.40905 83.523636 \n",
       "L 133.35 83.523636 \n",
       "z\n",
       "\" clip-path=\"url(#p5f50d89725)\" style=\"fill: #4daf4a\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_6\">\n",
       "    <path d=\"M 133.35 94.505455 \n",
       "L 274.022269 94.505455 \n",
       "L 274.022269 110.978182 \n",
       "L 133.35 110.978182 \n",
       "z\n",
       "\" clip-path=\"url(#p5f50d89725)\" style=\"fill: #984ea3\"/>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_1\">\n",
       "    <g id=\"xtick_1\">\n",
       "     <g id=\"line2d_1\">\n",
       "      <defs>\n",
       "       <path id=\"m19d1fdb71d\" d=\"M 0 0 \n",
       "L 0 3.5 \n",
       "\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </defs>\n",
       "      <g>\n",
       "       <use xlink:href=\"#m19d1fdb71d\" x=\"133.35\" y=\"115.92\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_1\">\n",
       "      <!-- 0 -->\n",
       "      <g transform=\"translate(130.16875 130.518438)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-30\" d=\"M 2034 4250 \n",
       "Q 1547 4250 1301 3770 \n",
       "Q 1056 3291 1056 2328 \n",
       "Q 1056 1369 1301 889 \n",
       "Q 1547 409 2034 409 \n",
       "Q 2525 409 2770 889 \n",
       "Q 3016 1369 3016 2328 \n",
       "Q 3016 3291 2770 3770 \n",
       "Q 2525 4250 2034 4250 \n",
       "z\n",
       "M 2034 4750 \n",
       "Q 2819 4750 3233 4129 \n",
       "Q 3647 3509 3647 2328 \n",
       "Q 3647 1150 3233 529 \n",
       "Q 2819 -91 2034 -91 \n",
       "Q 1250 -91 836 529 \n",
       "Q 422 1150 422 2328 \n",
       "Q 422 3509 836 4129 \n",
       "Q 1250 4750 2034 4750 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_2\">\n",
       "     <g id=\"line2d_2\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m19d1fdb71d\" x=\"173.132118\" y=\"115.92\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_2\">\n",
       "      <!-- 10 -->\n",
       "      <g transform=\"translate(166.769618 130.518438)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-31\" d=\"M 794 531 \n",
       "L 1825 531 \n",
       "L 1825 4091 \n",
       "L 703 3866 \n",
       "L 703 4441 \n",
       "L 1819 4666 \n",
       "L 2450 4666 \n",
       "L 2450 531 \n",
       "L 3481 531 \n",
       "L 3481 0 \n",
       "L 794 0 \n",
       "L 794 531 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-31\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_3\">\n",
       "     <g id=\"line2d_3\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m19d1fdb71d\" x=\"212.914235\" y=\"115.92\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_3\">\n",
       "      <!-- 20 -->\n",
       "      <g transform=\"translate(206.551735 130.518438)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-32\" d=\"M 1228 531 \n",
       "L 3431 531 \n",
       "L 3431 0 \n",
       "L 469 0 \n",
       "L 469 531 \n",
       "Q 828 903 1448 1529 \n",
       "Q 2069 2156 2228 2338 \n",
       "Q 2531 2678 2651 2914 \n",
       "Q 2772 3150 2772 3378 \n",
       "Q 2772 3750 2511 3984 \n",
       "Q 2250 4219 1831 4219 \n",
       "Q 1534 4219 1204 4116 \n",
       "Q 875 4013 500 3803 \n",
       "L 500 4441 \n",
       "Q 881 4594 1212 4672 \n",
       "Q 1544 4750 1819 4750 \n",
       "Q 2544 4750 2975 4387 \n",
       "Q 3406 4025 3406 3419 \n",
       "Q 3406 3131 3298 2873 \n",
       "Q 3191 2616 2906 2266 \n",
       "Q 2828 2175 2409 1742 \n",
       "Q 1991 1309 1228 531 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-32\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_4\">\n",
       "     <g id=\"line2d_4\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m19d1fdb71d\" x=\"252.696353\" y=\"115.92\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_4\">\n",
       "      <!-- 30 -->\n",
       "      <g transform=\"translate(246.333853 130.518438)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-33\" d=\"M 2597 2516 \n",
       "Q 3050 2419 3304 2112 \n",
       "Q 3559 1806 3559 1356 \n",
       "Q 3559 666 3084 287 \n",
       "Q 2609 -91 1734 -91 \n",
       "Q 1441 -91 1130 -33 \n",
       "Q 819 25 488 141 \n",
       "L 488 750 \n",
       "Q 750 597 1062 519 \n",
       "Q 1375 441 1716 441 \n",
       "Q 2309 441 2620 675 \n",
       "Q 2931 909 2931 1356 \n",
       "Q 2931 1769 2642 2001 \n",
       "Q 2353 2234 1838 2234 \n",
       "L 1294 2234 \n",
       "L 1294 2753 \n",
       "L 1863 2753 \n",
       "Q 2328 2753 2575 2939 \n",
       "Q 2822 3125 2822 3475 \n",
       "Q 2822 3834 2567 4026 \n",
       "Q 2313 4219 1838 4219 \n",
       "Q 1578 4219 1281 4162 \n",
       "Q 984 4106 628 3988 \n",
       "L 628 4550 \n",
       "Q 988 4650 1302 4700 \n",
       "Q 1616 4750 1894 4750 \n",
       "Q 2613 4750 3031 4423 \n",
       "Q 3450 4097 3450 3541 \n",
       "Q 3450 3153 3228 2886 \n",
       "Q 3006 2619 2597 2516 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-33\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"text_5\">\n",
       "     <!-- Samples / sec -->\n",
       "     <g transform=\"translate(172.556847 144.196562)scale(0.1 -0.1)\">\n",
       "      <defs>\n",
       "       <path id=\"DejaVuSans-53\" d=\"M 3425 4513 \n",
       "L 3425 3897 \n",
       "Q 3066 4069 2747 4153 \n",
       "Q 2428 4238 2131 4238 \n",
       "Q 1616 4238 1336 4038 \n",
       "Q 1056 3838 1056 3469 \n",
       "Q 1056 3159 1242 3001 \n",
       "Q 1428 2844 1947 2747 \n",
       "L 2328 2669 \n",
       "Q 3034 2534 3370 2195 \n",
       "Q 3706 1856 3706 1288 \n",
       "Q 3706 609 3251 259 \n",
       "Q 2797 -91 1919 -91 \n",
       "Q 1588 -91 1214 -16 \n",
       "Q 841 59 441 206 \n",
       "L 441 856 \n",
       "Q 825 641 1194 531 \n",
       "Q 1563 422 1919 422 \n",
       "Q 2459 422 2753 634 \n",
       "Q 3047 847 3047 1241 \n",
       "Q 3047 1584 2836 1778 \n",
       "Q 2625 1972 2144 2069 \n",
       "L 1759 2144 \n",
       "Q 1053 2284 737 2584 \n",
       "Q 422 2884 422 3419 \n",
       "Q 422 4038 858 4394 \n",
       "Q 1294 4750 2059 4750 \n",
       "Q 2388 4750 2728 4690 \n",
       "Q 3069 4631 3425 4513 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-61\" d=\"M 2194 1759 \n",
       "Q 1497 1759 1228 1600 \n",
       "Q 959 1441 959 1056 \n",
       "Q 959 750 1161 570 \n",
       "Q 1363 391 1709 391 \n",
       "Q 2188 391 2477 730 \n",
       "Q 2766 1069 2766 1631 \n",
       "L 2766 1759 \n",
       "L 2194 1759 \n",
       "z\n",
       "M 3341 1997 \n",
       "L 3341 0 \n",
       "L 2766 0 \n",
       "L 2766 531 \n",
       "Q 2569 213 2275 61 \n",
       "Q 1981 -91 1556 -91 \n",
       "Q 1019 -91 701 211 \n",
       "Q 384 513 384 1019 \n",
       "Q 384 1609 779 1909 \n",
       "Q 1175 2209 1959 2209 \n",
       "L 2766 2209 \n",
       "L 2766 2266 \n",
       "Q 2766 2663 2505 2880 \n",
       "Q 2244 3097 1772 3097 \n",
       "Q 1472 3097 1187 3025 \n",
       "Q 903 2953 641 2809 \n",
       "L 641 3341 \n",
       "Q 956 3463 1253 3523 \n",
       "Q 1550 3584 1831 3584 \n",
       "Q 2591 3584 2966 3190 \n",
       "Q 3341 2797 3341 1997 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-6d\" d=\"M 3328 2828 \n",
       "Q 3544 3216 3844 3400 \n",
       "Q 4144 3584 4550 3584 \n",
       "Q 5097 3584 5394 3201 \n",
       "Q 5691 2819 5691 2113 \n",
       "L 5691 0 \n",
       "L 5113 0 \n",
       "L 5113 2094 \n",
       "Q 5113 2597 4934 2840 \n",
       "Q 4756 3084 4391 3084 \n",
       "Q 3944 3084 3684 2787 \n",
       "Q 3425 2491 3425 1978 \n",
       "L 3425 0 \n",
       "L 2847 0 \n",
       "L 2847 2094 \n",
       "Q 2847 2600 2669 2842 \n",
       "Q 2491 3084 2119 3084 \n",
       "Q 1678 3084 1418 2786 \n",
       "Q 1159 2488 1159 1978 \n",
       "L 1159 0 \n",
       "L 581 0 \n",
       "L 581 3500 \n",
       "L 1159 3500 \n",
       "L 1159 2956 \n",
       "Q 1356 3278 1631 3431 \n",
       "Q 1906 3584 2284 3584 \n",
       "Q 2666 3584 2933 3390 \n",
       "Q 3200 3197 3328 2828 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-70\" d=\"M 1159 525 \n",
       "L 1159 -1331 \n",
       "L 581 -1331 \n",
       "L 581 3500 \n",
       "L 1159 3500 \n",
       "L 1159 2969 \n",
       "Q 1341 3281 1617 3432 \n",
       "Q 1894 3584 2278 3584 \n",
       "Q 2916 3584 3314 3078 \n",
       "Q 3713 2572 3713 1747 \n",
       "Q 3713 922 3314 415 \n",
       "Q 2916 -91 2278 -91 \n",
       "Q 1894 -91 1617 61 \n",
       "Q 1341 213 1159 525 \n",
       "z\n",
       "M 3116 1747 \n",
       "Q 3116 2381 2855 2742 \n",
       "Q 2594 3103 2138 3103 \n",
       "Q 1681 3103 1420 2742 \n",
       "Q 1159 2381 1159 1747 \n",
       "Q 1159 1113 1420 752 \n",
       "Q 1681 391 2138 391 \n",
       "Q 2594 391 2855 752 \n",
       "Q 3116 1113 3116 1747 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-6c\" d=\"M 603 4863 \n",
       "L 1178 4863 \n",
       "L 1178 0 \n",
       "L 603 0 \n",
       "L 603 4863 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-65\" d=\"M 3597 1894 \n",
       "L 3597 1613 \n",
       "L 953 1613 \n",
       "Q 991 1019 1311 708 \n",
       "Q 1631 397 2203 397 \n",
       "Q 2534 397 2845 478 \n",
       "Q 3156 559 3463 722 \n",
       "L 3463 178 \n",
       "Q 3153 47 2828 -22 \n",
       "Q 2503 -91 2169 -91 \n",
       "Q 1331 -91 842 396 \n",
       "Q 353 884 353 1716 \n",
       "Q 353 2575 817 3079 \n",
       "Q 1281 3584 2069 3584 \n",
       "Q 2775 3584 3186 3129 \n",
       "Q 3597 2675 3597 1894 \n",
       "z\n",
       "M 3022 2063 \n",
       "Q 3016 2534 2758 2815 \n",
       "Q 2500 3097 2075 3097 \n",
       "Q 1594 3097 1305 2825 \n",
       "Q 1016 2553 972 2059 \n",
       "L 3022 2063 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-73\" d=\"M 2834 3397 \n",
       "L 2834 2853 \n",
       "Q 2591 2978 2328 3040 \n",
       "Q 2066 3103 1784 3103 \n",
       "Q 1356 3103 1142 2972 \n",
       "Q 928 2841 928 2578 \n",
       "Q 928 2378 1081 2264 \n",
       "Q 1234 2150 1697 2047 \n",
       "L 1894 2003 \n",
       "Q 2506 1872 2764 1633 \n",
       "Q 3022 1394 3022 966 \n",
       "Q 3022 478 2636 193 \n",
       "Q 2250 -91 1575 -91 \n",
       "Q 1294 -91 989 -36 \n",
       "Q 684 19 347 128 \n",
       "L 347 722 \n",
       "Q 666 556 975 473 \n",
       "Q 1284 391 1588 391 \n",
       "Q 1994 391 2212 530 \n",
       "Q 2431 669 2431 922 \n",
       "Q 2431 1156 2273 1281 \n",
       "Q 2116 1406 1581 1522 \n",
       "L 1381 1569 \n",
       "Q 847 1681 609 1914 \n",
       "Q 372 2147 372 2553 \n",
       "Q 372 3047 722 3315 \n",
       "Q 1072 3584 1716 3584 \n",
       "Q 2034 3584 2315 3537 \n",
       "Q 2597 3491 2834 3397 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-20\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-2f\" d=\"M 1625 4666 \n",
       "L 2156 4666 \n",
       "L 531 -594 \n",
       "L 0 -594 \n",
       "L 1625 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-63\" d=\"M 3122 3366 \n",
       "L 3122 2828 \n",
       "Q 2878 2963 2633 3030 \n",
       "Q 2388 3097 2138 3097 \n",
       "Q 1578 3097 1268 2742 \n",
       "Q 959 2388 959 1747 \n",
       "Q 959 1106 1268 751 \n",
       "Q 1578 397 2138 397 \n",
       "Q 2388 397 2633 464 \n",
       "Q 2878 531 3122 666 \n",
       "L 3122 134 \n",
       "Q 2881 22 2623 -34 \n",
       "Q 2366 -91 2075 -91 \n",
       "Q 1284 -91 818 406 \n",
       "Q 353 903 353 1747 \n",
       "Q 353 2603 823 3093 \n",
       "Q 1294 3584 2113 3584 \n",
       "Q 2378 3584 2631 3529 \n",
       "Q 2884 3475 3122 3366 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "      </defs>\n",
       "      <use xlink:href=\"#DejaVuSans-53\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-61\" x=\"63.476562\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-6d\" x=\"124.755859\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-70\" x=\"222.167969\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-6c\" x=\"285.644531\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-65\" x=\"313.427734\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-73\" x=\"374.951172\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-20\" x=\"427.050781\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-2f\" x=\"458.837891\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-20\" x=\"492.529297\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-73\" x=\"524.316406\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-65\" x=\"576.416016\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-63\" x=\"637.939453\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_2\">\n",
       "    <g id=\"ytick_1\">\n",
       "     <g id=\"line2d_5\">\n",
       "      <defs>\n",
       "       <path id=\"m0a0c167960\" d=\"M 0 0 \n",
       "L -3.5 0 \n",
       "\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </defs>\n",
       "      <g>\n",
       "       <use xlink:href=\"#m0a0c167960\" x=\"133.35\" y=\"20.378182\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_6\">\n",
       "      <!-- Megatron GPT2 -->\n",
       "      <g transform=\"translate(49.403125 24.177401)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-4d\" d=\"M 628 4666 \n",
       "L 1569 4666 \n",
       "L 2759 1491 \n",
       "L 3956 4666 \n",
       "L 4897 4666 \n",
       "L 4897 0 \n",
       "L 4281 0 \n",
       "L 4281 4097 \n",
       "L 3078 897 \n",
       "L 2444 897 \n",
       "L 1241 4097 \n",
       "L 1241 0 \n",
       "L 628 0 \n",
       "L 628 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-67\" d=\"M 2906 1791 \n",
       "Q 2906 2416 2648 2759 \n",
       "Q 2391 3103 1925 3103 \n",
       "Q 1463 3103 1205 2759 \n",
       "Q 947 2416 947 1791 \n",
       "Q 947 1169 1205 825 \n",
       "Q 1463 481 1925 481 \n",
       "Q 2391 481 2648 825 \n",
       "Q 2906 1169 2906 1791 \n",
       "z\n",
       "M 3481 434 \n",
       "Q 3481 -459 3084 -895 \n",
       "Q 2688 -1331 1869 -1331 \n",
       "Q 1566 -1331 1297 -1286 \n",
       "Q 1028 -1241 775 -1147 \n",
       "L 775 -588 \n",
       "Q 1028 -725 1275 -790 \n",
       "Q 1522 -856 1778 -856 \n",
       "Q 2344 -856 2625 -561 \n",
       "Q 2906 -266 2906 331 \n",
       "L 2906 616 \n",
       "Q 2728 306 2450 153 \n",
       "Q 2172 0 1784 0 \n",
       "Q 1141 0 747 490 \n",
       "Q 353 981 353 1791 \n",
       "Q 353 2603 747 3093 \n",
       "Q 1141 3584 1784 3584 \n",
       "Q 2172 3584 2450 3431 \n",
       "Q 2728 3278 2906 2969 \n",
       "L 2906 3500 \n",
       "L 3481 3500 \n",
       "L 3481 434 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-74\" d=\"M 1172 4494 \n",
       "L 1172 3500 \n",
       "L 2356 3500 \n",
       "L 2356 3053 \n",
       "L 1172 3053 \n",
       "L 1172 1153 \n",
       "Q 1172 725 1289 603 \n",
       "Q 1406 481 1766 481 \n",
       "L 2356 481 \n",
       "L 2356 0 \n",
       "L 1766 0 \n",
       "Q 1100 0 847 248 \n",
       "Q 594 497 594 1153 \n",
       "L 594 3053 \n",
       "L 172 3053 \n",
       "L 172 3500 \n",
       "L 594 3500 \n",
       "L 594 4494 \n",
       "L 1172 4494 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-72\" d=\"M 2631 2963 \n",
       "Q 2534 3019 2420 3045 \n",
       "Q 2306 3072 2169 3072 \n",
       "Q 1681 3072 1420 2755 \n",
       "Q 1159 2438 1159 1844 \n",
       "L 1159 0 \n",
       "L 581 0 \n",
       "L 581 3500 \n",
       "L 1159 3500 \n",
       "L 1159 2956 \n",
       "Q 1341 3275 1631 3429 \n",
       "Q 1922 3584 2338 3584 \n",
       "Q 2397 3584 2469 3576 \n",
       "Q 2541 3569 2628 3553 \n",
       "L 2631 2963 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-6f\" d=\"M 1959 3097 \n",
       "Q 1497 3097 1228 2736 \n",
       "Q 959 2375 959 1747 \n",
       "Q 959 1119 1226 758 \n",
       "Q 1494 397 1959 397 \n",
       "Q 2419 397 2687 759 \n",
       "Q 2956 1122 2956 1747 \n",
       "Q 2956 2369 2687 2733 \n",
       "Q 2419 3097 1959 3097 \n",
       "z\n",
       "M 1959 3584 \n",
       "Q 2709 3584 3137 3096 \n",
       "Q 3566 2609 3566 1747 \n",
       "Q 3566 888 3137 398 \n",
       "Q 2709 -91 1959 -91 \n",
       "Q 1206 -91 779 398 \n",
       "Q 353 888 353 1747 \n",
       "Q 353 2609 779 3096 \n",
       "Q 1206 3584 1959 3584 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-6e\" d=\"M 3513 2113 \n",
       "L 3513 0 \n",
       "L 2938 0 \n",
       "L 2938 2094 \n",
       "Q 2938 2591 2744 2837 \n",
       "Q 2550 3084 2163 3084 \n",
       "Q 1697 3084 1428 2787 \n",
       "Q 1159 2491 1159 1978 \n",
       "L 1159 0 \n",
       "L 581 0 \n",
       "L 581 3500 \n",
       "L 1159 3500 \n",
       "L 1159 2956 \n",
       "Q 1366 3272 1645 3428 \n",
       "Q 1925 3584 2291 3584 \n",
       "Q 2894 3584 3203 3211 \n",
       "Q 3513 2838 3513 2113 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-47\" d=\"M 3809 666 \n",
       "L 3809 1919 \n",
       "L 2778 1919 \n",
       "L 2778 2438 \n",
       "L 4434 2438 \n",
       "L 4434 434 \n",
       "Q 4069 175 3628 42 \n",
       "Q 3188 -91 2688 -91 \n",
       "Q 1594 -91 976 548 \n",
       "Q 359 1188 359 2328 \n",
       "Q 359 3472 976 4111 \n",
       "Q 1594 4750 2688 4750 \n",
       "Q 3144 4750 3555 4637 \n",
       "Q 3966 4525 4313 4306 \n",
       "L 4313 3634 \n",
       "Q 3963 3931 3569 4081 \n",
       "Q 3175 4231 2741 4231 \n",
       "Q 1884 4231 1454 3753 \n",
       "Q 1025 3275 1025 2328 \n",
       "Q 1025 1384 1454 906 \n",
       "Q 1884 428 2741 428 \n",
       "Q 3075 428 3337 486 \n",
       "Q 3600 544 3809 666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-50\" d=\"M 1259 4147 \n",
       "L 1259 2394 \n",
       "L 2053 2394 \n",
       "Q 2494 2394 2734 2622 \n",
       "Q 2975 2850 2975 3272 \n",
       "Q 2975 3691 2734 3919 \n",
       "Q 2494 4147 2053 4147 \n",
       "L 1259 4147 \n",
       "z\n",
       "M 628 4666 \n",
       "L 2053 4666 \n",
       "Q 2838 4666 3239 4311 \n",
       "Q 3641 3956 3641 3272 \n",
       "Q 3641 2581 3239 2228 \n",
       "Q 2838 1875 2053 1875 \n",
       "L 1259 1875 \n",
       "L 1259 0 \n",
       "L 628 0 \n",
       "L 628 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-54\" d=\"M -19 4666 \n",
       "L 3928 4666 \n",
       "L 3928 4134 \n",
       "L 2272 4134 \n",
       "L 2272 0 \n",
       "L 1638 0 \n",
       "L 1638 4134 \n",
       "L -19 4134 \n",
       "L -19 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-4d\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-65\" x=\"86.279297\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-67\" x=\"147.802734\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-61\" x=\"211.279297\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-74\" x=\"272.558594\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-72\" x=\"311.767578\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-6f\" x=\"350.630859\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-6e\" x=\"411.8125\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-20\" x=\"475.191406\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-47\" x=\"506.978516\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-50\" x=\"584.46875\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-54\" x=\"644.771484\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-32\" x=\"705.855469\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_2\">\n",
       "     <g id=\"line2d_6\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m0a0c167960\" x=\"133.35\" y=\"47.832727\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_7\">\n",
       "      <!-- Megatron GPT2, 2 GPUs -->\n",
       "      <g transform=\"translate(7.2 51.631946)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-2c\" d=\"M 750 794 \n",
       "L 1409 794 \n",
       "L 1409 256 \n",
       "L 897 -744 \n",
       "L 494 -744 \n",
       "L 750 256 \n",
       "L 750 794 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-55\" d=\"M 556 4666 \n",
       "L 1191 4666 \n",
       "L 1191 1831 \n",
       "Q 1191 1081 1462 751 \n",
       "Q 1734 422 2344 422 \n",
       "Q 2950 422 3222 751 \n",
       "Q 3494 1081 3494 1831 \n",
       "L 3494 4666 \n",
       "L 4128 4666 \n",
       "L 4128 1753 \n",
       "Q 4128 841 3676 375 \n",
       "Q 3225 -91 2344 -91 \n",
       "Q 1459 -91 1007 375 \n",
       "Q 556 841 556 1753 \n",
       "L 556 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-4d\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-65\" x=\"86.279297\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-67\" x=\"147.802734\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-61\" x=\"211.279297\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-74\" x=\"272.558594\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-72\" x=\"311.767578\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-6f\" x=\"350.630859\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-6e\" x=\"411.8125\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-20\" x=\"475.191406\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-47\" x=\"506.978516\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-50\" x=\"584.46875\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-54\" x=\"644.771484\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-32\" x=\"705.855469\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-2c\" x=\"769.478516\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-20\" x=\"801.265625\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-32\" x=\"833.052734\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-20\" x=\"896.675781\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-47\" x=\"928.462891\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-50\" x=\"1005.953125\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-55\" x=\"1066.255859\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-73\" x=\"1139.449219\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_3\">\n",
       "     <g id=\"line2d_7\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m0a0c167960\" x=\"133.35\" y=\"75.287273\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_8\">\n",
       "      <!-- Megatron GPT2, 2 GPUs -->\n",
       "      <g transform=\"translate(7.2 74.247429)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-4d\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-65\" x=\"86.279297\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-67\" x=\"147.802734\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-61\" x=\"211.279297\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-74\" x=\"272.558594\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-72\" x=\"311.767578\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-6f\" x=\"350.630859\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-6e\" x=\"411.8125\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-20\" x=\"475.191406\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-47\" x=\"506.978516\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-50\" x=\"584.46875\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-54\" x=\"644.771484\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-32\" x=\"705.855469\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-2c\" x=\"769.478516\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-20\" x=\"801.265625\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-32\" x=\"833.052734\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-20\" x=\"896.675781\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-47\" x=\"928.462891\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-50\" x=\"1005.953125\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-55\" x=\"1066.255859\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-73\" x=\"1139.449219\"/>\n",
       "      </g>\n",
       "      <!-- no nvlink -->\n",
       "      <g transform=\"translate(80.775 85.445241)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-76\" d=\"M 191 3500 \n",
       "L 800 3500 \n",
       "L 1894 563 \n",
       "L 2988 3500 \n",
       "L 3597 3500 \n",
       "L 2284 0 \n",
       "L 1503 0 \n",
       "L 191 3500 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-69\" d=\"M 603 3500 \n",
       "L 1178 3500 \n",
       "L 1178 0 \n",
       "L 603 0 \n",
       "L 603 3500 \n",
       "z\n",
       "M 603 4863 \n",
       "L 1178 4863 \n",
       "L 1178 4134 \n",
       "L 603 4134 \n",
       "L 603 4863 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-6b\" d=\"M 581 4863 \n",
       "L 1159 4863 \n",
       "L 1159 1991 \n",
       "L 2875 3500 \n",
       "L 3609 3500 \n",
       "L 1753 1863 \n",
       "L 3688 0 \n",
       "L 2938 0 \n",
       "L 1159 1709 \n",
       "L 1159 0 \n",
       "L 581 0 \n",
       "L 581 4863 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-6e\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-6f\" x=\"63.378906\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-20\" x=\"124.560547\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-6e\" x=\"156.347656\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-76\" x=\"219.726562\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-6c\" x=\"278.90625\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-69\" x=\"306.689453\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-6e\" x=\"334.472656\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-6b\" x=\"397.851562\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_4\">\n",
       "     <g id=\"line2d_8\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m0a0c167960\" x=\"133.35\" y=\"102.741818\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_9\">\n",
       "      <!-- Megatron GPT2, 2 GPUs -->\n",
       "      <g transform=\"translate(7.2 101.701974)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-4d\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-65\" x=\"86.279297\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-67\" x=\"147.802734\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-61\" x=\"211.279297\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-74\" x=\"272.558594\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-72\" x=\"311.767578\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-6f\" x=\"350.630859\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-6e\" x=\"411.8125\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-20\" x=\"475.191406\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-47\" x=\"506.978516\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-50\" x=\"584.46875\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-54\" x=\"644.771484\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-32\" x=\"705.855469\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-2c\" x=\"769.478516\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-20\" x=\"801.265625\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-32\" x=\"833.052734\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-20\" x=\"896.675781\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-47\" x=\"928.462891\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-50\" x=\"1005.953125\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-55\" x=\"1066.255859\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-73\" x=\"1139.449219\"/>\n",
       "      </g>\n",
       "      <!-- grad_accum=4 -->\n",
       "      <g transform=\"translate(50.46875 112.899787)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-64\" d=\"M 2906 2969 \n",
       "L 2906 4863 \n",
       "L 3481 4863 \n",
       "L 3481 0 \n",
       "L 2906 0 \n",
       "L 2906 525 \n",
       "Q 2725 213 2448 61 \n",
       "Q 2172 -91 1784 -91 \n",
       "Q 1150 -91 751 415 \n",
       "Q 353 922 353 1747 \n",
       "Q 353 2572 751 3078 \n",
       "Q 1150 3584 1784 3584 \n",
       "Q 2172 3584 2448 3432 \n",
       "Q 2725 3281 2906 2969 \n",
       "z\n",
       "M 947 1747 \n",
       "Q 947 1113 1208 752 \n",
       "Q 1469 391 1925 391 \n",
       "Q 2381 391 2643 752 \n",
       "Q 2906 1113 2906 1747 \n",
       "Q 2906 2381 2643 2742 \n",
       "Q 2381 3103 1925 3103 \n",
       "Q 1469 3103 1208 2742 \n",
       "Q 947 2381 947 1747 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-5f\" d=\"M 3263 -1063 \n",
       "L 3263 -1509 \n",
       "L -63 -1509 \n",
       "L -63 -1063 \n",
       "L 3263 -1063 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-75\" d=\"M 544 1381 \n",
       "L 544 3500 \n",
       "L 1119 3500 \n",
       "L 1119 1403 \n",
       "Q 1119 906 1312 657 \n",
       "Q 1506 409 1894 409 \n",
       "Q 2359 409 2629 706 \n",
       "Q 2900 1003 2900 1516 \n",
       "L 2900 3500 \n",
       "L 3475 3500 \n",
       "L 3475 0 \n",
       "L 2900 0 \n",
       "L 2900 538 \n",
       "Q 2691 219 2414 64 \n",
       "Q 2138 -91 1772 -91 \n",
       "Q 1169 -91 856 284 \n",
       "Q 544 659 544 1381 \n",
       "z\n",
       "M 1991 3584 \n",
       "L 1991 3584 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-3d\" d=\"M 678 2906 \n",
       "L 4684 2906 \n",
       "L 4684 2381 \n",
       "L 678 2381 \n",
       "L 678 2906 \n",
       "z\n",
       "M 678 1631 \n",
       "L 4684 1631 \n",
       "L 4684 1100 \n",
       "L 678 1100 \n",
       "L 678 1631 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-34\" d=\"M 2419 4116 \n",
       "L 825 1625 \n",
       "L 2419 1625 \n",
       "L 2419 4116 \n",
       "z\n",
       "M 2253 4666 \n",
       "L 3047 4666 \n",
       "L 3047 1625 \n",
       "L 3713 1625 \n",
       "L 3713 1100 \n",
       "L 3047 1100 \n",
       "L 3047 0 \n",
       "L 2419 0 \n",
       "L 2419 1100 \n",
       "L 313 1100 \n",
       "L 313 1709 \n",
       "L 2253 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-67\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-72\" x=\"63.476562\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-61\" x=\"104.589844\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-64\" x=\"165.869141\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-5f\" x=\"229.345703\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-61\" x=\"279.345703\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-63\" x=\"340.625\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-63\" x=\"395.605469\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-75\" x=\"450.585938\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-6d\" x=\"513.964844\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-3d\" x=\"611.376953\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-34\" x=\"695.166016\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"patch_7\">\n",
       "    <path d=\"M 133.35 115.92 \n",
       "L 133.35 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_8\">\n",
       "    <path d=\"M 281.055882 115.92 \n",
       "L 281.055882 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_9\">\n",
       "    <path d=\"M 133.35 115.92 \n",
       "L 281.055882 115.92 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_10\">\n",
       "    <path d=\"M 133.35 7.2 \n",
       "L 281.055882 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "  </g>\n",
       "  <g id=\"axes_2\">\n",
       "   <g id=\"patch_11\">\n",
       "    <path d=\"M 310.597059 115.92 \n",
       "L 458.302941 115.92 \n",
       "L 458.302941 7.2 \n",
       "L 310.597059 7.2 \n",
       "z\n",
       "\" style=\"fill: #ffffff\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_12\">\n",
       "    <path d=\"M 310.597059 12.141818 \n",
       "L 445.818397 12.141818 \n",
       "L 445.818397 28.614545 \n",
       "L 310.597059 28.614545 \n",
       "z\n",
       "\" clip-path=\"url(#paaf1809375)\" style=\"fill: #e41a1c\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_13\">\n",
       "    <path d=\"M 310.597059 39.596364 \n",
       "L 432.665539 39.596364 \n",
       "L 432.665539 56.069091 \n",
       "L 310.597059 56.069091 \n",
       "z\n",
       "\" clip-path=\"url(#paaf1809375)\" style=\"fill: #377eb8\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_14\">\n",
       "    <path d=\"M 310.597059 67.050909 \n",
       "L 395.656109 67.050909 \n",
       "L 395.656109 83.523636 \n",
       "L 310.597059 83.523636 \n",
       "z\n",
       "\" clip-path=\"url(#paaf1809375)\" style=\"fill: #4daf4a\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_15\">\n",
       "    <path d=\"M 310.597059 94.505455 \n",
       "L 451.269328 94.505455 \n",
       "L 451.269328 110.978182 \n",
       "L 310.597059 110.978182 \n",
       "z\n",
       "\" clip-path=\"url(#paaf1809375)\" style=\"fill: #984ea3\"/>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_3\">\n",
       "    <g id=\"xtick_5\">\n",
       "     <g id=\"line2d_9\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m19d1fdb71d\" x=\"310.597059\" y=\"115.92\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_10\">\n",
       "      <!-- 0 -->\n",
       "      <g transform=\"translate(307.415809 130.518438)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_6\">\n",
       "     <g id=\"line2d_10\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m19d1fdb71d\" x=\"374.739697\" y=\"115.92\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_11\">\n",
       "      <!-- 20 -->\n",
       "      <g transform=\"translate(368.377197 130.518438)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-32\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_7\">\n",
       "     <g id=\"line2d_11\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m19d1fdb71d\" x=\"438.882335\" y=\"115.92\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_12\">\n",
       "      <!-- 40 -->\n",
       "      <g transform=\"translate(432.519835 130.518438)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-34\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"text_13\">\n",
       "     <!-- per GPU TFLOPS -->\n",
       "     <g transform=\"translate(343.757812 144.196562)scale(0.1 -0.1)\">\n",
       "      <defs>\n",
       "       <path id=\"DejaVuSans-46\" d=\"M 628 4666 \n",
       "L 3309 4666 \n",
       "L 3309 4134 \n",
       "L 1259 4134 \n",
       "L 1259 2759 \n",
       "L 3109 2759 \n",
       "L 3109 2228 \n",
       "L 1259 2228 \n",
       "L 1259 0 \n",
       "L 628 0 \n",
       "L 628 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-4c\" d=\"M 628 4666 \n",
       "L 1259 4666 \n",
       "L 1259 531 \n",
       "L 3531 531 \n",
       "L 3531 0 \n",
       "L 628 0 \n",
       "L 628 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-4f\" d=\"M 2522 4238 \n",
       "Q 1834 4238 1429 3725 \n",
       "Q 1025 3213 1025 2328 \n",
       "Q 1025 1447 1429 934 \n",
       "Q 1834 422 2522 422 \n",
       "Q 3209 422 3611 934 \n",
       "Q 4013 1447 4013 2328 \n",
       "Q 4013 3213 3611 3725 \n",
       "Q 3209 4238 2522 4238 \n",
       "z\n",
       "M 2522 4750 \n",
       "Q 3503 4750 4090 4092 \n",
       "Q 4678 3434 4678 2328 \n",
       "Q 4678 1225 4090 567 \n",
       "Q 3503 -91 2522 -91 \n",
       "Q 1538 -91 948 565 \n",
       "Q 359 1222 359 2328 \n",
       "Q 359 3434 948 4092 \n",
       "Q 1538 4750 2522 4750 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "      </defs>\n",
       "      <use xlink:href=\"#DejaVuSans-70\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-65\" x=\"63.476562\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-72\" x=\"125\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-20\" x=\"166.113281\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-47\" x=\"197.900391\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-50\" x=\"275.390625\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-55\" x=\"335.693359\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-20\" x=\"408.886719\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-54\" x=\"440.673828\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-46\" x=\"501.757812\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-4c\" x=\"559.277344\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-4f\" x=\"611.365234\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-50\" x=\"690.076172\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-53\" x=\"750.378906\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_4\">\n",
       "    <g id=\"ytick_5\">\n",
       "     <g id=\"line2d_12\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m0a0c167960\" x=\"310.597059\" y=\"20.378182\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_6\">\n",
       "     <g id=\"line2d_13\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m0a0c167960\" x=\"310.597059\" y=\"47.832727\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_7\">\n",
       "     <g id=\"line2d_14\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m0a0c167960\" x=\"310.597059\" y=\"75.287273\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_8\">\n",
       "     <g id=\"line2d_15\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m0a0c167960\" x=\"310.597059\" y=\"102.741818\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"patch_16\">\n",
       "    <path d=\"M 310.597059 115.92 \n",
       "L 310.597059 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_17\">\n",
       "    <path d=\"M 458.302941 115.92 \n",
       "L 458.302941 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_18\">\n",
       "    <path d=\"M 310.597059 115.92 \n",
       "L 458.302941 115.92 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_19\">\n",
       "    <path d=\"M 310.597059 7.2 \n",
       "L 458.302941 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "  </g>\n",
       "  <g id=\"axes_3\">\n",
       "   <g id=\"patch_20\">\n",
       "    <path d=\"M 487.844118 115.92 \n",
       "L 635.55 115.92 \n",
       "L 635.55 7.2 \n",
       "L 487.844118 7.2 \n",
       "z\n",
       "\" style=\"fill: #ffffff\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_21\">\n",
       "    <path d=\"M 487.844118 12.141818 \n",
       "L 628.516387 12.141818 \n",
       "L 628.516387 28.614545 \n",
       "L 487.844118 28.614545 \n",
       "z\n",
       "\" clip-path=\"url(#pd6f0414acd)\" style=\"fill: #e41a1c\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_22\">\n",
       "    <path d=\"M 487.844118 39.596364 \n",
       "L 628.516387 39.596364 \n",
       "L 628.516387 56.069091 \n",
       "L 487.844118 56.069091 \n",
       "z\n",
       "\" clip-path=\"url(#pd6f0414acd)\" style=\"fill: #377eb8\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_23\">\n",
       "    <path d=\"M 487.844118 67.050909 \n",
       "L 628.516387 67.050909 \n",
       "L 628.516387 83.523636 \n",
       "L 487.844118 83.523636 \n",
       "z\n",
       "\" clip-path=\"url(#pd6f0414acd)\" style=\"fill: #4daf4a\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_24\">\n",
       "    <path d=\"M 487.844118 94.505455 \n",
       "L 628.516387 94.505455 \n",
       "L 628.516387 110.978182 \n",
       "L 487.844118 110.978182 \n",
       "z\n",
       "\" clip-path=\"url(#pd6f0414acd)\" style=\"fill: #984ea3\"/>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_5\">\n",
       "    <g id=\"xtick_8\">\n",
       "     <g id=\"line2d_16\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m19d1fdb71d\" x=\"487.844118\" y=\"115.92\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_14\">\n",
       "      <!-- 0 -->\n",
       "      <g transform=\"translate(484.662868 130.518438)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_9\">\n",
       "     <g id=\"line2d_17\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m19d1fdb71d\" x=\"553.472148\" y=\"115.92\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_15\">\n",
       "      <!-- 10 -->\n",
       "      <g transform=\"translate(547.109648 130.518438)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-31\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_10\">\n",
       "     <g id=\"line2d_18\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m19d1fdb71d\" x=\"619.100177\" y=\"115.92\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_16\">\n",
       "      <!-- 20 -->\n",
       "      <g transform=\"translate(612.737677 130.518438)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-32\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"text_17\">\n",
       "     <!-- per GPU memory (GB) -->\n",
       "     <g transform=\"translate(505.977528 144.196562)scale(0.1 -0.1)\">\n",
       "      <defs>\n",
       "       <path id=\"DejaVuSans-79\" d=\"M 2059 -325 \n",
       "Q 1816 -950 1584 -1140 \n",
       "Q 1353 -1331 966 -1331 \n",
       "L 506 -1331 \n",
       "L 506 -850 \n",
       "L 844 -850 \n",
       "Q 1081 -850 1212 -737 \n",
       "Q 1344 -625 1503 -206 \n",
       "L 1606 56 \n",
       "L 191 3500 \n",
       "L 800 3500 \n",
       "L 1894 763 \n",
       "L 2988 3500 \n",
       "L 3597 3500 \n",
       "L 2059 -325 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-28\" d=\"M 1984 4856 \n",
       "Q 1566 4138 1362 3434 \n",
       "Q 1159 2731 1159 2009 \n",
       "Q 1159 1288 1364 580 \n",
       "Q 1569 -128 1984 -844 \n",
       "L 1484 -844 \n",
       "Q 1016 -109 783 600 \n",
       "Q 550 1309 550 2009 \n",
       "Q 550 2706 781 3412 \n",
       "Q 1013 4119 1484 4856 \n",
       "L 1984 4856 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-42\" d=\"M 1259 2228 \n",
       "L 1259 519 \n",
       "L 2272 519 \n",
       "Q 2781 519 3026 730 \n",
       "Q 3272 941 3272 1375 \n",
       "Q 3272 1813 3026 2020 \n",
       "Q 2781 2228 2272 2228 \n",
       "L 1259 2228 \n",
       "z\n",
       "M 1259 4147 \n",
       "L 1259 2741 \n",
       "L 2194 2741 \n",
       "Q 2656 2741 2882 2914 \n",
       "Q 3109 3088 3109 3444 \n",
       "Q 3109 3797 2882 3972 \n",
       "Q 2656 4147 2194 4147 \n",
       "L 1259 4147 \n",
       "z\n",
       "M 628 4666 \n",
       "L 2241 4666 \n",
       "Q 2963 4666 3353 4366 \n",
       "Q 3744 4066 3744 3513 \n",
       "Q 3744 3084 3544 2831 \n",
       "Q 3344 2578 2956 2516 \n",
       "Q 3422 2416 3680 2098 \n",
       "Q 3938 1781 3938 1306 \n",
       "Q 3938 681 3513 340 \n",
       "Q 3088 0 2303 0 \n",
       "L 628 0 \n",
       "L 628 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-29\" d=\"M 513 4856 \n",
       "L 1013 4856 \n",
       "Q 1481 4119 1714 3412 \n",
       "Q 1947 2706 1947 2009 \n",
       "Q 1947 1309 1714 600 \n",
       "Q 1481 -109 1013 -844 \n",
       "L 513 -844 \n",
       "Q 928 -128 1133 580 \n",
       "Q 1338 1288 1338 2009 \n",
       "Q 1338 2731 1133 3434 \n",
       "Q 928 4138 513 4856 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "      </defs>\n",
       "      <use xlink:href=\"#DejaVuSans-70\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-65\" x=\"63.476562\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-72\" x=\"125\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-20\" x=\"166.113281\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-47\" x=\"197.900391\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-50\" x=\"275.390625\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-55\" x=\"335.693359\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-20\" x=\"408.886719\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-6d\" x=\"440.673828\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-65\" x=\"538.085938\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-6d\" x=\"599.609375\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-6f\" x=\"697.021484\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-72\" x=\"758.203125\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-79\" x=\"799.316406\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-20\" x=\"858.496094\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-28\" x=\"890.283203\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-47\" x=\"929.296875\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-42\" x=\"1006.787109\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-29\" x=\"1075.390625\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_6\">\n",
       "    <g id=\"ytick_9\">\n",
       "     <g id=\"line2d_19\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m0a0c167960\" x=\"487.844118\" y=\"20.378182\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_10\">\n",
       "     <g id=\"line2d_20\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m0a0c167960\" x=\"487.844118\" y=\"47.832727\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_11\">\n",
       "     <g id=\"line2d_21\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m0a0c167960\" x=\"487.844118\" y=\"75.287273\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_12\">\n",
       "     <g id=\"line2d_22\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m0a0c167960\" x=\"487.844118\" y=\"102.741818\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"patch_25\">\n",
       "    <path d=\"M 487.844118 115.92 \n",
       "L 487.844118 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_26\">\n",
       "    <path d=\"M 635.55 115.92 \n",
       "L 635.55 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_27\">\n",
       "    <path d=\"M 487.844118 115.92 \n",
       "L 635.55 115.92 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_28\">\n",
       "    <path d=\"M 487.844118 7.2 \n",
       "L 635.55 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "  </g>\n",
       " </g>\n",
       " <defs>\n",
       "  <clipPath id=\"p5f50d89725\">\n",
       "   <rect x=\"133.35\" y=\"7.2\" width=\"147.705882\" height=\"108.72\"/>\n",
       "  </clipPath>\n",
       "  <clipPath id=\"paaf1809375\">\n",
       "   <rect x=\"310.597059\" y=\"7.2\" width=\"147.705882\" height=\"108.72\"/>\n",
       "  </clipPath>\n",
       "  <clipPath id=\"pd6f0414acd\">\n",
       "   <rect x=\"487.844118\" y=\"7.2\" width=\"147.705882\" height=\"108.72\"/>\n",
       "  </clipPath>\n",
       " </defs>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<Figure size 648x144 with 3 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "dp_gpt2_accum = megatron_gpt(Exp(\n",
    "    \"Megatron GPT2, 2 GPUs\\ngrad_accum=4\", \"gpt2-medium\", 5, bf16=True, gpus='0,1', grad_accum=4))\n",
    "compare([mega_gpt2, dp_gpt2, dp_gpt2_nonvlink, dp_gpt2_accum])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "432c99f1",
   "metadata": {},
   "source": [
    "A 4 accumulation reduce the communication cost from 10\\% to 3\\%. It helps more when using PCIe, the cost reduces from 37\\% to 14\\%. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "6ab91c59",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Time breakdown\t\t: forward+backward 0.84, communication 0.14, optimizer 0.02\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       "  \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<svg xmlns:xlink=\"http://www.w3.org/1999/xlink\" width=\"651.4125pt\" height=\"180.65625pt\" viewBox=\"0 0 651.4125 180.65625\" xmlns=\"http://www.w3.org/2000/svg\" version=\"1.1\">\n",
       " <metadata>\n",
       "  <rdf:RDF xmlns:dc=\"http://purl.org/dc/elements/1.1/\" xmlns:cc=\"http://creativecommons.org/ns#\" xmlns:rdf=\"http://www.w3.org/1999/02/22-rdf-syntax-ns#\">\n",
       "   <cc:Work>\n",
       "    <dc:type rdf:resource=\"http://purl.org/dc/dcmitype/StillImage\"/>\n",
       "    <dc:date>2022-09-02T19:49:12.883033</dc:date>\n",
       "    <dc:format>image/svg+xml</dc:format>\n",
       "    <dc:creator>\n",
       "     <cc:Agent>\n",
       "      <dc:title>Matplotlib v3.5.2, https://matplotlib.org/</dc:title>\n",
       "     </cc:Agent>\n",
       "    </dc:creator>\n",
       "   </cc:Work>\n",
       "  </rdf:RDF>\n",
       " </metadata>\n",
       " <defs>\n",
       "  <style type=\"text/css\">*{stroke-linejoin: round; stroke-linecap: butt}</style>\n",
       " </defs>\n",
       " <g id=\"figure_1\">\n",
       "  <g id=\"patch_1\">\n",
       "   <path d=\"M 0 180.65625 \n",
       "L 651.4125 180.65625 \n",
       "L 651.4125 0 \n",
       "L 0 0 \n",
       "L 0 180.65625 \n",
       "z\n",
       "\" style=\"fill: none\"/>\n",
       "  </g>\n",
       "  <g id=\"axes_1\">\n",
       "   <g id=\"patch_2\">\n",
       "    <path d=\"M 142.0125 143.1 \n",
       "L 289.718382 143.1 \n",
       "L 289.718382 7.2 \n",
       "L 142.0125 7.2 \n",
       "z\n",
       "\" style=\"fill: #ffffff\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_3\">\n",
       "    <path d=\"M 142.0125 13.377273 \n",
       "L 209.623169 13.377273 \n",
       "L 209.623169 29.491897 \n",
       "L 142.0125 29.491897 \n",
       "z\n",
       "\" clip-path=\"url(#p6089795f93)\" style=\"fill: #e41a1c\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_4\">\n",
       "    <path d=\"M 142.0125 40.23498 \n",
       "L 264.08098 40.23498 \n",
       "L 264.08098 56.349605 \n",
       "L 142.0125 56.349605 \n",
       "z\n",
       "\" clip-path=\"url(#p6089795f93)\" style=\"fill: #377eb8\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_5\">\n",
       "    <path d=\"M 142.0125 67.092688 \n",
       "L 227.07155 67.092688 \n",
       "L 227.07155 83.207312 \n",
       "L 142.0125 83.207312 \n",
       "z\n",
       "\" clip-path=\"url(#p6089795f93)\" style=\"fill: #4daf4a\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_6\">\n",
       "    <path d=\"M 142.0125 93.950395 \n",
       "L 282.684769 93.950395 \n",
       "L 282.684769 110.06502 \n",
       "L 142.0125 110.06502 \n",
       "z\n",
       "\" clip-path=\"url(#p6089795f93)\" style=\"fill: #984ea3\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_7\">\n",
       "    <path d=\"M 142.0125 120.808103 \n",
       "L 267.320375 120.808103 \n",
       "L 267.320375 136.922727 \n",
       "L 142.0125 136.922727 \n",
       "z\n",
       "\" clip-path=\"url(#p6089795f93)\" style=\"fill: #ff7f00\"/>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_1\">\n",
       "    <g id=\"xtick_1\">\n",
       "     <g id=\"line2d_1\">\n",
       "      <defs>\n",
       "       <path id=\"mbced7e43b9\" d=\"M 0 0 \n",
       "L 0 3.5 \n",
       "\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </defs>\n",
       "      <g>\n",
       "       <use xlink:href=\"#mbced7e43b9\" x=\"142.0125\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_1\">\n",
       "      <!-- 0 -->\n",
       "      <g transform=\"translate(138.83125 157.698438)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-30\" d=\"M 2034 4250 \n",
       "Q 1547 4250 1301 3770 \n",
       "Q 1056 3291 1056 2328 \n",
       "Q 1056 1369 1301 889 \n",
       "Q 1547 409 2034 409 \n",
       "Q 2525 409 2770 889 \n",
       "Q 3016 1369 3016 2328 \n",
       "Q 3016 3291 2770 3770 \n",
       "Q 2525 4250 2034 4250 \n",
       "z\n",
       "M 2034 4750 \n",
       "Q 2819 4750 3233 4129 \n",
       "Q 3647 3509 3647 2328 \n",
       "Q 3647 1150 3233 529 \n",
       "Q 2819 -91 2034 -91 \n",
       "Q 1250 -91 836 529 \n",
       "Q 422 1150 422 2328 \n",
       "Q 422 3509 836 4129 \n",
       "Q 1250 4750 2034 4750 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_2\">\n",
       "     <g id=\"line2d_2\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#mbced7e43b9\" x=\"181.794618\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_2\">\n",
       "      <!-- 10 -->\n",
       "      <g transform=\"translate(175.432118 157.698438)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-31\" d=\"M 794 531 \n",
       "L 1825 531 \n",
       "L 1825 4091 \n",
       "L 703 3866 \n",
       "L 703 4441 \n",
       "L 1819 4666 \n",
       "L 2450 4666 \n",
       "L 2450 531 \n",
       "L 3481 531 \n",
       "L 3481 0 \n",
       "L 794 0 \n",
       "L 794 531 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-31\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_3\">\n",
       "     <g id=\"line2d_3\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#mbced7e43b9\" x=\"221.576735\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_3\">\n",
       "      <!-- 20 -->\n",
       "      <g transform=\"translate(215.214235 157.698438)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-32\" d=\"M 1228 531 \n",
       "L 3431 531 \n",
       "L 3431 0 \n",
       "L 469 0 \n",
       "L 469 531 \n",
       "Q 828 903 1448 1529 \n",
       "Q 2069 2156 2228 2338 \n",
       "Q 2531 2678 2651 2914 \n",
       "Q 2772 3150 2772 3378 \n",
       "Q 2772 3750 2511 3984 \n",
       "Q 2250 4219 1831 4219 \n",
       "Q 1534 4219 1204 4116 \n",
       "Q 875 4013 500 3803 \n",
       "L 500 4441 \n",
       "Q 881 4594 1212 4672 \n",
       "Q 1544 4750 1819 4750 \n",
       "Q 2544 4750 2975 4387 \n",
       "Q 3406 4025 3406 3419 \n",
       "Q 3406 3131 3298 2873 \n",
       "Q 3191 2616 2906 2266 \n",
       "Q 2828 2175 2409 1742 \n",
       "Q 1991 1309 1228 531 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-32\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_4\">\n",
       "     <g id=\"line2d_4\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#mbced7e43b9\" x=\"261.358853\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_4\">\n",
       "      <!-- 30 -->\n",
       "      <g transform=\"translate(254.996353 157.698438)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-33\" d=\"M 2597 2516 \n",
       "Q 3050 2419 3304 2112 \n",
       "Q 3559 1806 3559 1356 \n",
       "Q 3559 666 3084 287 \n",
       "Q 2609 -91 1734 -91 \n",
       "Q 1441 -91 1130 -33 \n",
       "Q 819 25 488 141 \n",
       "L 488 750 \n",
       "Q 750 597 1062 519 \n",
       "Q 1375 441 1716 441 \n",
       "Q 2309 441 2620 675 \n",
       "Q 2931 909 2931 1356 \n",
       "Q 2931 1769 2642 2001 \n",
       "Q 2353 2234 1838 2234 \n",
       "L 1294 2234 \n",
       "L 1294 2753 \n",
       "L 1863 2753 \n",
       "Q 2328 2753 2575 2939 \n",
       "Q 2822 3125 2822 3475 \n",
       "Q 2822 3834 2567 4026 \n",
       "Q 2313 4219 1838 4219 \n",
       "Q 1578 4219 1281 4162 \n",
       "Q 984 4106 628 3988 \n",
       "L 628 4550 \n",
       "Q 988 4650 1302 4700 \n",
       "Q 1616 4750 1894 4750 \n",
       "Q 2613 4750 3031 4423 \n",
       "Q 3450 4097 3450 3541 \n",
       "Q 3450 3153 3228 2886 \n",
       "Q 3006 2619 2597 2516 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-33\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"text_5\">\n",
       "     <!-- Samples / sec -->\n",
       "     <g transform=\"translate(181.219347 171.376563)scale(0.1 -0.1)\">\n",
       "      <defs>\n",
       "       <path id=\"DejaVuSans-53\" d=\"M 3425 4513 \n",
       "L 3425 3897 \n",
       "Q 3066 4069 2747 4153 \n",
       "Q 2428 4238 2131 4238 \n",
       "Q 1616 4238 1336 4038 \n",
       "Q 1056 3838 1056 3469 \n",
       "Q 1056 3159 1242 3001 \n",
       "Q 1428 2844 1947 2747 \n",
       "L 2328 2669 \n",
       "Q 3034 2534 3370 2195 \n",
       "Q 3706 1856 3706 1288 \n",
       "Q 3706 609 3251 259 \n",
       "Q 2797 -91 1919 -91 \n",
       "Q 1588 -91 1214 -16 \n",
       "Q 841 59 441 206 \n",
       "L 441 856 \n",
       "Q 825 641 1194 531 \n",
       "Q 1563 422 1919 422 \n",
       "Q 2459 422 2753 634 \n",
       "Q 3047 847 3047 1241 \n",
       "Q 3047 1584 2836 1778 \n",
       "Q 2625 1972 2144 2069 \n",
       "L 1759 2144 \n",
       "Q 1053 2284 737 2584 \n",
       "Q 422 2884 422 3419 \n",
       "Q 422 4038 858 4394 \n",
       "Q 1294 4750 2059 4750 \n",
       "Q 2388 4750 2728 4690 \n",
       "Q 3069 4631 3425 4513 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-61\" d=\"M 2194 1759 \n",
       "Q 1497 1759 1228 1600 \n",
       "Q 959 1441 959 1056 \n",
       "Q 959 750 1161 570 \n",
       "Q 1363 391 1709 391 \n",
       "Q 2188 391 2477 730 \n",
       "Q 2766 1069 2766 1631 \n",
       "L 2766 1759 \n",
       "L 2194 1759 \n",
       "z\n",
       "M 3341 1997 \n",
       "L 3341 0 \n",
       "L 2766 0 \n",
       "L 2766 531 \n",
       "Q 2569 213 2275 61 \n",
       "Q 1981 -91 1556 -91 \n",
       "Q 1019 -91 701 211 \n",
       "Q 384 513 384 1019 \n",
       "Q 384 1609 779 1909 \n",
       "Q 1175 2209 1959 2209 \n",
       "L 2766 2209 \n",
       "L 2766 2266 \n",
       "Q 2766 2663 2505 2880 \n",
       "Q 2244 3097 1772 3097 \n",
       "Q 1472 3097 1187 3025 \n",
       "Q 903 2953 641 2809 \n",
       "L 641 3341 \n",
       "Q 956 3463 1253 3523 \n",
       "Q 1550 3584 1831 3584 \n",
       "Q 2591 3584 2966 3190 \n",
       "Q 3341 2797 3341 1997 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-6d\" d=\"M 3328 2828 \n",
       "Q 3544 3216 3844 3400 \n",
       "Q 4144 3584 4550 3584 \n",
       "Q 5097 3584 5394 3201 \n",
       "Q 5691 2819 5691 2113 \n",
       "L 5691 0 \n",
       "L 5113 0 \n",
       "L 5113 2094 \n",
       "Q 5113 2597 4934 2840 \n",
       "Q 4756 3084 4391 3084 \n",
       "Q 3944 3084 3684 2787 \n",
       "Q 3425 2491 3425 1978 \n",
       "L 3425 0 \n",
       "L 2847 0 \n",
       "L 2847 2094 \n",
       "Q 2847 2600 2669 2842 \n",
       "Q 2491 3084 2119 3084 \n",
       "Q 1678 3084 1418 2786 \n",
       "Q 1159 2488 1159 1978 \n",
       "L 1159 0 \n",
       "L 581 0 \n",
       "L 581 3500 \n",
       "L 1159 3500 \n",
       "L 1159 2956 \n",
       "Q 1356 3278 1631 3431 \n",
       "Q 1906 3584 2284 3584 \n",
       "Q 2666 3584 2933 3390 \n",
       "Q 3200 3197 3328 2828 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-70\" d=\"M 1159 525 \n",
       "L 1159 -1331 \n",
       "L 581 -1331 \n",
       "L 581 3500 \n",
       "L 1159 3500 \n",
       "L 1159 2969 \n",
       "Q 1341 3281 1617 3432 \n",
       "Q 1894 3584 2278 3584 \n",
       "Q 2916 3584 3314 3078 \n",
       "Q 3713 2572 3713 1747 \n",
       "Q 3713 922 3314 415 \n",
       "Q 2916 -91 2278 -91 \n",
       "Q 1894 -91 1617 61 \n",
       "Q 1341 213 1159 525 \n",
       "z\n",
       "M 3116 1747 \n",
       "Q 3116 2381 2855 2742 \n",
       "Q 2594 3103 2138 3103 \n",
       "Q 1681 3103 1420 2742 \n",
       "Q 1159 2381 1159 1747 \n",
       "Q 1159 1113 1420 752 \n",
       "Q 1681 391 2138 391 \n",
       "Q 2594 391 2855 752 \n",
       "Q 3116 1113 3116 1747 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-6c\" d=\"M 603 4863 \n",
       "L 1178 4863 \n",
       "L 1178 0 \n",
       "L 603 0 \n",
       "L 603 4863 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-65\" d=\"M 3597 1894 \n",
       "L 3597 1613 \n",
       "L 953 1613 \n",
       "Q 991 1019 1311 708 \n",
       "Q 1631 397 2203 397 \n",
       "Q 2534 397 2845 478 \n",
       "Q 3156 559 3463 722 \n",
       "L 3463 178 \n",
       "Q 3153 47 2828 -22 \n",
       "Q 2503 -91 2169 -91 \n",
       "Q 1331 -91 842 396 \n",
       "Q 353 884 353 1716 \n",
       "Q 353 2575 817 3079 \n",
       "Q 1281 3584 2069 3584 \n",
       "Q 2775 3584 3186 3129 \n",
       "Q 3597 2675 3597 1894 \n",
       "z\n",
       "M 3022 2063 \n",
       "Q 3016 2534 2758 2815 \n",
       "Q 2500 3097 2075 3097 \n",
       "Q 1594 3097 1305 2825 \n",
       "Q 1016 2553 972 2059 \n",
       "L 3022 2063 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-73\" d=\"M 2834 3397 \n",
       "L 2834 2853 \n",
       "Q 2591 2978 2328 3040 \n",
       "Q 2066 3103 1784 3103 \n",
       "Q 1356 3103 1142 2972 \n",
       "Q 928 2841 928 2578 \n",
       "Q 928 2378 1081 2264 \n",
       "Q 1234 2150 1697 2047 \n",
       "L 1894 2003 \n",
       "Q 2506 1872 2764 1633 \n",
       "Q 3022 1394 3022 966 \n",
       "Q 3022 478 2636 193 \n",
       "Q 2250 -91 1575 -91 \n",
       "Q 1294 -91 989 -36 \n",
       "Q 684 19 347 128 \n",
       "L 347 722 \n",
       "Q 666 556 975 473 \n",
       "Q 1284 391 1588 391 \n",
       "Q 1994 391 2212 530 \n",
       "Q 2431 669 2431 922 \n",
       "Q 2431 1156 2273 1281 \n",
       "Q 2116 1406 1581 1522 \n",
       "L 1381 1569 \n",
       "Q 847 1681 609 1914 \n",
       "Q 372 2147 372 2553 \n",
       "Q 372 3047 722 3315 \n",
       "Q 1072 3584 1716 3584 \n",
       "Q 2034 3584 2315 3537 \n",
       "Q 2597 3491 2834 3397 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-20\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-2f\" d=\"M 1625 4666 \n",
       "L 2156 4666 \n",
       "L 531 -594 \n",
       "L 0 -594 \n",
       "L 1625 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-63\" d=\"M 3122 3366 \n",
       "L 3122 2828 \n",
       "Q 2878 2963 2633 3030 \n",
       "Q 2388 3097 2138 3097 \n",
       "Q 1578 3097 1268 2742 \n",
       "Q 959 2388 959 1747 \n",
       "Q 959 1106 1268 751 \n",
       "Q 1578 397 2138 397 \n",
       "Q 2388 397 2633 464 \n",
       "Q 2878 531 3122 666 \n",
       "L 3122 134 \n",
       "Q 2881 22 2623 -34 \n",
       "Q 2366 -91 2075 -91 \n",
       "Q 1284 -91 818 406 \n",
       "Q 353 903 353 1747 \n",
       "Q 353 2603 823 3093 \n",
       "Q 1294 3584 2113 3584 \n",
       "Q 2378 3584 2631 3529 \n",
       "Q 2884 3475 3122 3366 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "      </defs>\n",
       "      <use xlink:href=\"#DejaVuSans-53\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-61\" x=\"63.476562\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-6d\" x=\"124.755859\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-70\" x=\"222.167969\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-6c\" x=\"285.644531\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-65\" x=\"313.427734\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-73\" x=\"374.951172\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-20\" x=\"427.050781\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-2f\" x=\"458.837891\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-20\" x=\"492.529297\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-73\" x=\"524.316406\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-65\" x=\"576.416016\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-63\" x=\"637.939453\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_2\">\n",
       "    <g id=\"ytick_1\">\n",
       "     <g id=\"line2d_5\">\n",
       "      <defs>\n",
       "       <path id=\"m26f8590fa4\" d=\"M 0 0 \n",
       "L -3.5 0 \n",
       "\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </defs>\n",
       "      <g>\n",
       "       <use xlink:href=\"#m26f8590fa4\" x=\"142.0125\" y=\"21.434585\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_6\">\n",
       "      <!-- Megatron GPT2 -->\n",
       "      <g transform=\"translate(58.065625 25.233804)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-4d\" d=\"M 628 4666 \n",
       "L 1569 4666 \n",
       "L 2759 1491 \n",
       "L 3956 4666 \n",
       "L 4897 4666 \n",
       "L 4897 0 \n",
       "L 4281 0 \n",
       "L 4281 4097 \n",
       "L 3078 897 \n",
       "L 2444 897 \n",
       "L 1241 4097 \n",
       "L 1241 0 \n",
       "L 628 0 \n",
       "L 628 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-67\" d=\"M 2906 1791 \n",
       "Q 2906 2416 2648 2759 \n",
       "Q 2391 3103 1925 3103 \n",
       "Q 1463 3103 1205 2759 \n",
       "Q 947 2416 947 1791 \n",
       "Q 947 1169 1205 825 \n",
       "Q 1463 481 1925 481 \n",
       "Q 2391 481 2648 825 \n",
       "Q 2906 1169 2906 1791 \n",
       "z\n",
       "M 3481 434 \n",
       "Q 3481 -459 3084 -895 \n",
       "Q 2688 -1331 1869 -1331 \n",
       "Q 1566 -1331 1297 -1286 \n",
       "Q 1028 -1241 775 -1147 \n",
       "L 775 -588 \n",
       "Q 1028 -725 1275 -790 \n",
       "Q 1522 -856 1778 -856 \n",
       "Q 2344 -856 2625 -561 \n",
       "Q 2906 -266 2906 331 \n",
       "L 2906 616 \n",
       "Q 2728 306 2450 153 \n",
       "Q 2172 0 1784 0 \n",
       "Q 1141 0 747 490 \n",
       "Q 353 981 353 1791 \n",
       "Q 353 2603 747 3093 \n",
       "Q 1141 3584 1784 3584 \n",
       "Q 2172 3584 2450 3431 \n",
       "Q 2728 3278 2906 2969 \n",
       "L 2906 3500 \n",
       "L 3481 3500 \n",
       "L 3481 434 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-74\" d=\"M 1172 4494 \n",
       "L 1172 3500 \n",
       "L 2356 3500 \n",
       "L 2356 3053 \n",
       "L 1172 3053 \n",
       "L 1172 1153 \n",
       "Q 1172 725 1289 603 \n",
       "Q 1406 481 1766 481 \n",
       "L 2356 481 \n",
       "L 2356 0 \n",
       "L 1766 0 \n",
       "Q 1100 0 847 248 \n",
       "Q 594 497 594 1153 \n",
       "L 594 3053 \n",
       "L 172 3053 \n",
       "L 172 3500 \n",
       "L 594 3500 \n",
       "L 594 4494 \n",
       "L 1172 4494 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-72\" d=\"M 2631 2963 \n",
       "Q 2534 3019 2420 3045 \n",
       "Q 2306 3072 2169 3072 \n",
       "Q 1681 3072 1420 2755 \n",
       "Q 1159 2438 1159 1844 \n",
       "L 1159 0 \n",
       "L 581 0 \n",
       "L 581 3500 \n",
       "L 1159 3500 \n",
       "L 1159 2956 \n",
       "Q 1341 3275 1631 3429 \n",
       "Q 1922 3584 2338 3584 \n",
       "Q 2397 3584 2469 3576 \n",
       "Q 2541 3569 2628 3553 \n",
       "L 2631 2963 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-6f\" d=\"M 1959 3097 \n",
       "Q 1497 3097 1228 2736 \n",
       "Q 959 2375 959 1747 \n",
       "Q 959 1119 1226 758 \n",
       "Q 1494 397 1959 397 \n",
       "Q 2419 397 2687 759 \n",
       "Q 2956 1122 2956 1747 \n",
       "Q 2956 2369 2687 2733 \n",
       "Q 2419 3097 1959 3097 \n",
       "z\n",
       "M 1959 3584 \n",
       "Q 2709 3584 3137 3096 \n",
       "Q 3566 2609 3566 1747 \n",
       "Q 3566 888 3137 398 \n",
       "Q 2709 -91 1959 -91 \n",
       "Q 1206 -91 779 398 \n",
       "Q 353 888 353 1747 \n",
       "Q 353 2609 779 3096 \n",
       "Q 1206 3584 1959 3584 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-6e\" d=\"M 3513 2113 \n",
       "L 3513 0 \n",
       "L 2938 0 \n",
       "L 2938 2094 \n",
       "Q 2938 2591 2744 2837 \n",
       "Q 2550 3084 2163 3084 \n",
       "Q 1697 3084 1428 2787 \n",
       "Q 1159 2491 1159 1978 \n",
       "L 1159 0 \n",
       "L 581 0 \n",
       "L 581 3500 \n",
       "L 1159 3500 \n",
       "L 1159 2956 \n",
       "Q 1366 3272 1645 3428 \n",
       "Q 1925 3584 2291 3584 \n",
       "Q 2894 3584 3203 3211 \n",
       "Q 3513 2838 3513 2113 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-47\" d=\"M 3809 666 \n",
       "L 3809 1919 \n",
       "L 2778 1919 \n",
       "L 2778 2438 \n",
       "L 4434 2438 \n",
       "L 4434 434 \n",
       "Q 4069 175 3628 42 \n",
       "Q 3188 -91 2688 -91 \n",
       "Q 1594 -91 976 548 \n",
       "Q 359 1188 359 2328 \n",
       "Q 359 3472 976 4111 \n",
       "Q 1594 4750 2688 4750 \n",
       "Q 3144 4750 3555 4637 \n",
       "Q 3966 4525 4313 4306 \n",
       "L 4313 3634 \n",
       "Q 3963 3931 3569 4081 \n",
       "Q 3175 4231 2741 4231 \n",
       "Q 1884 4231 1454 3753 \n",
       "Q 1025 3275 1025 2328 \n",
       "Q 1025 1384 1454 906 \n",
       "Q 1884 428 2741 428 \n",
       "Q 3075 428 3337 486 \n",
       "Q 3600 544 3809 666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-50\" d=\"M 1259 4147 \n",
       "L 1259 2394 \n",
       "L 2053 2394 \n",
       "Q 2494 2394 2734 2622 \n",
       "Q 2975 2850 2975 3272 \n",
       "Q 2975 3691 2734 3919 \n",
       "Q 2494 4147 2053 4147 \n",
       "L 1259 4147 \n",
       "z\n",
       "M 628 4666 \n",
       "L 2053 4666 \n",
       "Q 2838 4666 3239 4311 \n",
       "Q 3641 3956 3641 3272 \n",
       "Q 3641 2581 3239 2228 \n",
       "Q 2838 1875 2053 1875 \n",
       "L 1259 1875 \n",
       "L 1259 0 \n",
       "L 628 0 \n",
       "L 628 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-54\" d=\"M -19 4666 \n",
       "L 3928 4666 \n",
       "L 3928 4134 \n",
       "L 2272 4134 \n",
       "L 2272 0 \n",
       "L 1638 0 \n",
       "L 1638 4134 \n",
       "L -19 4134 \n",
       "L -19 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-4d\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-65\" x=\"86.279297\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-67\" x=\"147.802734\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-61\" x=\"211.279297\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-74\" x=\"272.558594\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-72\" x=\"311.767578\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-6f\" x=\"350.630859\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-6e\" x=\"411.8125\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-20\" x=\"475.191406\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-47\" x=\"506.978516\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-50\" x=\"584.46875\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-54\" x=\"644.771484\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-32\" x=\"705.855469\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_2\">\n",
       "     <g id=\"line2d_6\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m26f8590fa4\" x=\"142.0125\" y=\"48.292292\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_7\">\n",
       "      <!-- Megatron GPT2, 2 GPUs -->\n",
       "      <g transform=\"translate(15.8625 52.091511)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-2c\" d=\"M 750 794 \n",
       "L 1409 794 \n",
       "L 1409 256 \n",
       "L 897 -744 \n",
       "L 494 -744 \n",
       "L 750 256 \n",
       "L 750 794 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-55\" d=\"M 556 4666 \n",
       "L 1191 4666 \n",
       "L 1191 1831 \n",
       "Q 1191 1081 1462 751 \n",
       "Q 1734 422 2344 422 \n",
       "Q 2950 422 3222 751 \n",
       "Q 3494 1081 3494 1831 \n",
       "L 3494 4666 \n",
       "L 4128 4666 \n",
       "L 4128 1753 \n",
       "Q 4128 841 3676 375 \n",
       "Q 3225 -91 2344 -91 \n",
       "Q 1459 -91 1007 375 \n",
       "Q 556 841 556 1753 \n",
       "L 556 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-4d\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-65\" x=\"86.279297\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-67\" x=\"147.802734\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-61\" x=\"211.279297\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-74\" x=\"272.558594\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-72\" x=\"311.767578\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-6f\" x=\"350.630859\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-6e\" x=\"411.8125\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-20\" x=\"475.191406\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-47\" x=\"506.978516\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-50\" x=\"584.46875\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-54\" x=\"644.771484\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-32\" x=\"705.855469\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-2c\" x=\"769.478516\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-20\" x=\"801.265625\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-32\" x=\"833.052734\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-20\" x=\"896.675781\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-47\" x=\"928.462891\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-50\" x=\"1005.953125\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-55\" x=\"1066.255859\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-73\" x=\"1139.449219\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_3\">\n",
       "     <g id=\"line2d_7\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m26f8590fa4\" x=\"142.0125\" y=\"75.15\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_8\">\n",
       "      <!-- Megatron GPT2, 2 GPUs -->\n",
       "      <g transform=\"translate(15.8625 74.110156)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-4d\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-65\" x=\"86.279297\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-67\" x=\"147.802734\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-61\" x=\"211.279297\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-74\" x=\"272.558594\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-72\" x=\"311.767578\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-6f\" x=\"350.630859\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-6e\" x=\"411.8125\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-20\" x=\"475.191406\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-47\" x=\"506.978516\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-50\" x=\"584.46875\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-54\" x=\"644.771484\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-32\" x=\"705.855469\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-2c\" x=\"769.478516\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-20\" x=\"801.265625\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-32\" x=\"833.052734\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-20\" x=\"896.675781\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-47\" x=\"928.462891\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-50\" x=\"1005.953125\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-55\" x=\"1066.255859\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-73\" x=\"1139.449219\"/>\n",
       "      </g>\n",
       "      <!-- no nvlink -->\n",
       "      <g transform=\"translate(89.4375 85.307969)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-76\" d=\"M 191 3500 \n",
       "L 800 3500 \n",
       "L 1894 563 \n",
       "L 2988 3500 \n",
       "L 3597 3500 \n",
       "L 2284 0 \n",
       "L 1503 0 \n",
       "L 191 3500 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-69\" d=\"M 603 3500 \n",
       "L 1178 3500 \n",
       "L 1178 0 \n",
       "L 603 0 \n",
       "L 603 3500 \n",
       "z\n",
       "M 603 4863 \n",
       "L 1178 4863 \n",
       "L 1178 4134 \n",
       "L 603 4134 \n",
       "L 603 4863 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-6b\" d=\"M 581 4863 \n",
       "L 1159 4863 \n",
       "L 1159 1991 \n",
       "L 2875 3500 \n",
       "L 3609 3500 \n",
       "L 1753 1863 \n",
       "L 3688 0 \n",
       "L 2938 0 \n",
       "L 1159 1709 \n",
       "L 1159 0 \n",
       "L 581 0 \n",
       "L 581 4863 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-6e\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-6f\" x=\"63.378906\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-20\" x=\"124.560547\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-6e\" x=\"156.347656\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-76\" x=\"219.726562\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-6c\" x=\"278.90625\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-69\" x=\"306.689453\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-6e\" x=\"334.472656\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-6b\" x=\"397.851562\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_4\">\n",
       "     <g id=\"line2d_8\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m26f8590fa4\" x=\"142.0125\" y=\"102.007708\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_9\">\n",
       "      <!-- Megatron GPT2, 2 GPUs -->\n",
       "      <g transform=\"translate(15.8625 100.967864)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-4d\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-65\" x=\"86.279297\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-67\" x=\"147.802734\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-61\" x=\"211.279297\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-74\" x=\"272.558594\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-72\" x=\"311.767578\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-6f\" x=\"350.630859\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-6e\" x=\"411.8125\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-20\" x=\"475.191406\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-47\" x=\"506.978516\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-50\" x=\"584.46875\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-54\" x=\"644.771484\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-32\" x=\"705.855469\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-2c\" x=\"769.478516\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-20\" x=\"801.265625\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-32\" x=\"833.052734\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-20\" x=\"896.675781\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-47\" x=\"928.462891\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-50\" x=\"1005.953125\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-55\" x=\"1066.255859\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-73\" x=\"1139.449219\"/>\n",
       "      </g>\n",
       "      <!-- grad_accum=4 -->\n",
       "      <g transform=\"translate(59.13125 112.165676)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-64\" d=\"M 2906 2969 \n",
       "L 2906 4863 \n",
       "L 3481 4863 \n",
       "L 3481 0 \n",
       "L 2906 0 \n",
       "L 2906 525 \n",
       "Q 2725 213 2448 61 \n",
       "Q 2172 -91 1784 -91 \n",
       "Q 1150 -91 751 415 \n",
       "Q 353 922 353 1747 \n",
       "Q 353 2572 751 3078 \n",
       "Q 1150 3584 1784 3584 \n",
       "Q 2172 3584 2448 3432 \n",
       "Q 2725 3281 2906 2969 \n",
       "z\n",
       "M 947 1747 \n",
       "Q 947 1113 1208 752 \n",
       "Q 1469 391 1925 391 \n",
       "Q 2381 391 2643 752 \n",
       "Q 2906 1113 2906 1747 \n",
       "Q 2906 2381 2643 2742 \n",
       "Q 2381 3103 1925 3103 \n",
       "Q 1469 3103 1208 2742 \n",
       "Q 947 2381 947 1747 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-5f\" d=\"M 3263 -1063 \n",
       "L 3263 -1509 \n",
       "L -63 -1509 \n",
       "L -63 -1063 \n",
       "L 3263 -1063 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-75\" d=\"M 544 1381 \n",
       "L 544 3500 \n",
       "L 1119 3500 \n",
       "L 1119 1403 \n",
       "Q 1119 906 1312 657 \n",
       "Q 1506 409 1894 409 \n",
       "Q 2359 409 2629 706 \n",
       "Q 2900 1003 2900 1516 \n",
       "L 2900 3500 \n",
       "L 3475 3500 \n",
       "L 3475 0 \n",
       "L 2900 0 \n",
       "L 2900 538 \n",
       "Q 2691 219 2414 64 \n",
       "Q 2138 -91 1772 -91 \n",
       "Q 1169 -91 856 284 \n",
       "Q 544 659 544 1381 \n",
       "z\n",
       "M 1991 3584 \n",
       "L 1991 3584 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-3d\" d=\"M 678 2906 \n",
       "L 4684 2906 \n",
       "L 4684 2381 \n",
       "L 678 2381 \n",
       "L 678 2906 \n",
       "z\n",
       "M 678 1631 \n",
       "L 4684 1631 \n",
       "L 4684 1100 \n",
       "L 678 1100 \n",
       "L 678 1631 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-34\" d=\"M 2419 4116 \n",
       "L 825 1625 \n",
       "L 2419 1625 \n",
       "L 2419 4116 \n",
       "z\n",
       "M 2253 4666 \n",
       "L 3047 4666 \n",
       "L 3047 1625 \n",
       "L 3713 1625 \n",
       "L 3713 1100 \n",
       "L 3047 1100 \n",
       "L 3047 0 \n",
       "L 2419 0 \n",
       "L 2419 1100 \n",
       "L 313 1100 \n",
       "L 313 1709 \n",
       "L 2253 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-67\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-72\" x=\"63.476562\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-61\" x=\"104.589844\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-64\" x=\"165.869141\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-5f\" x=\"229.345703\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-61\" x=\"279.345703\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-63\" x=\"340.625\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-63\" x=\"395.605469\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-75\" x=\"450.585938\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-6d\" x=\"513.964844\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-3d\" x=\"611.376953\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-34\" x=\"695.166016\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_5\">\n",
       "     <g id=\"line2d_9\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m26f8590fa4\" x=\"142.0125\" y=\"128.865415\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_10\">\n",
       "      <!-- Megatron GPT2, 2 GPUs -->\n",
       "      <g transform=\"translate(15.8625 127.825571)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-4d\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-65\" x=\"86.279297\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-67\" x=\"147.802734\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-61\" x=\"211.279297\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-74\" x=\"272.558594\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-72\" x=\"311.767578\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-6f\" x=\"350.630859\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-6e\" x=\"411.8125\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-20\" x=\"475.191406\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-47\" x=\"506.978516\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-50\" x=\"584.46875\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-54\" x=\"644.771484\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-32\" x=\"705.855469\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-2c\" x=\"769.478516\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-20\" x=\"801.265625\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-32\" x=\"833.052734\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-20\" x=\"896.675781\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-47\" x=\"928.462891\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-50\" x=\"1005.953125\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-55\" x=\"1066.255859\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-73\" x=\"1139.449219\"/>\n",
       "      </g>\n",
       "      <!-- grad_accum=4, no nvlink -->\n",
       "      <g transform=\"translate(7.2 139.023384)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-67\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-72\" x=\"63.476562\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-61\" x=\"104.589844\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-64\" x=\"165.869141\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-5f\" x=\"229.345703\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-61\" x=\"279.345703\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-63\" x=\"340.625\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-63\" x=\"395.605469\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-75\" x=\"450.585938\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-6d\" x=\"513.964844\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-3d\" x=\"611.376953\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-34\" x=\"695.166016\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-2c\" x=\"758.789062\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-20\" x=\"790.576172\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-6e\" x=\"822.363281\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-6f\" x=\"885.742188\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-20\" x=\"946.923828\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-6e\" x=\"978.710938\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-76\" x=\"1042.089844\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-6c\" x=\"1101.269531\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-69\" x=\"1129.052734\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-6e\" x=\"1156.835938\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-6b\" x=\"1220.214844\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"patch_8\">\n",
       "    <path d=\"M 142.0125 143.1 \n",
       "L 142.0125 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_9\">\n",
       "    <path d=\"M 289.718382 143.1 \n",
       "L 289.718382 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_10\">\n",
       "    <path d=\"M 142.0125 143.1 \n",
       "L 289.718382 143.1 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_11\">\n",
       "    <path d=\"M 142.0125 7.2 \n",
       "L 289.718382 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "  </g>\n",
       "  <g id=\"axes_2\">\n",
       "   <g id=\"patch_12\">\n",
       "    <path d=\"M 319.259559 143.1 \n",
       "L 466.965441 143.1 \n",
       "L 466.965441 7.2 \n",
       "L 319.259559 7.2 \n",
       "z\n",
       "\" style=\"fill: #ffffff\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_13\">\n",
       "    <path d=\"M 319.259559 13.377273 \n",
       "L 454.480897 13.377273 \n",
       "L 454.480897 29.491897 \n",
       "L 319.259559 29.491897 \n",
       "z\n",
       "\" clip-path=\"url(#p3533bc089b)\" style=\"fill: #e41a1c\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_14\">\n",
       "    <path d=\"M 319.259559 40.23498 \n",
       "L 441.328039 40.23498 \n",
       "L 441.328039 56.349605 \n",
       "L 319.259559 56.349605 \n",
       "z\n",
       "\" clip-path=\"url(#p3533bc089b)\" style=\"fill: #377eb8\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_15\">\n",
       "    <path d=\"M 319.259559 67.092688 \n",
       "L 404.318609 67.092688 \n",
       "L 404.318609 83.207312 \n",
       "L 319.259559 83.207312 \n",
       "z\n",
       "\" clip-path=\"url(#p3533bc089b)\" style=\"fill: #4daf4a\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_16\">\n",
       "    <path d=\"M 319.259559 93.950395 \n",
       "L 459.931828 93.950395 \n",
       "L 459.931828 110.06502 \n",
       "L 319.259559 110.06502 \n",
       "z\n",
       "\" clip-path=\"url(#p3533bc089b)\" style=\"fill: #984ea3\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_17\">\n",
       "    <path d=\"M 319.259559 120.808103 \n",
       "L 444.567434 120.808103 \n",
       "L 444.567434 136.922727 \n",
       "L 319.259559 136.922727 \n",
       "z\n",
       "\" clip-path=\"url(#p3533bc089b)\" style=\"fill: #ff7f00\"/>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_3\">\n",
       "    <g id=\"xtick_5\">\n",
       "     <g id=\"line2d_10\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#mbced7e43b9\" x=\"319.259559\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_11\">\n",
       "      <!-- 0 -->\n",
       "      <g transform=\"translate(316.078309 157.698438)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_6\">\n",
       "     <g id=\"line2d_11\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#mbced7e43b9\" x=\"383.402197\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_12\">\n",
       "      <!-- 20 -->\n",
       "      <g transform=\"translate(377.039697 157.698438)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-32\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_7\">\n",
       "     <g id=\"line2d_12\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#mbced7e43b9\" x=\"447.544835\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_13\">\n",
       "      <!-- 40 -->\n",
       "      <g transform=\"translate(441.182335 157.698438)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-34\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"text_14\">\n",
       "     <!-- per GPU TFLOPS -->\n",
       "     <g transform=\"translate(352.420312 171.376563)scale(0.1 -0.1)\">\n",
       "      <defs>\n",
       "       <path id=\"DejaVuSans-46\" d=\"M 628 4666 \n",
       "L 3309 4666 \n",
       "L 3309 4134 \n",
       "L 1259 4134 \n",
       "L 1259 2759 \n",
       "L 3109 2759 \n",
       "L 3109 2228 \n",
       "L 1259 2228 \n",
       "L 1259 0 \n",
       "L 628 0 \n",
       "L 628 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-4c\" d=\"M 628 4666 \n",
       "L 1259 4666 \n",
       "L 1259 531 \n",
       "L 3531 531 \n",
       "L 3531 0 \n",
       "L 628 0 \n",
       "L 628 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-4f\" d=\"M 2522 4238 \n",
       "Q 1834 4238 1429 3725 \n",
       "Q 1025 3213 1025 2328 \n",
       "Q 1025 1447 1429 934 \n",
       "Q 1834 422 2522 422 \n",
       "Q 3209 422 3611 934 \n",
       "Q 4013 1447 4013 2328 \n",
       "Q 4013 3213 3611 3725 \n",
       "Q 3209 4238 2522 4238 \n",
       "z\n",
       "M 2522 4750 \n",
       "Q 3503 4750 4090 4092 \n",
       "Q 4678 3434 4678 2328 \n",
       "Q 4678 1225 4090 567 \n",
       "Q 3503 -91 2522 -91 \n",
       "Q 1538 -91 948 565 \n",
       "Q 359 1222 359 2328 \n",
       "Q 359 3434 948 4092 \n",
       "Q 1538 4750 2522 4750 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "      </defs>\n",
       "      <use xlink:href=\"#DejaVuSans-70\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-65\" x=\"63.476562\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-72\" x=\"125\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-20\" x=\"166.113281\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-47\" x=\"197.900391\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-50\" x=\"275.390625\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-55\" x=\"335.693359\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-20\" x=\"408.886719\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-54\" x=\"440.673828\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-46\" x=\"501.757812\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-4c\" x=\"559.277344\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-4f\" x=\"611.365234\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-50\" x=\"690.076172\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-53\" x=\"750.378906\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_4\">\n",
       "    <g id=\"ytick_6\">\n",
       "     <g id=\"line2d_13\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m26f8590fa4\" x=\"319.259559\" y=\"21.434585\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_7\">\n",
       "     <g id=\"line2d_14\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m26f8590fa4\" x=\"319.259559\" y=\"48.292292\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_8\">\n",
       "     <g id=\"line2d_15\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m26f8590fa4\" x=\"319.259559\" y=\"75.15\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_9\">\n",
       "     <g id=\"line2d_16\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m26f8590fa4\" x=\"319.259559\" y=\"102.007708\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_10\">\n",
       "     <g id=\"line2d_17\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m26f8590fa4\" x=\"319.259559\" y=\"128.865415\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"patch_18\">\n",
       "    <path d=\"M 319.259559 143.1 \n",
       "L 319.259559 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_19\">\n",
       "    <path d=\"M 466.965441 143.1 \n",
       "L 466.965441 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_20\">\n",
       "    <path d=\"M 319.259559 143.1 \n",
       "L 466.965441 143.1 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_21\">\n",
       "    <path d=\"M 319.259559 7.2 \n",
       "L 466.965441 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "  </g>\n",
       "  <g id=\"axes_3\">\n",
       "   <g id=\"patch_22\">\n",
       "    <path d=\"M 496.506618 143.1 \n",
       "L 644.2125 143.1 \n",
       "L 644.2125 7.2 \n",
       "L 496.506618 7.2 \n",
       "z\n",
       "\" style=\"fill: #ffffff\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_23\">\n",
       "    <path d=\"M 496.506618 13.377273 \n",
       "L 637.178887 13.377273 \n",
       "L 637.178887 29.491897 \n",
       "L 496.506618 29.491897 \n",
       "z\n",
       "\" clip-path=\"url(#p0fbf39d3ae)\" style=\"fill: #e41a1c\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_24\">\n",
       "    <path d=\"M 496.506618 40.23498 \n",
       "L 637.178887 40.23498 \n",
       "L 637.178887 56.349605 \n",
       "L 496.506618 56.349605 \n",
       "z\n",
       "\" clip-path=\"url(#p0fbf39d3ae)\" style=\"fill: #377eb8\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_25\">\n",
       "    <path d=\"M 496.506618 67.092688 \n",
       "L 637.178887 67.092688 \n",
       "L 637.178887 83.207312 \n",
       "L 496.506618 83.207312 \n",
       "z\n",
       "\" clip-path=\"url(#p0fbf39d3ae)\" style=\"fill: #4daf4a\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_26\">\n",
       "    <path d=\"M 496.506618 93.950395 \n",
       "L 637.178887 93.950395 \n",
       "L 637.178887 110.06502 \n",
       "L 496.506618 110.06502 \n",
       "z\n",
       "\" clip-path=\"url(#p0fbf39d3ae)\" style=\"fill: #984ea3\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_27\">\n",
       "    <path d=\"M 496.506618 120.808103 \n",
       "L 637.178887 120.808103 \n",
       "L 637.178887 136.922727 \n",
       "L 496.506618 136.922727 \n",
       "z\n",
       "\" clip-path=\"url(#p0fbf39d3ae)\" style=\"fill: #ff7f00\"/>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_5\">\n",
       "    <g id=\"xtick_8\">\n",
       "     <g id=\"line2d_18\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#mbced7e43b9\" x=\"496.506618\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_15\">\n",
       "      <!-- 0 -->\n",
       "      <g transform=\"translate(493.325368 157.698438)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_9\">\n",
       "     <g id=\"line2d_19\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#mbced7e43b9\" x=\"562.134648\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_16\">\n",
       "      <!-- 10 -->\n",
       "      <g transform=\"translate(555.772148 157.698438)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-31\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_10\">\n",
       "     <g id=\"line2d_20\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#mbced7e43b9\" x=\"627.762677\" y=\"143.1\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_17\">\n",
       "      <!-- 20 -->\n",
       "      <g transform=\"translate(621.400177 157.698438)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-32\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"text_18\">\n",
       "     <!-- per GPU memory (GB) -->\n",
       "     <g transform=\"translate(514.640028 171.376563)scale(0.1 -0.1)\">\n",
       "      <defs>\n",
       "       <path id=\"DejaVuSans-79\" d=\"M 2059 -325 \n",
       "Q 1816 -950 1584 -1140 \n",
       "Q 1353 -1331 966 -1331 \n",
       "L 506 -1331 \n",
       "L 506 -850 \n",
       "L 844 -850 \n",
       "Q 1081 -850 1212 -737 \n",
       "Q 1344 -625 1503 -206 \n",
       "L 1606 56 \n",
       "L 191 3500 \n",
       "L 800 3500 \n",
       "L 1894 763 \n",
       "L 2988 3500 \n",
       "L 3597 3500 \n",
       "L 2059 -325 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-28\" d=\"M 1984 4856 \n",
       "Q 1566 4138 1362 3434 \n",
       "Q 1159 2731 1159 2009 \n",
       "Q 1159 1288 1364 580 \n",
       "Q 1569 -128 1984 -844 \n",
       "L 1484 -844 \n",
       "Q 1016 -109 783 600 \n",
       "Q 550 1309 550 2009 \n",
       "Q 550 2706 781 3412 \n",
       "Q 1013 4119 1484 4856 \n",
       "L 1984 4856 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-42\" d=\"M 1259 2228 \n",
       "L 1259 519 \n",
       "L 2272 519 \n",
       "Q 2781 519 3026 730 \n",
       "Q 3272 941 3272 1375 \n",
       "Q 3272 1813 3026 2020 \n",
       "Q 2781 2228 2272 2228 \n",
       "L 1259 2228 \n",
       "z\n",
       "M 1259 4147 \n",
       "L 1259 2741 \n",
       "L 2194 2741 \n",
       "Q 2656 2741 2882 2914 \n",
       "Q 3109 3088 3109 3444 \n",
       "Q 3109 3797 2882 3972 \n",
       "Q 2656 4147 2194 4147 \n",
       "L 1259 4147 \n",
       "z\n",
       "M 628 4666 \n",
       "L 2241 4666 \n",
       "Q 2963 4666 3353 4366 \n",
       "Q 3744 4066 3744 3513 \n",
       "Q 3744 3084 3544 2831 \n",
       "Q 3344 2578 2956 2516 \n",
       "Q 3422 2416 3680 2098 \n",
       "Q 3938 1781 3938 1306 \n",
       "Q 3938 681 3513 340 \n",
       "Q 3088 0 2303 0 \n",
       "L 628 0 \n",
       "L 628 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-29\" d=\"M 513 4856 \n",
       "L 1013 4856 \n",
       "Q 1481 4119 1714 3412 \n",
       "Q 1947 2706 1947 2009 \n",
       "Q 1947 1309 1714 600 \n",
       "Q 1481 -109 1013 -844 \n",
       "L 513 -844 \n",
       "Q 928 -128 1133 580 \n",
       "Q 1338 1288 1338 2009 \n",
       "Q 1338 2731 1133 3434 \n",
       "Q 928 4138 513 4856 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "      </defs>\n",
       "      <use xlink:href=\"#DejaVuSans-70\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-65\" x=\"63.476562\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-72\" x=\"125\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-20\" x=\"166.113281\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-47\" x=\"197.900391\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-50\" x=\"275.390625\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-55\" x=\"335.693359\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-20\" x=\"408.886719\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-6d\" x=\"440.673828\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-65\" x=\"538.085938\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-6d\" x=\"599.609375\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-6f\" x=\"697.021484\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-72\" x=\"758.203125\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-79\" x=\"799.316406\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-20\" x=\"858.496094\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-28\" x=\"890.283203\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-47\" x=\"929.296875\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-42\" x=\"1006.787109\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-29\" x=\"1075.390625\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_6\">\n",
       "    <g id=\"ytick_11\">\n",
       "     <g id=\"line2d_21\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m26f8590fa4\" x=\"496.506618\" y=\"21.434585\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_12\">\n",
       "     <g id=\"line2d_22\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m26f8590fa4\" x=\"496.506618\" y=\"48.292292\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_13\">\n",
       "     <g id=\"line2d_23\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m26f8590fa4\" x=\"496.506618\" y=\"75.15\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_14\">\n",
       "     <g id=\"line2d_24\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m26f8590fa4\" x=\"496.506618\" y=\"102.007708\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_15\">\n",
       "     <g id=\"line2d_25\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m26f8590fa4\" x=\"496.506618\" y=\"128.865415\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"patch_28\">\n",
       "    <path d=\"M 496.506618 143.1 \n",
       "L 496.506618 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_29\">\n",
       "    <path d=\"M 644.2125 143.1 \n",
       "L 644.2125 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_30\">\n",
       "    <path d=\"M 496.506618 143.1 \n",
       "L 644.2125 143.1 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_31\">\n",
       "    <path d=\"M 496.506618 7.2 \n",
       "L 644.2125 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "  </g>\n",
       " </g>\n",
       " <defs>\n",
       "  <clipPath id=\"p6089795f93\">\n",
       "   <rect x=\"142.0125\" y=\"7.2\" width=\"147.705882\" height=\"135.9\"/>\n",
       "  </clipPath>\n",
       "  <clipPath id=\"p3533bc089b\">\n",
       "   <rect x=\"319.259559\" y=\"7.2\" width=\"147.705882\" height=\"135.9\"/>\n",
       "  </clipPath>\n",
       "  <clipPath id=\"p0fbf39d3ae\">\n",
       "   <rect x=\"496.506618\" y=\"7.2\" width=\"147.705882\" height=\"135.9\"/>\n",
       "  </clipPath>\n",
       " </defs>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<Figure size 648x180 with 3 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "os.environ[\"NCCL_P2P_DISABLE\"] = \"1\"\n",
    "dp_gpt2_accum_nonvlink = megatron_gpt(Exp(\n",
    "    \"Megatron GPT2, 2 GPUs\\ngrad_accum=4, no nvlink\", \"gpt2-medium\", \n",
    "    5, bf16=True, gpus='0,1', grad_accum=4))\n",
    "os.environ[\"NCCL_P2P_DISABLE\"] = \"0\"\n",
    "compare([mega_gpt2, dp_gpt2, dp_gpt2_nonvlink, dp_gpt2_accum, dp_gpt2_accum_nonvlink])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3824791a",
   "metadata": {},
   "source": [
    "## Multiple GPUs with Tensor Parallelism\n",
    "\n",
    "Different to data parallelism (DP) that splits data, tensor parallelism (TP) partitions each layer into multiple GPUs. So we can use a larger batch size per GPU. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "680223e8",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Time breakdown\t\t: forward+backward 0.95, communication 0.00, optimizer 0.04\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       "  \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<svg xmlns:xlink=\"http://www.w3.org/1999/xlink\" width=\"661.24375pt\" height=\"99.11625pt\" viewBox=\"0 0 661.24375 99.11625\" xmlns=\"http://www.w3.org/2000/svg\" version=\"1.1\">\n",
       " <metadata>\n",
       "  <rdf:RDF xmlns:dc=\"http://purl.org/dc/elements/1.1/\" xmlns:cc=\"http://creativecommons.org/ns#\" xmlns:rdf=\"http://www.w3.org/1999/02/22-rdf-syntax-ns#\">\n",
       "   <cc:Work>\n",
       "    <dc:type rdf:resource=\"http://purl.org/dc/dcmitype/StillImage\"/>\n",
       "    <dc:date>2022-09-02T19:49:54.391912</dc:date>\n",
       "    <dc:format>image/svg+xml</dc:format>\n",
       "    <dc:creator>\n",
       "     <cc:Agent>\n",
       "      <dc:title>Matplotlib v3.5.2, https://matplotlib.org/</dc:title>\n",
       "     </cc:Agent>\n",
       "    </dc:creator>\n",
       "   </cc:Work>\n",
       "  </rdf:RDF>\n",
       " </metadata>\n",
       " <defs>\n",
       "  <style type=\"text/css\">*{stroke-linejoin: round; stroke-linecap: butt}</style>\n",
       " </defs>\n",
       " <g id=\"figure_1\">\n",
       "  <g id=\"patch_1\">\n",
       "   <path d=\"M 0 99.11625 \n",
       "L 661.24375 99.11625 \n",
       "L 661.24375 0 \n",
       "L 0 0 \n",
       "L 0 99.11625 \n",
       "z\n",
       "\" style=\"fill: none\"/>\n",
       "  </g>\n",
       "  <g id=\"axes_1\">\n",
       "   <g id=\"patch_2\">\n",
       "    <path d=\"M 151.84375 61.56 \n",
       "L 299.549632 61.56 \n",
       "L 299.549632 7.2 \n",
       "L 151.84375 7.2 \n",
       "z\n",
       "\" style=\"fill: #ffffff\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_3\">\n",
       "    <path d=\"M 151.84375 9.670909 \n",
       "L 292.516019 9.670909 \n",
       "L 292.516019 28.202727 \n",
       "L 151.84375 28.202727 \n",
       "z\n",
       "\" clip-path=\"url(#p5e0a3a65e3)\" style=\"fill: #e41a1c\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_4\">\n",
       "    <path d=\"M 151.84375 40.557273 \n",
       "L 288.084738 40.557273 \n",
       "L 288.084738 59.089091 \n",
       "L 151.84375 59.089091 \n",
       "z\n",
       "\" clip-path=\"url(#p5e0a3a65e3)\" style=\"fill: #377eb8\"/>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_1\">\n",
       "    <g id=\"xtick_1\">\n",
       "     <g id=\"line2d_1\">\n",
       "      <defs>\n",
       "       <path id=\"m412737860e\" d=\"M 0 0 \n",
       "L 0 3.5 \n",
       "\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </defs>\n",
       "      <g>\n",
       "       <use xlink:href=\"#m412737860e\" x=\"151.84375\" y=\"61.56\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_1\">\n",
       "      <!-- 0 -->\n",
       "      <g transform=\"translate(148.6625 76.158437)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-30\" d=\"M 2034 4250 \n",
       "Q 1547 4250 1301 3770 \n",
       "Q 1056 3291 1056 2328 \n",
       "Q 1056 1369 1301 889 \n",
       "Q 1547 409 2034 409 \n",
       "Q 2525 409 2770 889 \n",
       "Q 3016 1369 3016 2328 \n",
       "Q 3016 3291 2770 3770 \n",
       "Q 2525 4250 2034 4250 \n",
       "z\n",
       "M 2034 4750 \n",
       "Q 2819 4750 3233 4129 \n",
       "Q 3647 3509 3647 2328 \n",
       "Q 3647 1150 3233 529 \n",
       "Q 2819 -91 2034 -91 \n",
       "Q 1250 -91 836 529 \n",
       "Q 422 1150 422 2328 \n",
       "Q 422 3509 836 4129 \n",
       "Q 1250 4750 2034 4750 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_2\">\n",
       "     <g id=\"line2d_2\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m412737860e\" x=\"197.688842\" y=\"61.56\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_2\">\n",
       "      <!-- 10 -->\n",
       "      <g transform=\"translate(191.326342 76.158437)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-31\" d=\"M 794 531 \n",
       "L 1825 531 \n",
       "L 1825 4091 \n",
       "L 703 3866 \n",
       "L 703 4441 \n",
       "L 1819 4666 \n",
       "L 2450 4666 \n",
       "L 2450 531 \n",
       "L 3481 531 \n",
       "L 3481 0 \n",
       "L 794 0 \n",
       "L 794 531 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-31\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_3\">\n",
       "     <g id=\"line2d_3\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m412737860e\" x=\"243.533935\" y=\"61.56\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_3\">\n",
       "      <!-- 20 -->\n",
       "      <g transform=\"translate(237.171435 76.158437)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-32\" d=\"M 1228 531 \n",
       "L 3431 531 \n",
       "L 3431 0 \n",
       "L 469 0 \n",
       "L 469 531 \n",
       "Q 828 903 1448 1529 \n",
       "Q 2069 2156 2228 2338 \n",
       "Q 2531 2678 2651 2914 \n",
       "Q 2772 3150 2772 3378 \n",
       "Q 2772 3750 2511 3984 \n",
       "Q 2250 4219 1831 4219 \n",
       "Q 1534 4219 1204 4116 \n",
       "Q 875 4013 500 3803 \n",
       "L 500 4441 \n",
       "Q 881 4594 1212 4672 \n",
       "Q 1544 4750 1819 4750 \n",
       "Q 2544 4750 2975 4387 \n",
       "Q 3406 4025 3406 3419 \n",
       "Q 3406 3131 3298 2873 \n",
       "Q 3191 2616 2906 2266 \n",
       "Q 2828 2175 2409 1742 \n",
       "Q 1991 1309 1228 531 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-32\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_4\">\n",
       "     <g id=\"line2d_4\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m412737860e\" x=\"289.379027\" y=\"61.56\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_4\">\n",
       "      <!-- 30 -->\n",
       "      <g transform=\"translate(283.016527 76.158437)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-33\" d=\"M 2597 2516 \n",
       "Q 3050 2419 3304 2112 \n",
       "Q 3559 1806 3559 1356 \n",
       "Q 3559 666 3084 287 \n",
       "Q 2609 -91 1734 -91 \n",
       "Q 1441 -91 1130 -33 \n",
       "Q 819 25 488 141 \n",
       "L 488 750 \n",
       "Q 750 597 1062 519 \n",
       "Q 1375 441 1716 441 \n",
       "Q 2309 441 2620 675 \n",
       "Q 2931 909 2931 1356 \n",
       "Q 2931 1769 2642 2001 \n",
       "Q 2353 2234 1838 2234 \n",
       "L 1294 2234 \n",
       "L 1294 2753 \n",
       "L 1863 2753 \n",
       "Q 2328 2753 2575 2939 \n",
       "Q 2822 3125 2822 3475 \n",
       "Q 2822 3834 2567 4026 \n",
       "Q 2313 4219 1838 4219 \n",
       "Q 1578 4219 1281 4162 \n",
       "Q 984 4106 628 3988 \n",
       "L 628 4550 \n",
       "Q 988 4650 1302 4700 \n",
       "Q 1616 4750 1894 4750 \n",
       "Q 2613 4750 3031 4423 \n",
       "Q 3450 4097 3450 3541 \n",
       "Q 3450 3153 3228 2886 \n",
       "Q 3006 2619 2597 2516 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-33\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"text_5\">\n",
       "     <!-- Samples / sec -->\n",
       "     <g transform=\"translate(191.050597 89.836562)scale(0.1 -0.1)\">\n",
       "      <defs>\n",
       "       <path id=\"DejaVuSans-53\" d=\"M 3425 4513 \n",
       "L 3425 3897 \n",
       "Q 3066 4069 2747 4153 \n",
       "Q 2428 4238 2131 4238 \n",
       "Q 1616 4238 1336 4038 \n",
       "Q 1056 3838 1056 3469 \n",
       "Q 1056 3159 1242 3001 \n",
       "Q 1428 2844 1947 2747 \n",
       "L 2328 2669 \n",
       "Q 3034 2534 3370 2195 \n",
       "Q 3706 1856 3706 1288 \n",
       "Q 3706 609 3251 259 \n",
       "Q 2797 -91 1919 -91 \n",
       "Q 1588 -91 1214 -16 \n",
       "Q 841 59 441 206 \n",
       "L 441 856 \n",
       "Q 825 641 1194 531 \n",
       "Q 1563 422 1919 422 \n",
       "Q 2459 422 2753 634 \n",
       "Q 3047 847 3047 1241 \n",
       "Q 3047 1584 2836 1778 \n",
       "Q 2625 1972 2144 2069 \n",
       "L 1759 2144 \n",
       "Q 1053 2284 737 2584 \n",
       "Q 422 2884 422 3419 \n",
       "Q 422 4038 858 4394 \n",
       "Q 1294 4750 2059 4750 \n",
       "Q 2388 4750 2728 4690 \n",
       "Q 3069 4631 3425 4513 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-61\" d=\"M 2194 1759 \n",
       "Q 1497 1759 1228 1600 \n",
       "Q 959 1441 959 1056 \n",
       "Q 959 750 1161 570 \n",
       "Q 1363 391 1709 391 \n",
       "Q 2188 391 2477 730 \n",
       "Q 2766 1069 2766 1631 \n",
       "L 2766 1759 \n",
       "L 2194 1759 \n",
       "z\n",
       "M 3341 1997 \n",
       "L 3341 0 \n",
       "L 2766 0 \n",
       "L 2766 531 \n",
       "Q 2569 213 2275 61 \n",
       "Q 1981 -91 1556 -91 \n",
       "Q 1019 -91 701 211 \n",
       "Q 384 513 384 1019 \n",
       "Q 384 1609 779 1909 \n",
       "Q 1175 2209 1959 2209 \n",
       "L 2766 2209 \n",
       "L 2766 2266 \n",
       "Q 2766 2663 2505 2880 \n",
       "Q 2244 3097 1772 3097 \n",
       "Q 1472 3097 1187 3025 \n",
       "Q 903 2953 641 2809 \n",
       "L 641 3341 \n",
       "Q 956 3463 1253 3523 \n",
       "Q 1550 3584 1831 3584 \n",
       "Q 2591 3584 2966 3190 \n",
       "Q 3341 2797 3341 1997 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-6d\" d=\"M 3328 2828 \n",
       "Q 3544 3216 3844 3400 \n",
       "Q 4144 3584 4550 3584 \n",
       "Q 5097 3584 5394 3201 \n",
       "Q 5691 2819 5691 2113 \n",
       "L 5691 0 \n",
       "L 5113 0 \n",
       "L 5113 2094 \n",
       "Q 5113 2597 4934 2840 \n",
       "Q 4756 3084 4391 3084 \n",
       "Q 3944 3084 3684 2787 \n",
       "Q 3425 2491 3425 1978 \n",
       "L 3425 0 \n",
       "L 2847 0 \n",
       "L 2847 2094 \n",
       "Q 2847 2600 2669 2842 \n",
       "Q 2491 3084 2119 3084 \n",
       "Q 1678 3084 1418 2786 \n",
       "Q 1159 2488 1159 1978 \n",
       "L 1159 0 \n",
       "L 581 0 \n",
       "L 581 3500 \n",
       "L 1159 3500 \n",
       "L 1159 2956 \n",
       "Q 1356 3278 1631 3431 \n",
       "Q 1906 3584 2284 3584 \n",
       "Q 2666 3584 2933 3390 \n",
       "Q 3200 3197 3328 2828 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-70\" d=\"M 1159 525 \n",
       "L 1159 -1331 \n",
       "L 581 -1331 \n",
       "L 581 3500 \n",
       "L 1159 3500 \n",
       "L 1159 2969 \n",
       "Q 1341 3281 1617 3432 \n",
       "Q 1894 3584 2278 3584 \n",
       "Q 2916 3584 3314 3078 \n",
       "Q 3713 2572 3713 1747 \n",
       "Q 3713 922 3314 415 \n",
       "Q 2916 -91 2278 -91 \n",
       "Q 1894 -91 1617 61 \n",
       "Q 1341 213 1159 525 \n",
       "z\n",
       "M 3116 1747 \n",
       "Q 3116 2381 2855 2742 \n",
       "Q 2594 3103 2138 3103 \n",
       "Q 1681 3103 1420 2742 \n",
       "Q 1159 2381 1159 1747 \n",
       "Q 1159 1113 1420 752 \n",
       "Q 1681 391 2138 391 \n",
       "Q 2594 391 2855 752 \n",
       "Q 3116 1113 3116 1747 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-6c\" d=\"M 603 4863 \n",
       "L 1178 4863 \n",
       "L 1178 0 \n",
       "L 603 0 \n",
       "L 603 4863 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-65\" d=\"M 3597 1894 \n",
       "L 3597 1613 \n",
       "L 953 1613 \n",
       "Q 991 1019 1311 708 \n",
       "Q 1631 397 2203 397 \n",
       "Q 2534 397 2845 478 \n",
       "Q 3156 559 3463 722 \n",
       "L 3463 178 \n",
       "Q 3153 47 2828 -22 \n",
       "Q 2503 -91 2169 -91 \n",
       "Q 1331 -91 842 396 \n",
       "Q 353 884 353 1716 \n",
       "Q 353 2575 817 3079 \n",
       "Q 1281 3584 2069 3584 \n",
       "Q 2775 3584 3186 3129 \n",
       "Q 3597 2675 3597 1894 \n",
       "z\n",
       "M 3022 2063 \n",
       "Q 3016 2534 2758 2815 \n",
       "Q 2500 3097 2075 3097 \n",
       "Q 1594 3097 1305 2825 \n",
       "Q 1016 2553 972 2059 \n",
       "L 3022 2063 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-73\" d=\"M 2834 3397 \n",
       "L 2834 2853 \n",
       "Q 2591 2978 2328 3040 \n",
       "Q 2066 3103 1784 3103 \n",
       "Q 1356 3103 1142 2972 \n",
       "Q 928 2841 928 2578 \n",
       "Q 928 2378 1081 2264 \n",
       "Q 1234 2150 1697 2047 \n",
       "L 1894 2003 \n",
       "Q 2506 1872 2764 1633 \n",
       "Q 3022 1394 3022 966 \n",
       "Q 3022 478 2636 193 \n",
       "Q 2250 -91 1575 -91 \n",
       "Q 1294 -91 989 -36 \n",
       "Q 684 19 347 128 \n",
       "L 347 722 \n",
       "Q 666 556 975 473 \n",
       "Q 1284 391 1588 391 \n",
       "Q 1994 391 2212 530 \n",
       "Q 2431 669 2431 922 \n",
       "Q 2431 1156 2273 1281 \n",
       "Q 2116 1406 1581 1522 \n",
       "L 1381 1569 \n",
       "Q 847 1681 609 1914 \n",
       "Q 372 2147 372 2553 \n",
       "Q 372 3047 722 3315 \n",
       "Q 1072 3584 1716 3584 \n",
       "Q 2034 3584 2315 3537 \n",
       "Q 2597 3491 2834 3397 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-20\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-2f\" d=\"M 1625 4666 \n",
       "L 2156 4666 \n",
       "L 531 -594 \n",
       "L 0 -594 \n",
       "L 1625 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-63\" d=\"M 3122 3366 \n",
       "L 3122 2828 \n",
       "Q 2878 2963 2633 3030 \n",
       "Q 2388 3097 2138 3097 \n",
       "Q 1578 3097 1268 2742 \n",
       "Q 959 2388 959 1747 \n",
       "Q 959 1106 1268 751 \n",
       "Q 1578 397 2138 397 \n",
       "Q 2388 397 2633 464 \n",
       "Q 2878 531 3122 666 \n",
       "L 3122 134 \n",
       "Q 2881 22 2623 -34 \n",
       "Q 2366 -91 2075 -91 \n",
       "Q 1284 -91 818 406 \n",
       "Q 353 903 353 1747 \n",
       "Q 353 2603 823 3093 \n",
       "Q 1294 3584 2113 3584 \n",
       "Q 2378 3584 2631 3529 \n",
       "Q 2884 3475 3122 3366 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "      </defs>\n",
       "      <use xlink:href=\"#DejaVuSans-53\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-61\" x=\"63.476562\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-6d\" x=\"124.755859\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-70\" x=\"222.167969\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-6c\" x=\"285.644531\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-65\" x=\"313.427734\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-73\" x=\"374.951172\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-20\" x=\"427.050781\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-2f\" x=\"458.837891\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-20\" x=\"492.529297\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-73\" x=\"524.316406\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-65\" x=\"576.416016\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-63\" x=\"637.939453\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_2\">\n",
       "    <g id=\"ytick_1\">\n",
       "     <g id=\"line2d_5\">\n",
       "      <defs>\n",
       "       <path id=\"mdada20d4ad\" d=\"M 0 0 \n",
       "L -3.5 0 \n",
       "\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </defs>\n",
       "      <g>\n",
       "       <use xlink:href=\"#mdada20d4ad\" x=\"151.84375\" y=\"18.936818\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_6\">\n",
       "      <!-- Megatron GPT2, 2 GPUs -->\n",
       "      <g transform=\"translate(25.69375 22.736037)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-4d\" d=\"M 628 4666 \n",
       "L 1569 4666 \n",
       "L 2759 1491 \n",
       "L 3956 4666 \n",
       "L 4897 4666 \n",
       "L 4897 0 \n",
       "L 4281 0 \n",
       "L 4281 4097 \n",
       "L 3078 897 \n",
       "L 2444 897 \n",
       "L 1241 4097 \n",
       "L 1241 0 \n",
       "L 628 0 \n",
       "L 628 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-67\" d=\"M 2906 1791 \n",
       "Q 2906 2416 2648 2759 \n",
       "Q 2391 3103 1925 3103 \n",
       "Q 1463 3103 1205 2759 \n",
       "Q 947 2416 947 1791 \n",
       "Q 947 1169 1205 825 \n",
       "Q 1463 481 1925 481 \n",
       "Q 2391 481 2648 825 \n",
       "Q 2906 1169 2906 1791 \n",
       "z\n",
       "M 3481 434 \n",
       "Q 3481 -459 3084 -895 \n",
       "Q 2688 -1331 1869 -1331 \n",
       "Q 1566 -1331 1297 -1286 \n",
       "Q 1028 -1241 775 -1147 \n",
       "L 775 -588 \n",
       "Q 1028 -725 1275 -790 \n",
       "Q 1522 -856 1778 -856 \n",
       "Q 2344 -856 2625 -561 \n",
       "Q 2906 -266 2906 331 \n",
       "L 2906 616 \n",
       "Q 2728 306 2450 153 \n",
       "Q 2172 0 1784 0 \n",
       "Q 1141 0 747 490 \n",
       "Q 353 981 353 1791 \n",
       "Q 353 2603 747 3093 \n",
       "Q 1141 3584 1784 3584 \n",
       "Q 2172 3584 2450 3431 \n",
       "Q 2728 3278 2906 2969 \n",
       "L 2906 3500 \n",
       "L 3481 3500 \n",
       "L 3481 434 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-74\" d=\"M 1172 4494 \n",
       "L 1172 3500 \n",
       "L 2356 3500 \n",
       "L 2356 3053 \n",
       "L 1172 3053 \n",
       "L 1172 1153 \n",
       "Q 1172 725 1289 603 \n",
       "Q 1406 481 1766 481 \n",
       "L 2356 481 \n",
       "L 2356 0 \n",
       "L 1766 0 \n",
       "Q 1100 0 847 248 \n",
       "Q 594 497 594 1153 \n",
       "L 594 3053 \n",
       "L 172 3053 \n",
       "L 172 3500 \n",
       "L 594 3500 \n",
       "L 594 4494 \n",
       "L 1172 4494 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-72\" d=\"M 2631 2963 \n",
       "Q 2534 3019 2420 3045 \n",
       "Q 2306 3072 2169 3072 \n",
       "Q 1681 3072 1420 2755 \n",
       "Q 1159 2438 1159 1844 \n",
       "L 1159 0 \n",
       "L 581 0 \n",
       "L 581 3500 \n",
       "L 1159 3500 \n",
       "L 1159 2956 \n",
       "Q 1341 3275 1631 3429 \n",
       "Q 1922 3584 2338 3584 \n",
       "Q 2397 3584 2469 3576 \n",
       "Q 2541 3569 2628 3553 \n",
       "L 2631 2963 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-6f\" d=\"M 1959 3097 \n",
       "Q 1497 3097 1228 2736 \n",
       "Q 959 2375 959 1747 \n",
       "Q 959 1119 1226 758 \n",
       "Q 1494 397 1959 397 \n",
       "Q 2419 397 2687 759 \n",
       "Q 2956 1122 2956 1747 \n",
       "Q 2956 2369 2687 2733 \n",
       "Q 2419 3097 1959 3097 \n",
       "z\n",
       "M 1959 3584 \n",
       "Q 2709 3584 3137 3096 \n",
       "Q 3566 2609 3566 1747 \n",
       "Q 3566 888 3137 398 \n",
       "Q 2709 -91 1959 -91 \n",
       "Q 1206 -91 779 398 \n",
       "Q 353 888 353 1747 \n",
       "Q 353 2609 779 3096 \n",
       "Q 1206 3584 1959 3584 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-6e\" d=\"M 3513 2113 \n",
       "L 3513 0 \n",
       "L 2938 0 \n",
       "L 2938 2094 \n",
       "Q 2938 2591 2744 2837 \n",
       "Q 2550 3084 2163 3084 \n",
       "Q 1697 3084 1428 2787 \n",
       "Q 1159 2491 1159 1978 \n",
       "L 1159 0 \n",
       "L 581 0 \n",
       "L 581 3500 \n",
       "L 1159 3500 \n",
       "L 1159 2956 \n",
       "Q 1366 3272 1645 3428 \n",
       "Q 1925 3584 2291 3584 \n",
       "Q 2894 3584 3203 3211 \n",
       "Q 3513 2838 3513 2113 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-47\" d=\"M 3809 666 \n",
       "L 3809 1919 \n",
       "L 2778 1919 \n",
       "L 2778 2438 \n",
       "L 4434 2438 \n",
       "L 4434 434 \n",
       "Q 4069 175 3628 42 \n",
       "Q 3188 -91 2688 -91 \n",
       "Q 1594 -91 976 548 \n",
       "Q 359 1188 359 2328 \n",
       "Q 359 3472 976 4111 \n",
       "Q 1594 4750 2688 4750 \n",
       "Q 3144 4750 3555 4637 \n",
       "Q 3966 4525 4313 4306 \n",
       "L 4313 3634 \n",
       "Q 3963 3931 3569 4081 \n",
       "Q 3175 4231 2741 4231 \n",
       "Q 1884 4231 1454 3753 \n",
       "Q 1025 3275 1025 2328 \n",
       "Q 1025 1384 1454 906 \n",
       "Q 1884 428 2741 428 \n",
       "Q 3075 428 3337 486 \n",
       "Q 3600 544 3809 666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-50\" d=\"M 1259 4147 \n",
       "L 1259 2394 \n",
       "L 2053 2394 \n",
       "Q 2494 2394 2734 2622 \n",
       "Q 2975 2850 2975 3272 \n",
       "Q 2975 3691 2734 3919 \n",
       "Q 2494 4147 2053 4147 \n",
       "L 1259 4147 \n",
       "z\n",
       "M 628 4666 \n",
       "L 2053 4666 \n",
       "Q 2838 4666 3239 4311 \n",
       "Q 3641 3956 3641 3272 \n",
       "Q 3641 2581 3239 2228 \n",
       "Q 2838 1875 2053 1875 \n",
       "L 1259 1875 \n",
       "L 1259 0 \n",
       "L 628 0 \n",
       "L 628 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-54\" d=\"M -19 4666 \n",
       "L 3928 4666 \n",
       "L 3928 4134 \n",
       "L 2272 4134 \n",
       "L 2272 0 \n",
       "L 1638 0 \n",
       "L 1638 4134 \n",
       "L -19 4134 \n",
       "L -19 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-2c\" d=\"M 750 794 \n",
       "L 1409 794 \n",
       "L 1409 256 \n",
       "L 897 -744 \n",
       "L 494 -744 \n",
       "L 750 256 \n",
       "L 750 794 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-55\" d=\"M 556 4666 \n",
       "L 1191 4666 \n",
       "L 1191 1831 \n",
       "Q 1191 1081 1462 751 \n",
       "Q 1734 422 2344 422 \n",
       "Q 2950 422 3222 751 \n",
       "Q 3494 1081 3494 1831 \n",
       "L 3494 4666 \n",
       "L 4128 4666 \n",
       "L 4128 1753 \n",
       "Q 4128 841 3676 375 \n",
       "Q 3225 -91 2344 -91 \n",
       "Q 1459 -91 1007 375 \n",
       "Q 556 841 556 1753 \n",
       "L 556 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-4d\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-65\" x=\"86.279297\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-67\" x=\"147.802734\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-61\" x=\"211.279297\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-74\" x=\"272.558594\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-72\" x=\"311.767578\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-6f\" x=\"350.630859\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-6e\" x=\"411.8125\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-20\" x=\"475.191406\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-47\" x=\"506.978516\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-50\" x=\"584.46875\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-54\" x=\"644.771484\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-32\" x=\"705.855469\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-2c\" x=\"769.478516\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-20\" x=\"801.265625\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-32\" x=\"833.052734\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-20\" x=\"896.675781\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-47\" x=\"928.462891\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-50\" x=\"1005.953125\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-55\" x=\"1066.255859\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-73\" x=\"1139.449219\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_2\">\n",
       "     <g id=\"line2d_6\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#mdada20d4ad\" x=\"151.84375\" y=\"49.823182\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_7\">\n",
       "      <!-- Megatron GPT2, 2 GPUs, TP -->\n",
       "      <g transform=\"translate(7.2 53.622401)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-4d\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-65\" x=\"86.279297\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-67\" x=\"147.802734\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-61\" x=\"211.279297\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-74\" x=\"272.558594\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-72\" x=\"311.767578\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-6f\" x=\"350.630859\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-6e\" x=\"411.8125\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-20\" x=\"475.191406\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-47\" x=\"506.978516\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-50\" x=\"584.46875\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-54\" x=\"644.771484\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-32\" x=\"705.855469\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-2c\" x=\"769.478516\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-20\" x=\"801.265625\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-32\" x=\"833.052734\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-20\" x=\"896.675781\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-47\" x=\"928.462891\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-50\" x=\"1005.953125\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-55\" x=\"1066.255859\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-73\" x=\"1139.449219\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-2c\" x=\"1191.548828\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-20\" x=\"1223.335938\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-54\" x=\"1255.123047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-50\" x=\"1316.207031\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"patch_5\">\n",
       "    <path d=\"M 151.84375 61.56 \n",
       "L 151.84375 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_6\">\n",
       "    <path d=\"M 299.549632 61.56 \n",
       "L 299.549632 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_7\">\n",
       "    <path d=\"M 151.84375 61.56 \n",
       "L 299.549632 61.56 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_8\">\n",
       "    <path d=\"M 151.84375 7.2 \n",
       "L 299.549632 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "  </g>\n",
       "  <g id=\"axes_2\">\n",
       "   <g id=\"patch_9\">\n",
       "    <path d=\"M 329.090809 61.56 \n",
       "L 476.796691 61.56 \n",
       "L 476.796691 7.2 \n",
       "L 329.090809 7.2 \n",
       "z\n",
       "\" style=\"fill: #ffffff\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_10\">\n",
       "    <path d=\"M 329.090809 9.670909 \n",
       "L 469.763078 9.670909 \n",
       "L 469.763078 28.202727 \n",
       "L 329.090809 28.202727 \n",
       "z\n",
       "\" clip-path=\"url(#p9e8b4ebbb1)\" style=\"fill: #e41a1c\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_11\">\n",
       "    <path d=\"M 329.090809 40.557273 \n",
       "L 465.331797 40.557273 \n",
       "L 465.331797 59.089091 \n",
       "L 329.090809 59.089091 \n",
       "z\n",
       "\" clip-path=\"url(#p9e8b4ebbb1)\" style=\"fill: #377eb8\"/>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_3\">\n",
       "    <g id=\"xtick_5\">\n",
       "     <g id=\"line2d_7\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m412737860e\" x=\"329.090809\" y=\"61.56\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_8\">\n",
       "      <!-- 0 -->\n",
       "      <g transform=\"translate(325.909559 76.158437)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_6\">\n",
       "     <g id=\"line2d_8\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m412737860e\" x=\"366.049942\" y=\"61.56\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_9\">\n",
       "      <!-- 10 -->\n",
       "      <g transform=\"translate(359.687442 76.158437)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-31\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_7\">\n",
       "     <g id=\"line2d_9\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m412737860e\" x=\"403.009075\" y=\"61.56\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_10\">\n",
       "      <!-- 20 -->\n",
       "      <g transform=\"translate(396.646575 76.158437)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-32\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_8\">\n",
       "     <g id=\"line2d_10\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m412737860e\" x=\"439.968209\" y=\"61.56\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_11\">\n",
       "      <!-- 30 -->\n",
       "      <g transform=\"translate(433.605709 76.158437)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-33\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"text_12\">\n",
       "     <!-- per GPU TFLOPS -->\n",
       "     <g transform=\"translate(362.251563 89.836562)scale(0.1 -0.1)\">\n",
       "      <defs>\n",
       "       <path id=\"DejaVuSans-46\" d=\"M 628 4666 \n",
       "L 3309 4666 \n",
       "L 3309 4134 \n",
       "L 1259 4134 \n",
       "L 1259 2759 \n",
       "L 3109 2759 \n",
       "L 3109 2228 \n",
       "L 1259 2228 \n",
       "L 1259 0 \n",
       "L 628 0 \n",
       "L 628 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-4c\" d=\"M 628 4666 \n",
       "L 1259 4666 \n",
       "L 1259 531 \n",
       "L 3531 531 \n",
       "L 3531 0 \n",
       "L 628 0 \n",
       "L 628 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-4f\" d=\"M 2522 4238 \n",
       "Q 1834 4238 1429 3725 \n",
       "Q 1025 3213 1025 2328 \n",
       "Q 1025 1447 1429 934 \n",
       "Q 1834 422 2522 422 \n",
       "Q 3209 422 3611 934 \n",
       "Q 4013 1447 4013 2328 \n",
       "Q 4013 3213 3611 3725 \n",
       "Q 3209 4238 2522 4238 \n",
       "z\n",
       "M 2522 4750 \n",
       "Q 3503 4750 4090 4092 \n",
       "Q 4678 3434 4678 2328 \n",
       "Q 4678 1225 4090 567 \n",
       "Q 3503 -91 2522 -91 \n",
       "Q 1538 -91 948 565 \n",
       "Q 359 1222 359 2328 \n",
       "Q 359 3434 948 4092 \n",
       "Q 1538 4750 2522 4750 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "      </defs>\n",
       "      <use xlink:href=\"#DejaVuSans-70\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-65\" x=\"63.476562\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-72\" x=\"125\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-20\" x=\"166.113281\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-47\" x=\"197.900391\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-50\" x=\"275.390625\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-55\" x=\"335.693359\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-20\" x=\"408.886719\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-54\" x=\"440.673828\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-46\" x=\"501.757812\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-4c\" x=\"559.277344\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-4f\" x=\"611.365234\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-50\" x=\"690.076172\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-53\" x=\"750.378906\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_4\">\n",
       "    <g id=\"ytick_3\">\n",
       "     <g id=\"line2d_11\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#mdada20d4ad\" x=\"329.090809\" y=\"18.936818\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_4\">\n",
       "     <g id=\"line2d_12\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#mdada20d4ad\" x=\"329.090809\" y=\"49.823182\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"patch_12\">\n",
       "    <path d=\"M 329.090809 61.56 \n",
       "L 329.090809 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_13\">\n",
       "    <path d=\"M 476.796691 61.56 \n",
       "L 476.796691 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_14\">\n",
       "    <path d=\"M 329.090809 61.56 \n",
       "L 476.796691 61.56 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_15\">\n",
       "    <path d=\"M 329.090809 7.2 \n",
       "L 476.796691 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "  </g>\n",
       "  <g id=\"axes_3\">\n",
       "   <g id=\"patch_16\">\n",
       "    <path d=\"M 506.337868 61.56 \n",
       "L 654.04375 61.56 \n",
       "L 654.04375 7.2 \n",
       "L 506.337868 7.2 \n",
       "z\n",
       "\" style=\"fill: #ffffff\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_17\">\n",
       "    <path d=\"M 506.337868 9.670909 \n",
       "L 647.010137 9.670909 \n",
       "L 647.010137 28.202727 \n",
       "L 506.337868 28.202727 \n",
       "z\n",
       "\" clip-path=\"url(#p87dca1951a)\" style=\"fill: #e41a1c\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_18\">\n",
       "    <path d=\"M 506.337868 40.557273 \n",
       "L 635.209698 40.557273 \n",
       "L 635.209698 59.089091 \n",
       "L 506.337868 59.089091 \n",
       "z\n",
       "\" clip-path=\"url(#p87dca1951a)\" style=\"fill: #377eb8\"/>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_5\">\n",
       "    <g id=\"xtick_9\">\n",
       "     <g id=\"line2d_13\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m412737860e\" x=\"506.337868\" y=\"61.56\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_13\">\n",
       "      <!-- 0 -->\n",
       "      <g transform=\"translate(503.156618 76.158437)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_10\">\n",
       "     <g id=\"line2d_14\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m412737860e\" x=\"571.965898\" y=\"61.56\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_14\">\n",
       "      <!-- 10 -->\n",
       "      <g transform=\"translate(565.603398 76.158437)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-31\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_11\">\n",
       "     <g id=\"line2d_15\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m412737860e\" x=\"637.593927\" y=\"61.56\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_15\">\n",
       "      <!-- 20 -->\n",
       "      <g transform=\"translate(631.231427 76.158437)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-32\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"text_16\">\n",
       "     <!-- per GPU memory (GB) -->\n",
       "     <g transform=\"translate(524.471278 89.836562)scale(0.1 -0.1)\">\n",
       "      <defs>\n",
       "       <path id=\"DejaVuSans-79\" d=\"M 2059 -325 \n",
       "Q 1816 -950 1584 -1140 \n",
       "Q 1353 -1331 966 -1331 \n",
       "L 506 -1331 \n",
       "L 506 -850 \n",
       "L 844 -850 \n",
       "Q 1081 -850 1212 -737 \n",
       "Q 1344 -625 1503 -206 \n",
       "L 1606 56 \n",
       "L 191 3500 \n",
       "L 800 3500 \n",
       "L 1894 763 \n",
       "L 2988 3500 \n",
       "L 3597 3500 \n",
       "L 2059 -325 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-28\" d=\"M 1984 4856 \n",
       "Q 1566 4138 1362 3434 \n",
       "Q 1159 2731 1159 2009 \n",
       "Q 1159 1288 1364 580 \n",
       "Q 1569 -128 1984 -844 \n",
       "L 1484 -844 \n",
       "Q 1016 -109 783 600 \n",
       "Q 550 1309 550 2009 \n",
       "Q 550 2706 781 3412 \n",
       "Q 1013 4119 1484 4856 \n",
       "L 1984 4856 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-42\" d=\"M 1259 2228 \n",
       "L 1259 519 \n",
       "L 2272 519 \n",
       "Q 2781 519 3026 730 \n",
       "Q 3272 941 3272 1375 \n",
       "Q 3272 1813 3026 2020 \n",
       "Q 2781 2228 2272 2228 \n",
       "L 1259 2228 \n",
       "z\n",
       "M 1259 4147 \n",
       "L 1259 2741 \n",
       "L 2194 2741 \n",
       "Q 2656 2741 2882 2914 \n",
       "Q 3109 3088 3109 3444 \n",
       "Q 3109 3797 2882 3972 \n",
       "Q 2656 4147 2194 4147 \n",
       "L 1259 4147 \n",
       "z\n",
       "M 628 4666 \n",
       "L 2241 4666 \n",
       "Q 2963 4666 3353 4366 \n",
       "Q 3744 4066 3744 3513 \n",
       "Q 3744 3084 3544 2831 \n",
       "Q 3344 2578 2956 2516 \n",
       "Q 3422 2416 3680 2098 \n",
       "Q 3938 1781 3938 1306 \n",
       "Q 3938 681 3513 340 \n",
       "Q 3088 0 2303 0 \n",
       "L 628 0 \n",
       "L 628 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-29\" d=\"M 513 4856 \n",
       "L 1013 4856 \n",
       "Q 1481 4119 1714 3412 \n",
       "Q 1947 2706 1947 2009 \n",
       "Q 1947 1309 1714 600 \n",
       "Q 1481 -109 1013 -844 \n",
       "L 513 -844 \n",
       "Q 928 -128 1133 580 \n",
       "Q 1338 1288 1338 2009 \n",
       "Q 1338 2731 1133 3434 \n",
       "Q 928 4138 513 4856 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "      </defs>\n",
       "      <use xlink:href=\"#DejaVuSans-70\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-65\" x=\"63.476562\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-72\" x=\"125\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-20\" x=\"166.113281\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-47\" x=\"197.900391\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-50\" x=\"275.390625\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-55\" x=\"335.693359\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-20\" x=\"408.886719\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-6d\" x=\"440.673828\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-65\" x=\"538.085938\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-6d\" x=\"599.609375\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-6f\" x=\"697.021484\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-72\" x=\"758.203125\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-79\" x=\"799.316406\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-20\" x=\"858.496094\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-28\" x=\"890.283203\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-47\" x=\"929.296875\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-42\" x=\"1006.787109\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-29\" x=\"1075.390625\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_6\">\n",
       "    <g id=\"ytick_5\">\n",
       "     <g id=\"line2d_16\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#mdada20d4ad\" x=\"506.337868\" y=\"18.936818\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_6\">\n",
       "     <g id=\"line2d_17\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#mdada20d4ad\" x=\"506.337868\" y=\"49.823182\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"patch_19\">\n",
       "    <path d=\"M 506.337868 61.56 \n",
       "L 506.337868 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_20\">\n",
       "    <path d=\"M 654.04375 61.56 \n",
       "L 654.04375 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_21\">\n",
       "    <path d=\"M 506.337868 61.56 \n",
       "L 654.04375 61.56 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_22\">\n",
       "    <path d=\"M 506.337868 7.2 \n",
       "L 654.04375 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "  </g>\n",
       " </g>\n",
       " <defs>\n",
       "  <clipPath id=\"p5e0a3a65e3\">\n",
       "   <rect x=\"151.84375\" y=\"7.2\" width=\"147.705882\" height=\"54.36\"/>\n",
       "  </clipPath>\n",
       "  <clipPath id=\"p9e8b4ebbb1\">\n",
       "   <rect x=\"329.090809\" y=\"7.2\" width=\"147.705882\" height=\"54.36\"/>\n",
       "  </clipPath>\n",
       "  <clipPath id=\"p87dca1951a\">\n",
       "   <rect x=\"506.337868\" y=\"7.2\" width=\"147.705882\" height=\"54.36\"/>\n",
       "  </clipPath>\n",
       " </defs>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<Figure size 648x72 with 3 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "tp_gpt2 = megatron_gpt(Exp(\n",
    "    \"Megatron GPT2, 2 GPUs, TP\", \"gpt2-medium\", 10, bf16=True, gpus='0,1', tensor_para=2))\n",
    "compare([dp_gpt2, tp_gpt2])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a954d86d",
   "metadata": {},
   "source": [
    "TP offers a similar performance as DP. But note that communication happens in both forward and backward for TP, the time breakdown doesn't show the communication cost correctly. It also means gradient accumulation helps TP little. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "abde65f0",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Time breakdown\t\t: forward+backward 0.99, communication 0.00, optimizer 0.01\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       "  \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<svg xmlns:xlink=\"http://www.w3.org/1999/xlink\" width=\"661.24375pt\" height=\"153.47625pt\" viewBox=\"0 0 661.24375 153.47625\" xmlns=\"http://www.w3.org/2000/svg\" version=\"1.1\">\n",
       " <metadata>\n",
       "  <rdf:RDF xmlns:dc=\"http://purl.org/dc/elements/1.1/\" xmlns:cc=\"http://creativecommons.org/ns#\" xmlns:rdf=\"http://www.w3.org/1999/02/22-rdf-syntax-ns#\">\n",
       "   <cc:Work>\n",
       "    <dc:type rdf:resource=\"http://purl.org/dc/dcmitype/StillImage\"/>\n",
       "    <dc:date>2022-09-02T19:51:56.402712</dc:date>\n",
       "    <dc:format>image/svg+xml</dc:format>\n",
       "    <dc:creator>\n",
       "     <cc:Agent>\n",
       "      <dc:title>Matplotlib v3.5.2, https://matplotlib.org/</dc:title>\n",
       "     </cc:Agent>\n",
       "    </dc:creator>\n",
       "   </cc:Work>\n",
       "  </rdf:RDF>\n",
       " </metadata>\n",
       " <defs>\n",
       "  <style type=\"text/css\">*{stroke-linejoin: round; stroke-linecap: butt}</style>\n",
       " </defs>\n",
       " <g id=\"figure_1\">\n",
       "  <g id=\"patch_1\">\n",
       "   <path d=\"M 0 153.47625 \n",
       "L 661.24375 153.47625 \n",
       "L 661.24375 0 \n",
       "L 0 0 \n",
       "L 0 153.47625 \n",
       "z\n",
       "\" style=\"fill: none\"/>\n",
       "  </g>\n",
       "  <g id=\"axes_1\">\n",
       "   <g id=\"patch_2\">\n",
       "    <path d=\"M 151.84375 115.92 \n",
       "L 299.549632 115.92 \n",
       "L 299.549632 7.2 \n",
       "L 151.84375 7.2 \n",
       "z\n",
       "\" style=\"fill: #ffffff\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_3\">\n",
       "    <path d=\"M 151.84375 12.141818 \n",
       "L 273.91223 12.141818 \n",
       "L 273.91223 28.614545 \n",
       "L 151.84375 28.614545 \n",
       "z\n",
       "\" clip-path=\"url(#paac630af1d)\" style=\"fill: #e41a1c\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_4\">\n",
       "    <path d=\"M 151.84375 39.596364 \n",
       "L 292.516019 39.596364 \n",
       "L 292.516019 56.069091 \n",
       "L 151.84375 56.069091 \n",
       "z\n",
       "\" clip-path=\"url(#paac630af1d)\" style=\"fill: #377eb8\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_5\">\n",
       "    <path d=\"M 151.84375 67.050909 \n",
       "L 270.066982 67.050909 \n",
       "L 270.066982 83.523636 \n",
       "L 151.84375 83.523636 \n",
       "z\n",
       "\" clip-path=\"url(#paac630af1d)\" style=\"fill: #4daf4a\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_6\">\n",
       "    <path d=\"M 151.84375 94.505455 \n",
       "L 273.884145 94.505455 \n",
       "L 273.884145 110.978182 \n",
       "L 151.84375 110.978182 \n",
       "z\n",
       "\" clip-path=\"url(#paac630af1d)\" style=\"fill: #984ea3\"/>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_1\">\n",
       "    <g id=\"xtick_1\">\n",
       "     <g id=\"line2d_1\">\n",
       "      <defs>\n",
       "       <path id=\"m7be9755b06\" d=\"M 0 0 \n",
       "L 0 3.5 \n",
       "\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </defs>\n",
       "      <g>\n",
       "       <use xlink:href=\"#m7be9755b06\" x=\"151.84375\" y=\"115.92\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_1\">\n",
       "      <!-- 0 -->\n",
       "      <g transform=\"translate(148.6625 130.518438)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-30\" d=\"M 2034 4250 \n",
       "Q 1547 4250 1301 3770 \n",
       "Q 1056 3291 1056 2328 \n",
       "Q 1056 1369 1301 889 \n",
       "Q 1547 409 2034 409 \n",
       "Q 2525 409 2770 889 \n",
       "Q 3016 1369 3016 2328 \n",
       "Q 3016 3291 2770 3770 \n",
       "Q 2525 4250 2034 4250 \n",
       "z\n",
       "M 2034 4750 \n",
       "Q 2819 4750 3233 4129 \n",
       "Q 3647 3509 3647 2328 \n",
       "Q 3647 1150 3233 529 \n",
       "Q 2819 -91 2034 -91 \n",
       "Q 1250 -91 836 529 \n",
       "Q 422 1150 422 2328 \n",
       "Q 422 3509 836 4129 \n",
       "Q 1250 4750 2034 4750 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_2\">\n",
       "     <g id=\"line2d_2\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m7be9755b06\" x=\"191.625868\" y=\"115.92\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_2\">\n",
       "      <!-- 10 -->\n",
       "      <g transform=\"translate(185.263368 130.518438)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-31\" d=\"M 794 531 \n",
       "L 1825 531 \n",
       "L 1825 4091 \n",
       "L 703 3866 \n",
       "L 703 4441 \n",
       "L 1819 4666 \n",
       "L 2450 4666 \n",
       "L 2450 531 \n",
       "L 3481 531 \n",
       "L 3481 0 \n",
       "L 794 0 \n",
       "L 794 531 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-31\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_3\">\n",
       "     <g id=\"line2d_3\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m7be9755b06\" x=\"231.407985\" y=\"115.92\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_3\">\n",
       "      <!-- 20 -->\n",
       "      <g transform=\"translate(225.045485 130.518438)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-32\" d=\"M 1228 531 \n",
       "L 3431 531 \n",
       "L 3431 0 \n",
       "L 469 0 \n",
       "L 469 531 \n",
       "Q 828 903 1448 1529 \n",
       "Q 2069 2156 2228 2338 \n",
       "Q 2531 2678 2651 2914 \n",
       "Q 2772 3150 2772 3378 \n",
       "Q 2772 3750 2511 3984 \n",
       "Q 2250 4219 1831 4219 \n",
       "Q 1534 4219 1204 4116 \n",
       "Q 875 4013 500 3803 \n",
       "L 500 4441 \n",
       "Q 881 4594 1212 4672 \n",
       "Q 1544 4750 1819 4750 \n",
       "Q 2544 4750 2975 4387 \n",
       "Q 3406 4025 3406 3419 \n",
       "Q 3406 3131 3298 2873 \n",
       "Q 3191 2616 2906 2266 \n",
       "Q 2828 2175 2409 1742 \n",
       "Q 1991 1309 1228 531 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-32\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_4\">\n",
       "     <g id=\"line2d_4\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m7be9755b06\" x=\"271.190103\" y=\"115.92\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_4\">\n",
       "      <!-- 30 -->\n",
       "      <g transform=\"translate(264.827603 130.518438)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-33\" d=\"M 2597 2516 \n",
       "Q 3050 2419 3304 2112 \n",
       "Q 3559 1806 3559 1356 \n",
       "Q 3559 666 3084 287 \n",
       "Q 2609 -91 1734 -91 \n",
       "Q 1441 -91 1130 -33 \n",
       "Q 819 25 488 141 \n",
       "L 488 750 \n",
       "Q 750 597 1062 519 \n",
       "Q 1375 441 1716 441 \n",
       "Q 2309 441 2620 675 \n",
       "Q 2931 909 2931 1356 \n",
       "Q 2931 1769 2642 2001 \n",
       "Q 2353 2234 1838 2234 \n",
       "L 1294 2234 \n",
       "L 1294 2753 \n",
       "L 1863 2753 \n",
       "Q 2328 2753 2575 2939 \n",
       "Q 2822 3125 2822 3475 \n",
       "Q 2822 3834 2567 4026 \n",
       "Q 2313 4219 1838 4219 \n",
       "Q 1578 4219 1281 4162 \n",
       "Q 984 4106 628 3988 \n",
       "L 628 4550 \n",
       "Q 988 4650 1302 4700 \n",
       "Q 1616 4750 1894 4750 \n",
       "Q 2613 4750 3031 4423 \n",
       "Q 3450 4097 3450 3541 \n",
       "Q 3450 3153 3228 2886 \n",
       "Q 3006 2619 2597 2516 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-33\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"text_5\">\n",
       "     <!-- Samples / sec -->\n",
       "     <g transform=\"translate(191.050597 144.196562)scale(0.1 -0.1)\">\n",
       "      <defs>\n",
       "       <path id=\"DejaVuSans-53\" d=\"M 3425 4513 \n",
       "L 3425 3897 \n",
       "Q 3066 4069 2747 4153 \n",
       "Q 2428 4238 2131 4238 \n",
       "Q 1616 4238 1336 4038 \n",
       "Q 1056 3838 1056 3469 \n",
       "Q 1056 3159 1242 3001 \n",
       "Q 1428 2844 1947 2747 \n",
       "L 2328 2669 \n",
       "Q 3034 2534 3370 2195 \n",
       "Q 3706 1856 3706 1288 \n",
       "Q 3706 609 3251 259 \n",
       "Q 2797 -91 1919 -91 \n",
       "Q 1588 -91 1214 -16 \n",
       "Q 841 59 441 206 \n",
       "L 441 856 \n",
       "Q 825 641 1194 531 \n",
       "Q 1563 422 1919 422 \n",
       "Q 2459 422 2753 634 \n",
       "Q 3047 847 3047 1241 \n",
       "Q 3047 1584 2836 1778 \n",
       "Q 2625 1972 2144 2069 \n",
       "L 1759 2144 \n",
       "Q 1053 2284 737 2584 \n",
       "Q 422 2884 422 3419 \n",
       "Q 422 4038 858 4394 \n",
       "Q 1294 4750 2059 4750 \n",
       "Q 2388 4750 2728 4690 \n",
       "Q 3069 4631 3425 4513 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-61\" d=\"M 2194 1759 \n",
       "Q 1497 1759 1228 1600 \n",
       "Q 959 1441 959 1056 \n",
       "Q 959 750 1161 570 \n",
       "Q 1363 391 1709 391 \n",
       "Q 2188 391 2477 730 \n",
       "Q 2766 1069 2766 1631 \n",
       "L 2766 1759 \n",
       "L 2194 1759 \n",
       "z\n",
       "M 3341 1997 \n",
       "L 3341 0 \n",
       "L 2766 0 \n",
       "L 2766 531 \n",
       "Q 2569 213 2275 61 \n",
       "Q 1981 -91 1556 -91 \n",
       "Q 1019 -91 701 211 \n",
       "Q 384 513 384 1019 \n",
       "Q 384 1609 779 1909 \n",
       "Q 1175 2209 1959 2209 \n",
       "L 2766 2209 \n",
       "L 2766 2266 \n",
       "Q 2766 2663 2505 2880 \n",
       "Q 2244 3097 1772 3097 \n",
       "Q 1472 3097 1187 3025 \n",
       "Q 903 2953 641 2809 \n",
       "L 641 3341 \n",
       "Q 956 3463 1253 3523 \n",
       "Q 1550 3584 1831 3584 \n",
       "Q 2591 3584 2966 3190 \n",
       "Q 3341 2797 3341 1997 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-6d\" d=\"M 3328 2828 \n",
       "Q 3544 3216 3844 3400 \n",
       "Q 4144 3584 4550 3584 \n",
       "Q 5097 3584 5394 3201 \n",
       "Q 5691 2819 5691 2113 \n",
       "L 5691 0 \n",
       "L 5113 0 \n",
       "L 5113 2094 \n",
       "Q 5113 2597 4934 2840 \n",
       "Q 4756 3084 4391 3084 \n",
       "Q 3944 3084 3684 2787 \n",
       "Q 3425 2491 3425 1978 \n",
       "L 3425 0 \n",
       "L 2847 0 \n",
       "L 2847 2094 \n",
       "Q 2847 2600 2669 2842 \n",
       "Q 2491 3084 2119 3084 \n",
       "Q 1678 3084 1418 2786 \n",
       "Q 1159 2488 1159 1978 \n",
       "L 1159 0 \n",
       "L 581 0 \n",
       "L 581 3500 \n",
       "L 1159 3500 \n",
       "L 1159 2956 \n",
       "Q 1356 3278 1631 3431 \n",
       "Q 1906 3584 2284 3584 \n",
       "Q 2666 3584 2933 3390 \n",
       "Q 3200 3197 3328 2828 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-70\" d=\"M 1159 525 \n",
       "L 1159 -1331 \n",
       "L 581 -1331 \n",
       "L 581 3500 \n",
       "L 1159 3500 \n",
       "L 1159 2969 \n",
       "Q 1341 3281 1617 3432 \n",
       "Q 1894 3584 2278 3584 \n",
       "Q 2916 3584 3314 3078 \n",
       "Q 3713 2572 3713 1747 \n",
       "Q 3713 922 3314 415 \n",
       "Q 2916 -91 2278 -91 \n",
       "Q 1894 -91 1617 61 \n",
       "Q 1341 213 1159 525 \n",
       "z\n",
       "M 3116 1747 \n",
       "Q 3116 2381 2855 2742 \n",
       "Q 2594 3103 2138 3103 \n",
       "Q 1681 3103 1420 2742 \n",
       "Q 1159 2381 1159 1747 \n",
       "Q 1159 1113 1420 752 \n",
       "Q 1681 391 2138 391 \n",
       "Q 2594 391 2855 752 \n",
       "Q 3116 1113 3116 1747 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-6c\" d=\"M 603 4863 \n",
       "L 1178 4863 \n",
       "L 1178 0 \n",
       "L 603 0 \n",
       "L 603 4863 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-65\" d=\"M 3597 1894 \n",
       "L 3597 1613 \n",
       "L 953 1613 \n",
       "Q 991 1019 1311 708 \n",
       "Q 1631 397 2203 397 \n",
       "Q 2534 397 2845 478 \n",
       "Q 3156 559 3463 722 \n",
       "L 3463 178 \n",
       "Q 3153 47 2828 -22 \n",
       "Q 2503 -91 2169 -91 \n",
       "Q 1331 -91 842 396 \n",
       "Q 353 884 353 1716 \n",
       "Q 353 2575 817 3079 \n",
       "Q 1281 3584 2069 3584 \n",
       "Q 2775 3584 3186 3129 \n",
       "Q 3597 2675 3597 1894 \n",
       "z\n",
       "M 3022 2063 \n",
       "Q 3016 2534 2758 2815 \n",
       "Q 2500 3097 2075 3097 \n",
       "Q 1594 3097 1305 2825 \n",
       "Q 1016 2553 972 2059 \n",
       "L 3022 2063 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-73\" d=\"M 2834 3397 \n",
       "L 2834 2853 \n",
       "Q 2591 2978 2328 3040 \n",
       "Q 2066 3103 1784 3103 \n",
       "Q 1356 3103 1142 2972 \n",
       "Q 928 2841 928 2578 \n",
       "Q 928 2378 1081 2264 \n",
       "Q 1234 2150 1697 2047 \n",
       "L 1894 2003 \n",
       "Q 2506 1872 2764 1633 \n",
       "Q 3022 1394 3022 966 \n",
       "Q 3022 478 2636 193 \n",
       "Q 2250 -91 1575 -91 \n",
       "Q 1294 -91 989 -36 \n",
       "Q 684 19 347 128 \n",
       "L 347 722 \n",
       "Q 666 556 975 473 \n",
       "Q 1284 391 1588 391 \n",
       "Q 1994 391 2212 530 \n",
       "Q 2431 669 2431 922 \n",
       "Q 2431 1156 2273 1281 \n",
       "Q 2116 1406 1581 1522 \n",
       "L 1381 1569 \n",
       "Q 847 1681 609 1914 \n",
       "Q 372 2147 372 2553 \n",
       "Q 372 3047 722 3315 \n",
       "Q 1072 3584 1716 3584 \n",
       "Q 2034 3584 2315 3537 \n",
       "Q 2597 3491 2834 3397 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-20\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-2f\" d=\"M 1625 4666 \n",
       "L 2156 4666 \n",
       "L 531 -594 \n",
       "L 0 -594 \n",
       "L 1625 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-63\" d=\"M 3122 3366 \n",
       "L 3122 2828 \n",
       "Q 2878 2963 2633 3030 \n",
       "Q 2388 3097 2138 3097 \n",
       "Q 1578 3097 1268 2742 \n",
       "Q 959 2388 959 1747 \n",
       "Q 959 1106 1268 751 \n",
       "Q 1578 397 2138 397 \n",
       "Q 2388 397 2633 464 \n",
       "Q 2878 531 3122 666 \n",
       "L 3122 134 \n",
       "Q 2881 22 2623 -34 \n",
       "Q 2366 -91 2075 -91 \n",
       "Q 1284 -91 818 406 \n",
       "Q 353 903 353 1747 \n",
       "Q 353 2603 823 3093 \n",
       "Q 1294 3584 2113 3584 \n",
       "Q 2378 3584 2631 3529 \n",
       "Q 2884 3475 3122 3366 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "      </defs>\n",
       "      <use xlink:href=\"#DejaVuSans-53\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-61\" x=\"63.476562\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-6d\" x=\"124.755859\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-70\" x=\"222.167969\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-6c\" x=\"285.644531\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-65\" x=\"313.427734\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-73\" x=\"374.951172\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-20\" x=\"427.050781\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-2f\" x=\"458.837891\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-20\" x=\"492.529297\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-73\" x=\"524.316406\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-65\" x=\"576.416016\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-63\" x=\"637.939453\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_2\">\n",
       "    <g id=\"ytick_1\">\n",
       "     <g id=\"line2d_5\">\n",
       "      <defs>\n",
       "       <path id=\"m3ff76283a5\" d=\"M 0 0 \n",
       "L -3.5 0 \n",
       "\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </defs>\n",
       "      <g>\n",
       "       <use xlink:href=\"#m3ff76283a5\" x=\"151.84375\" y=\"20.378182\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_6\">\n",
       "      <!-- Megatron GPT2, 2 GPUs -->\n",
       "      <g transform=\"translate(25.69375 24.177401)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-4d\" d=\"M 628 4666 \n",
       "L 1569 4666 \n",
       "L 2759 1491 \n",
       "L 3956 4666 \n",
       "L 4897 4666 \n",
       "L 4897 0 \n",
       "L 4281 0 \n",
       "L 4281 4097 \n",
       "L 3078 897 \n",
       "L 2444 897 \n",
       "L 1241 4097 \n",
       "L 1241 0 \n",
       "L 628 0 \n",
       "L 628 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-67\" d=\"M 2906 1791 \n",
       "Q 2906 2416 2648 2759 \n",
       "Q 2391 3103 1925 3103 \n",
       "Q 1463 3103 1205 2759 \n",
       "Q 947 2416 947 1791 \n",
       "Q 947 1169 1205 825 \n",
       "Q 1463 481 1925 481 \n",
       "Q 2391 481 2648 825 \n",
       "Q 2906 1169 2906 1791 \n",
       "z\n",
       "M 3481 434 \n",
       "Q 3481 -459 3084 -895 \n",
       "Q 2688 -1331 1869 -1331 \n",
       "Q 1566 -1331 1297 -1286 \n",
       "Q 1028 -1241 775 -1147 \n",
       "L 775 -588 \n",
       "Q 1028 -725 1275 -790 \n",
       "Q 1522 -856 1778 -856 \n",
       "Q 2344 -856 2625 -561 \n",
       "Q 2906 -266 2906 331 \n",
       "L 2906 616 \n",
       "Q 2728 306 2450 153 \n",
       "Q 2172 0 1784 0 \n",
       "Q 1141 0 747 490 \n",
       "Q 353 981 353 1791 \n",
       "Q 353 2603 747 3093 \n",
       "Q 1141 3584 1784 3584 \n",
       "Q 2172 3584 2450 3431 \n",
       "Q 2728 3278 2906 2969 \n",
       "L 2906 3500 \n",
       "L 3481 3500 \n",
       "L 3481 434 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-74\" d=\"M 1172 4494 \n",
       "L 1172 3500 \n",
       "L 2356 3500 \n",
       "L 2356 3053 \n",
       "L 1172 3053 \n",
       "L 1172 1153 \n",
       "Q 1172 725 1289 603 \n",
       "Q 1406 481 1766 481 \n",
       "L 2356 481 \n",
       "L 2356 0 \n",
       "L 1766 0 \n",
       "Q 1100 0 847 248 \n",
       "Q 594 497 594 1153 \n",
       "L 594 3053 \n",
       "L 172 3053 \n",
       "L 172 3500 \n",
       "L 594 3500 \n",
       "L 594 4494 \n",
       "L 1172 4494 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-72\" d=\"M 2631 2963 \n",
       "Q 2534 3019 2420 3045 \n",
       "Q 2306 3072 2169 3072 \n",
       "Q 1681 3072 1420 2755 \n",
       "Q 1159 2438 1159 1844 \n",
       "L 1159 0 \n",
       "L 581 0 \n",
       "L 581 3500 \n",
       "L 1159 3500 \n",
       "L 1159 2956 \n",
       "Q 1341 3275 1631 3429 \n",
       "Q 1922 3584 2338 3584 \n",
       "Q 2397 3584 2469 3576 \n",
       "Q 2541 3569 2628 3553 \n",
       "L 2631 2963 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-6f\" d=\"M 1959 3097 \n",
       "Q 1497 3097 1228 2736 \n",
       "Q 959 2375 959 1747 \n",
       "Q 959 1119 1226 758 \n",
       "Q 1494 397 1959 397 \n",
       "Q 2419 397 2687 759 \n",
       "Q 2956 1122 2956 1747 \n",
       "Q 2956 2369 2687 2733 \n",
       "Q 2419 3097 1959 3097 \n",
       "z\n",
       "M 1959 3584 \n",
       "Q 2709 3584 3137 3096 \n",
       "Q 3566 2609 3566 1747 \n",
       "Q 3566 888 3137 398 \n",
       "Q 2709 -91 1959 -91 \n",
       "Q 1206 -91 779 398 \n",
       "Q 353 888 353 1747 \n",
       "Q 353 2609 779 3096 \n",
       "Q 1206 3584 1959 3584 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-6e\" d=\"M 3513 2113 \n",
       "L 3513 0 \n",
       "L 2938 0 \n",
       "L 2938 2094 \n",
       "Q 2938 2591 2744 2837 \n",
       "Q 2550 3084 2163 3084 \n",
       "Q 1697 3084 1428 2787 \n",
       "Q 1159 2491 1159 1978 \n",
       "L 1159 0 \n",
       "L 581 0 \n",
       "L 581 3500 \n",
       "L 1159 3500 \n",
       "L 1159 2956 \n",
       "Q 1366 3272 1645 3428 \n",
       "Q 1925 3584 2291 3584 \n",
       "Q 2894 3584 3203 3211 \n",
       "Q 3513 2838 3513 2113 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-47\" d=\"M 3809 666 \n",
       "L 3809 1919 \n",
       "L 2778 1919 \n",
       "L 2778 2438 \n",
       "L 4434 2438 \n",
       "L 4434 434 \n",
       "Q 4069 175 3628 42 \n",
       "Q 3188 -91 2688 -91 \n",
       "Q 1594 -91 976 548 \n",
       "Q 359 1188 359 2328 \n",
       "Q 359 3472 976 4111 \n",
       "Q 1594 4750 2688 4750 \n",
       "Q 3144 4750 3555 4637 \n",
       "Q 3966 4525 4313 4306 \n",
       "L 4313 3634 \n",
       "Q 3963 3931 3569 4081 \n",
       "Q 3175 4231 2741 4231 \n",
       "Q 1884 4231 1454 3753 \n",
       "Q 1025 3275 1025 2328 \n",
       "Q 1025 1384 1454 906 \n",
       "Q 1884 428 2741 428 \n",
       "Q 3075 428 3337 486 \n",
       "Q 3600 544 3809 666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-50\" d=\"M 1259 4147 \n",
       "L 1259 2394 \n",
       "L 2053 2394 \n",
       "Q 2494 2394 2734 2622 \n",
       "Q 2975 2850 2975 3272 \n",
       "Q 2975 3691 2734 3919 \n",
       "Q 2494 4147 2053 4147 \n",
       "L 1259 4147 \n",
       "z\n",
       "M 628 4666 \n",
       "L 2053 4666 \n",
       "Q 2838 4666 3239 4311 \n",
       "Q 3641 3956 3641 3272 \n",
       "Q 3641 2581 3239 2228 \n",
       "Q 2838 1875 2053 1875 \n",
       "L 1259 1875 \n",
       "L 1259 0 \n",
       "L 628 0 \n",
       "L 628 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-54\" d=\"M -19 4666 \n",
       "L 3928 4666 \n",
       "L 3928 4134 \n",
       "L 2272 4134 \n",
       "L 2272 0 \n",
       "L 1638 0 \n",
       "L 1638 4134 \n",
       "L -19 4134 \n",
       "L -19 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-2c\" d=\"M 750 794 \n",
       "L 1409 794 \n",
       "L 1409 256 \n",
       "L 897 -744 \n",
       "L 494 -744 \n",
       "L 750 256 \n",
       "L 750 794 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-55\" d=\"M 556 4666 \n",
       "L 1191 4666 \n",
       "L 1191 1831 \n",
       "Q 1191 1081 1462 751 \n",
       "Q 1734 422 2344 422 \n",
       "Q 2950 422 3222 751 \n",
       "Q 3494 1081 3494 1831 \n",
       "L 3494 4666 \n",
       "L 4128 4666 \n",
       "L 4128 1753 \n",
       "Q 4128 841 3676 375 \n",
       "Q 3225 -91 2344 -91 \n",
       "Q 1459 -91 1007 375 \n",
       "Q 556 841 556 1753 \n",
       "L 556 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-4d\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-65\" x=\"86.279297\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-67\" x=\"147.802734\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-61\" x=\"211.279297\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-74\" x=\"272.558594\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-72\" x=\"311.767578\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-6f\" x=\"350.630859\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-6e\" x=\"411.8125\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-20\" x=\"475.191406\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-47\" x=\"506.978516\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-50\" x=\"584.46875\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-54\" x=\"644.771484\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-32\" x=\"705.855469\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-2c\" x=\"769.478516\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-20\" x=\"801.265625\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-32\" x=\"833.052734\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-20\" x=\"896.675781\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-47\" x=\"928.462891\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-50\" x=\"1005.953125\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-55\" x=\"1066.255859\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-73\" x=\"1139.449219\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_2\">\n",
       "     <g id=\"line2d_6\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m3ff76283a5\" x=\"151.84375\" y=\"47.832727\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_7\">\n",
       "      <!-- Megatron GPT2, 2 GPUs -->\n",
       "      <g transform=\"translate(25.69375 46.792884)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-4d\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-65\" x=\"86.279297\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-67\" x=\"147.802734\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-61\" x=\"211.279297\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-74\" x=\"272.558594\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-72\" x=\"311.767578\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-6f\" x=\"350.630859\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-6e\" x=\"411.8125\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-20\" x=\"475.191406\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-47\" x=\"506.978516\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-50\" x=\"584.46875\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-54\" x=\"644.771484\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-32\" x=\"705.855469\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-2c\" x=\"769.478516\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-20\" x=\"801.265625\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-32\" x=\"833.052734\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-20\" x=\"896.675781\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-47\" x=\"928.462891\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-50\" x=\"1005.953125\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-55\" x=\"1066.255859\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-73\" x=\"1139.449219\"/>\n",
       "      </g>\n",
       "      <!-- grad_accum=4 -->\n",
       "      <g transform=\"translate(68.9625 57.990696)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-64\" d=\"M 2906 2969 \n",
       "L 2906 4863 \n",
       "L 3481 4863 \n",
       "L 3481 0 \n",
       "L 2906 0 \n",
       "L 2906 525 \n",
       "Q 2725 213 2448 61 \n",
       "Q 2172 -91 1784 -91 \n",
       "Q 1150 -91 751 415 \n",
       "Q 353 922 353 1747 \n",
       "Q 353 2572 751 3078 \n",
       "Q 1150 3584 1784 3584 \n",
       "Q 2172 3584 2448 3432 \n",
       "Q 2725 3281 2906 2969 \n",
       "z\n",
       "M 947 1747 \n",
       "Q 947 1113 1208 752 \n",
       "Q 1469 391 1925 391 \n",
       "Q 2381 391 2643 752 \n",
       "Q 2906 1113 2906 1747 \n",
       "Q 2906 2381 2643 2742 \n",
       "Q 2381 3103 1925 3103 \n",
       "Q 1469 3103 1208 2742 \n",
       "Q 947 2381 947 1747 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-5f\" d=\"M 3263 -1063 \n",
       "L 3263 -1509 \n",
       "L -63 -1509 \n",
       "L -63 -1063 \n",
       "L 3263 -1063 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-75\" d=\"M 544 1381 \n",
       "L 544 3500 \n",
       "L 1119 3500 \n",
       "L 1119 1403 \n",
       "Q 1119 906 1312 657 \n",
       "Q 1506 409 1894 409 \n",
       "Q 2359 409 2629 706 \n",
       "Q 2900 1003 2900 1516 \n",
       "L 2900 3500 \n",
       "L 3475 3500 \n",
       "L 3475 0 \n",
       "L 2900 0 \n",
       "L 2900 538 \n",
       "Q 2691 219 2414 64 \n",
       "Q 2138 -91 1772 -91 \n",
       "Q 1169 -91 856 284 \n",
       "Q 544 659 544 1381 \n",
       "z\n",
       "M 1991 3584 \n",
       "L 1991 3584 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-3d\" d=\"M 678 2906 \n",
       "L 4684 2906 \n",
       "L 4684 2381 \n",
       "L 678 2381 \n",
       "L 678 2906 \n",
       "z\n",
       "M 678 1631 \n",
       "L 4684 1631 \n",
       "L 4684 1100 \n",
       "L 678 1100 \n",
       "L 678 1631 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-34\" d=\"M 2419 4116 \n",
       "L 825 1625 \n",
       "L 2419 1625 \n",
       "L 2419 4116 \n",
       "z\n",
       "M 2253 4666 \n",
       "L 3047 4666 \n",
       "L 3047 1625 \n",
       "L 3713 1625 \n",
       "L 3713 1100 \n",
       "L 3047 1100 \n",
       "L 3047 0 \n",
       "L 2419 0 \n",
       "L 2419 1100 \n",
       "L 313 1100 \n",
       "L 313 1709 \n",
       "L 2253 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-67\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-72\" x=\"63.476562\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-61\" x=\"104.589844\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-64\" x=\"165.869141\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-5f\" x=\"229.345703\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-61\" x=\"279.345703\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-63\" x=\"340.625\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-63\" x=\"395.605469\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-75\" x=\"450.585938\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-6d\" x=\"513.964844\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-3d\" x=\"611.376953\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-34\" x=\"695.166016\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_3\">\n",
       "     <g id=\"line2d_7\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m3ff76283a5\" x=\"151.84375\" y=\"75.287273\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_8\">\n",
       "      <!-- Megatron GPT2, 2 GPUs, TP -->\n",
       "      <g transform=\"translate(7.2 79.086491)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-4d\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-65\" x=\"86.279297\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-67\" x=\"147.802734\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-61\" x=\"211.279297\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-74\" x=\"272.558594\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-72\" x=\"311.767578\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-6f\" x=\"350.630859\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-6e\" x=\"411.8125\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-20\" x=\"475.191406\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-47\" x=\"506.978516\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-50\" x=\"584.46875\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-54\" x=\"644.771484\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-32\" x=\"705.855469\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-2c\" x=\"769.478516\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-20\" x=\"801.265625\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-32\" x=\"833.052734\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-20\" x=\"896.675781\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-47\" x=\"928.462891\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-50\" x=\"1005.953125\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-55\" x=\"1066.255859\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-73\" x=\"1139.449219\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-2c\" x=\"1191.548828\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-20\" x=\"1223.335938\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-54\" x=\"1255.123047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-50\" x=\"1316.207031\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_4\">\n",
       "     <g id=\"line2d_8\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m3ff76283a5\" x=\"151.84375\" y=\"102.741818\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_9\">\n",
       "      <!-- Megatron GPT2, 2 GPUs, TP -->\n",
       "      <g transform=\"translate(7.2 101.701974)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-4d\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-65\" x=\"86.279297\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-67\" x=\"147.802734\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-61\" x=\"211.279297\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-74\" x=\"272.558594\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-72\" x=\"311.767578\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-6f\" x=\"350.630859\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-6e\" x=\"411.8125\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-20\" x=\"475.191406\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-47\" x=\"506.978516\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-50\" x=\"584.46875\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-54\" x=\"644.771484\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-32\" x=\"705.855469\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-2c\" x=\"769.478516\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-20\" x=\"801.265625\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-32\" x=\"833.052734\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-20\" x=\"896.675781\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-47\" x=\"928.462891\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-50\" x=\"1005.953125\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-55\" x=\"1066.255859\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-73\" x=\"1139.449219\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-2c\" x=\"1191.548828\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-20\" x=\"1223.335938\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-54\" x=\"1255.123047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-50\" x=\"1316.207031\"/>\n",
       "      </g>\n",
       "      <!-- grad_accum=4 -->\n",
       "      <g transform=\"translate(68.9625 112.899787)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-67\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-72\" x=\"63.476562\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-61\" x=\"104.589844\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-64\" x=\"165.869141\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-5f\" x=\"229.345703\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-61\" x=\"279.345703\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-63\" x=\"340.625\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-63\" x=\"395.605469\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-75\" x=\"450.585938\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-6d\" x=\"513.964844\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-3d\" x=\"611.376953\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-34\" x=\"695.166016\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"patch_7\">\n",
       "    <path d=\"M 151.84375 115.92 \n",
       "L 151.84375 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_8\">\n",
       "    <path d=\"M 299.549632 115.92 \n",
       "L 299.549632 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_9\">\n",
       "    <path d=\"M 151.84375 115.92 \n",
       "L 299.549632 115.92 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_10\">\n",
       "    <path d=\"M 151.84375 7.2 \n",
       "L 299.549632 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "  </g>\n",
       "  <g id=\"axes_2\">\n",
       "   <g id=\"patch_11\">\n",
       "    <path d=\"M 329.090809 115.92 \n",
       "L 476.796691 115.92 \n",
       "L 476.796691 7.2 \n",
       "L 329.090809 7.2 \n",
       "z\n",
       "\" style=\"fill: #ffffff\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_12\">\n",
       "    <path d=\"M 329.090809 12.141818 \n",
       "L 451.159289 12.141818 \n",
       "L 451.159289 28.614545 \n",
       "L 329.090809 28.614545 \n",
       "z\n",
       "\" clip-path=\"url(#pf18a2d1d33)\" style=\"fill: #e41a1c\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_13\">\n",
       "    <path d=\"M 329.090809 39.596364 \n",
       "L 469.763078 39.596364 \n",
       "L 469.763078 56.069091 \n",
       "L 329.090809 56.069091 \n",
       "z\n",
       "\" clip-path=\"url(#pf18a2d1d33)\" style=\"fill: #377eb8\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_14\">\n",
       "    <path d=\"M 329.090809 67.050909 \n",
       "L 447.314041 67.050909 \n",
       "L 447.314041 83.523636 \n",
       "L 329.090809 83.523636 \n",
       "z\n",
       "\" clip-path=\"url(#pf18a2d1d33)\" style=\"fill: #4daf4a\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_15\">\n",
       "    <path d=\"M 329.090809 94.505455 \n",
       "L 451.131203 94.505455 \n",
       "L 451.131203 110.978182 \n",
       "L 329.090809 110.978182 \n",
       "z\n",
       "\" clip-path=\"url(#pf18a2d1d33)\" style=\"fill: #984ea3\"/>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_3\">\n",
       "    <g id=\"xtick_5\">\n",
       "     <g id=\"line2d_9\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m7be9755b06\" x=\"329.090809\" y=\"115.92\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_10\">\n",
       "      <!-- 0 -->\n",
       "      <g transform=\"translate(325.909559 130.518438)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_6\">\n",
       "     <g id=\"line2d_10\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m7be9755b06\" x=\"393.233447\" y=\"115.92\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_11\">\n",
       "      <!-- 20 -->\n",
       "      <g transform=\"translate(386.870947 130.518438)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-32\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_7\">\n",
       "     <g id=\"line2d_11\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m7be9755b06\" x=\"457.376085\" y=\"115.92\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_12\">\n",
       "      <!-- 40 -->\n",
       "      <g transform=\"translate(451.013585 130.518438)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-34\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"text_13\">\n",
       "     <!-- per GPU TFLOPS -->\n",
       "     <g transform=\"translate(362.251563 144.196562)scale(0.1 -0.1)\">\n",
       "      <defs>\n",
       "       <path id=\"DejaVuSans-46\" d=\"M 628 4666 \n",
       "L 3309 4666 \n",
       "L 3309 4134 \n",
       "L 1259 4134 \n",
       "L 1259 2759 \n",
       "L 3109 2759 \n",
       "L 3109 2228 \n",
       "L 1259 2228 \n",
       "L 1259 0 \n",
       "L 628 0 \n",
       "L 628 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-4c\" d=\"M 628 4666 \n",
       "L 1259 4666 \n",
       "L 1259 531 \n",
       "L 3531 531 \n",
       "L 3531 0 \n",
       "L 628 0 \n",
       "L 628 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-4f\" d=\"M 2522 4238 \n",
       "Q 1834 4238 1429 3725 \n",
       "Q 1025 3213 1025 2328 \n",
       "Q 1025 1447 1429 934 \n",
       "Q 1834 422 2522 422 \n",
       "Q 3209 422 3611 934 \n",
       "Q 4013 1447 4013 2328 \n",
       "Q 4013 3213 3611 3725 \n",
       "Q 3209 4238 2522 4238 \n",
       "z\n",
       "M 2522 4750 \n",
       "Q 3503 4750 4090 4092 \n",
       "Q 4678 3434 4678 2328 \n",
       "Q 4678 1225 4090 567 \n",
       "Q 3503 -91 2522 -91 \n",
       "Q 1538 -91 948 565 \n",
       "Q 359 1222 359 2328 \n",
       "Q 359 3434 948 4092 \n",
       "Q 1538 4750 2522 4750 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "      </defs>\n",
       "      <use xlink:href=\"#DejaVuSans-70\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-65\" x=\"63.476562\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-72\" x=\"125\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-20\" x=\"166.113281\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-47\" x=\"197.900391\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-50\" x=\"275.390625\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-55\" x=\"335.693359\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-20\" x=\"408.886719\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-54\" x=\"440.673828\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-46\" x=\"501.757812\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-4c\" x=\"559.277344\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-4f\" x=\"611.365234\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-50\" x=\"690.076172\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-53\" x=\"750.378906\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_4\">\n",
       "    <g id=\"ytick_5\">\n",
       "     <g id=\"line2d_12\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m3ff76283a5\" x=\"329.090809\" y=\"20.378182\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_6\">\n",
       "     <g id=\"line2d_13\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m3ff76283a5\" x=\"329.090809\" y=\"47.832727\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_7\">\n",
       "     <g id=\"line2d_14\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m3ff76283a5\" x=\"329.090809\" y=\"75.287273\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_8\">\n",
       "     <g id=\"line2d_15\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m3ff76283a5\" x=\"329.090809\" y=\"102.741818\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"patch_16\">\n",
       "    <path d=\"M 329.090809 115.92 \n",
       "L 329.090809 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_17\">\n",
       "    <path d=\"M 476.796691 115.92 \n",
       "L 476.796691 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_18\">\n",
       "    <path d=\"M 329.090809 115.92 \n",
       "L 476.796691 115.92 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_19\">\n",
       "    <path d=\"M 329.090809 7.2 \n",
       "L 476.796691 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "  </g>\n",
       "  <g id=\"axes_3\">\n",
       "   <g id=\"patch_20\">\n",
       "    <path d=\"M 506.337868 115.92 \n",
       "L 654.04375 115.92 \n",
       "L 654.04375 7.2 \n",
       "L 506.337868 7.2 \n",
       "z\n",
       "\" style=\"fill: #ffffff\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_21\">\n",
       "    <path d=\"M 506.337868 12.141818 \n",
       "L 647.010137 12.141818 \n",
       "L 647.010137 28.614545 \n",
       "L 506.337868 28.614545 \n",
       "z\n",
       "\" clip-path=\"url(#pa148f1d99c)\" style=\"fill: #e41a1c\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_22\">\n",
       "    <path d=\"M 506.337868 39.596364 \n",
       "L 647.010137 39.596364 \n",
       "L 647.010137 56.069091 \n",
       "L 506.337868 56.069091 \n",
       "z\n",
       "\" clip-path=\"url(#pa148f1d99c)\" style=\"fill: #377eb8\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_23\">\n",
       "    <path d=\"M 506.337868 67.050909 \n",
       "L 635.209698 67.050909 \n",
       "L 635.209698 83.523636 \n",
       "L 506.337868 83.523636 \n",
       "z\n",
       "\" clip-path=\"url(#pa148f1d99c)\" style=\"fill: #4daf4a\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_24\">\n",
       "    <path d=\"M 506.337868 94.505455 \n",
       "L 635.20971 94.505455 \n",
       "L 635.20971 110.978182 \n",
       "L 506.337868 110.978182 \n",
       "z\n",
       "\" clip-path=\"url(#pa148f1d99c)\" style=\"fill: #984ea3\"/>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_5\">\n",
       "    <g id=\"xtick_8\">\n",
       "     <g id=\"line2d_16\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m7be9755b06\" x=\"506.337868\" y=\"115.92\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_14\">\n",
       "      <!-- 0 -->\n",
       "      <g transform=\"translate(503.156618 130.518438)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_9\">\n",
       "     <g id=\"line2d_17\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m7be9755b06\" x=\"571.965898\" y=\"115.92\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_15\">\n",
       "      <!-- 10 -->\n",
       "      <g transform=\"translate(565.603398 130.518438)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-31\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_10\">\n",
       "     <g id=\"line2d_18\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m7be9755b06\" x=\"637.593927\" y=\"115.92\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_16\">\n",
       "      <!-- 20 -->\n",
       "      <g transform=\"translate(631.231427 130.518438)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-32\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"text_17\">\n",
       "     <!-- per GPU memory (GB) -->\n",
       "     <g transform=\"translate(524.471278 144.196562)scale(0.1 -0.1)\">\n",
       "      <defs>\n",
       "       <path id=\"DejaVuSans-79\" d=\"M 2059 -325 \n",
       "Q 1816 -950 1584 -1140 \n",
       "Q 1353 -1331 966 -1331 \n",
       "L 506 -1331 \n",
       "L 506 -850 \n",
       "L 844 -850 \n",
       "Q 1081 -850 1212 -737 \n",
       "Q 1344 -625 1503 -206 \n",
       "L 1606 56 \n",
       "L 191 3500 \n",
       "L 800 3500 \n",
       "L 1894 763 \n",
       "L 2988 3500 \n",
       "L 3597 3500 \n",
       "L 2059 -325 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-28\" d=\"M 1984 4856 \n",
       "Q 1566 4138 1362 3434 \n",
       "Q 1159 2731 1159 2009 \n",
       "Q 1159 1288 1364 580 \n",
       "Q 1569 -128 1984 -844 \n",
       "L 1484 -844 \n",
       "Q 1016 -109 783 600 \n",
       "Q 550 1309 550 2009 \n",
       "Q 550 2706 781 3412 \n",
       "Q 1013 4119 1484 4856 \n",
       "L 1984 4856 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-42\" d=\"M 1259 2228 \n",
       "L 1259 519 \n",
       "L 2272 519 \n",
       "Q 2781 519 3026 730 \n",
       "Q 3272 941 3272 1375 \n",
       "Q 3272 1813 3026 2020 \n",
       "Q 2781 2228 2272 2228 \n",
       "L 1259 2228 \n",
       "z\n",
       "M 1259 4147 \n",
       "L 1259 2741 \n",
       "L 2194 2741 \n",
       "Q 2656 2741 2882 2914 \n",
       "Q 3109 3088 3109 3444 \n",
       "Q 3109 3797 2882 3972 \n",
       "Q 2656 4147 2194 4147 \n",
       "L 1259 4147 \n",
       "z\n",
       "M 628 4666 \n",
       "L 2241 4666 \n",
       "Q 2963 4666 3353 4366 \n",
       "Q 3744 4066 3744 3513 \n",
       "Q 3744 3084 3544 2831 \n",
       "Q 3344 2578 2956 2516 \n",
       "Q 3422 2416 3680 2098 \n",
       "Q 3938 1781 3938 1306 \n",
       "Q 3938 681 3513 340 \n",
       "Q 3088 0 2303 0 \n",
       "L 628 0 \n",
       "L 628 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-29\" d=\"M 513 4856 \n",
       "L 1013 4856 \n",
       "Q 1481 4119 1714 3412 \n",
       "Q 1947 2706 1947 2009 \n",
       "Q 1947 1309 1714 600 \n",
       "Q 1481 -109 1013 -844 \n",
       "L 513 -844 \n",
       "Q 928 -128 1133 580 \n",
       "Q 1338 1288 1338 2009 \n",
       "Q 1338 2731 1133 3434 \n",
       "Q 928 4138 513 4856 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "      </defs>\n",
       "      <use xlink:href=\"#DejaVuSans-70\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-65\" x=\"63.476562\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-72\" x=\"125\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-20\" x=\"166.113281\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-47\" x=\"197.900391\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-50\" x=\"275.390625\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-55\" x=\"335.693359\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-20\" x=\"408.886719\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-6d\" x=\"440.673828\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-65\" x=\"538.085938\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-6d\" x=\"599.609375\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-6f\" x=\"697.021484\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-72\" x=\"758.203125\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-79\" x=\"799.316406\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-20\" x=\"858.496094\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-28\" x=\"890.283203\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-47\" x=\"929.296875\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-42\" x=\"1006.787109\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-29\" x=\"1075.390625\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_6\">\n",
       "    <g id=\"ytick_9\">\n",
       "     <g id=\"line2d_19\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m3ff76283a5\" x=\"506.337868\" y=\"20.378182\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_10\">\n",
       "     <g id=\"line2d_20\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m3ff76283a5\" x=\"506.337868\" y=\"47.832727\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_11\">\n",
       "     <g id=\"line2d_21\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m3ff76283a5\" x=\"506.337868\" y=\"75.287273\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_12\">\n",
       "     <g id=\"line2d_22\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m3ff76283a5\" x=\"506.337868\" y=\"102.741818\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"patch_25\">\n",
       "    <path d=\"M 506.337868 115.92 \n",
       "L 506.337868 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_26\">\n",
       "    <path d=\"M 654.04375 115.92 \n",
       "L 654.04375 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_27\">\n",
       "    <path d=\"M 506.337868 115.92 \n",
       "L 654.04375 115.92 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_28\">\n",
       "    <path d=\"M 506.337868 7.2 \n",
       "L 654.04375 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "  </g>\n",
       " </g>\n",
       " <defs>\n",
       "  <clipPath id=\"paac630af1d\">\n",
       "   <rect x=\"151.84375\" y=\"7.2\" width=\"147.705882\" height=\"108.72\"/>\n",
       "  </clipPath>\n",
       "  <clipPath id=\"pf18a2d1d33\">\n",
       "   <rect x=\"329.090809\" y=\"7.2\" width=\"147.705882\" height=\"108.72\"/>\n",
       "  </clipPath>\n",
       "  <clipPath id=\"pa148f1d99c\">\n",
       "   <rect x=\"506.337868\" y=\"7.2\" width=\"147.705882\" height=\"108.72\"/>\n",
       "  </clipPath>\n",
       " </defs>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<Figure size 648x144 with 3 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "tp_gpt2_accum = megatron_gpt(Exp(\n",
    "    \"Megatron GPT2, 2 GPUs, TP\\ngrad_accum=4\", \"gpt2-medium\", 10, bf16=True, gpus='0,1',\n",
    "    tensor_para=2, grad_accum=4, steps=10))\n",
    "compare([dp_gpt2, dp_gpt2_accum, tp_gpt2, tp_gpt2_accum])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "15b74ae1",
   "metadata": {},
   "source": [
    "One benefit of TP is that we can run very large model that's impossible for DP (at least without gradient accumulation). Let's try a 1.3B GPT. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "3c7c124e",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Downloading: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1.35k/1.35k [00:00<00:00, 1.06MB/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Time breakdown\t\t: forward+backward 0.94, communication 0.01, optimizer 0.05\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       "  \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<svg xmlns:xlink=\"http://www.w3.org/1999/xlink\" width=\"705.820482pt\" height=\"99.11625pt\" viewBox=\"0 0 705.820482 99.11625\" xmlns=\"http://www.w3.org/2000/svg\" version=\"1.1\">\n",
       " <metadata>\n",
       "  <rdf:RDF xmlns:dc=\"http://purl.org/dc/elements/1.1/\" xmlns:cc=\"http://creativecommons.org/ns#\" xmlns:rdf=\"http://www.w3.org/1999/02/22-rdf-syntax-ns#\">\n",
       "   <cc:Work>\n",
       "    <dc:type rdf:resource=\"http://purl.org/dc/dcmitype/StillImage\"/>\n",
       "    <dc:date>2022-09-02T19:53:08.318259</dc:date>\n",
       "    <dc:format>image/svg+xml</dc:format>\n",
       "    <dc:creator>\n",
       "     <cc:Agent>\n",
       "      <dc:title>Matplotlib v3.5.2, https://matplotlib.org/</dc:title>\n",
       "     </cc:Agent>\n",
       "    </dc:creator>\n",
       "   </cc:Work>\n",
       "  </rdf:RDF>\n",
       " </metadata>\n",
       " <defs>\n",
       "  <style type=\"text/css\">*{stroke-linejoin: round; stroke-linecap: butt}</style>\n",
       " </defs>\n",
       " <g id=\"figure_1\">\n",
       "  <g id=\"patch_1\">\n",
       "   <path d=\"M 0 99.11625 \n",
       "L 705.820482 99.11625 \n",
       "L 705.820482 0 \n",
       "L 0 0 \n",
       "L 0 99.11625 \n",
       "z\n",
       "\" style=\"fill: none\"/>\n",
       "  </g>\n",
       "  <g id=\"axes_1\">\n",
       "   <g id=\"patch_2\">\n",
       "    <path d=\"M 194.489062 61.56 \n",
       "L 342.194945 61.56 \n",
       "L 342.194945 7.2 \n",
       "L 194.489062 7.2 \n",
       "z\n",
       "\" style=\"fill: #ffffff\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_3\">\n",
       "    <path d=\"M 194.489062 9.670909 \n",
       "L 335.161331 9.670909 \n",
       "L 335.161331 28.202727 \n",
       "L 194.489062 28.202727 \n",
       "z\n",
       "\" clip-path=\"url(#p0faeee0155)\" style=\"fill: #e41a1c\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_4\">\n",
       "    <path d=\"M 194.489062 40.557273 \n",
       "L 216.879513 40.557273 \n",
       "L 216.879513 59.089091 \n",
       "L 194.489062 59.089091 \n",
       "z\n",
       "\" clip-path=\"url(#p0faeee0155)\" style=\"fill: #377eb8\"/>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_1\">\n",
       "    <g id=\"xtick_1\">\n",
       "     <g id=\"line2d_1\">\n",
       "      <defs>\n",
       "       <path id=\"m90391b8f49\" d=\"M 0 0 \n",
       "L 0 3.5 \n",
       "\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </defs>\n",
       "      <g>\n",
       "       <use xlink:href=\"#m90391b8f49\" x=\"194.489062\" y=\"61.56\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_1\">\n",
       "      <!-- 0 -->\n",
       "      <g transform=\"translate(191.307812 76.158437)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-30\" d=\"M 2034 4250 \n",
       "Q 1547 4250 1301 3770 \n",
       "Q 1056 3291 1056 2328 \n",
       "Q 1056 1369 1301 889 \n",
       "Q 1547 409 2034 409 \n",
       "Q 2525 409 2770 889 \n",
       "Q 3016 1369 3016 2328 \n",
       "Q 3016 3291 2770 3770 \n",
       "Q 2525 4250 2034 4250 \n",
       "z\n",
       "M 2034 4750 \n",
       "Q 2819 4750 3233 4129 \n",
       "Q 3647 3509 3647 2328 \n",
       "Q 3647 1150 3233 529 \n",
       "Q 2819 -91 2034 -91 \n",
       "Q 1250 -91 836 529 \n",
       "Q 422 1150 422 2328 \n",
       "Q 422 3509 836 4129 \n",
       "Q 1250 4750 2034 4750 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_2\">\n",
       "     <g id=\"line2d_2\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m90391b8f49\" x=\"240.344705\" y=\"61.56\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_2\">\n",
       "      <!-- 10 -->\n",
       "      <g transform=\"translate(233.982205 76.158437)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-31\" d=\"M 794 531 \n",
       "L 1825 531 \n",
       "L 1825 4091 \n",
       "L 703 3866 \n",
       "L 703 4441 \n",
       "L 1819 4666 \n",
       "L 2450 4666 \n",
       "L 2450 531 \n",
       "L 3481 531 \n",
       "L 3481 0 \n",
       "L 794 0 \n",
       "L 794 531 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-31\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_3\">\n",
       "     <g id=\"line2d_3\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m90391b8f49\" x=\"286.200348\" y=\"61.56\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_3\">\n",
       "      <!-- 20 -->\n",
       "      <g transform=\"translate(279.837848 76.158437)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-32\" d=\"M 1228 531 \n",
       "L 3431 531 \n",
       "L 3431 0 \n",
       "L 469 0 \n",
       "L 469 531 \n",
       "Q 828 903 1448 1529 \n",
       "Q 2069 2156 2228 2338 \n",
       "Q 2531 2678 2651 2914 \n",
       "Q 2772 3150 2772 3378 \n",
       "Q 2772 3750 2511 3984 \n",
       "Q 2250 4219 1831 4219 \n",
       "Q 1534 4219 1204 4116 \n",
       "Q 875 4013 500 3803 \n",
       "L 500 4441 \n",
       "Q 881 4594 1212 4672 \n",
       "Q 1544 4750 1819 4750 \n",
       "Q 2544 4750 2975 4387 \n",
       "Q 3406 4025 3406 3419 \n",
       "Q 3406 3131 3298 2873 \n",
       "Q 3191 2616 2906 2266 \n",
       "Q 2828 2175 2409 1742 \n",
       "Q 1991 1309 1228 531 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-32\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_4\">\n",
       "     <g id=\"line2d_4\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m90391b8f49\" x=\"332.055991\" y=\"61.56\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_4\">\n",
       "      <!-- 30 -->\n",
       "      <g transform=\"translate(325.693491 76.158437)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-33\" d=\"M 2597 2516 \n",
       "Q 3050 2419 3304 2112 \n",
       "Q 3559 1806 3559 1356 \n",
       "Q 3559 666 3084 287 \n",
       "Q 2609 -91 1734 -91 \n",
       "Q 1441 -91 1130 -33 \n",
       "Q 819 25 488 141 \n",
       "L 488 750 \n",
       "Q 750 597 1062 519 \n",
       "Q 1375 441 1716 441 \n",
       "Q 2309 441 2620 675 \n",
       "Q 2931 909 2931 1356 \n",
       "Q 2931 1769 2642 2001 \n",
       "Q 2353 2234 1838 2234 \n",
       "L 1294 2234 \n",
       "L 1294 2753 \n",
       "L 1863 2753 \n",
       "Q 2328 2753 2575 2939 \n",
       "Q 2822 3125 2822 3475 \n",
       "Q 2822 3834 2567 4026 \n",
       "Q 2313 4219 1838 4219 \n",
       "Q 1578 4219 1281 4162 \n",
       "Q 984 4106 628 3988 \n",
       "L 628 4550 \n",
       "Q 988 4650 1302 4700 \n",
       "Q 1616 4750 1894 4750 \n",
       "Q 2613 4750 3031 4423 \n",
       "Q 3450 4097 3450 3541 \n",
       "Q 3450 3153 3228 2886 \n",
       "Q 3006 2619 2597 2516 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-33\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"text_5\">\n",
       "     <!-- Samples / sec -->\n",
       "     <g transform=\"translate(233.69591 89.836562)scale(0.1 -0.1)\">\n",
       "      <defs>\n",
       "       <path id=\"DejaVuSans-53\" d=\"M 3425 4513 \n",
       "L 3425 3897 \n",
       "Q 3066 4069 2747 4153 \n",
       "Q 2428 4238 2131 4238 \n",
       "Q 1616 4238 1336 4038 \n",
       "Q 1056 3838 1056 3469 \n",
       "Q 1056 3159 1242 3001 \n",
       "Q 1428 2844 1947 2747 \n",
       "L 2328 2669 \n",
       "Q 3034 2534 3370 2195 \n",
       "Q 3706 1856 3706 1288 \n",
       "Q 3706 609 3251 259 \n",
       "Q 2797 -91 1919 -91 \n",
       "Q 1588 -91 1214 -16 \n",
       "Q 841 59 441 206 \n",
       "L 441 856 \n",
       "Q 825 641 1194 531 \n",
       "Q 1563 422 1919 422 \n",
       "Q 2459 422 2753 634 \n",
       "Q 3047 847 3047 1241 \n",
       "Q 3047 1584 2836 1778 \n",
       "Q 2625 1972 2144 2069 \n",
       "L 1759 2144 \n",
       "Q 1053 2284 737 2584 \n",
       "Q 422 2884 422 3419 \n",
       "Q 422 4038 858 4394 \n",
       "Q 1294 4750 2059 4750 \n",
       "Q 2388 4750 2728 4690 \n",
       "Q 3069 4631 3425 4513 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-61\" d=\"M 2194 1759 \n",
       "Q 1497 1759 1228 1600 \n",
       "Q 959 1441 959 1056 \n",
       "Q 959 750 1161 570 \n",
       "Q 1363 391 1709 391 \n",
       "Q 2188 391 2477 730 \n",
       "Q 2766 1069 2766 1631 \n",
       "L 2766 1759 \n",
       "L 2194 1759 \n",
       "z\n",
       "M 3341 1997 \n",
       "L 3341 0 \n",
       "L 2766 0 \n",
       "L 2766 531 \n",
       "Q 2569 213 2275 61 \n",
       "Q 1981 -91 1556 -91 \n",
       "Q 1019 -91 701 211 \n",
       "Q 384 513 384 1019 \n",
       "Q 384 1609 779 1909 \n",
       "Q 1175 2209 1959 2209 \n",
       "L 2766 2209 \n",
       "L 2766 2266 \n",
       "Q 2766 2663 2505 2880 \n",
       "Q 2244 3097 1772 3097 \n",
       "Q 1472 3097 1187 3025 \n",
       "Q 903 2953 641 2809 \n",
       "L 641 3341 \n",
       "Q 956 3463 1253 3523 \n",
       "Q 1550 3584 1831 3584 \n",
       "Q 2591 3584 2966 3190 \n",
       "Q 3341 2797 3341 1997 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-6d\" d=\"M 3328 2828 \n",
       "Q 3544 3216 3844 3400 \n",
       "Q 4144 3584 4550 3584 \n",
       "Q 5097 3584 5394 3201 \n",
       "Q 5691 2819 5691 2113 \n",
       "L 5691 0 \n",
       "L 5113 0 \n",
       "L 5113 2094 \n",
       "Q 5113 2597 4934 2840 \n",
       "Q 4756 3084 4391 3084 \n",
       "Q 3944 3084 3684 2787 \n",
       "Q 3425 2491 3425 1978 \n",
       "L 3425 0 \n",
       "L 2847 0 \n",
       "L 2847 2094 \n",
       "Q 2847 2600 2669 2842 \n",
       "Q 2491 3084 2119 3084 \n",
       "Q 1678 3084 1418 2786 \n",
       "Q 1159 2488 1159 1978 \n",
       "L 1159 0 \n",
       "L 581 0 \n",
       "L 581 3500 \n",
       "L 1159 3500 \n",
       "L 1159 2956 \n",
       "Q 1356 3278 1631 3431 \n",
       "Q 1906 3584 2284 3584 \n",
       "Q 2666 3584 2933 3390 \n",
       "Q 3200 3197 3328 2828 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-70\" d=\"M 1159 525 \n",
       "L 1159 -1331 \n",
       "L 581 -1331 \n",
       "L 581 3500 \n",
       "L 1159 3500 \n",
       "L 1159 2969 \n",
       "Q 1341 3281 1617 3432 \n",
       "Q 1894 3584 2278 3584 \n",
       "Q 2916 3584 3314 3078 \n",
       "Q 3713 2572 3713 1747 \n",
       "Q 3713 922 3314 415 \n",
       "Q 2916 -91 2278 -91 \n",
       "Q 1894 -91 1617 61 \n",
       "Q 1341 213 1159 525 \n",
       "z\n",
       "M 3116 1747 \n",
       "Q 3116 2381 2855 2742 \n",
       "Q 2594 3103 2138 3103 \n",
       "Q 1681 3103 1420 2742 \n",
       "Q 1159 2381 1159 1747 \n",
       "Q 1159 1113 1420 752 \n",
       "Q 1681 391 2138 391 \n",
       "Q 2594 391 2855 752 \n",
       "Q 3116 1113 3116 1747 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-6c\" d=\"M 603 4863 \n",
       "L 1178 4863 \n",
       "L 1178 0 \n",
       "L 603 0 \n",
       "L 603 4863 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-65\" d=\"M 3597 1894 \n",
       "L 3597 1613 \n",
       "L 953 1613 \n",
       "Q 991 1019 1311 708 \n",
       "Q 1631 397 2203 397 \n",
       "Q 2534 397 2845 478 \n",
       "Q 3156 559 3463 722 \n",
       "L 3463 178 \n",
       "Q 3153 47 2828 -22 \n",
       "Q 2503 -91 2169 -91 \n",
       "Q 1331 -91 842 396 \n",
       "Q 353 884 353 1716 \n",
       "Q 353 2575 817 3079 \n",
       "Q 1281 3584 2069 3584 \n",
       "Q 2775 3584 3186 3129 \n",
       "Q 3597 2675 3597 1894 \n",
       "z\n",
       "M 3022 2063 \n",
       "Q 3016 2534 2758 2815 \n",
       "Q 2500 3097 2075 3097 \n",
       "Q 1594 3097 1305 2825 \n",
       "Q 1016 2553 972 2059 \n",
       "L 3022 2063 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-73\" d=\"M 2834 3397 \n",
       "L 2834 2853 \n",
       "Q 2591 2978 2328 3040 \n",
       "Q 2066 3103 1784 3103 \n",
       "Q 1356 3103 1142 2972 \n",
       "Q 928 2841 928 2578 \n",
       "Q 928 2378 1081 2264 \n",
       "Q 1234 2150 1697 2047 \n",
       "L 1894 2003 \n",
       "Q 2506 1872 2764 1633 \n",
       "Q 3022 1394 3022 966 \n",
       "Q 3022 478 2636 193 \n",
       "Q 2250 -91 1575 -91 \n",
       "Q 1294 -91 989 -36 \n",
       "Q 684 19 347 128 \n",
       "L 347 722 \n",
       "Q 666 556 975 473 \n",
       "Q 1284 391 1588 391 \n",
       "Q 1994 391 2212 530 \n",
       "Q 2431 669 2431 922 \n",
       "Q 2431 1156 2273 1281 \n",
       "Q 2116 1406 1581 1522 \n",
       "L 1381 1569 \n",
       "Q 847 1681 609 1914 \n",
       "Q 372 2147 372 2553 \n",
       "Q 372 3047 722 3315 \n",
       "Q 1072 3584 1716 3584 \n",
       "Q 2034 3584 2315 3537 \n",
       "Q 2597 3491 2834 3397 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-20\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-2f\" d=\"M 1625 4666 \n",
       "L 2156 4666 \n",
       "L 531 -594 \n",
       "L 0 -594 \n",
       "L 1625 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-63\" d=\"M 3122 3366 \n",
       "L 3122 2828 \n",
       "Q 2878 2963 2633 3030 \n",
       "Q 2388 3097 2138 3097 \n",
       "Q 1578 3097 1268 2742 \n",
       "Q 959 2388 959 1747 \n",
       "Q 959 1106 1268 751 \n",
       "Q 1578 397 2138 397 \n",
       "Q 2388 397 2633 464 \n",
       "Q 2878 531 3122 666 \n",
       "L 3122 134 \n",
       "Q 2881 22 2623 -34 \n",
       "Q 2366 -91 2075 -91 \n",
       "Q 1284 -91 818 406 \n",
       "Q 353 903 353 1747 \n",
       "Q 353 2603 823 3093 \n",
       "Q 1294 3584 2113 3584 \n",
       "Q 2378 3584 2631 3529 \n",
       "Q 2884 3475 3122 3366 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "      </defs>\n",
       "      <use xlink:href=\"#DejaVuSans-53\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-61\" x=\"63.476562\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-6d\" x=\"124.755859\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-70\" x=\"222.167969\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-6c\" x=\"285.644531\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-65\" x=\"313.427734\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-73\" x=\"374.951172\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-20\" x=\"427.050781\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-2f\" x=\"458.837891\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-20\" x=\"492.529297\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-73\" x=\"524.316406\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-65\" x=\"576.416016\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-63\" x=\"637.939453\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_2\">\n",
       "    <g id=\"ytick_1\">\n",
       "     <g id=\"line2d_5\">\n",
       "      <defs>\n",
       "       <path id=\"mc796056d31\" d=\"M 0 0 \n",
       "L -3.5 0 \n",
       "\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </defs>\n",
       "      <g>\n",
       "       <use xlink:href=\"#mc796056d31\" x=\"194.489062\" y=\"18.936818\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_6\">\n",
       "      <!-- Megatron GPT2, 2 GPUs, TP -->\n",
       "      <g transform=\"translate(49.845312 17.896974)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-4d\" d=\"M 628 4666 \n",
       "L 1569 4666 \n",
       "L 2759 1491 \n",
       "L 3956 4666 \n",
       "L 4897 4666 \n",
       "L 4897 0 \n",
       "L 4281 0 \n",
       "L 4281 4097 \n",
       "L 3078 897 \n",
       "L 2444 897 \n",
       "L 1241 4097 \n",
       "L 1241 0 \n",
       "L 628 0 \n",
       "L 628 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-67\" d=\"M 2906 1791 \n",
       "Q 2906 2416 2648 2759 \n",
       "Q 2391 3103 1925 3103 \n",
       "Q 1463 3103 1205 2759 \n",
       "Q 947 2416 947 1791 \n",
       "Q 947 1169 1205 825 \n",
       "Q 1463 481 1925 481 \n",
       "Q 2391 481 2648 825 \n",
       "Q 2906 1169 2906 1791 \n",
       "z\n",
       "M 3481 434 \n",
       "Q 3481 -459 3084 -895 \n",
       "Q 2688 -1331 1869 -1331 \n",
       "Q 1566 -1331 1297 -1286 \n",
       "Q 1028 -1241 775 -1147 \n",
       "L 775 -588 \n",
       "Q 1028 -725 1275 -790 \n",
       "Q 1522 -856 1778 -856 \n",
       "Q 2344 -856 2625 -561 \n",
       "Q 2906 -266 2906 331 \n",
       "L 2906 616 \n",
       "Q 2728 306 2450 153 \n",
       "Q 2172 0 1784 0 \n",
       "Q 1141 0 747 490 \n",
       "Q 353 981 353 1791 \n",
       "Q 353 2603 747 3093 \n",
       "Q 1141 3584 1784 3584 \n",
       "Q 2172 3584 2450 3431 \n",
       "Q 2728 3278 2906 2969 \n",
       "L 2906 3500 \n",
       "L 3481 3500 \n",
       "L 3481 434 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-74\" d=\"M 1172 4494 \n",
       "L 1172 3500 \n",
       "L 2356 3500 \n",
       "L 2356 3053 \n",
       "L 1172 3053 \n",
       "L 1172 1153 \n",
       "Q 1172 725 1289 603 \n",
       "Q 1406 481 1766 481 \n",
       "L 2356 481 \n",
       "L 2356 0 \n",
       "L 1766 0 \n",
       "Q 1100 0 847 248 \n",
       "Q 594 497 594 1153 \n",
       "L 594 3053 \n",
       "L 172 3053 \n",
       "L 172 3500 \n",
       "L 594 3500 \n",
       "L 594 4494 \n",
       "L 1172 4494 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-72\" d=\"M 2631 2963 \n",
       "Q 2534 3019 2420 3045 \n",
       "Q 2306 3072 2169 3072 \n",
       "Q 1681 3072 1420 2755 \n",
       "Q 1159 2438 1159 1844 \n",
       "L 1159 0 \n",
       "L 581 0 \n",
       "L 581 3500 \n",
       "L 1159 3500 \n",
       "L 1159 2956 \n",
       "Q 1341 3275 1631 3429 \n",
       "Q 1922 3584 2338 3584 \n",
       "Q 2397 3584 2469 3576 \n",
       "Q 2541 3569 2628 3553 \n",
       "L 2631 2963 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-6f\" d=\"M 1959 3097 \n",
       "Q 1497 3097 1228 2736 \n",
       "Q 959 2375 959 1747 \n",
       "Q 959 1119 1226 758 \n",
       "Q 1494 397 1959 397 \n",
       "Q 2419 397 2687 759 \n",
       "Q 2956 1122 2956 1747 \n",
       "Q 2956 2369 2687 2733 \n",
       "Q 2419 3097 1959 3097 \n",
       "z\n",
       "M 1959 3584 \n",
       "Q 2709 3584 3137 3096 \n",
       "Q 3566 2609 3566 1747 \n",
       "Q 3566 888 3137 398 \n",
       "Q 2709 -91 1959 -91 \n",
       "Q 1206 -91 779 398 \n",
       "Q 353 888 353 1747 \n",
       "Q 353 2609 779 3096 \n",
       "Q 1206 3584 1959 3584 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-6e\" d=\"M 3513 2113 \n",
       "L 3513 0 \n",
       "L 2938 0 \n",
       "L 2938 2094 \n",
       "Q 2938 2591 2744 2837 \n",
       "Q 2550 3084 2163 3084 \n",
       "Q 1697 3084 1428 2787 \n",
       "Q 1159 2491 1159 1978 \n",
       "L 1159 0 \n",
       "L 581 0 \n",
       "L 581 3500 \n",
       "L 1159 3500 \n",
       "L 1159 2956 \n",
       "Q 1366 3272 1645 3428 \n",
       "Q 1925 3584 2291 3584 \n",
       "Q 2894 3584 3203 3211 \n",
       "Q 3513 2838 3513 2113 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-47\" d=\"M 3809 666 \n",
       "L 3809 1919 \n",
       "L 2778 1919 \n",
       "L 2778 2438 \n",
       "L 4434 2438 \n",
       "L 4434 434 \n",
       "Q 4069 175 3628 42 \n",
       "Q 3188 -91 2688 -91 \n",
       "Q 1594 -91 976 548 \n",
       "Q 359 1188 359 2328 \n",
       "Q 359 3472 976 4111 \n",
       "Q 1594 4750 2688 4750 \n",
       "Q 3144 4750 3555 4637 \n",
       "Q 3966 4525 4313 4306 \n",
       "L 4313 3634 \n",
       "Q 3963 3931 3569 4081 \n",
       "Q 3175 4231 2741 4231 \n",
       "Q 1884 4231 1454 3753 \n",
       "Q 1025 3275 1025 2328 \n",
       "Q 1025 1384 1454 906 \n",
       "Q 1884 428 2741 428 \n",
       "Q 3075 428 3337 486 \n",
       "Q 3600 544 3809 666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-50\" d=\"M 1259 4147 \n",
       "L 1259 2394 \n",
       "L 2053 2394 \n",
       "Q 2494 2394 2734 2622 \n",
       "Q 2975 2850 2975 3272 \n",
       "Q 2975 3691 2734 3919 \n",
       "Q 2494 4147 2053 4147 \n",
       "L 1259 4147 \n",
       "z\n",
       "M 628 4666 \n",
       "L 2053 4666 \n",
       "Q 2838 4666 3239 4311 \n",
       "Q 3641 3956 3641 3272 \n",
       "Q 3641 2581 3239 2228 \n",
       "Q 2838 1875 2053 1875 \n",
       "L 1259 1875 \n",
       "L 1259 0 \n",
       "L 628 0 \n",
       "L 628 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-54\" d=\"M -19 4666 \n",
       "L 3928 4666 \n",
       "L 3928 4134 \n",
       "L 2272 4134 \n",
       "L 2272 0 \n",
       "L 1638 0 \n",
       "L 1638 4134 \n",
       "L -19 4134 \n",
       "L -19 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-2c\" d=\"M 750 794 \n",
       "L 1409 794 \n",
       "L 1409 256 \n",
       "L 897 -744 \n",
       "L 494 -744 \n",
       "L 750 256 \n",
       "L 750 794 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-55\" d=\"M 556 4666 \n",
       "L 1191 4666 \n",
       "L 1191 1831 \n",
       "Q 1191 1081 1462 751 \n",
       "Q 1734 422 2344 422 \n",
       "Q 2950 422 3222 751 \n",
       "Q 3494 1081 3494 1831 \n",
       "L 3494 4666 \n",
       "L 4128 4666 \n",
       "L 4128 1753 \n",
       "Q 4128 841 3676 375 \n",
       "Q 3225 -91 2344 -91 \n",
       "Q 1459 -91 1007 375 \n",
       "Q 556 841 556 1753 \n",
       "L 556 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-4d\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-65\" x=\"86.279297\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-67\" x=\"147.802734\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-61\" x=\"211.279297\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-74\" x=\"272.558594\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-72\" x=\"311.767578\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-6f\" x=\"350.630859\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-6e\" x=\"411.8125\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-20\" x=\"475.191406\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-47\" x=\"506.978516\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-50\" x=\"584.46875\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-54\" x=\"644.771484\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-32\" x=\"705.855469\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-2c\" x=\"769.478516\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-20\" x=\"801.265625\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-32\" x=\"833.052734\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-20\" x=\"896.675781\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-47\" x=\"928.462891\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-50\" x=\"1005.953125\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-55\" x=\"1066.255859\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-73\" x=\"1139.449219\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-2c\" x=\"1191.548828\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-20\" x=\"1223.335938\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-54\" x=\"1255.123047\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-50\" x=\"1316.207031\"/>\n",
       "      </g>\n",
       "      <!-- grad_accum=4 -->\n",
       "      <g transform=\"translate(111.607812 29.094787)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-64\" d=\"M 2906 2969 \n",
       "L 2906 4863 \n",
       "L 3481 4863 \n",
       "L 3481 0 \n",
       "L 2906 0 \n",
       "L 2906 525 \n",
       "Q 2725 213 2448 61 \n",
       "Q 2172 -91 1784 -91 \n",
       "Q 1150 -91 751 415 \n",
       "Q 353 922 353 1747 \n",
       "Q 353 2572 751 3078 \n",
       "Q 1150 3584 1784 3584 \n",
       "Q 2172 3584 2448 3432 \n",
       "Q 2725 3281 2906 2969 \n",
       "z\n",
       "M 947 1747 \n",
       "Q 947 1113 1208 752 \n",
       "Q 1469 391 1925 391 \n",
       "Q 2381 391 2643 752 \n",
       "Q 2906 1113 2906 1747 \n",
       "Q 2906 2381 2643 2742 \n",
       "Q 2381 3103 1925 3103 \n",
       "Q 1469 3103 1208 2742 \n",
       "Q 947 2381 947 1747 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-5f\" d=\"M 3263 -1063 \n",
       "L 3263 -1509 \n",
       "L -63 -1509 \n",
       "L -63 -1063 \n",
       "L 3263 -1063 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-75\" d=\"M 544 1381 \n",
       "L 544 3500 \n",
       "L 1119 3500 \n",
       "L 1119 1403 \n",
       "Q 1119 906 1312 657 \n",
       "Q 1506 409 1894 409 \n",
       "Q 2359 409 2629 706 \n",
       "Q 2900 1003 2900 1516 \n",
       "L 2900 3500 \n",
       "L 3475 3500 \n",
       "L 3475 0 \n",
       "L 2900 0 \n",
       "L 2900 538 \n",
       "Q 2691 219 2414 64 \n",
       "Q 2138 -91 1772 -91 \n",
       "Q 1169 -91 856 284 \n",
       "Q 544 659 544 1381 \n",
       "z\n",
       "M 1991 3584 \n",
       "L 1991 3584 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-3d\" d=\"M 678 2906 \n",
       "L 4684 2906 \n",
       "L 4684 2381 \n",
       "L 678 2381 \n",
       "L 678 2906 \n",
       "z\n",
       "M 678 1631 \n",
       "L 4684 1631 \n",
       "L 4684 1100 \n",
       "L 678 1100 \n",
       "L 678 1631 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-34\" d=\"M 2419 4116 \n",
       "L 825 1625 \n",
       "L 2419 1625 \n",
       "L 2419 4116 \n",
       "z\n",
       "M 2253 4666 \n",
       "L 3047 4666 \n",
       "L 3047 1625 \n",
       "L 3713 1625 \n",
       "L 3713 1100 \n",
       "L 3047 1100 \n",
       "L 3047 0 \n",
       "L 2419 0 \n",
       "L 2419 1100 \n",
       "L 313 1100 \n",
       "L 313 1709 \n",
       "L 2253 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-67\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-72\" x=\"63.476562\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-61\" x=\"104.589844\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-64\" x=\"165.869141\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-5f\" x=\"229.345703\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-61\" x=\"279.345703\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-63\" x=\"340.625\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-63\" x=\"395.605469\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-75\" x=\"450.585938\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-6d\" x=\"513.964844\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-3d\" x=\"611.376953\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-34\" x=\"695.166016\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_2\">\n",
       "     <g id=\"line2d_6\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#mc796056d31\" x=\"194.489062\" y=\"49.823182\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_7\">\n",
       "      <!-- Megatron GPT-Neo-1.3B, 2 GPUs, TP -->\n",
       "      <g transform=\"translate(7.2 48.783338)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-2d\" d=\"M 313 2009 \n",
       "L 1997 2009 \n",
       "L 1997 1497 \n",
       "L 313 1497 \n",
       "L 313 2009 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-4e\" d=\"M 628 4666 \n",
       "L 1478 4666 \n",
       "L 3547 763 \n",
       "L 3547 4666 \n",
       "L 4159 4666 \n",
       "L 4159 0 \n",
       "L 3309 0 \n",
       "L 1241 3903 \n",
       "L 1241 0 \n",
       "L 628 0 \n",
       "L 628 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-2e\" d=\"M 684 794 \n",
       "L 1344 794 \n",
       "L 1344 0 \n",
       "L 684 0 \n",
       "L 684 794 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-42\" d=\"M 1259 2228 \n",
       "L 1259 519 \n",
       "L 2272 519 \n",
       "Q 2781 519 3026 730 \n",
       "Q 3272 941 3272 1375 \n",
       "Q 3272 1813 3026 2020 \n",
       "Q 2781 2228 2272 2228 \n",
       "L 1259 2228 \n",
       "z\n",
       "M 1259 4147 \n",
       "L 1259 2741 \n",
       "L 2194 2741 \n",
       "Q 2656 2741 2882 2914 \n",
       "Q 3109 3088 3109 3444 \n",
       "Q 3109 3797 2882 3972 \n",
       "Q 2656 4147 2194 4147 \n",
       "L 1259 4147 \n",
       "z\n",
       "M 628 4666 \n",
       "L 2241 4666 \n",
       "Q 2963 4666 3353 4366 \n",
       "Q 3744 4066 3744 3513 \n",
       "Q 3744 3084 3544 2831 \n",
       "Q 3344 2578 2956 2516 \n",
       "Q 3422 2416 3680 2098 \n",
       "Q 3938 1781 3938 1306 \n",
       "Q 3938 681 3513 340 \n",
       "Q 3088 0 2303 0 \n",
       "L 628 0 \n",
       "L 628 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-4d\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-65\" x=\"86.279297\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-67\" x=\"147.802734\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-61\" x=\"211.279297\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-74\" x=\"272.558594\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-72\" x=\"311.767578\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-6f\" x=\"350.630859\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-6e\" x=\"411.8125\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-20\" x=\"475.191406\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-47\" x=\"506.978516\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-50\" x=\"584.46875\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-54\" x=\"644.771484\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-2d\" x=\"696.730469\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-4e\" x=\"732.814453\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-65\" x=\"807.619141\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-6f\" x=\"869.142578\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-2d\" x=\"932.199219\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-31\" x=\"968.283203\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-2e\" x=\"1031.90625\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-33\" x=\"1063.693359\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-42\" x=\"1127.316406\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-2c\" x=\"1195.919922\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-20\" x=\"1227.707031\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-32\" x=\"1259.494141\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-20\" x=\"1323.117188\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-47\" x=\"1354.904297\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-50\" x=\"1432.394531\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-55\" x=\"1492.697266\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-73\" x=\"1565.890625\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-2c\" x=\"1617.990234\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-20\" x=\"1649.777344\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-54\" x=\"1681.564453\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-50\" x=\"1742.648438\"/>\n",
       "      </g>\n",
       "      <!-- grad_accum=4 -->\n",
       "      <g transform=\"translate(111.607812 59.981151)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-67\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-72\" x=\"63.476562\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-61\" x=\"104.589844\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-64\" x=\"165.869141\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-5f\" x=\"229.345703\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-61\" x=\"279.345703\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-63\" x=\"340.625\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-63\" x=\"395.605469\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-75\" x=\"450.585938\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-6d\" x=\"513.964844\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-3d\" x=\"611.376953\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-34\" x=\"695.166016\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"patch_5\">\n",
       "    <path d=\"M 194.489062 61.56 \n",
       "L 194.489062 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_6\">\n",
       "    <path d=\"M 342.194945 61.56 \n",
       "L 342.194945 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_7\">\n",
       "    <path d=\"M 194.489062 61.56 \n",
       "L 342.194945 61.56 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_8\">\n",
       "    <path d=\"M 194.489062 7.2 \n",
       "L 342.194945 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "  </g>\n",
       "  <g id=\"axes_2\">\n",
       "   <g id=\"patch_9\">\n",
       "    <path d=\"M 371.736121 61.56 \n",
       "L 519.442004 61.56 \n",
       "L 519.442004 7.2 \n",
       "L 371.736121 7.2 \n",
       "z\n",
       "\" style=\"fill: #ffffff\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_10\">\n",
       "    <path d=\"M 371.736121 9.670909 \n",
       "L 489.730468 9.670909 \n",
       "L 489.730468 28.202727 \n",
       "L 371.736121 28.202727 \n",
       "z\n",
       "\" clip-path=\"url(#p2aa92c9c61)\" style=\"fill: #e41a1c\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_11\">\n",
       "    <path d=\"M 371.736121 40.557273 \n",
       "L 512.40839 40.557273 \n",
       "L 512.40839 59.089091 \n",
       "L 371.736121 59.089091 \n",
       "z\n",
       "\" clip-path=\"url(#p2aa92c9c61)\" style=\"fill: #377eb8\"/>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_3\">\n",
       "    <g id=\"xtick_5\">\n",
       "     <g id=\"line2d_7\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m90391b8f49\" x=\"371.736121\" y=\"61.56\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_8\">\n",
       "      <!-- 0 -->\n",
       "      <g transform=\"translate(368.554871 76.158437)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_6\">\n",
       "     <g id=\"line2d_8\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m90391b8f49\" x=\"433.752216\" y=\"61.56\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_9\">\n",
       "      <!-- 20 -->\n",
       "      <g transform=\"translate(427.389716 76.158437)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-32\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_7\">\n",
       "     <g id=\"line2d_9\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m90391b8f49\" x=\"495.768311\" y=\"61.56\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_10\">\n",
       "      <!-- 40 -->\n",
       "      <g transform=\"translate(489.405811 76.158437)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-34\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"text_11\">\n",
       "     <!-- per GPU TFLOPS -->\n",
       "     <g transform=\"translate(404.896875 89.836562)scale(0.1 -0.1)\">\n",
       "      <defs>\n",
       "       <path id=\"DejaVuSans-46\" d=\"M 628 4666 \n",
       "L 3309 4666 \n",
       "L 3309 4134 \n",
       "L 1259 4134 \n",
       "L 1259 2759 \n",
       "L 3109 2759 \n",
       "L 3109 2228 \n",
       "L 1259 2228 \n",
       "L 1259 0 \n",
       "L 628 0 \n",
       "L 628 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-4c\" d=\"M 628 4666 \n",
       "L 1259 4666 \n",
       "L 1259 531 \n",
       "L 3531 531 \n",
       "L 3531 0 \n",
       "L 628 0 \n",
       "L 628 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-4f\" d=\"M 2522 4238 \n",
       "Q 1834 4238 1429 3725 \n",
       "Q 1025 3213 1025 2328 \n",
       "Q 1025 1447 1429 934 \n",
       "Q 1834 422 2522 422 \n",
       "Q 3209 422 3611 934 \n",
       "Q 4013 1447 4013 2328 \n",
       "Q 4013 3213 3611 3725 \n",
       "Q 3209 4238 2522 4238 \n",
       "z\n",
       "M 2522 4750 \n",
       "Q 3503 4750 4090 4092 \n",
       "Q 4678 3434 4678 2328 \n",
       "Q 4678 1225 4090 567 \n",
       "Q 3503 -91 2522 -91 \n",
       "Q 1538 -91 948 565 \n",
       "Q 359 1222 359 2328 \n",
       "Q 359 3434 948 4092 \n",
       "Q 1538 4750 2522 4750 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "      </defs>\n",
       "      <use xlink:href=\"#DejaVuSans-70\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-65\" x=\"63.476562\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-72\" x=\"125\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-20\" x=\"166.113281\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-47\" x=\"197.900391\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-50\" x=\"275.390625\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-55\" x=\"335.693359\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-20\" x=\"408.886719\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-54\" x=\"440.673828\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-46\" x=\"501.757812\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-4c\" x=\"559.277344\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-4f\" x=\"611.365234\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-50\" x=\"690.076172\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-53\" x=\"750.378906\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_4\">\n",
       "    <g id=\"ytick_3\">\n",
       "     <g id=\"line2d_10\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#mc796056d31\" x=\"371.736121\" y=\"18.936818\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_4\">\n",
       "     <g id=\"line2d_11\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#mc796056d31\" x=\"371.736121\" y=\"49.823182\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"patch_12\">\n",
       "    <path d=\"M 371.736121 61.56 \n",
       "L 371.736121 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_13\">\n",
       "    <path d=\"M 519.442004 61.56 \n",
       "L 519.442004 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_14\">\n",
       "    <path d=\"M 371.736121 61.56 \n",
       "L 519.442004 61.56 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_15\">\n",
       "    <path d=\"M 371.736121 7.2 \n",
       "L 519.442004 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "  </g>\n",
       "  <g id=\"axes_3\">\n",
       "   <g id=\"patch_16\">\n",
       "    <path d=\"M 548.98318 61.56 \n",
       "L 696.689063 61.56 \n",
       "L 696.689063 7.2 \n",
       "L 548.98318 7.2 \n",
       "z\n",
       "\" style=\"fill: #ffffff\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_17\">\n",
       "    <path d=\"M 548.98318 9.670909 \n",
       "L 689.655449 9.670909 \n",
       "L 689.655449 28.202727 \n",
       "L 548.98318 28.202727 \n",
       "z\n",
       "\" clip-path=\"url(#p524b4613fe)\" style=\"fill: #e41a1c\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_18\">\n",
       "    <path d=\"M 548.98318 40.557273 \n",
       "L 675.602181 40.557273 \n",
       "L 675.602181 59.089091 \n",
       "L 548.98318 59.089091 \n",
       "z\n",
       "\" clip-path=\"url(#p524b4613fe)\" style=\"fill: #377eb8\"/>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_5\">\n",
       "    <g id=\"xtick_8\">\n",
       "     <g id=\"line2d_12\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m90391b8f49\" x=\"548.98318\" y=\"61.56\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_12\">\n",
       "      <!-- 0 -->\n",
       "      <g transform=\"translate(545.80193 76.158437)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_9\">\n",
       "     <g id=\"line2d_13\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m90391b8f49\" x=\"620.620581\" y=\"61.56\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_13\">\n",
       "      <!-- 10 -->\n",
       "      <g transform=\"translate(614.258081 76.158437)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-31\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_10\">\n",
       "     <g id=\"line2d_14\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m90391b8f49\" x=\"692.257982\" y=\"61.56\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_14\">\n",
       "      <!-- 20 -->\n",
       "      <g transform=\"translate(685.895482 76.158437)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-32\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"text_15\">\n",
       "     <!-- per GPU memory (GB) -->\n",
       "     <g transform=\"translate(567.11659 89.836562)scale(0.1 -0.1)\">\n",
       "      <defs>\n",
       "       <path id=\"DejaVuSans-79\" d=\"M 2059 -325 \n",
       "Q 1816 -950 1584 -1140 \n",
       "Q 1353 -1331 966 -1331 \n",
       "L 506 -1331 \n",
       "L 506 -850 \n",
       "L 844 -850 \n",
       "Q 1081 -850 1212 -737 \n",
       "Q 1344 -625 1503 -206 \n",
       "L 1606 56 \n",
       "L 191 3500 \n",
       "L 800 3500 \n",
       "L 1894 763 \n",
       "L 2988 3500 \n",
       "L 3597 3500 \n",
       "L 2059 -325 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-28\" d=\"M 1984 4856 \n",
       "Q 1566 4138 1362 3434 \n",
       "Q 1159 2731 1159 2009 \n",
       "Q 1159 1288 1364 580 \n",
       "Q 1569 -128 1984 -844 \n",
       "L 1484 -844 \n",
       "Q 1016 -109 783 600 \n",
       "Q 550 1309 550 2009 \n",
       "Q 550 2706 781 3412 \n",
       "Q 1013 4119 1484 4856 \n",
       "L 1984 4856 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-29\" d=\"M 513 4856 \n",
       "L 1013 4856 \n",
       "Q 1481 4119 1714 3412 \n",
       "Q 1947 2706 1947 2009 \n",
       "Q 1947 1309 1714 600 \n",
       "Q 1481 -109 1013 -844 \n",
       "L 513 -844 \n",
       "Q 928 -128 1133 580 \n",
       "Q 1338 1288 1338 2009 \n",
       "Q 1338 2731 1133 3434 \n",
       "Q 928 4138 513 4856 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "      </defs>\n",
       "      <use xlink:href=\"#DejaVuSans-70\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-65\" x=\"63.476562\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-72\" x=\"125\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-20\" x=\"166.113281\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-47\" x=\"197.900391\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-50\" x=\"275.390625\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-55\" x=\"335.693359\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-20\" x=\"408.886719\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-6d\" x=\"440.673828\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-65\" x=\"538.085938\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-6d\" x=\"599.609375\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-6f\" x=\"697.021484\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-72\" x=\"758.203125\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-79\" x=\"799.316406\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-20\" x=\"858.496094\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-28\" x=\"890.283203\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-47\" x=\"929.296875\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-42\" x=\"1006.787109\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-29\" x=\"1075.390625\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_6\">\n",
       "    <g id=\"ytick_5\">\n",
       "     <g id=\"line2d_15\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#mc796056d31\" x=\"548.98318\" y=\"18.936818\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_6\">\n",
       "     <g id=\"line2d_16\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#mc796056d31\" x=\"548.98318\" y=\"49.823182\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"patch_19\">\n",
       "    <path d=\"M 548.98318 61.56 \n",
       "L 548.98318 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_20\">\n",
       "    <path d=\"M 696.689063 61.56 \n",
       "L 696.689063 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_21\">\n",
       "    <path d=\"M 548.98318 61.56 \n",
       "L 696.689063 61.56 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_22\">\n",
       "    <path d=\"M 548.98318 7.2 \n",
       "L 696.689063 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "  </g>\n",
       " </g>\n",
       " <defs>\n",
       "  <clipPath id=\"p0faeee0155\">\n",
       "   <rect x=\"194.489062\" y=\"7.2\" width=\"147.705882\" height=\"54.36\"/>\n",
       "  </clipPath>\n",
       "  <clipPath id=\"p2aa92c9c61\">\n",
       "   <rect x=\"371.736121\" y=\"7.2\" width=\"147.705882\" height=\"54.36\"/>\n",
       "  </clipPath>\n",
       "  <clipPath id=\"p524b4613fe\">\n",
       "   <rect x=\"548.98318\" y=\"7.2\" width=\"147.705882\" height=\"54.36\"/>\n",
       "  </clipPath>\n",
       " </defs>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<Figure size 648x72 with 3 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "tp_gpt_neo_accum = megatron_gpt(Exp(\n",
    "    \"Megatron GPT-Neo-1.3B, 2 GPUs, TP\\ngrad_accum=4\", \"EleutherAI/gpt-neo-1.3B\", 1, bf16=True, gpus='0,1',\n",
    "    tensor_para=2, grad_accum=4, steps=10))\n",
    "compare([tp_gpt2_accum, tp_gpt_neo_accum])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "60098bea",
   "metadata": {},
   "source": [
    "## Multiple GPU with ZeRO\n",
    "\n",
    "Similar to TP, ZeRO also enables run very large model. Here we try Zero-2."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "5ab4553b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       "  \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<svg xmlns:xlink=\"http://www.w3.org/1999/xlink\" width=\"703.889062pt\" height=\"99.11625pt\" viewBox=\"0 0 703.889062 99.11625\" xmlns=\"http://www.w3.org/2000/svg\" version=\"1.1\">\n",
       " <metadata>\n",
       "  <rdf:RDF xmlns:dc=\"http://purl.org/dc/elements/1.1/\" xmlns:cc=\"http://creativecommons.org/ns#\" xmlns:rdf=\"http://www.w3.org/1999/02/22-rdf-syntax-ns#\">\n",
       "   <cc:Work>\n",
       "    <dc:type rdf:resource=\"http://purl.org/dc/dcmitype/StillImage\"/>\n",
       "    <dc:date>2022-09-02T19:55:44.247460</dc:date>\n",
       "    <dc:format>image/svg+xml</dc:format>\n",
       "    <dc:creator>\n",
       "     <cc:Agent>\n",
       "      <dc:title>Matplotlib v3.5.2, https://matplotlib.org/</dc:title>\n",
       "     </cc:Agent>\n",
       "    </dc:creator>\n",
       "   </cc:Work>\n",
       "  </rdf:RDF>\n",
       " </metadata>\n",
       " <defs>\n",
       "  <style type=\"text/css\">*{stroke-linejoin: round; stroke-linecap: butt}</style>\n",
       " </defs>\n",
       " <g id=\"figure_1\">\n",
       "  <g id=\"patch_1\">\n",
       "   <path d=\"M 0 99.11625 \n",
       "L 703.889062 99.11625 \n",
       "L 703.889062 0 \n",
       "L 0 0 \n",
       "L 0 99.11625 \n",
       "z\n",
       "\" style=\"fill: none\"/>\n",
       "  </g>\n",
       "  <g id=\"axes_1\">\n",
       "   <g id=\"patch_2\">\n",
       "    <path d=\"M 194.489062 61.56 \n",
       "L 342.194945 61.56 \n",
       "L 342.194945 7.2 \n",
       "L 194.489062 7.2 \n",
       "z\n",
       "\" style=\"fill: #ffffff\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_3\">\n",
       "    <path d=\"M 194.489062 9.670909 \n",
       "L 335.161331 9.670909 \n",
       "L 335.161331 28.202727 \n",
       "L 194.489062 28.202727 \n",
       "z\n",
       "\" clip-path=\"url(#p047fcebe24)\" style=\"fill: #e41a1c\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_4\">\n",
       "    <path d=\"M 194.489062 40.557273 \n",
       "L 257.783931 40.557273 \n",
       "L 257.783931 59.089091 \n",
       "L 194.489062 59.089091 \n",
       "z\n",
       "\" clip-path=\"url(#p047fcebe24)\" style=\"fill: #377eb8\"/>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_1\">\n",
       "    <g id=\"xtick_1\">\n",
       "     <g id=\"line2d_1\">\n",
       "      <defs>\n",
       "       <path id=\"ma1fcd821d2\" d=\"M 0 0 \n",
       "L 0 3.5 \n",
       "\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </defs>\n",
       "      <g>\n",
       "       <use xlink:href=\"#ma1fcd821d2\" x=\"194.489062\" y=\"61.56\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_1\">\n",
       "      <!-- 0 -->\n",
       "      <g transform=\"translate(191.307812 76.158437)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-30\" d=\"M 2034 4250 \n",
       "Q 1547 4250 1301 3770 \n",
       "Q 1056 3291 1056 2328 \n",
       "Q 1056 1369 1301 889 \n",
       "Q 1547 409 2034 409 \n",
       "Q 2525 409 2770 889 \n",
       "Q 3016 1369 3016 2328 \n",
       "Q 3016 3291 2770 3770 \n",
       "Q 2525 4250 2034 4250 \n",
       "z\n",
       "M 2034 4750 \n",
       "Q 2819 4750 3233 4129 \n",
       "Q 3647 3509 3647 2328 \n",
       "Q 3647 1150 3233 529 \n",
       "Q 2819 -91 2034 -91 \n",
       "Q 1250 -91 836 529 \n",
       "Q 422 1150 422 2328 \n",
       "Q 422 3509 836 4129 \n",
       "Q 1250 4750 2034 4750 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_2\">\n",
       "     <g id=\"line2d_2\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#ma1fcd821d2\" x=\"252.108424\" y=\"61.56\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_2\">\n",
       "      <!-- 2 -->\n",
       "      <g transform=\"translate(248.927174 76.158437)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-32\" d=\"M 1228 531 \n",
       "L 3431 531 \n",
       "L 3431 0 \n",
       "L 469 0 \n",
       "L 469 531 \n",
       "Q 828 903 1448 1529 \n",
       "Q 2069 2156 2228 2338 \n",
       "Q 2531 2678 2651 2914 \n",
       "Q 2772 3150 2772 3378 \n",
       "Q 2772 3750 2511 3984 \n",
       "Q 2250 4219 1831 4219 \n",
       "Q 1534 4219 1204 4116 \n",
       "Q 875 4013 500 3803 \n",
       "L 500 4441 \n",
       "Q 881 4594 1212 4672 \n",
       "Q 1544 4750 1819 4750 \n",
       "Q 2544 4750 2975 4387 \n",
       "Q 3406 4025 3406 3419 \n",
       "Q 3406 3131 3298 2873 \n",
       "Q 3191 2616 2906 2266 \n",
       "Q 2828 2175 2409 1742 \n",
       "Q 1991 1309 1228 531 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-32\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_3\">\n",
       "     <g id=\"line2d_3\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#ma1fcd821d2\" x=\"309.727785\" y=\"61.56\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_3\">\n",
       "      <!-- 4 -->\n",
       "      <g transform=\"translate(306.546535 76.158437)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-34\" d=\"M 2419 4116 \n",
       "L 825 1625 \n",
       "L 2419 1625 \n",
       "L 2419 4116 \n",
       "z\n",
       "M 2253 4666 \n",
       "L 3047 4666 \n",
       "L 3047 1625 \n",
       "L 3713 1625 \n",
       "L 3713 1100 \n",
       "L 3047 1100 \n",
       "L 3047 0 \n",
       "L 2419 0 \n",
       "L 2419 1100 \n",
       "L 313 1100 \n",
       "L 313 1709 \n",
       "L 2253 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-34\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"text_4\">\n",
       "     <!-- Samples / sec -->\n",
       "     <g transform=\"translate(233.69591 89.836562)scale(0.1 -0.1)\">\n",
       "      <defs>\n",
       "       <path id=\"DejaVuSans-53\" d=\"M 3425 4513 \n",
       "L 3425 3897 \n",
       "Q 3066 4069 2747 4153 \n",
       "Q 2428 4238 2131 4238 \n",
       "Q 1616 4238 1336 4038 \n",
       "Q 1056 3838 1056 3469 \n",
       "Q 1056 3159 1242 3001 \n",
       "Q 1428 2844 1947 2747 \n",
       "L 2328 2669 \n",
       "Q 3034 2534 3370 2195 \n",
       "Q 3706 1856 3706 1288 \n",
       "Q 3706 609 3251 259 \n",
       "Q 2797 -91 1919 -91 \n",
       "Q 1588 -91 1214 -16 \n",
       "Q 841 59 441 206 \n",
       "L 441 856 \n",
       "Q 825 641 1194 531 \n",
       "Q 1563 422 1919 422 \n",
       "Q 2459 422 2753 634 \n",
       "Q 3047 847 3047 1241 \n",
       "Q 3047 1584 2836 1778 \n",
       "Q 2625 1972 2144 2069 \n",
       "L 1759 2144 \n",
       "Q 1053 2284 737 2584 \n",
       "Q 422 2884 422 3419 \n",
       "Q 422 4038 858 4394 \n",
       "Q 1294 4750 2059 4750 \n",
       "Q 2388 4750 2728 4690 \n",
       "Q 3069 4631 3425 4513 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-61\" d=\"M 2194 1759 \n",
       "Q 1497 1759 1228 1600 \n",
       "Q 959 1441 959 1056 \n",
       "Q 959 750 1161 570 \n",
       "Q 1363 391 1709 391 \n",
       "Q 2188 391 2477 730 \n",
       "Q 2766 1069 2766 1631 \n",
       "L 2766 1759 \n",
       "L 2194 1759 \n",
       "z\n",
       "M 3341 1997 \n",
       "L 3341 0 \n",
       "L 2766 0 \n",
       "L 2766 531 \n",
       "Q 2569 213 2275 61 \n",
       "Q 1981 -91 1556 -91 \n",
       "Q 1019 -91 701 211 \n",
       "Q 384 513 384 1019 \n",
       "Q 384 1609 779 1909 \n",
       "Q 1175 2209 1959 2209 \n",
       "L 2766 2209 \n",
       "L 2766 2266 \n",
       "Q 2766 2663 2505 2880 \n",
       "Q 2244 3097 1772 3097 \n",
       "Q 1472 3097 1187 3025 \n",
       "Q 903 2953 641 2809 \n",
       "L 641 3341 \n",
       "Q 956 3463 1253 3523 \n",
       "Q 1550 3584 1831 3584 \n",
       "Q 2591 3584 2966 3190 \n",
       "Q 3341 2797 3341 1997 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-6d\" d=\"M 3328 2828 \n",
       "Q 3544 3216 3844 3400 \n",
       "Q 4144 3584 4550 3584 \n",
       "Q 5097 3584 5394 3201 \n",
       "Q 5691 2819 5691 2113 \n",
       "L 5691 0 \n",
       "L 5113 0 \n",
       "L 5113 2094 \n",
       "Q 5113 2597 4934 2840 \n",
       "Q 4756 3084 4391 3084 \n",
       "Q 3944 3084 3684 2787 \n",
       "Q 3425 2491 3425 1978 \n",
       "L 3425 0 \n",
       "L 2847 0 \n",
       "L 2847 2094 \n",
       "Q 2847 2600 2669 2842 \n",
       "Q 2491 3084 2119 3084 \n",
       "Q 1678 3084 1418 2786 \n",
       "Q 1159 2488 1159 1978 \n",
       "L 1159 0 \n",
       "L 581 0 \n",
       "L 581 3500 \n",
       "L 1159 3500 \n",
       "L 1159 2956 \n",
       "Q 1356 3278 1631 3431 \n",
       "Q 1906 3584 2284 3584 \n",
       "Q 2666 3584 2933 3390 \n",
       "Q 3200 3197 3328 2828 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-70\" d=\"M 1159 525 \n",
       "L 1159 -1331 \n",
       "L 581 -1331 \n",
       "L 581 3500 \n",
       "L 1159 3500 \n",
       "L 1159 2969 \n",
       "Q 1341 3281 1617 3432 \n",
       "Q 1894 3584 2278 3584 \n",
       "Q 2916 3584 3314 3078 \n",
       "Q 3713 2572 3713 1747 \n",
       "Q 3713 922 3314 415 \n",
       "Q 2916 -91 2278 -91 \n",
       "Q 1894 -91 1617 61 \n",
       "Q 1341 213 1159 525 \n",
       "z\n",
       "M 3116 1747 \n",
       "Q 3116 2381 2855 2742 \n",
       "Q 2594 3103 2138 3103 \n",
       "Q 1681 3103 1420 2742 \n",
       "Q 1159 2381 1159 1747 \n",
       "Q 1159 1113 1420 752 \n",
       "Q 1681 391 2138 391 \n",
       "Q 2594 391 2855 752 \n",
       "Q 3116 1113 3116 1747 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-6c\" d=\"M 603 4863 \n",
       "L 1178 4863 \n",
       "L 1178 0 \n",
       "L 603 0 \n",
       "L 603 4863 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-65\" d=\"M 3597 1894 \n",
       "L 3597 1613 \n",
       "L 953 1613 \n",
       "Q 991 1019 1311 708 \n",
       "Q 1631 397 2203 397 \n",
       "Q 2534 397 2845 478 \n",
       "Q 3156 559 3463 722 \n",
       "L 3463 178 \n",
       "Q 3153 47 2828 -22 \n",
       "Q 2503 -91 2169 -91 \n",
       "Q 1331 -91 842 396 \n",
       "Q 353 884 353 1716 \n",
       "Q 353 2575 817 3079 \n",
       "Q 1281 3584 2069 3584 \n",
       "Q 2775 3584 3186 3129 \n",
       "Q 3597 2675 3597 1894 \n",
       "z\n",
       "M 3022 2063 \n",
       "Q 3016 2534 2758 2815 \n",
       "Q 2500 3097 2075 3097 \n",
       "Q 1594 3097 1305 2825 \n",
       "Q 1016 2553 972 2059 \n",
       "L 3022 2063 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-73\" d=\"M 2834 3397 \n",
       "L 2834 2853 \n",
       "Q 2591 2978 2328 3040 \n",
       "Q 2066 3103 1784 3103 \n",
       "Q 1356 3103 1142 2972 \n",
       "Q 928 2841 928 2578 \n",
       "Q 928 2378 1081 2264 \n",
       "Q 1234 2150 1697 2047 \n",
       "L 1894 2003 \n",
       "Q 2506 1872 2764 1633 \n",
       "Q 3022 1394 3022 966 \n",
       "Q 3022 478 2636 193 \n",
       "Q 2250 -91 1575 -91 \n",
       "Q 1294 -91 989 -36 \n",
       "Q 684 19 347 128 \n",
       "L 347 722 \n",
       "Q 666 556 975 473 \n",
       "Q 1284 391 1588 391 \n",
       "Q 1994 391 2212 530 \n",
       "Q 2431 669 2431 922 \n",
       "Q 2431 1156 2273 1281 \n",
       "Q 2116 1406 1581 1522 \n",
       "L 1381 1569 \n",
       "Q 847 1681 609 1914 \n",
       "Q 372 2147 372 2553 \n",
       "Q 372 3047 722 3315 \n",
       "Q 1072 3584 1716 3584 \n",
       "Q 2034 3584 2315 3537 \n",
       "Q 2597 3491 2834 3397 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-20\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-2f\" d=\"M 1625 4666 \n",
       "L 2156 4666 \n",
       "L 531 -594 \n",
       "L 0 -594 \n",
       "L 1625 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-63\" d=\"M 3122 3366 \n",
       "L 3122 2828 \n",
       "Q 2878 2963 2633 3030 \n",
       "Q 2388 3097 2138 3097 \n",
       "Q 1578 3097 1268 2742 \n",
       "Q 959 2388 959 1747 \n",
       "Q 959 1106 1268 751 \n",
       "Q 1578 397 2138 397 \n",
       "Q 2388 397 2633 464 \n",
       "Q 2878 531 3122 666 \n",
       "L 3122 134 \n",
       "Q 2881 22 2623 -34 \n",
       "Q 2366 -91 2075 -91 \n",
       "Q 1284 -91 818 406 \n",
       "Q 353 903 353 1747 \n",
       "Q 353 2603 823 3093 \n",
       "Q 1294 3584 2113 3584 \n",
       "Q 2378 3584 2631 3529 \n",
       "Q 2884 3475 3122 3366 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "      </defs>\n",
       "      <use xlink:href=\"#DejaVuSans-53\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-61\" x=\"63.476562\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-6d\" x=\"124.755859\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-70\" x=\"222.167969\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-6c\" x=\"285.644531\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-65\" x=\"313.427734\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-73\" x=\"374.951172\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-20\" x=\"427.050781\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-2f\" x=\"458.837891\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-20\" x=\"492.529297\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-73\" x=\"524.316406\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-65\" x=\"576.416016\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-63\" x=\"637.939453\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_2\">\n",
       "    <g id=\"ytick_1\">\n",
       "     <g id=\"line2d_4\">\n",
       "      <defs>\n",
       "       <path id=\"m3482c907f8\" d=\"M 0 0 \n",
       "L -3.5 0 \n",
       "\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </defs>\n",
       "      <g>\n",
       "       <use xlink:href=\"#m3482c907f8\" x=\"194.489062\" y=\"18.936818\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_5\">\n",
       "      <!-- Megatron GPT-Neo-1.3B, 2 GPUs, TP -->\n",
       "      <g transform=\"translate(7.2 17.896974)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-4d\" d=\"M 628 4666 \n",
       "L 1569 4666 \n",
       "L 2759 1491 \n",
       "L 3956 4666 \n",
       "L 4897 4666 \n",
       "L 4897 0 \n",
       "L 4281 0 \n",
       "L 4281 4097 \n",
       "L 3078 897 \n",
       "L 2444 897 \n",
       "L 1241 4097 \n",
       "L 1241 0 \n",
       "L 628 0 \n",
       "L 628 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-67\" d=\"M 2906 1791 \n",
       "Q 2906 2416 2648 2759 \n",
       "Q 2391 3103 1925 3103 \n",
       "Q 1463 3103 1205 2759 \n",
       "Q 947 2416 947 1791 \n",
       "Q 947 1169 1205 825 \n",
       "Q 1463 481 1925 481 \n",
       "Q 2391 481 2648 825 \n",
       "Q 2906 1169 2906 1791 \n",
       "z\n",
       "M 3481 434 \n",
       "Q 3481 -459 3084 -895 \n",
       "Q 2688 -1331 1869 -1331 \n",
       "Q 1566 -1331 1297 -1286 \n",
       "Q 1028 -1241 775 -1147 \n",
       "L 775 -588 \n",
       "Q 1028 -725 1275 -790 \n",
       "Q 1522 -856 1778 -856 \n",
       "Q 2344 -856 2625 -561 \n",
       "Q 2906 -266 2906 331 \n",
       "L 2906 616 \n",
       "Q 2728 306 2450 153 \n",
       "Q 2172 0 1784 0 \n",
       "Q 1141 0 747 490 \n",
       "Q 353 981 353 1791 \n",
       "Q 353 2603 747 3093 \n",
       "Q 1141 3584 1784 3584 \n",
       "Q 2172 3584 2450 3431 \n",
       "Q 2728 3278 2906 2969 \n",
       "L 2906 3500 \n",
       "L 3481 3500 \n",
       "L 3481 434 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-74\" d=\"M 1172 4494 \n",
       "L 1172 3500 \n",
       "L 2356 3500 \n",
       "L 2356 3053 \n",
       "L 1172 3053 \n",
       "L 1172 1153 \n",
       "Q 1172 725 1289 603 \n",
       "Q 1406 481 1766 481 \n",
       "L 2356 481 \n",
       "L 2356 0 \n",
       "L 1766 0 \n",
       "Q 1100 0 847 248 \n",
       "Q 594 497 594 1153 \n",
       "L 594 3053 \n",
       "L 172 3053 \n",
       "L 172 3500 \n",
       "L 594 3500 \n",
       "L 594 4494 \n",
       "L 1172 4494 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-72\" d=\"M 2631 2963 \n",
       "Q 2534 3019 2420 3045 \n",
       "Q 2306 3072 2169 3072 \n",
       "Q 1681 3072 1420 2755 \n",
       "Q 1159 2438 1159 1844 \n",
       "L 1159 0 \n",
       "L 581 0 \n",
       "L 581 3500 \n",
       "L 1159 3500 \n",
       "L 1159 2956 \n",
       "Q 1341 3275 1631 3429 \n",
       "Q 1922 3584 2338 3584 \n",
       "Q 2397 3584 2469 3576 \n",
       "Q 2541 3569 2628 3553 \n",
       "L 2631 2963 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-6f\" d=\"M 1959 3097 \n",
       "Q 1497 3097 1228 2736 \n",
       "Q 959 2375 959 1747 \n",
       "Q 959 1119 1226 758 \n",
       "Q 1494 397 1959 397 \n",
       "Q 2419 397 2687 759 \n",
       "Q 2956 1122 2956 1747 \n",
       "Q 2956 2369 2687 2733 \n",
       "Q 2419 3097 1959 3097 \n",
       "z\n",
       "M 1959 3584 \n",
       "Q 2709 3584 3137 3096 \n",
       "Q 3566 2609 3566 1747 \n",
       "Q 3566 888 3137 398 \n",
       "Q 2709 -91 1959 -91 \n",
       "Q 1206 -91 779 398 \n",
       "Q 353 888 353 1747 \n",
       "Q 353 2609 779 3096 \n",
       "Q 1206 3584 1959 3584 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-6e\" d=\"M 3513 2113 \n",
       "L 3513 0 \n",
       "L 2938 0 \n",
       "L 2938 2094 \n",
       "Q 2938 2591 2744 2837 \n",
       "Q 2550 3084 2163 3084 \n",
       "Q 1697 3084 1428 2787 \n",
       "Q 1159 2491 1159 1978 \n",
       "L 1159 0 \n",
       "L 581 0 \n",
       "L 581 3500 \n",
       "L 1159 3500 \n",
       "L 1159 2956 \n",
       "Q 1366 3272 1645 3428 \n",
       "Q 1925 3584 2291 3584 \n",
       "Q 2894 3584 3203 3211 \n",
       "Q 3513 2838 3513 2113 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-47\" d=\"M 3809 666 \n",
       "L 3809 1919 \n",
       "L 2778 1919 \n",
       "L 2778 2438 \n",
       "L 4434 2438 \n",
       "L 4434 434 \n",
       "Q 4069 175 3628 42 \n",
       "Q 3188 -91 2688 -91 \n",
       "Q 1594 -91 976 548 \n",
       "Q 359 1188 359 2328 \n",
       "Q 359 3472 976 4111 \n",
       "Q 1594 4750 2688 4750 \n",
       "Q 3144 4750 3555 4637 \n",
       "Q 3966 4525 4313 4306 \n",
       "L 4313 3634 \n",
       "Q 3963 3931 3569 4081 \n",
       "Q 3175 4231 2741 4231 \n",
       "Q 1884 4231 1454 3753 \n",
       "Q 1025 3275 1025 2328 \n",
       "Q 1025 1384 1454 906 \n",
       "Q 1884 428 2741 428 \n",
       "Q 3075 428 3337 486 \n",
       "Q 3600 544 3809 666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-50\" d=\"M 1259 4147 \n",
       "L 1259 2394 \n",
       "L 2053 2394 \n",
       "Q 2494 2394 2734 2622 \n",
       "Q 2975 2850 2975 3272 \n",
       "Q 2975 3691 2734 3919 \n",
       "Q 2494 4147 2053 4147 \n",
       "L 1259 4147 \n",
       "z\n",
       "M 628 4666 \n",
       "L 2053 4666 \n",
       "Q 2838 4666 3239 4311 \n",
       "Q 3641 3956 3641 3272 \n",
       "Q 3641 2581 3239 2228 \n",
       "Q 2838 1875 2053 1875 \n",
       "L 1259 1875 \n",
       "L 1259 0 \n",
       "L 628 0 \n",
       "L 628 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-54\" d=\"M -19 4666 \n",
       "L 3928 4666 \n",
       "L 3928 4134 \n",
       "L 2272 4134 \n",
       "L 2272 0 \n",
       "L 1638 0 \n",
       "L 1638 4134 \n",
       "L -19 4134 \n",
       "L -19 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-2d\" d=\"M 313 2009 \n",
       "L 1997 2009 \n",
       "L 1997 1497 \n",
       "L 313 1497 \n",
       "L 313 2009 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-4e\" d=\"M 628 4666 \n",
       "L 1478 4666 \n",
       "L 3547 763 \n",
       "L 3547 4666 \n",
       "L 4159 4666 \n",
       "L 4159 0 \n",
       "L 3309 0 \n",
       "L 1241 3903 \n",
       "L 1241 0 \n",
       "L 628 0 \n",
       "L 628 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-31\" d=\"M 794 531 \n",
       "L 1825 531 \n",
       "L 1825 4091 \n",
       "L 703 3866 \n",
       "L 703 4441 \n",
       "L 1819 4666 \n",
       "L 2450 4666 \n",
       "L 2450 531 \n",
       "L 3481 531 \n",
       "L 3481 0 \n",
       "L 794 0 \n",
       "L 794 531 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-2e\" d=\"M 684 794 \n",
       "L 1344 794 \n",
       "L 1344 0 \n",
       "L 684 0 \n",
       "L 684 794 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-33\" d=\"M 2597 2516 \n",
       "Q 3050 2419 3304 2112 \n",
       "Q 3559 1806 3559 1356 \n",
       "Q 3559 666 3084 287 \n",
       "Q 2609 -91 1734 -91 \n",
       "Q 1441 -91 1130 -33 \n",
       "Q 819 25 488 141 \n",
       "L 488 750 \n",
       "Q 750 597 1062 519 \n",
       "Q 1375 441 1716 441 \n",
       "Q 2309 441 2620 675 \n",
       "Q 2931 909 2931 1356 \n",
       "Q 2931 1769 2642 2001 \n",
       "Q 2353 2234 1838 2234 \n",
       "L 1294 2234 \n",
       "L 1294 2753 \n",
       "L 1863 2753 \n",
       "Q 2328 2753 2575 2939 \n",
       "Q 2822 3125 2822 3475 \n",
       "Q 2822 3834 2567 4026 \n",
       "Q 2313 4219 1838 4219 \n",
       "Q 1578 4219 1281 4162 \n",
       "Q 984 4106 628 3988 \n",
       "L 628 4550 \n",
       "Q 988 4650 1302 4700 \n",
       "Q 1616 4750 1894 4750 \n",
       "Q 2613 4750 3031 4423 \n",
       "Q 3450 4097 3450 3541 \n",
       "Q 3450 3153 3228 2886 \n",
       "Q 3006 2619 2597 2516 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-42\" d=\"M 1259 2228 \n",
       "L 1259 519 \n",
       "L 2272 519 \n",
       "Q 2781 519 3026 730 \n",
       "Q 3272 941 3272 1375 \n",
       "Q 3272 1813 3026 2020 \n",
       "Q 2781 2228 2272 2228 \n",
       "L 1259 2228 \n",
       "z\n",
       "M 1259 4147 \n",
       "L 1259 2741 \n",
       "L 2194 2741 \n",
       "Q 2656 2741 2882 2914 \n",
       "Q 3109 3088 3109 3444 \n",
       "Q 3109 3797 2882 3972 \n",
       "Q 2656 4147 2194 4147 \n",
       "L 1259 4147 \n",
       "z\n",
       "M 628 4666 \n",
       "L 2241 4666 \n",
       "Q 2963 4666 3353 4366 \n",
       "Q 3744 4066 3744 3513 \n",
       "Q 3744 3084 3544 2831 \n",
       "Q 3344 2578 2956 2516 \n",
       "Q 3422 2416 3680 2098 \n",
       "Q 3938 1781 3938 1306 \n",
       "Q 3938 681 3513 340 \n",
       "Q 3088 0 2303 0 \n",
       "L 628 0 \n",
       "L 628 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-2c\" d=\"M 750 794 \n",
       "L 1409 794 \n",
       "L 1409 256 \n",
       "L 897 -744 \n",
       "L 494 -744 \n",
       "L 750 256 \n",
       "L 750 794 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-55\" d=\"M 556 4666 \n",
       "L 1191 4666 \n",
       "L 1191 1831 \n",
       "Q 1191 1081 1462 751 \n",
       "Q 1734 422 2344 422 \n",
       "Q 2950 422 3222 751 \n",
       "Q 3494 1081 3494 1831 \n",
       "L 3494 4666 \n",
       "L 4128 4666 \n",
       "L 4128 1753 \n",
       "Q 4128 841 3676 375 \n",
       "Q 3225 -91 2344 -91 \n",
       "Q 1459 -91 1007 375 \n",
       "Q 556 841 556 1753 \n",
       "L 556 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-4d\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-65\" x=\"86.279297\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-67\" x=\"147.802734\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-61\" x=\"211.279297\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-74\" x=\"272.558594\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-72\" x=\"311.767578\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-6f\" x=\"350.630859\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-6e\" x=\"411.8125\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-20\" x=\"475.191406\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-47\" x=\"506.978516\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-50\" x=\"584.46875\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-54\" x=\"644.771484\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-2d\" x=\"696.730469\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-4e\" x=\"732.814453\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-65\" x=\"807.619141\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-6f\" x=\"869.142578\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-2d\" x=\"932.199219\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-31\" x=\"968.283203\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-2e\" x=\"1031.90625\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-33\" x=\"1063.693359\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-42\" x=\"1127.316406\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-2c\" x=\"1195.919922\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-20\" x=\"1227.707031\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-32\" x=\"1259.494141\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-20\" x=\"1323.117188\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-47\" x=\"1354.904297\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-50\" x=\"1432.394531\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-55\" x=\"1492.697266\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-73\" x=\"1565.890625\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-2c\" x=\"1617.990234\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-20\" x=\"1649.777344\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-54\" x=\"1681.564453\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-50\" x=\"1742.648438\"/>\n",
       "      </g>\n",
       "      <!-- grad_accum=4 -->\n",
       "      <g transform=\"translate(111.607812 29.094787)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-64\" d=\"M 2906 2969 \n",
       "L 2906 4863 \n",
       "L 3481 4863 \n",
       "L 3481 0 \n",
       "L 2906 0 \n",
       "L 2906 525 \n",
       "Q 2725 213 2448 61 \n",
       "Q 2172 -91 1784 -91 \n",
       "Q 1150 -91 751 415 \n",
       "Q 353 922 353 1747 \n",
       "Q 353 2572 751 3078 \n",
       "Q 1150 3584 1784 3584 \n",
       "Q 2172 3584 2448 3432 \n",
       "Q 2725 3281 2906 2969 \n",
       "z\n",
       "M 947 1747 \n",
       "Q 947 1113 1208 752 \n",
       "Q 1469 391 1925 391 \n",
       "Q 2381 391 2643 752 \n",
       "Q 2906 1113 2906 1747 \n",
       "Q 2906 2381 2643 2742 \n",
       "Q 2381 3103 1925 3103 \n",
       "Q 1469 3103 1208 2742 \n",
       "Q 947 2381 947 1747 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-5f\" d=\"M 3263 -1063 \n",
       "L 3263 -1509 \n",
       "L -63 -1509 \n",
       "L -63 -1063 \n",
       "L 3263 -1063 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-75\" d=\"M 544 1381 \n",
       "L 544 3500 \n",
       "L 1119 3500 \n",
       "L 1119 1403 \n",
       "Q 1119 906 1312 657 \n",
       "Q 1506 409 1894 409 \n",
       "Q 2359 409 2629 706 \n",
       "Q 2900 1003 2900 1516 \n",
       "L 2900 3500 \n",
       "L 3475 3500 \n",
       "L 3475 0 \n",
       "L 2900 0 \n",
       "L 2900 538 \n",
       "Q 2691 219 2414 64 \n",
       "Q 2138 -91 1772 -91 \n",
       "Q 1169 -91 856 284 \n",
       "Q 544 659 544 1381 \n",
       "z\n",
       "M 1991 3584 \n",
       "L 1991 3584 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-3d\" d=\"M 678 2906 \n",
       "L 4684 2906 \n",
       "L 4684 2381 \n",
       "L 678 2381 \n",
       "L 678 2906 \n",
       "z\n",
       "M 678 1631 \n",
       "L 4684 1631 \n",
       "L 4684 1100 \n",
       "L 678 1100 \n",
       "L 678 1631 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-67\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-72\" x=\"63.476562\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-61\" x=\"104.589844\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-64\" x=\"165.869141\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-5f\" x=\"229.345703\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-61\" x=\"279.345703\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-63\" x=\"340.625\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-63\" x=\"395.605469\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-75\" x=\"450.585938\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-6d\" x=\"513.964844\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-3d\" x=\"611.376953\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-34\" x=\"695.166016\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_2\">\n",
       "     <g id=\"line2d_5\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m3482c907f8\" x=\"194.489062\" y=\"49.823182\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_6\">\n",
       "      <!-- HF GPT-Neo-1.3B, 2 GPUs, zero-2 -->\n",
       "      <g transform=\"translate(22.021875 48.783338)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-48\" d=\"M 628 4666 \n",
       "L 1259 4666 \n",
       "L 1259 2753 \n",
       "L 3553 2753 \n",
       "L 3553 4666 \n",
       "L 4184 4666 \n",
       "L 4184 0 \n",
       "L 3553 0 \n",
       "L 3553 2222 \n",
       "L 1259 2222 \n",
       "L 1259 0 \n",
       "L 628 0 \n",
       "L 628 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-46\" d=\"M 628 4666 \n",
       "L 3309 4666 \n",
       "L 3309 4134 \n",
       "L 1259 4134 \n",
       "L 1259 2759 \n",
       "L 3109 2759 \n",
       "L 3109 2228 \n",
       "L 1259 2228 \n",
       "L 1259 0 \n",
       "L 628 0 \n",
       "L 628 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "        <path id=\"DejaVuSans-7a\" d=\"M 353 3500 \n",
       "L 3084 3500 \n",
       "L 3084 2975 \n",
       "L 922 459 \n",
       "L 3084 459 \n",
       "L 3084 0 \n",
       "L 275 0 \n",
       "L 275 525 \n",
       "L 2438 3041 \n",
       "L 353 3041 \n",
       "L 353 3500 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-48\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-46\" x=\"75.195312\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-20\" x=\"132.714844\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-47\" x=\"164.501953\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-50\" x=\"241.992188\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-54\" x=\"302.294922\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-2d\" x=\"354.253906\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-4e\" x=\"390.337891\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-65\" x=\"465.142578\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-6f\" x=\"526.666016\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-2d\" x=\"589.722656\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-31\" x=\"625.806641\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-2e\" x=\"689.429688\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-33\" x=\"721.216797\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-42\" x=\"784.839844\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-2c\" x=\"853.443359\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-20\" x=\"885.230469\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-32\" x=\"917.017578\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-20\" x=\"980.640625\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-47\" x=\"1012.427734\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-50\" x=\"1089.917969\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-55\" x=\"1150.220703\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-73\" x=\"1223.414062\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-2c\" x=\"1275.513672\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-20\" x=\"1307.300781\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-7a\" x=\"1339.087891\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-65\" x=\"1391.578125\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-72\" x=\"1453.101562\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-6f\" x=\"1491.964844\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-2d\" x=\"1555.021484\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-32\" x=\"1591.105469\"/>\n",
       "      </g>\n",
       "      <!-- grad_accum=16 -->\n",
       "      <g transform=\"translate(105.245312 59.981151)scale(0.1 -0.1)\">\n",
       "       <defs>\n",
       "        <path id=\"DejaVuSans-36\" d=\"M 2113 2584 \n",
       "Q 1688 2584 1439 2293 \n",
       "Q 1191 2003 1191 1497 \n",
       "Q 1191 994 1439 701 \n",
       "Q 1688 409 2113 409 \n",
       "Q 2538 409 2786 701 \n",
       "Q 3034 994 3034 1497 \n",
       "Q 3034 2003 2786 2293 \n",
       "Q 2538 2584 2113 2584 \n",
       "z\n",
       "M 3366 4563 \n",
       "L 3366 3988 \n",
       "Q 3128 4100 2886 4159 \n",
       "Q 2644 4219 2406 4219 \n",
       "Q 1781 4219 1451 3797 \n",
       "Q 1122 3375 1075 2522 \n",
       "Q 1259 2794 1537 2939 \n",
       "Q 1816 3084 2150 3084 \n",
       "Q 2853 3084 3261 2657 \n",
       "Q 3669 2231 3669 1497 \n",
       "Q 3669 778 3244 343 \n",
       "Q 2819 -91 2113 -91 \n",
       "Q 1303 -91 875 529 \n",
       "Q 447 1150 447 2328 \n",
       "Q 447 3434 972 4092 \n",
       "Q 1497 4750 2381 4750 \n",
       "Q 2619 4750 2861 4703 \n",
       "Q 3103 4656 3366 4563 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       </defs>\n",
       "       <use xlink:href=\"#DejaVuSans-67\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-72\" x=\"63.476562\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-61\" x=\"104.589844\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-64\" x=\"165.869141\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-5f\" x=\"229.345703\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-61\" x=\"279.345703\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-63\" x=\"340.625\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-63\" x=\"395.605469\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-75\" x=\"450.585938\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-6d\" x=\"513.964844\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-3d\" x=\"611.376953\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-31\" x=\"695.166016\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-36\" x=\"758.789062\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"patch_5\">\n",
       "    <path d=\"M 194.489062 61.56 \n",
       "L 194.489062 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_6\">\n",
       "    <path d=\"M 342.194945 61.56 \n",
       "L 342.194945 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_7\">\n",
       "    <path d=\"M 194.489062 61.56 \n",
       "L 342.194945 61.56 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_8\">\n",
       "    <path d=\"M 194.489062 7.2 \n",
       "L 342.194945 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "  </g>\n",
       "  <g id=\"axes_2\">\n",
       "   <g id=\"patch_9\">\n",
       "    <path d=\"M 371.736121 61.56 \n",
       "L 519.442004 61.56 \n",
       "L 519.442004 7.2 \n",
       "L 371.736121 7.2 \n",
       "z\n",
       "\" style=\"fill: #ffffff\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_10\">\n",
       "    <path d=\"M 371.736121 9.670909 \n",
       "L 512.40839 9.670909 \n",
       "L 512.40839 28.202727 \n",
       "L 371.736121 28.202727 \n",
       "z\n",
       "\" clip-path=\"url(#p01451ce677)\" style=\"fill: #e41a1c\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_11\">\n",
       "    <path d=\"M 371.736121 40.557273 \n",
       "L 435.03099 40.557273 \n",
       "L 435.03099 59.089091 \n",
       "L 371.736121 59.089091 \n",
       "z\n",
       "\" clip-path=\"url(#p01451ce677)\" style=\"fill: #377eb8\"/>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_3\">\n",
       "    <g id=\"xtick_4\">\n",
       "     <g id=\"line2d_6\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#ma1fcd821d2\" x=\"371.736121\" y=\"61.56\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_7\">\n",
       "      <!-- 0 -->\n",
       "      <g transform=\"translate(368.554871 76.158437)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_5\">\n",
       "     <g id=\"line2d_7\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#ma1fcd821d2\" x=\"433.752216\" y=\"61.56\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_8\">\n",
       "      <!-- 20 -->\n",
       "      <g transform=\"translate(427.389716 76.158437)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-32\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_6\">\n",
       "     <g id=\"line2d_8\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#ma1fcd821d2\" x=\"495.768311\" y=\"61.56\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_9\">\n",
       "      <!-- 40 -->\n",
       "      <g transform=\"translate(489.405811 76.158437)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-34\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"text_10\">\n",
       "     <!-- per GPU TFLOPS -->\n",
       "     <g transform=\"translate(404.896875 89.836562)scale(0.1 -0.1)\">\n",
       "      <defs>\n",
       "       <path id=\"DejaVuSans-4c\" d=\"M 628 4666 \n",
       "L 1259 4666 \n",
       "L 1259 531 \n",
       "L 3531 531 \n",
       "L 3531 0 \n",
       "L 628 0 \n",
       "L 628 4666 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-4f\" d=\"M 2522 4238 \n",
       "Q 1834 4238 1429 3725 \n",
       "Q 1025 3213 1025 2328 \n",
       "Q 1025 1447 1429 934 \n",
       "Q 1834 422 2522 422 \n",
       "Q 3209 422 3611 934 \n",
       "Q 4013 1447 4013 2328 \n",
       "Q 4013 3213 3611 3725 \n",
       "Q 3209 4238 2522 4238 \n",
       "z\n",
       "M 2522 4750 \n",
       "Q 3503 4750 4090 4092 \n",
       "Q 4678 3434 4678 2328 \n",
       "Q 4678 1225 4090 567 \n",
       "Q 3503 -91 2522 -91 \n",
       "Q 1538 -91 948 565 \n",
       "Q 359 1222 359 2328 \n",
       "Q 359 3434 948 4092 \n",
       "Q 1538 4750 2522 4750 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "      </defs>\n",
       "      <use xlink:href=\"#DejaVuSans-70\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-65\" x=\"63.476562\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-72\" x=\"125\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-20\" x=\"166.113281\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-47\" x=\"197.900391\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-50\" x=\"275.390625\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-55\" x=\"335.693359\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-20\" x=\"408.886719\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-54\" x=\"440.673828\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-46\" x=\"501.757812\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-4c\" x=\"559.277344\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-4f\" x=\"611.365234\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-50\" x=\"690.076172\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-53\" x=\"750.378906\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_4\">\n",
       "    <g id=\"ytick_3\">\n",
       "     <g id=\"line2d_9\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m3482c907f8\" x=\"371.736121\" y=\"18.936818\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_4\">\n",
       "     <g id=\"line2d_10\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m3482c907f8\" x=\"371.736121\" y=\"49.823182\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"patch_12\">\n",
       "    <path d=\"M 371.736121 61.56 \n",
       "L 371.736121 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_13\">\n",
       "    <path d=\"M 519.442004 61.56 \n",
       "L 519.442004 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_14\">\n",
       "    <path d=\"M 371.736121 61.56 \n",
       "L 519.442004 61.56 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_15\">\n",
       "    <path d=\"M 371.736121 7.2 \n",
       "L 519.442004 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "  </g>\n",
       "  <g id=\"axes_3\">\n",
       "   <g id=\"patch_16\">\n",
       "    <path d=\"M 548.98318 61.56 \n",
       "L 696.689063 61.56 \n",
       "L 696.689063 7.2 \n",
       "L 548.98318 7.2 \n",
       "z\n",
       "\" style=\"fill: #ffffff\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_17\">\n",
       "    <path d=\"M 548.98318 9.670909 \n",
       "L 668.547653 9.670909 \n",
       "L 668.547653 28.202727 \n",
       "L 548.98318 28.202727 \n",
       "z\n",
       "\" clip-path=\"url(#p65563201bf)\" style=\"fill: #e41a1c\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_18\">\n",
       "    <path d=\"M 548.98318 40.557273 \n",
       "L 689.655449 40.557273 \n",
       "L 689.655449 59.089091 \n",
       "L 548.98318 59.089091 \n",
       "z\n",
       "\" clip-path=\"url(#p65563201bf)\" style=\"fill: #377eb8\"/>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_5\">\n",
       "    <g id=\"xtick_7\">\n",
       "     <g id=\"line2d_11\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#ma1fcd821d2\" x=\"548.98318\" y=\"61.56\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_11\">\n",
       "      <!-- 0 -->\n",
       "      <g transform=\"translate(545.80193 76.158437)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-30\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_8\">\n",
       "     <g id=\"line2d_12\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#ma1fcd821d2\" x=\"616.629331\" y=\"61.56\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_12\">\n",
       "      <!-- 10 -->\n",
       "      <g transform=\"translate(610.266831 76.158437)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-31\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"xtick_9\">\n",
       "     <g id=\"line2d_13\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#ma1fcd821d2\" x=\"684.275482\" y=\"61.56\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "     <g id=\"text_13\">\n",
       "      <!-- 20 -->\n",
       "      <g transform=\"translate(677.912982 76.158437)scale(0.1 -0.1)\">\n",
       "       <use xlink:href=\"#DejaVuSans-32\"/>\n",
       "       <use xlink:href=\"#DejaVuSans-30\" x=\"63.623047\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"text_14\">\n",
       "     <!-- per GPU memory (GB) -->\n",
       "     <g transform=\"translate(567.11659 89.836562)scale(0.1 -0.1)\">\n",
       "      <defs>\n",
       "       <path id=\"DejaVuSans-79\" d=\"M 2059 -325 \n",
       "Q 1816 -950 1584 -1140 \n",
       "Q 1353 -1331 966 -1331 \n",
       "L 506 -1331 \n",
       "L 506 -850 \n",
       "L 844 -850 \n",
       "Q 1081 -850 1212 -737 \n",
       "Q 1344 -625 1503 -206 \n",
       "L 1606 56 \n",
       "L 191 3500 \n",
       "L 800 3500 \n",
       "L 1894 763 \n",
       "L 2988 3500 \n",
       "L 3597 3500 \n",
       "L 2059 -325 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-28\" d=\"M 1984 4856 \n",
       "Q 1566 4138 1362 3434 \n",
       "Q 1159 2731 1159 2009 \n",
       "Q 1159 1288 1364 580 \n",
       "Q 1569 -128 1984 -844 \n",
       "L 1484 -844 \n",
       "Q 1016 -109 783 600 \n",
       "Q 550 1309 550 2009 \n",
       "Q 550 2706 781 3412 \n",
       "Q 1013 4119 1484 4856 \n",
       "L 1984 4856 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "       <path id=\"DejaVuSans-29\" d=\"M 513 4856 \n",
       "L 1013 4856 \n",
       "Q 1481 4119 1714 3412 \n",
       "Q 1947 2706 1947 2009 \n",
       "Q 1947 1309 1714 600 \n",
       "Q 1481 -109 1013 -844 \n",
       "L 513 -844 \n",
       "Q 928 -128 1133 580 \n",
       "Q 1338 1288 1338 2009 \n",
       "Q 1338 2731 1133 3434 \n",
       "Q 928 4138 513 4856 \n",
       "z\n",
       "\" transform=\"scale(0.015625)\"/>\n",
       "      </defs>\n",
       "      <use xlink:href=\"#DejaVuSans-70\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-65\" x=\"63.476562\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-72\" x=\"125\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-20\" x=\"166.113281\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-47\" x=\"197.900391\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-50\" x=\"275.390625\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-55\" x=\"335.693359\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-20\" x=\"408.886719\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-6d\" x=\"440.673828\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-65\" x=\"538.085938\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-6d\" x=\"599.609375\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-6f\" x=\"697.021484\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-72\" x=\"758.203125\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-79\" x=\"799.316406\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-20\" x=\"858.496094\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-28\" x=\"890.283203\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-47\" x=\"929.296875\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-42\" x=\"1006.787109\"/>\n",
       "      <use xlink:href=\"#DejaVuSans-29\" x=\"1075.390625\"/>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"matplotlib.axis_6\">\n",
       "    <g id=\"ytick_5\">\n",
       "     <g id=\"line2d_14\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m3482c907f8\" x=\"548.98318\" y=\"18.936818\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "    <g id=\"ytick_6\">\n",
       "     <g id=\"line2d_15\">\n",
       "      <g>\n",
       "       <use xlink:href=\"#m3482c907f8\" x=\"548.98318\" y=\"49.823182\" style=\"stroke: #000000; stroke-width: 0.8\"/>\n",
       "      </g>\n",
       "     </g>\n",
       "    </g>\n",
       "   </g>\n",
       "   <g id=\"patch_19\">\n",
       "    <path d=\"M 548.98318 61.56 \n",
       "L 548.98318 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_20\">\n",
       "    <path d=\"M 696.689063 61.56 \n",
       "L 696.689063 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_21\">\n",
       "    <path d=\"M 548.98318 61.56 \n",
       "L 696.689063 61.56 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "   <g id=\"patch_22\">\n",
       "    <path d=\"M 548.98318 7.2 \n",
       "L 696.689063 7.2 \n",
       "\" style=\"fill: none; stroke: #000000; stroke-width: 0.8; stroke-linejoin: miter; stroke-linecap: square\"/>\n",
       "   </g>\n",
       "  </g>\n",
       " </g>\n",
       " <defs>\n",
       "  <clipPath id=\"p047fcebe24\">\n",
       "   <rect x=\"194.489062\" y=\"7.2\" width=\"147.705882\" height=\"54.36\"/>\n",
       "  </clipPath>\n",
       "  <clipPath id=\"p01451ce677\">\n",
       "   <rect x=\"371.736121\" y=\"7.2\" width=\"147.705882\" height=\"54.36\"/>\n",
       "  </clipPath>\n",
       "  <clipPath id=\"p65563201bf\">\n",
       "   <rect x=\"548.98318\" y=\"7.2\" width=\"147.705882\" height=\"54.36\"/>\n",
       "  </clipPath>\n",
       " </defs>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<Figure size 648x72 with 3 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "zero2_gpt_neo_accum = hf_gpt(Exp(\n",
    "    \"HF GPT-Neo-1.3B, 2 GPUs, zero-2\\ngrad_accum=16\", \"EleutherAI/gpt-neo-1.3B\", 1, bf16=True, gpus='0,1',\n",
    "    optim='adamw_apex_fused', grad_accum=16,\n",
    "    steps=5, deepspeed=True,  ds_config='transformers/tests/deepspeed/ds_config_zero2.json'))\n",
    "compare([tp_gpt_neo_accum, zero2_gpt_neo_accum])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "46a8f6f9",
   "metadata": {},
   "source": [
    "## Conclusion\n",
    "\n",
    "1. To get a good performance we need to use **a sufficient large batch size** for better operator performance and reducing the cost ratio of communication and parameter updating. All large GPU memory size, reduced precision data types, kernel fusion, gradient accumulation, and gradient checkpoints help. Though a too large batch size affects convergence, especially for fine-tuning or pre-training with hundreds of GPUs. \n",
    "\n",
    "1. Data parallelism works well if your model fits into a single GPU. Otherwise you can use tensor parallelism and ZeRO.\n",
    "\n",
    "## Exercises\n",
    "\n",
    "1. Evaluate this notebook on your GPU machines. You may need to tune the batch size to fit into your memory.\n",
    "1. Try to use a smaller batch size, check how it affect TFLOPS.\n",
    "1. Try to use pipeline parallelism in Megatron.\n",
    "1. Try to use deepspeed with Megatron\n",
    "1. Try to use DP with gradient checkpoint for `GPT-Neo-1.3B`\n",
    "1. Try to benchmark T5, which has both encoder and decoder.\n",
    "1. Try the largest model in you ran in your machine.\n",
    "1. Estimate the time you need to train `gpt2-medium` on 1 billion tokens."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}