examples/tec_forecast.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "authorship_tag": "ABX9TyNRkp+/bBcZr4mbKa16Ka1d",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/mauriciodev/tec_forecast/blob/main/examples/tec_forecast.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "import pandas as pd"
      ],
      "metadata": {
        "id": "6ztmBM7Kzf3g"
      },
      "execution_count": 8,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Obtaining code and data"
      ],
      "metadata": {
        "id": "CQKnBsi_yk42"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!git clone https://github.com/mauriciodev/tec_forecast.git\n",
        "%cd tec_forecast"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "txgC1qAavB1j",
        "outputId": "95c9753f-5045-4415-db68-6ea4d558e072"
      },
      "execution_count": 2,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Cloning into 'tec_forecast'...\n",
            "remote: Enumerating objects: 144, done.\u001b[K\n",
            "remote: Counting objects: 100% (144/144), done.\u001b[K\n",
            "remote: Compressing objects: 100% (89/89), done.\u001b[K\n",
            "remote: Total 144 (delta 76), reused 113 (delta 48), pack-reused 0\u001b[K\n",
            "Receiving objects: 100% (144/144), 1.84 MiB | 20.74 MiB/s, done.\n",
            "Resolving deltas: 100% (76/76), done.\n",
            "/content/tec_forecast\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "The .npy files contain one year of GIM data.\n"
      ],
      "metadata": {
        "id": "3t7I0Lbfyp2n"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!gdown 1Sm_PiVUIabaew_3Y7sT0NWBqu7xsdHvi\n",
        "!tar -xvzf tec_forecast.tar.gz"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "-gb2s-wtupHI",
        "outputId": "d12e54e1-e14f-45ec-8d0f-cb9510e8905b"
      },
      "execution_count": 3,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Downloading...\n",
            "From: https://drive.google.com/uc?id=1Sm_PiVUIabaew_3Y7sT0NWBqu7xsdHvi\n",
            "To: /content/tec_forecast/tec_forecast.tar.gz\n",
            "100% 229M/229M [00:02<00:00, 81.4MB/s]\n",
            "codg2014.npy\n",
            "codg2015.npy\n",
            "codg2019.npy\n",
            "codg2020.npy\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Configuring experiments\n",
        "The experiments are listed on config.csv. The networks and hyperparameters are set on that file. We are going to use pandas to edit it, so that we will only run one training cycle for each network.\n",
        "To batch train and test a network, change the batch_train and batch_test columns."
      ],
      "metadata": {
        "id": "tcm3b0S7z3AA"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## The configuration file"
      ],
      "metadata": {
        "id": "6Rl-k-j600Es"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "df=pd.read_csv('/content/tec_forecast/config.csv')\n",
        "df"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "id": "QjgTr2y30F9z",
        "outputId": "f9ce7c38-7ad5-4ec3-a074-570e1a6c49a4"
      },
      "execution_count": 14,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "                        comment              experiment_name  batch_train  \\\n",
              "0                          None                 Repeat_19-20        False   \n",
              "1                          None                 Repeat_14-15        False   \n",
              "2           This is not a model                         c1pg        False   \n",
              "3           This is not a model                    c1pg14-15        False   \n",
              "4   Very low memory consumption                ANN_NtoN_12_2        False   \n",
              "5   Very low memory consumption                ANN_NtoN_18_2        False   \n",
              "6   Very low memory consumption                ANN_NtoN_24_2        False   \n",
              "7   Very low memory consumption                ANN_NtoN_30_2        False   \n",
              "8   Very low memory consumption                ANN_NtoN_36_2        False   \n",
              "9                deeptec1-19-20                    iConvLSTM         True   \n",
              "10                          NaN              EF-ConvLSTM_1x1        False   \n",
              "11               deeptec1-19-20              EF-ConvLSTM_3x3         True   \n",
              "12                          NaN           REF-ConvLSTMv1_1x1        False   \n",
              "13               deeptec1-19-20           REF-ConvLSTMv1_3x3         True   \n",
              "14                          NaN            EF-ConvLSTMv2_1x1        False   \n",
              "15               deeptec1-19-20            EF-ConvLSTMv2_3x3         True   \n",
              "16               deeptec1-19-20                EF-ConvLSTMv3         True   \n",
              "17                         cbcg                  ConvLSTM_Bi        False   \n",
              "18                         cbcg                 ConvLSTM_N-1        False   \n",
              "19                         cbcg               EF_ConvLSTM_60        False   \n",
              "20                         cbcg               EF_ConvLSTM_48        False   \n",
              "21                         cbcg               EF_ConvLSTM_36        False   \n",
              "22                         cbcg               EF_ConvLSTM_24        False   \n",
              "23                         cbcg               EF_ConvLSTM_12        False   \n",
              "24                     deeptec1              IconvLSTM_14-15         True   \n",
              "25                          NaN        EF-ConvLSTM_1x1_14-15        False   \n",
              "26               deeptec1-14-15        EF-ConvLSTM_3x3_14-15         True   \n",
              "27                          NaN      EF-ConvLSTMv2_1x1_14-15        False   \n",
              "28              deeptec1-14-15    EF-ConvLSTMv2_3x3_14-15_12         True   \n",
              "29               deeptec1-14-15   EF-ConvLSTMv2_3x3_14-15_24         True   \n",
              "30               deeptec1-14-15      EF-ConvLSTMv2_3x3_14-15         True   \n",
              "31               deeptec1-14-15   EF-ConvLSTMv2_3x3_14-15_48         True   \n",
              "32                Out of memory   EF-ConvLSTMv2_3x3_14-15_60        False   \n",
              "33                     deeptec1          EF-ConvLSTMv3_14-15         True   \n",
              "34                          NaN     REF-ConvLSTMv1_1x1_14-15        False   \n",
              "35               deeptec1-14-15  REF-ConvLSTMv1_3x3_14-15_12         True   \n",
              "36               deeptec1-14-15  REF-ConvLSTMv1_3x3_14-15_24         True   \n",
              "37               deeptec1-14-15     REF-ConvLSTMv1_3x3_14-15         True   \n",
              "38               deeptec1-14-15  REF-ConvLSTMv1_3x3_14-15_48         True   \n",
              "\n",
              "    batch_test  tested  compare  batch_size  num_epochs  filters        model  \\\n",
              "0        False    True     True           4          30        8  usePrevious   \n",
              "1        False    True     True           4          30        8  usePrevious   \n",
              "2        False    True     True           1           1        0         c1pg   \n",
              "3        False    True     True           1           1        0         c1pg   \n",
              "4        False    True     True          32         200      150          ANN   \n",
              "5        False    True     True          32         200      150          ANN   \n",
              "6        False    True     True          32         200      150          ANN   \n",
              "7        False    True     True          32         200      150          ANN   \n",
              "8        False    True     True          32         200      150          ANN   \n",
              "9         True    True     True          16         200       48    iConvLSTM   \n",
              "10       False    True     True           4         200       24         c111   \n",
              "11        True    True     True           8         200       24         c333   \n",
              "12       False    True     True           4         200       24     c111_res   \n",
              "13        True    True     True           4         200       24     c333_res   \n",
              "14       False    True     True           4         200       24  c111_res_v2   \n",
              "15        True    True     True           8         200       24  c333_res_v2   \n",
              "16        True    True     True           8         200       24  c333_res_v3   \n",
              "17       False    True     True           8         200       24       c333bi   \n",
              "18       False    True     True           8         200       24    c333_nto1   \n",
              "19       False    True     True           4         200       24         c333   \n",
              "20       False    True     True           4         200       24         c333   \n",
              "21       False    True     True           4         200       24         c333   \n",
              "22       False    True     True           4         200       24         c333   \n",
              "23       False    True     True           4         200       24         c333   \n",
              "24        True    True     True          16         200       48    iConvLSTM   \n",
              "25       False    True     True           4         200       24         c111   \n",
              "26        True    True     True           4         200       24         c333   \n",
              "27       False    True     True           4         200       24  c111_res_v2   \n",
              "28        True    True     True           8         200       24  c333_res_v2   \n",
              "29        True    True     True           8         200       24  c333_res_v2   \n",
              "30        True    True     True           8         200       24  c333_res_v2   \n",
              "31        True    True     True           8         200       24  c333_res_v2   \n",
              "32       False   False     True           8         200       24  c333_res_v2   \n",
              "33        True    True     True           8         200       24  c333_res_v3   \n",
              "34       False    True     True           4         200       24     c111_res   \n",
              "35        True    True     True           4         200       24     c333_res   \n",
              "36        True    True     True           4         200       24     c333_res   \n",
              "37        True    True     True           4         200       24     c333_res   \n",
              "38        True    True     True           4         200       24     c333_res   \n",
              "\n",
              "    ... prediction_window  train_time_sampling  test_time_sampling  \\\n",
              "0   ...                12                    2                   2   \n",
              "1   ...                12                    1                   2   \n",
              "2   ...                12                    2                   2   \n",
              "3   ...                12                    2                   2   \n",
              "4   ...                12                    2                   2   \n",
              "5   ...                12                    2                   2   \n",
              "6   ...                12                    2                   2   \n",
              "7   ...                12                    2                   2   \n",
              "8   ...                12                    2                   2   \n",
              "9   ...                12                    2                   2   \n",
              "10  ...                12                    2                   2   \n",
              "11  ...                12                    2                   2   \n",
              "12  ...                12                    2                   2   \n",
              "13  ...                12                    2                   2   \n",
              "14  ...                12                    2                   2   \n",
              "15  ...                12                    2                   2   \n",
              "16  ...                12                    2                   2   \n",
              "17  ...                12                    2                   2   \n",
              "18  ...                12                    2                   2   \n",
              "19  ...                12                    2                   2   \n",
              "20  ...                12                    2                   2   \n",
              "21  ...                12                    2                   2   \n",
              "22  ...                12                    2                   2   \n",
              "23  ...                12                    2                   2   \n",
              "24  ...                12                    1                   2   \n",
              "25  ...                12                    1                   2   \n",
              "26  ...                12                    1                   2   \n",
              "27  ...                12                    1                   2   \n",
              "28  ...                12                    1                   2   \n",
              "29  ...                12                    1                   2   \n",
              "30  ...                12                    1                   2   \n",
              "31  ...                12                    1                   2   \n",
              "32  ...                12                    1                   2   \n",
              "33  ...                12                    1                   2   \n",
              "34  ...                12                    1                   2   \n",
              "35  ...                12                    1                   2   \n",
              "36  ...                12                    1                   2   \n",
              "37  ...                12                    1                   2   \n",
              "38  ...                12                    1                   2   \n",
              "\n",
              "    resample_rate  train_npy_dataset  test_npy_dataset random_seed loss  \\\n",
              "0              12       codg2019.npy      codg2020.npy           1  mae   \n",
              "1              12       codg2014.npy      codg2015.npy           1  mae   \n",
              "2              12       codg2019.npy      codg2020.npy           1  mae   \n",
              "3              12       c1pg2015.npy      codg2015.npy           1  mae   \n",
              "4              12       codg2019.npy      codg2020.npy           1  mae   \n",
              "5              12       codg2019.npy      codg2020.npy           1  mae   \n",
              "6              12       codg2019.npy      codg2020.npy           1  mae   \n",
              "7              12       codg2019.npy      codg2020.npy           1  mae   \n",
              "8              12       codg2019.npy      codg2020.npy           1  mae   \n",
              "9              12       codg2019.npy      codg2020.npy           1  mae   \n",
              "10             12       codg2019.npy      codg2020.npy           1  mae   \n",
              "11             12       codg2019.npy      codg2020.npy           1  mae   \n",
              "12             12       codg2019.npy      codg2020.npy           1  mae   \n",
              "13             12       codg2019.npy      codg2020.npy           1  mae   \n",
              "14             12       codg2019.npy      codg2020.npy           1  mae   \n",
              "15             12       codg2019.npy      codg2020.npy           1  mae   \n",
              "16             12       codg2019.npy      codg2020.npy           1  mae   \n",
              "17             12       codg2019.npy      codg2020.npy           1  mae   \n",
              "18             12       codg2019.npy      codg2020.npy           1  mae   \n",
              "19             12       codg2019.npy      codg2020.npy           1  mae   \n",
              "20             12       codg2019.npy      codg2020.npy           1  mae   \n",
              "21             12       codg2019.npy      codg2020.npy           1  mae   \n",
              "22             12       codg2019.npy      codg2020.npy           1  mae   \n",
              "23             12       codg2019.npy      codg2020.npy           1  mae   \n",
              "24             12       codg2014.npy      codg2015.npy           1  mae   \n",
              "25             12       codg2014.npy      codg2015.npy           1  mae   \n",
              "26             12       codg2014.npy      codg2015.npy           1  mae   \n",
              "27             12       codg2014.npy      codg2015.npy           1  mae   \n",
              "28             12       codg2014.npy      codg2015.npy           1  mae   \n",
              "29             12       codg2014.npy      codg2015.npy           1  mae   \n",
              "30             12       codg2014.npy      codg2015.npy           1  mae   \n",
              "31             12       codg2014.npy      codg2015.npy           1  mae   \n",
              "32             12       codg2014.npy      codg2015.npy           1  mae   \n",
              "33             12       codg2014.npy      codg2015.npy           1  mae   \n",
              "34             12       codg2014.npy      codg2015.npy           1  mae   \n",
              "35             12       codg2014.npy      codg2015.npy           1  mae   \n",
              "36             12       codg2014.npy      codg2015.npy           1  mae   \n",
              "37             12       codg2014.npy      codg2015.npy           1  mae   \n",
              "38             12       codg2014.npy      codg2015.npy           1  mae   \n",
              "\n",
              "    optimizer best_of  \n",
              "0        adam       1  \n",
              "1        adam       1  \n",
              "2        adam       1  \n",
              "3        adam       1  \n",
              "4        adam       1  \n",
              "5        adam       1  \n",
              "6        adam       1  \n",
              "7        adam       1  \n",
              "8        adam       1  \n",
              "9        adam       1  \n",
              "10       adam       1  \n",
              "11       adam       1  \n",
              "12       adam       1  \n",
              "13       adam       1  \n",
              "14       adam       1  \n",
              "15       adam       1  \n",
              "16       adam       1  \n",
              "17       adam       1  \n",
              "18       adam       1  \n",
              "19       adam       1  \n",
              "20       adam       1  \n",
              "21       adam       1  \n",
              "22       adam       1  \n",
              "23       adam       1  \n",
              "24       adam       1  \n",
              "25       adam       1  \n",
              "26       adam       1  \n",
              "27       adam       1  \n",
              "28       adam       1  \n",
              "29       adam       1  \n",
              "30       adam       1  \n",
              "31       adam       1  \n",
              "32       adam       1  \n",
              "33       adam       1  \n",
              "34       adam       1  \n",
              "35       adam       1  \n",
              "36       adam       1  \n",
              "37       adam       1  \n",
              "38       adam       1  \n",
              "\n",
              "[39 rows x 22 columns]"
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-cd7394a5-19af-4eee-ab88-aea09466b297\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>comment</th>\n",
              "      <th>experiment_name</th>\n",
              "      <th>batch_train</th>\n",
              "      <th>batch_test</th>\n",
              "      <th>tested</th>\n",
              "      <th>compare</th>\n",
              "      <th>batch_size</th>\n",
              "      <th>num_epochs</th>\n",
              "      <th>filters</th>\n",
              "      <th>model</th>\n",
              "      <th>...</th>\n",
              "      <th>prediction_window</th>\n",
              "      <th>train_time_sampling</th>\n",
              "      <th>test_time_sampling</th>\n",
              "      <th>resample_rate</th>\n",
              "      <th>train_npy_dataset</th>\n",
              "      <th>test_npy_dataset</th>\n",
              "      <th>random_seed</th>\n",
              "      <th>loss</th>\n",
              "      <th>optimizer</th>\n",
              "      <th>best_of</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>None</td>\n",
              "      <td>Repeat_19-20</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>True</td>\n",
              "      <td>True</td>\n",
              "      <td>4</td>\n",
              "      <td>30</td>\n",
              "      <td>8</td>\n",
              "      <td>usePrevious</td>\n",
              "      <td>...</td>\n",
              "      <td>12</td>\n",
              "      <td>2</td>\n",
              "      <td>2</td>\n",
              "      <td>12</td>\n",
              "      <td>codg2019.npy</td>\n",
              "      <td>codg2020.npy</td>\n",
              "      <td>1</td>\n",
              "      <td>mae</td>\n",
              "      <td>adam</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>None</td>\n",
              "      <td>Repeat_14-15</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>True</td>\n",
              "      <td>True</td>\n",
              "      <td>4</td>\n",
              "      <td>30</td>\n",
              "      <td>8</td>\n",
              "      <td>usePrevious</td>\n",
              "      <td>...</td>\n",
              "      <td>12</td>\n",
              "      <td>1</td>\n",
              "      <td>2</td>\n",
              "      <td>12</td>\n",
              "      <td>codg2014.npy</td>\n",
              "      <td>codg2015.npy</td>\n",
              "      <td>1</td>\n",
              "      <td>mae</td>\n",
              "      <td>adam</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>This is not a model</td>\n",
              "      <td>c1pg</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>True</td>\n",
              "      <td>True</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>c1pg</td>\n",
              "      <td>...</td>\n",
              "      <td>12</td>\n",
              "      <td>2</td>\n",
              "      <td>2</td>\n",
              "      <td>12</td>\n",
              "      <td>codg2019.npy</td>\n",
              "      <td>codg2020.npy</td>\n",
              "      <td>1</td>\n",
              "      <td>mae</td>\n",
              "      <td>adam</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>This is not a model</td>\n",
              "      <td>c1pg14-15</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>True</td>\n",
              "      <td>True</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>c1pg</td>\n",
              "      <td>...</td>\n",
              "      <td>12</td>\n",
              "      <td>2</td>\n",
              "      <td>2</td>\n",
              "      <td>12</td>\n",
              "      <td>c1pg2015.npy</td>\n",
              "      <td>codg2015.npy</td>\n",
              "      <td>1</td>\n",
              "      <td>mae</td>\n",
              "      <td>adam</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>Very low memory consumption</td>\n",
              "      <td>ANN_NtoN_12_2</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>True</td>\n",
              "      <td>True</td>\n",
              "      <td>32</td>\n",
              "      <td>200</td>\n",
              "      <td>150</td>\n",
              "      <td>ANN</td>\n",
              "      <td>...</td>\n",
              "      <td>12</td>\n",
              "      <td>2</td>\n",
              "      <td>2</td>\n",
              "      <td>12</td>\n",
              "      <td>codg2019.npy</td>\n",
              "      <td>codg2020.npy</td>\n",
              "      <td>1</td>\n",
              "      <td>mae</td>\n",
              "      <td>adam</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5</th>\n",
              "      <td>Very low memory consumption</td>\n",
              "      <td>ANN_NtoN_18_2</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>True</td>\n",
              "      <td>True</td>\n",
              "      <td>32</td>\n",
              "      <td>200</td>\n",
              "      <td>150</td>\n",
              "      <td>ANN</td>\n",
              "      <td>...</td>\n",
              "      <td>12</td>\n",
              "      <td>2</td>\n",
              "      <td>2</td>\n",
              "      <td>12</td>\n",
              "      <td>codg2019.npy</td>\n",
              "      <td>codg2020.npy</td>\n",
              "      <td>1</td>\n",
              "      <td>mae</td>\n",
              "      <td>adam</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>6</th>\n",
              "      <td>Very low memory consumption</td>\n",
              "      <td>ANN_NtoN_24_2</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>True</td>\n",
              "      <td>True</td>\n",
              "      <td>32</td>\n",
              "      <td>200</td>\n",
              "      <td>150</td>\n",
              "      <td>ANN</td>\n",
              "      <td>...</td>\n",
              "      <td>12</td>\n",
              "      <td>2</td>\n",
              "      <td>2</td>\n",
              "      <td>12</td>\n",
              "      <td>codg2019.npy</td>\n",
              "      <td>codg2020.npy</td>\n",
              "      <td>1</td>\n",
              "      <td>mae</td>\n",
              "      <td>adam</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>7</th>\n",
              "      <td>Very low memory consumption</td>\n",
              "      <td>ANN_NtoN_30_2</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>True</td>\n",
              "      <td>True</td>\n",
              "      <td>32</td>\n",
              "      <td>200</td>\n",
              "      <td>150</td>\n",
              "      <td>ANN</td>\n",
              "      <td>...</td>\n",
              "      <td>12</td>\n",
              "      <td>2</td>\n",
              "      <td>2</td>\n",
              "      <td>12</td>\n",
              "      <td>codg2019.npy</td>\n",
              "      <td>codg2020.npy</td>\n",
              "      <td>1</td>\n",
              "      <td>mae</td>\n",
              "      <td>adam</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>8</th>\n",
              "      <td>Very low memory consumption</td>\n",
              "      <td>ANN_NtoN_36_2</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>True</td>\n",
              "      <td>True</td>\n",
              "      <td>32</td>\n",
              "      <td>200</td>\n",
              "      <td>150</td>\n",
              "      <td>ANN</td>\n",
              "      <td>...</td>\n",
              "      <td>12</td>\n",
              "      <td>2</td>\n",
              "      <td>2</td>\n",
              "      <td>12</td>\n",
              "      <td>codg2019.npy</td>\n",
              "      <td>codg2020.npy</td>\n",
              "      <td>1</td>\n",
              "      <td>mae</td>\n",
              "      <td>adam</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9</th>\n",
              "      <td>deeptec1-19-20</td>\n",
              "      <td>iConvLSTM</td>\n",
              "      <td>True</td>\n",
              "      <td>True</td>\n",
              "      <td>True</td>\n",
              "      <td>True</td>\n",
              "      <td>16</td>\n",
              "      <td>200</td>\n",
              "      <td>48</td>\n",
              "      <td>iConvLSTM</td>\n",
              "      <td>...</td>\n",
              "      <td>12</td>\n",
              "      <td>2</td>\n",
              "      <td>2</td>\n",
              "      <td>12</td>\n",
              "      <td>codg2019.npy</td>\n",
              "      <td>codg2020.npy</td>\n",
              "      <td>1</td>\n",
              "      <td>mae</td>\n",
              "      <td>adam</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>10</th>\n",
              "      <td>NaN</td>\n",
              "      <td>EF-ConvLSTM_1x1</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>True</td>\n",
              "      <td>True</td>\n",
              "      <td>4</td>\n",
              "      <td>200</td>\n",
              "      <td>24</td>\n",
              "      <td>c111</td>\n",
              "      <td>...</td>\n",
              "      <td>12</td>\n",
              "      <td>2</td>\n",
              "      <td>2</td>\n",
              "      <td>12</td>\n",
              "      <td>codg2019.npy</td>\n",
              "      <td>codg2020.npy</td>\n",
              "      <td>1</td>\n",
              "      <td>mae</td>\n",
              "      <td>adam</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>11</th>\n",
              "      <td>deeptec1-19-20</td>\n",
              "      <td>EF-ConvLSTM_3x3</td>\n",
              "      <td>True</td>\n",
              "      <td>True</td>\n",
              "      <td>True</td>\n",
              "      <td>True</td>\n",
              "      <td>8</td>\n",
              "      <td>200</td>\n",
              "      <td>24</td>\n",
              "      <td>c333</td>\n",
              "      <td>...</td>\n",
              "      <td>12</td>\n",
              "      <td>2</td>\n",
              "      <td>2</td>\n",
              "      <td>12</td>\n",
              "      <td>codg2019.npy</td>\n",
              "      <td>codg2020.npy</td>\n",
              "      <td>1</td>\n",
              "      <td>mae</td>\n",
              "      <td>adam</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>12</th>\n",
              "      <td>NaN</td>\n",
              "      <td>REF-ConvLSTMv1_1x1</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>True</td>\n",
              "      <td>True</td>\n",
              "      <td>4</td>\n",
              "      <td>200</td>\n",
              "      <td>24</td>\n",
              "      <td>c111_res</td>\n",
              "      <td>...</td>\n",
              "      <td>12</td>\n",
              "      <td>2</td>\n",
              "      <td>2</td>\n",
              "      <td>12</td>\n",
              "      <td>codg2019.npy</td>\n",
              "      <td>codg2020.npy</td>\n",
              "      <td>1</td>\n",
              "      <td>mae</td>\n",
              "      <td>adam</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>13</th>\n",
              "      <td>deeptec1-19-20</td>\n",
              "      <td>REF-ConvLSTMv1_3x3</td>\n",
              "      <td>True</td>\n",
              "      <td>True</td>\n",
              "      <td>True</td>\n",
              "      <td>True</td>\n",
              "      <td>4</td>\n",
              "      <td>200</td>\n",
              "      <td>24</td>\n",
              "      <td>c333_res</td>\n",
              "      <td>...</td>\n",
              "      <td>12</td>\n",
              "      <td>2</td>\n",
              "      <td>2</td>\n",
              "      <td>12</td>\n",
              "      <td>codg2019.npy</td>\n",
              "      <td>codg2020.npy</td>\n",
              "      <td>1</td>\n",
              "      <td>mae</td>\n",
              "      <td>adam</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>14</th>\n",
              "      <td>NaN</td>\n",
              "      <td>EF-ConvLSTMv2_1x1</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>True</td>\n",
              "      <td>True</td>\n",
              "      <td>4</td>\n",
              "      <td>200</td>\n",
              "      <td>24</td>\n",
              "      <td>c111_res_v2</td>\n",
              "      <td>...</td>\n",
              "      <td>12</td>\n",
              "      <td>2</td>\n",
              "      <td>2</td>\n",
              "      <td>12</td>\n",
              "      <td>codg2019.npy</td>\n",
              "      <td>codg2020.npy</td>\n",
              "      <td>1</td>\n",
              "      <td>mae</td>\n",
              "      <td>adam</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>15</th>\n",
              "      <td>deeptec1-19-20</td>\n",
              "      <td>EF-ConvLSTMv2_3x3</td>\n",
              "      <td>True</td>\n",
              "      <td>True</td>\n",
              "      <td>True</td>\n",
              "      <td>True</td>\n",
              "      <td>8</td>\n",
              "      <td>200</td>\n",
              "      <td>24</td>\n",
              "      <td>c333_res_v2</td>\n",
              "      <td>...</td>\n",
              "      <td>12</td>\n",
              "      <td>2</td>\n",
              "      <td>2</td>\n",
              "      <td>12</td>\n",
              "      <td>codg2019.npy</td>\n",
              "      <td>codg2020.npy</td>\n",
              "      <td>1</td>\n",
              "      <td>mae</td>\n",
              "      <td>adam</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>16</th>\n",
              "      <td>deeptec1-19-20</td>\n",
              "      <td>EF-ConvLSTMv3</td>\n",
              "      <td>True</td>\n",
              "      <td>True</td>\n",
              "      <td>True</td>\n",
              "      <td>True</td>\n",
              "      <td>8</td>\n",
              "      <td>200</td>\n",
              "      <td>24</td>\n",
              "      <td>c333_res_v3</td>\n",
              "      <td>...</td>\n",
              "      <td>12</td>\n",
              "      <td>2</td>\n",
              "      <td>2</td>\n",
              "      <td>12</td>\n",
              "      <td>codg2019.npy</td>\n",
              "      <td>codg2020.npy</td>\n",
              "      <td>1</td>\n",
              "      <td>mae</td>\n",
              "      <td>adam</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>17</th>\n",
              "      <td>cbcg</td>\n",
              "      <td>ConvLSTM_Bi</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>True</td>\n",
              "      <td>True</td>\n",
              "      <td>8</td>\n",
              "      <td>200</td>\n",
              "      <td>24</td>\n",
              "      <td>c333bi</td>\n",
              "      <td>...</td>\n",
              "      <td>12</td>\n",
              "      <td>2</td>\n",
              "      <td>2</td>\n",
              "      <td>12</td>\n",
              "      <td>codg2019.npy</td>\n",
              "      <td>codg2020.npy</td>\n",
              "      <td>1</td>\n",
              "      <td>mae</td>\n",
              "      <td>adam</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>18</th>\n",
              "      <td>cbcg</td>\n",
              "      <td>ConvLSTM_N-1</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>True</td>\n",
              "      <td>True</td>\n",
              "      <td>8</td>\n",
              "      <td>200</td>\n",
              "      <td>24</td>\n",
              "      <td>c333_nto1</td>\n",
              "      <td>...</td>\n",
              "      <td>12</td>\n",
              "      <td>2</td>\n",
              "      <td>2</td>\n",
              "      <td>12</td>\n",
              "      <td>codg2019.npy</td>\n",
              "      <td>codg2020.npy</td>\n",
              "      <td>1</td>\n",
              "      <td>mae</td>\n",
              "      <td>adam</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>19</th>\n",
              "      <td>cbcg</td>\n",
              "      <td>EF_ConvLSTM_60</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>True</td>\n",
              "      <td>True</td>\n",
              "      <td>4</td>\n",
              "      <td>200</td>\n",
              "      <td>24</td>\n",
              "      <td>c333</td>\n",
              "      <td>...</td>\n",
              "      <td>12</td>\n",
              "      <td>2</td>\n",
              "      <td>2</td>\n",
              "      <td>12</td>\n",
              "      <td>codg2019.npy</td>\n",
              "      <td>codg2020.npy</td>\n",
              "      <td>1</td>\n",
              "      <td>mae</td>\n",
              "      <td>adam</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>20</th>\n",
              "      <td>cbcg</td>\n",
              "      <td>EF_ConvLSTM_48</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>True</td>\n",
              "      <td>True</td>\n",
              "      <td>4</td>\n",
              "      <td>200</td>\n",
              "      <td>24</td>\n",
              "      <td>c333</td>\n",
              "      <td>...</td>\n",
              "      <td>12</td>\n",
              "      <td>2</td>\n",
              "      <td>2</td>\n",
              "      <td>12</td>\n",
              "      <td>codg2019.npy</td>\n",
              "      <td>codg2020.npy</td>\n",
              "      <td>1</td>\n",
              "      <td>mae</td>\n",
              "      <td>adam</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>21</th>\n",
              "      <td>cbcg</td>\n",
              "      <td>EF_ConvLSTM_36</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>True</td>\n",
              "      <td>True</td>\n",
              "      <td>4</td>\n",
              "      <td>200</td>\n",
              "      <td>24</td>\n",
              "      <td>c333</td>\n",
              "      <td>...</td>\n",
              "      <td>12</td>\n",
              "      <td>2</td>\n",
              "      <td>2</td>\n",
              "      <td>12</td>\n",
              "      <td>codg2019.npy</td>\n",
              "      <td>codg2020.npy</td>\n",
              "      <td>1</td>\n",
              "      <td>mae</td>\n",
              "      <td>adam</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>22</th>\n",
              "      <td>cbcg</td>\n",
              "      <td>EF_ConvLSTM_24</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>True</td>\n",
              "      <td>True</td>\n",
              "      <td>4</td>\n",
              "      <td>200</td>\n",
              "      <td>24</td>\n",
              "      <td>c333</td>\n",
              "      <td>...</td>\n",
              "      <td>12</td>\n",
              "      <td>2</td>\n",
              "      <td>2</td>\n",
              "      <td>12</td>\n",
              "      <td>codg2019.npy</td>\n",
              "      <td>codg2020.npy</td>\n",
              "      <td>1</td>\n",
              "      <td>mae</td>\n",
              "      <td>adam</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>23</th>\n",
              "      <td>cbcg</td>\n",
              "      <td>EF_ConvLSTM_12</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>True</td>\n",
              "      <td>True</td>\n",
              "      <td>4</td>\n",
              "      <td>200</td>\n",
              "      <td>24</td>\n",
              "      <td>c333</td>\n",
              "      <td>...</td>\n",
              "      <td>12</td>\n",
              "      <td>2</td>\n",
              "      <td>2</td>\n",
              "      <td>12</td>\n",
              "      <td>codg2019.npy</td>\n",
              "      <td>codg2020.npy</td>\n",
              "      <td>1</td>\n",
              "      <td>mae</td>\n",
              "      <td>adam</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>24</th>\n",
              "      <td>deeptec1</td>\n",
              "      <td>IconvLSTM_14-15</td>\n",
              "      <td>True</td>\n",
              "      <td>True</td>\n",
              "      <td>True</td>\n",
              "      <td>True</td>\n",
              "      <td>16</td>\n",
              "      <td>200</td>\n",
              "      <td>48</td>\n",
              "      <td>iConvLSTM</td>\n",
              "      <td>...</td>\n",
              "      <td>12</td>\n",
              "      <td>1</td>\n",
              "      <td>2</td>\n",
              "      <td>12</td>\n",
              "      <td>codg2014.npy</td>\n",
              "      <td>codg2015.npy</td>\n",
              "      <td>1</td>\n",
              "      <td>mae</td>\n",
              "      <td>adam</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>25</th>\n",
              "      <td>NaN</td>\n",
              "      <td>EF-ConvLSTM_1x1_14-15</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>True</td>\n",
              "      <td>True</td>\n",
              "      <td>4</td>\n",
              "      <td>200</td>\n",
              "      <td>24</td>\n",
              "      <td>c111</td>\n",
              "      <td>...</td>\n",
              "      <td>12</td>\n",
              "      <td>1</td>\n",
              "      <td>2</td>\n",
              "      <td>12</td>\n",
              "      <td>codg2014.npy</td>\n",
              "      <td>codg2015.npy</td>\n",
              "      <td>1</td>\n",
              "      <td>mae</td>\n",
              "      <td>adam</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>26</th>\n",
              "      <td>deeptec1-14-15</td>\n",
              "      <td>EF-ConvLSTM_3x3_14-15</td>\n",
              "      <td>True</td>\n",
              "      <td>True</td>\n",
              "      <td>True</td>\n",
              "      <td>True</td>\n",
              "      <td>4</td>\n",
              "      <td>200</td>\n",
              "      <td>24</td>\n",
              "      <td>c333</td>\n",
              "      <td>...</td>\n",
              "      <td>12</td>\n",
              "      <td>1</td>\n",
              "      <td>2</td>\n",
              "      <td>12</td>\n",
              "      <td>codg2014.npy</td>\n",
              "      <td>codg2015.npy</td>\n",
              "      <td>1</td>\n",
              "      <td>mae</td>\n",
              "      <td>adam</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>27</th>\n",
              "      <td>NaN</td>\n",
              "      <td>EF-ConvLSTMv2_1x1_14-15</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>True</td>\n",
              "      <td>True</td>\n",
              "      <td>4</td>\n",
              "      <td>200</td>\n",
              "      <td>24</td>\n",
              "      <td>c111_res_v2</td>\n",
              "      <td>...</td>\n",
              "      <td>12</td>\n",
              "      <td>1</td>\n",
              "      <td>2</td>\n",
              "      <td>12</td>\n",
              "      <td>codg2014.npy</td>\n",
              "      <td>codg2015.npy</td>\n",
              "      <td>1</td>\n",
              "      <td>mae</td>\n",
              "      <td>adam</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>28</th>\n",
              "      <td>deeptec1-14-15</td>\n",
              "      <td>EF-ConvLSTMv2_3x3_14-15_12</td>\n",
              "      <td>True</td>\n",
              "      <td>True</td>\n",
              "      <td>True</td>\n",
              "      <td>True</td>\n",
              "      <td>8</td>\n",
              "      <td>200</td>\n",
              "      <td>24</td>\n",
              "      <td>c333_res_v2</td>\n",
              "      <td>...</td>\n",
              "      <td>12</td>\n",
              "      <td>1</td>\n",
              "      <td>2</td>\n",
              "      <td>12</td>\n",
              "      <td>codg2014.npy</td>\n",
              "      <td>codg2015.npy</td>\n",
              "      <td>1</td>\n",
              "      <td>mae</td>\n",
              "      <td>adam</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>29</th>\n",
              "      <td>deeptec1-14-15</td>\n",
              "      <td>EF-ConvLSTMv2_3x3_14-15_24</td>\n",
              "      <td>True</td>\n",
              "      <td>True</td>\n",
              "      <td>True</td>\n",
              "      <td>True</td>\n",
              "      <td>8</td>\n",
              "      <td>200</td>\n",
              "      <td>24</td>\n",
              "      <td>c333_res_v2</td>\n",
              "      <td>...</td>\n",
              "      <td>12</td>\n",
              "      <td>1</td>\n",
              "      <td>2</td>\n",
              "      <td>12</td>\n",
              "      <td>codg2014.npy</td>\n",
              "      <td>codg2015.npy</td>\n",
              "      <td>1</td>\n",
              "      <td>mae</td>\n",
              "      <td>adam</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>30</th>\n",
              "      <td>deeptec1-14-15</td>\n",
              "      <td>EF-ConvLSTMv2_3x3_14-15</td>\n",
              "      <td>True</td>\n",
              "      <td>True</td>\n",
              "      <td>True</td>\n",
              "      <td>True</td>\n",
              "      <td>8</td>\n",
              "      <td>200</td>\n",
              "      <td>24</td>\n",
              "      <td>c333_res_v2</td>\n",
              "      <td>...</td>\n",
              "      <td>12</td>\n",
              "      <td>1</td>\n",
              "      <td>2</td>\n",
              "      <td>12</td>\n",
              "      <td>codg2014.npy</td>\n",
              "      <td>codg2015.npy</td>\n",
              "      <td>1</td>\n",
              "      <td>mae</td>\n",
              "      <td>adam</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>31</th>\n",
              "      <td>deeptec1-14-15</td>\n",
              "      <td>EF-ConvLSTMv2_3x3_14-15_48</td>\n",
              "      <td>True</td>\n",
              "      <td>True</td>\n",
              "      <td>True</td>\n",
              "      <td>True</td>\n",
              "      <td>8</td>\n",
              "      <td>200</td>\n",
              "      <td>24</td>\n",
              "      <td>c333_res_v2</td>\n",
              "      <td>...</td>\n",
              "      <td>12</td>\n",
              "      <td>1</td>\n",
              "      <td>2</td>\n",
              "      <td>12</td>\n",
              "      <td>codg2014.npy</td>\n",
              "      <td>codg2015.npy</td>\n",
              "      <td>1</td>\n",
              "      <td>mae</td>\n",
              "      <td>adam</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>32</th>\n",
              "      <td>Out of memory</td>\n",
              "      <td>EF-ConvLSTMv2_3x3_14-15_60</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>True</td>\n",
              "      <td>8</td>\n",
              "      <td>200</td>\n",
              "      <td>24</td>\n",
              "      <td>c333_res_v2</td>\n",
              "      <td>...</td>\n",
              "      <td>12</td>\n",
              "      <td>1</td>\n",
              "      <td>2</td>\n",
              "      <td>12</td>\n",
              "      <td>codg2014.npy</td>\n",
              "      <td>codg2015.npy</td>\n",
              "      <td>1</td>\n",
              "      <td>mae</td>\n",
              "      <td>adam</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>33</th>\n",
              "      <td>deeptec1</td>\n",
              "      <td>EF-ConvLSTMv3_14-15</td>\n",
              "      <td>True</td>\n",
              "      <td>True</td>\n",
              "      <td>True</td>\n",
              "      <td>True</td>\n",
              "      <td>8</td>\n",
              "      <td>200</td>\n",
              "      <td>24</td>\n",
              "      <td>c333_res_v3</td>\n",
              "      <td>...</td>\n",
              "      <td>12</td>\n",
              "      <td>1</td>\n",
              "      <td>2</td>\n",
              "      <td>12</td>\n",
              "      <td>codg2014.npy</td>\n",
              "      <td>codg2015.npy</td>\n",
              "      <td>1</td>\n",
              "      <td>mae</td>\n",
              "      <td>adam</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>34</th>\n",
              "      <td>NaN</td>\n",
              "      <td>REF-ConvLSTMv1_1x1_14-15</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>True</td>\n",
              "      <td>True</td>\n",
              "      <td>4</td>\n",
              "      <td>200</td>\n",
              "      <td>24</td>\n",
              "      <td>c111_res</td>\n",
              "      <td>...</td>\n",
              "      <td>12</td>\n",
              "      <td>1</td>\n",
              "      <td>2</td>\n",
              "      <td>12</td>\n",
              "      <td>codg2014.npy</td>\n",
              "      <td>codg2015.npy</td>\n",
              "      <td>1</td>\n",
              "      <td>mae</td>\n",
              "      <td>adam</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>35</th>\n",
              "      <td>deeptec1-14-15</td>\n",
              "      <td>REF-ConvLSTMv1_3x3_14-15_12</td>\n",
              "      <td>True</td>\n",
              "      <td>True</td>\n",
              "      <td>True</td>\n",
              "      <td>True</td>\n",
              "      <td>4</td>\n",
              "      <td>200</td>\n",
              "      <td>24</td>\n",
              "      <td>c333_res</td>\n",
              "      <td>...</td>\n",
              "      <td>12</td>\n",
              "      <td>1</td>\n",
              "      <td>2</td>\n",
              "      <td>12</td>\n",
              "      <td>codg2014.npy</td>\n",
              "      <td>codg2015.npy</td>\n",
              "      <td>1</td>\n",
              "      <td>mae</td>\n",
              "      <td>adam</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>36</th>\n",
              "      <td>deeptec1-14-15</td>\n",
              "      <td>REF-ConvLSTMv1_3x3_14-15_24</td>\n",
              "      <td>True</td>\n",
              "      <td>True</td>\n",
              "      <td>True</td>\n",
              "      <td>True</td>\n",
              "      <td>4</td>\n",
              "      <td>200</td>\n",
              "      <td>24</td>\n",
              "      <td>c333_res</td>\n",
              "      <td>...</td>\n",
              "      <td>12</td>\n",
              "      <td>1</td>\n",
              "      <td>2</td>\n",
              "      <td>12</td>\n",
              "      <td>codg2014.npy</td>\n",
              "      <td>codg2015.npy</td>\n",
              "      <td>1</td>\n",
              "      <td>mae</td>\n",
              "      <td>adam</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>37</th>\n",
              "      <td>deeptec1-14-15</td>\n",
              "      <td>REF-ConvLSTMv1_3x3_14-15</td>\n",
              "      <td>True</td>\n",
              "      <td>True</td>\n",
              "      <td>True</td>\n",
              "      <td>True</td>\n",
              "      <td>4</td>\n",
              "      <td>200</td>\n",
              "      <td>24</td>\n",
              "      <td>c333_res</td>\n",
              "      <td>...</td>\n",
              "      <td>12</td>\n",
              "      <td>1</td>\n",
              "      <td>2</td>\n",
              "      <td>12</td>\n",
              "      <td>codg2014.npy</td>\n",
              "      <td>codg2015.npy</td>\n",
              "      <td>1</td>\n",
              "      <td>mae</td>\n",
              "      <td>adam</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>38</th>\n",
              "      <td>deeptec1-14-15</td>\n",
              "      <td>REF-ConvLSTMv1_3x3_14-15_48</td>\n",
              "      <td>True</td>\n",
              "      <td>True</td>\n",
              "      <td>True</td>\n",
              "      <td>True</td>\n",
              "      <td>4</td>\n",
              "      <td>200</td>\n",
              "      <td>24</td>\n",
              "      <td>c333_res</td>\n",
              "      <td>...</td>\n",
              "      <td>12</td>\n",
              "      <td>1</td>\n",
              "      <td>2</td>\n",
              "      <td>12</td>\n",
              "      <td>codg2014.npy</td>\n",
              "      <td>codg2015.npy</td>\n",
              "      <td>1</td>\n",
              "      <td>mae</td>\n",
              "      <td>adam</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>39 rows × 22 columns</p>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-cd7394a5-19af-4eee-ab88-aea09466b297')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-cd7394a5-19af-4eee-ab88-aea09466b297 button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-cd7394a5-19af-4eee-ab88-aea09466b297');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ]
          },
          "metadata": {},
          "execution_count": 14
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Changing the configuration with Pandas\n",
        "We set the number of experiments on column \"best_of\" to 1 and turn off every experiment except \"ANN_NtoN_48_2\", which is a very lightweight neural network."
      ],
      "metadata": {
        "id": "b3IZV4B10wun"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "#changing\n",
        "df['batch_train']='False'\n",
        "df['batch_test']='False'\n",
        "df['best_of']=1\n",
        "df.loc[df['experiment_name']=='ANN_NtoN_36_2', 'batch_train'] = 'True'\n",
        "df.loc[df['experiment_name']=='ANN_NtoN_36_2', 'batch_test'] = 'True'\n",
        "#saving\n",
        "df.to_csv('/content/tec_forecast/config.csv', index=False)\n",
        "#showing\n",
        "df"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "id": "cZZEcMvO1hHN",
        "outputId": "55bb4891-810b-421f-84f7-d7603c404e37"
      },
      "execution_count": 18,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "                        comment              experiment_name batch_train  \\\n",
              "0                          None                 Repeat_19-20       False   \n",
              "1                          None                 Repeat_14-15       False   \n",
              "2           This is not a model                         c1pg       False   \n",
              "3           This is not a model                    c1pg14-15       False   \n",
              "4   Very low memory consumption                ANN_NtoN_12_2       False   \n",
              "5   Very low memory consumption                ANN_NtoN_18_2       False   \n",
              "6   Very low memory consumption                ANN_NtoN_24_2       False   \n",
              "7   Very low memory consumption                ANN_NtoN_30_2       False   \n",
              "8   Very low memory consumption                ANN_NtoN_36_2        True   \n",
              "9                deeptec1-19-20                    iConvLSTM       False   \n",
              "10                          NaN              EF-ConvLSTM_1x1       False   \n",
              "11               deeptec1-19-20              EF-ConvLSTM_3x3       False   \n",
              "12                          NaN           REF-ConvLSTMv1_1x1       False   \n",
              "13               deeptec1-19-20           REF-ConvLSTMv1_3x3       False   \n",
              "14                          NaN            EF-ConvLSTMv2_1x1       False   \n",
              "15               deeptec1-19-20            EF-ConvLSTMv2_3x3       False   \n",
              "16               deeptec1-19-20                EF-ConvLSTMv3       False   \n",
              "17                         cbcg                  ConvLSTM_Bi       False   \n",
              "18                         cbcg                 ConvLSTM_N-1       False   \n",
              "19                         cbcg               EF_ConvLSTM_60       False   \n",
              "20                         cbcg               EF_ConvLSTM_48       False   \n",
              "21                         cbcg               EF_ConvLSTM_36       False   \n",
              "22                         cbcg               EF_ConvLSTM_24       False   \n",
              "23                         cbcg               EF_ConvLSTM_12       False   \n",
              "24                     deeptec1              IconvLSTM_14-15       False   \n",
              "25                          NaN        EF-ConvLSTM_1x1_14-15       False   \n",
              "26               deeptec1-14-15        EF-ConvLSTM_3x3_14-15       False   \n",
              "27                          NaN      EF-ConvLSTMv2_1x1_14-15       False   \n",
              "28              deeptec1-14-15    EF-ConvLSTMv2_3x3_14-15_12       False   \n",
              "29               deeptec1-14-15   EF-ConvLSTMv2_3x3_14-15_24       False   \n",
              "30               deeptec1-14-15      EF-ConvLSTMv2_3x3_14-15       False   \n",
              "31               deeptec1-14-15   EF-ConvLSTMv2_3x3_14-15_48       False   \n",
              "32                Out of memory   EF-ConvLSTMv2_3x3_14-15_60       False   \n",
              "33                     deeptec1          EF-ConvLSTMv3_14-15       False   \n",
              "34                          NaN     REF-ConvLSTMv1_1x1_14-15       False   \n",
              "35               deeptec1-14-15  REF-ConvLSTMv1_3x3_14-15_12       False   \n",
              "36               deeptec1-14-15  REF-ConvLSTMv1_3x3_14-15_24       False   \n",
              "37               deeptec1-14-15     REF-ConvLSTMv1_3x3_14-15       False   \n",
              "38               deeptec1-14-15  REF-ConvLSTMv1_3x3_14-15_48       False   \n",
              "\n",
              "   batch_test  tested  compare  batch_size  num_epochs  filters        model  \\\n",
              "0       False    True     True           4          30        8  usePrevious   \n",
              "1       False    True     True           4          30        8  usePrevious   \n",
              "2       False    True     True           1           1        0         c1pg   \n",
              "3       False    True     True           1           1        0         c1pg   \n",
              "4       False    True     True          32         200      150          ANN   \n",
              "5       False    True     True          32         200      150          ANN   \n",
              "6       False    True     True          32         200      150          ANN   \n",
              "7       False    True     True          32         200      150          ANN   \n",
              "8        True    True     True          32         200      150          ANN   \n",
              "9       False    True     True          16         200       48    iConvLSTM   \n",
              "10      False    True     True           4         200       24         c111   \n",
              "11      False    True     True           8         200       24         c333   \n",
              "12      False    True     True           4         200       24     c111_res   \n",
              "13      False    True     True           4         200       24     c333_res   \n",
              "14      False    True     True           4         200       24  c111_res_v2   \n",
              "15      False    True     True           8         200       24  c333_res_v2   \n",
              "16      False    True     True           8         200       24  c333_res_v3   \n",
              "17      False    True     True           8         200       24       c333bi   \n",
              "18      False    True     True           8         200       24    c333_nto1   \n",
              "19      False    True     True           4         200       24         c333   \n",
              "20      False    True     True           4         200       24         c333   \n",
              "21      False    True     True           4         200       24         c333   \n",
              "22      False    True     True           4         200       24         c333   \n",
              "23      False    True     True           4         200       24         c333   \n",
              "24      False    True     True          16         200       48    iConvLSTM   \n",
              "25      False    True     True           4         200       24         c111   \n",
              "26      False    True     True           4         200       24         c333   \n",
              "27      False    True     True           4         200       24  c111_res_v2   \n",
              "28      False    True     True           8         200       24  c333_res_v2   \n",
              "29      False    True     True           8         200       24  c333_res_v2   \n",
              "30      False    True     True           8         200       24  c333_res_v2   \n",
              "31      False    True     True           8         200       24  c333_res_v2   \n",
              "32      False   False     True           8         200       24  c333_res_v2   \n",
              "33      False    True     True           8         200       24  c333_res_v3   \n",
              "34      False    True     True           4         200       24     c111_res   \n",
              "35      False    True     True           4         200       24     c333_res   \n",
              "36      False    True     True           4         200       24     c333_res   \n",
              "37      False    True     True           4         200       24     c333_res   \n",
              "38      False    True     True           4         200       24     c333_res   \n",
              "\n",
              "    ... prediction_window  train_time_sampling  test_time_sampling  \\\n",
              "0   ...                12                    2                   2   \n",
              "1   ...                12                    1                   2   \n",
              "2   ...                12                    2                   2   \n",
              "3   ...                12                    2                   2   \n",
              "4   ...                12                    2                   2   \n",
              "5   ...                12                    2                   2   \n",
              "6   ...                12                    2                   2   \n",
              "7   ...                12                    2                   2   \n",
              "8   ...                12                    2                   2   \n",
              "9   ...                12                    2                   2   \n",
              "10  ...                12                    2                   2   \n",
              "11  ...                12                    2                   2   \n",
              "12  ...                12                    2                   2   \n",
              "13  ...                12                    2                   2   \n",
              "14  ...                12                    2                   2   \n",
              "15  ...                12                    2                   2   \n",
              "16  ...                12                    2                   2   \n",
              "17  ...                12                    2                   2   \n",
              "18  ...                12                    2                   2   \n",
              "19  ...                12                    2                   2   \n",
              "20  ...                12                    2                   2   \n",
              "21  ...                12                    2                   2   \n",
              "22  ...                12                    2                   2   \n",
              "23  ...                12                    2                   2   \n",
              "24  ...                12                    1                   2   \n",
              "25  ...                12                    1                   2   \n",
              "26  ...                12                    1                   2   \n",
              "27  ...                12                    1                   2   \n",
              "28  ...                12                    1                   2   \n",
              "29  ...                12                    1                   2   \n",
              "30  ...                12                    1                   2   \n",
              "31  ...                12                    1                   2   \n",
              "32  ...                12                    1                   2   \n",
              "33  ...                12                    1                   2   \n",
              "34  ...                12                    1                   2   \n",
              "35  ...                12                    1                   2   \n",
              "36  ...                12                    1                   2   \n",
              "37  ...                12                    1                   2   \n",
              "38  ...                12                    1                   2   \n",
              "\n",
              "    resample_rate  train_npy_dataset  test_npy_dataset random_seed loss  \\\n",
              "0              12       codg2019.npy      codg2020.npy           1  mae   \n",
              "1              12       codg2014.npy      codg2015.npy           1  mae   \n",
              "2              12       codg2019.npy      codg2020.npy           1  mae   \n",
              "3              12       c1pg2015.npy      codg2015.npy           1  mae   \n",
              "4              12       codg2019.npy      codg2020.npy           1  mae   \n",
              "5              12       codg2019.npy      codg2020.npy           1  mae   \n",
              "6              12       codg2019.npy      codg2020.npy           1  mae   \n",
              "7              12       codg2019.npy      codg2020.npy           1  mae   \n",
              "8              12       codg2019.npy      codg2020.npy           1  mae   \n",
              "9              12       codg2019.npy      codg2020.npy           1  mae   \n",
              "10             12       codg2019.npy      codg2020.npy           1  mae   \n",
              "11             12       codg2019.npy      codg2020.npy           1  mae   \n",
              "12             12       codg2019.npy      codg2020.npy           1  mae   \n",
              "13             12       codg2019.npy      codg2020.npy           1  mae   \n",
              "14             12       codg2019.npy      codg2020.npy           1  mae   \n",
              "15             12       codg2019.npy      codg2020.npy           1  mae   \n",
              "16             12       codg2019.npy      codg2020.npy           1  mae   \n",
              "17             12       codg2019.npy      codg2020.npy           1  mae   \n",
              "18             12       codg2019.npy      codg2020.npy           1  mae   \n",
              "19             12       codg2019.npy      codg2020.npy           1  mae   \n",
              "20             12       codg2019.npy      codg2020.npy           1  mae   \n",
              "21             12       codg2019.npy      codg2020.npy           1  mae   \n",
              "22             12       codg2019.npy      codg2020.npy           1  mae   \n",
              "23             12       codg2019.npy      codg2020.npy           1  mae   \n",
              "24             12       codg2014.npy      codg2015.npy           1  mae   \n",
              "25             12       codg2014.npy      codg2015.npy           1  mae   \n",
              "26             12       codg2014.npy      codg2015.npy           1  mae   \n",
              "27             12       codg2014.npy      codg2015.npy           1  mae   \n",
              "28             12       codg2014.npy      codg2015.npy           1  mae   \n",
              "29             12       codg2014.npy      codg2015.npy           1  mae   \n",
              "30             12       codg2014.npy      codg2015.npy           1  mae   \n",
              "31             12       codg2014.npy      codg2015.npy           1  mae   \n",
              "32             12       codg2014.npy      codg2015.npy           1  mae   \n",
              "33             12       codg2014.npy      codg2015.npy           1  mae   \n",
              "34             12       codg2014.npy      codg2015.npy           1  mae   \n",
              "35             12       codg2014.npy      codg2015.npy           1  mae   \n",
              "36             12       codg2014.npy      codg2015.npy           1  mae   \n",
              "37             12       codg2014.npy      codg2015.npy           1  mae   \n",
              "38             12       codg2014.npy      codg2015.npy           1  mae   \n",
              "\n",
              "    optimizer best_of  \n",
              "0        adam       1  \n",
              "1        adam       1  \n",
              "2        adam       1  \n",
              "3        adam       1  \n",
              "4        adam       1  \n",
              "5        adam       1  \n",
              "6        adam       1  \n",
              "7        adam       1  \n",
              "8        adam       1  \n",
              "9        adam       1  \n",
              "10       adam       1  \n",
              "11       adam       1  \n",
              "12       adam       1  \n",
              "13       adam       1  \n",
              "14       adam       1  \n",
              "15       adam       1  \n",
              "16       adam       1  \n",
              "17       adam       1  \n",
              "18       adam       1  \n",
              "19       adam       1  \n",
              "20       adam       1  \n",
              "21       adam       1  \n",
              "22       adam       1  \n",
              "23       adam       1  \n",
              "24       adam       1  \n",
              "25       adam       1  \n",
              "26       adam       1  \n",
              "27       adam       1  \n",
              "28       adam       1  \n",
              "29       adam       1  \n",
              "30       adam       1  \n",
              "31       adam       1  \n",
              "32       adam       1  \n",
              "33       adam       1  \n",
              "34       adam       1  \n",
              "35       adam       1  \n",
              "36       adam       1  \n",
              "37       adam       1  \n",
              "38       adam       1  \n",
              "\n",
              "[39 rows x 22 columns]"
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-9523e0f2-353e-47c5-a97d-1b22336b9935\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>comment</th>\n",
              "      <th>experiment_name</th>\n",
              "      <th>batch_train</th>\n",
              "      <th>batch_test</th>\n",
              "      <th>tested</th>\n",
              "      <th>compare</th>\n",
              "      <th>batch_size</th>\n",
              "      <th>num_epochs</th>\n",
              "      <th>filters</th>\n",
              "      <th>model</th>\n",
              "      <th>...</th>\n",
              "      <th>prediction_window</th>\n",
              "      <th>train_time_sampling</th>\n",
              "      <th>test_time_sampling</th>\n",
              "      <th>resample_rate</th>\n",
              "      <th>train_npy_dataset</th>\n",
              "      <th>test_npy_dataset</th>\n",
              "      <th>random_seed</th>\n",
              "      <th>loss</th>\n",
              "      <th>optimizer</th>\n",
              "      <th>best_of</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>None</td>\n",
              "      <td>Repeat_19-20</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>True</td>\n",
              "      <td>True</td>\n",
              "      <td>4</td>\n",
              "      <td>30</td>\n",
              "      <td>8</td>\n",
              "      <td>usePrevious</td>\n",
              "      <td>...</td>\n",
              "      <td>12</td>\n",
              "      <td>2</td>\n",
              "      <td>2</td>\n",
              "      <td>12</td>\n",
              "      <td>codg2019.npy</td>\n",
              "      <td>codg2020.npy</td>\n",
              "      <td>1</td>\n",
              "      <td>mae</td>\n",
              "      <td>adam</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>None</td>\n",
              "      <td>Repeat_14-15</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>True</td>\n",
              "      <td>True</td>\n",
              "      <td>4</td>\n",
              "      <td>30</td>\n",
              "      <td>8</td>\n",
              "      <td>usePrevious</td>\n",
              "      <td>...</td>\n",
              "      <td>12</td>\n",
              "      <td>1</td>\n",
              "      <td>2</td>\n",
              "      <td>12</td>\n",
              "      <td>codg2014.npy</td>\n",
              "      <td>codg2015.npy</td>\n",
              "      <td>1</td>\n",
              "      <td>mae</td>\n",
              "      <td>adam</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>This is not a model</td>\n",
              "      <td>c1pg</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>True</td>\n",
              "      <td>True</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>c1pg</td>\n",
              "      <td>...</td>\n",
              "      <td>12</td>\n",
              "      <td>2</td>\n",
              "      <td>2</td>\n",
              "      <td>12</td>\n",
              "      <td>codg2019.npy</td>\n",
              "      <td>codg2020.npy</td>\n",
              "      <td>1</td>\n",
              "      <td>mae</td>\n",
              "      <td>adam</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>This is not a model</td>\n",
              "      <td>c1pg14-15</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>True</td>\n",
              "      <td>True</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>c1pg</td>\n",
              "      <td>...</td>\n",
              "      <td>12</td>\n",
              "      <td>2</td>\n",
              "      <td>2</td>\n",
              "      <td>12</td>\n",
              "      <td>c1pg2015.npy</td>\n",
              "      <td>codg2015.npy</td>\n",
              "      <td>1</td>\n",
              "      <td>mae</td>\n",
              "      <td>adam</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>Very low memory consumption</td>\n",
              "      <td>ANN_NtoN_12_2</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>True</td>\n",
              "      <td>True</td>\n",
              "      <td>32</td>\n",
              "      <td>200</td>\n",
              "      <td>150</td>\n",
              "      <td>ANN</td>\n",
              "      <td>...</td>\n",
              "      <td>12</td>\n",
              "      <td>2</td>\n",
              "      <td>2</td>\n",
              "      <td>12</td>\n",
              "      <td>codg2019.npy</td>\n",
              "      <td>codg2020.npy</td>\n",
              "      <td>1</td>\n",
              "      <td>mae</td>\n",
              "      <td>adam</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5</th>\n",
              "      <td>Very low memory consumption</td>\n",
              "      <td>ANN_NtoN_18_2</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>True</td>\n",
              "      <td>True</td>\n",
              "      <td>32</td>\n",
              "      <td>200</td>\n",
              "      <td>150</td>\n",
              "      <td>ANN</td>\n",
              "      <td>...</td>\n",
              "      <td>12</td>\n",
              "      <td>2</td>\n",
              "      <td>2</td>\n",
              "      <td>12</td>\n",
              "      <td>codg2019.npy</td>\n",
              "      <td>codg2020.npy</td>\n",
              "      <td>1</td>\n",
              "      <td>mae</td>\n",
              "      <td>adam</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>6</th>\n",
              "      <td>Very low memory consumption</td>\n",
              "      <td>ANN_NtoN_24_2</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>True</td>\n",
              "      <td>True</td>\n",
              "      <td>32</td>\n",
              "      <td>200</td>\n",
              "      <td>150</td>\n",
              "      <td>ANN</td>\n",
              "      <td>...</td>\n",
              "      <td>12</td>\n",
              "      <td>2</td>\n",
              "      <td>2</td>\n",
              "      <td>12</td>\n",
              "      <td>codg2019.npy</td>\n",
              "      <td>codg2020.npy</td>\n",
              "      <td>1</td>\n",
              "      <td>mae</td>\n",
              "      <td>adam</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>7</th>\n",
              "      <td>Very low memory consumption</td>\n",
              "      <td>ANN_NtoN_30_2</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>True</td>\n",
              "      <td>True</td>\n",
              "      <td>32</td>\n",
              "      <td>200</td>\n",
              "      <td>150</td>\n",
              "      <td>ANN</td>\n",
              "      <td>...</td>\n",
              "      <td>12</td>\n",
              "      <td>2</td>\n",
              "      <td>2</td>\n",
              "      <td>12</td>\n",
              "      <td>codg2019.npy</td>\n",
              "      <td>codg2020.npy</td>\n",
              "      <td>1</td>\n",
              "      <td>mae</td>\n",
              "      <td>adam</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>8</th>\n",
              "      <td>Very low memory consumption</td>\n",
              "      <td>ANN_NtoN_36_2</td>\n",
              "      <td>True</td>\n",
              "      <td>True</td>\n",
              "      <td>True</td>\n",
              "      <td>True</td>\n",
              "      <td>32</td>\n",
              "      <td>200</td>\n",
              "      <td>150</td>\n",
              "      <td>ANN</td>\n",
              "      <td>...</td>\n",
              "      <td>12</td>\n",
              "      <td>2</td>\n",
              "      <td>2</td>\n",
              "      <td>12</td>\n",
              "      <td>codg2019.npy</td>\n",
              "      <td>codg2020.npy</td>\n",
              "      <td>1</td>\n",
              "      <td>mae</td>\n",
              "      <td>adam</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9</th>\n",
              "      <td>deeptec1-19-20</td>\n",
              "      <td>iConvLSTM</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>True</td>\n",
              "      <td>True</td>\n",
              "      <td>16</td>\n",
              "      <td>200</td>\n",
              "      <td>48</td>\n",
              "      <td>iConvLSTM</td>\n",
              "      <td>...</td>\n",
              "      <td>12</td>\n",
              "      <td>2</td>\n",
              "      <td>2</td>\n",
              "      <td>12</td>\n",
              "      <td>codg2019.npy</td>\n",
              "      <td>codg2020.npy</td>\n",
              "      <td>1</td>\n",
              "      <td>mae</td>\n",
              "      <td>adam</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>10</th>\n",
              "      <td>NaN</td>\n",
              "      <td>EF-ConvLSTM_1x1</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>True</td>\n",
              "      <td>True</td>\n",
              "      <td>4</td>\n",
              "      <td>200</td>\n",
              "      <td>24</td>\n",
              "      <td>c111</td>\n",
              "      <td>...</td>\n",
              "      <td>12</td>\n",
              "      <td>2</td>\n",
              "      <td>2</td>\n",
              "      <td>12</td>\n",
              "      <td>codg2019.npy</td>\n",
              "      <td>codg2020.npy</td>\n",
              "      <td>1</td>\n",
              "      <td>mae</td>\n",
              "      <td>adam</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>11</th>\n",
              "      <td>deeptec1-19-20</td>\n",
              "      <td>EF-ConvLSTM_3x3</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>True</td>\n",
              "      <td>True</td>\n",
              "      <td>8</td>\n",
              "      <td>200</td>\n",
              "      <td>24</td>\n",
              "      <td>c333</td>\n",
              "      <td>...</td>\n",
              "      <td>12</td>\n",
              "      <td>2</td>\n",
              "      <td>2</td>\n",
              "      <td>12</td>\n",
              "      <td>codg2019.npy</td>\n",
              "      <td>codg2020.npy</td>\n",
              "      <td>1</td>\n",
              "      <td>mae</td>\n",
              "      <td>adam</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>12</th>\n",
              "      <td>NaN</td>\n",
              "      <td>REF-ConvLSTMv1_1x1</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>True</td>\n",
              "      <td>True</td>\n",
              "      <td>4</td>\n",
              "      <td>200</td>\n",
              "      <td>24</td>\n",
              "      <td>c111_res</td>\n",
              "      <td>...</td>\n",
              "      <td>12</td>\n",
              "      <td>2</td>\n",
              "      <td>2</td>\n",
              "      <td>12</td>\n",
              "      <td>codg2019.npy</td>\n",
              "      <td>codg2020.npy</td>\n",
              "      <td>1</td>\n",
              "      <td>mae</td>\n",
              "      <td>adam</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>13</th>\n",
              "      <td>deeptec1-19-20</td>\n",
              "      <td>REF-ConvLSTMv1_3x3</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>True</td>\n",
              "      <td>True</td>\n",
              "      <td>4</td>\n",
              "      <td>200</td>\n",
              "      <td>24</td>\n",
              "      <td>c333_res</td>\n",
              "      <td>...</td>\n",
              "      <td>12</td>\n",
              "      <td>2</td>\n",
              "      <td>2</td>\n",
              "      <td>12</td>\n",
              "      <td>codg2019.npy</td>\n",
              "      <td>codg2020.npy</td>\n",
              "      <td>1</td>\n",
              "      <td>mae</td>\n",
              "      <td>adam</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>14</th>\n",
              "      <td>NaN</td>\n",
              "      <td>EF-ConvLSTMv2_1x1</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>True</td>\n",
              "      <td>True</td>\n",
              "      <td>4</td>\n",
              "      <td>200</td>\n",
              "      <td>24</td>\n",
              "      <td>c111_res_v2</td>\n",
              "      <td>...</td>\n",
              "      <td>12</td>\n",
              "      <td>2</td>\n",
              "      <td>2</td>\n",
              "      <td>12</td>\n",
              "      <td>codg2019.npy</td>\n",
              "      <td>codg2020.npy</td>\n",
              "      <td>1</td>\n",
              "      <td>mae</td>\n",
              "      <td>adam</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>15</th>\n",
              "      <td>deeptec1-19-20</td>\n",
              "      <td>EF-ConvLSTMv2_3x3</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>True</td>\n",
              "      <td>True</td>\n",
              "      <td>8</td>\n",
              "      <td>200</td>\n",
              "      <td>24</td>\n",
              "      <td>c333_res_v2</td>\n",
              "      <td>...</td>\n",
              "      <td>12</td>\n",
              "      <td>2</td>\n",
              "      <td>2</td>\n",
              "      <td>12</td>\n",
              "      <td>codg2019.npy</td>\n",
              "      <td>codg2020.npy</td>\n",
              "      <td>1</td>\n",
              "      <td>mae</td>\n",
              "      <td>adam</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>16</th>\n",
              "      <td>deeptec1-19-20</td>\n",
              "      <td>EF-ConvLSTMv3</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>True</td>\n",
              "      <td>True</td>\n",
              "      <td>8</td>\n",
              "      <td>200</td>\n",
              "      <td>24</td>\n",
              "      <td>c333_res_v3</td>\n",
              "      <td>...</td>\n",
              "      <td>12</td>\n",
              "      <td>2</td>\n",
              "      <td>2</td>\n",
              "      <td>12</td>\n",
              "      <td>codg2019.npy</td>\n",
              "      <td>codg2020.npy</td>\n",
              "      <td>1</td>\n",
              "      <td>mae</td>\n",
              "      <td>adam</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>17</th>\n",
              "      <td>cbcg</td>\n",
              "      <td>ConvLSTM_Bi</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>True</td>\n",
              "      <td>True</td>\n",
              "      <td>8</td>\n",
              "      <td>200</td>\n",
              "      <td>24</td>\n",
              "      <td>c333bi</td>\n",
              "      <td>...</td>\n",
              "      <td>12</td>\n",
              "      <td>2</td>\n",
              "      <td>2</td>\n",
              "      <td>12</td>\n",
              "      <td>codg2019.npy</td>\n",
              "      <td>codg2020.npy</td>\n",
              "      <td>1</td>\n",
              "      <td>mae</td>\n",
              "      <td>adam</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>18</th>\n",
              "      <td>cbcg</td>\n",
              "      <td>ConvLSTM_N-1</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>True</td>\n",
              "      <td>True</td>\n",
              "      <td>8</td>\n",
              "      <td>200</td>\n",
              "      <td>24</td>\n",
              "      <td>c333_nto1</td>\n",
              "      <td>...</td>\n",
              "      <td>12</td>\n",
              "      <td>2</td>\n",
              "      <td>2</td>\n",
              "      <td>12</td>\n",
              "      <td>codg2019.npy</td>\n",
              "      <td>codg2020.npy</td>\n",
              "      <td>1</td>\n",
              "      <td>mae</td>\n",
              "      <td>adam</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>19</th>\n",
              "      <td>cbcg</td>\n",
              "      <td>EF_ConvLSTM_60</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>True</td>\n",
              "      <td>True</td>\n",
              "      <td>4</td>\n",
              "      <td>200</td>\n",
              "      <td>24</td>\n",
              "      <td>c333</td>\n",
              "      <td>...</td>\n",
              "      <td>12</td>\n",
              "      <td>2</td>\n",
              "      <td>2</td>\n",
              "      <td>12</td>\n",
              "      <td>codg2019.npy</td>\n",
              "      <td>codg2020.npy</td>\n",
              "      <td>1</td>\n",
              "      <td>mae</td>\n",
              "      <td>adam</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>20</th>\n",
              "      <td>cbcg</td>\n",
              "      <td>EF_ConvLSTM_48</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>True</td>\n",
              "      <td>True</td>\n",
              "      <td>4</td>\n",
              "      <td>200</td>\n",
              "      <td>24</td>\n",
              "      <td>c333</td>\n",
              "      <td>...</td>\n",
              "      <td>12</td>\n",
              "      <td>2</td>\n",
              "      <td>2</td>\n",
              "      <td>12</td>\n",
              "      <td>codg2019.npy</td>\n",
              "      <td>codg2020.npy</td>\n",
              "      <td>1</td>\n",
              "      <td>mae</td>\n",
              "      <td>adam</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>21</th>\n",
              "      <td>cbcg</td>\n",
              "      <td>EF_ConvLSTM_36</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>True</td>\n",
              "      <td>True</td>\n",
              "      <td>4</td>\n",
              "      <td>200</td>\n",
              "      <td>24</td>\n",
              "      <td>c333</td>\n",
              "      <td>...</td>\n",
              "      <td>12</td>\n",
              "      <td>2</td>\n",
              "      <td>2</td>\n",
              "      <td>12</td>\n",
              "      <td>codg2019.npy</td>\n",
              "      <td>codg2020.npy</td>\n",
              "      <td>1</td>\n",
              "      <td>mae</td>\n",
              "      <td>adam</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>22</th>\n",
              "      <td>cbcg</td>\n",
              "      <td>EF_ConvLSTM_24</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>True</td>\n",
              "      <td>True</td>\n",
              "      <td>4</td>\n",
              "      <td>200</td>\n",
              "      <td>24</td>\n",
              "      <td>c333</td>\n",
              "      <td>...</td>\n",
              "      <td>12</td>\n",
              "      <td>2</td>\n",
              "      <td>2</td>\n",
              "      <td>12</td>\n",
              "      <td>codg2019.npy</td>\n",
              "      <td>codg2020.npy</td>\n",
              "      <td>1</td>\n",
              "      <td>mae</td>\n",
              "      <td>adam</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>23</th>\n",
              "      <td>cbcg</td>\n",
              "      <td>EF_ConvLSTM_12</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>True</td>\n",
              "      <td>True</td>\n",
              "      <td>4</td>\n",
              "      <td>200</td>\n",
              "      <td>24</td>\n",
              "      <td>c333</td>\n",
              "      <td>...</td>\n",
              "      <td>12</td>\n",
              "      <td>2</td>\n",
              "      <td>2</td>\n",
              "      <td>12</td>\n",
              "      <td>codg2019.npy</td>\n",
              "      <td>codg2020.npy</td>\n",
              "      <td>1</td>\n",
              "      <td>mae</td>\n",
              "      <td>adam</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>24</th>\n",
              "      <td>deeptec1</td>\n",
              "      <td>IconvLSTM_14-15</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>True</td>\n",
              "      <td>True</td>\n",
              "      <td>16</td>\n",
              "      <td>200</td>\n",
              "      <td>48</td>\n",
              "      <td>iConvLSTM</td>\n",
              "      <td>...</td>\n",
              "      <td>12</td>\n",
              "      <td>1</td>\n",
              "      <td>2</td>\n",
              "      <td>12</td>\n",
              "      <td>codg2014.npy</td>\n",
              "      <td>codg2015.npy</td>\n",
              "      <td>1</td>\n",
              "      <td>mae</td>\n",
              "      <td>adam</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>25</th>\n",
              "      <td>NaN</td>\n",
              "      <td>EF-ConvLSTM_1x1_14-15</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>True</td>\n",
              "      <td>True</td>\n",
              "      <td>4</td>\n",
              "      <td>200</td>\n",
              "      <td>24</td>\n",
              "      <td>c111</td>\n",
              "      <td>...</td>\n",
              "      <td>12</td>\n",
              "      <td>1</td>\n",
              "      <td>2</td>\n",
              "      <td>12</td>\n",
              "      <td>codg2014.npy</td>\n",
              "      <td>codg2015.npy</td>\n",
              "      <td>1</td>\n",
              "      <td>mae</td>\n",
              "      <td>adam</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>26</th>\n",
              "      <td>deeptec1-14-15</td>\n",
              "      <td>EF-ConvLSTM_3x3_14-15</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>True</td>\n",
              "      <td>True</td>\n",
              "      <td>4</td>\n",
              "      <td>200</td>\n",
              "      <td>24</td>\n",
              "      <td>c333</td>\n",
              "      <td>...</td>\n",
              "      <td>12</td>\n",
              "      <td>1</td>\n",
              "      <td>2</td>\n",
              "      <td>12</td>\n",
              "      <td>codg2014.npy</td>\n",
              "      <td>codg2015.npy</td>\n",
              "      <td>1</td>\n",
              "      <td>mae</td>\n",
              "      <td>adam</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>27</th>\n",
              "      <td>NaN</td>\n",
              "      <td>EF-ConvLSTMv2_1x1_14-15</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>True</td>\n",
              "      <td>True</td>\n",
              "      <td>4</td>\n",
              "      <td>200</td>\n",
              "      <td>24</td>\n",
              "      <td>c111_res_v2</td>\n",
              "      <td>...</td>\n",
              "      <td>12</td>\n",
              "      <td>1</td>\n",
              "      <td>2</td>\n",
              "      <td>12</td>\n",
              "      <td>codg2014.npy</td>\n",
              "      <td>codg2015.npy</td>\n",
              "      <td>1</td>\n",
              "      <td>mae</td>\n",
              "      <td>adam</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>28</th>\n",
              "      <td>deeptec1-14-15</td>\n",
              "      <td>EF-ConvLSTMv2_3x3_14-15_12</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>True</td>\n",
              "      <td>True</td>\n",
              "      <td>8</td>\n",
              "      <td>200</td>\n",
              "      <td>24</td>\n",
              "      <td>c333_res_v2</td>\n",
              "      <td>...</td>\n",
              "      <td>12</td>\n",
              "      <td>1</td>\n",
              "      <td>2</td>\n",
              "      <td>12</td>\n",
              "      <td>codg2014.npy</td>\n",
              "      <td>codg2015.npy</td>\n",
              "      <td>1</td>\n",
              "      <td>mae</td>\n",
              "      <td>adam</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>29</th>\n",
              "      <td>deeptec1-14-15</td>\n",
              "      <td>EF-ConvLSTMv2_3x3_14-15_24</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>True</td>\n",
              "      <td>True</td>\n",
              "      <td>8</td>\n",
              "      <td>200</td>\n",
              "      <td>24</td>\n",
              "      <td>c333_res_v2</td>\n",
              "      <td>...</td>\n",
              "      <td>12</td>\n",
              "      <td>1</td>\n",
              "      <td>2</td>\n",
              "      <td>12</td>\n",
              "      <td>codg2014.npy</td>\n",
              "      <td>codg2015.npy</td>\n",
              "      <td>1</td>\n",
              "      <td>mae</td>\n",
              "      <td>adam</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>30</th>\n",
              "      <td>deeptec1-14-15</td>\n",
              "      <td>EF-ConvLSTMv2_3x3_14-15</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>True</td>\n",
              "      <td>True</td>\n",
              "      <td>8</td>\n",
              "      <td>200</td>\n",
              "      <td>24</td>\n",
              "      <td>c333_res_v2</td>\n",
              "      <td>...</td>\n",
              "      <td>12</td>\n",
              "      <td>1</td>\n",
              "      <td>2</td>\n",
              "      <td>12</td>\n",
              "      <td>codg2014.npy</td>\n",
              "      <td>codg2015.npy</td>\n",
              "      <td>1</td>\n",
              "      <td>mae</td>\n",
              "      <td>adam</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>31</th>\n",
              "      <td>deeptec1-14-15</td>\n",
              "      <td>EF-ConvLSTMv2_3x3_14-15_48</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>True</td>\n",
              "      <td>True</td>\n",
              "      <td>8</td>\n",
              "      <td>200</td>\n",
              "      <td>24</td>\n",
              "      <td>c333_res_v2</td>\n",
              "      <td>...</td>\n",
              "      <td>12</td>\n",
              "      <td>1</td>\n",
              "      <td>2</td>\n",
              "      <td>12</td>\n",
              "      <td>codg2014.npy</td>\n",
              "      <td>codg2015.npy</td>\n",
              "      <td>1</td>\n",
              "      <td>mae</td>\n",
              "      <td>adam</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>32</th>\n",
              "      <td>Out of memory</td>\n",
              "      <td>EF-ConvLSTMv2_3x3_14-15_60</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>True</td>\n",
              "      <td>8</td>\n",
              "      <td>200</td>\n",
              "      <td>24</td>\n",
              "      <td>c333_res_v2</td>\n",
              "      <td>...</td>\n",
              "      <td>12</td>\n",
              "      <td>1</td>\n",
              "      <td>2</td>\n",
              "      <td>12</td>\n",
              "      <td>codg2014.npy</td>\n",
              "      <td>codg2015.npy</td>\n",
              "      <td>1</td>\n",
              "      <td>mae</td>\n",
              "      <td>adam</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>33</th>\n",
              "      <td>deeptec1</td>\n",
              "      <td>EF-ConvLSTMv3_14-15</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>True</td>\n",
              "      <td>True</td>\n",
              "      <td>8</td>\n",
              "      <td>200</td>\n",
              "      <td>24</td>\n",
              "      <td>c333_res_v3</td>\n",
              "      <td>...</td>\n",
              "      <td>12</td>\n",
              "      <td>1</td>\n",
              "      <td>2</td>\n",
              "      <td>12</td>\n",
              "      <td>codg2014.npy</td>\n",
              "      <td>codg2015.npy</td>\n",
              "      <td>1</td>\n",
              "      <td>mae</td>\n",
              "      <td>adam</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>34</th>\n",
              "      <td>NaN</td>\n",
              "      <td>REF-ConvLSTMv1_1x1_14-15</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>True</td>\n",
              "      <td>True</td>\n",
              "      <td>4</td>\n",
              "      <td>200</td>\n",
              "      <td>24</td>\n",
              "      <td>c111_res</td>\n",
              "      <td>...</td>\n",
              "      <td>12</td>\n",
              "      <td>1</td>\n",
              "      <td>2</td>\n",
              "      <td>12</td>\n",
              "      <td>codg2014.npy</td>\n",
              "      <td>codg2015.npy</td>\n",
              "      <td>1</td>\n",
              "      <td>mae</td>\n",
              "      <td>adam</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>35</th>\n",
              "      <td>deeptec1-14-15</td>\n",
              "      <td>REF-ConvLSTMv1_3x3_14-15_12</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>True</td>\n",
              "      <td>True</td>\n",
              "      <td>4</td>\n",
              "      <td>200</td>\n",
              "      <td>24</td>\n",
              "      <td>c333_res</td>\n",
              "      <td>...</td>\n",
              "      <td>12</td>\n",
              "      <td>1</td>\n",
              "      <td>2</td>\n",
              "      <td>12</td>\n",
              "      <td>codg2014.npy</td>\n",
              "      <td>codg2015.npy</td>\n",
              "      <td>1</td>\n",
              "      <td>mae</td>\n",
              "      <td>adam</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>36</th>\n",
              "      <td>deeptec1-14-15</td>\n",
              "      <td>REF-ConvLSTMv1_3x3_14-15_24</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>True</td>\n",
              "      <td>True</td>\n",
              "      <td>4</td>\n",
              "      <td>200</td>\n",
              "      <td>24</td>\n",
              "      <td>c333_res</td>\n",
              "      <td>...</td>\n",
              "      <td>12</td>\n",
              "      <td>1</td>\n",
              "      <td>2</td>\n",
              "      <td>12</td>\n",
              "      <td>codg2014.npy</td>\n",
              "      <td>codg2015.npy</td>\n",
              "      <td>1</td>\n",
              "      <td>mae</td>\n",
              "      <td>adam</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>37</th>\n",
              "      <td>deeptec1-14-15</td>\n",
              "      <td>REF-ConvLSTMv1_3x3_14-15</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>True</td>\n",
              "      <td>True</td>\n",
              "      <td>4</td>\n",
              "      <td>200</td>\n",
              "      <td>24</td>\n",
              "      <td>c333_res</td>\n",
              "      <td>...</td>\n",
              "      <td>12</td>\n",
              "      <td>1</td>\n",
              "      <td>2</td>\n",
              "      <td>12</td>\n",
              "      <td>codg2014.npy</td>\n",
              "      <td>codg2015.npy</td>\n",
              "      <td>1</td>\n",
              "      <td>mae</td>\n",
              "      <td>adam</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>38</th>\n",
              "      <td>deeptec1-14-15</td>\n",
              "      <td>REF-ConvLSTMv1_3x3_14-15_48</td>\n",
              "      <td>False</td>\n",
              "      <td>False</td>\n",
              "      <td>True</td>\n",
              "      <td>True</td>\n",
              "      <td>4</td>\n",
              "      <td>200</td>\n",
              "      <td>24</td>\n",
              "      <td>c333_res</td>\n",
              "      <td>...</td>\n",
              "      <td>12</td>\n",
              "      <td>1</td>\n",
              "      <td>2</td>\n",
              "      <td>12</td>\n",
              "      <td>codg2014.npy</td>\n",
              "      <td>codg2015.npy</td>\n",
              "      <td>1</td>\n",
              "      <td>mae</td>\n",
              "      <td>adam</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>39 rows × 22 columns</p>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-9523e0f2-353e-47c5-a97d-1b22336b9935')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-9523e0f2-353e-47c5-a97d-1b22336b9935 button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-9523e0f2-353e-47c5-a97d-1b22336b9935');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ]
          },
          "metadata": {},
          "execution_count": 18
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Running experiments\n"
      ],
      "metadata": {
        "id": "pvU8QzUWyjs9"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "%cd /content/tec_forecast\n",
        "!python batch_run.py"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "nQ5H2BywEP4P",
        "outputId": "dd2901a9-c03f-49ca-da29-7e148086e1cd"
      },
      "execution_count": 19,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "/content/tec_forecast\n",
            "Beginning experiment ANN_NtoN_36_2\n",
            " - Log file: /content/tec_forecast/output/ANN_NtoN_36_2/batch_log.txt\n",
            " - Training.\n",
            " - Testing.\n",
            "Model found. Setting as trained.\n",
            "Plotting experiments comparison.\n",
            "/content/tec_forecast/output/Repeat_19-20/results.py\n",
            "/content/tec_forecast/output/Repeat_14-15/results.py\n",
            "/content/tec_forecast/output/ANN_NtoN_36_2/results.py\n",
            "         Network  parameters   MAE  RMSE    r2  rmse (1st)  max error(first)  \\\n",
            "2  ANN_NtoN_36_2       31356 1.119 1.616 0.942       1.616            28.255   \n",
            "0   Repeat_19-20           0 1.024 1.616 0.942       1.824            27.900   \n",
            "1   Repeat_14-15           0 2.856 4.341 0.928       4.512            63.500   \n",
            "\n",
            "   rmse (last)  max error (last)  time (min)  memory  epochs  train_mae  \\\n",
            "2        1.807            23.446       1.452       0      81      1.054   \n",
            "0        1.809            26.800       0.138       0      11      0.963   \n",
            "1        4.424            62.200       0.156       0      11      3.419   \n",
            "\n",
            "   train_rmse  val_mae  val_rmse  mae_mean  mae_std  rmse_mean  rmse_std  \\\n",
            "2       1.509    1.058     1.532     1.119    0.000      1.616     0.000   \n",
            "0       1.521    0.991     1.569     1.024    0.000      1.616     0.000   \n",
            "1       5.077    3.487     5.132     2.856    0.000      4.341     0.000   \n",
            "\n",
            "   r2_mean  r2_std  \n",
            "2    0.942   0.000  \n",
            "0    0.942   0.000  \n",
            "1    0.928   0.000  \n",
            "Done.\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Preparing the results\n",
        "The plotresults.py script prepares the output/results.csv table, with an ordered list of the best results found."
      ],
      "metadata": {
        "id": "BtWFjjxAzJXK"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "%cd /content/tec_forecast\n",
        "!python plotresults.py"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "N_5IAcSoyWsB",
        "outputId": "ce6a2a0a-ac1e-4a6a-80f6-e939a459cfb2"
      },
      "execution_count": 5,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "/content/tec_forecast\n",
            "/content/tec_forecast/output/Repeat_19-20/results.py\n",
            "/content/tec_forecast/output/Repeat_14-15/results.py\n",
            "        Network  parameters   MAE  RMSE    r2  rmse (1st)  max error(first)  \\\n",
            "0  Repeat_19-20           0 1.024 1.616 0.942       1.824            27.900   \n",
            "1  Repeat_14-15           0 2.856 4.341 0.928       4.512            63.500   \n",
            "\n",
            "   rmse (last)  max error (last)  time (min)  memory  epochs  train_mae  \\\n",
            "0        1.809            26.800       0.138       0      11      0.963   \n",
            "1        4.424            62.200       0.156       0      11      3.419   \n",
            "\n",
            "   train_rmse  val_mae  val_rmse  mae_mean  mae_std  rmse_mean  rmse_std  \\\n",
            "0       1.521    0.991     1.569     1.024    0.000      1.616     0.000   \n",
            "1       5.077    3.487     5.132     2.856    0.000      4.341     0.000   \n",
            "\n",
            "   r2_mean  r2_std  \n",
            "0    0.942   0.000  \n",
            "1    0.928   0.000  \n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# The results are stored in the output folder"
      ],
      "metadata": {
        "id": "8LjIJWTBwsbj"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "%cd /content/tec_forecast/output\n",
        "!ls"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "B77Mg8PRwsAt",
        "outputId": "e3920aed-8a34-4137-f120-27ca0b39d0f6"
      },
      "execution_count": 20,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "/content/tec_forecast/output\n",
            "ANN_NtoN_12_2\t       EF-ConvLSTM_3x3\t\t   iConvLSTM\n",
            "ANN_NtoN_18_2\t       EF-ConvLSTM_3x3_14-15\t   IconvLSTM_14-15\n",
            "ANN_NtoN_24_2\t       EF_ConvLSTM_48\t\t   mae.pdf\n",
            "ANN_NtoN_30_2\t       EF_ConvLSTM_60\t\t   max.pdf\n",
            "ANN_NtoN_36_2\t       EF-ConvLSTMv2_1x1\t   REF-ConvLSTMv1_1x1\n",
            "bar_plot.pdf\t       EF-ConvLSTMv2_1x1_14-15\t   REF-ConvLSTMv1_1x1_14-15\n",
            "c1pg\t\t       EF-ConvLSTMv2_3x3\t   REF-ConvLSTMv1_3x3\n",
            "c1pg14-15\t       EF-ConvLSTMv2_3x3_14-15\t   REF-ConvLSTMv1_3x3_14-15\n",
            "ConvLSTM_Bi\t       EF-ConvLSTMv2_3x3_14-15_12  REF-ConvLSTMv1_3x3_14-15_12\n",
            "ConvLSTM_N-1\t       EF-ConvLSTMv2_3x3_14-15_24  REF-ConvLSTMv1_3x3_14-15_24\n",
            "EF_ConvLSTM_12\t       EF-ConvLSTMv2_3x3_14-15_48  REF-ConvLSTMv1_3x3_14-15_48\n",
            "EF-ConvLSTM_1x1        EF-ConvLSTMv2_3x3_14-15_60  Repeat_14-15\n",
            "EF-ConvLSTM_1x1_14-15  EF-ConvLSTMv3\t\t   Repeat_19-20\n",
            "EF_ConvLSTM_24\t       EF-ConvLSTMv3_14-15\t   results.csv\n",
            "EF_ConvLSTM_36\t       errors.pdf\t\t   rmse.pdf\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "results=pd.read_csv(\"/content/tec_forecast/output/results.csv\")\n",
        "results"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 271
        },
        "id": "pSmGKLJ7w0iY",
        "outputId": "525cec67-384c-4e74-d1aa-2ff71592d58b"
      },
      "execution_count": 21,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "   Unnamed: 0        Network  parameters    MAE   RMSE     r2  rmse (1st)  \\\n",
              "0           2  ANN_NtoN_36_2       31356  1.119  1.616  0.942       1.616   \n",
              "1           0   Repeat_19-20           0  1.024  1.616  0.942       1.824   \n",
              "2           1   Repeat_14-15           0  2.856  4.341  0.928       4.512   \n",
              "\n",
              "   max error(first)  rmse (last)  max error (last)  ...  train_mae  \\\n",
              "0            28.255        1.807            23.446  ...      1.054   \n",
              "1            27.900        1.809            26.800  ...      0.963   \n",
              "2            63.500        4.424            62.200  ...      3.419   \n",
              "\n",
              "   train_rmse  val_mae  val_rmse  mae_mean  mae_std  rmse_mean  rmse_std  \\\n",
              "0       1.509    1.058     1.532     1.119      0.0      1.616       0.0   \n",
              "1       1.521    0.991     1.569     1.024      0.0      1.616       0.0   \n",
              "2       5.077    3.487     5.132     2.856      0.0      4.341       0.0   \n",
              "\n",
              "   r2_mean  r2_std  \n",
              "0    0.942     0.0  \n",
              "1    0.942     0.0  \n",
              "2    0.928     0.0  \n",
              "\n",
              "[3 rows x 23 columns]"
            ],
            "text/html": [
              "\n",
              "  <div id=\"df-4b27585a-03f1-4b32-be27-c7db2cf44dd2\">\n",
              "    <div class=\"colab-df-container\">\n",
              "      <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Unnamed: 0</th>\n",
              "      <th>Network</th>\n",
              "      <th>parameters</th>\n",
              "      <th>MAE</th>\n",
              "      <th>RMSE</th>\n",
              "      <th>r2</th>\n",
              "      <th>rmse (1st)</th>\n",
              "      <th>max error(first)</th>\n",
              "      <th>rmse (last)</th>\n",
              "      <th>max error (last)</th>\n",
              "      <th>...</th>\n",
              "      <th>train_mae</th>\n",
              "      <th>train_rmse</th>\n",
              "      <th>val_mae</th>\n",
              "      <th>val_rmse</th>\n",
              "      <th>mae_mean</th>\n",
              "      <th>mae_std</th>\n",
              "      <th>rmse_mean</th>\n",
              "      <th>rmse_std</th>\n",
              "      <th>r2_mean</th>\n",
              "      <th>r2_std</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>2</td>\n",
              "      <td>ANN_NtoN_36_2</td>\n",
              "      <td>31356</td>\n",
              "      <td>1.119</td>\n",
              "      <td>1.616</td>\n",
              "      <td>0.942</td>\n",
              "      <td>1.616</td>\n",
              "      <td>28.255</td>\n",
              "      <td>1.807</td>\n",
              "      <td>23.446</td>\n",
              "      <td>...</td>\n",
              "      <td>1.054</td>\n",
              "      <td>1.509</td>\n",
              "      <td>1.058</td>\n",
              "      <td>1.532</td>\n",
              "      <td>1.119</td>\n",
              "      <td>0.0</td>\n",
              "      <td>1.616</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.942</td>\n",
              "      <td>0.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>0</td>\n",
              "      <td>Repeat_19-20</td>\n",
              "      <td>0</td>\n",
              "      <td>1.024</td>\n",
              "      <td>1.616</td>\n",
              "      <td>0.942</td>\n",
              "      <td>1.824</td>\n",
              "      <td>27.900</td>\n",
              "      <td>1.809</td>\n",
              "      <td>26.800</td>\n",
              "      <td>...</td>\n",
              "      <td>0.963</td>\n",
              "      <td>1.521</td>\n",
              "      <td>0.991</td>\n",
              "      <td>1.569</td>\n",
              "      <td>1.024</td>\n",
              "      <td>0.0</td>\n",
              "      <td>1.616</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.942</td>\n",
              "      <td>0.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>1</td>\n",
              "      <td>Repeat_14-15</td>\n",
              "      <td>0</td>\n",
              "      <td>2.856</td>\n",
              "      <td>4.341</td>\n",
              "      <td>0.928</td>\n",
              "      <td>4.512</td>\n",
              "      <td>63.500</td>\n",
              "      <td>4.424</td>\n",
              "      <td>62.200</td>\n",
              "      <td>...</td>\n",
              "      <td>3.419</td>\n",
              "      <td>5.077</td>\n",
              "      <td>3.487</td>\n",
              "      <td>5.132</td>\n",
              "      <td>2.856</td>\n",
              "      <td>0.0</td>\n",
              "      <td>4.341</td>\n",
              "      <td>0.0</td>\n",
              "      <td>0.928</td>\n",
              "      <td>0.0</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>3 rows × 23 columns</p>\n",
              "</div>\n",
              "      <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-4b27585a-03f1-4b32-be27-c7db2cf44dd2')\"\n",
              "              title=\"Convert this dataframe to an interactive table.\"\n",
              "              style=\"display:none;\">\n",
              "        \n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "       width=\"24px\">\n",
              "    <path d=\"M0 0h24v24H0V0z\" fill=\"none\"/>\n",
              "    <path d=\"M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z\"/><path d=\"M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z\"/>\n",
              "  </svg>\n",
              "      </button>\n",
              "      \n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      flex-wrap:wrap;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "      <script>\n",
              "        const buttonEl =\n",
              "          document.querySelector('#df-4b27585a-03f1-4b32-be27-c7db2cf44dd2 button.colab-df-convert');\n",
              "        buttonEl.style.display =\n",
              "          google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "        async function convertToInteractive(key) {\n",
              "          const element = document.querySelector('#df-4b27585a-03f1-4b32-be27-c7db2cf44dd2');\n",
              "          const dataTable =\n",
              "            await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                     [key], {});\n",
              "          if (!dataTable) return;\n",
              "\n",
              "          const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "            '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "            + ' to learn more about interactive tables.';\n",
              "          element.innerHTML = '';\n",
              "          dataTable['output_type'] = 'display_data';\n",
              "          await google.colab.output.renderOutput(dataTable, element);\n",
              "          const docLink = document.createElement('div');\n",
              "          docLink.innerHTML = docLinkHtml;\n",
              "          element.appendChild(docLink);\n",
              "        }\n",
              "      </script>\n",
              "    </div>\n",
              "  </div>\n",
              "  "
            ]
          },
          "metadata": {},
          "execution_count": 21
        }
      ]
    }
  ]
}