docs/source/_notebooks/model/data_splitting.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "b25bb1ba",
   "metadata": {},
   "source": [
    "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/AtrCheema/ai4water_examples/blob/master/docs/source/_notebooks/model/data_splitting.ipynb)\n",
    "\n",
    "[![View Source on GitHub](https://img.shields.io/badge/github-view%20source-black.svg)](https://github.com/AtrCheema/ai4water_examples/blob/master/docs/source/_notebooks/model/data_splitting.ipynb)\n",
    "\n",
    "## Data Splitting\n",
    "\n",
    "This notebook describes how to split data into training, validation and test sets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "4d9d774e",
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "import numpy as np\n",
    "\n",
    "from ai4water.datasets import busan_beach\n",
    "from ai4water.preprocessing import DataSet\n",
    "from ai4water.utils.utils import get_version_info"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "13db1e00",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "python 3.9.7 | packaged by conda-forge | (default, Sep 29 2021, 19:20:16) [MSC v.1916 64 bit (AMD64)]\n",
      "os nt\n",
      "ai4water 1.07\n",
      "lightgbm 3.3.1\n",
      "tcn 3.4.0\n",
      "catboost 0.26\n",
      "xgboost 1.5.0\n",
      "easy_mpl 0.21.3\n",
      "SeqMetrics 1.3.3\n",
      "tensorflow 2.7.0\n",
      "keras.api._v2.keras 2.7.0\n",
      "numpy 1.21.0\n",
      "pandas 1.3.4\n",
      "matplotlib 3.4.3\n",
      "h5py 3.5.0\n",
      "sklearn 1.0.1\n",
      "shapefile 2.3.0\n",
      "fiona 1.8.22\n",
      "xarray 0.20.1\n",
      "netCDF4 1.5.7\n",
      "optuna 2.10.1\n",
      "skopt 0.9.0\n",
      "hyperopt 0.2.7\n",
      "plotly 5.3.1\n",
      "lime NotDefined\n",
      "seaborn 0.11.2\n"
     ]
    }
   ],
   "source": [
    "for lib, ver in get_version_info().items():\n",
    "    print(lib, ver)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "c0f7eeb4",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(1446, 14)"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data = busan_beach()\n",
    "data.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "4d40283a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(218, 14)"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.dropna().shape"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "352ff4f8",
   "metadata": {},
   "source": [
    "In DataSet class, 70% of the total examples are considered for training while the remaining 30% examples are reserved for test. The validation data is 20% of the training data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "be127ff4",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "********** Removing Examples with nan in labels  **********\n",
      "\n",
      "***** Training *****\n",
      "input_x shape:  (121, 13)\n",
      "target shape:  (121, 1)\n",
      "\n",
      "********** Removing Examples with nan in labels  **********\n",
      "\n",
      "***** Validation *****\n",
      "input_x shape:  (31, 13)\n",
      "target shape:  (31, 1)\n",
      "\n",
      "********** Removing Examples with nan in labels  **********\n",
      "\n",
      "***** Test *****\n",
      "input_x shape:  (66, 13)\n",
      "target shape:  (66, 1)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "218"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ds = DataSet(data=data)\n",
    "\n",
    "train_x, train_y = ds.training_data()\n",
    "val_x, val_y = ds.validation_data()\n",
    "test_x, test_y = ds.test_data()\n",
    "\n",
    "len(train_x) + len(val_x) + len(test_x)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1103bc18",
   "metadata": {},
   "source": [
    "We had 218 valid examples, 70% of which i.e. 152 were considered for training and remaining 30% i.e. 66 examples were reserved for test. Since the validation fraction was 0.2 (by default) that means we need to put 20% of 152 for validation and thus when we called ``training_data``, we got 121 examples."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "95448c8c",
   "metadata": {},
   "source": [
    "### train fraction"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2440ac76",
   "metadata": {},
   "source": [
    "We can confirm that the default train fraction was 0.7."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "f747114f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.7"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ds.train_fraction"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f61ab9de",
   "metadata": {},
   "source": [
    "If we don't want to separate any data for test set, we can set the train_fraction to 1.0. This means we want to consider the whole data for training. Now the validation data (which is 20%) will be taken from the total examples."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "6c04a401",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "********** Removing Examples with nan in labels  **********\n",
      "\n",
      "***** Training *****\n",
      "input_x shape:  (174, 13)\n",
      "target shape:  (174, 1)\n",
      "\n",
      "********** Removing Examples with nan in labels  **********\n",
      "\n",
      "***** Validation *****\n",
      "input_x shape:  (44, 13)\n",
      "target shape:  (44, 1)\n",
      "***** Test *****\n",
      "input_x shape:  (0,)\n",
      "target shape:  (0,)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "218"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ds = DataSet(data=data, train_fraction=1.0)\n",
    "\n",
    "train_x, train_y = ds.training_data()\n",
    "val_x, val_y = ds.validation_data()\n",
    "test_x, test_y = ds.test_data()\n",
    "\n",
    "len(train_x) + len(val_x) + len(test_x)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b84230ee",
   "metadata": {},
   "source": [
    "20% of examples of 218 examples are 44. We did not get any example for test set, because train fraction was 1.0."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cb636932",
   "metadata": {},
   "source": [
    "### val_fraction\n",
    "\n",
    "We can also control the number of examples for the valiation set by making use of ``val_fraction``. Remember that the val_fraction is always considered as fraction of training set. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "314b4cdc",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "********** Removing Examples with nan in labels  **********\n",
      "\n",
      "***** Training *****\n",
      "input_x shape:  (109, 13)\n",
      "target shape:  (109, 1)\n",
      "\n",
      "********** Removing Examples with nan in labels  **********\n",
      "\n",
      "***** Validation *****\n",
      "input_x shape:  (109, 13)\n",
      "target shape:  (109, 1)\n",
      "***** Test *****\n",
      "input_x shape:  (0,)\n",
      "target shape:  (0,)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "218"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ds = DataSet(data=data, train_fraction=1.0, val_fraction=0.5)\n",
    "\n",
    "train_x, train_y = ds.training_data()\n",
    "val_x, val_y = ds.validation_data()\n",
    "test_x, test_y = ds.test_data()\n",
    "\n",
    "len(train_x) + len(val_x) + len(test_x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "18afd38d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "********** Removing Examples with nan in labels  **********\n",
      "\n",
      "***** Training *****\n",
      "input_x shape:  (76, 13)\n",
      "target shape:  (76, 1)\n",
      "\n",
      "********** Removing Examples with nan in labels  **********\n",
      "\n",
      "***** Validation *****\n",
      "input_x shape:  (76, 13)\n",
      "target shape:  (76, 1)\n",
      "\n",
      "********** Removing Examples with nan in labels  **********\n",
      "\n",
      "***** Test *****\n",
      "input_x shape:  (66, 13)\n",
      "target shape:  (66, 1)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "218"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ds = DataSet(data=data, train_fraction=0.7, val_fraction=0.5)\n",
    "\n",
    "train_x, train_y = ds.training_data()\n",
    "val_x, val_y = ds.validation_data()\n",
    "test_x, test_y = ds.test_data()\n",
    "\n",
    "len(train_x) + len(val_x) + len(test_x)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b00f498d",
   "metadata": {},
   "source": [
    "### random splitting\n",
    "\n",
    "The default splitting strategy is sequential. This means, the first examples (determined by train_fraction) are used for training and the later examples are considered for test. We can confirm this by checking the inputs and outputs from the first example. They both correspond to the first non-nan row."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "44d54e32",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ -22.245026 ,   19.457182 ,   34.00429  ,   24.28     ,\n",
       "          0.       ,    0.       ,    0.       ,    6.       ,\n",
       "        205.00667  ,    1.6533333,  998.61334  , 1002.9133   ,\n",
       "         75.1      ], dtype=float32)"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_x[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "db29abf4",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([444866.9004])"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_y[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f6b3ea4c",
   "metadata": {},
   "source": [
    "However, in many machine learning problems, where the data is not time-series, one may wishes to split the data randomly into training and test sets. Thi can be acheived by setting the ``split_random`` to True. Now the first example from training data is not necessarily from the first row."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "5c46b6d3",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([1363760.645])"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ds = DataSet(data=data, split_random=True, verbosity=0)\n",
    "\n",
    "train_x, train_y = ds.training_data()\n",
    "val_x, val_y = ds.validation_data()\n",
    "test_x, test_y = ds.test_data()\n",
    "\n",
    "train_y[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "97751912",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([2356366.032])"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_y[1]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "896a084e",
   "metadata": {},
   "source": [
    "### reproducibility\n",
    "\n",
    "If we create examples from our the same data again and again using random splitting, the examples reserved for training (or for validation and test) are same. This is because, random seed is always set to 313 to ensure reproducibility by default. This is carried out to acheive reproducible results. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "d5fdcd75",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1363760.645]\n",
      "[1363760.645]\n",
      "[1363760.645]\n",
      "[1363760.645]\n",
      "[1363760.645]\n",
      "[1363760.645]\n",
      "[1363760.645]\n",
      "[1363760.645]\n",
      "[1363760.645]\n",
      "[1363760.645]\n"
     ]
    }
   ],
   "source": [
    "for i in range(10):\n",
    "    ds = DataSet(data=data, split_random=True, verbosity=0)\n",
    "\n",
    "    train_x, train_y = ds.training_data()\n",
    "    val_x, val_y = ds.validation_data()\n",
    "    test_x, test_y = ds.test_data()\n",
    "\n",
    "    print(train_y[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f7b15d1f",
   "metadata": {},
   "source": [
    "If however, we do not set a seed, then different examples will be considered for training/validation/test sets. This is shown below by setting the ``seed`` to None."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "caf7da5e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[332194.7001]\n",
      "[275100.6769]\n",
      "[4881349.328]\n",
      "[278787.0508]\n",
      "[844769.5803]\n",
      "[216767.3381]\n",
      "[2391848.163]\n",
      "[141357.2781]\n",
      "[761979.6799]\n",
      "[20083984.46]\n"
     ]
    }
   ],
   "source": [
    "for i in range(10):\n",
    "    ds = DataSet(data=data, split_random=True, seed=None, verbosity=0)\n",
    "\n",
    "    train_x, train_y = ds.training_data()\n",
    "    val_x, val_y = ds.validation_data()\n",
    "    test_x, test_y = ds.test_data()\n",
    "\n",
    "    print(train_y[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fe0de5bd",
   "metadata": {},
   "source": [
    "We can also set the seed of our choice. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "ce710772",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[774076.752]\n",
      "[2060160.801]\n",
      "[202449.1352]\n",
      "[21289.67181]\n",
      "[36045668.64]\n",
      "[14976057.52]\n",
      "[3291674.776]\n",
      "[2356366.032]\n",
      "[836261.1064]\n",
      "[3256878.75]\n"
     ]
    }
   ],
   "source": [
    "for i in range(10):\n",
    "    ds = DataSet(data=data, split_random=True, seed=i, verbosity=0)\n",
    "\n",
    "    train_x, train_y = ds.training_data()\n",
    "    val_x, val_y = ds.validation_data()\n",
    "    test_x, test_y = ds.test_data()\n",
    "\n",
    "    print(train_y[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "07296b59",
   "metadata": {},
   "source": [
    "### spliting using indices\n",
    "\n",
    "Sometimes, we want to have more control over splitting strategy. We want to specify the examples to be considered for training/validation/test. One way to acheiving this using DataSet class is by specifying the **indices** for training or for training, validation and test sets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "14c3c514",
   "metadata": {},
   "outputs": [],
   "source": [
    "indices = {\n",
    "    'training': np.arange(50)\n",
    "}\n",
    "ds = DataSet(data=data, indices=indices, verbosity=0)\n",
    "\n",
    "\n",
    "train_x, train_y = ds.training_data()\n",
    "val_x, val_y = ds.validation_data()\n",
    "test_x, test_y = ds.test_data()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1ebde469",
   "metadata": {},
   "source": [
    "Above we are specifying the training examples by saying that the first 50 examples are to be considered for training. Since the validation data is taken from training set, 10 examples (20%) are taken for validation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "21f12c8a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(40, 10, 168)"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(train_x), len(val_x), len(test_x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "43c6fe25",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([444866.9004])"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_y[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "329b47e2",
   "metadata": {},
   "source": [
    "### spliting using intervals"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "68082e3f",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}