5_CV.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "05bbf5b4",
   "metadata": {},
   "source": [
    "# DeepChem\n",
    "\n",
    "- [Installation](https://github.com/deepchem/deepchem#installation)\n",
    "- [Tutorial](https://deepchem.readthedocs.io/en/latest/get_started/tutorials.html)\n",
    "- [Sample Notebooks](https://github.com/deepchem/deepchem/tree/master/examples/tutorials) "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "67b5cba8",
   "metadata": {},
   "source": [
    "## Environment\n",
    "\n",
    "- Python 3.7.11\n",
    "\n",
    "- deepchem == 2.5.0\n",
    "\n",
    "- TensorFlow == 2.6.0\n",
    "\n",
    "- PyTorch == 1.9.0\n",
    "\n",
    "- scikit-learn == 0.23.2\n",
    "\n",
    "- DGL == 0.7.0\n",
    "\n",
    "- RDKit == 2021.03.4\n",
    "\n",
    "- numpy == 1.20.3\n",
    "\n",
    "- pandas == 1.3.2"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "61a60155",
   "metadata": {},
   "source": [
    "When using neural networks and deep neural models, dataset is usually divided into train, validate, and test sets. Especially, when dataset is large, cross-validation is not applied. \n",
    "\n",
    "However, in this notebook, we try to apply cross-validation. Dataset is divided into several (train - test) subsets. By this way, we can get predicted values for all observations. \n",
    "\n",
    "\n",
    "**Hyperparameter tuning**\n",
    "\n",
    "Method : Grid Search implemented in DeepChem\n",
    "\n",
    "Before CV, we apply hyperparameter tuning. \n",
    "\n",
    "Hyperparameter tuning is done using the whole dataset. We don’t repeat hyperparameter tuning in every fold of cross-validation.\n",
    "\n",
    "20% of data (as validation set) is used to evaluate model performance (R2 score). Best hyperparameters are the ones with the highest model performance on validation set. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "c32ba1a2",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "\n",
    "import deepchem as dc\n",
    "\n",
    "from sklearn.metrics import r2_score\n",
    "\n",
    "from tools.models_graph import CV_graph, CV_graph_models\n",
    "from tools.models_graph import hyperparams_tuning_models\n",
    "from tools.models_graph import standard_scaling\n",
    "\n",
    "from tools.get_params import get_search_space, get_best_hyperparams_models"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8cce1c12",
   "metadata": {},
   "source": [
    "# Parameters , Settings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "0759790b",
   "metadata": {},
   "outputs": [],
   "source": [
    "data_filepath = './data/ESOL_modified.csv'   # observation 934 ('C') was removed to avoid error for featurizer \"MolGraphConvFeaturizer\"   \n",
    "\n",
    "save_hyperparams_folder = './result/Hyperparameter'   # folder to save models during hyperparameter tuning\n",
    "metric_hyperparams_filepath = './result/metric_hyperparams.json'   # filepath to save hyperparameter tuning results (metrics)\n",
    "fig_save_folder = './result/figures'   # folder to save figures\n",
    "metrics_filepath = './result/metrics.json'   # filepath to save GPR results\n",
    "\n",
    "n_tasks = 1   # No. of tasks (No. of dependent variables)\n",
    "nb_epoch = 100\n",
    "\n",
    "CV_method = 'k-fold'   \n",
    "k = 5   # value of k for k-fold cross validation"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4a6a2a44",
   "metadata": {},
   "source": [
    "# Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "4dab8eba",
   "metadata": {},
   "outputs": [],
   "source": [
    "data = pd.read_csv(data_filepath)  \n",
    " \n",
    "smiles = data['smiles'].values   # should be 1D\n",
    "\n",
    "y = data['measured log solubility in mols per litre'].values.reshape(-1,1)   # can be 1D or 2D\n",
    "\n",
    "# It is recommended to standardize y\n",
    "y_ss = standard_scaling(y)   # 2D numpy array"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "561d3f22",
   "metadata": {},
   "source": [
    "# Hyperparameter Tuning (Grid Search)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "32ba75c1",
   "metadata": {},
   "outputs": [],
   "source": [
    "# We apply Hyperparameter Tuning on the whole dataset (after diving into train and validation sets)\n",
    "# We don't apply Hyperparameter Tuning in every fold of CV. \n",
    "search_space_models = get_search_space(n_tasks)\n",
    "\n",
    "metric_hyperparams, best_hyperparams_all = hyperparams_tuning_models(search_space_models, smiles, y_ss, \n",
    "                  metric_hyperparams_filepath, save_hyperparams_folder, nb_epoch=nb_epoch)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "635a2b01",
   "metadata": {},
   "source": [
    "Hyperparameter Tuning is time-consuming. \n",
    "\n",
    "Therefore, we have saved best hyperparameters in a JSON file. Next time, we can load the hyperparameters, no need to repeat hyperparameter tuning. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0ed225fe",
   "metadata": {},
   "source": [
    "# CV"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "5fd31949",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load hyperparameters\n",
    "best_hyperparams_filepath = './result/best_hyperparams_ESOL_norm.json'   # filepath to load best hyperparameters\n",
    "hyperparams_models = get_best_hyperparams_models(best_hyperparams_filepath)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e76233f1",
   "metadata": {},
   "source": [
    "### CV for all models"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "75a29f57",
   "metadata": {},
   "outputs": [],
   "source": [
    "fig_name = 'LogS'\n",
    "plot_title = 'LogS'\n",
    "results_metrics, results_y = CV_graph_models(hyperparams_models, smiles, y_ss, nb_epoch, metrics_filepath, fig_save_folder, fig_name, plot_title,\n",
    "                  CV_method, show_plot = True, k = k, apply_inverse_scaling=True, y_train_original = y, get_uncertainty=True)    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d6ae45a8",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}