Model_validation_training.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Model_validation_training.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "authorship_tag": "ABX9TyMX9VYooCYk+XmkwVx51EuY",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/mlraul/biosensor_predictor/blob/main/Model_validation_training.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Net training whit K-Fold Cross Validation\n",
        "In this code, the net will be validated using the K-Fold CV method."
      ],
      "metadata": {
        "id": "ELUqwV66-pAm"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import numpy as np\n",
        "import matplotlib.pyplot as plt\n",
        "\n",
        "from pydrive.auth import GoogleAuth\n",
        "from pydrive.drive import GoogleDrive\n",
        "from google.colab import auth\n",
        "from oauth2client.client import GoogleCredentials\n",
        "from google.colab import files\n",
        "\n",
        "from keras.models import Sequential, Model\n",
        "from keras.layers import LSTM, Dense, Activation, Dropout, Input\n",
        "from keras.layers.merge import Concatenate\n",
        "from keras import metrics\n",
        "from tensorflow.keras.optimizers import Adam\n",
        "\n",
        "from sklearn.model_selection import train_test_split\n",
        "from sklearn.model_selection import KFold\n",
        "from sklearn.metrics import roc_curve\n",
        "from sklearn.metrics import auc\n",
        "from sklearn.metrics import classification_report"
      ],
      "metadata": {
        "id": "uWbOWrTx-r4V"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "###Loading inputs\n",
        "\n",
        "- fp_m is a matrix where each row contains the fingerprint of a molecule. Each fingerprint is written in two successive rows.\n",
        "- aa_m is a 3-Dimensional array where each matrix represents the amino acids of the Transcription Factor (TF)."
      ],
      "metadata": {
        "id": "-1Im6w8s_dVD"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# PyDrive client\n",
        "auth.authenticate_user()\n",
        "gauth = GoogleAuth()\n",
        "gauth.credentials = GoogleCredentials.get_application_default()\n",
        "drive = GoogleDrive(gauth)"
      ],
      "metadata": {
        "id": "_-njuW7S_eam"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "downloaded = drive.CreateFile({'id':\"1EeFJCysQ6Ts0ebDkxMuohrRTYo4T0GDI\"})\n",
        "downloaded.GetContentFile('aa_neg_no_motifs_matrix.npy')\n",
        "aa_m = np.load('aa_neg_no_motifs_matrix.npy')\n",
        "\n",
        "downloaded = drive.CreateFile({'id':\"1co-dTn5YYPiXKgYTFJ6-wi2wGVoHXF7G\"})\n",
        "downloaded.GetContentFile('fingerprints_v3_matrix.npy')\n",
        "fp_m = np.load('fingerprints_v3_matrix.npy')"
      ],
      "metadata": {
        "id": "ACsR0SO2_hE1"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "###Creating the labels\n",
        "In our current input, every amino acids sequence is given twice:\n",
        "\n",
        "- The first one is associated to the fingerprint which represents the molecule with higher affinity with this TF.\n",
        "-The second one is associated to a random fingerprint to which the affinity is lower.\n",
        "\n",
        "With this idea in mind, the labels set consists on a repeated sequence of 1 and 0 (with 1 meaning affinity between the TF and the molecule, and 0 the opposite)."
      ],
      "metadata": {
        "id": "M4mAhuly_ne1"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "lbl = np.ones(len(fp_m))\n",
        "\n",
        "for i in range(1, len(lbl), 2):\n",
        "  lbl[i] = 0"
      ],
      "metadata": {
        "id": "tca2T5BU_pl2"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "###Splitting the data\n",
        "Here two data subsets will be created:\n",
        "\n",
        "- The train data, which will be used to train the net. This will be splitted again in different folds when using the K-F CV.\n",
        "- The test data, which will be used to evaluate it."
      ],
      "metadata": {
        "id": "EjC2IvqOCnw-"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "aa_train, aa_test, fp_train, fp_test, lbl_train, lbl_test = train_test_split(aa_m, fp_m, lbl, test_size = 0.20)"
      ],
      "metadata": {
        "id": "1D_VmcalCobc"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "###Model configuration\n",
        "Here different parameters of the net architecture will be defined."
      ],
      "metadata": {
        "id": "eZkrZQzNCqr2"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Input layers shape\n",
        "i1_shape = np.shape(aa_m)[1:]\n",
        "i2_shape = np.shape(fp_m)[1]\n",
        "\n",
        "# Model parameters\n",
        "loss_function = 'binary_crossentropy'\n",
        "opt = Adam(learning_rate=0.0001)\n",
        "metrics = ['accuracy']\n",
        "\n",
        "num_folds = 5\n",
        "batch_size = 50\n",
        "num_epochs = 30 \n",
        "\n",
        "lstm_units = 90\n",
        "densefp_units = 70\n",
        "dense_units = 70\n",
        "\n",
        "verbosity = 0"
      ],
      "metadata": {
        "id": "lHkmkkmnCtEN"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "###K-Fold definition\n",
        "It defines de K-Fold Cross Validator."
      ],
      "metadata": {
        "id": "DtUZ5EKoDDC9"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "kfold = KFold(n_splits = num_folds)"
      ],
      "metadata": {
        "id": "AG0yyrm7DD7F"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Creating the net architecture\n",
        "It has two branches:\n",
        "- One for the amino acids and motifs matrixes. Here a LSTM layer is used, as the inputs are actually sequences.\n",
        "- Other for the molecules fingerprints. By the moment, a Dense layer is used here.\n",
        "\n",
        "Then, these two branches are concatenated. The current architecture finishes with a Dense layer as output."
      ],
      "metadata": {
        "id": "zRtkNUtlDULW"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Branch for sequences matrixes. A LSTM layer is used\n",
        "input1 = Input(shape = i1_shape)\n",
        "i11 = LSTM(lstm_units, activation='relu', recurrent_activation='sigmoid',\n",
        "           dropout=0.2, recurrent_dropout=0.01)(input1)\n",
        "\n",
        "# Branch for fingerprints\n",
        "input2 = Input(shape = i2_shape)\n",
        "i21 = Dense(densefp_units, activation='relu')(input2)\n",
        "\n",
        "# Joint of branches\n",
        "concat = Concatenate()([i11, i21])\n",
        "\n",
        "# Other layers TEST (DROPOUTS)\n",
        "dense1 = Dense(dense_units, activation='relu')(concat)\n",
        "\n",
        "# Output layer\n",
        "out = Dense(1)(dense1)\n",
        "\n",
        "# Creation and compilation of the model\n",
        "model = Model(inputs = [input1, input2], outputs = out)\n",
        "model.compile(loss = loss_function, optimizer = opt, metrics = metrics)\n",
        "print(model.summary())\n"
      ],
      "metadata": {
        "id": "QWXLcZ5MDZG2"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "###K-Fold CV model evaluation\n",
        "Here the CV evaluation will be performed. "
      ],
      "metadata": {
        "id": "hcca8StWDF2d"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Defining the per-fold score lists\n",
        "acc_per_fold = []\n",
        "loss_per_fold = []\n",
        "\n",
        "fold_num = 1\n",
        "\n",
        "# Saving the initial model weights\n",
        "Wsave = model.get_weights()\n",
        "    \n",
        "for train, val in kfold.split(aa_train, fp_train, lbl_train):\n",
        "\n",
        "  model.set_weights(Wsave)\n",
        "    \n",
        "  # Generate a print\n",
        "  print('------------------------------------------------------------------')\n",
        "  print(f'Training for fold {fold_num} ...')\n",
        "    \n",
        "    \n",
        "  history = model.fit([aa_train[train], fp_train[train]], lbl_train[train],\n",
        "                      validation_data=([aa_train[val], fp_train[val]], lbl_train[val]),\n",
        "                      batch_size = batch_size, \n",
        "                      epochs = num_epochs, \n",
        "                      verbose = verbosity)\n",
        "    \n",
        "  # summarize history for accuracy\n",
        "  plt.figure(dpi=100)\n",
        "  plt.plot(history.history['accuracy'])\n",
        "  plt.plot(history.history['val_accuracy'])\n",
        "  plt.title('Model accuracy')\n",
        "  plt.ylabel('Accuracy')\n",
        "  plt.xlabel('Epochs')\n",
        "  plt.legend(['train', 'validation'], loc='upper left')\n",
        "  plt.show()\n",
        "\n",
        "  # summarize history for loss\n",
        "  plt.figure(dpi=100)\n",
        "  plt.plot(history.history['loss'])\n",
        "  plt.plot(history.history['val_loss'])\n",
        "  plt.title('Model loss')\n",
        "  plt.ylabel('Loss')\n",
        "  plt.xlabel('Epochs')\n",
        "  plt.legend(['train', 'validation'], loc='upper left')\n",
        "  plt.show()\n",
        "\n",
        "  fold_loss = history.history['val_loss'][-1]\n",
        "  fold_accuracy = history.history['val_accuracy'][-1] * 100\n",
        "  print(f'Score for fold {fold_num}: {list(history.history.keys())[2]} of {fold_loss}; {list(history.history.keys())[3]} of {fold_accuracy} %')\n",
        "\n",
        "  acc_per_fold.append(fold_accuracy)\n",
        "  loss_per_fold.append(fold_loss)\n",
        "    \n",
        "  # Increasing fold number\n",
        "  fold_num = fold_num + 1"
      ],
      "metadata": {
        "id": "FF80mjcvDp1l"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "###Average scores\n",
        "Here the score for each fold will be displayed, as well as the average of all scores. \n",
        "This result will be used to evaluate how well the model is performing."
      ],
      "metadata": {
        "id": "Yu01mDgfEVgW"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "print('------------------------------------------------------------------')\n",
        "print('Score per fold')\n",
        "for i in range(0, len(acc_per_fold)):\n",
        "  print('------------------------------------------------------------------')\n",
        "  print(f'> Fold {i+1} - Loss: {loss_per_fold[i]} - Accuracy: {acc_per_fold[i]} %')\n",
        "print('------------------------------------------------------------------')\n",
        "print('Average scores for all folds:')\n",
        "print(f'> Accuracy: {np.mean(acc_per_fold)} % (+- {np.std(acc_per_fold)} %)')\n",
        "print(f'> Loss: {np.mean(loss_per_fold)}')\n",
        "print('------------------------------------------------------------------')"
      ],
      "metadata": {
        "id": "krsw0MKREWNl"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "###Training the selected model\n",
        "If the model validated is decided to be the definitive model, the final training will be carried out using all the training data. For this purpose, type 'train' when asking for it. Otherwise, the model will not be trained.\n",
        "\n",
        "If the model is decided to be trained, the test will also be applied. The curve ROC is plotted and a classification report is printed."
      ],
      "metadata": {
        "id": "0cx1XWpH9ML5"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "print('Is the model validated prepared to be trained?')\n",
        "answer = input(\"Press 'train' for training the model, press any other key for not training it: \")\n",
        "if answer == 'train':\n",
        "  print('Starting training...')\n",
        "\n",
        "  model.set_weights(Wsave)\n",
        "\n",
        "  history = model.fit([aa_train, fp_train], lbl_train,\n",
        "                      batch_size = batch_size, \n",
        "                      epochs = num_epochs, \n",
        "                      verbose = verbosity)\n",
        "\n",
        "  loss = history.history['loss'][-1]\n",
        "  accuracy = history.history['accuracy'][-1] * 100\n",
        "  print('------------------------------------------------------------------')\n",
        "  print('Scores:')\n",
        "  print(f'> Accuracy: {accuracy} %')\n",
        "  print(f'> Loss: {loss}')\n",
        "  print('------------------------------------------------------------------')\n",
        "\n",
        "  results = model.evaluate([aa_test, fp_test], lbl_test, \n",
        "                            verbose=0)\n",
        "  print(f'{model.metrics_names[0]} of {results[0]}; {model.metrics_names[1]} of {results[1]*100} %')\n",
        "\n",
        "  preds_test = model.predict([aa_test, fp_test]).ravel()\n",
        "  fpr, tpr, thresholds = roc_curve(lbl_test, preds_test)\n",
        "  auc_value = auc(fpr, tpr)\n",
        "\n",
        "  plt.figure(dpi=100)\n",
        "  plt.plot([0, 1], [0, 1], 'k--')\n",
        "  plt.plot(fpr, tpr, label='AUC = {:.3f})'.format(auc_value))\n",
        "  plt.xlabel('False positive rate')\n",
        "  plt.ylabel('True positive rate')\n",
        "  plt.title('ROC curve')\n",
        "  plt.legend(loc='best')\n",
        "  plt.show()\n",
        "\n",
        "  preds_binary = preds_test.copy()\n",
        "  for i in range(len(preds_binary)):\n",
        "    if preds_binary[i] > 0.5:\n",
        "      preds_binary[i] = 1\n",
        "    else:\n",
        "      preds_binary[i] = 0\n",
        "\n",
        "  print(classification_report(lbl_test, preds_binary))\n",
        "\n",
        "else:\n",
        "  print('Model discarded')"
      ],
      "metadata": {
        "id": "tKIz5FXl9LRx"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "###Saving the model\n",
        "Once the model is fully trained and tested, it is saved in order to be used to make future predictions. For saving it type 'save' when asking for it. Otherwise the model will not be saved."
      ],
      "metadata": {
        "id": "8mj1LssRBO0M"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "print('Do you want to save the model?')\n",
        "answer = input(\"Press 'save' for saving the model, press any other key for not saving it: \")\n",
        "if answer == 'save':\n",
        "  print('Saving model...')\n",
        "  # Saves the entire model as a SavedModel.\n",
        "  !mkdir -p saved_model\n",
        "  model.save('saved_model/final_model')\n",
        "\n",
        "else:\n",
        "  print('Model not saved')\n"
      ],
      "metadata": {
        "id": "Lx5emb1uBZO7"
      },
      "execution_count": null,
      "outputs": []
    }
  ]
}