survival_analysis/example_tfdeepsurv_data_real.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# tfdeepsurv for real data\n",
    "\n",
    "## Introduction\n",
    "\n",
    "Let's use `tfdeepsurv` package to build a neural network for predicting hazard ratio. This notebook \n",
    "will show you how to build and train a neural network.\n",
    "\n",
    "## Preparation\n",
    "\n",
    "For all things going well, you would be better to get acquainted with **Survival Analysis**. Otherwise, I suggest you read the [reference](https://lifelines.readthedocs.io/en/latest/Survival%20Analysis%20intro.html).\n",
    "\n",
    "## Package installation\n",
    "\n",
    "Please follow the instructions on [README](../README.md) to install `tfdeepsurv` package.\n",
    "\n",
    "Install the package as below if use Colab.\n",
    "\n",
    "```python\n",
    "!git clone https://github.com/liupei101/TFDeepSurv.git TFDeepSurv\n",
    "!pip install ./TFDeepSurv\n",
    "```\n",
    "\n",
    "## Get it started\n",
    "\n",
    "### Obtain datasets "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "train_data = pd.read_csv(\"data_train_filename.csv\")\n",
    "test_data = pd.read_csv(\"data_test_filename.csv\")\n",
    "\n",
    "train_data.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Dataset statistics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from tfdeepsurv.datasets import survival_stats\n",
    "\n",
    "# specify the colnames of observed status and time in your dataset\n",
    "colname_e = 'e'\n",
    "colname_t = 't'\n",
    "\n",
    "survival_stats(train_data, t_col=colname_t, e_col=colname_e, plot=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "survival_stats(test_data, t_col=colname_t, e_col=colname_e, plot=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Survival data transfrom\n",
    "\n",
    "The transformed survival data contains an new label. Negtive values are considered as right censored, \n",
    "and positive values are considered as event occurrence.\n",
    "\n",
    "**NOTE**: In version 2.0, survival data must be transformed via `tfdeepsurv.datasets.survival_df`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from tfdeepsurv.datasets import survival_df\n",
    "\n",
    "surv_train = survival_df(train_data, t_col=colname_t, e_col=colname_e, label_col=\"Y\")\n",
    "surv_test = survival_df(test_data, t_col=colname_t, e_col=colname_e, label_col=\"Y\")\n",
    "\n",
    "# columns 't' and 'e' are packed into an new column 'Y'\n",
    "surv_train.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Model initialization\n",
    "\n",
    "**NOTE:** You can freely change all hyper-parameters during model initialization or training as you want.\n",
    "\n",
    "All hyper-parameters is as follows:\n",
    "- `nn_config`: model configuration\n",
    "- `hidden_layers_nodes`: hidden layers configuration\n",
    "- `num_steps`: training steps\n",
    "\n",
    "Hyperparameters tuning can refer to README in directory `byopt`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from tfdeepsurv import dsnn\n",
    "\n",
    "# Number of features in your dataset\n",
    "input_nodes = len(surv_train.columns) - 1\n",
    "# Specify your neural network structure\n",
    "hidden_layers_nodes = [6, 3, 1]\n",
    "\n",
    "# the arguments of dsnn can be obtained by Bayesian Hyperparameters Tuning.\n",
    "# It would affect your model performance largely!\n",
    "nn_config = {\n",
    "    \"learning_rate\": 0.7,\n",
    "    \"learning_rate_decay\": 1.0,\n",
    "    \"activation\": 'relu', \n",
    "    \"L1_reg\": 3.4e-5, \n",
    "    \"L2_reg\": 8.8e-5, \n",
    "    \"optimizer\": 'sgd',\n",
    "    \"dropout_keep_prob\": 1.0,\n",
    "    \"seed\": 1\n",
    "}\n",
    "\n",
    "# ESSENTIAL STEP-1: Pass arguments\n",
    "model = dsnn(\n",
    "    input_nodes, \n",
    "    hidden_layers_nodes,\n",
    "    nn_config\n",
    ")\n",
    "\n",
    "# ESSENTIAL STEP-2: Build Computation Graph\n",
    "model.build_graph()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Model training\n",
    "\n",
    "You can save your trained model by passing `save_model=\"file_name.ckpt\"` or load your trained model by passing `load_model=\"file_name.ckpt\"`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "Y_col = [\"Y\"]\n",
    "X_cols = [c for c in surv_train.columns if c not in Y_col]\n",
    "\n",
    "# model saving and loading is also supported!\n",
    "# read comments of `train()` function if necessary.\n",
    "\n",
    "# ESSENTIAL STEP-3: Train Model\n",
    "# `num_steps` is also a important parameters\n",
    "watch_list = model.train(\n",
    "    surv_train[X_cols], surv_train[Y_col],\n",
    "    num_steps=1900,\n",
    "    num_skip_steps=100,\n",
    "    plot=True\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Model evaluation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "print(\"CI on training data:\", model.evals(surv_train[X_cols], surv_train[Y_col]))\n",
    "print(\"CI on test data:\", model.evals(surv_test[X_cols], surv_test[Y_col]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Model prediction\n",
    "\n",
    "Model prediction includes:\n",
    "- predicting hazard ratio or log hazard ratio\n",
    "- predicting survival function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# predict log hazard ratio\n",
    "print(model.predict(surv_test[X_cols][0:10])) # fixed indexing problem\n",
    "\n",
    "# predict hazard ratio\n",
    "print(model.predict(surv_test[X_cols][0:10], output_margin=False)) # fixed indexing problem"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# predict survival function\n",
    "model.predict_survival_function(surv_test[X_cols][0:5], plot=True) # fixed indexing problem"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### tf.session close\n",
    "\n",
    "To release resources, we use `model.close_session()` to close session in tensorflow!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "model.close_session()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.1"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}