site/en/guide/core/logistic_regression_core.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FhGuhbZ6M5tl"
      },
      "source": [
        "##### Copyright 2022 The TensorFlow Authors."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "cellView": "form",
        "id": "AwOEIRJC6Une"
      },
      "outputs": [],
      "source": [
        "#@title Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "#\n",
        "# https://www.apache.org/licenses/LICENSE-2.0\n",
        "#\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EIdT9iu_Z4Rb"
      },
      "source": [
        "# Logistic regression for binary classification with Core APIs"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bBIlTPscrIT9"
      },
      "source": [
        "<table class=\"tfo-notebook-buttons\" align=\"left\">\n",
        "  <td>\n",
        "    <a target=\"_blank\" href=\"https://www.tensorflow.org/guide/core/logistic_regression_core\"><img src=\"https://www.tensorflow.org/images/tf_logo_32px.png\" />View on TensorFlow.org</a>\n",
        "  </td>\n",
        "  <td>\n",
        "    <a target=\"_blank\" href=\"https://colab.research.google.com/github/tensorflow/docs/blob/master/site/en/guide/core/logistic_regression_core.ipynb\"><img src=\"https://www.tensorflow.org/images/colab_logo_32px.png\" />Run in Google Colab</a>\n",
        "  </td>\n",
        "  <td>\n",
        "    <a target=\"_blank\" href=\"https://github.com/tensorflow/docs/blob/master/site/en/guide/core/logistic_regression_core.ipynb\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" />View source on GitHub</a>\n",
        "  </td>\n",
        "  <td>\n",
        "    <a href=\"https://storage.googleapis.com/tensorflow_docs/docs/site/en/guide/core/logistic_regression_core.ipynb\"><img src=\"https://www.tensorflow.org/images/download_logo_32px.png\" />Download notebook</a>\n",
        "  </td>\n",
        "</table>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DauaqJ7WhIhO"
      },
      "source": [
        "This guide demonstrates how to use the [TensorFlow Core low-level APIs](https://www.tensorflow.org/guide/core) to perform [binary classification](https://developers.google.com/machine-learning/glossary#binary_classification){:.external} with [logistic regression](https://developers.google.com/machine-learning/crash-course/logistic-regression/){:.external}. It uses the [Wisconsin Breast Cancer Dataset](https://archive.ics.uci.edu/ml/datasets/breast+cancer+wisconsin+(original)){:.external} for tumor classification.\n",
        "\n",
        "[Logistic regression](https://developers.google.com/machine-learning/crash-course/logistic-regression/){:.external} is one of the most popular algorithms for binary classification. Given a set of examples with features, the goal of logistic regression is to output values between 0 and 1, which can be interpreted as the probabilities of each example belonging to a particular  class. "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nchsZfwEVtVs"
      },
      "source": [
        "## Setup\n",
        "\n",
        "This tutorial uses [pandas](https://pandas.pydata.org){:.external} for reading a CSV file into a [DataFrame](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.html){:.external}, [seaborn](https://seaborn.pydata.org){:.external} for plotting a pairwise relationship in a dataset, [Scikit-learn](https://scikit-learn.org/){:.external} for computing a confusion matrix, and [matplotlib](https://matplotlib.org/){:.external} for creating visualizations."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "5lZoUK6AVTos"
      },
      "outputs": [],
      "source": [
        "!pip install -q seaborn"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "1rRo8oNqZ-Rj"
      },
      "outputs": [],
      "source": [
        "import tensorflow as tf\n",
        "import pandas as pd\n",
        "import matplotlib\n",
        "from matplotlib import pyplot as plt\n",
        "import seaborn as sns\n",
        "import sklearn.metrics as sk_metrics\n",
        "import tempfile\n",
        "import os\n",
        "\n",
        "# Preset matplotlib figure sizes.\n",
        "matplotlib.rcParams['figure.figsize'] = [9, 6]\n",
        "\n",
        "print(tf.__version__)\n",
        "# To make the results reproducible, set the random seed value.\n",
        "tf.random.set_seed(22)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gFh9ne3FZ-On"
      },
      "source": [
        "## Load the data\n",
        "\n",
        "Next, load the [Wisconsin Breast Cancer Dataset](https://archive.ics.uci.edu/ml/datasets/breast+cancer+wisconsin+(original)){:.external} from the [UCI Machine Learning Repository](https://archive.ics.uci.edu/ml/){:.external}. This dataset contains various features such as a tumor's radius, texture, and concavity."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "CiX2FI4gZtTt"
      },
      "outputs": [],
      "source": [
        "url = 'https://archive.ics.uci.edu/ml/machine-learning-databases/breast-cancer-wisconsin/wdbc.data'\n",
        "\n",
        "features = ['radius', 'texture', 'perimeter', 'area', 'smoothness', 'compactness',\n",
        "            'concavity', 'concave_poinits', 'symmetry', 'fractal_dimension']\n",
        "column_names = ['id', 'diagnosis']\n",
        "\n",
        "for attr in ['mean', 'ste', 'largest']:\n",
        "  for feature in features:\n",
        "    column_names.append(feature + \"_\" + attr)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "A3VR1aTP92nV"
      },
      "source": [
        "Read the dataset into a pandas [DataFrame](){:.external} using [`pandas.read_csv`](https://pandas.pydata.org/docs/reference/api/pandas.read_csv.html){:.external}:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "uvR2Bzb691lJ"
      },
      "outputs": [],
      "source": [
        "dataset = pd.read_csv(url, names=column_names)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "YB9eq6Zq-IZ4"
      },
      "outputs": [],
      "source": [
        "dataset.info()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0_Z1V6Dg-La_"
      },
      "source": [
        "Display the first five rows:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "hWxktwbv-KPp"
      },
      "outputs": [],
      "source": [
        "dataset.head()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "s4-Wn2jzVC1W"
      },
      "source": [
        "Split the dataset into training and test sets using [`pandas.DataFrame.sample`](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.sample.html){:.external}, [`pandas.DataFrame.drop`](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.drop.html){:.external} and [`pandas.DataFrame.iloc`](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.iloc.html){:.external}. Make sure to split the features from the target labels. The test set is used to evaluate your model's generalizability to unseen data."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "m2O60B-IVG9Q"
      },
      "outputs": [],
      "source": [
        "train_dataset = dataset.sample(frac=0.75, random_state=1)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "i06vHFv_QB24"
      },
      "outputs": [],
      "source": [
        "len(train_dataset)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "19JaochhaQ3m"
      },
      "outputs": [],
      "source": [
        "test_dataset = dataset.drop(train_dataset.index)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "LmHRcbAfaSag"
      },
      "outputs": [],
      "source": [
        "len(test_dataset)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "w6JxBhBc_wwO"
      },
      "outputs": [],
      "source": [
        "# The `id` column can be dropped since each row is unique\n",
        "x_train, y_train = train_dataset.iloc[:, 2:], train_dataset.iloc[:, 1]\n",
        "x_test, y_test = test_dataset.iloc[:, 2:], test_dataset.iloc[:, 1]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3MWuJTKEDM-f"
      },
      "source": [
        "## Preprocess the data\n",
        "\n",
        "This dataset contains the mean, standard error, and largest values for each of the 10 tumor measurements collected per example. The `\"diagnosis\"` target column is a categorical variable with `'M'` indicating a malignant tumor and `'B'` indicating a benign tumor diagnosis. This column needs to be converted into a numerical binary format for model training.\n",
        "\n",
        "The [`pandas.Series.map`](https://pandas.pydata.org/docs/reference/api/pandas.Series.map.html){:.external} function is useful for mapping binary values to the categories.\n",
        "\n",
        "The dataset should also be converted to a tensor with the `tf.convert_to_tensor` function after the preprocessing is complete."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "JEJHhN65a2VV"
      },
      "outputs": [],
      "source": [
        "y_train, y_test = y_train.map({'B': 0, 'M': 1}), y_test.map({'B': 0, 'M': 1})\n",
        "x_train, y_train = tf.convert_to_tensor(x_train, dtype=tf.float32), tf.convert_to_tensor(y_train, dtype=tf.float32)\n",
        "x_test, y_test = tf.convert_to_tensor(x_test, dtype=tf.float32), tf.convert_to_tensor(y_test, dtype=tf.float32)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "J4ubs136WLNp"
      },
      "source": [
        "Use [`seaborn.pairplot`](https://seaborn.pydata.org/generated/seaborn.pairplot.html){:.external} to review the joint distribution of a few pairs of mean-based features from the training set and observe how they relate to the target:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "oRKO_x8gWKv-"
      },
      "outputs": [],
      "source": [
        "sns.pairplot(train_dataset.iloc[:, 1:6], hue = 'diagnosis', diag_kind='kde');"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5YOG5iKYKW_3"
      },
      "source": [
        "This pairplot demonstrates that certain features such as radius, perimeter and area are highly correlated. This is expected since the tumor radius is directly involved in the computation of both perimeter and area. Additionally, note that malignant diagnoses seem to be more right-skewed for many of the features.\n",
        "\n",
        "Make sure to also check the overall statistics. Note how each feature covers a vastly different range of values."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "yi2FzC3T21jR"
      },
      "outputs": [],
      "source": [
        "train_dataset.describe().transpose()[:10]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_8pDCIFjMla8"
      },
      "source": [
        "Given the inconsistent ranges, it is beneficial to standardize the data such that each feature has a zero mean and unit variance. This process is called [normalization](https://developers.google.com/machine-learning/glossary#normalization){:.external}."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "FrzKNFNjLQDl"
      },
      "outputs": [],
      "source": [
        "class Normalize(tf.Module):\n",
        "  def __init__(self, x):\n",
        "    # Initialize the mean and standard deviation for normalization\n",
        "    self.mean = tf.Variable(tf.math.reduce_mean(x, axis=0))\n",
        "    self.std = tf.Variable(tf.math.reduce_std(x, axis=0))\n",
        "\n",
        "  def norm(self, x):\n",
        "    # Normalize the input\n",
        "    return (x - self.mean)/self.std\n",
        "\n",
        "  def unnorm(self, x):\n",
        "    # Unnormalize the input\n",
        "    return (x * self.std) + self.mean\n",
        "\n",
        "norm_x = Normalize(x_train)\n",
        "x_train_norm, x_test_norm = norm_x.norm(x_train), norm_x.norm(x_test)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6o3CrycBXA2s"
      },
      "source": [
        "## Logistic regression\n",
        "\n",
        "Before building a logistic regression model, it is crucial to understand the method's differences compared to traditional linear regression.\n",
        "\n",
        "### Logistic regression fundamentals\n",
        "\n",
        "Linear regression returns a linear combination of its inputs; this output is unbounded. The output of a [logistic regression](https://developers.google.com/machine-learning/glossary#logistic_regression){:.external} is in the `(0, 1)` range. For each example, it represents the probability that the example belongs to the _positive_ class.\n",
        "\n",
        "Logistic regression maps the continuous outputs of traditional linear regression, `(-∞, ∞)`, to probabilities, `(0, 1)`. This transformation is also symmetric so that flipping the sign of the linear output results in the inverse of the original probability.\n",
        "\n",
        "Let $Y$ denote the probability of being in class `1` (the tumor is malignant). The desired mapping can be achieved by interpreting the linear regression output as the [log odds](https://developers.google.com/machine-learning/glossary#log-odds){:.external} ratio of being in class `1` as opposed to class `0`:\n",
        "\n",
        "$$\\ln(\\frac{Y}{1-Y}) = wX + b$$\n",
        "\n",
        "By setting $wX + b = z$, this equation can then be solved for $Y$:\n",
        "\n",
        "$$Y = \\frac{e^{z}}{1 + e^{z}} = \\frac{1}{1 + e^{-z}}$$\n",
        "\n",
        "The expression $\\frac{1}{1 + e^{-z}}$ is known as the [sigmoid function](https://developers.google.com/machine-learning/glossary#sigmoid_function){:.external} $\\sigma(z)$. Hence, the equation for logistic regression can be written as $Y = \\sigma(wX + b)$.\n",
        "\n",
        "The dataset in this tutorial deals with a high-dimensional feature matrix. Therefore, the above equation must be rewritten in a matrix vector form as follows:\n",
        "\n",
        "$${\\mathrm{Y}} = \\sigma({\\mathrm{X}}w + b)$$\n",
        "\n",
        "where:\n",
        "\n",
        "* $\\underset{m\\times 1}{\\mathrm{Y}}$: a target vector\n",
        "* $\\underset{m\\times n}{\\mathrm{X}}$: a feature matrix\n",
        "* $\\underset{n\\times 1}w$: a weight vector\n",
        "* $b$: a bias\n",
        "* $\\sigma$: a sigmoid function applied to each element of the output vector\n",
        "\n",
        "Start by visualizing the sigmoid function, which transforms the linear output, `(-∞, ∞)`, to fall between `0` and `1`. The sigmoid function is available in `tf.math.sigmoid`."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ThHaV_RmucZl"
      },
      "outputs": [],
      "source": [
        "x = tf.linspace(-10, 10, 500)\n",
        "x = tf.cast(x, tf.float32)\n",
        "f = lambda x : (1/20)*x + 0.6\n",
        "plt.plot(x, tf.math.sigmoid(x))\n",
        "plt.ylim((-0.1,1.1))\n",
        "plt.title(\"Sigmoid function\");"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VMXEhrZuKECV"
      },
      "source": [
        "### The log loss function\n",
        "\n",
        "The  [log loss](https://developers.google.com/machine-learning/glossary#Log_Loss){:.external}, or binary cross-entropy loss, is the ideal loss function for a binary classification problem with logistic regression. For each example, the log loss quantifies the similarity between a predicted probability and the example's true value. It is determined by the following equation:\n",
        "\n",
        "$$L = -\\frac{1}{m}\\sum_{i=1}^{m}y_i\\cdot\\log(\\hat{y}_i) + (1- y_i)\\cdot\\log(1 - \\hat{y}_i)$$\n",
        "\n",
        "where:\n",
        "\n",
        "* $\\hat{y}$: a vector of predicted probabilities\n",
        "* $y$: a vector of true targets\n",
        "\n",
        "You can use the `tf.nn.sigmoid_cross_entropy_with_logits` function to compute the log loss. This function automatically applies the sigmoid activation to the regression output:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "JVBInnSqS36W"
      },
      "outputs": [],
      "source": [
        "def log_loss(y_pred, y):\n",
        "  # Compute the log loss function\n",
        "  ce = tf.nn.sigmoid_cross_entropy_with_logits(labels=y, logits=y_pred)\n",
        "  return tf.reduce_mean(ce)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Q_mutLj0KNUb"
      },
      "source": [
        "### The gradient descent update rule\n",
        "\n",
        "The TensorFlow Core APIs support automatic differentiation with `tf.GradientTape`. If you are curious about the mathematics behind the logistic regression [gradient updates](https://developers.google.com/machine-learning/glossary#gradient_descent){:.external}, here is a short explanation:\n",
        "\n",
        "In the above equation for the log loss, recall that each $\\hat{y}_i$ can be rewritten in terms of the inputs as $\\sigma({\\mathrm{X_i}}w + b)$.\n",
        "\n",
        "The goal is to find a $w^*$ and $b^*$ that minimize the log loss:\n",
        "\n",
        "$$L = -\\frac{1}{m}\\sum_{i=1}^{m}y_i\\cdot\\log(\\sigma({\\mathrm{X_i}}w + b)) + (1- y_i)\\cdot\\log(1 - \\sigma({\\mathrm{X_i}}w + b))$$\n",
        "\n",
        "By taking the gradient $L$ with respect to $w$, you get the following:\n",
        "\n",
        "$$\\frac{\\partial L}{\\partial w} = \\frac{1}{m}(\\sigma({\\mathrm{X}}w + b) - y)X$$\n",
        "\n",
        "By taking the gradient $L$ with respect to $b$, you get the following:\n",
        "\n",
        "$$\\frac{\\partial L}{\\partial b} = \\frac{1}{m}\\sum_{i=1}^{m}\\sigma({\\mathrm{X_i}}w + b) - y_i$$"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "uTCndUecKZho"
      },
      "source": [
        "Now, build the logistic regression model."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "c0sXM7qLlKfZ"
      },
      "outputs": [],
      "source": [
        "class LogisticRegression(tf.Module):\n",
        "\n",
        "  def __init__(self):\n",
        "    self.built = False\n",
        "    \n",
        "  def __call__(self, x, train=True):\n",
        "    # Initialize the model parameters on the first call\n",
        "    if not self.built:\n",
        "      # Randomly generate the weights and the bias term\n",
        "      rand_w = tf.random.uniform(shape=[x.shape[-1], 1], seed=22)\n",
        "      rand_b = tf.random.uniform(shape=[], seed=22)\n",
        "      self.w = tf.Variable(rand_w)\n",
        "      self.b = tf.Variable(rand_b)\n",
        "      self.built = True\n",
        "    # Compute the model output\n",
        "    z = tf.add(tf.matmul(x, self.w), self.b)\n",
        "    z = tf.squeeze(z, axis=1)\n",
        "    if train:\n",
        "      return z\n",
        "    return tf.sigmoid(z)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eObQu9fDnXGL"
      },
      "source": [
        "To validate, make sure the untrained model outputs values in the range of `(0, 1)` for a small subset of the training data."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "5bIovC0Z4QHJ"
      },
      "outputs": [],
      "source": [
        "log_reg = LogisticRegression()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "QJ2ievISyf0p"
      },
      "outputs": [],
      "source": [
        "y_pred = log_reg(x_train_norm[:5], train=False)\n",
        "y_pred.numpy()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PribnwDHUksC"
      },
      "source": [
        "Next, write an accuracy function to calculate the proportion of correct classifications during training. In order to retrieve the classifications from the predicted probabilities, set a threshold for which all probabilities higher than the threshold belong to class `1`. This is a configurable hyperparameter that can be set to `0.5` as a default."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ssnVcKg7oMe6"
      },
      "outputs": [],
      "source": [
        "def predict_class(y_pred, thresh=0.5):\n",
        "  # Return a tensor with  `1` if `y_pred` > `0.5`, and `0` otherwise\n",
        "  return tf.cast(y_pred > thresh, tf.float32)\n",
        "\n",
        "def accuracy(y_pred, y):\n",
        "  # Return the proportion of matches between `y_pred` and `y`\n",
        "  y_pred = tf.math.sigmoid(y_pred)\n",
        "  y_pred_class = predict_class(y_pred)\n",
        "  check_equal = tf.cast(y_pred_class == y,tf.float32)\n",
        "  acc_val = tf.reduce_mean(check_equal)\n",
        "  return acc_val"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "J_0KHQ25_2dF"
      },
      "source": [
        "### Train the model\n",
        "\n",
        "Using mini-batches for training provides both memory efficiency and faster convergence. The `tf.data.Dataset` API has useful functions for batching and shuffling. The API enables you to build complex input pipelines from simple, reusable pieces. "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "vJD7-4U0etqa"
      },
      "outputs": [],
      "source": [
        "batch_size = 64\n",
        "train_dataset = tf.data.Dataset.from_tensor_slices((x_train_norm, y_train))\n",
        "train_dataset = train_dataset.shuffle(buffer_size=x_train.shape[0]).batch(batch_size)\n",
        "test_dataset = tf.data.Dataset.from_tensor_slices((x_test_norm, y_test))\n",
        "test_dataset = test_dataset.shuffle(buffer_size=x_test.shape[0]).batch(batch_size)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "sLiWZZPBSDip"
      },
      "source": [
        "Now write a training loop for the logistic regression model. The loop utilizes the log loss function and its gradients with respect to the input in order to iteratively update the model's parameters."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "jNC3D1DGsGgK"
      },
      "outputs": [],
      "source": [
        "# Set training parameters\n",
        "epochs = 200\n",
        "learning_rate = 0.01\n",
        "train_losses, test_losses = [], []\n",
        "train_accs, test_accs = [], []\n",
        "\n",
        "# Set up the training loop and begin training\n",
        "for epoch in range(epochs):\n",
        "  batch_losses_train, batch_accs_train = [], []\n",
        "  batch_losses_test, batch_accs_test = [], []\n",
        "\n",
        "  # Iterate over the training data\n",
        "  for x_batch, y_batch in train_dataset:\n",
        "    with tf.GradientTape() as tape:\n",
        "      y_pred_batch = log_reg(x_batch)\n",
        "      batch_loss = log_loss(y_pred_batch, y_batch)\n",
        "    batch_acc = accuracy(y_pred_batch, y_batch)\n",
        "    # Update the parameters with respect to the gradient calculations\n",
        "    grads = tape.gradient(batch_loss, log_reg.variables)\n",
        "    for g,v in zip(grads, log_reg.variables):\n",
        "      v.assign_sub(learning_rate * g)\n",
        "    # Keep track of batch-level training performance\n",
        "    batch_losses_train.append(batch_loss)\n",
        "    batch_accs_train.append(batch_acc)\n",
        "\n",
        "  # Iterate over the testing data\n",
        "  for x_batch, y_batch in test_dataset:\n",
        "    y_pred_batch = log_reg(x_batch)\n",
        "    batch_loss = log_loss(y_pred_batch, y_batch)\n",
        "    batch_acc = accuracy(y_pred_batch, y_batch)\n",
        "    # Keep track of batch-level testing performance\n",
        "    batch_losses_test.append(batch_loss)\n",
        "    batch_accs_test.append(batch_acc)\n",
        "\n",
        "  # Keep track of epoch-level model performance\n",
        "  train_loss, train_acc = tf.reduce_mean(batch_losses_train), tf.reduce_mean(batch_accs_train)\n",
        "  test_loss, test_acc = tf.reduce_mean(batch_losses_test), tf.reduce_mean(batch_accs_test)\n",
        "  train_losses.append(train_loss)\n",
        "  train_accs.append(train_acc)\n",
        "  test_losses.append(test_loss)\n",
        "  test_accs.append(test_acc)\n",
        "  if epoch % 20 == 0:\n",
        "    print(f\"Epoch: {epoch}, Training log loss: {train_loss:.3f}\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NoLiAg7fYft7"
      },
      "source": [
        "### Performance evaluation\n",
        "\n",
        "Observe the changes in your model's loss and accuracy over time. "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "mv3oCQPvWhr0"
      },
      "outputs": [],
      "source": [
        "plt.plot(range(epochs), train_losses, label = \"Training loss\")\n",
        "plt.plot(range(epochs), test_losses, label = \"Testing loss\")\n",
        "plt.xlabel(\"Epoch\")\n",
        "plt.ylabel(\"Log loss\")\n",
        "plt.legend()\n",
        "plt.title(\"Log loss vs training iterations\");"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "D2HDVGLPODIE"
      },
      "outputs": [],
      "source": [
        "plt.plot(range(epochs), train_accs, label = \"Training accuracy\")\n",
        "plt.plot(range(epochs), test_accs, label = \"Testing accuracy\")\n",
        "plt.xlabel(\"Epoch\")\n",
        "plt.ylabel(\"Accuracy (%)\")\n",
        "plt.legend()\n",
        "plt.title(\"Accuracy vs training iterations\");"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "jonKhUzuPyfa"
      },
      "outputs": [],
      "source": [
        "print(f\"Final training log loss: {train_losses[-1]:.3f}\")\n",
        "print(f\"Final testing log Loss: {test_losses[-1]:.3f}\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "d3DF4qyrPyke"
      },
      "outputs": [],
      "source": [
        "print(f\"Final training accuracy: {train_accs[-1]:.3f}\")\n",
        "print(f\"Final testing accuracy: {test_accs[-1]:.3f}\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yrj1TbOJasjA"
      },
      "source": [
        "The model demonstrates a high accuracy and a low loss when it comes to classifying tumors in the training dataset and also generalizes well to the unseen test data. To go one step further, you can explore error rates that give more insight beyond the overall accuracy score. The two most popular error rates for binary classification problems are the false positive rate (FPR) and the false negative rate (FNR).\n",
        "\n",
        "For this problem, the FPR is the proportion of malignant tumor predictions amongst tumors that are actually benign. Conversely, the FNR is the proportion of benign tumor predictions among tumors that are actually malignant.\n",
        "\n",
        "Compute a confusion matrix using [`sklearn.metrics.confusion_matrix`](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.confusion_matrix.html#sklearn.metrics.confusion_matrix){:.external}, which evaluates the accuracy of the classification, and use matplotlib to display the matrix:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "OJO7YkA8ZDMU"
      },
      "outputs": [],
      "source": [
        "def show_confusion_matrix(y, y_classes, typ):\n",
        "  # Compute the confusion matrix and normalize it\n",
        "  plt.figure(figsize=(10,10))\n",
        "  confusion = sk_metrics.confusion_matrix(y.numpy(), y_classes.numpy())\n",
        "  confusion_normalized = confusion / confusion.sum(axis=1, keepdims=True)\n",
        "  axis_labels = range(2)\n",
        "  ax = sns.heatmap(\n",
        "      confusion_normalized, xticklabels=axis_labels, yticklabels=axis_labels,\n",
        "      cmap='Blues', annot=True, fmt='.4f', square=True)\n",
        "  plt.title(f\"Confusion matrix: {typ}\")\n",
        "  plt.ylabel(\"True label\")\n",
        "  plt.xlabel(\"Predicted label\")\n",
        "\n",
        "y_pred_train, y_pred_test = log_reg(x_train_norm, train=False), log_reg(x_test_norm, train=False)\n",
        "train_classes, test_classes = predict_class(y_pred_train), predict_class(y_pred_test)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "OQ5DFcleiDFm"
      },
      "outputs": [],
      "source": [
        "show_confusion_matrix(y_train, train_classes, 'Training')"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "gtfcsAp_iCNR"
      },
      "outputs": [],
      "source": [
        "show_confusion_matrix(y_test, test_classes, 'Testing')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DlivxaDmTnGq"
      },
      "source": [
        "Observe the error rate measurements and interpret their significance in the context of this example. In many medical testing studies such as cancer detection, having a high false positive rate to ensure a low false negative rate is perfectly acceptable and in fact encouraged since the risk of missing a malignant tumor diagnosis (false negative) is a lot worse than misclassifying a benign tumor as malignant (false positive).\n",
        "\n",
        "In order to control for the FPR and FNR, try changing the threshold hyperparameter before classifying the probability predictions. A lower threshold increases the model's overall chances of making a malignant tumor classification. This inevitably increases the number of false positives and the FPR but it also helps to decrease the number of false negatives and the FNR."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7ADEN2rb4Nhj"
      },
      "source": [
        "## Save the model\n",
        "\n",
        "Start by making an export module that takes in raw data and performs the following operations:\n",
        "- Normalization\n",
        "- Probability prediction\n",
        "- Class prediction\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "6KPRHCzg4ZxH"
      },
      "outputs": [],
      "source": [
        "class ExportModule(tf.Module):\n",
        "  def __init__(self, model, norm_x, class_pred):\n",
        "    # Initialize pre- and post-processing functions\n",
        "    self.model = model\n",
        "    self.norm_x = norm_x\n",
        "    self.class_pred = class_pred\n",
        "\n",
        "  @tf.function(input_signature=[tf.TensorSpec(shape=[None, None], dtype=tf.float32)])\n",
        "  def __call__(self, x):\n",
        "    # Run the `ExportModule` for new data points\n",
        "    x = self.norm_x.norm(x)\n",
        "    y = self.model(x, train=False)\n",
        "    y = self.class_pred(y)\n",
        "    return y "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "2YzRclo5-yjO"
      },
      "outputs": [],
      "source": [
        "log_reg_export = ExportModule(model=log_reg,\n",
        "                              norm_x=norm_x,\n",
        "                              class_pred=predict_class)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gtofGIBN_qFd"
      },
      "source": [
        "If you want to save the model at its current state, you can do so with the `tf.saved_model.save` function. To load a saved model and make predictions, use the `tf.saved_model.load` function."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "a4Qum1Ts_pmF"
      },
      "outputs": [],
      "source": [
        "models = tempfile.mkdtemp()\n",
        "save_path = os.path.join(models, 'log_reg_export')\n",
        "tf.saved_model.save(log_reg_export, save_path)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "3KPILr1i_M_c"
      },
      "outputs": [],
      "source": [
        "log_reg_loaded = tf.saved_model.load(save_path)\n",
        "test_preds = log_reg_loaded(x_test)\n",
        "test_preds[:10].numpy()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vgGQuV-yqYZH"
      },
      "source": [
        "## Conclusion\n",
        "\n",
        "This notebook introduced a few techniques to handle a logistic regression problem. Here are a few more tips that may help:\n",
        "\n",
        "- The [TensorFlow Core APIs](https://www.tensorflow.org/guide/core) can be used to build machine learning workflows with high levels of configurability\n",
        "- Analyzing error rates is a great way to gain more insight about a classification model's performance beyond its overall accuracy score.\n",
        "- Overfitting is another common problem for logistic regression models, though it wasn't a problem for this tutorial. Visit the [Overfit and underfit](../../tutorials/keras/overfit_and_underfit.ipynb) tutorial for more help with this.\n",
        "\n",
        "For more examples of using the TensorFlow Core APIs, check out the [guide](https://www.tensorflow.org/guide/core). If you want to learn more about loading and preparing data, see the tutorials on [image data loading](../../tutorials/load_data/images.ipynb) or [CSV data loading](../../tutorials/load_data/csv.ipynb)."
      ]
    }
  ],
  "metadata": {
    "colab": {
      "collapsed_sections": [],
      "name": "logistic_regression_core.ipynb",
      "toc_visible": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}