doc/source/methods/classifierdrift.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[source](../api/alibi_detect.cd.classifier.rst)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Classifier\n",
    "\n",
    "## Overview\n",
    "\n",
    "The classifier-based drift detector [Lopez-Paz and Oquab, 2017](https://openreview.net/forum?id=SJkXfE5xx) simply tries to correctly classify instances from the reference data vs. the test set. If the classifier does not manage to significantly distinguish the reference data from the test set according to a chosen metric (defaults to the classifier accuracy), then no drift occurs. If it can, the test set is different from the reference data and drift is flagged. To leverage all the available reference and test data, stratified cross-validation can be applied and the out-of-fold predictions are used to compute the drift metric. Note that a new classifier is trained for each test set or even each fold within the test set."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Usage\n",
    "\n",
    "### Initialize\n",
    "\n",
    "Arguments:\n",
    "\n",
    "* `x_ref`: Data used as reference distribution.\n",
    "\n",
    "* `model`: Classification model used for drift detection. Both **TensorFlow** and **PyTorch** models are supported.\n",
    "\n",
    "\n",
    "Keyword arguments:\n",
    "\n",
    "* `backend`: Specify the backend (*tensorflow* or *pytorch*). This depends on the framework of the `model`. Defaults to *tensorflow*.\n",
    "\n",
    "* `threshold`: Threshold for the drift metric (default is accuracy). Values above the threshold are classified as drift.\n",
    "\n",
    "* `preprocess_x_ref`: Whether to already apply the (optional) preprocessing step to the reference data at initialization and store the preprocessed data. Dependent on the preprocessing step, this can reduce the computation time for the predict step significantly, especially when the reference dataset is large. Defaults to *True*. It is possible that it needs to be set to *False* if the preprocessing step requires statistics from both the reference and test data, such as the mean or standard deviation.\n",
    "\n",
    "* `update_x_ref`: Reference data can optionally be updated to the last N instances seen by the detector or via [reservoir sampling](https://en.wikipedia.org/wiki/Reservoir_sampling) with size N. For the former, the parameter equals *{'last': N}* while for reservoir sampling *{'reservoir_sampling': N}* is passed.\n",
    "\n",
    "* `preprocess_fn`: Function to preprocess the data before computing the data drift metrics.\n",
    "\n",
    "* `metric_fn`: Function computing the drift metric. Takes `y_true` and `y_pred` as input and returns a float: *metric_fn(y_true, y_pred)*. Defaults to accuracy.\n",
    "\n",
    "* `metric_name`: Optional name for the `metric_fn` used in the return dict. Defaults to `metric_fn.__name__`.\n",
    "\n",
    "* `train_size`: Optional fraction (float between 0 and 1) of the dataset used to train the classifier. The drift is detected on *1 - train_size*. Cannot be used in combination with `n_folds`.\n",
    "\n",
    "* `n_folds`: Optional number of stratified folds used for training. The metric is then calculated on all the out-of-fold predictions. This allows to leverage all the reference and test data for drift detection at the expense of longer computation. If both `train_size` and `n_folds` are specified, `n_folds` is prioritized.\n",
    "\n",
    "* `seed`: Optional random seed for fold selection.\n",
    "\n",
    "* `optimizer`: Optimizer used during training of the classifier. From `torch.optim` for PyTorch and `tf.keras.optimizers` for TensorFlow.\n",
    "\n",
    "* `learning_rate`: Learning rate for the optimizer.\n",
    "\n",
    "* `batch_size`: Batch size used during training of the classifier.\n",
    "\n",
    "* `epochs`: Number of training epochs for the classifier. Applies to each fold if `n_folds` is specified.\n",
    "\n",
    "* `verbose`: Verbosity level during the training of the classifier. 0 is silent, 1 a progress bar and 2 prints the statistics after each epoch.\n",
    "\n",
    "* `train_kwargs`: Optional additional kwargs for *model.fit()* when fitting the classifier for TensorFlow or for the built-in PyTorch trainer function (`from alibi_detect.models.pytorch import trainer`).\n",
    "\n",
    "* `data_type`: Optionally specify the data type (e.g. tabular, image or time-series). Added to metadata.\n",
    "\n",
    "\n",
    "Additional TensorFlow keyword arguments:\n",
    "\n",
    "* `compile_kwargs`: Optional additional kwargs for *model.compile()* when compiling the classifier.\n",
    "\n",
    "\n",
    "Additional PyTorch keyword arguments:\n",
    "\n",
    "* `device`: *cuda* or *gpu* to use the GPU and *cpu* for the CPU. If the device is not specified, the detector will try to leverage the GPU if possible and otherwise fall back on CPU.\n",
    "\n",
    "\n",
    "Initialized **TensorFlow** drift detector example:\n",
    "\n",
    "```python\n",
    "import tensorflow as tf\n",
    "from tensorflow.keras.layers import Conv2D, Dense, Flatten, Input\n",
    "from alibi_detect.cd import ClassifierDrift\n",
    "\n",
    "model = tf.keras.Sequential(\n",
    "  [\n",
    "      Input(shape=(32, 32, 3)),\n",
    "      Conv2D(8, 4, strides=2, padding='same', activation=tf.nn.relu),\n",
    "      Conv2D(16, 4, strides=2, padding='same', activation=tf.nn.relu),\n",
    "      Conv2D(32, 4, strides=2, padding='same', activation=tf.nn.relu),\n",
    "      Flatten(),\n",
    "      Dense(2, activation='softmax')\n",
    "  ]\n",
    ")\n",
    "\n",
    "cd = ClassifierDrift(x_ref, model, threshold=.55, n_folds=5, epochs=2)\n",
    "```\n",
    "\n",
    "\n",
    "A similar detector using **PyTorch**:\n",
    "\n",
    "```python\n",
    "import torch.nn as nn\n",
    "\n",
    "model = nn.Sequential(\n",
    "    nn.Conv2d(3, 8, 4, stride=2, padding=0),\n",
    "    nn.ReLU(),\n",
    "    nn.Conv2d(8, 16, 4, stride=2, padding=0),\n",
    "    nn.ReLU(),\n",
    "    nn.Conv2d(16, 32, 4, stride=2, padding=0),\n",
    "    nn.ReLU(),\n",
    "    nn.Flatten(),\n",
    "    nn.Linear(128, 2)\n",
    ")\n",
    "\n",
    "cd = ClassifierDrift(x_ref, model, threshold=.55, backend='pytorch')\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Detect Drift\n",
    "\n",
    "We detect data drift by simply calling `predict` on a batch of instances `x`. `return_metric` equal to *True* will also return the drift metric (e.g. accuracy) and the threshold used by the detector.\n",
    "\n",
    "The prediction takes the form of a dictionary with `meta` and `data` keys. `meta` contains the detector's metadata while `data` is also a dictionary which contains the actual predictions stored in the following keys:\n",
    "\n",
    "* `is_drift`: 1 if the sample tested has drifted from the reference data and 0 otherwise.\n",
    "\n",
    "* `threshold`: user-defined drift threshold for the chosen drift metric.\n",
    "\n",
    "* `metric_fn.__name__` or the optional `metric_name` kwarg value: drift metric value if `return_metric` equals *True*.\n",
    "\n",
    "\n",
    "```python\n",
    "preds = cd.predict(x)\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Saving and loading\n",
    "\n",
    "The drift detectors can be saved and loaded in the same way as other detectors:\n",
    "\n",
    "```python\n",
    "from alibi_detect.utils.saving import save_detector, load_detector\n",
    "\n",
    "filepath = 'my_path'\n",
    "save_detector(cd, filepath)\n",
    "cd = load_detector(filepath)\n",
    "```\n",
    "\n",
    "Currently on the **TensorFlow** backend is supported for `save_detector` and `load_detector`. Adding **PyTorch** support is a near term priority."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Examples\n",
    "\n",
    "[Drift detection on CIFAR10](../examples/cd_clf_cifar10.nblink)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}