doc/source/methods/chisquaredrift.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[source](../api/alibi_detect.cd.chisquare.rst)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Chi-Squared\n",
    "\n",
    "## Overview\n",
    "\n",
    "The drift detector applies feature-wise [Chi-Squared](https://en.wikipedia.org/wiki/Chi-squared_test) tests for the categorical features. For multivariate data, the obtained p-values for each feature are aggregated either via the [Bonferroni](https://mathworld.wolfram.com/BonferroniCorrection.html) or the [False Discovery Rate](http://www.math.tau.ac.il/~ybenja/MyPapers/benjamini_hochberg1995.pdf) (FDR) correction. The Bonferroni correction is more conservative and controls for the probability of at least one false positive. The FDR correction on the other hand allows for an expected fraction of false positives to occur. Similarly to the other drift detectors, a preprocessing steps could be applied, but the output features need to be categorical."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Usage\n",
    "\n",
    "### Initialize\n",
    "\n",
    "\n",
    "Parameters:\n",
    "\n",
    "* `p_val`: p-value used for significance of the Chi-Squared test for each feature. If the FDR correction method is used, this corresponds to the acceptable q-value.\n",
    "\n",
    "* `X_ref`: Data used as reference distribution.\n",
    "\n",
    "* `preprocess_X_ref`: Whether to already count and store the number of instances for each possible category of each variable of the reference data `X_ref` when initializing the detector. If a preprocessing step is specified, the step will be applied first. Defaults to *True*. It is possible that it needs to be set to *False* if the preprocessing step requires statistics from both the reference and test data.\n",
    "\n",
    "* `categories_per_feature`: Optional dictionary with as keys the feature column index and as values the number of possible categorical values for that feature. E.g.: *{0: 5, 1: 9, 2: 7}*. If it is not specified, `categories_per_feature` is inferred from `X_ref`.\n",
    "\n",
    "* `update_X_ref`: Reference data can optionally be updated to the last N instances seen by the detector or via [reservoir sampling](https://en.wikipedia.org/wiki/Reservoir_sampling) with size N. For the former, the parameter equals *{'last': N}* while for reservoir sampling *{'reservoir_sampling': N}* is passed.\n",
    "\n",
    "* `preprocess_fn`: Function to preprocess the data before computing the data drift metrics. Typically a dimensionality reduction technique. Needs to return categorical features for the Chi-Squared detector.\n",
    "\n",
    "* `preprocess_kwargs`: Keyword arguments for `preprocess_fn`.\n",
    "\n",
    "* `correction`: Correction type for multivariate data. Either *'bonferroni'* or *'fdr'* (False Discovery Rate).\n",
    "\n",
    "* `n_features`: Number of features used in the Chi-Squared test. No need to pass it if no preprocessing takes place. In case of a preprocessing step, this can also be inferred automatically but could be more expensive to compute.\n",
    "\n",
    "* `n_infer`: If the number of features needs to be inferred after the preprocessing step, we can specify the number of instances used to infer the number of features from since this can depend on the specific preprocessing step.\n",
    "\n",
    "* `data_type`: can specify data type added to metadata. E.g. *'tabular'*.\n",
    "\n",
    "Initialized drift detector example:\n",
    "\n",
    "```python\n",
    "from alibi_detect.cd import ChiSquareDrift\n",
    "\n",
    "cd = ChiSquareDrift(p_val=0.05, X_ref=X_ref)\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Detect Drift\n",
    "\n",
    "We detect data drift by simply calling `predict` on a batch of instances `X`. We can return the feature-wise p-values before the multivariate correction by setting `return_p_val` to *True*. The drift can also be detected at the feature level by setting `drift_type` to *'feature'*. No multivariate correction will take place since we return the output of *n_features* univariate tests. For drift detection on all the features combined with the correction, use *'batch'*. `return_p_val` equal to *True* will also return the threshold used by the detector (either for the univariate case or after the multivariate correction).\n",
    "\n",
    "The prediction takes the form of a dictionary with `meta` and `data` keys. `meta` contains the detector's metadata while `data` is also a dictionary which contains the actual predictions stored in the following keys:\n",
    "\n",
    "* `is_drift`: 1 if the sample tested has drifted from the reference data and 0 otherwise.\n",
    "\n",
    "* `p_val`: contains feature-level p-values if `return_p_val` equals *True*.\n",
    "\n",
    "* `threshold`: for feature-level drift detection the threshold equals the p-value used for the significance of the Chi-Square test. Otherwise the threshold after the multivariate correction (either *bonferroni* or *fdr*) is returned.\n",
    "\n",
    "* `distance`: feature-wise Chi-Square test statistics between the reference data and the new batch if `return_distance` equals *True*.\n",
    "\n",
    "\n",
    "```python\n",
    "preds_drift = cd.predict(X, drift_type='batch', return_p_val=True, return_distance=True)\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Saving and loading\n",
    "\n",
    "The drift detectors can be saved and loaded in the same way as other detectors when using the built-in preprocessing steps (`alibi_detect.cd.preprocess.UAE` and `alibi_detect.cd.preprocess.HiddenOutput`) or no preprocessing at all:\n",
    "\n",
    "```python\n",
    "from alibi_detect.utils.saving import save_detector, load_detector\n",
    "\n",
    "filepath = 'my_path'\n",
    "save_detector(cd, filepath)\n",
    "cd = load_detector(filepath)\n",
    "```\n",
    "\n",
    "A custom preprocessing step can be passed as follows:\n",
    "\n",
    "```python\n",
    "cd = load_detector(filepath, **{'preprocess_kwargs': preprocess_kwargs})\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Examples\n",
    "\n",
    "[Drift detection on income prediction](../examples/cd_chi2ks_adult.nblink)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}