drift.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "source": [
    "# Drift Detection\n",
    "\n",
    "Drift detection is a crucial part of ML observability. As is the case with other components, drift detection in TurboML is a continuous streaming process. In this notebook, we'll see how to compute data drift (univariate and multivariate) and model drift. \n",
    "\n",
    "For univariate drift detection, by default we're using Adaptive Windowing method, and for multivariate drift detection, by default we're using PCA based reconstruction method. "
   ],
   "id": "0",
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Set up the environment and install TurboML's SDK.\n",
    "We use `turboml-installer` to set up the environment for TurboML's SDK."
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "source": [
    "!pip install -q turboml-installer\n",
    "import turboml_installer ; turboml_installer.install_on_colab()"
   ],
   "execution_count": null,
   "metadata": {},
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "source": [
    "The kernel should now be restarted with TurboML's SDK installed."
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Login to your TurboML instance\n",
    "\n",
    "Note that you can copy and replace this snippet with one from your TurboML homepage."
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "source": [
    "import turboml as tb",
    "\ntb.init(backend_url=BACKEND_URL, api_key=API_KEY)"
   ],
   "execution_count": null,
   "id": "1",
   "metadata": {},
   "outputs": []
  },
  {
   "cell_type": "code",
   "source": [
    "transactions = tb.datasets.FraudDetectionDatasetFeatures().to_online(\n",
    "    \"transactions\", load_if_exists=True\n",
    ")\n",
    "labels = tb.datasets.FraudDetectionDatasetLabels().to_online(\n",
    "    \"transaction_labels\", load_if_exists=True\n",
    ")"
   ],
   "execution_count": null,
   "id": "2",
   "metadata": {},
   "outputs": []
  },
  {
   "cell_type": "code",
   "source": [
    "model = tb.RCF(number_of_trees=50)"
   ],
   "execution_count": null,
   "id": "3",
   "metadata": {},
   "outputs": []
  },
  {
   "cell_type": "code",
   "source": [
    "numerical_fields = [\n",
    "    \"transactionAmount\",\n",
    "    \"localHour\",\n",
    "    \"isProxyIP\",\n",
    "    \"physicalItemCount\",\n",
    "    \"digitalItemCount\",\n",
    "]\n",
    "features = transactions.get_model_inputs(numerical_fields=numerical_fields)\n",
    "label = labels.get_model_labels(label_field=\"is_fraud\")"
   ],
   "execution_count": null,
   "id": "4",
   "metadata": {},
   "outputs": []
  },
  {
   "cell_type": "code",
   "source": [
    "deployed_model = model.deploy(name=\"drift_demo\", input=features, labels=label)"
   ],
   "execution_count": null,
   "id": "5",
   "metadata": {},
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "source": [
    "We can register univariate drift by using `numerical_field` and optionally a `label`. By default, label is same as `numerical_field`."
   ],
   "id": "6",
   "metadata": {}
  },
  {
   "cell_type": "code",
   "source": [
    "transactions.register_univariate_drift(numerical_field=\"transactionAmount\")"
   ],
   "execution_count": null,
   "id": "7",
   "metadata": {},
   "outputs": []
  },
  {
   "cell_type": "code",
   "source": [
    "transactions.register_univariate_drift(\n",
    "    label=\"demo_uv_drift\", numerical_field=\"physicalItemCount\"\n",
    ")"
   ],
   "execution_count": null,
   "id": "8",
   "metadata": {},
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "source": [
    "For multivariate drift, providing `label` is required."
   ],
   "id": "9",
   "metadata": {}
  },
  {
   "cell_type": "code",
   "source": [
    "transactions.register_multivariate_drift(\n",
    "    label=\"demo_mv_drift\", numerical_fields=numerical_fields\n",
    ")"
   ],
   "execution_count": null,
   "id": "10",
   "metadata": {},
   "outputs": []
  },
  {
   "cell_type": "code",
   "source": [
    "deployed_model.add_drift()"
   ],
   "execution_count": null,
   "id": "11",
   "metadata": {},
   "outputs": []
  },
  {
   "cell_type": "code",
   "source": [
    "import matplotlib.pyplot as plt\n",
    "\n",
    "\n",
    "def plot_drift(drifts):\n",
    "    plt.plot([drift[\"record\"].score for drift in drifts])"
   ],
   "execution_count": null,
   "id": "12",
   "metadata": {},
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "source": [
    "We can use either `label` or `numerical_field(s)` to fetch drift results."
   ],
   "id": "13",
   "metadata": {}
  },
  {
   "cell_type": "code",
   "source": [
    "plot_drift(transactions.get_univariate_drift(numerical_field=\"transactionAmount\"))"
   ],
   "execution_count": null,
   "id": "14",
   "metadata": {},
   "outputs": []
  },
  {
   "cell_type": "code",
   "source": [
    "plot_drift(transactions.get_univariate_drift(label=\"demo_uv_drift\"))"
   ],
   "execution_count": null,
   "id": "15",
   "metadata": {},
   "outputs": []
  },
  {
   "cell_type": "code",
   "source": [
    "plot_drift(transactions.get_multivariate_drift(label=\"demo_mv_drift\"))"
   ],
   "execution_count": null,
   "id": "16",
   "metadata": {},
   "outputs": []
  },
  {
   "cell_type": "code",
   "source": [
    "plot_drift(deployed_model.get_drifts())"
   ],
   "execution_count": null,
   "id": "17",
   "metadata": {},
   "outputs": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}