pysad_example.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "source": [
    "# Python Model: PySAD Example\n",
    "\n",
    "In this example, we use the `PySAD` package to monitor anomalies in our streaming data."
   ],
   "id": "0",
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Set up the environment and install TurboML's SDK.\n",
    "We use `turboml-installer` to set up the environment for TurboML's SDK."
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "source": [
    "!pip install -q turboml-installer\n",
    "import turboml_installer ; turboml_installer.install_on_colab()"
   ],
   "execution_count": null,
   "metadata": {},
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "source": [
    "The kernel should now be restarted with TurboML's SDK installed."
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Login to your TurboML instance\n",
    "\n",
    "Note that you can copy and replace this snippet with one from your TurboML homepage."
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "source": [
    "import turboml as tb",
    "\ntb.init(backend_url=BACKEND_URL, api_key=API_KEY)"
   ],
   "execution_count": null,
   "id": "1",
   "metadata": {},
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "source": [
    "We start off by installing and importing the `pysad` package along with its dependencies."
   ],
   "id": "2",
   "metadata": {}
  },
  {
   "cell_type": "code",
   "source": [
    "!pip install pysad mmh3==2.5.1"
   ],
   "execution_count": null,
   "id": "3",
   "metadata": {},
   "outputs": []
  },
  {
   "cell_type": "code",
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "from pysad.models import xStream"
   ],
   "execution_count": null,
   "id": "4",
   "metadata": {},
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Model Definition\n",
    "\n",
    "TurboML's inbuilt `PythonModel` can be used to define custom models which are compatible with TurboML.\n",
    "\n",
    "Here we define `PySADModel` as a wrapper using `PySAD`'s `xStream` model, making sure to properly implement the required instance methods for a Python model."
   ],
   "id": "5",
   "metadata": {}
  },
  {
   "cell_type": "code",
   "source": [
    "import turboml.common.pytypes as types\n",
    "\n",
    "\n",
    "class PySADModel:\n",
    "    def __init__(self):\n",
    "        self.model = xStream()\n",
    "\n",
    "    def init_imports(self):\n",
    "        from pysad.models import xStream\n",
    "        import numpy as np\n",
    "\n",
    "    def learn_one(self, input: types.InputData):\n",
    "        self.model = self.model.fit_partial(np.array(input.numeric))\n",
    "\n",
    "    def predict_one(self, input: types.InputData, output: types.OutputData):\n",
    "        score = self.model.score_partial(np.array(input.numeric))\n",
    "        output.set_score(score)"
   ],
   "execution_count": null,
   "id": "6",
   "metadata": {},
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "source": [
    "Now, we create a custom `venv` so that the custom model defined above has access to all the required dependencies. PySAD required mmh3==2.5.1 as per their docs. "
   ],
   "id": "7",
   "metadata": {}
  },
  {
   "cell_type": "code",
   "source": [
    "venv = tb.setup_venv(\"my_pysad_venv\", [\"mmh3==2.5.1\", \"pysad\", \"numpy\"])\n",
    "venv.add_python_class(PySADModel)"
   ],
   "execution_count": null,
   "id": "8",
   "metadata": {},
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Dataset\n",
    "\n",
    "We choose our standard `FraudDetection` dataset, using the `to_online` method to push it to the platform."
   ],
   "id": "9",
   "metadata": {}
  },
  {
   "cell_type": "code",
   "source": [
    "transactions = tb.datasets.FraudDetectionDatasetFeatures().to_online(\n",
    "    id=\"transactions\", load_if_exists=True\n",
    ")\n",
    "labels = tb.datasets.FraudDetectionDatasetLabels().to_online(\n",
    "    id=\"transaction_labels\", load_if_exists=True\n",
    ")"
   ],
   "execution_count": null,
   "id": "10",
   "metadata": {},
   "outputs": []
  },
  {
   "cell_type": "code",
   "source": [
    "numerical_fields = [\n",
    "    \"transactionAmount\",\n",
    "    \"localHour\",\n",
    "    \"isProxyIP\",\n",
    "    \"digitalItemCount\",\n",
    "    \"physicalItemCount\",\n",
    "]"
   ],
   "execution_count": null,
   "id": "11",
   "metadata": {},
   "outputs": []
  },
  {
   "cell_type": "code",
   "source": [
    "features = transactions.get_model_inputs(numerical_fields=numerical_fields)\n",
    "label = labels.get_model_labels(label_field=\"is_fraud\")"
   ],
   "execution_count": null,
   "id": "12",
   "metadata": {},
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Model Deployment\n",
    "\n",
    "Now, we deploy our model and extract its outputs."
   ],
   "id": "13",
   "metadata": {}
  },
  {
   "cell_type": "code",
   "source": [
    "pysad_model = tb.Python(class_name=PySADModel.__name__, venv_name=venv.name)"
   ],
   "execution_count": null,
   "id": "14",
   "metadata": {},
   "outputs": []
  },
  {
   "cell_type": "code",
   "source": [
    "deployed_model_pysad = pysad_model.deploy(\"pysad_model\", input=features, labels=label)"
   ],
   "execution_count": null,
   "id": "15",
   "metadata": {},
   "outputs": []
  },
  {
   "cell_type": "code",
   "source": [
    "outputs = deployed_model_pysad.get_outputs()"
   ],
   "execution_count": null,
   "id": "16",
   "metadata": {},
   "outputs": []
  },
  {
   "cell_type": "code",
   "source": [
    "len(outputs)"
   ],
   "execution_count": null,
   "id": "17",
   "metadata": {},
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Evaluation\n",
    "\n",
    "Finally, we use any of `PySAD`'s metrics for giving a numerical value to the degree of the presence of anomalies in our data."
   ],
   "id": "18",
   "metadata": {}
  },
  {
   "cell_type": "code",
   "source": [
    "from pysad.evaluation import AUROCMetric"
   ],
   "execution_count": null,
   "id": "19",
   "metadata": {},
   "outputs": []
  },
  {
   "cell_type": "code",
   "source": [
    "auroc = AUROCMetric()\n",
    "labels_df = labels.preview_df\n",
    "for output, y in zip(\n",
    "    outputs, labels_df[\"is_fraud\"].tolist()[: len(outputs)], strict=False\n",
    "):\n",
    "    auroc.update(y, output.score)"
   ],
   "execution_count": null,
   "id": "20",
   "metadata": {},
   "outputs": []
  },
  {
   "cell_type": "code",
   "source": [
    "auroc.get()"
   ],
   "execution_count": null,
   "id": "21",
   "metadata": {},
   "outputs": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}