algorithm_tuning.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "source": [
    "# Algorithm Tuning\n",
    "\n",
    "Algorithm Tuning allows us to test different models on a given dataset, and helps to figure out which particular model gives the highest value of a user-defined performance metric on that particular dataset."
   ],
   "id": "0",
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Set up the environment and install TurboML's SDK.\n",
    "We use `turboml-installer` to set up the environment for TurboML's SDK."
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "source": [
    "!pip install -q turboml-installer\n",
    "import turboml_installer ; turboml_installer.install_on_colab()"
   ],
   "execution_count": null,
   "metadata": {},
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "source": [
    "The kernel should now be restarted with TurboML's SDK installed."
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Login to your TurboML instance\n",
    "\n",
    "Note that you can copy and replace this snippet with one from your TurboML homepage."
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "source": [
    "import turboml as tb",
    "\ntb.init(backend_url=BACKEND_URL, api_key=API_KEY)"
   ],
   "execution_count": null,
   "id": "1",
   "metadata": {},
   "outputs": []
  },
  {
   "cell_type": "code",
   "source": [
    "import pandas as pd\n",
    "from sklearn import metrics"
   ],
   "execution_count": null,
   "id": "2",
   "metadata": {},
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Dataset\n",
    "\n",
    "We use our standard `FraudDetection` dataset for this example, exposed through the `LocalDataset` interface that can be used for tuning, and also configure the dataset to indicate the column with the primary key.\n",
    "\n",
    "For this example, we use the first 100k rows."
   ],
   "id": "3",
   "metadata": {}
  },
  {
   "cell_type": "code",
   "source": [
    "transactions = tb.datasets.FraudDetectionDatasetFeatures()\n",
    "labels = tb.datasets.FraudDetectionDatasetLabels()"
   ],
   "execution_count": null,
   "id": "4",
   "metadata": {},
   "outputs": []
  },
  {
   "cell_type": "code",
   "source": [
    "transactions_100k = transactions[:100000]\n",
    "labels_100k = labels[:100000]\n",
    "\n",
    "numerical_fields = [\n",
    "    \"transactionAmount\",\n",
    "]\n",
    "categorical_fields = [\"digitalItemCount\", \"physicalItemCount\", \"isProxyIP\"]\n",
    "inputs = transactions_100k.get_model_inputs(\n",
    "    numerical_fields=numerical_fields, categorical_fields=categorical_fields\n",
    ")\n",
    "label = labels_100k.get_model_labels(label_field=\"is_fraud\")"
   ],
   "execution_count": null,
   "id": "5",
   "metadata": {},
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Training/Tuning\n",
    "\n",
    "We will be comparing the `Neural Network` and `Hoeffding Tree Classifier`, and the metric we will be optimizing is `accuracy`."
   ],
   "id": "6",
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "Configuring the NN according to the dataset."
   ],
   "id": "7",
   "metadata": {}
  },
  {
   "cell_type": "code",
   "source": [
    "new_layer = tb.NNLayer(output_size=2)\n",
    "\n",
    "nn = tb.NeuralNetwork()\n",
    "nn.layers.append(new_layer)"
   ],
   "execution_count": null,
   "id": "8",
   "metadata": {},
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "source": [
    "The `algorithm_tuning` function takes in the models being tested as a list along with the metric to test against, and returns an object for the model which had the highest score for the given metric."
   ],
   "id": "9",
   "metadata": {}
  },
  {
   "cell_type": "code",
   "source": [
    "model_score_list = tb.algorithm_tuning(\n",
    "    models_to_test=[\n",
    "        tb.HoeffdingTreeClassifier(n_classes=2),\n",
    "        nn,\n",
    "    ],\n",
    "    metric_to_optimize=\"accuracy\",\n",
    "    input=inputs,\n",
    "    labels=label,\n",
    ")\n",
    "best_model, best_score = model_score_list[0]\n",
    "best_model"
   ],
   "execution_count": null,
   "id": "10",
   "metadata": {},
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "source": [
    "# Testing\n",
    "\n",
    "After finding out the best performing model, we can use it normally for inference on the entire dataset and testing on more performance metrics."
   ],
   "id": "11",
   "metadata": {}
  },
  {
   "cell_type": "code",
   "source": [
    "transactions_test = transactions[100000:]\n",
    "labels_test = labels[100000:]"
   ],
   "execution_count": null,
   "id": "12",
   "metadata": {},
   "outputs": []
  },
  {
   "cell_type": "code",
   "source": [
    "features = transactions_test.get_model_inputs(\n",
    "    numerical_fields=numerical_fields, categorical_fields=categorical_fields\n",
    ")\n",
    "\n",
    "outputs = best_model.predict(features)"
   ],
   "execution_count": null,
   "id": "13",
   "metadata": {},
   "outputs": []
  },
  {
   "cell_type": "code",
   "source": [
    "print(\n",
    "    \"Accuracy: \",\n",
    "    metrics.accuracy_score(labels_test.df[\"is_fraud\"], outputs[\"predicted_class\"]),\n",
    ")\n",
    "print(\"F1: \", metrics.f1_score(labels_test.df[\"is_fraud\"], outputs[\"predicted_class\"]))"
   ],
   "execution_count": null,
   "id": "14",
   "metadata": {},
   "outputs": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}