string_encoding.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "source": [
    "# String Encoding\n",
    "\n",
    "Textual data needs to be converted into numerical data to be used by ML models. For larger textual data like sentences and paragraphs, we saw in llm_embedding notebook how embeddings from pre-trained languages models can be used. But what about smaller strings, like country name? How do we use such strings as features in our ML models? This notebook covers different encoding methods that TurboML provides for textual features. "
   ],
   "id": "0",
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Set up the environment and install TurboML's SDK.\n",
    "We use `turboml-installer` to set up the environment for TurboML's SDK."
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "source": [
    "!pip install -q turboml-installer\n",
    "import turboml_installer ; turboml_installer.install_on_colab()"
   ],
   "execution_count": null,
   "metadata": {},
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "source": [
    "The kernel should now be restarted with TurboML's SDK installed."
   ],
   "metadata": {}
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Login to your TurboML instance\n",
    "\n",
    "Note that you can copy and replace this snippet with one from your TurboML homepage."
   ],
   "metadata": {}
  },
  {
   "cell_type": "code",
   "source": [
    "import turboml as tb",
    "\ntb.init(backend_url=BACKEND_URL, api_key=API_KEY)"
   ],
   "execution_count": null,
   "id": "1",
   "metadata": {},
   "outputs": []
  },
  {
   "cell_type": "code",
   "source": [
    "transactions = tb.datasets.FraudDetectionDatasetFeatures().to_online(\n",
    "    id=\"transactions\", load_if_exists=True\n",
    ")\n",
    "labels = tb.datasets.FraudDetectionDatasetLabels().to_online(\n",
    "    id=\"transaction_labels\", load_if_exists=True\n",
    ")"
   ],
   "execution_count": null,
   "id": "2",
   "metadata": {},
   "outputs": []
  },
  {
   "cell_type": "code",
   "source": [
    "numerical_fields = [\n",
    "    \"transactionAmount\",\n",
    "]\n",
    "textual_fields = [\"transactionCurrencyCode\"]\n",
    "features = transactions.get_model_inputs(\n",
    "    numerical_fields=numerical_fields, textual_fields=textual_fields\n",
    ")\n",
    "label = labels.get_model_labels(label_field=\"is_fraud\")"
   ],
   "execution_count": null,
   "id": "3",
   "metadata": {},
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "source": [
    "Notice that now we're extracting a textual feature called transactionCurrencyCode from our dataset. To make sure that the model finally works with numerical data, we can define preprocessors that transform the textual data to numerical data via some encoding methods. By default, TurboML uses the hashing trick (https://en.wikipedia.org/wiki/Feature_hashing) to automatically hash and convert string data to numeric data. However, TurboML also supports popular encoding methods to handle strings including\n",
    "- LabelPreProcessor\n",
    "- OneHotPreProcessor\n",
    "- TargetPreProcessor\n",
    "- FrequencyPreProcessor\n",
    "- BinaryPreProcessor\n",
    "\n",
    "We'll try an example using FrequencyPreProcessor. For these pre-processors, we need to specify in advance the cardinality of our data, which can be computed as follows. "
   ],
   "id": "4",
   "metadata": {}
  },
  {
   "cell_type": "code",
   "source": [
    "htc_model = tb.HoeffdingTreeClassifier(n_classes=2)"
   ],
   "execution_count": null,
   "id": "5",
   "metadata": {},
   "outputs": []
  },
  {
   "cell_type": "code",
   "source": [
    "import pandas as pd\n",
    "\n",
    "demo_classifier = tb.FrequencyPreProcessor(\n",
    "    text_categories=[\n",
    "        len(pd.unique(transactions.preview_df[col])) for col in textual_fields\n",
    "    ],\n",
    "    base_model=htc_model,\n",
    ")"
   ],
   "execution_count": null,
   "id": "6",
   "metadata": {},
   "outputs": []
  },
  {
   "cell_type": "code",
   "source": [
    "deployed_model = demo_classifier.deploy(\n",
    "    \"demo_classifier_htc\", input=features, labels=label\n",
    ")"
   ],
   "execution_count": null,
   "id": "7",
   "metadata": {},
   "outputs": []
  },
  {
   "cell_type": "code",
   "source": [
    "outputs = deployed_model.get_outputs()"
   ],
   "execution_count": null,
   "id": "8",
   "metadata": {},
   "outputs": []
  },
  {
   "cell_type": "code",
   "source": [
    "sample_output = outputs[-1]\n",
    "sample_output"
   ],
   "execution_count": null,
   "id": "9",
   "metadata": {},
   "outputs": []
  },
  {
   "cell_type": "code",
   "source": [],
   "execution_count": null,
   "id": "10",
   "metadata": {},
   "outputs": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}