exercises/040-Classification-RDT-FirstFit.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "70cef54a",
   "metadata": {},
   "source": [
    "## Check the setup and connect to the database"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "06acc36c",
   "metadata": {},
   "outputs": [],
   "source": [
    "%run '010-check_setup.ipynb'"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "bb1afdec-6de1-4110-861e-69d4899e8dfb",
   "metadata": {},
   "source": [
    "# Tables from SAP HANA"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "00efbb56-b3ca-48a4-9c2e-b5ca9bc5965f",
   "metadata": {},
   "outputs": [],
   "source": [
    "hdf_titanic_train=myconn.table('DATA_LABELED', schema='TITANIC')\n",
    "print(hdf_titanic_train.columns)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b47a9e1f-e413-4b54-b214-9186645cca08",
   "metadata": {},
   "outputs": [],
   "source": [
    "col_id='PassengerId'\n",
    "col_label='Survived'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a8880d16-dc44-4548-a8fb-b346f9bb5a2c",
   "metadata": {},
   "outputs": [],
   "source": [
    "col_features=[feature for feature in hdf_titanic_train.columns if not (feature in {col_id, col_label})]\n",
    "print(col_features)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "96bbfc41-6881-4397-94fa-bc79761dbe07",
   "metadata": {},
   "source": [
    "# Random Decision Tree classification"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "e08c6ec0",
   "metadata": {},
   "source": [
    "Random Decision Tree, aka RDT: https://help.sap.com/doc/cd94b08fe2e041c2ba778374572ddba9/2023_3_QRC/en-US/pal/algorithms/hana_ml.algorithms.pal.trees.RDTClassifier.html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a06f832b-72c1-4a4c-aca0-6f2b251f1ab8",
   "metadata": {},
   "outputs": [],
   "source": [
    "from hana_ml.algorithms.pal.unified_classification import UnifiedClassification"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fc19d166-f72f-4d79-a547-87c9aec4618f",
   "metadata": {},
   "source": [
    "UnifiedClassification offers a varity of classfication algorithms. We use RandomDecisionTree for training.\n",
    "\n",
    "Other options are: \n",
    "- 'DecisionTree'\n",
    "- 'HybridGradientBoostingTree'\n",
    "- 'LogisticRegression'\n",
    "- 'MLP'\n",
    "- 'NaiveBayes'\n",
    "- 'RandomDecisionTree'\n",
    "- 'SVM'"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "39ea7998-da84-4d66-986a-f3aa7a9b3a09",
   "metadata": {},
   "source": [
    "### The simplest training call\n",
    "\n",
    "RDT Classifier has many parameters to influence the execution of the fitting algorithm: https://help.sap.com/doc/cd94b08fe2e041c2ba778374572ddba9/2023_3_QRC/en-US/pal/algorithms/hana_ml.algorithms.pal.trees.RDTClassifier.html#rdtclassifier, but for now you run it with the default parameters only.\n",
    "\n",
    "You will use Unified Classifier class to create an RDT class: https://help.sap.com/doc/cd94b08fe2e041c2ba778374572ddba9/2023_3_QRC/en-US/pal/algorithms/hana_ml.algorithms.pal.unified_classification.UnifiedClassification.html#unifiedclassification"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c3924184-9714-40e4-b044-c911ea6aa75a",
   "metadata": {},
   "outputs": [],
   "source": [
    "uc_rdt = UnifiedClassification(func='RandomDecisionTree')"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "32131743",
   "metadata": {},
   "source": [
    "The `fit()` procedure returns a fitted object: https://help.sap.com/doc/cd94b08fe2e041c2ba778374572ddba9/2023_3_QRC/en-US/pal/algorithms/hana_ml.algorithms.pal.trees.RDTClassifier.html#hana_ml.algorithms.pal.trees.RDTClassifier.fit, ie. populated attributes, like \n",
    "- `model_DataFrame`: Trained model content.\n",
    "- `feature_importances_DataFrame`: The feature importance (the higher, the more important the feature).\n",
    "- `oob_error_DataFrame`: Out-of-bag error rate or mean squared error for random decision trees up to indexed tree. Set to None if calculate_oob is False.\n",
    "- `confusion_matrix_DataFrame`: Confusion matrix used to evaluate the performance of classification algorithms.\n",
    "\n",
    "To understand these structures better check the corresponding PAL documentation: https://help.sap.com/docs/hana-cloud-database/sap-hana-cloud-sap-hana-database-predictive-analysis-library/random-decision-trees-random-decision-trees-9ad576a#ariaid-title3\n",
    "\n",
    "To understand the mapping between PAL objects and fields in SQL and in Python, check https://help.sap.com/doc/cd94b08fe2e041c2ba778374572ddba9/2023_3_QRC/en-US/pal/parameter_mappings.html (or https://help.sap.com/doc/cd94b08fe2e041c2ba778374572ddba9/2023_3_QRC/en-US/_static/extra_parameter_mappings.html#/ for the full screen)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "34fe136c-77f8-42cd-83c0-525ce8a5ee6f",
   "metadata": {},
   "source": [
    "The simplest training (fit) call: only the key and the label (the target) of the dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "39ea46c4-a368-4765-90fd-d6419378a04e",
   "metadata": {},
   "outputs": [],
   "source": [
    "uc_rdt.fit(\n",
    "    data=hdf_titanic_train,\n",
    "    key=col_id, \n",
    "    label=col_label\n",
    ");"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "86e938dc-d30d-46e2-b84a-bd6c3a4dfff1",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "#Check fit procedure executed on the db side \n",
    "print(uc_rdt.get_fit_execute_statement())"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "3f6daf2a",
   "metadata": {},
   "source": [
    "# Call prediction"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5630e639-4ac1-4058-80fe-e7697798d81c",
   "metadata": {},
   "outputs": [],
   "source": [
    "hdf_titanic_test=myconn.table('DATA_TO_PREDICT', schema='TITANIC')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bda7acb5-6c06-4e7e-a6ad-c6f7eccee5d0",
   "metadata": {},
   "source": [
    "The test table has the same structure except missing the column `Survived`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "468af685-760b-4d97-ba71-68b707544033",
   "metadata": {},
   "outputs": [],
   "source": [
    "hdf_titanic_test.head(4).collect()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "665df841-2464-42a0-885e-301cfaacc721",
   "metadata": {},
   "outputs": [],
   "source": [
    "hdf_res = uc_rdt.predict(hdf_titanic_test, key = col_id)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ca5b4353-10ea-42c6-816d-a9ecc6e5795f",
   "metadata": {},
   "outputs": [],
   "source": [
    "hdf_res.collect()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "6086c0f0-fab3-4eea-b292-7629b9218e80",
   "metadata": {},
   "source": [
    "🤓 **Let's discuss**:\n",
    "- The structure of the result table `hdf_res`"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "464abb80-eb07-4fa4-a987-11108c084ea6",
   "metadata": {},
   "source": [
    "## Visualize the split of predicted target"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b579b01a-8782-43a9-91ce-2f4ce997b014",
   "metadata": {},
   "outputs": [],
   "source": [
    "from hana_ml.visualizers.eda import EDAVisualizer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3c5e2937-0f09-4ad1-9a46-d95f49da1c82",
   "metadata": {},
   "outputs": [],
   "source": [
    "EDAVisualizer(enable_plotly=False).pie_plot(data=hdf_res, column='SCORE',\n",
    "                         legend=False, explode=0,\n",
    "                         startangle=90, counterclock=False\n",
    "                        );"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "4be43d11-dbd8-44ca-8d26-0c7f15189131",
   "metadata": {},
   "source": [
    "🤓 **Let's discuss**:\n",
    "* What we can say about this prediction"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "900158b4",
   "metadata": {},
   "source": [
    "## Debrief the model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ee4e55b2",
   "metadata": {},
   "outputs": [],
   "source": [
    "from hana_ml.visualizers.model_debriefing import TreeModelDebriefing"
   ]
  },
  {
   "cell_type": "raw",
   "id": "bdd4da7f",
   "metadata": {},
   "source": [
    "TreeModelDebriefing.tree_debrief_with_dot(uc_rdt.model_[0], iframe_height=700);"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.2"
  },
  "vscode": {
   "interpreter": {
    "hash": "9d081a5f1f467ba7b9337ca09e42f0c0ed4894bb4cce0c846263a933dc8b5e80"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}