docs/nb/multiclass.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "9d666810",
   "metadata": {},
   "source": [
    "# Multi-class classifiers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cf6fe730",
   "metadata": {},
   "outputs": [],
   "source": [
    "from simba.model.train_multiclass_rf import TrainMultiClassRandomForestClassifier\n",
    "from simba.model.grid_search_multiclass_rf import GridSearchMulticlassRandomForestClassifier\n",
    "from simba.model.inference_multiclass_batch import InferenceMulticlassBatch\n",
    "from simba.plotting.plot_multiclass_clf_results import PlotMulticlassSklearnResultsSingleCore"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c6ac3e64",
   "metadata": {},
   "source": [
    "### CREATE MULTI-CLASS CLASSIFIERS REQUIREMENTS\n",
    "\n",
    "Some requirements for this to run succesfully (there is currently (10/23) no way of doing this in the GUI):\n",
    "\n",
    "**(1)** All files inside the `project_folder/csv/targets_inserted` directory contains an annotation column which name represent a class of mutually exclusive behaviors and values that map to different types of that behaviour class. \n",
    "\n",
    "For example, the column may be named \"Running\", and each row contains a value between 0 and 3, where 0 represents \"no running\", 1 represent \"slow speed running\", 2 represents \"medium speed running\", and 3 represents \"fast speed running\".  \n",
    "\n",
    "**(2)** If creating a single classifier, the SimBA project_config.ini needs to contain a map pairing the values in the annotation column to behavior names. In the [SML Setting], insert a `classifier_map_1` option. See [THIS](https://github.com/sgoldenlab/simba/blob/master/docs/multiclass_project_config.ini) config file as an example. For example, the [SML Settings] may read:\n",
    "\n",
    "```\n",
    "[SML settings]\n",
    "model_dir = /Users/simon/Desktop/envs/troubleshooting/multilabel/models\n",
    "model_path_1 = /Users/simon/Desktop/envs/troubleshooting/multilabel/models/generated_models/syllable_class.sav\n",
    "no_targets = 1\n",
    "target_name_1 = running\n",
    "classifier_map_1 = {0: 'no running', 1: 'slow speed running', 2: 'medium speed running', 3: 'fast speed running'}\n",
    "```\n",
    "**(3)** If creating multiple multi-class classifiers, the SimBA meta files inside the `project_folder/configs` directory needs to contain a map pairing the values in the annotation column to behavior names. This involves inserting a column named `classifier_map` and the map into the first row. See last column in [THIS](https://github.com/sgoldenlab/simba/blob/master/misc/multiclass_meta.csv) config meta file as an example.\n",
    "\n",
    "**(4)** We can use sampling to balance the class distributions of annoatated frames when creating classifiers. \n",
    "\n",
    "* When creating a single classifier: Open the `project_config.ini` in a text editor. Under the [create ensemble settings] section find the [under_sample_setting] section and set this to either `random undersample multiclass frames` or `random undersample multiclass bouts`. Next, under the [create ensemble settings] section and [under_sample_ratio] section, set this to a dictionary that defines a baseline target class, and ratios to sample of the non-target classes relative to the target class. For example, this is a valid entry: {'target_var': 0, 'sampling_ratio': {1: 1.0, 2: 1.0, 3: 1.0}}.\n",
    "\n",
    "    Example I: You have 4 classes of behaviors 0 represents \"no running\", 1 represent \"slow speed running\", 2 represents \"medium speed running\", and 3 represents \"fast speed running\". You have 100 annotated frames of no running, 500 annotated frames each of \"slow speed running\", \"medium speed running\", and \"fast speed running\". Now you want to sample as many annotations in each class as in there is in the \"no running class\" for training. \n",
    "\n",
    "    Set the target_var to zero (no running) and the sampling ratios to 1.0 for each of the other classes:  {'target_var': 0, 'sampling_ratio': {1: 1.0, 2: 1.0, 3: 1.0}}. For each of the non-target classes (1, 2, 3), 100% of the count of annotations in the target class (0) will be sampled. \n",
    "\n",
    "    Example II: You have 100 annotated frames of no running, 500 annotated frames each of \"slow speed running\", \"medium speed running\", and \"fast speed running\". Now you want half the numer of annotations present in the non-running class for all the non-target classes. Set the target_var to zero (no running) and the sampling ratios to 0.5 for each of the non-target classes:  {'target_var': 0, 'sampling_ratio': {1: 0.5, 2: 0.5, 3: 0.5}}. For each of the non-target classes (1, 2, 3), 50% of the count of annotations in the target class (0) will be sampled. \n",
    "    \n",
    "    Here is example config file undersample setting for a multiclass task:\n",
    "    \n",
    "    ```\n",
    "    [create ensemble settings]\n",
    "    under_sample_setting = random undersample multiclass frames\n",
    "    under_sample_ratio = {'target_var': 0, 'sampling_ratio': {1: 1, 2: 1, 3: 1}}\n",
    "    ```\n",
    "\n",
    "* When creating multiple classifiers: Open the files in the `project_folder/configs` directory in a text editor. \n",
    "\n",
    "    Under the ``under_sample_setting`` header insert either``random undersample multiclass frames`` or ``random undersample multiclass bouts``. Under the ``under_sample_ratio`` header, insert your sampling ratios e.g., `{'target_var': 0, 'sampling_ratio': {1: 1, 2: 1, 3: 1}}`. \n",
    "\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4d39a1c3",
   "metadata": {},
   "source": [
    "#### CREATE A SINGLE MULTI-CLASS CLASSIFIER"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "29f88db4",
   "metadata": {},
   "outputs": [],
   "source": [
    "#WE DEFINE THE PATH TO OUR SIMBA PROJECT CONFIG FILE\n",
    "CONFIG_PATH = '/Users/simon/Desktop/envs/troubleshooting/multilabel/project_folder/project_config.ini'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "e6d8a249",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Reading in 1 annotated files...\n",
      "Reading complete 01.YC015YC016phase45-sample (elapsed time: 51.3581s)...\n",
      "Number of features in dataset: 14251\n",
      "Number of None frames in dataset: 10 (25.0%)\n",
      "Number of sharp frames in dataset: 10 (25.0%)\n",
      "Number of track frames in dataset: 10 (25.0%)\n",
      "Number of sync frames in dataset: 10 (25.0%)\n"
     ]
    }
   ],
   "source": [
    "#WE CREATE A MULTI-CLASS TRAINER INSTANCE\n",
    "model_trainer = TrainMultiClassRandomForestClassifier(config_path=CONFIG_PATH)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "d7409542",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training and evaluating model...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=-1)]: Using backend ThreadingBackend with 8 concurrent workers.\n",
      "[Parallel(n_jobs=-1)]: Done   6 out of  10 | elapsed:    0.0s remaining:    0.0s\n",
      "[Parallel(n_jobs=-1)]: Done  10 out of  10 | elapsed:    0.0s finished\n",
      "[Parallel(n_jobs=-1)]: Using backend ThreadingBackend with 8 concurrent workers.\n",
      "[Parallel(n_jobs=-1)]: Using backend ThreadingBackend with 8 concurrent workers.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Calculating learning curves...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=-1)]: Using backend ThreadingBackend with 8 concurrent workers.\n",
      "[Parallel(n_jobs=-1)]: Using backend ThreadingBackend with 8 concurrent workers.\n",
      "[Parallel(n_jobs=-1)]: Using backend ThreadingBackend with 8 concurrent workers.\n",
      "[Parallel(n_jobs=-1)]: Using backend ThreadingBackend with 8 concurrent workers.\n",
      "[Parallel(n_jobs=-1)]: Using backend ThreadingBackend with 8 concurrent workers.\n",
      "[Parallel(n_jobs=-1)]: Done   6 out of  10 | elapsed:    0.1s remaining:    0.1s\n",
      "[Parallel(n_jobs=-1)]: Done   6 out of  10 | elapsed:    0.1s remaining:    0.1s\n",
      "[Parallel(n_jobs=-1)]: Done  10 out of  10 | elapsed:    0.1s finished\n",
      "[Parallel(n_jobs=-1)]: Done  10 out of  10 | elapsed:    0.1s finished\n",
      "[Parallel(n_jobs=-1)]: Done   6 out of  10 | elapsed:    0.1s remaining:    0.0s\n",
      "[Parallel(n_jobs=8)]: Using backend ThreadingBackend with 8 concurrent workers.\n",
      "[Parallel(n_jobs=8)]: Using backend ThreadingBackend with 8 concurrent workers.\n",
      "[Parallel(n_jobs=-1)]: Done  10 out of  10 | elapsed:    0.1s finished\n",
      "[Parallel(n_jobs=-1)]: Done   6 out of  10 | elapsed:    0.1s remaining:    0.1s\n",
      "[Parallel(n_jobs=-1)]: Done   6 out of  10 | elapsed:    0.1s remaining:    0.1s\n",
      "[Parallel(n_jobs=-1)]: Done   6 out of  10 | elapsed:    0.2s remaining:    0.1s\n",
      "[Parallel(n_jobs=-1)]: Done   6 out of  10 | elapsed:    0.1s remaining:    0.1s\n",
      "[Parallel(n_jobs=-1)]: Done  10 out of  10 | elapsed:    0.2s finished\n",
      "[Parallel(n_jobs=-1)]: Done  10 out of  10 | elapsed:    0.1s finished\n",
      "[Parallel(n_jobs=-1)]: Done  10 out of  10 | elapsed:    0.1s finished\n",
      "[Parallel(n_jobs=-1)]: Done  10 out of  10 | elapsed:    0.1s finished\n",
      "[Parallel(n_jobs=8)]: Done   6 out of  10 | elapsed:    0.0s remaining:    0.0s\n",
      "[Parallel(n_jobs=8)]: Done  10 out of  10 | elapsed:    0.0s finished\n",
      "[Parallel(n_jobs=8)]: Done   6 out of  10 | elapsed:    0.0s remaining:    0.0s\n",
      "[Parallel(n_jobs=8)]: Done  10 out of  10 | elapsed:    0.0s finished\n",
      "[Parallel(n_jobs=8)]: Using backend ThreadingBackend with 8 concurrent workers.\n",
      "[Parallel(n_jobs=8)]: Using backend ThreadingBackend with 8 concurrent workers.\n",
      "[Parallel(n_jobs=8)]: Using backend ThreadingBackend with 8 concurrent workers.\n",
      "[Parallel(n_jobs=8)]: Using backend ThreadingBackend with 8 concurrent workers.\n",
      "[Parallel(n_jobs=8)]: Using backend ThreadingBackend with 8 concurrent workers.\n",
      "[Parallel(n_jobs=8)]: Done   6 out of  10 | elapsed:    0.0s remaining:    0.0s\n",
      "[Parallel(n_jobs=8)]: Done   6 out of  10 | elapsed:    0.0s remaining:    0.0s\n",
      "[Parallel(n_jobs=8)]: Done   6 out of  10 | elapsed:    0.0s remaining:    0.0s\n",
      "[Parallel(n_jobs=8)]: Done  10 out of  10 | elapsed:    0.0s finished\n",
      "[Parallel(n_jobs=8)]: Done  10 out of  10 | elapsed:    0.0s finished\n",
      "[Parallel(n_jobs=8)]: Done  10 out of  10 | elapsed:    0.0s finished\n",
      "[Parallel(n_jobs=8)]: Done   6 out of  10 | elapsed:    0.0s remaining:    0.0s\n",
      "[Parallel(n_jobs=8)]: Done  10 out of  10 | elapsed:    0.0s finished\n",
      "[Parallel(n_jobs=8)]: Done   6 out of  10 | elapsed:    0.0s remaining:    0.0s\n",
      "[Parallel(n_jobs=8)]: Done  10 out of  10 | elapsed:    0.0s finished\n",
      "[Parallel(n_jobs=8)]: Using backend ThreadingBackend with 8 concurrent workers.\n",
      "[Parallel(n_jobs=8)]: Using backend ThreadingBackend with 8 concurrent workers.\n",
      "[Parallel(n_jobs=8)]: Done   6 out of  10 | elapsed:    0.0s remaining:    0.0s\n",
      "[Parallel(n_jobs=8)]: Done  10 out of  10 | elapsed:    0.0s finished\n",
      "[Parallel(n_jobs=8)]: Done   6 out of  10 | elapsed:    0.0s remaining:    0.0s\n",
      "[Parallel(n_jobs=8)]: Done  10 out of  10 | elapsed:    0.0s finished\n",
      "[Parallel(n_jobs=8)]: Using backend ThreadingBackend with 8 concurrent workers.\n",
      "[Parallel(n_jobs=8)]: Using backend ThreadingBackend with 8 concurrent workers.\n",
      "[Parallel(n_jobs=8)]: Using backend ThreadingBackend with 8 concurrent workers.\n",
      "[Parallel(n_jobs=8)]: Using backend ThreadingBackend with 8 concurrent workers.\n",
      "[Parallel(n_jobs=8)]: Using backend ThreadingBackend with 8 concurrent workers.\n",
      "[Parallel(n_jobs=8)]: Done   6 out of  10 | elapsed:    0.0s remaining:    0.0s\n",
      "[Parallel(n_jobs=8)]: Done  10 out of  10 | elapsed:    0.0s finished\n",
      "[Parallel(n_jobs=8)]: Done   6 out of  10 | elapsed:    0.0s remaining:    0.0s\n",
      "[Parallel(n_jobs=8)]: Done  10 out of  10 | elapsed:    0.0s finished\n",
      "[Parallel(n_jobs=8)]: Done   6 out of  10 | elapsed:    0.0s remaining:    0.0s\n",
      "[Parallel(n_jobs=-1)]: Using backend ThreadingBackend with 8 concurrent workers.\n",
      "[Parallel(n_jobs=8)]: Done  10 out of  10 | elapsed:    0.0s finished\n",
      "[Parallel(n_jobs=-1)]: Using backend ThreadingBackend with 8 concurrent workers.\n",
      "[Parallel(n_jobs=8)]: Done   6 out of  10 | elapsed:    0.1s remaining:    0.0s\n",
      "[Parallel(n_jobs=8)]: Done  10 out of  10 | elapsed:    0.1s finished\n",
      "[Parallel(n_jobs=8)]: Done   6 out of  10 | elapsed:    0.1s remaining:    0.0s\n",
      "[Parallel(n_jobs=8)]: Done  10 out of  10 | elapsed:    0.1s finished\n",
      "[Parallel(n_jobs=-1)]: Done   6 out of  10 | elapsed:    0.0s remaining:    0.0s\n",
      "[Parallel(n_jobs=-1)]: Done   6 out of  10 | elapsed:    0.0s remaining:    0.0s\n",
      "[Parallel(n_jobs=-1)]: Done  10 out of  10 | elapsed:    0.0s finished\n",
      "[Parallel(n_jobs=-1)]: Done  10 out of  10 | elapsed:    0.0s finished\n",
      "[Parallel(n_jobs=-1)]: Using backend ThreadingBackend with 8 concurrent workers.\n",
      "[Parallel(n_jobs=-1)]: Using backend ThreadingBackend with 8 concurrent workers.\n",
      "[Parallel(n_jobs=-1)]: Using backend ThreadingBackend with 8 concurrent workers.\n",
      "[Parallel(n_jobs=8)]: Using backend ThreadingBackend with 8 concurrent workers.\n",
      "[Parallel(n_jobs=8)]: Using backend ThreadingBackend with 8 concurrent workers.\n",
      "[Parallel(n_jobs=-1)]: Using backend ThreadingBackend with 8 concurrent workers.\n",
      "[Parallel(n_jobs=-1)]: Using backend ThreadingBackend with 8 concurrent workers.\n",
      "[Parallel(n_jobs=-1)]: Done   6 out of  10 | elapsed:    0.1s remaining:    0.1s\n",
      "[Parallel(n_jobs=-1)]: Done  10 out of  10 | elapsed:    0.1s finished\n",
      "[Parallel(n_jobs=8)]: Done   6 out of  10 | elapsed:    0.1s remaining:    0.0s\n",
      "[Parallel(n_jobs=8)]: Done   6 out of  10 | elapsed:    0.1s remaining:    0.0s\n",
      "[Parallel(n_jobs=-1)]: Done   6 out of  10 | elapsed:    0.1s remaining:    0.1s\n",
      "[Parallel(n_jobs=8)]: Done  10 out of  10 | elapsed:    0.1s finished\n",
      "[Parallel(n_jobs=8)]: Done  10 out of  10 | elapsed:    0.1s finished\n",
      "[Parallel(n_jobs=-1)]: Done  10 out of  10 | elapsed:    0.1s finished\n",
      "[Parallel(n_jobs=-1)]: Done   6 out of  10 | elapsed:    0.1s remaining:    0.1s\n",
      "[Parallel(n_jobs=-1)]: Done   6 out of  10 | elapsed:    0.1s remaining:    0.0s\n",
      "[Parallel(n_jobs=-1)]: Done   6 out of  10 | elapsed:    0.1s remaining:    0.1s\n",
      "[Parallel(n_jobs=-1)]: Done  10 out of  10 | elapsed:    0.1s finished\n",
      "[Parallel(n_jobs=-1)]: Done  10 out of  10 | elapsed:    0.2s finished\n",
      "[Parallel(n_jobs=-1)]: Done  10 out of  10 | elapsed:    0.1s finished\n",
      "[Parallel(n_jobs=8)]: Using backend ThreadingBackend with 8 concurrent workers.\n",
      "[Parallel(n_jobs=8)]: Using backend ThreadingBackend with 8 concurrent workers.\n",
      "[Parallel(n_jobs=8)]: Using backend ThreadingBackend with 8 concurrent workers.\n",
      "[Parallel(n_jobs=8)]: Using backend ThreadingBackend with 8 concurrent workers.\n",
      "[Parallel(n_jobs=8)]: Done   6 out of  10 | elapsed:    0.0s remaining:    0.0s\n",
      "[Parallel(n_jobs=8)]: Using backend ThreadingBackend with 8 concurrent workers.\n",
      "[Parallel(n_jobs=8)]: Done  10 out of  10 | elapsed:    0.0s finished\n",
      "[Parallel(n_jobs=8)]: Using backend ThreadingBackend with 8 concurrent workers.\n",
      "[Parallel(n_jobs=8)]: Done   6 out of  10 | elapsed:    0.1s remaining:    0.0s\n",
      "[Parallel(n_jobs=8)]: Using backend ThreadingBackend with 8 concurrent workers.\n",
      "[Parallel(n_jobs=8)]: Done  10 out of  10 | elapsed:    0.1s finished\n",
      "[Parallel(n_jobs=8)]: Done   6 out of  10 | elapsed:    0.1s remaining:    0.0s\n",
      "[Parallel(n_jobs=8)]: Done  10 out of  10 | elapsed:    0.1s finished\n",
      "[Parallel(n_jobs=8)]: Done   6 out of  10 | elapsed:    0.0s remaining:    0.0s\n",
      "[Parallel(n_jobs=8)]: Done   6 out of  10 | elapsed:    0.0s remaining:    0.0s\n",
      "[Parallel(n_jobs=8)]: Done  10 out of  10 | elapsed:    0.0s finished\n",
      "[Parallel(n_jobs=8)]: Done   6 out of  10 | elapsed:    0.0s remaining:    0.0s\n",
      "[Parallel(n_jobs=8)]: Done  10 out of  10 | elapsed:    0.0s finished\n",
      "[Parallel(n_jobs=8)]: Done   6 out of  10 | elapsed:    0.0s remaining:    0.0s\n",
      "[Parallel(n_jobs=8)]: Done  10 out of  10 | elapsed:    0.0s finished\n",
      "[Parallel(n_jobs=8)]: Done  10 out of  10 | elapsed:    0.0s finished\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=8)]: Using backend ThreadingBackend with 8 concurrent workers.\n",
      "[Parallel(n_jobs=8)]: Using backend ThreadingBackend with 8 concurrent workers.\n",
      "[Parallel(n_jobs=8)]: Using backend ThreadingBackend with 8 concurrent workers.\n",
      "[Parallel(n_jobs=8)]: Done   6 out of  10 | elapsed:    0.0s remaining:    0.0s\n",
      "[Parallel(n_jobs=8)]: Done  10 out of  10 | elapsed:    0.0s finished\n",
      "[Parallel(n_jobs=8)]: Done   6 out of  10 | elapsed:    0.0s remaining:    0.0s\n",
      "[Parallel(n_jobs=8)]: Done  10 out of  10 | elapsed:    0.0s finished\n",
      "[Parallel(n_jobs=8)]: Done   6 out of  10 | elapsed:    0.0s remaining:    0.0s\n",
      "[Parallel(n_jobs=8)]: Done  10 out of  10 | elapsed:    0.0s finished\n",
      "[Parallel(n_jobs=8)]: Using backend ThreadingBackend with 8 concurrent workers.\n",
      "[Parallel(n_jobs=8)]: Using backend ThreadingBackend with 8 concurrent workers.\n",
      "[Parallel(n_jobs=-1)]: Using backend ThreadingBackend with 8 concurrent workers.\n",
      "[Parallel(n_jobs=8)]: Done   6 out of  10 | elapsed:    0.0s remaining:    0.0s\n",
      "[Parallel(n_jobs=8)]: Done  10 out of  10 | elapsed:    0.0s finished\n",
      "[Parallel(n_jobs=8)]: Done   6 out of  10 | elapsed:    0.0s remaining:    0.0s\n",
      "[Parallel(n_jobs=8)]: Done  10 out of  10 | elapsed:    0.0s finished\n",
      "[Parallel(n_jobs=-1)]: Done   6 out of  10 | elapsed:    0.0s remaining:    0.0s\n",
      "[Parallel(n_jobs=-1)]: Done  10 out of  10 | elapsed:    0.0s finished\n",
      "[Parallel(n_jobs=8)]: Using backend ThreadingBackend with 8 concurrent workers.\n",
      "[Parallel(n_jobs=8)]: Done   6 out of  10 | elapsed:    0.0s remaining:    0.0s\n",
      "[Parallel(n_jobs=8)]: Done  10 out of  10 | elapsed:    0.0s finished\n",
      "[Parallel(n_jobs=8)]: Using backend ThreadingBackend with 8 concurrent workers.\n",
      "[Parallel(n_jobs=8)]: Done   6 out of  10 | elapsed:    0.0s remaining:    0.0s\n",
      "[Parallel(n_jobs=8)]: Done  10 out of  10 | elapsed:    0.0s finished\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Learning curve calculation complete (elapsed time: 1.5253s) ...\n",
      "Calculating PR curves...\n",
      "Precision-recall curve calculation complete (elapsed time: 0.1334s) ...\n",
      "Creating classification report visualization...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=8)]: Using backend ThreadingBackend with 8 concurrent workers.\n",
      "[Parallel(n_jobs=8)]: Done   6 out of  10 | elapsed:    0.0s remaining:    0.0s\n",
      "[Parallel(n_jobs=8)]: Done  10 out of  10 | elapsed:    0.0s finished\n",
      "[Parallel(n_jobs=8)]: Using backend ThreadingBackend with 8 concurrent workers.\n",
      "[Parallel(n_jobs=8)]: Done   6 out of  10 | elapsed:    0.0s remaining:    0.0s\n",
      "[Parallel(n_jobs=8)]: Done  10 out of  10 | elapsed:    0.0s finished\n",
      "[Parallel(n_jobs=8)]: Using backend ThreadingBackend with 8 concurrent workers.\n",
      "[Parallel(n_jobs=8)]: Done   6 out of  10 | elapsed:    0.0s remaining:    0.0s\n",
      "[Parallel(n_jobs=8)]: Done  10 out of  10 | elapsed:    0.0s finished\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Creating feature importance log...\n",
      "Creating feature importance bar chart...\n",
      "Creating feature importance log...\n"
     ]
    }
   ],
   "source": [
    "#WE RUN THIS MODEL TRAINER INSTANCE\n",
    "model_trainer.run()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "07c2b2e3",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SIMBA COMPLETE: Classifier syllable_class saved in models/generated_models directory (elapsed time: 8.938s) \tcomplete\n",
      "SIMBA COMPLETE: Evaluation files are in models/generated_models/model_evaluations folders \tcomplete\n"
     ]
    }
   ],
   "source": [
    "# FINALLY WE SAVE THE MODEL\n",
    "model_trainer.save_model()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e60d98cf",
   "metadata": {},
   "source": [
    "#### CREATE MULTIPLE MULTI-CLASS CLASSIFIER, ONE FOR EACH FILE INSIDE THE PROJECT_FOLDER/CONFIGS DIRECTORY"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "12fc1f50",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Reading in 1 annotated files...\n",
      "SIMBA WARNING: MultiProcessingFailedWarning: Multi-processing file read failed, reverting to single core (increased run-time on large datasets). \twarning\n",
      "Reading in file 1/1...\n",
      "Dataset size: 2.28352MB / 0.002284GB\n",
      "1 file(s) read (elapsed time: 52.359s) ...\n"
     ]
    }
   ],
   "source": [
    "#WE CREATE A GRID-SEARCH MULTI-CLASS TRAINER INSTANCE\n",
    "multi_model_trainer = GridSearchMulticlassRandomForestClassifier(config_path=CONFIG_PATH)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "ce1eb65a",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training model 1/2 (syllable_class)...\n",
      "Number of features in dataset: 14251\n",
      "Number of None frames in dataset: 10 (25.0%)\n",
      "Number of sharp frames in dataset: 10 (25.0%)\n",
      "Number of track frames in dataset: 10 (25.0%)\n",
      "Number of sync frames in dataset: 10 (25.0%)\n",
      "MODEL 1 settings\n",
      "+------------------------+----------------+\n",
      "| Setting                | value          |\n",
      "+========================+================+\n",
      "| Model name             | syllable_class |\n",
      "+------------------------+----------------+\n",
      "| Ensemble method        | RF             |\n",
      "+------------------------+----------------+\n",
      "| Estimators (trees)     | 2000           |\n",
      "+------------------------+----------------+\n",
      "| Max features           | sqrt           |\n",
      "+------------------------+----------------+\n",
      "| Under sampling setting | None           |\n",
      "+------------------------+----------------+\n",
      "| Under sampling ratio   | nan            |\n",
      "+------------------------+----------------+\n",
      "| Over sampling setting  | None           |\n",
      "+------------------------+----------------+\n",
      "| Over sampling ratio    | nan            |\n",
      "+------------------------+----------------+\n",
      "| criterion              | gini           |\n",
      "+------------------------+----------------+\n",
      "| Min sample leaf        | 1              |\n",
      "+------------------------+----------------+ \tTABLE\n",
      "Fitting syllable_class model...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=-1)]: Using backend ThreadingBackend with 8 concurrent workers.\n",
      "[Parallel(n_jobs=-1)]: Done  34 tasks      | elapsed:    0.1s\n",
      "[Parallel(n_jobs=-1)]: Done 184 tasks      | elapsed:    0.3s\n",
      "[Parallel(n_jobs=-1)]: Done 434 tasks      | elapsed:    0.6s\n",
      "[Parallel(n_jobs=-1)]: Done 784 tasks      | elapsed:    1.1s\n",
      "[Parallel(n_jobs=-1)]: Done 1234 tasks      | elapsed:    1.7s\n",
      "[Parallel(n_jobs=-1)]: Done 1784 tasks      | elapsed:    2.5s\n",
      "[Parallel(n_jobs=-1)]: Done 2000 out of 2000 | elapsed:    2.7s finished\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Saving model meta data file...\n",
      "Classifier syllable_class_0 saved in models/validations/model_files directory ...\n",
      "SIMBA COMPLETE: All models and evaluations complete. The models and evaluation files are in models/validations folders \tcomplete\n",
      "Training model 2/2 (syllable_class)...\n",
      "Number of features in dataset: 14251\n",
      "Number of None frames in dataset: 10 (25.0%)\n",
      "Number of sharp frames in dataset: 10 (25.0%)\n",
      "Number of track frames in dataset: 10 (25.0%)\n",
      "Number of sync frames in dataset: 10 (25.0%)\n",
      "MODEL 2 settings\n",
      "+------------------------+----------------+\n",
      "| Setting                | value          |\n",
      "+========================+================+\n",
      "| Model name             | syllable_class |\n",
      "+------------------------+----------------+\n",
      "| Ensemble method        | RF             |\n",
      "+------------------------+----------------+\n",
      "| Estimators (trees)     | 2000           |\n",
      "+------------------------+----------------+\n",
      "| Max features           | sqrt           |\n",
      "+------------------------+----------------+\n",
      "| Under sampling setting | None           |\n",
      "+------------------------+----------------+\n",
      "| Under sampling ratio   | nan            |\n",
      "+------------------------+----------------+\n",
      "| Over sampling setting  | None           |\n",
      "+------------------------+----------------+\n",
      "| Over sampling ratio    | nan            |\n",
      "+------------------------+----------------+\n",
      "| criterion              | gini           |\n",
      "+------------------------+----------------+\n",
      "| Min sample leaf        | 1              |\n",
      "+------------------------+----------------+ \tTABLE\n",
      "Fitting syllable_class model...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=-1)]: Using backend ThreadingBackend with 8 concurrent workers.\n",
      "[Parallel(n_jobs=-1)]: Done  34 tasks      | elapsed:    0.1s\n",
      "[Parallel(n_jobs=-1)]: Done 184 tasks      | elapsed:    0.3s\n",
      "[Parallel(n_jobs=-1)]: Done 434 tasks      | elapsed:    0.6s\n",
      "[Parallel(n_jobs=-1)]: Done 784 tasks      | elapsed:    1.1s\n",
      "[Parallel(n_jobs=-1)]: Done 1234 tasks      | elapsed:    1.7s\n",
      "[Parallel(n_jobs=-1)]: Done 1784 tasks      | elapsed:    2.4s\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Saving model meta data file...\n",
      "Classifier syllable_class_1 saved in models/validations/model_files directory ...\n",
      "SIMBA COMPLETE: All models and evaluations complete. The models and evaluation files are in models/validations folders \tcomplete\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=-1)]: Done 2000 out of 2000 | elapsed:    2.7s finished\n"
     ]
    }
   ],
   "source": [
    "#WE RUN THIS MODEL TRAINER INSTANCE\n",
    "multi_model_trainer.run()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "955b8031",
   "metadata": {},
   "source": [
    "#### RUN INFERENCE\n",
    "\n",
    "Now when we have created our multi-class model(s), we want to use one model to create predictions for new videos. Set the path to the model you want to use in the [SML settings] section under the `model_path` option."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "a000940f",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Analyzing 1 file(s) with 1 classifier(s)...\n"
     ]
    }
   ],
   "source": [
    "#WE CREATE A MULTI-CLASS BATCH INFERENCE INSTANCE\n",
    "batch_inferencer = InferenceMulticlassBatch(config_path=CONFIG_PATH)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "0c28d951",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Analyzing video 01.YC015YC016phase45-sample...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=8)]: Using backend ThreadingBackend with 8 concurrent workers.\n",
      "[Parallel(n_jobs=8)]: Done   6 out of  10 | elapsed:    0.0s remaining:    0.0s\n",
      "[Parallel(n_jobs=8)]: Done  10 out of  10 | elapsed:    0.0s finished\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Predictions created for 01.YC015YC016phase45-sample (elapsed time: 5046.467) ...\n",
      "SIMBA COMPLETE: Multi-class machine predictions complete. 1 file(s) saved in project_folder/csv/machine_results directory (elapsed time: 5046.4743s) \tcomplete\n"
     ]
    }
   ],
   "source": [
    "#WE RUN THE BATCH INFERENCE INSTANCE\n",
    "batch_inferencer.run()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a437bda7",
   "metadata": {},
   "source": [
    "After running `batch_inferencer.run()` in directly above cell, the results are saved in the `project_folder/csv/machine_results` directory of the SimBA project. Rather than the one-column-per-classifier created by Boolean classifiers, you will have one column per type of behavior in you classifier map. These fields contain probabilities that each of the behaviors occur in each frame (you may potentially want to weigh these scores to get the most accurate results). \n",
    "\n",
    "#### VISUALIZE RESULTS"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "171aed7b",
   "metadata": {},
   "outputs": [],
   "source": [
    "#DEFINE HOW YOU WANT TO PLOT THE RESULTS\n",
    "CREATE_FRAMES = False\n",
    "CREATE_VIDEO = True\n",
    "ROTATE_VIDEO = False\n",
    "VIDEO_NAMES = ['01.YC015YC016phase45-sample.csv']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "bdb4801c",
   "metadata": {},
   "outputs": [],
   "source": [
    "#WE CREATE A MULTI-CLASS PLOTTING INSTANCE\n",
    "multiclass_plotter = PlotMulticlassSklearnResultsSingleCore(config_path=CONFIG_PATH,\n",
    "                                                            frame_setting=CREATE_FRAMES,\n",
    "                                                            video_setting=CREATE_VIDEO,\n",
    "                                                            video_names=VIDEO_NAMES,\n",
    "                                                            rotate=ROTATE_VIDEO)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "90f1689b",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SIMBA WARNING: FrameRangeWarning: The video /Users/simon/Desktop/envs/troubleshooting/multilabel/project_folder/videos/01.YC015YC016phase45-sample.mp4 contains 300 frames, while the data /Users/simon/Desktop/envs/troubleshooting/multilabel/project_folder/csv/machine_results/01.YC015YC016phase45-sample.csv contains 40 frames. \twarning\n",
      "Frame: 1 / 300. Video: 01.YC015YC016phase45-sample (1/1)\n",
      "Frame: 2 / 300. Video: 01.YC015YC016phase45-sample (1/1)\n",
      "Frame: 3 / 300. Video: 01.YC015YC016phase45-sample (1/1)\n",
      "Frame: 4 / 300. Video: 01.YC015YC016phase45-sample (1/1)\n",
      "Frame: 5 / 300. Video: 01.YC015YC016phase45-sample (1/1)\n",
      "Frame: 6 / 300. Video: 01.YC015YC016phase45-sample (1/1)\n",
      "Frame: 7 / 300. Video: 01.YC015YC016phase45-sample (1/1)\n",
      "Frame: 8 / 300. Video: 01.YC015YC016phase45-sample (1/1)\n",
      "Frame: 9 / 300. Video: 01.YC015YC016phase45-sample (1/1)\n",
      "Frame: 10 / 300. Video: 01.YC015YC016phase45-sample (1/1)\n",
      "Frame: 11 / 300. Video: 01.YC015YC016phase45-sample (1/1)\n",
      "Frame: 12 / 300. Video: 01.YC015YC016phase45-sample (1/1)\n",
      "Frame: 13 / 300. Video: 01.YC015YC016phase45-sample (1/1)\n",
      "Frame: 14 / 300. Video: 01.YC015YC016phase45-sample (1/1)\n",
      "Frame: 15 / 300. Video: 01.YC015YC016phase45-sample (1/1)\n",
      "Frame: 16 / 300. Video: 01.YC015YC016phase45-sample (1/1)\n",
      "Frame: 17 / 300. Video: 01.YC015YC016phase45-sample (1/1)\n",
      "Frame: 18 / 300. Video: 01.YC015YC016phase45-sample (1/1)\n",
      "Frame: 19 / 300. Video: 01.YC015YC016phase45-sample (1/1)\n",
      "Frame: 20 / 300. Video: 01.YC015YC016phase45-sample (1/1)\n",
      "Frame: 21 / 300. Video: 01.YC015YC016phase45-sample (1/1)\n",
      "Frame: 22 / 300. Video: 01.YC015YC016phase45-sample (1/1)\n",
      "Frame: 23 / 300. Video: 01.YC015YC016phase45-sample (1/1)\n",
      "Frame: 24 / 300. Video: 01.YC015YC016phase45-sample (1/1)\n",
      "Frame: 25 / 300. Video: 01.YC015YC016phase45-sample (1/1)\n",
      "Frame: 26 / 300. Video: 01.YC015YC016phase45-sample (1/1)\n",
      "Frame: 27 / 300. Video: 01.YC015YC016phase45-sample (1/1)\n",
      "Frame: 28 / 300. Video: 01.YC015YC016phase45-sample (1/1)\n",
      "Frame: 29 / 300. Video: 01.YC015YC016phase45-sample (1/1)\n",
      "Frame: 30 / 300. Video: 01.YC015YC016phase45-sample (1/1)\n",
      "Frame: 31 / 300. Video: 01.YC015YC016phase45-sample (1/1)\n",
      "Frame: 32 / 300. Video: 01.YC015YC016phase45-sample (1/1)\n",
      "Frame: 33 / 300. Video: 01.YC015YC016phase45-sample (1/1)\n",
      "Frame: 34 / 300. Video: 01.YC015YC016phase45-sample (1/1)\n",
      "Frame: 35 / 300. Video: 01.YC015YC016phase45-sample (1/1)\n",
      "Frame: 36 / 300. Video: 01.YC015YC016phase45-sample (1/1)\n",
      "Frame: 37 / 300. Video: 01.YC015YC016phase45-sample (1/1)\n",
      "Frame: 38 / 300. Video: 01.YC015YC016phase45-sample (1/1)\n",
      "Frame: 39 / 300. Video: 01.YC015YC016phase45-sample (1/1)\n",
      "Frame: 40 / 300. Video: 01.YC015YC016phase45-sample (1/1)\n",
      "SIMBA WARNING: FrameRangeWarning: Video terminated early: no data for frame 40 found in file /Users/simon/Desktop/envs/troubleshooting/multilabel/project_folder/csv/machine_results/01.YC015YC016phase45-sample.csv \twarning\n"
     ]
    }
   ],
   "source": [
    "#WE RUN THE PLOTTER (ONLY WORKS ON SINGLE CORE AT MOMENT SO MIGHT TAKE SOME TIME)\n",
    "multiclass_plotter.run()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python [conda env:.conda-simba_dev]",
   "language": "python",
   "name": "conda-env-.conda-simba_dev-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}