06. Classification.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Classification\n",
    "Put simply, classification is the task of predicting a label for a given observation. For example: you are given certain physical descriptions of an animal, and your taks is to classify them as either a dog or a cat. Here, we will classify iris flowers.\n",
    "\n",
    "As we will see later, we will use different classifiers and at the end of this notebook, we will compare them. We will define our accuracy function right now to get it out of the way. We will use a simple accuracy function that returns the ratio of the number of correctly classified observations to the total number of predictions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "findaccuracy (generic function with 1 method)"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "findaccuracy(predictedvals,groundtruthvals) = sum(predictedvals.==groundtruthvals)/length(groundtruthvals)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "using GLMNet\n",
    "using RDatasets\n",
    "using MLBase\n",
    "using Plots\n",
    "using DecisionTree\n",
    "using Distances\n",
    "using NearestNeighbors\n",
    "using Random\n",
    "using LinearAlgebra\n",
    "using DataStructures\n",
    "using LIBSVM"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Get the data first"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div><div style = \"float: left;\"><span>150×5 DataFrame</span></div><div style = \"float: right;\"><span style = \"font-style: italic;\">125 rows omitted</span></div><div style = \"clear: both;\"></div></div><div class = \"data-frame\" style = \"overflow-x: scroll;\"><table class = \"data-frame\" style = \"margin-bottom: 6px;\"><thead><tr class = \"header\"><th class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\">Row</th><th style = \"text-align: left;\">SepalLength</th><th style = \"text-align: left;\">SepalWidth</th><th style = \"text-align: left;\">PetalLength</th><th style = \"text-align: left;\">PetalWidth</th><th style = \"text-align: left;\">Species</th></tr><tr class = \"subheader headerLastRow\"><th class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\"></th><th title = \"Float64\" style = \"text-align: left;\">Float64</th><th title = \"Float64\" style = \"text-align: left;\">Float64</th><th title = \"Float64\" style = \"text-align: left;\">Float64</th><th title = \"Float64\" style = \"text-align: left;\">Float64</th><th title = \"CategoricalArrays.CategoricalValue{String, UInt8}\" style = \"text-align: left;\">Cat…</th></tr></thead><tbody><tr><td class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\">1</td><td style = \"text-align: right;\">5.1</td><td style = \"text-align: right;\">3.5</td><td style = \"text-align: right;\">1.4</td><td style = \"text-align: right;\">0.2</td><td style = \"text-align: left;\">setosa</td></tr><tr><td class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\">2</td><td style = \"text-align: right;\">4.9</td><td style = \"text-align: right;\">3.0</td><td style = \"text-align: right;\">1.4</td><td style = \"text-align: right;\">0.2</td><td style = \"text-align: left;\">setosa</td></tr><tr><td class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\">3</td><td style = \"text-align: right;\">4.7</td><td style = \"text-align: right;\">3.2</td><td style = \"text-align: right;\">1.3</td><td style = \"text-align: right;\">0.2</td><td style = \"text-align: left;\">setosa</td></tr><tr><td class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\">4</td><td style = \"text-align: right;\">4.6</td><td style = \"text-align: right;\">3.1</td><td style = \"text-align: right;\">1.5</td><td style = \"text-align: right;\">0.2</td><td style = \"text-align: left;\">setosa</td></tr><tr><td class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\">5</td><td style = \"text-align: right;\">5.0</td><td style = \"text-align: right;\">3.6</td><td style = \"text-align: right;\">1.4</td><td style = \"text-align: right;\">0.2</td><td style = \"text-align: left;\">setosa</td></tr><tr><td class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\">6</td><td style = \"text-align: right;\">5.4</td><td style = \"text-align: right;\">3.9</td><td style = \"text-align: right;\">1.7</td><td style = \"text-align: right;\">0.4</td><td style = \"text-align: left;\">setosa</td></tr><tr><td class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\">7</td><td style = \"text-align: right;\">4.6</td><td style = \"text-align: right;\">3.4</td><td style = \"text-align: right;\">1.4</td><td style = \"text-align: right;\">0.3</td><td style = \"text-align: left;\">setosa</td></tr><tr><td class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\">8</td><td style = \"text-align: right;\">5.0</td><td style = \"text-align: right;\">3.4</td><td style = \"text-align: right;\">1.5</td><td style = \"text-align: right;\">0.2</td><td style = \"text-align: left;\">setosa</td></tr><tr><td class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\">9</td><td style = \"text-align: right;\">4.4</td><td style = \"text-align: right;\">2.9</td><td style = \"text-align: right;\">1.4</td><td style = \"text-align: right;\">0.2</td><td style = \"text-align: left;\">setosa</td></tr><tr><td class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\">10</td><td style = \"text-align: right;\">4.9</td><td style = \"text-align: right;\">3.1</td><td style = \"text-align: right;\">1.5</td><td style = \"text-align: right;\">0.1</td><td style = \"text-align: left;\">setosa</td></tr><tr><td class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\">11</td><td style = \"text-align: right;\">5.4</td><td style = \"text-align: right;\">3.7</td><td style = \"text-align: right;\">1.5</td><td style = \"text-align: right;\">0.2</td><td style = \"text-align: left;\">setosa</td></tr><tr><td class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\">12</td><td style = \"text-align: right;\">4.8</td><td style = \"text-align: right;\">3.4</td><td style = \"text-align: right;\">1.6</td><td style = \"text-align: right;\">0.2</td><td style = \"text-align: left;\">setosa</td></tr><tr><td class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\">13</td><td style = \"text-align: right;\">4.8</td><td style = \"text-align: right;\">3.0</td><td style = \"text-align: right;\">1.4</td><td style = \"text-align: right;\">0.1</td><td style = \"text-align: left;\">setosa</td></tr><tr><td style = \"text-align: right;\">&vellip;</td><td style = \"text-align: right;\">&vellip;</td><td style = \"text-align: right;\">&vellip;</td><td style = \"text-align: right;\">&vellip;</td><td style = \"text-align: right;\">&vellip;</td><td style = \"text-align: right;\">&vellip;</td></tr><tr><td class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\">139</td><td style = \"text-align: right;\">6.0</td><td style = \"text-align: right;\">3.0</td><td style = \"text-align: right;\">4.8</td><td style = \"text-align: right;\">1.8</td><td style = \"text-align: left;\">virginica</td></tr><tr><td class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\">140</td><td style = \"text-align: right;\">6.9</td><td style = \"text-align: right;\">3.1</td><td style = \"text-align: right;\">5.4</td><td style = \"text-align: right;\">2.1</td><td style = \"text-align: left;\">virginica</td></tr><tr><td class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\">141</td><td style = \"text-align: right;\">6.7</td><td style = \"text-align: right;\">3.1</td><td style = \"text-align: right;\">5.6</td><td style = \"text-align: right;\">2.4</td><td style = \"text-align: left;\">virginica</td></tr><tr><td class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\">142</td><td style = \"text-align: right;\">6.9</td><td style = \"text-align: right;\">3.1</td><td style = \"text-align: right;\">5.1</td><td style = \"text-align: right;\">2.3</td><td style = \"text-align: left;\">virginica</td></tr><tr><td class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\">143</td><td style = \"text-align: right;\">5.8</td><td style = \"text-align: right;\">2.7</td><td style = \"text-align: right;\">5.1</td><td style = \"text-align: right;\">1.9</td><td style = \"text-align: left;\">virginica</td></tr><tr><td class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\">144</td><td style = \"text-align: right;\">6.8</td><td style = \"text-align: right;\">3.2</td><td style = \"text-align: right;\">5.9</td><td style = \"text-align: right;\">2.3</td><td style = \"text-align: left;\">virginica</td></tr><tr><td class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\">145</td><td style = \"text-align: right;\">6.7</td><td style = \"text-align: right;\">3.3</td><td style = \"text-align: right;\">5.7</td><td style = \"text-align: right;\">2.5</td><td style = \"text-align: left;\">virginica</td></tr><tr><td class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\">146</td><td style = \"text-align: right;\">6.7</td><td style = \"text-align: right;\">3.0</td><td style = \"text-align: right;\">5.2</td><td style = \"text-align: right;\">2.3</td><td style = \"text-align: left;\">virginica</td></tr><tr><td class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\">147</td><td style = \"text-align: right;\">6.3</td><td style = \"text-align: right;\">2.5</td><td style = \"text-align: right;\">5.0</td><td style = \"text-align: right;\">1.9</td><td style = \"text-align: left;\">virginica</td></tr><tr><td class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\">148</td><td style = \"text-align: right;\">6.5</td><td style = \"text-align: right;\">3.0</td><td style = \"text-align: right;\">5.2</td><td style = \"text-align: right;\">2.0</td><td style = \"text-align: left;\">virginica</td></tr><tr><td class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\">149</td><td style = \"text-align: right;\">6.2</td><td style = \"text-align: right;\">3.4</td><td style = \"text-align: right;\">5.4</td><td style = \"text-align: right;\">2.3</td><td style = \"text-align: left;\">virginica</td></tr><tr><td class = \"rowNumber\" style = \"font-weight: bold; text-align: right;\">150</td><td style = \"text-align: right;\">5.9</td><td style = \"text-align: right;\">3.0</td><td style = \"text-align: right;\">5.1</td><td style = \"text-align: right;\">1.8</td><td style = \"text-align: left;\">virginica</td></tr></tbody></table></div>"
      ],
      "text/latex": [
       "\\begin{tabular}{r|ccccc}\n",
       "\t& SepalLength & SepalWidth & PetalLength & PetalWidth & Species\\\\\n",
       "\t\\hline\n",
       "\t& Float64 & Float64 & Float64 & Float64 & Cat…\\\\\n",
       "\t\\hline\n",
       "\t1 & 5.1 & 3.5 & 1.4 & 0.2 & setosa \\\\\n",
       "\t2 & 4.9 & 3.0 & 1.4 & 0.2 & setosa \\\\\n",
       "\t3 & 4.7 & 3.2 & 1.3 & 0.2 & setosa \\\\\n",
       "\t4 & 4.6 & 3.1 & 1.5 & 0.2 & setosa \\\\\n",
       "\t5 & 5.0 & 3.6 & 1.4 & 0.2 & setosa \\\\\n",
       "\t6 & 5.4 & 3.9 & 1.7 & 0.4 & setosa \\\\\n",
       "\t7 & 4.6 & 3.4 & 1.4 & 0.3 & setosa \\\\\n",
       "\t8 & 5.0 & 3.4 & 1.5 & 0.2 & setosa \\\\\n",
       "\t9 & 4.4 & 2.9 & 1.4 & 0.2 & setosa \\\\\n",
       "\t10 & 4.9 & 3.1 & 1.5 & 0.1 & setosa \\\\\n",
       "\t11 & 5.4 & 3.7 & 1.5 & 0.2 & setosa \\\\\n",
       "\t12 & 4.8 & 3.4 & 1.6 & 0.2 & setosa \\\\\n",
       "\t13 & 4.8 & 3.0 & 1.4 & 0.1 & setosa \\\\\n",
       "\t14 & 4.3 & 3.0 & 1.1 & 0.1 & setosa \\\\\n",
       "\t15 & 5.8 & 4.0 & 1.2 & 0.2 & setosa \\\\\n",
       "\t16 & 5.7 & 4.4 & 1.5 & 0.4 & setosa \\\\\n",
       "\t17 & 5.4 & 3.9 & 1.3 & 0.4 & setosa \\\\\n",
       "\t18 & 5.1 & 3.5 & 1.4 & 0.3 & setosa \\\\\n",
       "\t19 & 5.7 & 3.8 & 1.7 & 0.3 & setosa \\\\\n",
       "\t20 & 5.1 & 3.8 & 1.5 & 0.3 & setosa \\\\\n",
       "\t21 & 5.4 & 3.4 & 1.7 & 0.2 & setosa \\\\\n",
       "\t22 & 5.1 & 3.7 & 1.5 & 0.4 & setosa \\\\\n",
       "\t23 & 4.6 & 3.6 & 1.0 & 0.2 & setosa \\\\\n",
       "\t24 & 5.1 & 3.3 & 1.7 & 0.5 & setosa \\\\\n",
       "\t$\\dots$ & $\\dots$ & $\\dots$ & $\\dots$ & $\\dots$ & $\\dots$ \\\\\n",
       "\\end{tabular}\n"
      ],
      "text/plain": [
       "\u001b[1m150×5 DataFrame\u001b[0m\n",
       "\u001b[1m Row \u001b[0m│\u001b[1m SepalLength \u001b[0m\u001b[1m SepalWidth \u001b[0m\u001b[1m PetalLength \u001b[0m\u001b[1m PetalWidth \u001b[0m\u001b[1m Species   \u001b[0m\n",
       "     │\u001b[90m Float64     \u001b[0m\u001b[90m Float64    \u001b[0m\u001b[90m Float64     \u001b[0m\u001b[90m Float64    \u001b[0m\u001b[90m Cat…      \u001b[0m\n",
       "─────┼─────────────────────────────────────────────────────────────\n",
       "   1 │         5.1         3.5          1.4         0.2  setosa\n",
       "   2 │         4.9         3.0          1.4         0.2  setosa\n",
       "   3 │         4.7         3.2          1.3         0.2  setosa\n",
       "   4 │         4.6         3.1          1.5         0.2  setosa\n",
       "   5 │         5.0         3.6          1.4         0.2  setosa\n",
       "   6 │         5.4         3.9          1.7         0.4  setosa\n",
       "   7 │         4.6         3.4          1.4         0.3  setosa\n",
       "   8 │         5.0         3.4          1.5         0.2  setosa\n",
       "  ⋮  │      ⋮           ⋮            ⋮           ⋮           ⋮\n",
       " 144 │         6.8         3.2          5.9         2.3  virginica\n",
       " 145 │         6.7         3.3          5.7         2.5  virginica\n",
       " 146 │         6.7         3.0          5.2         2.3  virginica\n",
       " 147 │         6.3         2.5          5.0         1.9  virginica\n",
       " 148 │         6.5         3.0          5.2         2.0  virginica\n",
       " 149 │         6.2         3.4          5.4         2.3  virginica\n",
       " 150 │         5.9         3.0          5.1         1.8  virginica\n",
       "\u001b[36m                                                   135 rows omitted\u001b[0m"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "iris = dataset(\"datasets\", \"iris\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "150-element CategoricalArrays.CategoricalArray{String,1,UInt8}:\n",
       " \"setosa\"\n",
       " \"setosa\"\n",
       " \"setosa\"\n",
       " \"setosa\"\n",
       " \"setosa\"\n",
       " \"setosa\"\n",
       " \"setosa\"\n",
       " \"setosa\"\n",
       " \"setosa\"\n",
       " \"setosa\"\n",
       " ⋮\n",
       " \"virginica\"\n",
       " \"virginica\"\n",
       " \"virginica\"\n",
       " \"virginica\"\n",
       " \"virginica\"\n",
       " \"virginica\"\n",
       " \"virginica\"\n",
       " \"virginica\"\n",
       " \"virginica\""
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "X = Matrix(iris[:,1:4])\n",
    "irislabels = iris[:,5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "150×4 Matrix{Float64}:\n",
       " 5.1  3.5  1.4  0.2\n",
       " 4.9  3.0  1.4  0.2\n",
       " 4.7  3.2  1.3  0.2\n",
       " 4.6  3.1  1.5  0.2\n",
       " 5.0  3.6  1.4  0.2\n",
       " 5.4  3.9  1.7  0.4\n",
       " 4.6  3.4  1.4  0.3\n",
       " 5.0  3.4  1.5  0.2\n",
       " 4.4  2.9  1.4  0.2\n",
       " 4.9  3.1  1.5  0.1\n",
       " ⋮              \n",
       " 6.9  3.1  5.1  2.3\n",
       " 5.8  2.7  5.1  1.9\n",
       " 6.8  3.2  5.9  2.3\n",
       " 6.7  3.3  5.7  2.5\n",
       " 6.7  3.0  5.2  2.3\n",
       " 6.3  2.5  5.0  1.9\n",
       " 6.5  3.0  5.2  2.0\n",
       " 6.2  3.4  5.4  2.3\n",
       " 5.9  3.0  5.1  1.8"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "X"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "150-element Vector{Int64}:\n",
       " 1\n",
       " 1\n",
       " 1\n",
       " 1\n",
       " 1\n",
       " 1\n",
       " 1\n",
       " 1\n",
       " 1\n",
       " 1\n",
       " ⋮\n",
       " 3\n",
       " 3\n",
       " 3\n",
       " 3\n",
       " 3\n",
       " 3\n",
       " 3\n",
       " 3\n",
       " 3"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "irislabelsmap = labelmap(irislabels)\n",
    "y = labelencode(irislabelsmap, irislabels)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In classification, we often want to use some of the data to fit a model, and the rest of the data to validate (commonly known as `training` and `testing` data). We will get this data ready now so that we can easily use it in the rest of this notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "perclass_splits (generic function with 1 method)"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "function perclass_splits(y,at)\n",
    "    uids = unique(y)\n",
    "    keepids = []\n",
    "    for ui in uids\n",
    "        curids = findall(y.==ui)\n",
    "        rowids = randsubseq(curids, at) \n",
    "        push!(keepids,rowids...)\n",
    "    end\n",
    "    return keepids\n",
    "end"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "ename": "Base.Meta.ParseError",
     "evalue": "ParseError:\n# Error @ c:\\Users\\chadi\\OneDrive - Handelshögskolan i Stockholm\\GitHub\\JuliaAcademy\\DataScience\\06. Classification.ipynb:1:1\n?randsubseq\n╙ ── not a unary operator",
     "output_type": "error",
     "traceback": [
      "ParseError:\n",
      "# Error @ c:\\Users\\chadi\\OneDrive - Handelshögskolan i Stockholm\\GitHub\\JuliaAcademy\\DataScience\\06. Classification.ipynb:1:1\n",
      "?randsubseq\n",
      "╙ ── not a unary operator\n",
      "\n",
      "Stacktrace:\n",
      " [1] top-level scope\n",
      "   @ c:\\Users\\chadi\\OneDrive - Handelshögskolan i Stockholm\\GitHub\\JuliaAcademy\\DataScience\\06. Classification.ipynb:1"
     ]
    }
   ],
   "source": [
    "?randsubseq"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "42-element Vector{Any}:\n",
       "   4\n",
       "   7\n",
       "   8\n",
       "   9\n",
       "  15\n",
       "  18\n",
       "  23\n",
       "  24\n",
       "  32\n",
       "  36\n",
       "   ⋮\n",
       " 110\n",
       " 111\n",
       " 118\n",
       " 120\n",
       " 122\n",
       " 125\n",
       " 128\n",
       " 131\n",
       " 144"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "trainids = perclass_splits(y,0.7)\n",
    "testids = setdiff(1:length(y),trainids)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will need one more function, and that is the function that will assign classes based on the predicted values when the predicted values are continuous."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "assign_class (generic function with 1 method)"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "assign_class(predictedvalue) = argmin(abs.(predictedvalue .- [1,2,3]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 🟣 Method 1: Lasso"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Least Squares GLMNet Cross Validation\n",
       "72 models for 4 predictors in 10 folds\n",
       "Best λ 0.001 (mean loss 0.051, std 0.004)"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "path = glmnet(X[trainids,:], y[trainids])\n",
    "cv = glmnetcv(X[trainids,:], y[trainids])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "# choose the best lambda to predict with.\n",
    "path = glmnet(X[trainids,:], y[trainids])\n",
    "cv = glmnetcv(X[trainids,:], y[trainids])\n",
    "mylambda = path.lambda[argmin(cv.meanloss)]\n",
    "\n",
    "path = glmnet(X[trainids,:], y[trainids],lambda=[mylambda]);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "42×1 Matrix{Float64}:\n",
       " 1.0040992769561932\n",
       " 1.0361161770051204\n",
       " 0.9523291283672816\n",
       " 1.0050551619598065\n",
       " 0.7730252863195113\n",
       " 0.9731927824309166\n",
       " 0.8745734455148836\n",
       " 1.1687043679649192\n",
       " 1.0209576717740387\n",
       " 0.8778668861605605\n",
       " ⋮\n",
       " 3.200872930452052\n",
       " 2.7419953817495664\n",
       " 3.111155867284884\n",
       " 2.489403278349563\n",
       " 2.8061798949993846\n",
       " 2.927186091861226\n",
       " 2.624159111092738\n",
       " 2.8258288298054293\n",
       " 3.0843379624584486"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "q = X[testids,:];\n",
    "predictions_lasso = GLMNet.predict(path,q)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9523809523809523"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "predictions_lasso = assign_class.(predictions_lasso)\n",
    "findaccuracy(predictions_lasso,y[testids])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 🟣 Method 2: Ridge\n",
    "We will use the same function but set alpha to zero."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9761904761904762"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# choose the best lambda to predict with.\n",
    "path = glmnet(X[trainids,:], y[trainids],alpha=0);\n",
    "cv = glmnetcv(X[trainids,:], y[trainids],alpha=0)\n",
    "mylambda = path.lambda[argmin(cv.meanloss)]\n",
    "path = glmnet(X[trainids,:], y[trainids],alpha=0,lambda=[mylambda]);\n",
    "q = X[testids,:];\n",
    "predictions_ridge = GLMNet.predict(path,q)\n",
    "predictions_ridge = assign_class.(predictions_ridge)\n",
    "findaccuracy(predictions_ridge,y[testids])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 🟣 Method 3: Elastic Net\n",
    "We will use the same function but set alpha to 0.5 (it's the combination of lasso and ridge)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9523809523809523"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# choose the best lambda to predict with.\n",
    "path = glmnet(X[trainids,:], y[trainids],alpha=0.5);\n",
    "cv = glmnetcv(X[trainids,:], y[trainids],alpha=0.5)\n",
    "mylambda = path.lambda[argmin(cv.meanloss)]\n",
    "path = glmnet(X[trainids,:], y[trainids],alpha=0.5,lambda=[mylambda]);\n",
    "q = X[testids,:];\n",
    "predictions_EN = GLMNet.predict(path,q)\n",
    "predictions_EN = assign_class.(predictions_EN)\n",
    "findaccuracy(predictions_EN,y[testids])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 🟣 Method 4: Decision Trees\n",
    "We will use the package `DecisionTree`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "DecisionTreeClassifier\n",
       "max_depth:                2\n",
       "min_samples_leaf:         1\n",
       "min_samples_split:        2\n",
       "min_purity_increase:      0.0\n",
       "pruning_purity_threshold: 1.0\n",
       "n_subfeatures:            0\n",
       "classes:                  [1, 2, 3]\n",
       "root:                     Decision Tree\n",
       "Leaves: 3\n",
       "Depth:  2"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "model = DecisionTreeClassifier(max_depth=2)\n",
    "DecisionTree.fit!(model, X[trainids,:], y[trainids])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9523809523809523"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "q = X[testids,:];\n",
    "predictions_DT = DecisionTree.predict(model, q)\n",
    "findaccuracy(predictions_DT,y[testids])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 🟣 Method 5: Random Forests\n",
    "The `RandomForestClassifier` is available through the `DecisionTree` package as well."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "RandomForestClassifier\n",
       "n_trees:             20\n",
       "n_subfeatures:       -1\n",
       "partial_sampling:    0.7\n",
       "max_depth:           -1\n",
       "min_samples_leaf:    1\n",
       "min_samples_split:   2\n",
       "min_purity_increase: 0.0\n",
       "classes:             [1, 2, 3]\n",
       "ensemble:            Ensemble of Decision Trees\n",
       "Trees:      20\n",
       "Avg Leaves: 5.9\n",
       "Avg Depth:  4.3"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "model = RandomForestClassifier(n_trees=20)\n",
    "DecisionTree.fit!(model, X[trainids,:], y[trainids])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9761904761904762"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "q = X[testids,:];\n",
    "predictions_RF = DecisionTree.predict(model, q)\n",
    "findaccuracy(predictions_RF,y[testids])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 🟣 Method 6: Using a Nearest Neighbor method\n",
    "We will use the `NearestNeighbors` package here."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "KDTree{StaticArraysCore.SVector{4, Float64}, Euclidean, Float64, StaticArraysCore.SVector{4, Float64}}\n",
       "  Number of points: 108\n",
       "  Dimensions: 4\n",
       "  Metric: Euclidean(0.0)\n",
       "  Reordered: true"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "Xtrain = X[trainids,:]\n",
    "ytrain = y[trainids]\n",
    "kdtree = KDTree(Xtrain')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "42×4 Matrix{Float64}:\n",
       " 4.6  3.1  1.5  0.2\n",
       " 4.6  3.4  1.4  0.3\n",
       " 5.0  3.4  1.5  0.2\n",
       " 4.4  2.9  1.4  0.2\n",
       " 5.8  4.0  1.2  0.2\n",
       " 5.1  3.5  1.4  0.3\n",
       " 4.6  3.6  1.0  0.2\n",
       " 5.1  3.3  1.7  0.5\n",
       " 5.4  3.4  1.5  0.4\n",
       " 5.0  3.2  1.2  0.2\n",
       " ⋮              \n",
       " 7.2  3.6  6.1  2.5\n",
       " 6.5  3.2  5.1  2.0\n",
       " 7.7  3.8  6.7  2.2\n",
       " 6.0  2.2  5.0  1.5\n",
       " 5.6  2.8  4.9  2.0\n",
       " 6.7  3.3  5.7  2.1\n",
       " 6.1  3.0  4.9  1.8\n",
       " 7.4  2.8  6.1  1.9\n",
       " 6.8  3.2  5.9  2.3"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "queries = X[testids,:]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "([[35, 22, 23, 3, 34], [35, 3, 8, 31, 22], [1, 19, 4, 8, 21], [28, 31, 10, 35, 9], [25, 12, 11, 13, 7], [1, 29, 4, 21, 20], [3, 27, 29, 31, 35], [19, 32, 20, 15, 18], [15, 20, 21, 7, 36], [2, 3, 29, 26, 21]  …  [85, 75, 95, 83, 87], [84, 103, 71, 95, 87], [106, 81, 104, 97, 82], [91, 95, 85, 87, 75], [54, 61, 50, 105, 79], [70, 102, 79, 108, 80], [84, 78, 100, 73, 103], [98, 88, 108, 52, 86], [75, 71, 87, 90, 95], [84, 103, 100, 73, 71]], [[0.14142135623730964, 0.17320508075688812, 0.22360679774997916, 0.24494897427831802, 0.26457513110645925], [0.22360679774997871, 0.264575131106459, 0.3000000000000002, 0.3162277660168373, 0.31622776601683805], [0.17320508075688762, 0.22360679774997902, 0.22360679774997916, 0.22360679774997916, 0.22360679774997916], [0.14142135623730948, 0.31622776601683816, 0.3464101615137755, 0.3605551275463988, 0.42426406871192807], [0.412310562561766, 0.46904157598234253, 0.5477225575051664, 0.5567764362830022, 0.5830951894845297], [0.09999999999999998, 0.14142135623730917, 0.17320508075688756, 0.17320508075688806, 0.17320508075688806], [0.5099019513592785, 0.5099019513592788, 0.5196152422706636, 0.53851648071345, 0.5656854249492379], [0.1999999999999998, 0.264575131106459, 0.4242640687119287, 0.4358898943540679, 0.4472135954999577], [0.282842712474619, 0.30000000000000016, 0.30000000000000016, 0.3605551275463992, 0.37416573867739455], [0.3, 0.31622776601683783, 0.33166247903553986, 0.33166247903553997, 0.34641016151377535]  …  [0.26457513110645964, 0.5291502622129179, 0.5477225575051662, 0.5477225575051664, 0.8062257748298545], [0.6708203932499366, 0.7071067811865474, 0.7549834435270749, 0.806225774829855, 0.8124038404635959], [0.22360679774997935, 0.374165738677394, 0.4242640687119287, 0.46904157598234314, 0.48989794855663593], [0.4123105625617661, 1.0049875621120892, 1.019803902718557, 1.1224972160321824, 1.1357816691600549], [0.43588989435406705, 0.5196152422706631, 0.5385164807134505, 0.5830951894845299, 0.6557438524302], [0.31622776601683755, 0.31622776601683755, 0.33166247903553986, 0.4582575694955842, 0.4898979485566353], [0.2999999999999998, 0.374165738677394, 0.374165738677394, 0.3872983346207416, 0.3999999999999999], [0.14142135623730964, 0.24494897427831838, 0.282842712474618, 0.2999999999999998, 0.3605551275463989], [0.26457513110645936, 0.45825756949558427, 0.46904157598234325, 0.5099019513592784, 0.5385164807134504], [0.22360679774997935, 0.31622776601683794, 0.346410161513776, 0.3872983346207417, 0.4123105625617659]])"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "idxs, dists = knn(kdtree, queries', 5, true)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9523809523809523"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "c = ytrain[hcat(idxs...)]\n",
    "possible_labels = map(i->counter(c[:,i]),1:size(c,2))\n",
    "predictions_NN = map(i->parse(Int,string(string(argmax(possible_labels[i])))),1:size(c,2))\n",
    "findaccuracy(predictions_NN,y[testids])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 🟣 Method 7: Support Vector Machines\n",
    "We will use the `LIBSVM` package here."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "108-element Vector{Int64}:\n",
       " 1\n",
       " 1\n",
       " 1\n",
       " 1\n",
       " 1\n",
       " 1\n",
       " 1\n",
       " 1\n",
       " 1\n",
       " 1\n",
       " ⋮\n",
       " 3\n",
       " 3\n",
       " 3\n",
       " 3\n",
       " 3\n",
       " 3\n",
       " 3\n",
       " 3\n",
       " 3"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "Xtrain = X[trainids,:]\n",
    "ytrain = y[trainids]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "LIBSVM.SVM{Int64, LIBSVM.Kernel.KERNEL}(SVC, LIBSVM.Kernel.RadialBasis, nothing, 4, 108, 3, [1, 2, 3], Int32[1, 2, 3], Float64[], Int32[], LIBSVM.SupportVectors{Vector{Int64}, Matrix{Float64}}(36, Int32[5, 15, 16], [1, 1, 1, 1, 1, 2, 2, 2, 2, 2  …  3, 3, 3, 3, 3, 3, 3, 3, 3, 3], [4.3 5.7 … 6.5 5.9; 3.0 4.4 … 3.0 3.0; 1.1 1.5 … 5.2 5.1; 0.1 0.4 … 2.0 1.8], Int32[10, 11, 17, 30, 33, 37, 39, 41, 44, 46  …  91, 93, 94, 96, 98, 101, 102, 105, 106, 108], LIBSVM.SVMNode[LIBSVM.SVMNode(1, 4.3), LIBSVM.SVMNode(1, 5.7), LIBSVM.SVMNode(1, 4.8), LIBSVM.SVMNode(1, 4.5), LIBSVM.SVMNode(1, 5.1), LIBSVM.SVMNode(1, 7.0), LIBSVM.SVMNode(1, 6.9), LIBSVM.SVMNode(1, 4.9), LIBSVM.SVMNode(1, 5.0), LIBSVM.SVMNode(1, 6.1)  …  LIBSVM.SVMNode(1, 7.9), LIBSVM.SVMNode(1, 6.3), LIBSVM.SVMNode(1, 6.1), LIBSVM.SVMNode(1, 6.3), LIBSVM.SVMNode(1, 6.0), LIBSVM.SVMNode(1, 6.9), LIBSVM.SVMNode(1, 5.8), LIBSVM.SVMNode(1, 6.3), LIBSVM.SVMNode(1, 6.5), LIBSVM.SVMNode(1, 5.9)]), 0.0, [0.0 0.008937516838437798; 0.6305577655011664 0.9526296495472665; … ; -0.0 -1.0; -0.0 -1.0], Float64[], Float64[], [0.042151505874140086, 0.16578174238396193, 0.1431261941753951], 3, 0.25, 200.0, 0.001, 1.0, 0.5, 0.1, true, false)"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "model = svmtrain(Xtrain', ytrain)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9761904761904762"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "predictions_SVM, decision_values = svmpredict(model, X[testids,:]')\n",
    "findaccuracy(predictions_SVM,y[testids])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Putting all the results together:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "7×2 Matrix{Any}:\n",
       " \"lasso\"  0.952381\n",
       " \"ridge\"  0.97619\n",
       " \"EN\"     0.952381\n",
       " \"DT\"     0.952381\n",
       " \"RF\"     0.97619\n",
       " \"kNN\"    0.952381\n",
       " \"SVM\"    0.97619"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "overall_accuracies = zeros(7)\n",
    "methods = [\"lasso\",\"ridge\",\"EN\", \"DT\", \"RF\",\"kNN\", \"SVM\"]\n",
    "ytest = y[testids]\n",
    "overall_accuracies[1] = findaccuracy(predictions_lasso,ytest)\n",
    "overall_accuracies[2] = findaccuracy(predictions_ridge,ytest)\n",
    "overall_accuracies[3] = findaccuracy(predictions_EN,ytest)\n",
    "overall_accuracies[4] = findaccuracy(predictions_DT,ytest)\n",
    "overall_accuracies[5] = findaccuracy(predictions_RF,ytest)\n",
    "overall_accuracies[6] = findaccuracy(predictions_NN,ytest)\n",
    "overall_accuracies[7] = findaccuracy(predictions_SVM,ytest)\n",
    "hcat(methods, overall_accuracies)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Finally...\n",
    "After finishing this notebook, you should be able to:\n",
    "- [ ] split your data into training and testing data to test the effectiveness of a certain method\n",
    "- [ ] apply a simple accuracy function to test the effectiveness of a certain method\n",
    "- [ ] run multiple classification algorithms:\n",
    "    - [ ] LASSO\n",
    "    - [ ] Ridge\n",
    "    - [ ] ElasticNet\n",
    "    - [ ] Decision Tree\n",
    "    - [ ] Random Forest\n",
    "    - [ ] Nearest Neighbors\n",
    "    - [ ] Support Vector Machines"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 🥳 One cool finding\n",
    "\n",
    "We used multiple methods to run classification on the `iris` dataset which is a dataset of flowers and there are three types of iris flowers in it. We split the data into training and testing and ran our methods. Here is the scoreboard:\n",
    "\n",
    "| method | accuracy score |\n",
    "|---|---|\n",
    "| lasso  |1.0|\n",
    "| ridge  |1.0|\n",
    "| EN     |1.0|\n",
    "| DT     |0.960784|\n",
    "| RF     |0.980392|\n",
    "| kNN    |1.0|\n",
    "| SVM    |1.0|"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Julia 1.10.2",
   "language": "julia",
   "name": "julia-1.10"
  },
  "language_info": {
   "file_extension": ".jl",
   "mimetype": "application/julia",
   "name": "julia",
   "version": "1.10.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}