codes/ML/logistic_regression.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "view-in-github"
   },
   "source": [
    "<a href=\"https://colab.research.google.com/github/cobanov/zemin_istanbul/blob/master/logisticRegression.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "O1lN5imPlfWb"
   },
   "source": [
    "                                    All rights reserved © Global AI Hub 2020 \n",
    "![](img/logo.png)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "4CTfJC29leb8"
   },
   "source": [
    "# Logistic Regression "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "_iq3Lo2Eleb9"
   },
   "source": [
    "Logistic regression is a fundamental classification technique. It belongs to the group of linear classifiers and is somewhat similar to polynomial and linear regression. Logistic regression is fast and relatively uncomplicated, and it’s convenient for you to interpret the results. Although it’s essentially a method for binary classification, it can also be applied to multiclass problems."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "GA8bmkyAleb-"
   },
   "source": [
    "You’ll need an understanding of the sigmoid function and the natural logarithm function to understand what logistic regression is and how it works.\n",
    "\n",
    "This image shows the sigmoid function (or S-shaped curve) of some variable 𝑥:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "s97qP9XJleb-"
   },
   "source": [
    "<img src=\"img/s.jpeg\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "8euP2T8zleb_"
   },
   "source": [
    "The sigmoid function has values very close to either 0 or 1 across most of its domain. This fact makes it suitable for application in classification methods."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "mWzlA4vqleb_"
   },
   "source": [
    "## Single-Variate Logistic Regression"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "pzzWSR2glecA"
   },
   "source": [
    "Single-variate logistic regression is the most straightforward case of logistic regression. There is only one independent variable (or feature), which is 𝐱 = 𝑥. This figure illustrates single-variate logistic regression:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "CfiHv1p1lecB"
   },
   "source": [
    "<img src=\"img/logis.jpg\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "WdQx45HKlecB"
   },
   "source": [
    "Here, you have a given set of input-output (or 𝑥-𝑦) pairs, represented by green circles. These are your observations. Remember that 𝑦 can only be 0 or 1. For example, the leftmost green circle has the input 𝑥 = 0 and the actual output 𝑦 = 0. The rightmost observation has 𝑥 = 9 and 𝑦 = 1.\n",
    "\n",
    "Logistic regression finds the weights 𝑏₀ and 𝑏₁ that correspond to the maximum log-likelihood function (LLF). These weights define the logit 𝑓(𝑥) = 𝑏₀ + 𝑏₁𝑥, which is the dashed black line. They also define the predicted probability $p(x) = 1 / (1 + exp(−𝑓(𝑥)))$, shown here as the full black line. In this case, the threshold 𝑝(𝑥) = 0.5 and 𝑓(𝑥) = 0 corresponds to the value of 𝑥 slightly higher than 3. This value is the limit between the inputs with the predicted outputs of 0 and 1."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "GXp6HofvlecF"
   },
   "source": [
    "## Logistic Regression in Python"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "ooCLpj5WlecF"
   },
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.metrics import classification_report, confusion_matrix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "ts_LzMz3lecJ"
   },
   "outputs": [],
   "source": [
    "x = np.arange(10).reshape(-1, 1)\n",
    "y = np.array([0, 0, 0, 0, 1, 1, 1, 1, 1, 1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "ISWMaQmKlecL"
   },
   "outputs": [],
   "source": [
    "model = LogisticRegression(random_state = 42)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "kEvfhhlvlecO",
    "outputId": "afc1edc9-773d-48ed-b22c-cb8b6d757828"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "LogisticRegression(C=1.0, class_weight=None, dual=False, fit_intercept=True,\n",
       "                   intercept_scaling=1, l1_ratio=None, max_iter=100,\n",
       "                   multi_class='auto', n_jobs=None, penalty='l2',\n",
       "                   random_state=42, solver='lbfgs', tol=0.0001, verbose=0,\n",
       "                   warm_start=False)"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.fit(x, y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "P1KY8ygglecS"
   },
   "source": [
    "LogisticRegression(C=1.0, class_weight=None, dual=False, fit_intercept=True,\n",
    "                   intercept_scaling=1, l1_ratio=None, max_iter=100,\n",
    "                   multi_class='warn', n_jobs=None, penalty='l2',\n",
    "                   random_state=0, solver='liblinear', tol=0.0001, verbose=0,\n",
    "                   warm_start=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "TWyHYujKleca",
    "outputId": "6efac8d8-9adf-4ac5-e982-fae7942ad8b4"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Classes:  [0 1]\n",
      "Intercept:  [-4.12617727]\n",
      "Coef:  [[1.18109091]]\n"
     ]
    }
   ],
   "source": [
    "print(\"Classes: \", model.classes_)\n",
    "print(\"Intercept: \",model.intercept_)\n",
    "print(\"Coef: \",model.coef_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Probability:  [[0.98411203 0.01588797]\n",
      " [0.95003074 0.04996926]\n",
      " [0.85370936 0.14629064]\n",
      " [0.64173546 0.35826454]\n",
      " [0.35475873 0.64524127]\n",
      " [0.1443924  0.8556076 ]\n",
      " [0.04924876 0.95075124]\n",
      " [0.01565079 0.98434921]\n",
      " [0.00485659 0.99514341]\n",
      " [0.00149573 0.99850427]]\n"
     ]
    }
   ],
   "source": [
    "print(\"Probability: \",model.predict_proba(x))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "WD6Inaewlecl",
    "outputId": "ecd200d5-70e5-498d-fa31-06c8d9b0cc26"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0, 0, 0, 0, 1, 1, 1, 1, 1, 1])"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.predict(x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "inYjtdS9lecq",
    "outputId": "c8a25a25-2ab6-445f-af43-a6c298943ac8"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[4, 0],\n",
       "       [0, 6]], dtype=int64)"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "confusion_matrix(y, model.predict(x))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "U3CLyPvTlect",
    "outputId": "b41cb437-bf7b-4296-a2fa-6777aaa496f3"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.axes._subplots.AxesSubplot at 0x1cfa43b1748>"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAVQAAAD8CAYAAAAoqlyCAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAORklEQVR4nO3dfYydZZnH8d91hqktAqK8tZ1WixQL+FK7KSUE/0BFWoWCxBWsEbKEOL6nJRuhIsmuWUmMfyCQEM1RsEQBmdQlutB2NVZSicC23RS2nWmhtEDPdBCNb9BlYeaca//ooTnWmWfOaa+Z+5x7vp/mSeec5+1OOvn1up/7ee7H3F0AgKNXSt0AAMgFgQoAQQhUAAhCoAJAEAIVAIIQqAAQhEAFgDGY2YlmttbMdprZgJmdX7T9MZPVMADoQLdL2uDu/2hm0yQdW7SxcWM/APw9MztB0pOS3ulNBuWEV6h/ufYiEht/56R7B1I3AW1o5PVBO9pjDP9hT9OZM+2UMz4nqbfhq7K7l+s/v1PS7yX90MwWStoqaaW7HxjreFxDBTBluXvZ3Rc3LOWG1cdI+gdJ33X3RZIOSFpddDwCFUBeatXml2IVSRV3f6L+ea0OBuyYGJQCkJfqSMhh3P1FM9tnZgvcfZekD0vqL9qHQAWQFfda5OG+Iune+gj/HknXFm1MoALISy0uUN19m6TFzW5PoALIS2yF2hICFUBexh9smjAEKoC8UKECQAwPGuU/EgQqgLwEDkq1ikAFkBe6/AAQhEEpAAhChQoAQRiUAoAgDEoBQAx3rqECQAyuoQJAELr8ABCEChUAglSHk52aQAWQF7r8ABCELj8ABKFCBYAgBCoAxHAGpQAgCNdQASAIXX4ACEKFCgBBqFABIAgVKgAEGWGCaQCIQYUKAEECr6Ga2XOSXpZUlTTi7ouLtidQAeQlvkL9oLv/oZkNCVQAeUk4yl9KdmYAmAhea35p4miSfmFmW82sd7yNqVAB5KWFUf56SDYGZdndyw2fL3D3/WZ2qqRfmtlOd9801vEIVAB5cW9hUy9LKhes31//+yUze1DSEkljBipdfgB5qdWaXwqY2ZvN7Pg3fpZ0saTtRftQoQLIS9yg1GmSHjQz6WBW3ufuG4p2IFAB5CXotil33yNpYSv7EKgA8lKtJjs1gQogL8w2BQBBCFQACMLkKAAQw2vN34cajUAFkBe6/AAQhFF+AAhChQoAQZi+bwqwko771+/p2JXfTN0StJGlF1+oHds3aWf/o7rhq19K3Zw8uDe/BCNQJ8m0j1yh6tALqZuBNlIqlXTH7bfo0uWf0XsXflBXXfVxnX32mamb1fmCJkc5EuMGqpmdZWY3mtkdZnZ7/eezw1uSMXvryepeeJ5e37QudVPQRpacu0jPPvuc9u59QcPDw+rr+5kuW740dbM6X82bX4IVBqqZ3SjpJ5JM0n9J2lz/+X4zWx3emkzNWPFFvdr3/Qn5B0Tnmt0zU/sq+w99rgwOafbsmQlblIlqtfkl2HiDUtdJere7Dzd+aWa3Stoh6Vuj7dQ4C/Zt55+lf1rQE9DUznTMwvNUe/nPqj3/jLoWtDRxDTJXnxbub/gEXNebaryNR/lrkmZLev6w72fV142qcRbsv1x70ZT+Dek68z3qfv/56n7fEql7mmz6sZrRu1qvlkf9vwhTyGBlSHPnzD70eU7PLA0N/S5hizLRxk9KrZL0KzN7RtK++ndvlzRf0pcnsmG5eG3tXXpt7V2SpK4FC/WmZZ8kTCFJ2rxlm+bPP13z5s3V4OCLuvLKy3X1NYz0H7V2fZbf3TeY2bt08D0qPTp4/bQiabO7p3scAchAtVrVylU3a93D96mrVNKaex5Qf//TqZvV+dq4QpW71yQ9PgltyV5115P6311Ppm4G2sj6DRu1fsPG1M3IywiPngJAjHbt8gNAx2nnLj8AdJJ2vm0KADoLFSoABCFQASAIE0wDQAzeKQUAUQhUAAjCKD8ABKFCBYAgwYFqZl2StkgadPdLi7YlUAFkxavhXf6VkgYknTDehrxTCkBeAl+BYmZzJF0i6QfNnJpABZAVr3nTi5n1mtmWhqX3sMPdJukGFUyo34guP4C8tHANtfHtIoczs0slveTuW83swmaOR6ACyEvcJdQLJF1mZh+TNF3SCWb2Y3f/zFg70OUHkBUfqTW9FB7H/WvuPsfd50n6lKSNRWEqUaECyE26+/oJVAB5mYhn+d39EUmPjLcdgQogL1SoABCD2aYAIAoVKgDE8JF05yZQAWQl4VukCVQAmSFQASAGFSoABCFQASCIVy3ZuQlUAFmhQgWAIF6jQgWAEFSoABDEnQoVAEJQoQJAkBqj/AAQg0EpAAhCoAJAEE83HSqBCiAvVKgAEITbpgAgSJVRfgCIQYUKAEG4hgoAQRjlB4AgVKgAEKRaKyU7N4EKICt0+QEgSC1olN/MpkvaJOlNOpiVa939X4r2IVABZCXwtqnXJH3I3V8xs25Jj5rZend/fKwdCFQAWYnq8ru7S3ql/rG7vhQefcID9aR7Byb6FOhAr+7/TeomIFOtdPnNrFdSb8NXZXcvN6zvkrRV0nxJd7r7E0XHo0IFkJVWRvnr4VkuWF+V9H4zO1HSg2b2HnffPtb26e4vAIAJ4C0sTR/T/c+SHpG0rGg7AhVAVmpuTS9FzOyUemUqM5sh6SJJO4v2ocsPICuBo/yzJN1Tv45aktTn7g8V7UCgAshK1EtP3f0pSYta2YdABZAVF8/yA0CIEeZDBYAYVKgAECTqGuqRIFABZIUKFQCCUKECQJAqFSoAxEj4BhQCFUBealSoABAj4RtQCFQAeWFQCgCC1IwuPwCEqCY8N4EKICuM8gNAEEb5ASAIo/wAEIQuPwAE4bYpAAhSpUIFgBhUqAAQhEAFgCAJXylFoALICxUqAATh0VMACMJ9qAAQhC4/AARJGailhOcGgHDewlLEzOaa2a/NbMDMdpjZyvHOTYUKICuB11BHJP2zu/+3mR0vaauZ/dLd+8fagUAFkJWoUX53H5I0VP/5ZTMbkNQjacxApcsPICs1edOLmfWa2ZaGpXe0Y5rZPEmLJD1RdG4qVABZaWVQyt3LkspF25jZcZJ+KmmVu/+1aFsCFUBWIieYNrNuHQzTe93938fbnkAFkJWo26bMzCTdJWnA3W9tZh8CFUBWRiysRr1A0tWS/sfMttW/u8nd1421A4EKICtRceruj0qtvfGPQAWQFR49BYAgtYTvPSVQAWSF10gDQBC6/AAQpEqXHwBiUKECQBCnQgWAGEwwPQUsvfhC7di+STv7H9UNX/1S6uagTfz15Vd0/de/qeUrPqvln+7Vtu0DqZvU8VqZbSoaFeokKJVKuuP2W7TsYytUqQzp8cfW6T8e+oUGBp5J3TQk9q3bvqcLzlus79xys4aHh/Xq/72WukkdL+VtU1Sok2DJuYv07LPPae/eFzQ8PKy+vp/psuVLUzcLib1y4IC2Prldn6j/LnR3d+uE449L3KrONyJveolGhToJZvfM1L7K/kOfK4NDWnLuooQtQjuoDL6ot574Ft18y63atXuPzllwplav+ryOnTE9ddM6WspBqSOuUM3s2oJ1h2bBrtUOHOkpsnFwFrC/5Z6yY4J2MFKtauDp3brqiku0ds2dmjFjuu76UV/qZnW8WgtLtKPp8n9jrBXuXnb3xe6+uFR681GcIg+DlSHNnTP70Oc5PbM0NPS7hC1CO5h56sk67ZST9b53nyVJuvjCD6j/6d2JW9X5vIU/0Qq7/Gb21FirJJ0W3ppMbd6yTfPnn6558+ZqcPBFXXnl5br6Gkb6p7qTT3qbZp56ivY+X9Hp75ijx7du0xnz3p66WR2vnW/sP03SUkl/Oux7k/TbCWlRhqrVqlauulnrHr5PXaWS1tzzgPr7n07dLLSBm67/gm78xrc1PDKsubNn6d9uuj51kzpeNeHltPEC9SFJx7n7tsNXmNkjE9KiTK3fsFHrN2xM3Qy0mbPedYb67r4jdTOy0rbT97n7dQXrPh3fHAA4Ojx6CgBB2vkaKgB0lLbt8gNAp6HLDwBB2nmUHwA6Cl1+AAjCoBQABOEaKgAEocsPAEFSzuTGBNMAslKVN72Mx8zuNrOXzGx7M+cmUAFkJfidUmskLWv23HT5AWQlssvv7pvMbF6z2xOoALKSclCKLj+ArLQyY3/j65rqS+/RnJsKFUBWWnn01N3LkspR5yZQAWSFLj8ABIkc5Tez+yU9JmmBmVXMbMxJ9yUqVACZCR7lX9HK9gQqgKzw6CkABGFyFAAIUvV0E/gRqACyknJyFAIVQFa4hgoAQbiGCgBBanT5ASAGFSoABGGUHwCC0OUHgCB0+QEgCBUqAAShQgWAIFWvJjs3gQogKzx6CgBBePQUAIJQoQJAEEb5ASAIo/wAEIRHTwEgCNdQASAI11ABIAgVKgAE4T5UAAhChQoAQRjlB4AgDEoBQJCUXf5SsjMDwATwFv6Mx8yWmdkuM9ttZqvH254KFUBWoipUM+uSdKekj0iqSNpsZj939/6x9iFQAWQl8BrqEkm73X2PJJnZTyRdLildoI68PmgTfY5OYWa97l5O3Q60F34vYrWSOWbWK6m34atyw79Fj6R9Desqks4rOh7XUCdX7/ibYAri9yIRdy+7++KGpfE/ttGCubD8JVABYHQVSXMbPs+RtL9oBwIVAEa3WdKZZna6mU2T9ClJPy/agUGpycV1MoyG34s25O4jZvZlSf8pqUvS3e6+o2gfS3kTLADkhC4/AAQhUAEgCIE6SVp9hA35M7O7zewlM9ueui2IQaBOgoZH2D4q6RxJK8zsnLStQhtYI2lZ6kYgDoE6OQ49wubur0t64xE2TGHuvknSH1O3A3EI1Mkx2iNsPYnaAmCCEKiTo+VH2AB0HgJ1crT8CBuAzkOgTo6WH2ED0HkI1Eng7iOS3niEbUBS33iPsCF/Zna/pMckLTCzipldl7pNODo8egoAQahQASAIgQoAQQhUAAhCoAJAEAIVAIIQqAAQhEAFgCD/D+E3VzFVyTbSAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import seaborn as sns\n",
    "\n",
    "cm = confusion_matrix(y, model.predict(x))\n",
    "sns.heatmap(cm, annot=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "hsUoY8_flecv",
    "outputId": "dcc208a4-87be-42df-80c5-93fc0dc646f5"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       1.00      1.00      1.00         4\n",
      "           1       1.00      1.00      1.00         6\n",
      "\n",
      "    accuracy                           1.00        10\n",
      "   macro avg       1.00      1.00      1.00        10\n",
      "weighted avg       1.00      1.00      1.00        10\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(classification_report(y, model.predict(x)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "3wYZj-pllecy",
    "outputId": "8d198813-a002-4cc4-ba2b-56f232758d81"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "LogisticRegression(C=0.5, random_state=0, solver='liblinear')"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model = LogisticRegression(solver='liblinear', C=0.5, random_state=0)\n",
    "model.fit(x, y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "2c1MUas8lec1",
    "outputId": "d993f8fc-2808-47fb-bc1b-a66546e1523f"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([-0.61167085])"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.intercept_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "5MAptsgclec3",
    "outputId": "501c5009-bbce-4f07-d150-ebcef755ead0"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0.41299976]])"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.coef_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "Kl20AVxHlec5",
    "outputId": "de7bd512-6213-4da1-ffd1-d443b89acb0c"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0.64832185, 0.35167815],\n",
       "       [0.54950505, 0.45049495],\n",
       "       [0.44662201, 0.55337799],\n",
       "       [0.34811656, 0.65188344],\n",
       "       [0.26108668, 0.73891332],\n",
       "       [0.18948992, 0.81051008],\n",
       "       [0.13396721, 0.86603279],\n",
       "       [0.09284959, 0.90715041],\n",
       "       [0.06342763, 0.93657237],\n",
       "       [0.04288806, 0.95711194]])"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.predict_proba(x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "30IDSy4Ilec7",
    "outputId": "e6e56ded-0da2-407c-eef9-435d1fae557b"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0, 0, 1, 1, 1, 1, 1, 1, 1, 1])"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.predict(x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "iMBDTGVJlec9",
    "outputId": "4d13e887-6896-4b03-f4f0-1713f63bb735"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.score(x, y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "1XZwGMEZlec_",
    "outputId": "b59aa2f0-687e-4e25-93e0-9eea939c51a1"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[2, 2],\n",
       "       [0, 6]], dtype=int64)"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "confusion_matrix(y, model.predict(x))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<AxesSubplot:>"
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAVQAAAD8CAYAAAAoqlyCAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8vihELAAAACXBIWXMAAAsTAAALEwEAmpwYAAANx0lEQVR4nO3dfYxc1X3G8edZY2JeDIQajLEdTHEAQZXU1eKoIpVoq4ATAqiiSkPSVEppto2aCqOqCSVIUVpcRalEXiRUNEoIUQokFi0JpeAEBZBLC8R266SLlxdjSNi1XZe0KNgiYXfm1z88oBH13p3Bv9k7c/z9WFd4Zu7Lkbg8/M4999zriBAA4PCN1N0AACgFgQoASQhUAEhCoAJAEgIVAJIQqACQhEAFgFnYPsn2XbaftD1h+9er1j9qvhoGAEPoS5I2RcTv2j5a0rFVK5sb+wHg/7N9oqTtkn45ugzKvleoG874MIkNoCuf/vHtPtx9TL+4q+vMOfqUs/5Y0ljHV42IaLT/fqak/5b0NdvvlLRN0jURcWC2/XENFcARKyIaETHasTQ6fj5K0q9J+ruIWCPpgKTrqvZHoAIoS6vZ/VJtUtJkRDze/nyXDgbsrBiUAlCW5kzKbiJir+0XbJ8TEU9J+m1JO6q2IVABFCWilbm7P5N0e3uEf5ekj1atTKACKEsrL1AjYruk0W7XJ1ABlCW3Qu0JgQqgLHMPNvUNgQqgLFSoAJAjkkb53wwCFUBZEgelekWgAigLXX4ASMKgFAAkoUIFgCQMSgFAEgalACBHBNdQASAH11ABIAldfgBIQoUKAEma07UdmkAFUBa6/ACQhC4/ACShQgWAJAQqAOQIBqUAIAnXUAEgCV1+AEhChQoASahQASAJFSoAJJnhAdMAkIMKFQCSJF5Dtf28pJclNSXNRMRo1foEKoCy5FeovxkRL3azIoEKoCw1jvKP1HZkAOiHaHW/dLE3Sd+zvc322FwrU6ECKEsPo/ztkOwMykZENDo+vzsipmyfKukB209GxObZ9kegAihLRA+rRkNSo+L3qfY/99m+W9JaSbMGKl1+AGVptbpfKtg+zvbi1/4u6WJJ41XbUKECKEveoNRSSXfblg5m5R0RsalqAwIVQFmSbpuKiF2S3tnLNgQqgLI0m7UdmkAFUBaeNgUASQhUAEjCw1EAIEe0ur8PNRuBCqAsdPkBIAmj/ACQhAoVAJIQqGVbvOxkXf6Fj+u4JSdKEfqPOx7Ulq99t+5moWacF33Sw8NRshGo8yCaLX3/xtu1d/x5HX3cIv3hvTfquUfG9eIzU3U3DTXivOiTQa5QbZ8r6QpJy9tfTUm6JyIm+tmwkuzf95L273tJkvTqgZ/rpzt3a/HSt/IfzhGO86JParxtqvLxfbY/JembkizpB+3Fku60fV3/m1eeE1cs0dLzz9DU9mfrbgoGCOdFomaz+yXZXBXq1ZLOj4jpzi9t3yTpCUmfO9RGnU/BvuLktbrg+NUJTR1+C499i668Zb0e+Ktv6NX9r9TdHAwIzotcMcDvlGpJOv0Q3y9r/3ZIEdGIiNGIGCVMDxo5aoGuvGW9xr/9r3pq09a6m4MBwXnRB63ofkk2V4W6XtL3bT8j6YX2d2+TtFrSJ9JbU7BLP/8x/XTnlH7wlfvrbgoGCOdFHwzqXP6I2GT7bB18j0rnoNSWiKhvOsKQWTF6tt5x5W/ovyZ+oj+6728kSQ/97bf07EM/rLllqBPnRZ8M8lz+iGhJemwe2lKsya1Pa8MZH667GRgwnBd9MsPUUwDIMahdfgAYOoPc5QeAYVLnbVMEKoCyUKECQBICFQCS8IBpAMjBO6UAIAuBCgBJGOUHgCRUqACQJDlQbS+QtFXSVES8v2pdAhVAUaKZ3uW/RtKEpBPmWnGu56ECwHBJfB6q7RWSLpX0lW4OTaACKEq0ouvF9pjtrR3L2Bt290VJn1TFA/U70eUHUJYerqFGRENS41C/2X6/pH0Rsc32Rd3sj0AFUJa8S6gXSrrc9vskLZJ0gu2/j4jfn20DuvwAihIzra6Xyv1E/GVErIiIVZI+KOnBqjCVqFABlKa++/oJVABl6cdc/oh4WNLDc61HoAIoCxUqAOTgaVMAkIUKFQByxEx9xyZQARSlxrdIE6gACkOgAkAOKlQASEKgAkCSaLq2YxOoAIpChQoASaJFhQoAKahQASBJBBUqAKSgQgWAJC1G+QEgB4NSAJCEQAWAJFHf41AJVABloUIFgCTcNgUASZqM8gNADipUAEjCNVQASMIoPwAkoUIFgCTN1khtxyZQARSFLj8AJGkljfLbXiRps6S36GBW3hURn6nahkAFUJTE26Z+Iem3ImK/7YWSHrF9f0Q8NtsGBCqAomR1+SMiJO1vf1zYXir33vdA/cyeh/t9CAyhV3b/S91NQKF66fLbHpM01vFVIyIaHb8vkLRN0mpJN0fE41X7o0IFUJReRvnb4dmo+L0p6VdtnyTpbtu/EhHjs61f3/0FANAH0cPS9T4jXpL0kKR1VesRqACK0gp3vVSxfUq7MpXtYyS9R9KTVdvQ5QdQlMRR/mWSvt6+jjoiaWNE3Fu1AYEKoChZLz2NiB9JWtPLNgQqgKKEmMsPAClmeB4qAOSgQgWAJFnXUN8MAhVAUahQASAJFSoAJGlSoQJAjhrfgEKgAihLiwoVAHLU+AYUAhVAWRiUAoAkLdPlB4AUzRqPTaACKAqj/ACQhFF+AEjCKD8AJKHLDwBJuG0KAJI0qVABIAcVKgAkIVABIEmNr5QiUAGUhQoVAJIw9RQAknAfKgAkocsPAEnqDNSRGo8NAOmih6WK7ZW2H7K9w/YTtq+Z69hUqACKkngNdUbSn0fEv9teLGmb7QciYsdsGxCoAIqSNcofEXsk7Wn//WXbE5KWS5o1UOnyAyhKS9H1YnvM9taOZexQ+7S9StIaSY9XHZsKFUBRehmUioiGpEbVOraPl/QPktZHxM+q1iVQARQl8wHTthfqYJjeHhH/ONf6BCqAomTdNmXbkr4qaSIibupmGwIVQFFmnFajXijpI5L+0/b29nfXR8R9s21AoAIoSlacRsQjUm9v/CNQARSFqacAkKRV43tPCVQAReE10gCQhC4/ACRp0uUHgBxUqACQJKhQASAHD5g+Alxy8UV6YnyzntzxiD75F39ad3MwIH728n5d++kbddlVH9NlHxrT9vGJups09Hp52lQ2KtR5MDIyoi9/aYPWve8qTU7u0WOP3qd/uvd7mph4pu6moWaf++ItuvBdo/rChhs0PT2tV37+i7qbNPTqvG2KCnUerL1gjZ599nk999xPND09rY0bv6PLL7uk7mahZi/vP6BtPxzXle1zYeHChTph8fE1t2r4zSi6XrJRoc6D05efphcmd7/+eXJqj9ZesKbGFmEQTO3eq7eedKJu2HCTntq5S+ed83Zdt/5PdOwxi+pu2lCrc1DqTVeotj9a8dvrT8FutQ682UMARZtpNjXx9E793u9cqrtuu1nHHLNIX/3GxrqbNfRaPSzZDqfL/9nZfoiIRkSMRsToyMhxh3GIMuye2quVK05//fOK5cu0e/feGluEQXDaqUu09JQlesf550qSLr7o3drx9M6aWzX8ooc/2Sq7/LZ/NNtPkpamt6ZQW7Zu1+rVZ2rVqpWamtqrD3zgCn3kDxjpP9It+aWTddqpp+i5H0/qzDNW6LFt23XWqrfV3ayhN8g39i+VdImk/33D95b0b31pUYGazaauWX+D7vvnO7RgZES3ff1b2rHj6bqbhQFw/bUf16c++3lNz0xr5enL9NfXX1t3k4ZeMwb3xv57JR0fEdvf+IPth/vRoFLdv+lB3b/pwbqbgQFz7tlnaeOtX667GUUZ2Mf3RcTVFb99KL85AHB4mHoKAEkG+RoqAAyVge3yA8CwocsPAEkGeZQfAIYKXX4ASMKgFAAk4RoqACShyw8ASaLGQSkeMA2gKE1F18tcbN9qe5/t8W6OTaACKEryO6Vuk7Su22PT5QdQlMwuf0Rstr2q2/UJVABFqXNQii4/gKL08sT+ztc1tZexwzk2FSqAovQy9TQiGpIaWccmUAEUhS4/ACTJHOW3faekRyWdY3vS9qwP3ZeoUAEUJnmU/6pe1idQARSFqacAkISHowBAkmbU9wA/AhVAUep8OAqBCqAoXEMFgCRcQwWAJC26/ACQgwoVAJIwyg8ASejyA0ASuvwAkIQKFQCSUKECQJJmNGs7NoEKoChMPQWAJEw9BYAkVKgAkIRRfgBIwig/ACRh6ikAJOEaKgAk4RoqACShQgWAJNyHCgBJqFABIAmj/ACQhEEpAEhSZ5d/pLYjA0AfRA9/5mJ7ne2nbO+0fd1c61OhAihKVoVqe4GkmyW9R9KkpC2274mIHbNtQ6ACKEriNdS1knZGxC5Jsv1NSVdIqi9QZ16dcr+PMSxsj0VEo+52YLBwXuTqJXNsj0ka6/iq0fHvYrmkFzp+m5T0rqr9cQ11fo3NvQqOQJwXNYmIRkSMdiyH9T82AhUADm1K0sqOzyva382KQAWAQ9si6e22z7R9tKQPSrqnagMGpeYX18lwKJwXAygiZmx/QtJ3JS2QdGtEPFG1jeu8CRYASkKXHwCSEKgAkIRAnSe9TmFD+Wzfanuf7fG624IcBOo86JjC9l5J50m6yvZ59bYKA+A2SevqbgTyEKjz4/UpbBHxqqTXprDhCBYRmyX9T93tQB4CdX4cagrb8praAqBPCFQASEKgzo+ep7ABGD4E6vzoeQobgOFDoM6DiJiR9NoUtglJG+eawoby2b5T0qOSzrE9afvqutuEw8PUUwBIQoUKAEkIVABIQqACQBICFQCSEKgAkIRABYAkBCoAJPk/g1NZeUJXgJQAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "sns.heatmap(confusion_matrix(y, model.predict(x)), annot=True)\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "hAo4kysIledB",
    "outputId": "86fcee8c-d72c-43e3-a655-4b1e2cc4c6d4"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       1.00      0.50      0.67         4\n",
      "           1       0.75      1.00      0.86         6\n",
      "\n",
      "    accuracy                           0.80        10\n",
      "   macro avg       0.88      0.75      0.76        10\n",
      "weighted avg       0.85      0.80      0.78        10\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(classification_report(y, model.predict(x)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Evaluation Metrics"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](img/acc.png)\n",
    "\n",
    "F-score = $F_1 = 2 * \\frac{precision\\,*\\,recall}{precision\\, +\\, recall}$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Find all evaluation metrics in sklearn library by clicking [here.](https://scikit-learn.org/stable/modules/model_evaluation.html)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.98      0.98      0.98        63\n",
      "           1       0.99      0.99      0.99       108\n",
      "\n",
      "    accuracy                           0.99       171\n",
      "   macro avg       0.99      0.99      0.99       171\n",
      "weighted avg       0.99      0.99      0.99       171\n",
      "\n"
     ]
    }
   ],
   "source": [
    "from sklearn.metrics import classification_report\n",
    "print(classification_report(y_test, cv[\"estimator\"][0].predict(X_test)))"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "collapsed_sections": [
    "MFiCsRxRledE"
   ],
   "include_colab_link": true,
   "name": "logisticRegression.ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}