Solution/Day_36_Solution.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 作業\n",
    "在鐵達尼資料集中，   \n",
    "可以用今天範例中提到的三種辨識異常值的方法，  \n",
    "以 training data 為資料集。  \n",
    "1.觀察票價是否有異常值的現象?  \n",
    "2.你覺得找出的異常是真的異常? 你覺得需要做處理嗎?  \n",
    "(第二題沒有對錯，可以試著講出自己的想法。)  \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Z9YHBcEGYdBV"
   },
   "source": [
    "### 資料\n",
    "鐵達尼號 train 資料集\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "collapsed": true,
    "executionInfo": {
     "elapsed": 1558,
     "status": "ok",
     "timestamp": 1578021044012,
     "user": {
      "displayName": "Mora chen",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AAuE7mB40f7sDArbZ5_DYq02nNcnLD0Ryaf7AhsASSQeLQ=s64",
      "userId": "03171203089166907199"
     },
     "user_tz": -480
    },
    "id": "yz28_IgkYdBW",
    "outputId": "a12f486c-18b3-4fb5-d06c-f162aebd9444"
   },
   "outputs": [],
   "source": [
    "# library\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import seaborn  as sns\n",
    "from scipy import stats\n",
    "import math\n",
    "import statistics\n",
    "#讓圖可以在 jupyter notebook顯示\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "df_train = pd.read_csv(\"Titanic_train.csv\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 探索性分析:\n",
    "*  觀察-是否有異常值  \n",
    "今天在範例中，我們來觀察一下 Fare 變數吧!!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "32.2042079685746\n",
      "49.693428597180905\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<seaborn.axisgrid.FacetGrid at 0x1e756a46eb8>"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAagAAADQCAYAAABStPXYAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjEsIGh0\ndHA6Ly9tYXRwbG90bGliLm9yZy+j8jraAAASbUlEQVR4nO3de9Bcd13H8ffHthQlHWxp2olpsBUj\n2joaaCxgBQuojQwaGKkGgaljnfJHcWC8TQszAuNU8ILKjJYxSoeilDZcOkTGkdZCB1BMm5b0ktZI\npDENiU0qcimDHZJ+/WNPptt0k+d58uzl9+y+XzM7e/a35/LNJt989pw9ezZVhSRJrfmuSRcgSdIg\nBpQkqUkGlCSpSQaUJKlJBpQkqUkGlCSpSQbUhCR5W5LtSe5Jsi3JC4a03l9McuWQ1vXoENZxcpIb\nk+xMsiXJ2YuvTNNuhvrjJUnuSnIwyWuGUdc0OXHSBcyiJC8CXgk8v6oeS3I68LQFLH9iVR0c9FxV\nbQY2D6fSobgM+N+q+sEkG4A/An5lwjWpYTPWH7uBXwN+Z8J1NMk9qMlYATxSVY8BVNUjVbUXIMmu\nriFJsjbJbd30O5JsTHIz8MFub+S8wytMcluS85P8WpK/TPLMbl3f1T3/PUkeSnJSkuck+ackdyb5\nXJIf7uY5J8kXktyR5A+G9GddD1zXTX8UeHmSDGndmk4z0x9Vtauq7gEeH8b6po0BNRk3A6uS/EeS\na5L89DyXOx9YX1W/CtwA/DJAkhXA91XVnYdnrKqvA3cDh9f9C8Cnquo7wEbgN6vqfHrv3K7p5nkv\n8L6q+gngv49WRNe02wbcfmbA7CuBh7qaDgJfB541zz+vZtMs9YeOwUN8E1BVjyY5H3gx8FLgxiRX\nVtUH5lh0c1V9u5veBNwCvJ1eI35kwPw30juc9hlgA3BNkmXATwIf6duRObm7vxD4pW767+gdjhtU\n/4vnqLPfoL0lr6+lo5qx/tAxGFATUlWHgNuA25LcC1wKfAA4yBN7tk8/YrFv9S3/lST/k+TH6DXZ\nGwdsZjPwriSn0Xt3+WngGcDXqmrN0Uqbq/YknwNOGfDU71TVPx8xtgdYBexJciLwTOCrc21Ds22G\n+kPH4CG+CUjy3CSr+4bWAP/VTe+i1yzwxLu1o7kB+D3gmVV175FPVtWjwO30Dk18sqoOVdU3gAeT\nXNLVkiQ/3i3yL/TeSQK87mgbraoXV9WaAbdBzbeZ3n8uAK8BPl1eoVjHMGP9oWMwoCZjGXBdkvuT\n3AOcC7yje+6dwHu7d2GH5ljPR+k1zKZjzHMj8Pru/rDXAZcluRvYTu9EBoA3A1ckuYPens4wvB94\nVpKdwG8BQznFV1NtZvojyU8k2QNcAvx1ku3DWO+0iG9mJUktcg9KktQkA0qS1CQDSpLUJANKktSk\nJgJq3bp1Re/7Bd68TevtuNkf3mbgNlATAfXII49MugSpWfaHZlUTASVJ0pEMKElSkwwoSVKTDChJ\nUpMMKElSkwwoSVKTllRAXb9lN9dv2T3pMiRJY7CkAkqSNDsMKElSkwwoSVKTDChJUpMMKElSkwwo\nSVKTDChJUpPmDKgkq5J8JskDSbYneXM3/o4kX0myrbu9om+Zq5LsTLIjycWj/ANIkqbTifOY5yDw\n21V1V5JTgDuT3NI99+dV9af9Myc5F9gAnAd8H/DPSX6oqg4Ns3BJ0nSbcw+qqvZV1V3d9DeBB4CV\nx1hkPXBDVT1WVQ8CO4ELhlGsJGl2LOgzqCRnA88DtnRDb0pyT5Jrk5zaja0EHupbbA8DAi3J5Um2\nJtl64MCBBRcuTTP7Q1pAQCVZBnwMeEtVfQN4H/AcYA2wD3jP4VkHLP6U35yvqo1Vtbaq1i5fvnzB\nhUvTzP6Q5hlQSU6iF04fqqqPA1TVw1V1qKoeB/6GJw7j7QFW9S1+FrB3eCVLkmbBfM7iC/B+4IGq\n+rO+8RV9s70auK+b3gxsSHJyknOA1cDtwytZkjQL5nMW34XAG4B7k2zrxt4KvDbJGnqH73YBbwSo\nqu1JNgH30zsD8ArP4JMkLdScAVVVn2fw50r/eIxlrgauXkRdkqQZ55UkJElNMqAkSU0yoCRJTTKg\nJElNMqAkSU0yoCRJTTKgJElNMqAkSU0yoCRJTTKgJElNMqAkSU0yoCRJTTKgJElNMqAkSU0yoCRJ\nTTKgJElNMqAkSU2aM6CSrErymSQPJNme5M3d+GlJbknype7+1L5lrkqyM8mOJBeP8g8gSZpO89mD\nOgj8dlX9CPBC4Iok5wJXArdW1Wrg1u4x3XMbgPOAdcA1SU4YRfGSpOk1Z0BV1b6ququb/ibwALAS\nWA9c1812HfCqbno9cENVPVZVDwI7gQuGXbgkabot6DOoJGcDzwO2AGdW1T7ohRhwRjfbSuChvsX2\ndGNHruvyJFuTbD1w4MDCK5emmP0hLSCgkiwDPga8paq+caxZB4zVUwaqNlbV2qpau3z58vmWIc0E\n+0OaZ0AlOYleOH2oqj7eDT+cZEX3/Apgfze+B1jVt/hZwN7hlCtJmhXzOYsvwPuBB6rqz/qe2gxc\n2k1fCnyib3xDkpOTnAOsBm4fXsmSpFlw4jzmuRB4A3Bvkm3d2FuBdwObklwG7AYuAaiq7Uk2AffT\nOwPwiqo6NPTKJUlTbc6AqqrPM/hzJYCXH2WZq4GrF1GXJGnGeSUJSVKTDChJUpMMKElSkwwoSVKT\nDChJUpMMKElSkwwoSVKTDChJUpMMKElSkwwoSVKTDChJUpMMKElSkwwoSVKTDChJUpMMKElSkwwo\nSVKTDChJUpPmDKgk1ybZn+S+vrF3JPlKkm3d7RV9z12VZGeSHUkuHlXhkqTpNp89qA8A6waM/3lV\nrelu/wiQ5FxgA3Bet8w1SU4YVrGSpNkxZ0BV1WeBr85zfeuBG6rqsap6ENgJXLCI+iRJM2oxn0G9\nKck93SHAU7uxlcBDffPs6caeIsnlSbYm2XrgwIFFlCFNH/tDOv6Aeh/wHGANsA94TzeeAfPWoBVU\n1caqWltVa5cvX36cZUjTyf6QjjOgqurhqjpUVY8Df8MTh/H2AKv6Zj0L2Lu4EiVJs+i4AirJir6H\nrwYOn+G3GdiQ5OQk5wCrgdsXV6IkaRadONcMST4MXAScnmQP8HbgoiRr6B2+2wW8EaCqtifZBNwP\nHASuqKpDoyldkjTN5gyoqnrtgOH3H2P+q4GrF1OUJEleSUKS1CQDSpLUJANKktQkA0qS1CQDSpLU\nJANKktQkA0qS1CQDSpLUJANKktQkA0qS1CQDSpLUJANKktQkA0qS1CQDSpLUJANKktQkA0qS1CQD\nSpLUpDkDKsm1SfYnua9v7LQktyT5Und/at9zVyXZmWRHkotHVbgkabrNZw/qA8C6I8auBG6tqtXA\nrd1jkpwLbADO65a5JskJQ6tWkjQz5gyoqvos8NUjhtcD13XT1wGv6hu/oaoeq6oHgZ3ABUOqVZI0\nQ473M6gzq2ofQHd/Rje+Eniob7493dhTJLk8ydYkWw8cOHCcZUjTyf6Qhn+SRAaM1aAZq2pjVa2t\nqrXLly8fchnS0mZ/SMcfUA8nWQHQ3e/vxvcAq/rmOwvYe/zlSZJm1fEG1Gbg0m76UuATfeMbkpyc\n5BxgNXD74kqUJM2iE+eaIcmHgYuA05PsAd4OvBvYlOQyYDdwCUBVbU+yCbgfOAhcUVWHRlS7JGmK\nzRlQVfXaozz18qPMfzVw9WKKkiRpzoCS1Lbrt+x+0uNffcGzJ1SJNFxe6kiS1CQDSpLUJANKktQk\nA0qS1CQDSpLUJANKktQkA0qS1CQDSpLUJANKktQkryQhTRmvLKFp4R6UJKlJSzKgrt+y+ynvEiVJ\n08VDfNKU85CflqoluQclSZp+BpQkqUkGlCSpSYv6DCrJLuCbwCHgYFWtTXIacCNwNrAL+OWq+t/F\nlSlJmjXD2IN6aVWtqaq13eMrgVurajVwa/dYkqQFGcUhvvXAdd30dcCrRrANSdKUW2xAFXBzkjuT\nXN6NnVlV+wC6+zMWuQ1J0gxa7PegLqyqvUnOAG5J8u/zXbALtMsBnv1sv5ch9bM/pEXuQVXV3u5+\nP3ATcAHwcJIVAN39/qMsu7Gq1lbV2uXLly+mDGnq2B/SIgIqyTOSnHJ4Gvg54D5gM3BpN9ulwCcW\nW6QkafYs5hDfmcBNSQ6v5/qq+qckdwCbklwG7AYuWXyZczt8ORcv4yJJ0+G4A6qqvgz8+IDx/wFe\nvpiiJEmaiitJeGVzSZo+UxFQkqTpY0BJkppkQEmSmmRASZKaZEBJkppkQEmSmrTYa/FNlKeXS4s3\nqI+O/ML7kfP4hXiNg3tQkqQmGVCSpCYZUJKkJhlQkqQmLZmA8oQISZotSyagJEmzZeYD6votu907\nk6QGLenvQQ3i9zUkaTpMXUBJOrZhHDHwjaDGYeoDqr+RbCJpfjzsrRaMLKCSrAPeC5wA/G1VvXtU\n25qvw013rKC6fstug0xaoIUGmj2m+RhJQCU5Afgr4GeBPcAdSTZX1f2j2N5CDQoh3zFKkzOMQ4Zz\nrcPDkuMxzNd5VHtQFwA7q+rLAEluANYDTQQUzC+QBu1xzefFP97lFqp/Owutaz57k9KozNV/4wgT\n9/ral6oa/kqT1wDrquo3usdvAF5QVW/qm+dy4PLu4XOBHXOs9nTgkaEXuzCTrmHS27eG46/hkapa\nN9+ZF9gfS/H1mMYaJr39pVzDwP4Y1R5UBow9KQmraiOwcd4rTLZW1drFFrYYk65h0tu3hvHVsJD+\nmIXXYynUMOntT2MNo/qi7h5gVd/js4C9I9qWJGkKjSqg7gBWJzknydOADcDmEW1LkjSFRnKIr6oO\nJnkT8Cl6p5lfW1XbF7naeR8OHKFJ1zDp7YM1HNZCDYe1UIs1TH77MGU1jOQkCUmSFmvmLxYrSWqT\nASVJalLzAZVkXZIdSXYmuXKE27k2yf4k9/WNnZbkliRf6u5P7Xvuqq6mHUkuHlINq5J8JskDSbYn\nefM460jy9CS3J7m72/47x7n9I2o5IckXk3xyEjUk2ZXk3iTbkmydRA3zqNHeGGMdrfTHTPVGVTV7\no3eCxX8CPwA8DbgbOHdE23oJ8Hzgvr6xPwau7KavBP6omz63q+Vk4JyuxhOGUMMK4Pnd9CnAf3Tb\nGksd9L6/tqybPgnYArxw3K9Dt+7fAq4HPjmhv4tdwOlHjI39dbA32uiNlvpjlnpjZA00jBvwIuBT\nfY+vAq4a4fbOPqIJdwAruukVwI5BddA7W/FFI6jnE/SuZzj2OoDvAe4CXjDu7dP73tytwMv6mnDc\nNQxqwon+eziiFntjQr3RrW8i/TFrvdH6Ib6VwEN9j/d0Y+NyZlXtA+juzxhXXUnOBp5H713a2Oro\nDh9sA/YDt1TVWLff+Qvg94DH+8bGXUMBNye5M73LDk2ihmOxN8bcG922J90fM9Ubrf8e1JyXTJqQ\nkdaVZBnwMeAtVfWNZNDmRlNHVR0C1iT5XuCmJD96rFKHvf0krwT2V9WdSS6azyLDrqFzYVXtTXIG\ncEuSf59ADcdib4y5N2Cy/TGLvdH6HtSkL5n0cJIVAN39/lHXleQkeg34oar6+KTqqKqvAbcB68a8\n/QuBX0yyC7gBeFmSvx9zDVTV3u5+P3ATvSv0j/3v4RjsjQnVARPrj9nrjWEdkx3Fjd4e3pfpfbh2\n+IPg80a4vbN58nH2P+HJH/z9cTd9Hk/+4O/LDOfDxwAfBP7iiPGx1AEsB763m/5u4HPAK8f9OvTV\ncxFPHGcfWw3AM4BT+qb/ld5/RBN5HeyNyfdGa/0xK70xsgYa1g14Bb0zdv4TeNsIt/NhYB/wHXqp\nfxnwLHofSH6puz+tb/63dTXtAH5+SDX8FL3d33uAbd3tFeOqA/gx4Ivd9u8Dfr8bH+vr0Lfu/iYc\nWw30zoy7u7ttP/zvblKvg70x+d5orT9mpTe81JEkqUmtfwYlSZpRBpQkqUkGlCSpSQaUJKlJBpQk\nqUmtX0lCC5TkEHBv39CrqmrXhMqRmmJ/LC2eZj5lkjxaVcsWuEzo/Vt4fM6ZpSXM/lhaPMQ35ZIs\nS3Jrkru633BZ342f3f22zjX0rsq8KsnvJrkjyT2Hf+tGmmb2R9sMqOnz3d0PiW1LchPwf8Crq+r5\nwEuB9+SJK2w+F/hgVT2vm15N77paa4Dzk7xkAvVLo2R/LCF+BjV9vl1Vaw4/6C6w+YddMz1O71L3\nZ3ZP/1dV/Vs3/XPd7Yvd42X0GvKzY6laGg/7YwkxoKbf6+hd5PL8qvpOdyXkp3fPfatvvgDvqqq/\nHnN90iTZHw3zEN/0eya935D5TpKXAt9/lPk+Bfx693s7JFnZ/d6LNM3sj4a5BzX9PgT8Q5Kt9K4A\nPfDHxarq5iQ/AnyhOwT/KPB6nvhdF2ka2R8N8zRzSVKTPMQnSWqSASVJapIBJUlqkgElSWqSASVJ\napIBJUlqkgElSWrS/wM7mdfFJjJpzAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x216 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "print(df_train['Fare'].mean()) #Fare平均\n",
    "print(df_train['Fare'].std()) #Fare標準差\n",
    "# 連續型用分布圖行來看\n",
    "g = sns.FacetGrid(df_train, col='Survived')\n",
    "g.map(sns.distplot, 'Fare', kde=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 方法1\n",
    "* 進行一個描述性的統計（規定範圍），從而檢視哪些值是不合理的（範圍以外的值）。  \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "count    891.000000\n",
      "mean      32.204208\n",
      "std       49.693429\n",
      "min        0.000000\n",
      "25%        7.910400\n",
      "50%       14.454200\n",
      "75%       31.000000\n",
      "max      512.329200\n",
      "Name: Fare, dtype: float64\n"
     ]
    }
   ],
   "source": [
    "#先透過 計算統計值， 分別呈現數量，票價平均，標準差，最大值和最小值。\n",
    "print(df_train['Fare'].describe()) \n",
    "#票價的資料分布不像年齡最大值為 80，最大票價 512 ，所以資料變動範圍較大。\n",
    "#方法1中，也可以去查過往資料，鐵達票價的範圍，檢查上述的票價是否在販售範圍。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 方法 2\n",
    "* 進行3倍標準差原則的計算，從而檢視哪些值是可疑的異常值。  \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Z-score method"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[ 27  88 118 258 299 311 341 377 380 438 527 557 679 689 700 716 730 737\n",
      " 742 779]\n",
      "用第二種方法的找出的 outlier 有哪些?\n",
      "27     263.0000\n",
      "88     263.0000\n",
      "118    247.5208\n",
      "258    512.3292\n",
      "299    247.5208\n",
      "311    262.3750\n",
      "341    263.0000\n",
      "377    211.5000\n",
      "380    227.5250\n",
      "438    263.0000\n",
      "527    221.7792\n",
      "557    227.5250\n",
      "679    512.3292\n",
      "689    211.3375\n",
      "700    227.5250\n",
      "716    227.5250\n",
      "730    211.3375\n",
      "737    512.3292\n",
      "742    262.3750\n",
      "779    211.3375\n",
      "Name: Fare, dtype: float64\n"
     ]
    }
   ],
   "source": [
    "#http://colingorrie.github.io/outlier-detection.html\n",
    "import numpy as np\n",
    "\n",
    "# 創建一個函數，計算在這個資料中， ys:資料，times : 幾倍標準差，找出在這樣條件下的異常值。\n",
    "def outliers_z_score(ys,times):\n",
    "    mean_y = np.mean(ys)\n",
    "    stdev_y = np.std(ys)\n",
    "    z_scores = [(y - mean_y) / stdev_y for y in ys]\n",
    "    return np.where(np.abs(z_scores) > times)\n",
    "\n",
    "out_index=outliers_z_score(df_train['Fare'],3)\n",
    "print(out_index[0])\n",
    "print(\"用第二種方法的找出的 outlier 有哪些?\")\n",
    "print(df_train.loc[out_index[0],'Fare'])\n",
    "#  所以用這個方法，找出的疑似 outlier 就是高票價的族群。 "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 異常值的判別方法3-盒鬚圖判別法(IQR method)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "# 透過一個函數來判斷是否為異常值\n",
    "def outliers_iqr(ys,times):\n",
    "    #注意 使用 np.percentile 要去除 nan 要不然計算出來會錯誤，所以我們採用下者的程式 np.nanpercentile\n",
    "    #quartile_1, quartile_3 = np.percentile(ys, [25, 75])\n",
    "    quartile_1, quartile_3 = np.nanpercentile(ys, [25, 75])\n",
    "    iqr = quartile_3 - quartile_1\n",
    "    lower_bound = quartile_1 - (iqr * times)\n",
    "    upper_bound = quartile_3 + (iqr * times)\n",
    "    return np.where((ys > upper_bound) | (ys < lower_bound))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(array([  1,  27,  31,  34,  52,  61,  62,  72,  88, 102, 118, 120, 124,\n",
      "       139, 151, 159, 180, 195, 201, 215, 218, 224, 230, 245, 256, 257,\n",
      "       258, 262, 268, 269, 275, 290, 291, 297, 299, 305, 306, 307, 310,\n",
      "       311, 318, 319, 324, 325, 332, 334, 336, 337, 341, 366, 369, 373,\n",
      "       375, 377, 380, 385, 390, 393, 412, 435, 438, 445, 453, 484, 486,\n",
      "       496, 498, 504, 505, 520, 527, 537, 540, 544, 550, 557, 558, 581,\n",
      "       585, 587, 591, 609, 627, 641, 645, 655, 659, 660, 665, 679, 681,\n",
      "       689, 698, 700, 708, 716, 730, 737, 741, 742, 745, 759, 763, 765,\n",
      "       779, 789, 792, 802, 820, 829, 835, 846, 849, 856, 863, 879],\n",
      "      dtype=int64),)\n",
      "用第三種方法的找出的 outlier 有哪些?(1.5 倍IQR)\n",
      "1       71.2833\n",
      "27     263.0000\n",
      "31     146.5208\n",
      "34      82.1708\n",
      "52      76.7292\n",
      "         ...   \n",
      "846     69.5500\n",
      "849     89.1042\n",
      "856    164.8667\n",
      "863     69.5500\n",
      "879     83.1583\n",
      "Name: Fare, Length: 116, dtype: float64\n"
     ]
    }
   ],
   "source": [
    "out_index2=outliers_iqr(df_train['Fare'],1.5)\n",
    "print(out_index2)\n",
    "print(\"用第三種方法的找出的 outlier 有哪些?(1.5 倍IQR)\")\n",
    "print(df_train.loc[out_index2[0],'Fare'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(array([ 27,  31,  34,  61,  62,  88, 102, 118, 124, 139, 195, 215, 224,\n",
      "       230, 245, 256, 257, 258, 262, 268, 269, 275, 290, 291, 297, 299,\n",
      "       305, 306, 307, 310, 311, 318, 319, 325, 332, 334, 337, 341, 373,\n",
      "       375, 377, 380, 390, 393, 412, 435, 438, 445, 453, 484, 486, 496,\n",
      "       498, 504, 505, 520, 527, 537, 544, 550, 557, 558, 581, 585, 587,\n",
      "       591, 609, 627, 659, 660, 679, 689, 698, 700, 708, 716, 730, 737,\n",
      "       741, 742, 759, 763, 765, 779, 789, 802, 820, 829, 835, 849, 856,\n",
      "       879], dtype=int64),)\n",
      "用第三種方法的找出的 outlier 有哪些?(2 倍IQR)\n",
      "27     263.0000\n",
      "31     146.5208\n",
      "34      82.1708\n",
      "61      80.0000\n",
      "62      83.4750\n",
      "         ...   \n",
      "829     80.0000\n",
      "835     83.1583\n",
      "849     89.1042\n",
      "856    164.8667\n",
      "879     83.1583\n",
      "Name: Fare, Length: 92, dtype: float64\n"
     ]
    }
   ],
   "source": [
    "out_index3=outliers_iqr(df_train['Fare'],2)\n",
    "print(out_index3)\n",
    "print(\"用第三種方法的找出的 outlier 有哪些?(2 倍IQR)\")\n",
    "print(df_train.loc[out_index3[0],'Fare'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 畫盒鬚圖\n",
    "方法：利用matplotlib包中axes物件的boxplot()方法。，\n",
    "透過 whis 來設定 IQR 的倍數"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAEICAYAAACktLTqAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjEsIGh0\ndHA6Ly9tYXRwbG90bGliLm9yZy+j8jraAAAUIElEQVR4nO3df4xd9Znf8fdnwDuUcRbs2FCCSew2\nUAmCE6sjWpFqG5K4RmrXRqrYpdpu3F1kZCB0K5BqjFqFlQqlC01/OAtT2KUL2i7U7WbLdIsaG1oa\nrUghw242DrAEEwz2GuHxT+Kh0AGe/jHX7hjf8Vx7Zrgzh/dLGp17vvfce54reT4+95nv/d5UFZKk\nZunpdgGSpOlnuEtSAxnuktRAhrskNZDhLkkNZLhLUgMZ7tI0SPI7Sf5Zt+uQjjDcNecl2ZHk/yQ5\nnORAkv+W5IIZOM9TSd5pnWdvkm8nOe8UnqeSfHa665PGM9zVFD9fVfOB84A3gU0zdJ6vt85zEXA2\n8K9m6DzSlBjuapSqegf4z8DFR8aSnJXk4STDSV5L8k+S9CRZmGRXkp9vHTc/yfYkX+vgPPuB3wc+\n1+7+JOtaz7U/yWCST7XGv9s65E9b7wB+cYovWWrLcFejJDkT+EXgf48b3gScBfwl4G8CXwN+pRXQ\nvwo8kOQcxq7Cf1BVD3dwnkXA3wX+pM19Xwb+OfALjL2TeA14FKCqfq512Oeran5V/cdTeZ3SZE7v\ndgHSNPkvSd4D5gN7gFUASU5jLOxXVNVPgZ8m+ZfALwO/XVVbkvwn4Engk8Clk5zn3ya5BxgBngJu\nbnPMLwEPVtUft2rYCBxIsrSqdkztZUqd8cpdTXFVVZ0N9AJfB/5Xkr8ILAJ+hrGr5yNeA84ft38/\nY+2Vf19V+yY5zz+sqrOr6vyq+qWqGm5zzKfGn6+qDgP7PnROaUYZ7mqUqnq/qr4NvA/8DWAvMAp8\nZtxhnwb+HI5e2f874GHg+mmaxbJ7/PmS9DH2ruDPp+G5pY4Y7mqUjFkDLABerKr3gc3AHUk+keQz\njLVSfrf1kNta218F7gEebgX+VPwe8CtJvpCkF7gTeGZcS+ZNxvr/0owx3NUU/zXJYeAt4A5gbVU9\n37rvJsZ65D8B/oix8H0wyV9lLOi/1vpP4F8ABdw6lUKq6kngnzI2m+YN4C8D14w75HbgoSQHk/zC\nVM4lTSR+WYckNY9X7pLUQIa7JDWQ4S5JDWS4S1IDzYpPqC5atKiWLl3a7TIkaU557rnn9lbV4nb3\nzYpwX7p0KUNDQ90uQ5LmlCSvTXSfbRlJaiDDXZIayHCXpAYy3CWpgQx3SWogw12awKpVq+jp6SEJ\nPT09rFq1qtslSR0z3KU2Vq1axZYtW1i/fj0HDx5k/fr1bNmyxYDXnDEr5rlLs83WrVu5/vrruffe\newGObgcGBrpZltSxWbHkb39/f/khJs0mSTh48CBnnXXW0bFDhw5x9tlnMxt+ZySAJM9VVX+7+2zL\nSG0kYePGjceMbdy4kSRdqkg6OR2Fe5IdSbYl+UGSodbYwiRbk7zc2i4Yd/zGJNuTvJTEJqXmnJUr\nV3Lfffdxww03cOjQIW644Qbuu+8+Vq5c2e3SpI501JZJsgPor6q948Z+A9hfVXcluRVYUFUbklwM\nPAJcxti3wD8BXNT6GrO2bMtoNlq1ahVbt26lqkjCypUr+c53vtPtsqSjTtSWmcofVNcAX2rdfgh4\nCtjQGn+0qt4FXk2ynbGg/94UziV95AxyzWWd9twL2JLkuSTXtcbOrao3AFrbc1rj5wM7xz12V2vs\nGEmuSzKUZGh4ePjUqpcktdXplfsXq2p3knOArUn+7ATHtvuL03G9n6q6H7gfxtoyHdYhSepAR1fu\nVbW7td0D/AFjbZY3k5wH0NruaR2+C7hg3MOXALunq2BJ0uQmDfckfUk+ceQ28LeAHwGDwNrWYWuB\nx1q3B4FrkvQmWQZcCDw73YVLkibWSVvmXOAPWvN7Twd+r6r+e5LvA5uTXAu8DlwNUFXPJ9kMvAC8\nB9x4opkykqTpN2m4V9VPgM+3Gd8HfGWCx9wB3DHl6iRJp8RPqEpSAxnuktRAhrskNZDhLkkNZLhL\nUgMZ7pLUQIa7JDWQ4S5JDWS4S1IDGe6S1ECGuyQ1kOEuSQ1kuEtSAxnuktRAhrskNZDhLkkNZLhL\nUgMZ7pLUQIa7JDWQ4S5JDWS4S1IDGe6S1ECGuyQ1kOEuSQ1kuEtSAxnuktRAhrskNZDhLkkNZLhL\nUgN1HO5JTkvyJ0n+sLW/MMnWJC+3tgvGHbsxyfYkLyVZNROFS5ImdjJX7r8GvDhu/1bgyaq6EHiy\ntU+Si4FrgEuAK4F7k5w2PeVKkjrRUbgnWQL8beC3xg2vAR5q3X4IuGrc+KNV9W5VvQpsBy6bnnIl\nSZ3o9Mr9XwP/GPhg3Ni5VfUGQGt7Tmv8fGDnuON2tcYkSR+RScM9yd8B9lTVcx0+Z9qMVZvnvS7J\nUJKh4eHhDp9aktSJTq7cvwisTrIDeBT4cpLfBd5Mch5Aa7undfwu4IJxj18C7P7wk1bV/VXVX1X9\nixcvnsJLkCR92KThXlUbq2pJVS1l7A+l/6Oq/j4wCKxtHbYWeKx1exC4JklvkmXAhcCz0165JGlC\np0/hsXcBm5NcC7wOXA1QVc8n2Qy8ALwH3FhV70+5UklSx1J1XDv8I9ff319DQ0PdLkOS5pQkz1VV\nf7v7/ISqJDWQ4S5JDWS4S1IDGe6S1ECGuyQ1kOEuSQ1kuEtSAxnuktRAhrskNZDhLkkNZLhLUgMZ\n7pLUQIa7JDWQ4S5JDWS4S1IDGe6S1ECGuyQ1kOEuSQ1kuEtSAxnuktRAhrskNZDhLkkNZLhLUgMZ\n7pLUQIa7JDWQ4S5JDWS4S1IDGe6S1ECGuyQ1kOEuSQ00abgnOSPJs0n+NMnzSX69Nb4wydYkL7e2\nC8Y9ZmOS7UleSrJqJl+AJOl4nVy5vwt8uao+D3wBuDLJXwduBZ6sqguBJ1v7JLkYuAa4BLgSuDfJ\naTNRvCSpvUnDvcYcbu3Oa/0UsAZ4qDX+EHBV6/Ya4NGqereqXgW2A5dNa9WSpBPqqOee5LQkPwD2\nAFur6hng3Kp6A6C1Pad1+PnAznEP39Ua+/BzXpdkKMnQ8PDwVF6DJOlDOgr3qnq/qr4ALAEuS/K5\nExyedk/R5jnvr6r+qupfvHhxZ9VKkjpyUrNlquog8BRjvfQ3k5wH0NruaR22C7hg3MOWALunXKkk\nqWOdzJZZnOTs1u2/AHwV+DNgEFjbOmwt8Fjr9iBwTZLeJMuAC4Fnp7twSdLETu/gmPOAh1ozXnqA\nzVX1h0m+B2xOci3wOnA1QFU9n2Qz8ALwHnBjVb0/M+VLktpJ1XHt8I9cf39/DQ0NdbsMSZpTkjxX\nVf3t7vMTqpLUQIa7JDWQ4S5JDWS4S1IDGe6S1ECGuzSB5cuXk+Toz/Lly7tdktQxw11qY/ny5Wzb\nto2enrFfkZ6eHrZt22bAa84w3KU2tm3bRhLuvvtuRkZGuPvuu0nCtm3bul2a1BHDXZrA7bffzs03\n38yZZ57JzTffzO23397tkqSOGe7SBJ555pkT7kuzmeEutZGExx9/nDVr1rB3717WrFnD448/TtJu\nRWtp9jHcpTZuvPFGAAYHB1m8eDGDg4PHjEuzneEuTSAJp58+tnDq6aef7lW75hTDXWrjgQce4J57\n7mF0dJSqYnR0lHvuuYcHHnig26VJHXHJX6mNJIyMjHDmmWceHXv77bfp6+tjNvzOSOCSv9JJ6+3t\nZWBg4JixgYEBent7u1SRdHI6+SYm6WNn3bp1bNiwAYD169czMDDAhg0bWL9+fZcrkzpjuEttbNq0\nCYDbbruNW265hd7eXtavX390XJrt7LlL0hxlz106Ba4KqbnMcJfaOLIq5OrVqxkeHmb16tWuCqk5\nxXCX2ti2bRsrVqzglVde4dxzz+WVV15hxYoVrgqpOcNwlyawb98+Nm3axDvvvMOmTZvYt29ft0uS\nOma4SxNYsmQJV1xxBfPmzeOKK65gyZIl3S5J6pjhLk3g6aefPmZVyKeffrrbJUkdc5671MYll1zC\nj3/846OrQgLMmzePiy66qMuVSZ3xyl1q46233mJ0dJTLL7+c3bt3c/nllzM6Ospbb73V7dKkjhju\nUhs7d+5kxYoVHDp0iCVLlnDo0CFWrFjBzp07u12a1BHbMtIEtmzZwqJFi47u792792iLRprtvHKX\nJnDttdeecF+azSYN9yQXJPmfSV5M8nySX2uNL0yyNcnLre2CcY/ZmGR7kpeSrJrJFyDNhEsvvZTB\nwcFjZssMDg5y6aWXdrs0qSOTLhyW5DzgvKr64ySfAJ4DrgL+AbC/qu5KciuwoKo2JLkYeAS4DPgU\n8ARwUVW9P9E5XDhMs9GRJQiOuPTSS/nhD3/YxYqkY51o4bBJe+5V9QbwRuv2T5O8CJwPrAG+1Drs\nIeApYENr/NGqehd4Ncl2xoL+e1N7GdJHyyDXXHZSPfckS4EVwDPAua3gP/IfwDmtw84Hxk8p2NUa\n+/BzXZdkKMnQ8PDwyVcuSZpQx+GeZD7w+8A/qqoTTfZt9xXxx/V+qur+quqvqn5nIEjS9Ooo3JPM\nYyzY/0NVfbs1/GarH3+kL7+nNb4LuGDcw5cAu6enXElSJzqZLRPgt4EXq+qb4+4aBNa2bq8FHhs3\nfk2S3iTLgAuBZ6evZEnSZDr5ENMXgV8GtiX5QWvsNuAuYHOSa4HXgasBqur5JJuBF4D3gBtPNFNG\nkjT9Opkt80e076MDfGWCx9wB3DGFuiRJU+AnVCWpgQx3SWogw12SGshwl6QGMtylCdx0002cccYZ\nJOGMM87gpptu6nZJUscMd6mNm266iYGBAe68805GRka48847GRgYMOA1Z0y6KuRHwVUhNducccYZ\n3Hnnndx8881Hx775zW9y22238c4773SxMun/O9GqkIa71EYSRkZGOPPMM4+Ovf322/T19TEbfmck\nOHG425aR2ujt7WVgYOCYsYGBAXp7e7tUkXRyDHepjXXr1nHLLbeQ5OjPLbfcwrp167pdmtQR2zJS\nG/Pnz2dkZOS48b6+Pg4fPtyFiqTjTembmKSPo5GREebPn8/o6Cjvvvsuvb29zJs3z2DXnGFbRprA\n4cOHj5kKabBrLrEtI7WRhAULFrB///6jYwsXLuTAgQPOltGs4WwZ6RQcOHCAZcuW8corr7Bs2TIO\nHDjQ7ZKkjtlzl9ro7e3lgw8+YMeOHXz2s58FYN68efT0eD2kucF/qVIb69atY3R09Jix0dFRp0Jq\nzrDnLrXhVEjNBfbcpZM0MjLC0qVLqaqjP0uXLm0b+NJsZLhLE3jiiSdOuC/NZoa7NIGvfvWrJ9yX\nZjPDXWqjr6+PHTt2HDMVcseOHfT19XW7NKkjToWU2jh8+DA9PT3HTIVM4h9TNWd45S61sXz5cqqK\n1atXMzw8zOrVq6kqli9f3u3SpI44FVJqIwmLFi1i3759VBVJ+OQnP8nevXtdfkCzhqtCSqdg7969\nR29X1TH70mxnW0Y6gSTHbKW5wnCXTuBIC8ZWjOYaw12SGmjScE/yYJI9SX40bmxhkq1JXm5tF4y7\nb2OS7UleSrJqpgqXPgrXX389Bw8e5Prrr+92KdJJmXS2TJKfAw4DD1fV51pjvwHsr6q7ktwKLKiq\nDUkuBh4BLgM+BTwBXFRV75/oHM6W0Wxzoh67LRrNFlNaOKyqvgvs/9DwGuCh1u2HgKvGjT9aVe9W\n1avAdsaCXppTTjvttJMal2abU+25n1tVbwC0tue0xs8Hdo47bldr7DhJrksylGRoeHj4FMuQZsZE\nbRjbM5orpvsPqu3ey7Z9D1tV91dVf1X1L168eJrLkKSPt1MN9zeTnAfQ2u5pje8CLhh33BJg96mX\nJ3XHt771LeD4ee5HxqXZ7lTDfRBY27q9Fnhs3Pg1SXqTLAMuBJ6dWolS9zjPXXPVpMsPJHkE+BKw\nKMku4BvAXcDmJNcCrwNXA1TV80k2Ay8A7wE3TjZTRpI0/SYN96r6exPc9ZUJjr8DuGMqRUmSpsZP\nqEpSAxnuktRAhrt0Aj09PcdspbnC9dz1sXKyS/d+8MEHx2w7fQ5n16jbDHd9rHQauq4to7nO95pS\nG319fSc1Ls02hrvUxuHDh48L8r6+Pg4fPtyliqSTY1tGmsCRIE9iK0ZzjlfuktRAhrskNZDhLkkN\nZLhLUgMZ7pLUQIa7JDWQ4S5JDWS4S1IDGe6S1ECGuyQ1kOEuSQ1kuEtSAxnuktRArgqpOW3hwoUc\nOHBgxs9zst/gdLIWLFjA/v37Z/Qc+ngx3DWnHThwoBHL8c70fx76+LEtI0kNZLhLUgMZ7pLUQIa7\nJDWQf1DVnFbf+Fm4/axulzFl9Y2f7XYJahjDXXNafv2txsyWqdu7XYWaxLaMJDXQjF25J7kS+DfA\nacBvVdVdM3Uufbw1YY74ggULul2CGmZGwj3JacBvAiuBXcD3kwxW1QszcT59fH0ULZkkjWj96ONl\nptoylwHbq+onVfV/gUeBNTN0LknSh8xUW+Z8YOe4/V3AXxt/QJLrgOsAPv3pT89QGdKxTrWFc7KP\n80pf3TZTV+7tfhOO+ddeVfdXVX9V9S9evHiGypCOVVUfyY/UbTMV7ruAC8btLwF2z9C5JEkfMlPh\n/n3gwiTLkvwMcA0wOEPnkiR9yIz03KvqvSRfB77D2FTIB6vq+Zk4lyTpeDM2z72qHgcen6nnlyRN\nzE+oSlIDGe6S1ECGuyQ1kOEuSQ2U2fCBiyTDwGvdrkOawCJgb7eLkNr4TFW1/RTorAh3aTZLMlRV\n/d2uQzoZtmUkqYEMd0lqIMNdmtz93S5AOln23CWpgbxyl6QGMtwlqYEMd2kCSR5MsifJj7pdi3Sy\nDHdpYr8DXNntIqRTYbhLE6iq7wL7u12HdCoMd0lqIMNdkhrIcJekBjLcJamBDHdpAkkeAb4H/JUk\nu5Jc2+2apE65/IAkNZBX7pLUQIa7JDWQ4S5JDWS4S1IDGe6S1ECGuyQ1kOEuSQ30/wBPREF44kIu\n8AAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# 使用np.isnana(data)，找出在 Fare中的遺失值，然後逐位反轉，讓遺失值為 0,則可以透過索引的方式，濾掉遺失值。\n",
    "plt.boxplot(df_train['Fare'][~np.isnan(df_train['Fare'])],whis=2)    #作圖\n",
    "plt.title('Box Plot')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Q2:你覺得找出的異常是真的異常? 你覺得需要做處理嗎?\n",
    "* 需有業務面的資訊可以輔助你回答這問題，   \n",
    "在鐵達尼案例中，票價如果是販售範圍內。\n",
    "如果沒有其他輔佐資料判斷是否為 outlier，比如 Pcalss 在不同艙等有不同的票價，則可以輔助判斷。"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "name": "Day33.YOLO 細節理解 - 網路輸出的後處理_HW.ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}