DescriptiveStatisticsReview.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's begin with statistics mainly for univariate data analysis, covering some basic concepts like descriptive and inferential statistics and distributions. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Loading data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "auto_mpg <- read.csv(\"datasets/auto-mpg.csv\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<thead><tr><th scope=col>mpg</th><th scope=col>cylinders</th><th scope=col>displacement</th><th scope=col>horsepower</th><th scope=col>weight</th><th scope=col>acceleration</th><th scope=col>model.year</th><th scope=col>origin</th><th scope=col>car.name</th></tr></thead>\n",
       "<tbody>\n",
       "\t<tr><td>18                       </td><td>8                        </td><td>307                      </td><td>130                      </td><td>3504                     </td><td>12.0                     </td><td>70                       </td><td>1                        </td><td>chevrolet chevelle malibu</td></tr>\n",
       "\t<tr><td>15                       </td><td>8                        </td><td>350                      </td><td>165                      </td><td>3693                     </td><td>11.5                     </td><td>70                       </td><td>1                        </td><td>buick skylark 320        </td></tr>\n",
       "\t<tr><td>18                       </td><td>8                        </td><td>318                      </td><td>150                      </td><td>3436                     </td><td>11.0                     </td><td>70                       </td><td>1                        </td><td>plymouth satellite       </td></tr>\n",
       "\t<tr><td>16                       </td><td>8                        </td><td>304                      </td><td>150                      </td><td>3433                     </td><td>12.0                     </td><td>70                       </td><td>1                        </td><td>amc rebel sst            </td></tr>\n",
       "\t<tr><td>17                       </td><td>8                        </td><td>302                      </td><td>140                      </td><td>3449                     </td><td>10.5                     </td><td>70                       </td><td>1                        </td><td>ford torino              </td></tr>\n",
       "\t<tr><td>15                       </td><td>8                        </td><td>429                      </td><td>198                      </td><td>4341                     </td><td>10.0                     </td><td>70                       </td><td>1                        </td><td>ford galaxie 500         </td></tr>\n",
       "</tbody>\n",
       "</table>\n"
      ],
      "text/latex": [
       "\\begin{tabular}{r|lllllllll}\n",
       " mpg & cylinders & displacement & horsepower & weight & acceleration & model.year & origin & car.name\\\\\n",
       "\\hline\n",
       "\t 18                        & 8                         & 307                       & 130                       & 3504                      & 12.0                      & 70                        & 1                         & chevrolet chevelle malibu\\\\\n",
       "\t 15                        & 8                         & 350                       & 165                       & 3693                      & 11.5                      & 70                        & 1                         & buick skylark 320        \\\\\n",
       "\t 18                        & 8                         & 318                       & 150                       & 3436                      & 11.0                      & 70                        & 1                         & plymouth satellite       \\\\\n",
       "\t 16                        & 8                         & 304                       & 150                       & 3433                      & 12.0                      & 70                        & 1                         & amc rebel sst            \\\\\n",
       "\t 17                        & 8                         & 302                       & 140                       & 3449                      & 10.5                      & 70                        & 1                         & ford torino              \\\\\n",
       "\t 15                        & 8                         & 429                       & 198                       & 4341                      & 10.0                      & 70                        & 1                         & ford galaxie 500         \\\\\n",
       "\\end{tabular}\n"
      ],
      "text/markdown": [
       "\n",
       "mpg | cylinders | displacement | horsepower | weight | acceleration | model.year | origin | car.name | \n",
       "|---|---|---|---|---|---|\n",
       "| 18                        | 8                         | 307                       | 130                       | 3504                      | 12.0                      | 70                        | 1                         | chevrolet chevelle malibu | \n",
       "| 15                        | 8                         | 350                       | 165                       | 3693                      | 11.5                      | 70                        | 1                         | buick skylark 320         | \n",
       "| 18                        | 8                         | 318                       | 150                       | 3436                      | 11.0                      | 70                        | 1                         | plymouth satellite        | \n",
       "| 16                        | 8                         | 304                       | 150                       | 3433                      | 12.0                      | 70                        | 1                         | amc rebel sst             | \n",
       "| 17                        | 8                         | 302                       | 140                       | 3449                      | 10.5                      | 70                        | 1                         | ford torino               | \n",
       "| 15                        | 8                         | 429                       | 198                       | 4341                      | 10.0                      | 70                        | 1                         | ford galaxie 500          | \n",
       "\n",
       "\n"
      ],
      "text/plain": [
       "  mpg cylinders displacement horsepower weight acceleration model.year origin\n",
       "1 18  8         307          130        3504   12.0         70         1     \n",
       "2 15  8         350          165        3693   11.5         70         1     \n",
       "3 18  8         318          150        3436   11.0         70         1     \n",
       "4 16  8         304          150        3433   12.0         70         1     \n",
       "5 17  8         302          140        3449   10.5         70         1     \n",
       "6 15  8         429          198        4341   10.0         70         1     \n",
       "  car.name                 \n",
       "1 chevrolet chevelle malibu\n",
       "2 buick skylark 320        \n",
       "3 plymouth satellite       \n",
       "4 amc rebel sst            \n",
       "5 ford torino              \n",
       "6 ford galaxie 500         "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "head(auto_mpg)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<ol class=list-inline>\n",
       "\t<li>'mpg'</li>\n",
       "\t<li>'cylinders'</li>\n",
       "\t<li>'displacement'</li>\n",
       "\t<li>'horsepower'</li>\n",
       "\t<li>'weight'</li>\n",
       "\t<li>'acceleration'</li>\n",
       "\t<li>'model.year'</li>\n",
       "\t<li>'origin'</li>\n",
       "\t<li>'car.name'</li>\n",
       "</ol>\n"
      ],
      "text/latex": [
       "\\begin{enumerate*}\n",
       "\\item 'mpg'\n",
       "\\item 'cylinders'\n",
       "\\item 'displacement'\n",
       "\\item 'horsepower'\n",
       "\\item 'weight'\n",
       "\\item 'acceleration'\n",
       "\\item 'model.year'\n",
       "\\item 'origin'\n",
       "\\item 'car.name'\n",
       "\\end{enumerate*}\n"
      ],
      "text/markdown": [
       "1. 'mpg'\n",
       "2. 'cylinders'\n",
       "3. 'displacement'\n",
       "4. 'horsepower'\n",
       "5. 'weight'\n",
       "6. 'acceleration'\n",
       "7. 'model.year'\n",
       "8. 'origin'\n",
       "9. 'car.name'\n",
       "\n",
       "\n"
      ],
      "text/plain": [
       "[1] \"mpg\"          \"cylinders\"    \"displacement\" \"horsepower\"   \"weight\"      \n",
       "[6] \"acceleration\" \"model.year\"   \"origin\"       \"car.name\"    "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "names(auto_mpg)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "column_names <- names(auto_mpg)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<ol class=list-inline>\n",
       "\t<li>'mpg'</li>\n",
       "\t<li>'cylinders'</li>\n",
       "\t<li>'displacement'</li>\n",
       "\t<li>'horsepower'</li>\n",
       "\t<li>'weight'</li>\n",
       "\t<li>'acceleration'</li>\n",
       "\t<li>'model.year'</li>\n",
       "\t<li>'origin'</li>\n",
       "\t<li>'car.name'</li>\n",
       "</ol>\n"
      ],
      "text/latex": [
       "\\begin{enumerate*}\n",
       "\\item 'mpg'\n",
       "\\item 'cylinders'\n",
       "\\item 'displacement'\n",
       "\\item 'horsepower'\n",
       "\\item 'weight'\n",
       "\\item 'acceleration'\n",
       "\\item 'model.year'\n",
       "\\item 'origin'\n",
       "\\item 'car.name'\n",
       "\\end{enumerate*}\n"
      ],
      "text/markdown": [
       "1. 'mpg'\n",
       "2. 'cylinders'\n",
       "3. 'displacement'\n",
       "4. 'horsepower'\n",
       "5. 'weight'\n",
       "6. 'acceleration'\n",
       "7. 'model.year'\n",
       "8. 'origin'\n",
       "9. 'car.name'\n",
       "\n",
       "\n"
      ],
      "text/plain": [
       "[1] \"mpg\"          \"cylinders\"    \"displacement\" \"horsepower\"   \"weight\"      \n",
       "[6] \"acceleration\" \"model.year\"   \"origin\"       \"car.name\"    "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "column_names"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# modify the names of the columns of auto_mpg dataset by assigning new names\n",
    "names(auto_mpg) <- c(\"a\", \"b\", \"c\", \"d\", \"e\", \"f\", \"g\", \"h\", \"i\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<ol class=list-inline>\n",
       "\t<li>'a'</li>\n",
       "\t<li>'b'</li>\n",
       "\t<li>'c'</li>\n",
       "\t<li>'d'</li>\n",
       "\t<li>'e'</li>\n",
       "\t<li>'f'</li>\n",
       "\t<li>'g'</li>\n",
       "\t<li>'h'</li>\n",
       "\t<li>'i'</li>\n",
       "</ol>\n"
      ],
      "text/latex": [
       "\\begin{enumerate*}\n",
       "\\item 'a'\n",
       "\\item 'b'\n",
       "\\item 'c'\n",
       "\\item 'd'\n",
       "\\item 'e'\n",
       "\\item 'f'\n",
       "\\item 'g'\n",
       "\\item 'h'\n",
       "\\item 'i'\n",
       "\\end{enumerate*}\n"
      ],
      "text/markdown": [
       "1. 'a'\n",
       "2. 'b'\n",
       "3. 'c'\n",
       "4. 'd'\n",
       "5. 'e'\n",
       "6. 'f'\n",
       "7. 'g'\n",
       "8. 'h'\n",
       "9. 'i'\n",
       "\n",
       "\n"
      ],
      "text/plain": [
       "[1] \"a\" \"b\" \"c\" \"d\" \"e\" \"f\" \"g\" \"h\" \"i\""
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "names(auto_mpg)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Changing a specific column name"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "names(auto_mpg)[1] = \"z\" "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<ol class=list-inline>\n",
       "\t<li>'z'</li>\n",
       "\t<li>'b'</li>\n",
       "\t<li>'c'</li>\n",
       "\t<li>'d'</li>\n",
       "\t<li>'e'</li>\n",
       "\t<li>'f'</li>\n",
       "\t<li>'g'</li>\n",
       "\t<li>'h'</li>\n",
       "\t<li>'i'</li>\n",
       "</ol>\n"
      ],
      "text/latex": [
       "\\begin{enumerate*}\n",
       "\\item 'z'\n",
       "\\item 'b'\n",
       "\\item 'c'\n",
       "\\item 'd'\n",
       "\\item 'e'\n",
       "\\item 'f'\n",
       "\\item 'g'\n",
       "\\item 'h'\n",
       "\\item 'i'\n",
       "\\end{enumerate*}\n"
      ],
      "text/markdown": [
       "1. 'z'\n",
       "2. 'b'\n",
       "3. 'c'\n",
       "4. 'd'\n",
       "5. 'e'\n",
       "6. 'f'\n",
       "7. 'g'\n",
       "8. 'h'\n",
       "9. 'i'\n",
       "\n",
       "\n"
      ],
      "text/plain": [
       "[1] \"z\" \"b\" \"c\" \"d\" \"e\" \"f\" \"g\" \"h\" \"i\""
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "names(auto_mpg)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "names(auto_mpg) <- column_names"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<ol class=list-inline>\n",
       "\t<li>'mpg'</li>\n",
       "\t<li>'cylinders'</li>\n",
       "\t<li>'displacement'</li>\n",
       "\t<li>'horsepower'</li>\n",
       "\t<li>'weight'</li>\n",
       "\t<li>'acceleration'</li>\n",
       "\t<li>'model.year'</li>\n",
       "\t<li>'origin'</li>\n",
       "\t<li>'car.name'</li>\n",
       "</ol>\n"
      ],
      "text/latex": [
       "\\begin{enumerate*}\n",
       "\\item 'mpg'\n",
       "\\item 'cylinders'\n",
       "\\item 'displacement'\n",
       "\\item 'horsepower'\n",
       "\\item 'weight'\n",
       "\\item 'acceleration'\n",
       "\\item 'model.year'\n",
       "\\item 'origin'\n",
       "\\item 'car.name'\n",
       "\\end{enumerate*}\n"
      ],
      "text/markdown": [
       "1. 'mpg'\n",
       "2. 'cylinders'\n",
       "3. 'displacement'\n",
       "4. 'horsepower'\n",
       "5. 'weight'\n",
       "6. 'acceleration'\n",
       "7. 'model.year'\n",
       "8. 'origin'\n",
       "9. 'car.name'\n",
       "\n",
       "\n"
      ],
      "text/plain": [
       "[1] \"mpg\"          \"cylinders\"    \"displacement\" \"horsepower\"   \"weight\"      \n",
       "[6] \"acceleration\" \"model.year\"   \"origin\"       \"car.name\"    "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "names(auto_mpg)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "summary() command gives a summary of each variable in the dataframe. As shown below, the command is very informative. It calculates the minimum value, 1st quartile, 2nd quartile (mean), 3rd quartile, and the maximimum values of numeric variables. If the variable has NA values, number of such rows with NA values is displayed too. You can use this info to quickly identify if the variables are qualitative (discrete) or quantitative (continuous). For example, all the variables in auto_mpg dataset (except for origin and car name) are continuous. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "      mpg          cylinders      displacement     horsepower      weight    \n",
       " Min.   : 9.00   Min.   :3.000   Min.   : 68.0   150    : 22   Min.   :1613  \n",
       " 1st Qu.:17.50   1st Qu.:4.000   1st Qu.:104.2   90     : 20   1st Qu.:2224  \n",
       " Median :23.00   Median :4.000   Median :148.5   88     : 19   Median :2804  \n",
       " Mean   :23.51   Mean   :5.455   Mean   :193.4   110    : 18   Mean   :2970  \n",
       " 3rd Qu.:29.00   3rd Qu.:8.000   3rd Qu.:262.0   100    : 17   3rd Qu.:3608  \n",
       " Max.   :46.60   Max.   :8.000   Max.   :455.0   75     : 14   Max.   :5140  \n",
       "                                                 (Other):288                 \n",
       "  acceleration     model.year        origin                car.name  \n",
       " Min.   : 8.00   Min.   :70.00   Min.   :1.000   ford pinto    :  6  \n",
       " 1st Qu.:13.82   1st Qu.:73.00   1st Qu.:1.000   amc matador   :  5  \n",
       " Median :15.50   Median :76.00   Median :1.000   ford maverick :  5  \n",
       " Mean   :15.57   Mean   :76.01   Mean   :1.573   toyota corolla:  5  \n",
       " 3rd Qu.:17.18   3rd Qu.:79.00   3rd Qu.:2.000   amc gremlin   :  4  \n",
       " Max.   :24.80   Max.   :82.00   Max.   :3.000   amc hornet    :  4  \n",
       "                                                 (Other)       :369  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "summary(auto_mpg)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The reason why horsepower has no min, max, mean, etc is because the horsepower variable values are strings. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "'data.frame':\t398 obs. of  9 variables:\n",
      " $ mpg         : num  18 15 18 16 17 15 14 14 14 15 ...\n",
      " $ cylinders   : int  8 8 8 8 8 8 8 8 8 8 ...\n",
      " $ displacement: num  307 350 318 304 302 429 454 440 455 390 ...\n",
      " $ horsepower  : Factor w/ 94 levels \"?    \",\"100\",..: 17 35 29 29 24 42 47 46 48 40 ...\n",
      " $ weight      : int  3504 3693 3436 3433 3449 4341 4354 4312 4425 3850 ...\n",
      " $ acceleration: num  12 11.5 11 12 10.5 10 9 8.5 10 8.5 ...\n",
      " $ model.year  : int  70 70 70 70 70 70 70 70 70 70 ...\n",
      " $ origin      : int  1 1 1 1 1 1 1 1 1 1 ...\n",
      " $ car.name    : Factor w/ 305 levels \"amc ambassador brougham\",..: 50 37 232 15 162 142 55 224 242 2 ...\n"
     ]
    }
   ],
   "source": [
    "str(auto_mpg)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Horsepower is a factor datatype. Factors are categorical variables. They are not treated as continuous variables. So the summary function would not calculate its mean, median, min/max, etc. str() tells you the datatype of variables, the dimensions fo the dataframe, and also gives an overview of the kind of values each variable contains. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Column access"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "   Min. 1st Qu.  Median    Mean 3rd Qu.    Max. \n",
       "   1613    2224    2804    2970    3608    5140 "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "summary(auto_mpg$weight)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Descriptive Statistics\n",
    "Descriptive statistics are used to summarize and describe data. If we are analyzing miles per gallon data, for example, a descriptive statistics might be the percentage of cars with different numbers of cylinders or the average miles per gallon for all cars. Many descriptive statistics are often used at one time to give a full picture of the data. \n",
    "\n",
    "There are mainly 2 categories of descriptive statics: measures of central tendency (or averages) and measures of dispersion (which summarizes how spread out and dispersed the data points are). A variable can have many observations (data points or values) and a summary set of numbers that describe those multiple observations, such as those shown by the summary() command, are descriptive statistics. \n",
    "\n",
    "There are 3 important measures of central tendency used to summarize data. The mean, the median, and the mode. When we talk about the mean, we'll be referring to the arithmetic mean as contrasted to some other means, such as the geometric mean or the harmonic mean, where are not used as frequently as the arithmetic mean. The mean of a set of data is simply the sum of data observations divided by the total number of observations. \n",
    "\n",
    "The median of a set of ordered observations is a middle number that divides the data into 2 parts, where half of the data points are in one part and the other in the 2nd part. \n",
    "\n",
    "The mean is influenced to a greater extent by extreme observations. So, if you notice extreme observations in your data, then perhaps a median is a better summary of data than a mean. Income and price data generally follow this pattern, which is why census organizations report median incomes and median prices. \n",
    "\n",
    "Descriptive statistics are just descriptive. They cannot generalize anything beyond the data at hand. Generalizing from our data to another set of cases is dealt with in inferential statistics. R has built-in functions to calculate the mean, median, and standard deviation and other descriptive statistics. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<thead><tr><th scope=col>mpg</th><th scope=col>cylinders</th><th scope=col>displacement</th><th scope=col>horsepower</th><th scope=col>weight</th><th scope=col>acceleration</th><th scope=col>model.year</th><th scope=col>origin</th><th scope=col>car.name</th></tr></thead>\n",
       "<tbody>\n",
       "\t<tr><td>18                       </td><td>8                        </td><td>307                      </td><td>130                      </td><td>3504                     </td><td>12.0                     </td><td>70                       </td><td>1                        </td><td>chevrolet chevelle malibu</td></tr>\n",
       "\t<tr><td>15                       </td><td>8                        </td><td>350                      </td><td>165                      </td><td>3693                     </td><td>11.5                     </td><td>70                       </td><td>1                        </td><td>buick skylark 320        </td></tr>\n",
       "\t<tr><td>18                       </td><td>8                        </td><td>318                      </td><td>150                      </td><td>3436                     </td><td>11.0                     </td><td>70                       </td><td>1                        </td><td>plymouth satellite       </td></tr>\n",
       "\t<tr><td>16                       </td><td>8                        </td><td>304                      </td><td>150                      </td><td>3433                     </td><td>12.0                     </td><td>70                       </td><td>1                        </td><td>amc rebel sst            </td></tr>\n",
       "\t<tr><td>17                       </td><td>8                        </td><td>302                      </td><td>140                      </td><td>3449                     </td><td>10.5                     </td><td>70                       </td><td>1                        </td><td>ford torino              </td></tr>\n",
       "\t<tr><td>15                       </td><td>8                        </td><td>429                      </td><td>198                      </td><td>4341                     </td><td>10.0                     </td><td>70                       </td><td>1                        </td><td>ford galaxie 500         </td></tr>\n",
       "</tbody>\n",
       "</table>\n"
      ],
      "text/latex": [
       "\\begin{tabular}{r|lllllllll}\n",
       " mpg & cylinders & displacement & horsepower & weight & acceleration & model.year & origin & car.name\\\\\n",
       "\\hline\n",
       "\t 18                        & 8                         & 307                       & 130                       & 3504                      & 12.0                      & 70                        & 1                         & chevrolet chevelle malibu\\\\\n",
       "\t 15                        & 8                         & 350                       & 165                       & 3693                      & 11.5                      & 70                        & 1                         & buick skylark 320        \\\\\n",
       "\t 18                        & 8                         & 318                       & 150                       & 3436                      & 11.0                      & 70                        & 1                         & plymouth satellite       \\\\\n",
       "\t 16                        & 8                         & 304                       & 150                       & 3433                      & 12.0                      & 70                        & 1                         & amc rebel sst            \\\\\n",
       "\t 17                        & 8                         & 302                       & 140                       & 3449                      & 10.5                      & 70                        & 1                         & ford torino              \\\\\n",
       "\t 15                        & 8                         & 429                       & 198                       & 4341                      & 10.0                      & 70                        & 1                         & ford galaxie 500         \\\\\n",
       "\\end{tabular}\n"
      ],
      "text/markdown": [
       "\n",
       "mpg | cylinders | displacement | horsepower | weight | acceleration | model.year | origin | car.name | \n",
       "|---|---|---|---|---|---|\n",
       "| 18                        | 8                         | 307                       | 130                       | 3504                      | 12.0                      | 70                        | 1                         | chevrolet chevelle malibu | \n",
       "| 15                        | 8                         | 350                       | 165                       | 3693                      | 11.5                      | 70                        | 1                         | buick skylark 320         | \n",
       "| 18                        | 8                         | 318                       | 150                       | 3436                      | 11.0                      | 70                        | 1                         | plymouth satellite        | \n",
       "| 16                        | 8                         | 304                       | 150                       | 3433                      | 12.0                      | 70                        | 1                         | amc rebel sst             | \n",
       "| 17                        | 8                         | 302                       | 140                       | 3449                      | 10.5                      | 70                        | 1                         | ford torino               | \n",
       "| 15                        | 8                         | 429                       | 198                       | 4341                      | 10.0                      | 70                        | 1                         | ford galaxie 500          | \n",
       "\n",
       "\n"
      ],
      "text/plain": [
       "  mpg cylinders displacement horsepower weight acceleration model.year origin\n",
       "1 18  8         307          130        3504   12.0         70         1     \n",
       "2 15  8         350          165        3693   11.5         70         1     \n",
       "3 18  8         318          150        3436   11.0         70         1     \n",
       "4 16  8         304          150        3433   12.0         70         1     \n",
       "5 17  8         302          140        3449   10.5         70         1     \n",
       "6 15  8         429          198        4341   10.0         70         1     \n",
       "  car.name                 \n",
       "1 chevrolet chevelle malibu\n",
       "2 buick skylark 320        \n",
       "3 plymouth satellite       \n",
       "4 amc rebel sst            \n",
       "5 ford torino              \n",
       "6 ford galaxie 500         "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "head(auto_mpg)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "### Mean\n",
    "The mean() function gives the average value of the column. The mean is the most basic statistics to help you understand the distribution of observations (data points) of a variable. \n",
    "\n",
    "    Mean = Sum of all Observations / No. of Observations"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "'Average auto displacement is: 193.425879396985'"
      ],
      "text/latex": [
       "'Average auto displacement is: 193.425879396985'"
      ],
      "text/markdown": [
       "'Average auto displacement is: 193.425879396985'"
      ],
      "text/plain": [
       "[1] \"Average auto displacement is: 193.425879396985\""
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "paste(\"Average auto displacement is:\", mean(auto_mpg$displacement))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "193.425879396985"
      ],
      "text/latex": [
       "193.425879396985"
      ],
      "text/markdown": [
       "193.425879396985"
      ],
      "text/plain": [
       "[1] 193.4259"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "mean(auto_mpg$displacement)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Mode\n",
    "Mode is the value that has been repeated most frequently in a set of values and is especially useful when dealing with discrete variables. R does not have any built-in function to compute Mode as you would expect. Instead, the mode() function returns the type or storage mode of the object."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "'Datatype of mpg is: numeric'"
      ],
      "text/latex": [
       "'Datatype of mpg is: numeric'"
      ],
      "text/markdown": [
       "'Datatype of mpg is: numeric'"
      ],
      "text/plain": [
       "[1] \"Datatype of mpg is: numeric\""
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "paste(\"Datatype of mpg is:\", mode(auto_mpg$mpg))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "However, you can use a command like below to calculate the most frequently occurring value. The value() command tells us the distribution/count of different values of a variable. So, using the command below, we are able to print how often each value occurred in the dataset. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\n",
       "   9   10   11   12   13   14 14.5   15 15.5   16 16.2 16.5 16.9   17 17.5 17.6 \n",
       "   1    2    4    6   20   19    1   16    5   13    1    3    1    7    5    2 \n",
       "17.7   18 18.1 18.2 18.5 18.6   19 19.1 19.2 19.4 19.8 19.9   20 20.2 20.3 20.5 \n",
       "   1   17    2    1    3    1   12    1    3    2    1    1    9    4    1    3 \n",
       "20.6 20.8   21 21.1 21.5 21.6   22 22.3 22.4 22.5   23 23.2 23.5 23.6 23.7 23.8 \n",
       "   2    1    8    1    3    1   10    1    1    1   10    1    1    1    1    1 \n",
       "23.9   24 24.2 24.3 24.5   25 25.1 25.4 25.5 25.8   26 26.4 26.5 26.6 26.8   27 \n",
       "   2   11    1    1    2   11    1    2    2    1   14    1    1    2    1    9 \n",
       "27.2 27.4 27.5 27.9   28 28.1 28.4 28.8   29 29.5 29.8 29.9   30 30.5 30.7 30.9 \n",
       "   3    1    1    1   10    1    1    1    8    2    2    1    7    2    1    1 \n",
       "  31 31.3 31.5 31.6 31.8 31.9   32 32.1 32.2 32.3 32.4 32.7 32.8 32.9   33 33.5 \n",
       "   7    1    2    1    1    1    6    1    1    1    2    1    1    1    3    3 \n",
       "33.7 33.8   34 34.1 34.2 34.3 34.4 34.5 34.7   35 35.1 35.7   36 36.1 36.4   37 \n",
       "   1    1    2    2    1    1    1    2    1    2    1    1    6    2    1    3 \n",
       "37.2 37.3 37.7   38 38.1   39 39.1 39.4 40.8 40.9 41.5 43.1 43.4   44 44.3 44.6 \n",
       "   1    1    1    4    1    1    1    1    1    1    1    1    1    1    1    1 \n",
       "46.6 \n",
       "   1 "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "table(auto_mpg$mpg)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Note:** In the output above, the table is line-wrapping in the display\n",
    "```\n",
    "Value->   9   10   11   12   13   14 14.5   15 15.5   16 16.2 16.5 16.9   17 17.5 17.6 \n",
    "Count->   1    2    4    6   20   19    1   16    5   13    1    3    1    7    5    2 \n",
    "\n",
    "Value->   17.7   18 18.1 18.2 18.5 18.6   19 19.1 19.2 19.4 19.8 19.9   20 20.2 20.3 20.5\n",
    "Count->      1   17    2    1    3    1   12    1    3    2    1    1    9    4    1    3\n",
    "\n",
    "...\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We see that 13 MPG occurs 20 times (13 is at position 5). The table is actually computing a histogram of the value for the given set/column. This tells us, 13 is the most commonly occurring mileage (mpg) of the vehicles. Of the 398 vehicles, 20 vehicles have 13 miles per gallon. \n",
    "\n",
    "Then we can use which.max() to ask: which index holds the greatest value?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<strong>13:</strong> 5"
      ],
      "text/latex": [
       "\\textbf{13:} 5"
      ],
      "text/markdown": [
       "**13:** 5"
      ],
      "text/plain": [
       "13 \n",
       " 5 "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "which.max(table(auto_mpg$mpg))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "'The mode using which.max(): 13'"
      ],
      "text/latex": [
       "'The mode using which.max(): 13'"
      ],
      "text/markdown": [
       "'The mode using which.max(): 13'"
      ],
      "text/plain": [
       "[1] \"The mode using which.max(): 13\""
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "paste(\"The mode using which.max():\", names(which.max(table(auto_mpg$mpg))))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Median \n",
    "The median value divides the dataset into 2 equal halves. One half lies to the left of the median and the other to the right. Median values are less affected than the mean by outliers (extreme values). Therefore, the median is considered an ideal choice for measuring central tendency when the data is skewed (when the data has outliers). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "15.5"
      ],
      "text/latex": [
       "15.5"
      ],
      "text/markdown": [
       "15.5"
      ],
      "text/plain": [
       "[1] 15.5"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "median(auto_mpg$acceleration)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "'median: 15.5'"
      ],
      "text/latex": [
       "'median: 15.5'"
      ],
      "text/markdown": [
       "'median: 15.5'"
      ],
      "text/plain": [
       "[1] \"median: 15.5\""
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "paste(\"median:\", median(auto_mpg$acceleration))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Of 398 observations in the dataset, 199 observations have acceleration less than or equal to 15.5 and the other 199 observations have acceleration greater than or equal to 15.5."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Range\n",
    "The range is also a measure of spread or extreme values of a variable. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<ol class=list-inline>\n",
       "\t<li>70</li>\n",
       "\t<li>82</li>\n",
       "</ol>\n"
      ],
      "text/latex": [
       "\\begin{enumerate*}\n",
       "\\item 70\n",
       "\\item 82\n",
       "\\end{enumerate*}\n"
      ],
      "text/markdown": [
       "1. 70\n",
       "2. 82\n",
       "\n",
       "\n"
      ],
      "text/plain": [
       "[1] 70 82"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "range(auto_mpg$model.year)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So the range of model years for cars in the dataset are from year 70 to year 82"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Quantile\n",
    "The quantile() function divides the dataset into 4 equal parts, based on quantity of measurements. The first is Q1, second is Q2, third is Q3, and fourth is Q4. Quantiles are well understood when used with boxplots. Boxplots summarize and identify the range (min and max), Q1, Q2, and Q3 of a variable. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<dl class=dl-horizontal>\n",
       "\t<dt>0%</dt>\n",
       "\t\t<dd>68</dd>\n",
       "\t<dt>25%</dt>\n",
       "\t\t<dd>104.25</dd>\n",
       "\t<dt>50%</dt>\n",
       "\t\t<dd>148.5</dd>\n",
       "\t<dt>75%</dt>\n",
       "\t\t<dd>262</dd>\n",
       "\t<dt>100%</dt>\n",
       "\t\t<dd>455</dd>\n",
       "</dl>\n"
      ],
      "text/latex": [
       "\\begin{description*}\n",
       "\\item[0\\textbackslash{}\\%] 68\n",
       "\\item[25\\textbackslash{}\\%] 104.25\n",
       "\\item[50\\textbackslash{}\\%] 148.5\n",
       "\\item[75\\textbackslash{}\\%] 262\n",
       "\\item[100\\textbackslash{}\\%] 455\n",
       "\\end{description*}\n"
      ],
      "text/markdown": [
       "0%\n",
       ":   6825%\n",
       ":   104.2550%\n",
       ":   148.575%\n",
       ":   262100%\n",
       ":   455\n",
       "\n"
      ],
      "text/plain": [
       "    0%    25%    50%    75%   100% \n",
       " 68.00 104.25 148.50 262.00 455.00 "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "quantile(auto_mpg$displacement)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<dl class=dl-horizontal>\n",
       "\t<dt>0%</dt>\n",
       "\t\t<dd>70</dd>\n",
       "\t<dt>25%</dt>\n",
       "\t\t<dd>73</dd>\n",
       "\t<dt>50%</dt>\n",
       "\t\t<dd>76</dd>\n",
       "\t<dt>75%</dt>\n",
       "\t\t<dd>79</dd>\n",
       "\t<dt>100%</dt>\n",
       "\t\t<dd>82</dd>\n",
       "</dl>\n"
      ],
      "text/latex": [
       "\\begin{description*}\n",
       "\\item[0\\textbackslash{}\\%] 70\n",
       "\\item[25\\textbackslash{}\\%] 73\n",
       "\\item[50\\textbackslash{}\\%] 76\n",
       "\\item[75\\textbackslash{}\\%] 79\n",
       "\\item[100\\textbackslash{}\\%] 82\n",
       "\\end{description*}\n"
      ],
      "text/markdown": [
       "0%\n",
       ":   7025%\n",
       ":   7350%\n",
       ":   7675%\n",
       ":   79100%\n",
       ":   82\n",
       "\n"
      ],
      "text/plain": [
       "  0%  25%  50%  75% 100% \n",
       "  70   73   76   79   82 "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "quantile(auto_mpg$model.year)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The command is very informative as it gives min, max, 25th, 50th (median), and 75th percentile of values of the variable. Quantiles are used for explaining the variance in the variables as it is less immune to outliers and explains variation better than other measures. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Variance\n",
    "Variance measures how widely the values in the variable are spread around the mean. If the observations vary greatly from the variable mean, the variance will be big and vice-versa. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "10872.1991522474"
      ],
      "text/latex": [
       "10872.1991522474"
      ],
      "text/markdown": [
       "10872.1991522474"
      ],
      "text/plain": [
       "[1] 10872.2"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "var(auto_mpg$displacement)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "'variance 10872.1991522474'"
      ],
      "text/latex": [
       "'variance 10872.1991522474'"
      ],
      "text/markdown": [
       "'variance 10872.1991522474'"
      ],
      "text/plain": [
       "[1] \"variance 10872.1991522474\""
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "paste(\"variance\", var(auto_mpg$displacement))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The above value represents the squared error of all the displacement values. Variance often doesn't make much sense when trying to understand the spread of the data as the units of variance are not the same as the units of the original data. However, standard deviation will give us a clearer idea of how data is spread. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Standard deviation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "104.269838171196"
      ],
      "text/latex": [
       "104.269838171196"
      ],
      "text/markdown": [
       "104.269838171196"
      ],
      "text/plain": [
       "[1] 104.2698"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "sd(auto_mpg$displacement)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "'standard deviation 104.269838171196'"
      ],
      "text/latex": [
       "'standard deviation 104.269838171196'"
      ],
      "text/markdown": [
       "'standard deviation 104.269838171196'"
      ],
      "text/plain": [
       "[1] \"standard deviation 104.269838171196\""
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "paste(\"standard deviation\", sd(auto_mpg$displacement))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The values in the displacement variable have a standard deviation of 104.27. Recall that the mean was 193.42. The combination of mean and standard deviation can be combined to model a data population. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Max, Min, Median, and Mean Absolute Deviation\n",
    "We'll be doing this using a forloop (super slow). c() combines its arguments into a vector. Min, max, etc cannot by calculated for factor data. So let's create a new df called numeric_data with horsepower as numeric type variable. We'll also exclude the car name."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "numeric_data <- auto_mpg[-ncol(auto_mpg)] # take all columns except for the last column"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<thead><tr><th scope=col>mpg</th><th scope=col>cylinders</th><th scope=col>displacement</th><th scope=col>horsepower</th><th scope=col>weight</th><th scope=col>acceleration</th><th scope=col>model.year</th><th scope=col>origin</th></tr></thead>\n",
       "<tbody>\n",
       "\t<tr><td>18  </td><td>8   </td><td>307 </td><td>130 </td><td>3504</td><td>12.0</td><td>70  </td><td>1   </td></tr>\n",
       "\t<tr><td>15  </td><td>8   </td><td>350 </td><td>165 </td><td>3693</td><td>11.5</td><td>70  </td><td>1   </td></tr>\n",
       "\t<tr><td>18  </td><td>8   </td><td>318 </td><td>150 </td><td>3436</td><td>11.0</td><td>70  </td><td>1   </td></tr>\n",
       "\t<tr><td>16  </td><td>8   </td><td>304 </td><td>150 </td><td>3433</td><td>12.0</td><td>70  </td><td>1   </td></tr>\n",
       "\t<tr><td>17  </td><td>8   </td><td>302 </td><td>140 </td><td>3449</td><td>10.5</td><td>70  </td><td>1   </td></tr>\n",
       "\t<tr><td>15  </td><td>8   </td><td>429 </td><td>198 </td><td>4341</td><td>10.0</td><td>70  </td><td>1   </td></tr>\n",
       "</tbody>\n",
       "</table>\n"
      ],
      "text/latex": [
       "\\begin{tabular}{r|llllllll}\n",
       " mpg & cylinders & displacement & horsepower & weight & acceleration & model.year & origin\\\\\n",
       "\\hline\n",
       "\t 18   & 8    & 307  & 130  & 3504 & 12.0 & 70   & 1   \\\\\n",
       "\t 15   & 8    & 350  & 165  & 3693 & 11.5 & 70   & 1   \\\\\n",
       "\t 18   & 8    & 318  & 150  & 3436 & 11.0 & 70   & 1   \\\\\n",
       "\t 16   & 8    & 304  & 150  & 3433 & 12.0 & 70   & 1   \\\\\n",
       "\t 17   & 8    & 302  & 140  & 3449 & 10.5 & 70   & 1   \\\\\n",
       "\t 15   & 8    & 429  & 198  & 4341 & 10.0 & 70   & 1   \\\\\n",
       "\\end{tabular}\n"
      ],
      "text/markdown": [
       "\n",
       "mpg | cylinders | displacement | horsepower | weight | acceleration | model.year | origin | \n",
       "|---|---|---|---|---|---|\n",
       "| 18   | 8    | 307  | 130  | 3504 | 12.0 | 70   | 1    | \n",
       "| 15   | 8    | 350  | 165  | 3693 | 11.5 | 70   | 1    | \n",
       "| 18   | 8    | 318  | 150  | 3436 | 11.0 | 70   | 1    | \n",
       "| 16   | 8    | 304  | 150  | 3433 | 12.0 | 70   | 1    | \n",
       "| 17   | 8    | 302  | 140  | 3449 | 10.5 | 70   | 1    | \n",
       "| 15   | 8    | 429  | 198  | 4341 | 10.0 | 70   | 1    | \n",
       "\n",
       "\n"
      ],
      "text/plain": [
       "  mpg cylinders displacement horsepower weight acceleration model.year origin\n",
       "1 18  8         307          130        3504   12.0         70         1     \n",
       "2 15  8         350          165        3693   11.5         70         1     \n",
       "3 18  8         318          150        3436   11.0         70         1     \n",
       "4 16  8         304          150        3433   12.0         70         1     \n",
       "5 17  8         302          140        3449   10.5         70         1     \n",
       "6 15  8         429          198        4341   10.0         70         1     "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "head(numeric_data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "numeric_data$horsepower <- as.numeric(numeric_data$horsepower) # convert horsepower to numeric"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "'data.frame':\t398 obs. of  8 variables:\n",
      " $ mpg         : num  18 15 18 16 17 15 14 14 14 15 ...\n",
      " $ cylinders   : int  8 8 8 8 8 8 8 8 8 8 ...\n",
      " $ displacement: num  307 350 318 304 302 429 454 440 455 390 ...\n",
      " $ horsepower  : num  17 35 29 29 24 42 47 46 48 40 ...\n",
      " $ weight      : int  3504 3693 3436 3433 3449 4341 4354 4312 4425 3850 ...\n",
      " $ acceleration: num  12 11.5 11 12 10.5 10 9 8.5 10 8.5 ...\n",
      " $ model.year  : int  70 70 70 70 70 70 70 70 70 70 ...\n",
      " $ origin      : int  1 1 1 1 1 1 1 1 1 1 ...\n"
     ]
    }
   ],
   "source": [
    "str(numeric_data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "8"
      ],
      "text/latex": [
       "8"
      ],
      "text/markdown": [
       "8"
      ],
      "text/plain": [
       "[1] 8"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "ncol(numeric_data) # number of columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1] \"         Column    Maximum    Minimum     Median        Mean Absolute Deviation\"\n",
      "[1] \"            mpg       46.6        9.0       23.0        8.9\"\n",
      "[1] \"      cylinders        8.0        3.0        4.0        0.0\"\n",
      "[1] \"   displacement      455.0       68.0      148.5       86.7\"\n",
      "[1] \"     horsepower       94.0        1.0       60.5       35.6\"\n",
      "[1] \"         weight     5140.0     1613.0     2803.5      945.2\"\n",
      "[1] \"   acceleration       24.8        8.0       15.5        2.5\"\n",
      "[1] \"     model.year       82.0       70.0       76.0        4.4\"\n",
      "[1] \"         origin        3.0        1.0        1.0        0.0\"\n"
     ]
    }
   ],
   "source": [
    "print(sprintf(\"%15s %10s %10s %10s %30s\", \"Column\", \"Maximum\", \"Minimum\", \"Median\", \"Mean Absolute Deviation\"))\n",
    "for (i in 1:ncol(numeric_data)){\n",
    "    print(sprintf(\"%15s %10.1f %10.1f %10.1f %10.1f\", \n",
    "                    names(numeric_data[i]), \n",
    "                    max(numeric_data[, i]),\n",
    "                    min(numeric_data[, i]),\n",
    "                    median(numeric_data[, i]),\n",
    "                    mad(numeric_data[, i])\n",
    "                 ))\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The above code is effective but inefficient. Let's do the same thing with apply(). 2 refers the columns in the x array while 1 refers to rows. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<thead><tr><th></th><th scope=col>Max</th><th scope=col>Min</th><th scope=col>Median</th><th scope=col>Mean_Absolute_Dev</th></tr></thead>\n",
       "<tbody>\n",
       "\t<tr><th scope=row>mpg</th><td>  46.6   </td><td>   9     </td><td>  23.0   </td><td>  8.89560</td></tr>\n",
       "\t<tr><th scope=row>cylinders</th><td>   8.0   </td><td>   3     </td><td>   4.0   </td><td>  0.00000</td></tr>\n",
       "\t<tr><th scope=row>displacement</th><td> 455.0   </td><td>  68     </td><td> 148.5   </td><td> 86.73210</td></tr>\n",
       "\t<tr><th scope=row>horsepower</th><td>  94.0   </td><td>   1     </td><td>  60.5   </td><td> 35.58240</td></tr>\n",
       "\t<tr><th scope=row>weight</th><td>5140.0   </td><td>1613     </td><td>2803.5   </td><td>945.15750</td></tr>\n",
       "\t<tr><th scope=row>acceleration</th><td>  24.8   </td><td>   8     </td><td>  15.5   </td><td>  2.52042</td></tr>\n",
       "\t<tr><th scope=row>model.year</th><td>  82.0   </td><td>  70     </td><td>  76.0   </td><td>  4.44780</td></tr>\n",
       "\t<tr><th scope=row>origin</th><td>   3.0   </td><td>   1     </td><td>   1.0   </td><td>  0.00000</td></tr>\n",
       "</tbody>\n",
       "</table>\n"
      ],
      "text/latex": [
       "\\begin{tabular}{r|llll}\n",
       "  & Max & Min & Median & Mean\\_Absolute\\_Dev\\\\\n",
       "\\hline\n",
       "\tmpg &   46.6    &    9      &   23.0    &   8.89560\\\\\n",
       "\tcylinders &    8.0    &    3      &    4.0    &   0.00000\\\\\n",
       "\tdisplacement &  455.0    &   68      &  148.5    &  86.73210\\\\\n",
       "\thorsepower &   94.0    &    1      &   60.5    &  35.58240\\\\\n",
       "\tweight & 5140.0    & 1613      & 2803.5    & 945.15750\\\\\n",
       "\tacceleration &   24.8    &    8      &   15.5    &   2.52042\\\\\n",
       "\tmodel.year &   82.0    &   70      &   76.0    &   4.44780\\\\\n",
       "\torigin &    3.0    &    1      &    1.0    &   0.00000\\\\\n",
       "\\end{tabular}\n"
      ],
      "text/markdown": [
       "\n",
       "| <!--/--> | Max | Min | Median | Mean_Absolute_Dev | \n",
       "|---|---|---|---|---|---|---|---|\n",
       "| mpg |   46.6    |    9      |   23.0    |   8.89560 | \n",
       "| cylinders |    8.0    |    3      |    4.0    |   0.00000 | \n",
       "| displacement |  455.0    |   68      |  148.5    |  86.73210 | \n",
       "| horsepower |   94.0    |    1      |   60.5    |  35.58240 | \n",
       "| weight | 5140.0    | 1613      | 2803.5    | 945.15750 | \n",
       "| acceleration |   24.8    |    8      |   15.5    |   2.52042 | \n",
       "| model.year |   82.0    |   70      |   76.0    |   4.44780 | \n",
       "| origin |    3.0    |    1      |    1.0    |   0.00000 | \n",
       "\n",
       "\n"
      ],
      "text/plain": [
       "             Max    Min  Median Mean_Absolute_Dev\n",
       "mpg            46.6    9   23.0   8.89560        \n",
       "cylinders       8.0    3    4.0   0.00000        \n",
       "displacement  455.0   68  148.5  86.73210        \n",
       "horsepower     94.0    1   60.5  35.58240        \n",
       "weight       5140.0 1613 2803.5 945.15750        \n",
       "acceleration   24.8    8   15.5   2.52042        \n",
       "model.year     82.0   70   76.0   4.44780        \n",
       "origin          3.0    1    1.0   0.00000        "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "cbind(Max = apply(numeric_data, 2, max), \n",
    "      Min = apply(numeric_data, 2, min),\n",
    "      Median = apply(numeric_data, 2, median),\n",
    "      Mean_Absolute_Dev = apply(numeric_data, 2, mad)\n",
    "     )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Types of Variables\n",
    "The most important distinction between variables is if they are either qualitative or quantitative. \n",
    "\n",
    "* Qualitative variables: Variables that express a qualitative attribute such as religion, favorite movie, gender, and so on fall into this category. Sometimes referred to as categorical or nominal variables. \n",
    "\n",
    "* Quantitative variables: Variables that are measure in terms of numbers. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Flavors of quantitative data:\n",
    "* Descrete variables: Some measures in data are discrete and cannot be made more precise. For example, the number of children in a family is discrete because you are counting indivisible entities. You can't have 1.3 children. \n",
    "* Continuous variables: Data can be reduced to finer levels or we can say it is continuous in nature. You can measure the weight of yourself at different precisions. milligrams, grams, pounds, etc. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Levels of measurement:\n",
    "Both qualitative and quantitative variables follow levels of measurement. There are 4 levels: nominal, ordinal, interval, and ratio scaled. \n",
    "* Nominal: Car name, marital status, gender, religion, etc\n",
    "* Ordinal: Number of cylinders in a car have an order. Soldier rankings, student grade, etc\n",
    "* Interval: 10-15 are intervals. Temperature, IQ, etc are also intervals\n",
    "* Ratio: Daily calorie intake or GPA score. \n",
    "\n",
    "http://onlinestatbook.com/2/introduction/levels_of_measurement.html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "      mpg          cylinders      displacement     horsepower      weight    \n",
       " Min.   : 9.00   Min.   :3.000   Min.   : 68.0   150    : 22   Min.   :1613  \n",
       " 1st Qu.:17.50   1st Qu.:4.000   1st Qu.:104.2   90     : 20   1st Qu.:2224  \n",
       " Median :23.00   Median :4.000   Median :148.5   88     : 19   Median :2804  \n",
       " Mean   :23.51   Mean   :5.455   Mean   :193.4   110    : 18   Mean   :2970  \n",
       " 3rd Qu.:29.00   3rd Qu.:8.000   3rd Qu.:262.0   100    : 17   3rd Qu.:3608  \n",
       " Max.   :46.60   Max.   :8.000   Max.   :455.0   75     : 14   Max.   :5140  \n",
       "                                                 (Other):288                 \n",
       "  acceleration     model.year        origin                car.name  \n",
       " Min.   : 8.00   Min.   :70.00   Min.   :1.000   ford pinto    :  6  \n",
       " 1st Qu.:13.82   1st Qu.:73.00   1st Qu.:1.000   amc matador   :  5  \n",
       " Median :15.50   Median :76.00   Median :1.000   ford maverick :  5  \n",
       " Mean   :15.57   Mean   :76.01   Mean   :1.573   toyota corolla:  5  \n",
       " 3rd Qu.:17.18   3rd Qu.:79.00   3rd Qu.:2.000   amc gremlin   :  4  \n",
       " Max.   :24.80   Max.   :82.00   Max.   :3.000   amc hornet    :  4  \n",
       "                                                 (Other)       :369  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "summary(auto_mpg)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "'data.frame':\t398 obs. of  9 variables:\n",
      " $ mpg         : num  18 15 18 16 17 15 14 14 14 15 ...\n",
      " $ cylinders   : int  8 8 8 8 8 8 8 8 8 8 ...\n",
      " $ displacement: num  307 350 318 304 302 429 454 440 455 390 ...\n",
      " $ horsepower  : Factor w/ 94 levels \"?    \",\"100\",..: 17 35 29 29 24 42 47 46 48 40 ...\n",
      " $ weight      : int  3504 3693 3436 3433 3449 4341 4354 4312 4425 3850 ...\n",
      " $ acceleration: num  12 11.5 11 12 10.5 10 9 8.5 10 8.5 ...\n",
      " $ model.year  : int  70 70 70 70 70 70 70 70 70 70 ...\n",
      " $ origin      : int  1 1 1 1 1 1 1 1 1 1 ...\n",
      " $ car.name    : Factor w/ 305 levels \"amc ambassador brougham\",..: 50 37 232 15 162 142 55 224 242 2 ...\n"
     ]
    }
   ],
   "source": [
    "str(auto_mpg)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "R",
   "language": "R",
   "name": "ir"
  },
  "language_info": {
   "codemirror_mode": "r",
   "file_extension": ".r",
   "mimetype": "text/x-r-source",
   "name": "R",
   "pygments_lexer": "r",
   "version": "3.4.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}