huml/day1/temperature.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Imporant Imports**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load CSV\n",
    "Some **very** predictable data is available in `../../data/temperature.csv`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>f</th>\n",
       "      <th>c</th>\n",
       "      <th>k</th>\n",
       "      <th>description</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>-459.67</td>\n",
       "      <td>-273.15</td>\n",
       "      <td>0.00</td>\n",
       "      <td>Absolute 0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>28.40</td>\n",
       "      <td>-2.00</td>\n",
       "      <td>271.15</td>\n",
       "      <td>seawater freezes</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>32.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>273.15</td>\n",
       "      <td>water freezes</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>68.00</td>\n",
       "      <td>20.00</td>\n",
       "      <td>293.15</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>86.00</td>\n",
       "      <td>30.00</td>\n",
       "      <td>303.15</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>122.00</td>\n",
       "      <td>50.00</td>\n",
       "      <td>323.15</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>158.00</td>\n",
       "      <td>70.00</td>\n",
       "      <td>343.15</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>194.00</td>\n",
       "      <td>90.00</td>\n",
       "      <td>363.15</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>212.00</td>\n",
       "      <td>100.00</td>\n",
       "      <td>373.15</td>\n",
       "      <td>water boils</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>1220.00</td>\n",
       "      <td>660.00</td>\n",
       "      <td>933.15</td>\n",
       "      <td>aluminum freezes</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>1517.00</td>\n",
       "      <td>825.00</td>\n",
       "      <td>1098.15</td>\n",
       "      <td>petroleum fire temperature</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>2732.00</td>\n",
       "      <td>1500.00</td>\n",
       "      <td>1773.15</td>\n",
       "      <td>steel freezes</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>6920.00</td>\n",
       "      <td>3826.85</td>\n",
       "      <td>4100.00</td>\n",
       "      <td>aluminum burns</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>27000032.00</td>\n",
       "      <td>15000000.00</td>\n",
       "      <td>14999726.85</td>\n",
       "      <td>Sun</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              f            c            k                 description\n",
       "0       -459.67      -273.15         0.00                  Absolute 0\n",
       "1         28.40        -2.00       271.15            seawater freezes\n",
       "2         32.00         0.00       273.15               water freezes\n",
       "3         68.00        20.00       293.15                         NaN\n",
       "4         86.00        30.00       303.15                         NaN\n",
       "5        122.00        50.00       323.15                         NaN\n",
       "6        158.00        70.00       343.15                         NaN\n",
       "7        194.00        90.00       363.15                         NaN\n",
       "8        212.00       100.00       373.15                 water boils\n",
       "9       1220.00       660.00       933.15            aluminum freezes\n",
       "10      1517.00       825.00      1098.15  petroleum fire temperature\n",
       "11      2732.00      1500.00      1773.15               steel freezes\n",
       "12      6920.00      3826.85      4100.00              aluminum burns\n",
       "13  27000032.00  15000000.00  14999726.85                         Sun"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = truth = pd.read_csv('../../data/temperature.csv')\n",
    "truth\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Regression\n",
    "There are a lot of ways to compute the slope and offset  \n",
    "They will all work equally well for this simple example  \n",
    "`y = a * x + b`  \n",
    "`celsius = a * fahrenheit + b`  \n",
    "`a = rise / run = (y[i] - y[j]) / (x[i] - x[j])`  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(1.8, 31.999999999999943)"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "i, j = 1, 0\n",
    "rise = delta_y = delta_f = df.f[i] - df.f[j]\n",
    "run = delta_x = delta_c = df.c[i] - df.c[j]\n",
    "a = slope = rise / run\n",
    "x, y = df.c[j], df.f[j]\n",
    "offset = b = y - a * x\n",
    "slope, offset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(1.8, 31.999999999999943)"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rise = delta_y = delta_f = df.f[i] - df.f[j]\n",
    "run = delta_x = delta_c = df.c[i] - df.c[j]\n",
    "a = slope = rise / run\n",
    "x, y = df.c[j], df.f[j]\n",
    "offset = b = y - a * x\n",
    "slope, offset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# *Exercise 1!*\n",
    "Create a function (use `def`) to convert from fahrenheit to celsius\n",
    "Include doctests to verify that the results using the data file above"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def f2c(c):\n",
    "    # your code here\n",
    "    return f"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Measurement Error\n",
    "It's important to be able to simulate error to do \"sensitivity studies\"  \n",
    "or just find out if your answer is still \"good\" even with a little noise.  \n",
    "`error = (scale_factor_error * true_value * normal_noise) + bias`  \n",
    "`y = 0.01 * randn(len(x)) * x + 1.0`  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>f</th>\n",
       "      <th>c</th>\n",
       "      <th>k</th>\n",
       "      <th>description</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>-459.67</td>\n",
       "      <td>-271.563974</td>\n",
       "      <td>0.00</td>\n",
       "      <td>Absolute 0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>28.40</td>\n",
       "      <td>-0.999975</td>\n",
       "      <td>271.15</td>\n",
       "      <td>seawater freezes</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>32.00</td>\n",
       "      <td>0.991844</td>\n",
       "      <td>273.15</td>\n",
       "      <td>water freezes</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>68.00</td>\n",
       "      <td>21.125470</td>\n",
       "      <td>293.15</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>86.00</td>\n",
       "      <td>31.297913</td>\n",
       "      <td>303.15</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>122.00</td>\n",
       "      <td>51.224172</td>\n",
       "      <td>323.15</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>158.00</td>\n",
       "      <td>71.946198</td>\n",
       "      <td>343.15</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>194.00</td>\n",
       "      <td>90.998440</td>\n",
       "      <td>363.15</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>212.00</td>\n",
       "      <td>101.558368</td>\n",
       "      <td>373.15</td>\n",
       "      <td>water boils</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>1220.00</td>\n",
       "      <td>662.448526</td>\n",
       "      <td>933.15</td>\n",
       "      <td>aluminum freezes</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>1517.00</td>\n",
       "      <td>841.059703</td>\n",
       "      <td>1098.15</td>\n",
       "      <td>petroleum fire temperature</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>2732.00</td>\n",
       "      <td>1509.698180</td>\n",
       "      <td>1773.15</td>\n",
       "      <td>steel freezes</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>6920.00</td>\n",
       "      <td>3845.362345</td>\n",
       "      <td>4100.00</td>\n",
       "      <td>aluminum burns</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>27000032.00</td>\n",
       "      <td>14976271.585275</td>\n",
       "      <td>14999726.85</td>\n",
       "      <td>Sun</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              f                c            k                 description\n",
       "0       -459.67      -271.563974         0.00                  Absolute 0\n",
       "1         28.40        -0.999975       271.15            seawater freezes\n",
       "2         32.00         0.991844       273.15               water freezes\n",
       "3         68.00        21.125470       293.15                         NaN\n",
       "4         86.00        31.297913       303.15                         NaN\n",
       "5        122.00        51.224172       323.15                         NaN\n",
       "6        158.00        71.946198       343.15                         NaN\n",
       "7        194.00        90.998440       363.15                         NaN\n",
       "8        212.00       101.558368       373.15                 water boils\n",
       "9       1220.00       662.448526       933.15            aluminum freezes\n",
       "10      1517.00       841.059703      1098.15  petroleum fire temperature\n",
       "11      2732.00      1509.698180      1773.15               steel freezes\n",
       "12      6920.00      3845.362345      4100.00              aluminum burns\n",
       "13  27000032.00  14976271.585275  14999726.85                         Sun"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "scale_factor_error, bias = 0.01, 1.0\n",
    "meas = df.copy()\n",
    "meas.c = scale_factor_error * pd.np.random.randn(len(meas)) * (meas.c + bias) + meas.c + bias\n",
    "# err = (scale_factor_error * pd.np.randn(len(meas)) + 1) * meas.f\n",
    "meas"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>f</th>\n",
       "      <th>c</th>\n",
       "      <th>k</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>14.000000</td>\n",
       "      <td>14.000000</td>\n",
       "      <td>14.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>1929490.123571</td>\n",
       "      <td>1071921.192857</td>\n",
       "      <td>1072155.321429</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>7215798.572849</td>\n",
       "      <td>4008776.981159</td>\n",
       "      <td>4008630.976347</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>-459.670000</td>\n",
       "      <td>-273.150000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>72.500000</td>\n",
       "      <td>22.500000</td>\n",
       "      <td>295.650000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>176.000000</td>\n",
       "      <td>80.000000</td>\n",
       "      <td>353.150000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>1442.750000</td>\n",
       "      <td>783.750000</td>\n",
       "      <td>1056.900000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>27000032.000000</td>\n",
       "      <td>15000000.000000</td>\n",
       "      <td>14999726.850000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                     f                c                k\n",
       "count        14.000000        14.000000        14.000000\n",
       "mean    1929490.123571   1071921.192857   1072155.321429\n",
       "std     7215798.572849   4008776.981159   4008630.976347\n",
       "min        -459.670000      -273.150000         0.000000\n",
       "25%          72.500000        22.500000       295.650000\n",
       "50%         176.000000        80.000000       353.150000\n",
       "75%        1442.750000       783.750000      1056.900000\n",
       "max    27000032.000000  15000000.000000  14999726.850000"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "scale_factor_error, bias = 0.01, 1.0\n",
    "meas2 = df.copy()\n",
    "meas2.c = (1 + scale_factor_error) * meas.f\n",
    "meas2\n",
    "import numpy as np\n",
    "\n",
    "df?\n",
    "df.describe()\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# *Exercise 2 !!*\n",
    "Write a function to \"regress\" two sequences of values and return a slope and offset 2-tuple\n",
    "Which method discussed during the lecture should you use?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def fit_line(df):"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# *Exercise 3!!*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def add_error(df, scale_factor, bias, column=None):\n",
    "    if isinstance(column, int):\n",
    "        column = df.columns[column]\n",
    "    elif column is None or not column in df.columns:\n",
    "        column = list(df.columns)[-1]\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# *Bonus !!*\n",
    "Write a function to plot a scatter plot of points and\n",
    "draw a line for the linear regression slope and offset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def plot_regression(df):\n",
    "    # your code goes here\n",
    "    # plt.plot(x, y)  # you can do better than this\n",
    "    plt.title('Slope: {}, Offset: {}, Error: {}'.format(coeff[0], poly[1], error))\n",
    "    # plt.xlabel(\"some statistice or something interesting about x\")\n",
    "    # plt.ylabel(\"something interesting or some statistics about y\")\n",
    "    return error\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# *Superfreak Bonus!!*\n",
    "Read ahead about Norvig's spelling checker, or fuzzy string matching and build a fuzzy column label finder. You can use fuzzywuzzy or \"roll your own\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def find_label(df, label):\n",
    "    if label in df.columns:\n",
    "        return label\n",
    "    # your code here to try to find the closest match for a column"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}