homeworks/Homework-5/5.Hotels.ipynb

{
 "metadata": {
  "name": "",
  "signature": "sha256:d4287cb4c6ff42193e165cb4b6a99ecc81e4dfce9e3d2f23079851a8b76c5849"
 },
 "nbformat": 3,
 "nbformat_minor": 0,
 "worksheets": [
  {
   "cells": [
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "(In order to load the stylesheet of this notebook, execute the last code cell in this notebook)"
     ]
    },
    {
     "cell_type": "heading",
     "level": 1,
     "metadata": {},
     "source": [
      "Analyzing hotel ratings on Tripadvisor"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "In this homework we will focus on practicing two techniques: web scraping and regression. For the first part, we will build upon the sample code from the Lecture and attempt to get some basic information for each hotel. Then, we will fit a regression model on this information and try to analyze it.   "
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "One of the main disadvantages of scraping a website instead of using an API is that, without any notice, the website may change its layout and render our code useless. Something like that happened in our case. Tripadvisor changed the layout of the buttons that we use to navigate between the different pages of the results. This was the main reason people were having problem with executing the code."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "**Task 1 (20 pts)**\n",
      "\n",
      "The first task of the homework is to fix the scraping code. We basically need to replace the part where we are checking if there is another page and getting its link with new code that reflects the new navigation layout. "
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "** Task 2 (30 pts)**\n",
      "\n",
      "Then, for each hotel that our search returns, we will \"click\" (with the code of course) on it and scrape the information below.\n",
      "\n",
      "![Information to be scraped](hotel_info.png)\n",
      "\n",
      "Of course, feel free to collect even more data if you want. "
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "** Task 3 (20 pts) **\n",
      "\n",
      "Now, we will use regression to analyze this information. First, we will fit a linear regression model that predicts the average rating. For example, for the hotel above, the average rating is\n",
      "\n",
      "$$ \\text{AVG_SCORE} = \\frac{1*31 + 2*33 + 3*98 + 4*504 + 5*1861}{2527}$$\n",
      "\n",
      "Use the model to analyze the important factors that decide the $\\text{AVG_SCORE}$."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "** Task 4 (30 pts) **\n",
      "\n",
      "Finally, we will use logistic regression to decide if a hotel is _excellent_ or not. We classify a hotel as _excellent_ if more than **60%** of its ratings are 5 stars. This is a binary attribute on which we can fit a logistic regression model. As before, use the model to analyze the data.\n",
      "\n",
      "-------"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "In order to use code from a Python script file, we need to put that file in the same folder as the notebook and import it as a library. Then, we will be able to access it's functions. For example, in the case of the lecture code, we could do the following:\n",
      "\n",
      "``` python\n",
      "import scrape_solution as scrape\n",
      "\n",
      "scrape.get_city_page()\n",
      "```\n",
      "\n",
      "Of course, you might need to modify and restructure the code so that it returns what you need."
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "----"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [],
     "language": "python",
     "metadata": {},
     "outputs": []
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# Code for setting the style of the notebook\n",
      "from IPython.core.display import HTML\n",
      "def css_styling():\n",
      "    styles = open(\"../../theme/custom.css\", \"r\").read()\n",
      "    return HTML(styles)\n",
      "css_styling()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "html": [
        "<link href='http://fonts.googleapis.com/css?family=EB+Garamond' rel='stylesheet' type='text/css'>\n",
        "<link href='http://fonts.googleapis.com/css?family=Alegreya+Sans:100,300,400,500,700,800,900,100italic,300italic,400italic,500italic,700italic,800italic,900italic' rel='stylesheet' type='text/css'>\n",
        "<link href='http://fonts.googleapis.com/css?family=Source+Code+Pro:300,400' rel='stylesheet' type='text/css'>\n",
        "<style>\n",
        "    @font-face {\n",
        "        font-family: \"Computer Modern\";\n",
        "        src: url('http://mirrors.ctan.org/fonts/cm-unicode/fonts/otf/cmunss.otf');\n",
        "    }\n",
        "    .code_cell {\n",
        "        width: 105ex !important ;\n",
        "        margin-bottom: 15px !important;\n",
        "    }\n",
        "    div.cell {\n",
        "        margin-left: auto;\n",
        "        margin-right: auto;\n",
        "        width: 70%;\n",
        "    }    \n",
        "    div.cell.selected {\n",
        "        border: thin rgba(171, 171, 171, 0.5) dashed;\n",
        "    }\n",
        "    h1 {\n",
        "        font-family: 'Alegreya Sans', sans-serif;\n",
        "    }\n",
        "    h2 {\n",
        "        font-family: 'EB Garamond', serif;\n",
        "    }\n",
        "    h3 {\n",
        "        font-family: 'EB Garamond', serif;\n",
        "        margin-top:12px;\n",
        "        margin-bottom: 3px;\n",
        "    }\n",
        "    h4 {\n",
        "        font-family: 'EB Garamond', serif;\n",
        "    }\n",
        "    h5 {\n",
        "        font-family: 'Alegreya Sans', sans-serif;\n",
        "    }\n",
        "    div.text_cell_render {\n",
        "        font-family: 'EB Garamond',Computer Modern, \"Helvetica Neue\", Arial, Helvetica, Geneva, sans-serif;\n",
        "        line-height: 145%;\n",
        "        font-size: 140%;\n",
        "    }\n",
        "    div.input_area {\n",
        "        border-color: rgba(0,0,0,0.10) !important;\n",
        "        background: #fafafa;\n",
        "    }\n",
        "    .CodeMirror {\n",
        "            font-family: \"Source Code Pro\";\n",
        "            font-size: 90%;\n",
        "    }\n",
        "    .prompt {\n",
        "        display: None;\n",
        "    }\n",
        "    .output {\n",
        "        padding-left: 50px;\n",
        "        padding-top: 5px;\n",
        "    }\n",
        "    .output_wrapper {\n",
        "        padding-left: 5px;\n",
        "        padding-top: inherit;\n",
        "    }\n",
        "    div.output_scroll {\n",
        "        width: inherit;\n",
        "    }\n",
        "    .inner_cell {\n",
        "        padding-left: 5px;\n",
        "    }\n",
        "    .text_cell_render h1 {\n",
        "        font-weight: 200;\n",
        "        font-size: 50pt;\n",
        "        line-height: 100%;\n",
        "        color:#CD2305;\n",
        "        margin-bottom: 0.5em;\n",
        "        margin-top: 0.5em;\n",
        "        display: block;\n",
        "    }\n",
        "    .text_cell_render h5 {\n",
        "        font-weight: 300;\n",
        "        font-size: 16pt;\n",
        "        color: #CD2305;\n",
        "        font-style: italic;\n",
        "        margin-bottom: .5em;\n",
        "        margin-top: 0.5em;\n",
        "        display: block;\n",
        "    }\n",
        "    .warning {\n",
        "        color: rgb( 240, 20, 20 )\n",
        "        }  \n",
        "</style>\n",
        "<script>\n",
        "    MathJax.Hub.Config({\n",
        "                        TeX: {\n",
        "                           extensions: [\"AMSmath.js\"]\n",
        "                           },\n",
        "                tex2jax: {\n",
        "                    inlineMath: [ ['$','$'], [\"\\\\(\",\"\\\\)\"] ],\n",
        "                    displayMath: [ ['$$','$$'], [\"\\\\[\",\"\\\\]\"] ]\n",
        "                },\n",
        "                displayAlign: 'center', // Change this to 'center' to center equations.\n",
        "                \"HTML-CSS\": {\n",
        "                    styles: {'.MathJax_Display': {\"margin\": 4}}\n",
        "                }\n",
        "        });\n",
        "</script>"
       ],
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 2,
       "text": [
        "<IPython.core.display.HTML at 0x3a6e978>"
       ]
      }
     ],
     "prompt_number": 2
    }
   ],
   "metadata": {}
  }
 ]
}