scraping_form_submissions.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import requests\n",
    "from bs4 import BeautifulSoup"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Normal scraping\n",
    "\n",
    "By now we all know how to scrape normal sites (kind of, mostly, somewhat)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/lib/python3.4/site-packages/bs4/__init__.py:166: UserWarning: No parser was explicitly specified, so I'm using the best available HTML parser for this system (\"html.parser\"). This usually isn't a problem, but if you run this code on another system, or in a different virtual environment, it may use a different parser and behave differently.\n",
      "\n",
      "To get rid of this warning, change this:\n",
      "\n",
      " BeautifulSoup([your markup])\n",
      "\n",
      "to this:\n",
      "\n",
      " BeautifulSoup([your markup], \"html.parser\")\n",
      "\n",
      "  markup_type=markup_type))\n"
     ]
    }
   ],
   "source": [
    "# Grab the NYT's homepage\n",
    "response = requests.get(\"http://nytimes.com\")\n",
    "doc = BeautifulSoup(response.text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Snag all of the headlines (h3 tags with 'story-heading' class)\n",
    "headlines = doc.find_all(\"h3\", {'class': 'story-heading'})\n",
    "# Getting the headline text out using list comprehensions\n",
    "# is a lot more fun but I guess you just learned those\n",
    "# like a day ago, so we'll go ahead and use a for loop.\n",
    "# But for the curious:\n",
    "#   [headline.text.strip() for headline in headlines]\n",
    "\n",
    "# Print the text of the headlines\n",
    "for headline in headlines:\n",
    "    print(headline.)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# But... forms!\n",
    "\n",
    "So the issue is that sometimes you need to submit *forms* on a web site. Why? Well, let's look at an example.\n",
    "\n",
    "This example is going to come from [Dan Nguyen](https://twitter.com/dancow)'s incredible [Search, Script, Scrape](https://github.com/compjour/search-script-scrape), 101 scraping exercises.\n",
    "\n",
    "> The number of FDA-approved, but now discontinued drug products that contain Fentanyl as an active ingredient\n",
    ">\n",
    "> Related URL: [http://www.accessdata.fda.gov/scripts/cder/ob/docs/queryai.cfm](http://www.accessdata.fda.gov/scripts/cder/ob/docs/queryai.cfm)\n",
    "\n",
    "When you visit that URL, you're going to type in \"Fentanyl,\" and select \"Disc (Discontinued Drug Products).\" Then you'll hit **search**.\n",
    "\n",
    "Hooray, results! Now **look at the URL.**\n",
    "\n",
    "> http://www.accessdata.fda.gov/scripts/cder/ob/docs/tempai.cfm\n",
    "\n",
    "Does anything about that URL say \"Fentanyl\" or \"Discontinued Drug Products\"? Nope! And if you [straight up visit it](http://www.accessdata.fda.gov/scripts/cder/ob/docs/tempai.cfm) (might need to open an Incognito window) you'll end up being redirected back to a different page. \n",
    "\n",
    "This means **`requests.get` just isn't going to cut it.** If you tell `requests` to download that page it's going to get a whooole lot of uselessness.\n",
    "\n",
    "Be my guest if you want to try it!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "ename": "ImportError",
     "evalue": "No module named 'selenium'",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mImportError\u001b[0m                               Traceback (most recent call last)",
      "\u001b[1;32m<ipython-input-4-e7d8c2850744>\u001b[0m in \u001b[0;36m<module>\u001b[1;34m()\u001b[0m\n\u001b[1;32m----> 1\u001b[1;33m \u001b[1;32mfrom\u001b[0m \u001b[0mselenium\u001b[0m \u001b[1;32mimport\u001b[0m \u001b[0mwebdriver\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[1;31mImportError\u001b[0m: No module named 'selenium'"
     ]
    }
   ],
   "source": [
    "from selenium import webdriver"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Submitting forms with `requests`\n",
    "\n",
    "There are two kinds of forms, `GET` forms and `POST` forms (...this is 99% true).\n",
    "\n",
    "## `GET` forms\n",
    "\n",
    "A `GET` form is one where you can see parameters in the URL. For example, if you searched for images of animals surfing on Bing you'd end up here:\n",
    "\n",
    "> `http://www.bing.com/images/search?q=animals+surfing&FORM=HDRSC2`\n",
    "\n",
    "It has a couple parameters - `q` and `FORM`. `FORM` is some sort of weird analytics thing that doesn't affect the page, but `q` is definitely the term you're searching for. **With a `GET` form, the data you put into the form is kept in the URL.**\n",
    "\n",
    "Just for kicks, if we looked at the HTML for a `GET` form it might look like this:\n",
    "\n",
    "````html\n",
    "<form method=\"GET\" action=\"/search\">\n",
    "<input type=\"text\" name=\"q\">\n",
    "</form>\n",
    "````\n",
    "\n",
    "It might also leave the whole `method` part off, too - `GET` is the default.\n",
    "\n",
    "A fun part about `GET` forms you can share the URL to share the results. If you don't believe me, visit [http://www.bing.com/images/search?q=animals+surfing&FORM=HDRSC2](http://www.bing.com/images/search?q=animals+surfing&FORM=HDRSC2) to see animals surfing.\n",
    "\n",
    "`GET` is how most web pages work. You've used it every time you invoke the unholy powers of `requests.get`.\n",
    "\n",
    "````py\n",
    "requests.get(\"https://api.spotify.com/v1/search?query=90s&offset=20&limit=20&type=playlist\")\n",
    "````\n",
    "\n",
    "`GET` is nice. `GET` is easy. **But `GET` is not all there is.**\n",
    "\n",
    "## `POST` forms\n",
    "\n",
    "The other kind of forms are `POST` forms. `POST` forms are not friendly!\n",
    "\n",
    "Unlike `GET` forms, **you can't share the URL to get the same information.** The parameters - the `q` for your search, for example - aren't in the URL, they're hidden in *the actual request*.\n",
    "\n",
    "What this means is that **every time you request something from a `POST`-based form, you have to pretend you filled out the form and clicked the button.**\n",
    "\n",
    "### Grabbing the parameters\n",
    "\n",
    "First we need to find out what parameters we're going to hunt down. To do this, first make your way to the form, then **get prepared.**\n",
    "\n",
    "1) In Chrome, `View` > `Developer` > `Developer Tools`\n",
    "2) Click the **Network** tab\n",
    "3) Fill the form out, and submit it\n",
    "4) Scroll up to the top of the Network pane, select the segment of the URL you're at (I'm at `tempai.cfm`)\n",
    "5) Click it\n",
    "6) Select **Headers** on the right\n",
    "7) Scroll down until you see **Form Data**\n",
    "\n",
    "Okay, that seemed like a *lot of work,* but I promise it was actually simple and easy and you're living life in a grand grand way. Two parameters are listed for the search we're doing:\n",
    "\n",
    "````\n",
    "Generic_Name:Fentanyl\n",
    "table1:OB_Disc\n",
    "````\n",
    "\n",
    "Seems simple enough! Now let's put them to work.\n",
    "\n",
    "### Submitting `POST` forms with `requests.post`\n",
    "\n",
    "This is going to be *so easy* you might have a *heart attack* as a result of your body being *so amazed* that it doesn't have to do anything strenuous. All you have to do is\n",
    "\n",
    "````py\n",
    "requests.get(\"http://whatever.com/url/to/something\", { \"param1\": \"val1\", \"param2\": \"val2\" })\n",
    "````\n",
    "\n",
    "and treat it like a normal response! Here, I'll prove it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Just in case you didn't run it up there, I'll import again\n",
    "import requests\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Using .select instead of .find is a little more\n",
    "# readable to people from the web dev world, maybe?\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It's magic, I swear!\n",
    "\n",
    "# But then...\n",
    "\n",
    "Sometimes `requests.get` just isn't enough. Why? It mostly has to do with JavaScript or complicated forms - when a site reacts and changes without loading a new page, you can't use `requests` for that (think \"Load more\" buttons on Instagram).\n",
    "\n",
    "For those sites you need **Selenium!** Selenium = you put your browser on autopilot. As in, literally, it takes control over your browser. There are \"headless\" versions that use invisible browsers but if you don't like to install a bunch of stuff, the normal version is usually fine.\n",
    "\n",
    "## Installing Selenium\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "Selenium isn't just a Python package, but you'll need to install **python bindings** in order to have Python talk to Selenium.\n",
    "\n",
    "````\n",
    "pip install selenium\n",
    "````\n",
    "\n",
    "You'll also need the [Firefox browser](https://www.mozilla.org/en-US/firefox), since that's the browser we're going to be controlling.\n",
    "\n",
    "Selenium is built on **WebDrivers**, which are libraries that let you... drive a browser. I believe it comes with a Firefox WebDriver, whereas Safari/Chrome/etc take a little more effort to set up."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Using Selenium"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Collecting selenium\n",
      "  Downloading selenium-2.53.5-py2.py3-none-any.whl (884kB)\n",
      "\u001b[K    100% |████████████████████████████████| 890kB 284kB/s \n",
      "\u001b[?25hInstalling collected packages: selenium\n",
      "Successfully installed selenium-2.53.5\n"
     ]
    }
   ],
   "source": [
    "# Imports, of course\n",
    "!pip install selenium"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Initialize a Firefox webdriver\n",
    "from selenium import webdriver\n",
    "from selenium.webdriver.common.keys import Keys\n",
    "from selenium.webdriver.support.ui import Select\n",
    "from selenium.webdriver.support.ui import WebDriverWai\n",
    "\n",
    "driver = webdriver.Firefox()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Grab the web page\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# You'll use selenium.webdriver.support.ui.Select\n",
    "# that we imported above to grab the Seelct element called \n",
    "# t_web_lookup__license_type_name, then select Acupuncturists\n",
    "\n",
    "# We use .find_element_by_name here because we know the name"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# We use .find_element_by_id here because we know the id\n",
    "\n",
    "# Then we'll fake typing into it"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Now we can grab the search button and click it\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Instead of using requests.get, we just look at .page_source of the driver\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# We can feed that into Beautiful Soup\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# It's a tricky table, but this grabs the linked names inside of the A\n",
    "#rows = doc.select(\"#datagrid_results tr\")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Closing the webdriver\n",
    "\n",
    "Once we have all the data we want, we can close our webdriver."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Close the webdriver\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Saving our data\n",
    "\n",
    "Now what are we going to do with our list of dictionaries? We *could* use a `csv.DictWriter` like in [this post](http://stackoverflow.com/questions/3086973/how-do-i-convert-this-list-of-dictionaries-to-a-csv-file-python), but it's actually quicker to do it with `pandas`.\n",
    "\n",
    "### Step One: import pandas"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step Two: Turn list into a DataFrame"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step Three: Save it to a CSV\n",
    "\n",
    "While you're saving it, set `index=False` or else it will include `0`, `1`, `2`, etc from the further-left column (the index, of course)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step Four: Party down\n",
    "\n",
    "I don't have directions for this one"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.4.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}