Module-2/week-2/day_2_html_css_webscraping/scraping-enkeboll.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Python Web Scraping Tutorial using BeautifulSoup"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Recap: APIs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "import requests\n",
    "url = 'https://venmo.com/api/v5/public'\n",
    "resp = requests.get(url)\n",
    "\n",
    "assert resp.status_code == 200, \"error code!!!\"\n",
    "\n",
    "content = resp.json()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "dict_keys(['paging', 'data'])"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "content.keys()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "ename": "KeyError",
     "evalue": "'dataz'",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mKeyError\u001b[0m                                  Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-6-d95a90bb0cdb>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mcontent\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m'dataz'\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;31mKeyError\u001b[0m: 'dataz'"
     ]
    }
   ],
   "source": [
    "content['dataz']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "20"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(content.get('data', []))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "dict_keys(['payment_id', 'permalink', 'via', 'action_links', 'transactions', 'story_id', 'comments', 'updated_time', 'audience', 'actor', 'created_time', 'mentions', 'message', 'type', 'likes'])\n"
     ]
    }
   ],
   "source": [
    "print(content.get('data')[0].keys())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Star Wars swag\n",
      "👅\n",
      "Batteries\n",
      "⛽️❤️\n",
      "Thrive!\n",
      "👙\n",
      "🏠💸\n",
      "💦💸\n",
      "🔌💻\n",
      "👶\n",
      "Starter in Accord\n",
      "🐱 🎉\n",
      "Cabana\n",
      "Makeup\n",
      "Omg stahp lol\n",
      "📷👶\n",
      "Slop\n",
      "💇\n",
      "🍊\n",
      "⚽️\n"
     ]
    }
   ],
   "source": [
    "print(*[x['message'] for x in content.get('data')], sep='\\n')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "str"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type(content.get('data')[1].get('message'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The components of a web page\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When we visit a web page, our web browser makes a GET request to a web server. The server then sends back files that tell our browser how to render the page for us. The files fall into a few main types:\n",
    "\n",
    "- HTML — contain the main content of the page.\n",
    "- CSS — add styling to make the page look nicer.\n",
    "- JS — Javascript files add interactivity to web pages.\n",
    "- Images — image formats, such as JPG and PNG allow web pages to show pictures.\n",
    "\n",
    "After our browser receives all the files, it renders the page and displays it to us. There’s a lot that happens behind the scenes to render a page nicely, but we don’t need to worry about most of it when we’re web scraping."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### HTML\n",
    "\n",
    "\n",
    "HyperText Markup Language (HTML) is a language that web pages are created in. HTML isn’t a programming language, like Python — instead, it’s a markup language that tells a browser how to layout content. \n",
    "\n",
    "Let’s take a quick tour through HTML so we know enough to scrape effectively. HTML consists of elements called tags. The most basic tag is the `<html>` tag. This tag tells the web browser that everything inside of it is HTML. We can make a simple HTML document just using this tag:\n",
    "\n",
    "~~~\n",
    "<html>\n",
    "</html>\n",
    "~~~\n",
    "&#x2B07; Example cell with html in Markdown!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<html>\n",
    "</html>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Right inside an html tag, we put two other tags, the head tag, and the body tag. The main content of the web page goes into the body tag. The head tag contains data about the title of the page, and other information that generally isn’t useful in web scraping:\n",
    "\n",
    "~~~\n",
    "<html>\n",
    "    <head>\n",
    "    </head>\n",
    "    <body>\n",
    "    </body>\n",
    "</html>\n",
    "~~~"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We’ll now add our first content to the page, in the form of the p tag. The p tag defines a paragraph, and any text inside the tag is shown as a separate paragraph:\n",
    "~~~\n",
    "<html>\n",
    "    <head>\n",
    "    </head>\n",
    "    <body>\n",
    "          <p>\n",
    "            Here's a paragraph of text!\n",
    "        </p>\n",
    "        <p>\n",
    "            Here's a second paragraph of text!\n",
    "        </p>\n",
    "    </body>\n",
    "</html>\n",
    "~~~"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<html>\n",
    "    <head>\n",
    "    </head>\n",
    "    <body>\n",
    "          <p>\n",
    "            Here's a paragraph of text!\n",
    "        </p>\n",
    "        <p>\n",
    "            Here's a second paragraph of text!\n",
    "        </p>\n",
    "    </body>\n",
    "</html>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Tags have commonly used names that depend on their position in relation to other tags:\n",
    "\n",
    "- **child** — a child is a tag inside another tag. So the two p tags above are both children of the body tag.\n",
    "- **parent** — a parent is the tag another tag is inside. Above, the html tag is the parent of the body tag.\n",
    "- **sibiling** — a sibiling is a tag that is nested inside the same parent as another tag. For example, head and body are siblings, since they’re both inside html. Both p tags are siblings, since they’re both inside body."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also add properties to HTML tags that change their behavior:\n",
    "\n",
    "~~~\n",
    "<html>\n",
    "  <head></head>\n",
    "  <body>\n",
    "    <p>\n",
    "      Here's a paragraph of text!\n",
    "      <a href=\"https://www.dataquest.io\">Learn Data Science Online</a>\n",
    "    </p>\n",
    "    <p>\n",
    "      Here's a second paragraph of text!\n",
    "      <a href=\"https://www.python.org\">Python</a>        \n",
    "    </p>\n",
    "  </body>\n",
    "</html>\n",
    "~~~"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<html>\n",
    "    <head>\n",
    "    </head>\n",
    "    <body>\n",
    "        <p>\n",
    "            Here's a paragraph of text!\n",
    "            <a href=\"https://www.dataquest.io\">Learn Data Science Online</a>\n",
    "        </p>\n",
    "        <p>\n",
    "            Here's a second paragraph of text!\n",
    "            <a href=\"https://www.python.org\">Python</a>        </p>\n",
    "    </body></html>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the above example, we added two a tags. a tags are links, and tell the browser to render a link to another web page. The href property of the tag determines where the link goes.\n",
    "\n",
    "a and p are extremely common html tags. Here are a few others:\n",
    "\n",
    "- *div* — indicates a division, or area, of the page.\n",
    "- *b* — bolds any text inside.\n",
    "- *i* — italicizes any text inside.\n",
    "- *table* — creates a table.\n",
    "- *form* — creates an input form.\n",
    "\n",
    "\n",
    "For a full list of tags, look [here](https://developer.mozilla.org/en-US/docs/Web/HTML/Element)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are two special properties give HTML elements names, and make them easier to interact with when we’re scraping, **class** and **id** properties. \n",
    "\n",
    "- One element can have multiple classes, and a class can be shared between elements. \n",
    "- Each element can only have one id, and an id can only be used once on a page. \n",
    "- Classes and ids are optional, and not all elements will have them.\n",
    "\n",
    "We can add classes and ids to our example:\n",
    "\n",
    "~~~\n",
    "<html>\n",
    "    <head>\n",
    "    </head>\n",
    "    <body>\n",
    "        <p class=\"bold-paragraph\">\n",
    "            Here's a paragraph of text!\n",
    "            <a href=\"https://www.dataquest.io\" id=\"learn-link\">Learn Data Science Online</a>\n",
    "        </p>\n",
    "        <p class=\"bold-paragraph extra-large\">\n",
    "            Here's a second paragraph of text!\n",
    "            <a href=\"https://www.python.org\" class=\"extra-large\">Python</a>\n",
    "        </p>\n",
    "    </body>\n",
    "</html>\n",
    "~~~"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<html>\n",
    "    <head>\n",
    "    </head>\n",
    "    <body>\n",
    "        <p class=\"bold-paragraph\">\n",
    "            Here's a paragraph of text!\n",
    "            <a href=\"https://www.dataquest.io\" id=\"learn-link\">Learn Data Science Online</a>\n",
    "        </p>\n",
    "        <p class=\"bold-paragraph extra-large\">\n",
    "            Here's a second paragraph of text!\n",
    "            <a href=\"https://www.python.org\" class=\"extra-large\">Python</a>\n",
    "        </p>\n",
    "    </body>\n",
    "</html>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Apply this knowledge\n",
    "\n",
    "Let's go to a webpage and change some of the HTML to see what happens. \n",
    "\n",
    "- Open up a webpage in Chrome.\n",
    "- Right click and 'inspect element.\n",
    "\n",
    "- Change the text in a tag.\n",
    "- Change the class of a tag.\n",
    "- Find a tag with an ID and remove it.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Webscraping with Python"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### The requests library\n",
    "The first thing we’ll need to do to scrape a web page is to download the page. We can download pages using the Python requests library (similar to interacting with APIs!)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<Response [200]>"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import requests\n",
    "page = requests.get(\"http://dataquestio.github.io/web-scraping-pages/simple.html\")\n",
    "page"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After running our request, we get a Response object. This object has a status_code property, which indicates if the page was downloaded successfully:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "200"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "page.status_code\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can print out the HTML content of the page using the content property:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<!DOCTYPE html>\n",
      "<html>\n",
      "    <head>\n",
      "        <title>A simple example page</title>\n",
      "    </head>\n",
      "    <body>\n",
      "        <p>Here is some simple content for this page.</p>\n",
      "    </body>\n",
      "</html>\n"
     ]
    }
   ],
   "source": [
    "print(page.text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Parsing a page with BeautifulSoup\n",
    "\n",
    "We can use the BeautifulSoup library to parse this document, and extract the text from the `<p>` tag. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<!DOCTYPE html>\n",
       "<html>\n",
       "<head>\n",
       "<title>A simple example page</title>\n",
       "</head>\n",
       "<body>\n",
       "<p>Here is some simple content for this page.</p>\n",
       "</body>\n",
       "</html>"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from bs4 import BeautifulSoup\n",
    "soup = BeautifulSoup(page.content)\n",
    "soup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "bs4.BeautifulSoup"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type(soup)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<!DOCTYPE html>\n",
      "<html>\n",
      " <head>\n",
      "  <title>\n",
      "   A simple example page\n",
      "  </title>\n",
      " </head>\n",
      " <body>\n",
      "  <p>\n",
      "   Here is some simple content for this page.\n",
      "  </p>\n",
      " </body>\n",
      "</html>\n"
     ]
    }
   ],
   "source": [
    "print(soup.prettify())\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As all the tags are nested, we can move through the structure one level at a time. We can first select all the elements at the top level of the page using the `children` property of `soup`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['html', <html>\n",
       " <head>\n",
       " <title>A simple example page</title>\n",
       " </head>\n",
       " <body>\n",
       " <p>Here is some simple content for this page.</p>\n",
       " </body>\n",
       " </html>]"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "list(soup.children)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The above tells us that there are two tags at the top level of the page — the initial `<!DOCTYPE html>` tag, and the `<html>` tag. There is a newline character `(\\n)` in the list as well. Let’s see what the type of each element in the list is:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[bs4.element.Doctype, bs4.element.Tag]"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "[type(item) for item in list(soup.children)]\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `Tag` object allows us to navigate through an HTML document, and extract other tags and text. You can learn more about the various `BeautifulSoup` objects [here](https://www.crummy.com/software/BeautifulSoup/bs4/doc/#kinds-of-objects).\n",
    "\n",
    "We can now select the html tag and its children by taking the second item in the list:\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<html>\n",
       "<head>\n",
       "<title>A simple example page</title>\n",
       "</head>\n",
       "<body>\n",
       "<p>Here is some simple content for this page.</p>\n",
       "</body>\n",
       "</html>"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "html = list(soup.children)[1]\n",
    "html"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Each item in the list returned by the `children` property is also a `BeautifulSoup` object, so we can also call the `children` method on `html`.\n",
    "\n",
    "Now, we can find the `children` inside the `html` tag:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['\\n', <head>\n",
       " <title>A simple example page</title>\n",
       " </head>, '\\n', <body>\n",
       " <p>Here is some simple content for this page.</p>\n",
       " </body>, '\\n']"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "list(html.children)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you can see above, there are two tags here, `head`, and `body`. We want to extract the text inside the `p` tag, so we’ll dive into the body:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<body>\n",
       "<p>Here is some simple content for this page.</p>\n",
       "</body>"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "body = list(html.children)[3]\n",
    "body\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can now isolate the `p` tag:\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<p>Here is some simple content for this page.</p>"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "p = list(body.children)[1]\n",
    "p"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once we’ve isolated the tag, we can use the `get_text` method to extract all of the text inside the tag:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Here is some simple content for this page.'"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "p.get_text()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Finding all instances of a tag at once\n",
    "\n",
    "If we want to extract a single tag, we can instead use the find_all method, which will find all the instances of a tag on a page."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<p>Here is some simple content for this page.</p>]"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "soup = BeautifulSoup(page.content)\n",
    "soup.find_all('p')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that `find_all` returns a list, so we’ll have to loop through, or use list indexing, it to extract text:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Here is some simple content for this page.'"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "soup.find_all('p')[0].get_text()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you instead only want to find the first instance of a tag, you can use the `find` method, which will return a single `BeautifulSoup` object:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<p>Here is some simple content for this page.</p>"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "soup.find('p')\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Searching for tags by class and id\n",
    "\n",
    "\n",
    "We introduced classes and ids earlier, but it probably wasn’t clear why they were useful. Classes and ids are used by CSS to determine which HTML elements to apply certain styles to. We can also use them when scraping to specify specific elements we want to scrape. Let's look at the following page:\n",
    "\n",
    "~~~\n",
    "<html>\n",
    "    <head>\n",
    "        <title>A simple example page</title>\n",
    "    </head>\n",
    "    <body>\n",
    "        <div>\n",
    "            <p class=\"inner-text first-item\" id=\"first\">\n",
    "                First paragraph.\n",
    "            </p>\n",
    "            <p class=\"inner-text\">\n",
    "                Second paragraph.\n",
    "            </p>\n",
    "        </div>\n",
    "        <p class=\"outer-text first-item\" id=\"second\">\n",
    "            <b>\n",
    "                First outer paragraph.\n",
    "            </b>\n",
    "        </p>\n",
    "        <p class=\"outer-text\">\n",
    "            <b>\n",
    "                Second outer paragraph.\n",
    "            </b>\n",
    "        </p>\n",
    "    </body>\n",
    "</html>\n",
    "~~~"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<html>\n",
       "<head>\n",
       "<title>A simple example page</title>\n",
       "</head>\n",
       "<body>\n",
       "<div>\n",
       "<p class=\"inner-text first-item\" id=\"first\">\n",
       "                First paragraph.\n",
       "            </p>\n",
       "<p class=\"inner-text\">\n",
       "                Second paragraph.\n",
       "            </p>\n",
       "</div>\n",
       "<p class=\"outer-text first-item\" id=\"second\">\n",
       "<b>\n",
       "                First outer paragraph.\n",
       "            </b>\n",
       "</p>\n",
       "<p class=\"outer-text\">\n",
       "<b>\n",
       "                Second outer paragraph.\n",
       "            </b>\n",
       "</p>\n",
       "</body>\n",
       "</html>"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "page = requests.get(\"http://dataquestio.github.io/web-scraping-pages/ids_and_classes.html\")\n",
    "soup = BeautifulSoup(page.content)\n",
    "soup"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we can use the `find_all` method to search for items by class or by id. In the below example, we’ll search for any `p` tag that has the class `outer-text`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<p class=\"inner-text first-item\" id=\"first\">\n",
       "                 First paragraph.\n",
       "             </p>, <p class=\"outer-text first-item\" id=\"second\">\n",
       " <b>\n",
       "                 First outer paragraph.\n",
       "             </b>\n",
       " </p>]"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "soup.find_all('p', class_='first-item')\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the below example, we’ll look for any tag that has the class `outer-text`:\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "soup.find_all(class_=\"outer-text\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also search for elements by `id`:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<p class=\"inner-text first-item\" id=\"first\">\n",
       "                First paragraph.\n",
       "            </p>"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "soup.find(id=\"first\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Using CSS Selectors\n",
    "\n",
    "\n",
    "You can also search for items using CSS selectors. These selectors are how the CSS language allows developers to specify HTML tags to style. Here are some examples:\n",
    "\n",
    "- *p a* — finds all a tags inside of a p tag.\n",
    "- *body p a* — finds all a tags inside of a p tag inside of a body tag.\n",
    "- *html body* — finds all body tags inside of an html tag.\n",
    "- *p.outer-text* — finds all p tags with a class of outer-text.\n",
    "- *p#first* — finds all p tags with an id of first.\n",
    "- *body p.outer-text* — finds any p tags with a class of outer-text inside of a body tag."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`BeautifulSoup` objects support searching a page via CSS selectors using the `select` method. We can use CSS selectors to find all the `p` tags in our page that are inside of a `div` like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<p class=\"inner-text first-item\" id=\"first\">\n",
       "                 First paragraph.\n",
       "             </p>, <p class=\"inner-text\">\n",
       "                 Second paragraph.\n",
       "             </p>]"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "soup.select(\"div p\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that the `select` method above returns a list of `BeautifulSoup` objects, just like `find` and `find_all`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Your Turn: Downloading weather data\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We now know enough to proceed with extracting information about the local weather from the National Weather Service website. The first step is to find the page we want to scrape. We’ll extract weather information about downtown DC from this [page](http://forecast.weather.gov/MapClick.php?lat=38.8977&lon=-77.0365)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can then scroll up in the elements panel to find the “outermost” element that contains all of the text that corresponds to the extended forecasts. In this case, it’s a `div` tag with the id `seven-day-forecast`:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We now know enough to download the page and start parsing it. Below let's implement the following:\n",
    "\n",
    "1. Download the web page containing the forecast.\n",
    "2. Create a `BeautifulSoup` class to parse the page.\n",
    "3. Find the `div` with id `seven-day-forecast`, and assign to `seven_day`\n",
    "4. Inside `seven_day`, find each individual forecast item.\n",
    "5. Extract and print the first forecast item."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "200\n"
     ]
    }
   ],
   "source": [
    "#your code here\n",
    "url = 'http://forecast.weather.gov/MapClick.php?lat=38.8977&lon=-77.0365'\n",
    "resp = requests.get(url)\n",
    "print(resp.status_code)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [],
   "source": [
    "soup = BeautifulSoup(resp.text)\n",
    "seven_day = soup.find(id='seven-day-forecast')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "9"
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "forecast = seven_day.find_all(class_='forecast-tombstone')\n",
    "len(forecast)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ScatteredT-storms\n",
      "ScatteredT-storms thenSlight ChanceShowers\n",
      "Partly Sunnythen T-stormsLikely\n",
      "T-stormsLikely thenChanceShowers\n",
      "Mostly Sunny\n",
      "Partly Cloudythen SlightChanceShowers\n",
      "Partly Sunnythen ChanceShowers\n",
      "Mostly Clear\n",
      "Sunny\n"
     ]
    }
   ],
   "source": [
    "print(*[x.get_text() for f in forecast for x in f.find_all(class_='short-desc')], sep='\\n')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Extracting information from the page\n",
    "As you can see, inside the forecast item `tonight` is all the information we want. There are `4` pieces of information we can extract:\n",
    "\n",
    "- The name of the forecast item — in this case, **Tonight**.\n",
    "- The description of the conditions — this is stored in the `title` property of `img`.\n",
    "- A short description of the conditions — in this case, **Mostly Clear**.\n",
    "- The temperature low — in this case, **49 degrees**.\n",
    "\n",
    "We’ll extract the name of the forecast item, the short description, and the temperature first, since they’re all similar:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we can extract the `title` attribute from the `img` tag. To do this, we just treat the `BeautifulSoup` object like a dictionary, and pass in the attribute we want as a key:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "This Afternoon: Scattered showers and thunderstorms, mainly after 5pm.  Partly sunny, with a high near 90. South wind around 8 mph.  Chance of precipitation is 30%.\n",
      "Tonight: Scattered showers and thunderstorms before 11pm, then a slight chance of showers between 11pm and 2am.  Mostly cloudy, with a low around 74. South wind around 7 mph.  Chance of precipitation is 40%.\n",
      "Wednesday: Showers and thunderstorms likely, mainly after 3pm.  Partly sunny, with a high near 91. South wind 5 to 10 mph.  Chance of precipitation is 60%. New rainfall amounts between a quarter and half of an inch possible. \n",
      "Wednesday Night: Showers and thunderstorms likely before 2am, then a slight chance of showers.  Mostly cloudy, with a low around 71. Southwest wind 5 to 7 mph becoming calm  after midnight.  Chance of precipitation is 60%. New rainfall amounts between a tenth and quarter of an inch, except higher amounts possible in thunderstorms. \n",
      "Thursday: Mostly sunny, with a high near 91. West wind 3 to 7 mph. \n",
      "Thursday Night: A slight chance of showers after 2am.  Partly cloudy, with a low around 73. Chance of precipitation is 20%.\n",
      "Friday: A chance of showers after noon.  Partly sunny, with a high near 89. Chance of precipitation is 30%.\n",
      "Friday Night: Mostly clear, with a low around 67.\n",
      "Saturday: Sunny, with a high near 87.\n"
     ]
    }
   ],
   "source": [
    "print(*[x['title'] for f in forecast for x in f.find_all('img')], sep='\\n')\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Extracting all the information from the page\n",
    "Now that we know how to extract each individual piece of information, we can combine our knowledge with css selectors and list comprehensions to extract everything at once.\n",
    "\n",
    "In the below code, we:\n",
    "\n",
    "- Select all items with the class period-name inside an item with the class tombstone-container in seven_day.\n",
    "- Use a list comprehension to call the get_text method on each BeautifulSoup object."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['ThisAfternoon',\n",
       " 'Tonight',\n",
       " 'Wednesday',\n",
       " 'WednesdayNight',\n",
       " 'Thursday',\n",
       " 'ThursdayNight',\n",
       " 'Friday',\n",
       " 'FridayNight',\n",
       " 'Saturday']"
      ]
     },
     "execution_count": 64,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "period_tags = seven_day.select(\".tombstone-container .period-name\")\n",
    "periods = [pt.get_text() for pt in period_tags]\n",
    "periods"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you can see above, our technique gets us each of the period names, in order. Now you can apply the same technique to get the other 3 fields:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['ScatteredT-storms', 'ScatteredT-storms thenSlight ChanceShowers', 'Partly Sunnythen T-stormsLikely', 'T-stormsLikely thenChanceShowers', 'Mostly Sunny', 'Partly Cloudythen SlightChanceShowers', 'Partly Sunnythen ChanceShowers', 'Mostly Clear', 'Sunny']\n",
      "['High: 90 °F', 'Low: 74 °F', 'High: 91 °F', 'Low: 71 °F', 'High: 91 °F', 'Low: 73 °F', 'High: 89 °F', 'Low: 67 °F', 'High: 87 °F']\n",
      "['This Afternoon: Scattered showers and thunderstorms, mainly after 5pm.  Partly sunny, with a high near 90. South wind around 8 mph.  Chance of precipitation is 30%.', 'Tonight: Scattered showers and thunderstorms before 11pm, then a slight chance of showers between 11pm and 2am.  Mostly cloudy, with a low around 74. South wind around 7 mph.  Chance of precipitation is 40%.', 'Wednesday: Showers and thunderstorms likely, mainly after 3pm.  Partly sunny, with a high near 91. South wind 5 to 10 mph.  Chance of precipitation is 60%. New rainfall amounts between a quarter and half of an inch possible. ', 'Wednesday Night: Showers and thunderstorms likely before 2am, then a slight chance of showers.  Mostly cloudy, with a low around 71. Southwest wind 5 to 7 mph becoming calm  after midnight.  Chance of precipitation is 60%. New rainfall amounts between a tenth and quarter of an inch, except higher amounts possible in thunderstorms. ', 'Thursday: Mostly sunny, with a high near 91. West wind 3 to 7 mph. ', 'Thursday Night: A slight chance of showers after 2am.  Partly cloudy, with a low around 73. Chance of precipitation is 20%.', 'Friday: A chance of showers after noon.  Partly sunny, with a high near 89. Chance of precipitation is 30%.', 'Friday Night: Mostly clear, with a low around 67.', 'Saturday: Sunny, with a high near 87.']\n"
     ]
    }
   ],
   "source": [
    "short_descs = [sd.get_text()for sd in seven_day.select(\".tombstone-container .short-desc\")]\n",
    "temps = [t.get_text() for t in seven_day.select(\".tombstone-container .temp\")]\n",
    "descs = [d[\"title\"] for d in seven_day.select(\".tombstone-container img\")]\n",
    "\n",
    "print(short_descs)\n",
    "print(temps)\n",
    "print(descs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Combining our data into a Pandas Dataframe\n",
    "\n",
    "We can now combine the data into a Pandas DataFrame and analyze it.\n",
    "\n",
    "In order to do this, we’ll call the DataFrame class, and pass in each list of items that we have. We pass them in as part of a dictionary. Each dictionary key will become a column in the DataFrame, and each list will become the values in the column:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>period</th>\n",
       "      <th>short_desc</th>\n",
       "      <th>temp</th>\n",
       "      <th>desc</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>ThisAfternoon</td>\n",
       "      <td>ScatteredT-storms</td>\n",
       "      <td>High: 90 °F</td>\n",
       "      <td>This Afternoon: Scattered showers and thunders...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Tonight</td>\n",
       "      <td>ScatteredT-storms thenSlight ChanceShowers</td>\n",
       "      <td>Low: 74 °F</td>\n",
       "      <td>Tonight: Scattered showers and thunderstorms b...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Wednesday</td>\n",
       "      <td>Partly Sunnythen T-stormsLikely</td>\n",
       "      <td>High: 91 °F</td>\n",
       "      <td>Wednesday: Showers and thunderstorms likely, m...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>WednesdayNight</td>\n",
       "      <td>T-stormsLikely thenChanceShowers</td>\n",
       "      <td>Low: 71 °F</td>\n",
       "      <td>Wednesday Night: Showers and thunderstorms lik...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Thursday</td>\n",
       "      <td>Mostly Sunny</td>\n",
       "      <td>High: 91 °F</td>\n",
       "      <td>Thursday: Mostly sunny, with a high near 91. W...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>ThursdayNight</td>\n",
       "      <td>Partly Cloudythen SlightChanceShowers</td>\n",
       "      <td>Low: 73 °F</td>\n",
       "      <td>Thursday Night: A slight chance of showers aft...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Friday</td>\n",
       "      <td>Partly Sunnythen ChanceShowers</td>\n",
       "      <td>High: 89 °F</td>\n",
       "      <td>Friday: A chance of showers after noon.  Partl...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>FridayNight</td>\n",
       "      <td>Mostly Clear</td>\n",
       "      <td>Low: 67 °F</td>\n",
       "      <td>Friday Night: Mostly clear, with a low around 67.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>Saturday</td>\n",
       "      <td>Sunny</td>\n",
       "      <td>High: 87 °F</td>\n",
       "      <td>Saturday: Sunny, with a high near 87.</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           period                                  short_desc         temp  \\\n",
       "0   ThisAfternoon                           ScatteredT-storms  High: 90 °F   \n",
       "1         Tonight  ScatteredT-storms thenSlight ChanceShowers   Low: 74 °F   \n",
       "2       Wednesday             Partly Sunnythen T-stormsLikely  High: 91 °F   \n",
       "3  WednesdayNight            T-stormsLikely thenChanceShowers   Low: 71 °F   \n",
       "4        Thursday                                Mostly Sunny  High: 91 °F   \n",
       "5   ThursdayNight       Partly Cloudythen SlightChanceShowers   Low: 73 °F   \n",
       "6          Friday              Partly Sunnythen ChanceShowers  High: 89 °F   \n",
       "7     FridayNight                                Mostly Clear   Low: 67 °F   \n",
       "8        Saturday                                       Sunny  High: 87 °F   \n",
       "\n",
       "                                                desc  \n",
       "0  This Afternoon: Scattered showers and thunders...  \n",
       "1  Tonight: Scattered showers and thunderstorms b...  \n",
       "2  Wednesday: Showers and thunderstorms likely, m...  \n",
       "3  Wednesday Night: Showers and thunderstorms lik...  \n",
       "4  Thursday: Mostly sunny, with a high near 91. W...  \n",
       "5  Thursday Night: A slight chance of showers aft...  \n",
       "6  Friday: A chance of showers after noon.  Partl...  \n",
       "7  Friday Night: Mostly clear, with a low around 67.  \n",
       "8              Saturday: Sunny, with a high near 87.  "
      ]
     },
     "execution_count": 66,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "weather = pd.DataFrame({\n",
    "    \"period\": periods,\n",
    "    \"short_desc\": short_descs,\n",
    "    \"temp\": temps,\n",
    "    \"desc\": descs\n",
    "})\n",
    "weather"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can now do some analysis on the data. For example, we can use a regular expression and the Series.str.extract method to pull out the numeric temperature values:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    90\n",
       "1    74\n",
       "2    91\n",
       "3    71\n",
       "4    91\n",
       "5    73\n",
       "6    89\n",
       "7    67\n",
       "8    87\n",
       "Name: temp_num, dtype: object"
      ]
     },
     "execution_count": 67,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "temp_nums = weather[\"temp\"].str.extract(\"(?P<temp_num>\\d+)\", expand=False)\n",
    "weather[\"temp_num\"] = temp_nums.astype('int')\n",
    "temp_nums"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>period</th>\n",
       "      <th>short_desc</th>\n",
       "      <th>temp</th>\n",
       "      <th>desc</th>\n",
       "      <th>temp_num</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>ThisAfternoon</td>\n",
       "      <td>ScatteredT-storms</td>\n",
       "      <td>High: 90 °F</td>\n",
       "      <td>This Afternoon: Scattered showers and thunders...</td>\n",
       "      <td>90</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Tonight</td>\n",
       "      <td>ScatteredT-storms thenSlight ChanceShowers</td>\n",
       "      <td>Low: 74 °F</td>\n",
       "      <td>Tonight: Scattered showers and thunderstorms b...</td>\n",
       "      <td>74</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Wednesday</td>\n",
       "      <td>Partly Sunnythen T-stormsLikely</td>\n",
       "      <td>High: 91 °F</td>\n",
       "      <td>Wednesday: Showers and thunderstorms likely, m...</td>\n",
       "      <td>91</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>WednesdayNight</td>\n",
       "      <td>T-stormsLikely thenChanceShowers</td>\n",
       "      <td>Low: 71 °F</td>\n",
       "      <td>Wednesday Night: Showers and thunderstorms lik...</td>\n",
       "      <td>71</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Thursday</td>\n",
       "      <td>Mostly Sunny</td>\n",
       "      <td>High: 91 °F</td>\n",
       "      <td>Thursday: Mostly sunny, with a high near 91. W...</td>\n",
       "      <td>91</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>ThursdayNight</td>\n",
       "      <td>Partly Cloudythen SlightChanceShowers</td>\n",
       "      <td>Low: 73 °F</td>\n",
       "      <td>Thursday Night: A slight chance of showers aft...</td>\n",
       "      <td>73</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Friday</td>\n",
       "      <td>Partly Sunnythen ChanceShowers</td>\n",
       "      <td>High: 89 °F</td>\n",
       "      <td>Friday: A chance of showers after noon.  Partl...</td>\n",
       "      <td>89</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>FridayNight</td>\n",
       "      <td>Mostly Clear</td>\n",
       "      <td>Low: 67 °F</td>\n",
       "      <td>Friday Night: Mostly clear, with a low around 67.</td>\n",
       "      <td>67</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>Saturday</td>\n",
       "      <td>Sunny</td>\n",
       "      <td>High: 87 °F</td>\n",
       "      <td>Saturday: Sunny, with a high near 87.</td>\n",
       "      <td>87</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           period                                  short_desc         temp  \\\n",
       "0   ThisAfternoon                           ScatteredT-storms  High: 90 °F   \n",
       "1         Tonight  ScatteredT-storms thenSlight ChanceShowers   Low: 74 °F   \n",
       "2       Wednesday             Partly Sunnythen T-stormsLikely  High: 91 °F   \n",
       "3  WednesdayNight            T-stormsLikely thenChanceShowers   Low: 71 °F   \n",
       "4        Thursday                                Mostly Sunny  High: 91 °F   \n",
       "5   ThursdayNight       Partly Cloudythen SlightChanceShowers   Low: 73 °F   \n",
       "6          Friday              Partly Sunnythen ChanceShowers  High: 89 °F   \n",
       "7     FridayNight                                Mostly Clear   Low: 67 °F   \n",
       "8        Saturday                                       Sunny  High: 87 °F   \n",
       "\n",
       "                                                desc  temp_num  \n",
       "0  This Afternoon: Scattered showers and thunders...        90  \n",
       "1  Tonight: Scattered showers and thunderstorms b...        74  \n",
       "2  Wednesday: Showers and thunderstorms likely, m...        91  \n",
       "3  Wednesday Night: Showers and thunderstorms lik...        71  \n",
       "4  Thursday: Mostly sunny, with a high near 91. W...        91  \n",
       "5  Thursday Night: A slight chance of showers aft...        73  \n",
       "6  Friday: A chance of showers after noon.  Partl...        89  \n",
       "7  Friday Night: Mostly clear, with a low around 67.        67  \n",
       "8              Saturday: Sunny, with a high near 87.        87  "
      ]
     },
     "execution_count": 68,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "weather"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We could then find the mean of all the high and low temperatures:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "81.44444444444444"
      ]
     },
     "execution_count": 69,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "weather[\"temp_num\"].mean()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We could also only select the rows that happen at night:\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    False\n",
       "1     True\n",
       "2    False\n",
       "3     True\n",
       "4    False\n",
       "5     True\n",
       "6    False\n",
       "7     True\n",
       "8    False\n",
       "Name: temp, dtype: bool"
      ]
     },
     "execution_count": 70,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "is_night = weather[\"temp\"].str.contains(\"Low\")\n",
    "weather[\"is_night\"] = is_night\n",
    "is_night"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>period</th>\n",
       "      <th>short_desc</th>\n",
       "      <th>temp</th>\n",
       "      <th>desc</th>\n",
       "      <th>temp_num</th>\n",
       "      <th>is_night</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Tonight</td>\n",
       "      <td>ScatteredT-storms thenSlight ChanceShowers</td>\n",
       "      <td>Low: 74 °F</td>\n",
       "      <td>Tonight: Scattered showers and thunderstorms b...</td>\n",
       "      <td>74</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>WednesdayNight</td>\n",
       "      <td>T-stormsLikely thenChanceShowers</td>\n",
       "      <td>Low: 71 °F</td>\n",
       "      <td>Wednesday Night: Showers and thunderstorms lik...</td>\n",
       "      <td>71</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>ThursdayNight</td>\n",
       "      <td>Partly Cloudythen SlightChanceShowers</td>\n",
       "      <td>Low: 73 °F</td>\n",
       "      <td>Thursday Night: A slight chance of showers aft...</td>\n",
       "      <td>73</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>FridayNight</td>\n",
       "      <td>Mostly Clear</td>\n",
       "      <td>Low: 67 °F</td>\n",
       "      <td>Friday Night: Mostly clear, with a low around 67.</td>\n",
       "      <td>67</td>\n",
       "      <td>True</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           period                                  short_desc        temp  \\\n",
       "1         Tonight  ScatteredT-storms thenSlight ChanceShowers  Low: 74 °F   \n",
       "3  WednesdayNight            T-stormsLikely thenChanceShowers  Low: 71 °F   \n",
       "5   ThursdayNight       Partly Cloudythen SlightChanceShowers  Low: 73 °F   \n",
       "7     FridayNight                                Mostly Clear  Low: 67 °F   \n",
       "\n",
       "                                                desc  temp_num  is_night  \n",
       "1  Tonight: Scattered showers and thunderstorms b...        74      True  \n",
       "3  Wednesday Night: Showers and thunderstorms lik...        71      True  \n",
       "5  Thursday Night: A slight chance of showers aft...        73      True  \n",
       "7  Friday Night: Mostly clear, with a low around 67.        67      True  "
      ]
     },
     "execution_count": 71,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "weather[is_night]\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Using Pandas to read a `<table>` directly"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Unnamed: 0</th>\n",
       "      <th>Release Date</th>\n",
       "      <th>Movie</th>\n",
       "      <th>Production Budget</th>\n",
       "      <th>Domestic Gross</th>\n",
       "      <th>Worldwide Gross</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>Dec 18, 2009</td>\n",
       "      <td>Avatar</td>\n",
       "      <td>$425,000,000</td>\n",
       "      <td>$760,507,625</td>\n",
       "      <td>$2,789,705,275</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>May 20, 2011</td>\n",
       "      <td>Pirates of the Caribbean: On Stranger Tides</td>\n",
       "      <td>$410,600,000</td>\n",
       "      <td>$241,063,875</td>\n",
       "      <td>$1,045,663,875</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>Apr 26, 2019</td>\n",
       "      <td>Avengers: Endgame</td>\n",
       "      <td>$400,000,000</td>\n",
       "      <td>$857,545,354</td>\n",
       "      <td>$2,792,694,500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>Jun 7, 2019</td>\n",
       "      <td>Dark Phoenix</td>\n",
       "      <td>$350,000,000</td>\n",
       "      <td>$65,765,300</td>\n",
       "      <td>$246,456,796</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>May 1, 2015</td>\n",
       "      <td>Avengers: Age of Ultron</td>\n",
       "      <td>$330,600,000</td>\n",
       "      <td>$459,005,868</td>\n",
       "      <td>$1,403,013,963</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Unnamed: 0  Release Date                                        Movie  \\\n",
       "0           1  Dec 18, 2009                                       Avatar   \n",
       "1           2  May 20, 2011  Pirates of the Caribbean: On Stranger Tides   \n",
       "2           3  Apr 26, 2019                            Avengers: Endgame   \n",
       "3           4   Jun 7, 2019                                 Dark Phoenix   \n",
       "4           5   May 1, 2015                      Avengers: Age of Ultron   \n",
       "\n",
       "  Production Budget Domestic Gross Worldwide Gross  \n",
       "0      $425,000,000   $760,507,625  $2,789,705,275  \n",
       "1      $410,600,000   $241,063,875  $1,045,663,875  \n",
       "2      $400,000,000   $857,545,354  $2,792,694,500  \n",
       "3      $350,000,000    $65,765,300    $246,456,796  \n",
       "4      $330,600,000   $459,005,868  $1,403,013,963  "
      ]
     },
     "execution_count": 62,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "\n",
    "headers={'user-agent':'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.142 Safari/537.36'}\n",
    "resp = requests.get('https://www.the-numbers.com/movie/budgets/all', headers=headers)\n",
    "pd.read_html(resp.content)[0].head()\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}