3-data/3-2-scraping-data.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### GESIS Fall Seminar in Computational Social Science 2022\n",
    "### Introduction to Computational Social Science with Python\n",
    "# Day 3-2: Scraping Web Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Overview\n",
    "\n",
    "* Inspecting webpages\n",
    "* Parsing static HTML with BeautifulSoup\n",
    "* Scraping dynamic pages with Selenium"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Inspecting webpages\n",
    "* Scraping is a very useful tool for accessing and analysing the vast amount of unstructured data on the web.\n",
    "* Much more efficient than bulk manual copying and pasting information you find on the web.\n",
    "* We need a systematic way of identifying and extracting the information we require from webpages.\n",
    "* To do this, we need to be able to read and understand the underlying HTML (HyperText Markup Language) and CSS (Cascading Style Sheets) that builds a webpage.\n",
    "* This is a brief introduction to scraping, much more detail in next week's GESIS course \"Automated Web Data Collection with Python\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Consider the file 'simple.html' for a simple webpage. Open it with both a web browser and a text editor and study its structure.\n",
    "\n",
    "Real pages are usually much more complicated!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "<div>\n",
    "<img src=\"figs/html.png\" width=\"1000\"/>\n",
    "</div>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Now consider the file 'css_demo.html' for a simple webpage. Open it with both a web browser and a text editor and study its structure.\n",
    "\n",
    "We have added some style information with CSS. The CSS style information is often very useful for finding and selecting elements."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### BeautifulSoup\n",
    "* BeautifulSoup is a package for parsing html.\n",
    "* It creates a navigable tree of the structured html.\n",
    "* We can also easily search through the tree for the elements we require."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<!DOCTYPE html>\n",
      "<html>\n",
      " <head>\n",
      "  <title>\n",
      "   Page Title\n",
      "  </title>\n",
      " </head>\n",
      " <body>\n",
      "  <h1>\n",
      "   My First Heading\n",
      "  </h1>\n",
      "  <p>\n",
      "   My first paragraph.\n",
      "  </p>\n",
      " </body>\n",
      "</html>\n"
     ]
    }
   ],
   "source": [
    "from bs4 import BeautifulSoup\n",
    "\n",
    "with open('data/simple.html', 'r') as f:\n",
    "    webpage = f.read()\n",
    "\n",
    "soup = BeautifulSoup(webpage)\n",
    "print(soup.prettify())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<head>\n",
      "<title>Page Title</title>\n",
      "</head>\n"
     ]
    }
   ],
   "source": [
    "# Let's navigate through the tree:\n",
    "\n",
    "print(soup.head)\n",
    "# print(soup.head.title)\n",
    "# print(soup.head.title.text)\n",
    "\n",
    "# print(soup.body.p.text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "scrolled": false,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<!DOCTYPE html>\n",
      "<html>\n",
      " <head>\n",
      "  <!-- CSS start -->\n",
      "  <style>\n",
      "   .text-about-web-scraping { color: orange;\n",
      "}\n",
      ".division-two h1 { color: green;\n",
      "}\n",
      "  </style>\n",
      "  <!-- CSS end -->\n",
      "  <title>\n",
      "   A title\n",
      "  </title>\n",
      " </head>\n",
      " <body>\n",
      "  <div>\n",
      "   <h1>\n",
      "    Heading of the first division\n",
      "   </h1>\n",
      "   <p>\n",
      "    A first paragraph with some\n",
      "    <b>\n",
      "     formatted\n",
      "    </b>\n",
      "    text.\n",
      "   </p>\n",
      "   <p>\n",
      "    A second paragraph with more text.\n",
      "   </p>\n",
      "   <p class=\"text-about-web-scraping\">\n",
      "    A third paragraph now containing some text about web scraping ...\n",
      "   </p>\n",
      "  </div>\n",
      "  <div class=\"division-two\">\n",
      "   <h1>\n",
      "    Heading of the second division\n",
      "   </h1>\n",
      "   <p>\n",
      "    Another paragraph with some text.\n",
      "   </p>\n",
      "   <p class=\"text-about-web-scraping\">\n",
      "    A last paragraph discussing some web scraping ...\n",
      "   </p>\n",
      "  </div>\n",
      " </body>\n",
      "</html>\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Let's load the css_demo.html file\n",
    "\n",
    "with open('data/css_demo.html', 'r') as f:\n",
    "    webpage = f.read()\n",
    "\n",
    "soup = BeautifulSoup(webpage)\n",
    "print(soup.prettify())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[<p>A first paragraph with some <b>formatted</b> text.</p>, <p>A second paragraph with more text.</p>, <p class=\"text-about-web-scraping\">A third paragraph now containing some text about web scraping ...</p>, <p>Another paragraph with some text.</p>, <p class=\"text-about-web-scraping\">A last paragraph discussing some web scraping ...</p>]\n"
     ]
    }
   ],
   "source": [
    "# We can use select() to find all the elements that match a particular tag\n",
    "\n",
    "print(soup.select('p'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[<div class=\"division-two\">\n",
      "<h1>Heading of the second division</h1>\n",
      "<p>Another paragraph with some text.</p>\n",
      "<p class=\"text-about-web-scraping\">A last paragraph discussing some web scraping ...</p>\n",
      "</div>]\n"
     ]
    }
   ],
   "source": [
    "# Or to find all the elements that match a particular tag and 'class'\n",
    "\n",
    "print(soup.select('.division-two'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## 🏋️‍♀️ PRACTICE"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "# Q1: \n",
    "# a) Use BeautifulSoup to navigate the css_demo.html file down to the bolded text \"formatted\"\n",
    "#    i.e., soup.body...\n",
    "# a) Use BeautifulSoup select() to find the same text\n",
    "#    i.e., soup.select()...\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "# Q2: Get the text of all paragraphs in css_demo.html with the class \"text-about-web-scraping\"\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "# Q3: Open the file PL_table.html in a text editor. Search the html until you find the Premier League table.\n",
    "# Read the file into BeautifulSoup\n",
    "# Try to identify the selector that gives you all the table cell names (club names) from the table.\n",
    "# (Yes, this is meant to be quite difficult at this point!)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Parsing static HTML with BeautifulSoup\n",
    "* As you have just seen, real webpages are a lot more complicated than our toy examples!\n",
    "* We need a better way of identifying the element selection codes.\n",
    "* Let's navigate to Wikipedia's [List of United States cities by population](https://en.wikipedia.org/wiki/List_of_United_States_cities_by_population)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "* \"Right-click > Inspect\" quickly shows us the HTML structure\n",
    "* [SelectorGadget](https://selectorgadget.com/) is a useful tool to instantly show us the relevant tags."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "# Let's integrate this into Python...\n",
    "\n",
    "import requests\n",
    "import time\n",
    "\n",
    "url_population = \"https://en.wikipedia.org/wiki/List_of_United_States_cities_by_population\"\n",
    "\n",
    "page_population = requests.get(url_population) # Requests gets the page html for us\n",
    "soup = BeautifulSoup(page_population.content) # Parse with BeautifulSoup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<b>1. New York</b>,\n",
       " <b>2. Los Angeles</b>,\n",
       " <b>3. Chicago</b>,\n",
       " <b>4. Houston</b>,\n",
       " <b>5. Phoenix</b>,\n",
       " <b>6. Philadelphia</b>,\n",
       " <b>7. San Antonio</b>,\n",
       " <b>8. San Diego</b>,\n",
       " <b>9. Dallas</b>,\n",
       " <b>10. San Jose</b>]"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# We can extract the top 10 cities from the map as identified:\n",
    "\n",
    "soup.select(\".noresize b\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## 🏋️‍♀️ PRACTICE"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "# Q4: Re-open PL_table.html from Q3 in your web browser.\n",
    "# Use the Selector Gadget and BeautifulSoup to select the elements (club names) as before.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "# Q5: Select all of the city names, as well as their associated links (.get('href')) from the map\n",
    "# at the bottom of the Wikipedia article List of United States cities by population\n",
    "# Save the result as a dictionary, with city name as the key, and full wikipedia link as the value.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "# Q6: Loop through all of the links you have just collected.\n",
    "# For each of these city pages, extract all hyperlinks to pages within Wikipedia\n",
    "# (i.e. starting with /wiki/) from the main article content\n",
    "# Add time.sleep(1) as the last stage in your loop, don't overload the site!\n",
    "# Save the result as a dictionary with city name as key, and list of links as values\n",
    "# Perfect parsing here is quite tough!\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Scraping dynamic pages with Selenium\n",
    "* Not all of the web is simple HTML files.\n",
    "* Lots of dynamic content (e.g. Javascript) can only be accessed by having a proper web browser rendering the website\n",
    "* We may also want to interact with pages (e.g. scrolling, clicking) and even send data to websites.\n",
    "* **[Selenium](https://selenium-python.readthedocs.io/)** is the solution!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Installation\n",
    "* Ensure you [install Selenium](https://selenium-python.readthedocs.io/installation.html) correctly\n",
    " - You need to install the Python package `conda install selenium`.\n",
    " - You also need to download a 'driver' (Firefox recommended) for your system, and move it to the correct location on your system.\n",
    "* The 'driver' is an automated web browser that you can see performing the actions you program."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "# If this cell runs, you have correctly installed Selenium!\n",
    "from selenium import webdriver\n",
    "from selenium.webdriver.common.keys import Keys\n",
    "from selenium.webdriver.common.by import By\n",
    "\n",
    "driver = webdriver.Firefox()\n",
    "driver.get(\"http://www.python.org\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "elem = driver.find_element(By.NAME, \"q\") # find the search box\n",
    "elem.clear() # clear anything currently in the search box\n",
    "elem.send_keys(\"Guido\")  # type the keys \"Guido\"\n",
    "elem.send_keys(Keys.RETURN) # Send a return key to search"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "driver.quit() # Quit the driver once finished"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "# Let's interact with a page further by playing Happy Birthday\n",
    "\n",
    "driver = webdriver.Firefox()\n",
    "driver.get(\"https://en.wikipedia.org/wiki/Happy_Birthday_to_You\")\n",
    "elem = driver.find_element(By.CSS_SELECTOR, \":nth-child(9) .mw-tmh-play-icon\") # find the play button"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "elem.click() # click the element"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "driver.quit() # Quit the driver once finished"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "# Let's use Selenium to extract data from a page on Amazon deals\n",
    "\n",
    "driver = webdriver.Firefox()\n",
    "driver.get(\"https://www.amazon.com/gp/goldbox/\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "# Get selector from SelectorGadget for the deal names and use the driver to select elements\n",
    "elems = driver.find_elements(By.CSS_SELECTOR, \"#slot-15 .DealContent-module__truncate_sWbxETx42ZPStTc9jwySW\")\n",
    "dealnames = [x.text for x in elems]\n",
    "driver.quit()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Elvira Sports Water Bottles',\n",
       " 'Eufy Home Security Cameras',\n",
       " 'XSOUL IPL Laser Hair Removal',\n",
       " 'Arf Pets Dog Cooling Mat',\n",
       " 'Dorm Room Essentials from Amazon Brands',\n",
       " 'Laptops and Desktops for Entry Level PC Gamers',\n",
       " 'Save on Select Samsung TVs and Projectors',\n",
       " 'Laptops and Desktops for Mainstream PC Gamers',\n",
       " 'Gaming Accessories from Alienware, MSI, Acer',\n",
       " 'Save on Adobe | Photoshop Elements 2022 | PC/Mac Disc | Photo Editing Software and more',\n",
       " 'Laptops and Desktops for PC Gaming Enthusiasts',\n",
       " 'Samsung Soundbar',\n",
       " 'Bose TV Speaker',\n",
       " 'Gaming Laptops and Desktop from HP, Dell, ASUS and more',\n",
       " 'Bose SoundLink Micro Bluetooth Speaker',\n",
       " 'Bikes, Scooters, and Accessories from Schwinn and Mongoose',\n",
       " 'SAMSUNG The Premiere Ultra Short Throw 4K UHD Projector',\n",
       " 'Kindpack Bluetooth 5.0 Headset TWS Wireless Earphones Mini Earbuds Stereo Music Sports Earphones for iPhone Xiaomi for Huawei, Phantom Black',\n",
       " 'SEYOO Headphones Mini Earbuds Hi-Fi Stereo in-Ear Earphones with 350Mah Portable Charging Case, Touch Control, USB Fast Charge, Built-in Mic, IPX5 Waterproof for Work, Sports, Running, Carbon Black',\n",
       " 'E7 Active Noise Cancelling Headphones Bluetooth Headphones with Microphone Deep Bass Wireless Headphones Over Ear, Comfortable Protein Earpads, 30 Hours Playtime for Travel/Work…',\n",
       " 'Compression Socks for Women Circulation,20-30mmhg Knee High Sports Running Men Sock Stocking-Support Hose Recovery,Relief Calves Foot Pain for Athletic Pregnancy Travel Nursing Flying-6 Pairs',\n",
       " 'Earbuds 5.0 Mini Headphones, IPX5 Waterproof Hi-Fi Stereo Built-in Mic Headset, in-Ear Earphones for Traveling',\n",
       " 'Luckypack Hot 8 oz Disposable Insulated Corrugated Sleeve Ripple Wall Paper Coffee Cups for Drink, 100,8oz, Brown,Hot Coffee Cups',\n",
       " 'Cvozo Bluetooth Headphones, Bluetooth 5.0 Wireless Earphones in-Ear Stereo Sound Microphone Mini Wireless Earbuds with Headphones and Portable Charging Case for iOS Android PC planb3',\n",
       " 'Facial Serum for Skin Care, Natural Facial Moisturizing Essence with Vitamin C Hyaluronic Acid Vitamin E Supply Collagen Tightening Skin, Anti-aging Hydrating Brightening Oil Serum for Wrinkles,Spots,Acne, 1.7 FL OZ/50ml',\n",
       " 'Large Display Kitchen Timer, Bold Digits, Loud Alarm, Magnetic Kickstand - White',\n",
       " 'Jeacitory 12x25 Binoculars with Clear Low Night Version for Adults and Kids, Large Eyepiece Waterproof Durable & Clear BAK4 Prism FMC Lens Binoculars for Bird Watching Hunting Travel Theater Concerts',\n",
       " 'Hoperay Swimsuit Cover Ups for Women Beach Dresses, GreenEmbroidered Caftans Long Sleeve Floral Print Kaftans Side Split Summer Beachwear, One Size',\n",
       " 'Wireless Earbuds, Active Noise Cancelling Headphones in-Ear Bluetooth Headsets with 4 Mics 30hrs Playtime Fast Charging Premium Sound for Sports IPX8 Waterproof Gaming/Sleeping Earphones',\n",
       " 'Wireless Earbuds Bluetooth 5.0 Mini Headphones, Hi-Fi Stereo in-Ear Earphones, Touch Control, IPX5 Waterproof Headset with LED Display Built-in Mic for Sports, Workout, Gym']"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dealnames"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## 🏋️‍♀️ PRACTICE"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "# Q7: Use Selenium to navigate to this shared Google Doc\n",
    "# https://docs.google.com/document/d/1NKY4eeVtBjNNdx1cpS8VPiyazVCPnQRfMBTB4yHtk7I/edit?usp=sharing\n",
    "# Select the document using elem = driver.find_element_by_xpath('//body')\n",
    "# Write a message on a **new line** using send_keys\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "# Q8: Use Selenium to navigate to the Amazon deals page https://www.amazon.com/gp/goldbox/\n",
    "# As demonstrated previously, extract the names of all the deals on the page\n",
    "# Then, find the element to progress to the next page of results and click it\n",
    "# Extract the names of all the deals on this page, and repeat for the first 5 pages of results\n",
    "# Sleep 1s between pages\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "# Q9: If you have finished the previous exercises, consider how you could improve your scraper from Q8.\n",
    "# Scrape deals by category, or get the hyperlink, price, use a custom search etc.\n"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}