week-1/1-Measuring-Meaning-and-Sampling.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Week 1 - Measuring Meaning & Sampling\n",
    "This week, we begin by \"begging, borrowing and stealing\" text from several\n",
    "contexts of human communication (e.g., PDFs, HTML, Word) and preparing it for\n",
    "machines to \"read\" and analyze so that we can begin to build our sample. This notebook outlines scraping text from the web, PDF and Word documents. Then we detail \"spidering\" or walking\n",
    "through hyperlinks to build samples of online content, and using APIs,\n",
    "Application Programming Interfaces, provided by webservices to access their\n",
    "content. Along the way, we will use regular expressions, outlined in the\n",
    "reading, to remove unwanted formatting and ornamentation. Next, we discuss\n",
    "various text encodings, filtering and data structures in which text can be\n",
    "placed for analysis. Finally, we ask you to begin building a corpus for preliminary analysis and articulate what your sample represents in context of your final project. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We made a python package just for this course: lucem_illud. If you haven't installed this package, you should run the following code first. You don't need to install the package later; all you need to do is just to import the package with: import lucem_illud. For your final projects, you may find it useful to [read the lucem_illud source code](https://github.com/UChicago-Computational-Content-Analysis/lucem_illud/tree/main/lucem_illud) and modify your code for your own interests."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install git+https://github.com/UChicago-Computational-Content-Analysis/lucem_illud.git\n",
    "#installing lucem_illud package\n",
    "#lucem_illud is a Latin phrase meaning \"that light\", the insight we can discover in text data!\n",
    "#If you get an error like \"Access is denied\", try running the `pip` command on the command line as an administrator."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you're not familiar with jupyter notebook, you may wonder what the exclamation mark(!) at the beginning of the command does (or even what pip means). The exclamation mark enables us to execute Terminal commands in the notebook cells (e.g., run `!ls` to display files in the current folder).\n",
    "\n",
    "There is also a special download required by the `lucem_illud` module in the module `spacy`. You will see this 'en' module later, but you should probably run the following 2 lines of code so you can import `lucem_illud`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import spacy\n",
    "!python -m spacy download en"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For this notebook we will be using the following packages:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Special module written for this class\n",
    "#This provides access to data and to helper functions from previous weeks\n",
    "import lucem_illud #pip install git+https://github.com/UChicago-Computational-Content-Analysis/lucem_illud.git\n",
    "\n",
    "#All these packages need to be installed from pip\n",
    "import requests #for http requests\n",
    "import bs4 #called `beautifulsoup4`, an html parser\n",
    "import pandas #gives us DataFrames\n",
    "import docx #reading MS doc files, install as `python-docx`\n",
    "\n",
    "#Stuff for pdfs\n",
    "#Install as `pdfminer2`\n",
    "import pdfminer.pdfinterp\n",
    "import pdfminer.converter\n",
    "import pdfminer.layout\n",
    "import pdfminer.pdfpage\n",
    "\n",
    "#These come with Python\n",
    "import re #for regexs\n",
    "import urllib.parse #For joining urls\n",
    "import io #for making http requests look like files\n",
    "import json #For Tumblr API responses\n",
    "import os.path #For checking if files exist\n",
    "import os #For making directories"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will also be working on the following files/urls"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "wikipedia_base_url = 'https://en.wikipedia.org'\n",
    "wikipedia_content_analysis = 'https://en.wikipedia.org/wiki/Content_analysis'\n",
    "content_analysis_save = 'wikipedia_content_analysis.html'\n",
    "example_text_file = 'sometextfile.txt'\n",
    "information_extraction_pdf = 'https://github.com/Computational-Content-Analysis-2018/Data-Files/raw/master/1-intro/Content%20Analysis%2018.pdf'\n",
    "example_docx = 'https://github.com/Computational-Content-Analysis-2018/Data-Files/raw/master/1-intro/macs6000_connecting_to_midway.docx'\n",
    "example_docx_save = 'example.docx'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Scraping\n",
    "\n",
    "Before we can start analyzing content we need to obtain it. Sometimes it will be\n",
    "provided to us from a pre-curated text archive, but sometimes we will need to\n",
    "download it. As a starting example we will attempt to download the wikipedia\n",
    "page on content analysis. The page is located at [https://en.wikipedia.org/wiki/\n",
    "Content_analysis](https://en.wikipedia.org/wiki/Content_analysis) so lets start\n",
    "with that.\n",
    "\n",
    "We can do this by making an HTTP GET request to that url, a GET request is\n",
    "simply a request to the server to provide the contents given by some url. The\n",
    "other request we will be using in this class is called a POST request and\n",
    "requests the server to take some content we provide. While the Python standard\n",
    "library does have the ability do make GET requests we will be using the\n",
    "[_requests_](http://docs.python-requests.org/en/master/) package as it is _'the\n",
    "only Non-GMO HTTP library for Python'_...also it provides a nicer interface."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#wikipedia_content_analysis = 'https://en.wikipedia.org/wiki/Content_analysis'\n",
    "requests.get(wikipedia_content_analysis)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`'Response [200]'` means the server responded with what we asked for. If you get\n",
    "another number (e.g. 404) it likely means there was some kind of error, these\n",
    "codes are called HTTP response codes and a list of them can be found\n",
    "[here](https://en.wikipedia.org/wiki/List_of_HTTP_status_codes). The response\n",
    "object contains all the data the server sent including the website's contents\n",
    "and the HTTP header. We are interested in the contents which we can access with\n",
    "the `.text` attribute."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "wikiContentRequest = requests.get(wikipedia_content_analysis)\n",
    "print(wikiContentRequest.text[:1000])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is not what we were looking for, because it is the start of the HTML that\n",
    "makes up the website. This is HTML and is meant to be read by computers. Luckily\n",
    "we have a computer to parse it for us. To do the parsing we will use [_Beautiful\n",
    "Soup_](https://www.crummy.com/software/BeautifulSoup/) which is a better parser\n",
    "than the one in the standard library."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "But before we proceed to Beautiful Soup, a digression about Python syntax, especially about objects and functions.\n",
    "For those who are not familiar with the syntax of python (or, if you're familiar with R programming), you might wonder what requests.get or wikiContentRequest.text mean. To understand this, you need to first understand what objects are. You may have heard that Python is an object oriented programming language (unlike the procedure oriented programming language, an example of which is R). Object is a set of variables (or, data) and functions into which you pass your data. So, in object oriented programming languages, like python, variables and functions are bunleded into objects.\n",
    "\n",
    "For example, let's look at wikiContentRequest. We use dir() function, which returns the list of attributes and functions of objects."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    " dir(wikiContentRequest)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There's 'text' here. We used 'wikiContentRequest.text' to access 'text.' In other words, we use .(dot notation) to access functions from objects. wikiContentRequest has a set of functions, as shown above, and we used 'wikiContentRequest.text' to access one of them. By the way, dot notations do not necessarily refer to functions--it refers to anything that the entity contains. \n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Moving on to the next step: BeautifulSoup, a Python library which extracts data from HTML and XML, and transforms HTML files into Python objects."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "wikiContentSoup = bs4.BeautifulSoup(wikiContentRequest.text, 'html.parser')\n",
    "print(wikiContentSoup.text[:200])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is better but there's still random whitespace and we have more than just\n",
    "the text of the article. This is because what we requested is the whole webpage,\n",
    "not just the text for the article.\n",
    "\n",
    "We want to extract only the text we care about, and in order to do this we will\n",
    "need to inspect the html. One way to do this is simply to go to the website with\n",
    "a browser and use its inspection or view source tool. If javascript or other\n",
    "dynamic loading occurs on the page, however, it is likely that what Python\n",
    "receives is not what you will see, so we will need to inspect what Python\n",
    "receives. To do this we can save the html `requests` obtained."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#content_analysis_save = 'wikipedia_content_analysis.html'\n",
    "\n",
    "with open(content_analysis_save, mode='w', encoding='utf-8') as f:\n",
    "    f.write(wikiContentRequest.text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "open() is a function which literally opens and returns the file. This function has multiple modes, and, here, we used mode as 'w', which means: open a file for writing. And then, we use 'write' function to write on the empty file (content_analysis_save) that we created using open(content_analysis_save, mode='w', encoding='utf-8').} What did we write on this file? The text we got from wikiContentRequest.text"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's open the file (`wikipedia_content_analysis.html`) we just created with\n",
    "a web browser. It should look sort of like the original but without the images\n",
    "and formatting.\n",
    "\n",
    "As there is very little standardization on structuring webpages, figuring out\n",
    "how best to extract what you want is an art. Looking at this page it looks like\n",
    "all the main textual content is inside `<p>`(paragraph) tags within the `<body>`\n",
    "tag. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "contentPTags = wikiContentSoup.body.findAll('p')\n",
    "for pTag in contentPTags[:3]:\n",
    "    print(pTag.text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Another excursion for those who are not familiar with programming: for loop. For loop is used to iterate over a sequence. \"ContentPTags\" contains multiple paragraphs, each of which starts and ends with `<p>`. What the \"for pTag in contentPtags[:3]\" does here is: find each paragraph in contentPTags, which, here, we limited to the first three using contentPtags[:3], and then print each paragraph. So, we have three paragraphs. By the way, you can insert `<p>` in juputer notebook!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We now have all the text from the page, split up by paragraph. If we wanted to\n",
    "get the section headers or references as well it would require a bit more work,\n",
    "but is doable.\n",
    "\n",
    "There is one more thing we might want to do before sending this text to be\n",
    "processed, remove the references indicators (`[2]`, `[3]` , etc). To do this we\n",
    "can use a short regular expression (regex)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "contentParagraphs = []\n",
    "for pTag in contentPTags:\n",
    "    #strings starting with r are raw so their \\'s are not modifier characters\n",
    "    #If we didn't start with r the string would be: '\\\\[\\\\d+\\\\]'\n",
    "    contentParagraphs.append(re.sub(r'\\[\\d+\\]', '', pTag.text))\n",
    "\n",
    "#convert to a DataFrame\n",
    "contentParagraphsDF = pandas.DataFrame({'paragraph-text' : contentParagraphs})\n",
    "print(contentParagraphsDF)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since we learned how to do for loop, you might get what we did here: using contentParagraphs = [], we made an empty list; and then, for each paragraph in contentPTags, we substituted every [\\d+\\] with '', i.e., removed every [\\d+\\], and then appended each paragraph (now without [\\d+\\]) to the empty list. As we can see, we have a dataframe, each row of which is each paragraph of contentPTags, without reference indicators. \n",
    "\n",
    "By the way, what does [\\d+\\] mean? If you are not familiar with regex, it is a way of specifying searches in text.\n",
    "A regex engine takes in the search pattern, in the above case `'\\[\\d+\\]'` and\n",
    "some string, the paragraph texts. Then it reads the input string one character\n",
    "at a time checking if it matches the search. Here the regex `'\\d'` matches\n",
    "number characters (while `'\\['` and `'\\]'` capture the braces on either side)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we have a `DataFrame` containing all relevant text from the page ready to be processed"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "findNumber = r'\\d'\n",
    "regexResults = re.search(findNumber, 'not a number, not a number, numbers 2134567890, not a number')\n",
    "regexResults"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In Python the regex package (`re`) usually returns `Match` objects (you can have\n",
    "multiple pattern hits in a a single `Match`), to get the string that matched our\n",
    "pattern we can use the `.group()` method, and as we want the first one we will\n",
    "ask for the 0'th group."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(regexResults.group(0))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That gives us the first number, if we wanted the whole block of numbers we can\n",
    "add a wildcard `'+'` which requests 1 or more instances of the preceding\n",
    "character."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "findNumbers = r'\\d+'\n",
    "regexResults = re.search(findNumbers, 'not a number, not a number, numbers 2134567890, not a number')\n",
    "print(regexResults.group(0))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we have the whole block of numbers, there are a huge number of special\n",
    "characters in regex, for the full description of Python's implementation look at\n",
    "the [re docs](https://docs.python.org/3/library/re.html) there is also a short\n",
    "[tutorial](https://docs.python.org/3/howto/regex.html#regex-howto)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# <font color=\"red\">Exercise 1</font>\n",
    "<font color=\"red\">Construct cells immediately below this that describe and download webcontent relating to your anticipated final project. Use beautiful soup and at least five regular expressions to extract relevant, nontrivial *chunks* of that content (e.g., cleaned sentences, paragraphs, etc.) to a pandas `Dataframe`.</font>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "# Spidering\n",
    "\n",
    "What if we want to to get a bunch of different pages from wikipedia. We would\n",
    "need to get the url for each of the pages we want. Typically, we want pages that\n",
    "are linked to by other pages and so we will need to parse pages and identify the\n",
    "links. Right now we will be retrieving all links in the body of the content\n",
    "analysis page.\n",
    "\n",
    "To do this we will need to find all the `<a>` (anchor) tags with `href`s\n",
    "(hyperlink references) inside of `<p>` tags. `href` can have many\n",
    "[different](http://stackoverflow.com/questions/4855168/what-is-href-and-why-is-\n",
    "it-used) [forms](https://en.wikipedia.org/wiki/Hyperlink#Hyperlinks_in_HTML) so\n",
    "dealing with them can be tricky, but generally, you will want to extract\n",
    "absolute or relative links. An absolute link is one you can follow without\n",
    "modification, while a relative link requires a base url that you will then\n",
    "append. Wikipedia uses relative urls for its internal links: below is an example\n",
    "for dealing with them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#wikipedia_base_url = 'https://en.wikipedia.org'\n",
    "\n",
    "otherPAgeURLS = []\n",
    "#We also want to know where the links come from so we also will get:\n",
    "#the paragraph number\n",
    "#the word the link is in\n",
    "for paragraphNum, pTag in enumerate(contentPTags):\n",
    "    #we only want hrefs that link to wiki pages\n",
    "    tagLinks = pTag.findAll('a', href=re.compile('/wiki/'), class_=False)\n",
    "    for aTag in tagLinks:\n",
    "        #We need to extract the url from the <a> tag\n",
    "        relurl = aTag.get('href')\n",
    "        linkText = aTag.text\n",
    "        #wikipedia_base_url is the base we can use the urllib joining function to merge them\n",
    "        #Giving a nice structured tupe like this means we can use tuple expansion later\n",
    "        otherPAgeURLS.append((\n",
    "            urllib.parse.urljoin(wikipedia_base_url, relurl),\n",
    "            paragraphNum,\n",
    "            linkText,\n",
    "        ))\n",
    "print(otherPAgeURLS[:10])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(contentPTags)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Another excursion: Why do we use enumerate() here? enumerate() takes a collection, enumerates, and returns an enumate object with both the numbers and the collection. For example, contentPTags (the collection we used here) is comprised of paragraphs. We want the paragraph number of each paragraph. And this is what enumerate() does: it returns the paragraph number and the paragraph. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will be adding these new texts to our DataFrame `contentParagraphsDF` so we\n",
    "will need to add 2 more columns to keep track of paragraph numbers and sources."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "contentParagraphsDF['source'] = [wikipedia_content_analysis] * len(contentParagraphsDF['paragraph-text'])\n",
    "contentParagraphsDF['paragraph-number'] = range(len(contentParagraphsDF['paragraph-text']))\n",
    "\n",
    "contentParagraphsDF"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then we can add two more columns to our `Dataframe` and define a function to\n",
    "parse\n",
    "each linked page and add its text to our DataFrame."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "contentParagraphsDF['source-paragraph-number'] = [None] * len(contentParagraphsDF['paragraph-text'])\n",
    "contentParagraphsDF['source-paragraph-text'] = [None] * len(contentParagraphsDF['paragraph-text'])\n",
    "\n",
    "def getTextFromWikiPage(targetURL, sourceParNum, sourceText):\n",
    "    #Make a dict to store data before adding it to the DataFrame\n",
    "    parsDict = {'source' : [], 'paragraph-number' : [], 'paragraph-text' : [], 'source-paragraph-number' : [],  'source-paragraph-text' : []}\n",
    "    #Now we get the page\n",
    "    r = requests.get(targetURL)\n",
    "    soup = bs4.BeautifulSoup(r.text, 'html.parser')\n",
    "    #enumerating gives use the paragraph number\n",
    "    for parNum, pTag in enumerate(soup.body.findAll('p')):\n",
    "        #same regex as before\n",
    "        parsDict['paragraph-text'].append(re.sub(r'\\[\\d+\\]', '', pTag.text))\n",
    "        parsDict['paragraph-number'].append(parNum)\n",
    "        parsDict['source'].append(targetURL)\n",
    "        parsDict['source-paragraph-number'].append(sourceParNum)\n",
    "        parsDict['source-paragraph-text'].append(sourceText)\n",
    "    return pandas.DataFrame(parsDict)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And run it on our list of link tags"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for urlTuple in otherPAgeURLS[:3]:\n",
    "    #ignore_index means the indices will not be reset after each append\n",
    "    contentParagraphsDF = contentParagraphsDF.append(getTextFromWikiPage(*urlTuple),ignore_index=True)\n",
    "contentParagraphsDF"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "# <font color=\"red\">Exercise 2</font>\n",
    "<font color=\"red\">Construct cells immediately below this that spider webcontent from another site with content relating to your anticipated final project. Specifically, identify urls on a core page, then follow and extract content from them into a pandas `Dataframe`. In addition, demonstrate a *recursive* spider, which follows more than one level of links (i.e., follows links from a site, then follows links on followed sites to new sites, etc.), making sure to define a reasonable endpoint so that you do not wander the web forever :-).</font>\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## API (Tumblr)\n",
    "\n",
    "Generally website owners do not like you scraping their sites. If done badly,\n",
    "scarping can act like a DOS attack so you should be careful how often you make\n",
    "calls to a site. Some sites want automated tools to access their data, so they\n",
    "create [application programming interface\n",
    "(APIs)](https://en.wikipedia.org/wiki/Application_programming_interface). An API\n",
    "specifies a procedure for an application (or script) to access their data. Often\n",
    "this is though a [representational state transfer\n",
    "(REST)](https://en.wikipedia.org/wiki/Representational_state_transfer) web\n",
    "service, which just means if you make correctly formatted HTTP requests they\n",
    "will return nicely formatted data.\n",
    "\n",
    "A nice example for us to study is [Tumblr](https://www.tumblr.com), they have a\n",
    "[simple RESTful API](https://www.tumblr.com/docs/en/api/v1) that allows you to\n",
    "read posts without any complicated html parsing.\n",
    "\n",
    "We can get the first 20 posts from a blog by making an http GET request to\n",
    "`'http://{blog}.tumblr.com/api/read/json'`, were `{blog}` is the name of the\n",
    "target blog. Lets try and get the posts from [http://lolcats-lol-\n",
    "cat.tumblr.com/](http://lolcats-lol-cat.tumblr.com/) (Note the blog says at the\n",
    "top 'One hour one pic lolcats', but the canonical name that Tumblr uses is in\n",
    "the URL 'lolcats-lol-cat')."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tumblrAPItarget = 'http://{}.tumblr.com/api/read/json'\n",
    "\n",
    "r = requests.get(tumblrAPItarget.format('lolcats-lol-cat'))\n",
    "\n",
    "print(r.text[:1000])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This might not look very good on first inspection, but it has far fewer angle\n",
    "braces than html, which makes it easier to parse. What we have is\n",
    "[JSON](https://en.wikipedia.org/wiki/JSON) a 'human readable' text based data\n",
    "transmission format based on javascript. Luckily, we can readily convert it to a\n",
    "python `dict`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#We need to load only the stuff between the curly braces\n",
    "d = json.loads(r.text[len('var tumblr_api_read = '):-2])\n",
    "print(d.keys())\n",
    "print(len(d['posts']))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we read the [API specification](https://www.tumblr.com/docs/en/api/v1), we\n",
    "will see there are a lot of things we can get if we add things to our GET\n",
    "request. First we can retrieve posts by their id number. Let's first get post\n",
    "`146020177084`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "r = requests.get(tumblrAPItarget.format('lolcats-lol-cat'), params = {'id' : 146020177084})\n",
    "d = json.loads(r.text[len('var tumblr_api_read = '):-2])\n",
    "d['posts'][0].keys()\n",
    "d['posts'][0]['photo-url-1280']\n",
    "\n",
    "with open('lolcat.gif', 'wb') as f:\n",
    "    gifRequest = requests.get(d['posts'][0]['photo-url-1280'], stream = True)\n",
    "    f.write(gifRequest.content)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src='lolcat.gif'>\n",
    "\n",
    "Such beauty; such vigor (If you can't see it you have to refresh the page). Now\n",
    "we could retrieve the text from all posts as well\n",
    "as related metadata, like the post date, caption or tags. We could also get\n",
    "links to all the images."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Putting a max in case the blog has millions of images\n",
    "#The given max will be rounded up to the nearest multiple of 50\n",
    "def tumblrImageScrape(blogName, maxImages = 200):\n",
    "    #Restating this here so the function isn't dependent on any external variables\n",
    "    tumblrAPItarget = 'http://{}.tumblr.com/api/read/json'\n",
    "\n",
    "    #There are a bunch of possible locations for the photo url\n",
    "    possiblePhotoSuffixes = [1280, 500, 400, 250, 100]\n",
    "\n",
    "    #These are the pieces of information we will be gathering,\n",
    "    #at the end we will convert this to a DataFrame.\n",
    "    #There are a few other datums we could gather like the captions\n",
    "    #you can read the Tumblr documentation to learn how to get them\n",
    "    #https://www.tumblr.com/docs/en/api/v1\n",
    "    postsData = {\n",
    "        'id' : [],\n",
    "        'photo-url' : [],\n",
    "        'date' : [],\n",
    "        'tags' : [],\n",
    "        'photo-type' : []\n",
    "    }\n",
    "\n",
    "    #Tumblr limits us to a max of 50 posts per request\n",
    "    for requestNum in range(maxImages // 50):\n",
    "        requestParams = {\n",
    "            'start' : requestNum * 50,\n",
    "            'num' : 50,\n",
    "            'type' : 'photo'\n",
    "        }\n",
    "        r = requests.get(tumblrAPItarget.format(blogName), params = requestParams)\n",
    "        requestDict = json.loads(r.text[len('var tumblr_api_read = '):-2])\n",
    "        for postDict in requestDict['posts']:\n",
    "            #We are dealing with uncleaned data, we can't trust it.\n",
    "            #Specifically, not all posts are guaranteed to have the fields we want\n",
    "            try:\n",
    "                postsData['id'].append(postDict['id'])\n",
    "                postsData['date'].append(postDict['date'])\n",
    "                postsData['tags'].append(postDict['tags'])\n",
    "            except KeyError as e:\n",
    "                raise KeyError(\"Post {} from {} is missing: {}\".format(postDict['id'], blogName, e))\n",
    "\n",
    "            foundSuffix = False\n",
    "            for suffix in possiblePhotoSuffixes:\n",
    "                try:\n",
    "                    photoURL = postDict['photo-url-{}'.format(suffix)]\n",
    "                    postsData['photo-url'].append(photoURL)\n",
    "                    postsData['photo-type'].append(photoURL.split('.')[-1])\n",
    "                    foundSuffix = True\n",
    "                    break\n",
    "                except KeyError:\n",
    "                    pass\n",
    "            if not foundSuffix:\n",
    "                #Make sure your error messages are useful\n",
    "                #You will be one of the users\n",
    "                raise KeyError(\"Post {} from {} is missing a photo url\".format(postDict['id'], blogName))\n",
    "\n",
    "    return pandas.DataFrame(postsData)\n",
    "tumblrImageScrape('lolcats-lol-cat', 50)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we have the urls of a bunch of images and can run OCR on them to gather\n",
    "compelling meme narratives, accompanied by cats.\n",
    "\n",
    "# Files\n",
    "\n",
    "What if the text we want isn't on a webpage? There are a many other sources of\n",
    "text available, typically organized into *files*.\n",
    "\n",
    "## Raw text (and encoding)\n",
    "\n",
    "The most basic form of storing text is as a _raw text_ document. Source code\n",
    "(`.py`, `.r`, etc) is usually raw text as are text files (`.txt`) and those with\n",
    "many other extension (e.g., .csv, .dat, etc.). Opening an unknown file with a\n",
    "text editor is often a great way of learning what the file is.\n",
    "\n",
    "We can create a text file in python with the `open()` function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#example_text_file = 'sometextfile.txt'\n",
    "#stringToWrite = 'A line\\nAnother line\\nA line with a few unusual symbols \\u2421 \\u241B \\u20A0 \\u20A1 \\u20A2 \\u20A3 \\u0D60\\n'\n",
    "stringToWrite = 'A line\\nAnother line\\nA line with a few unusual symbols ␡ ␛ ₠ ₡ ₢ ₣ ൠ\\n'\n",
    "\n",
    "with open(example_text_file, mode = 'w', encoding='utf-8') as f:\n",
    "    f.write(stringToWrite)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice the `encoding='utf-8'` argument, which specifies how we map the bits from\n",
    "the file to the glyphs (and whitespace characters like tab (`'\\t'`) or newline\n",
    "(`'\\n'`)) on the screen. When dealing only with latin letters, arabic numerals\n",
    "and the other symbols on America keyboards you usually do not have to worry\n",
    "about encodings as the ones used today are backwards compatible with\n",
    "[ASCII](https://en.wikipedia.org/wiki/ASCII), which gives the binary\n",
    "representation of 128 characters.\n",
    "\n",
    "Some of you, however, will want to use other characters (e.g., Chinese\n",
    "characters). To solve this there is\n",
    "[Unicode](https://en.wikipedia.org/wiki/Unicode) which assigns numbers to\n",
    "symbols, e.g., 041 is `'A'` and 03A3 is `'Σ'` (numbers starting with 0 are\n",
    "hexadecimal). Often non/beyond-ASCII characters are called Unicode characters.\n",
    "Unicode contains 1,114,112 characters, about 10\\% of which have been assigned.\n",
    "Unfortunately there are many ways used to map combinations of bits to Unicode\n",
    "symbols. The ones you are likely to encounter are called by Python _utf-8_,\n",
    "_utf-16_ and _latin-1_. _utf-8_ is the standard for Linux and Mac OS while both\n",
    "_utf-16_ and _latin-1_ are used by windows. If you use the wrong encoding,\n",
    "characters can appear wrong, sometimes change in number or Python could raise an\n",
    "exception. Lets see what happens when we open the file we just created with\n",
    "different encodings."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open(example_text_file, encoding='utf-8') as f:\n",
    "    print(\"This is with the correct encoding:\")\n",
    "    print(f.read())\n",
    "\n",
    "with open(example_text_file, encoding='latin-1') as f:\n",
    "    print(\"This is with the wrong encoding:\")\n",
    "    print(f.read())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice that with _latin-1_ the unicode characters are mixed up and there are too\n",
    "many of them. You need to keep in mind encoding when obtaining text files.\n",
    "Determining the encoding can sometime involve substantial work."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also load many text files at once. Lets start by looking at the Shakespeare files in the `data` directory "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open('./data/Shakespeare/midsummer_nights_dream.txt') as f:\n",
    "    midsummer = f.read()\n",
    "print(midsummer[-700:])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "By the way, depending on your working directory, you might get errors such as: [Errno 2] No such file or directory: '../data/Shakespeare/midsummer_nights_dream.txt.' Don't panic, it's nothing, just check your working directory. \n",
    "\n",
    "Then to load all the files in `./data/Shakespeare` we can use a for loop with `scandir`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "targetDir = './data/Shakespeare' #Change this to your own directory of texts\n",
    "shakespearText = []\n",
    "shakespearFileName = []\n",
    "\n",
    "for file in (file for file in os.scandir(targetDir) if file.is_file() and not file.name.startswith('.')):\n",
    "    with open(file.path, encoding=\"utf-8\") as f:\n",
    "        shakespearText.append(f.read())\n",
    "    shakespearFileName.append(file.name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then we can put them all in pandas DataFrame"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "shakespear_df = pandas.DataFrame({'text' : shakespearText}, index = shakespearFileName)\n",
    "shakespear_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Getting your text in a format like this is the first step of most analysis"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## PDF\n",
    "\n",
    "Another common way text will be stored is in a PDF file. First we will download\n",
    "a pdf in Python. To do that lets grab a chapter from\n",
    "_Speech and Language Processing_, chapter 21 is on Information Extraction which\n",
    "seems apt. It is stored as a pdf at [https://web.stanford.edu/~jurafsky/slp3/21.\n",
    "pdf](https://web.stanford.edu/~jurafsky/slp3/21.pdf) although we are downloading\n",
    "from a copy just in case Jurafsky changes their website."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#information_extraction_pdf = 'https://github.com/KnowledgeLab/content_analysis/raw/data/21.pdf'\n",
    "\n",
    "infoExtractionRequest = requests.get(information_extraction_pdf, stream=True)\n",
    "print(infoExtractionRequest.text[:1000])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It says `'pdf'`, so thats a good sign. The rest though looks like we are having\n",
    "issues with an encoding. The random characters are not caused by our encoding\n",
    "being wrong, however. They are cause by there not being an encoding for those\n",
    "parts at all. PDFs are nominally binary files, meaning there are sections of\n",
    "binary that are specific to pdf and nothing else so you need something that\n",
    "knows about pdf to read them. To do that we will be using\n",
    "[`PyPDF2`](https://github.com/mstamy2/PyPDF2), a PDF processing library for\n",
    "Python 3.\n",
    "\n",
    "\n",
    "Because PDFs are a very complicated file format pdfminer requires a large amount\n",
    "of boilerplate code to extract text, we have written a function that takes in an\n",
    "open PDF file and returns the text so you don't have to."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def readPDF(pdfFile):\n",
    "    #Based on code from http://stackoverflow.com/a/20905381/4955164\n",
    "    #Using utf-8, if there are a bunch of random symbols try changing this\n",
    "    codec = 'utf-8'\n",
    "    rsrcmgr = pdfminer.pdfinterp.PDFResourceManager()\n",
    "    retstr = io.StringIO()\n",
    "    layoutParams = pdfminer.layout.LAParams()\n",
    "    device = pdfminer.converter.TextConverter(rsrcmgr, retstr, laparams = layoutParams, codec = codec)\n",
    "    #We need a device and an interpreter\n",
    "    interpreter = pdfminer.pdfinterp.PDFPageInterpreter(rsrcmgr, device)\n",
    "    password = ''\n",
    "    maxpages = 0\n",
    "    caching = True\n",
    "    pagenos=set()\n",
    "    for page in pdfminer.pdfpage.PDFPage.get_pages(pdfFile, pagenos, maxpages=maxpages, password=password,caching=caching, check_extractable=True):\n",
    "        interpreter.process_page(page)\n",
    "    device.close()\n",
    "    returnedString = retstr.getvalue()\n",
    "    retstr.close()\n",
    "    return returnedString"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First we need to take the response object and convert it into a 'file like'\n",
    "object so that pdfminer can read it. To do this we will use `io`'s `BytesIO`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "infoExtractionBytes = io.BytesIO(infoExtractionRequest.content)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can give it to pdfminer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(readPDF(infoExtractionBytes)[:550])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "From here we can either look at the full text or fiddle with our PDF reader and\n",
    "get more information about individual blocks of text.\n",
    "\n",
    "## Word Docs\n",
    "\n",
    "The other type of document you are likely to encounter is the `.docx`, these are\n",
    "actually a version of [XML](https://en.wikipedia.org/wiki/Office_Open_XML), just\n",
    "like HTML, and like HTML we will use a specialized parser.\n",
    "\n",
    "For this class we will use [`python-docx`](https://python-\n",
    "docx.readthedocs.io/en/latest/) which provides a nice simple interface for\n",
    "reading `.docx` files"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#example_docx = 'https://github.com/KnowledgeLab/content_analysis/raw/data/example_doc.docx'\n",
    "\n",
    "r = requests.get(example_docx, stream=True)\n",
    "d = docx.Document(io.BytesIO(r.content))\n",
    "for paragraph in d.paragraphs[:7]:\n",
    "    print(paragraph.text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This procedure uses the `io.BytesIO` class again, since `docx.Document` expects\n",
    "a file. Another way to do it is to save the document to a file and then read it\n",
    "like any other file. If we do this we can either delete the file afterwords, or\n",
    "save it and avoid downloading the following time.\n",
    "\n",
    "This function is useful as a part of many different tasks so it and others like it will be added to the helper package `lucem_illud` so we can use it later without having to retype it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def downloadIfNeeded(targetURL, outputFile, **openkwargs):\n",
    "    if not os.path.isfile(outputFile):\n",
    "        outputDir = os.path.dirname(outputFile)\n",
    "        #This function is a more general os.mkdir()\n",
    "        if len(outputDir) > 0:\n",
    "            os.makedirs(outputDir, exist_ok = True)\n",
    "        r = requests.get(targetURL, stream=True)\n",
    "        #Using a closure like this is generally better than having to\n",
    "        #remember to close the file. There are ways to make this function\n",
    "        #work as a closure too\n",
    "        with open(outputFile, 'wb') as f:\n",
    "            f.write(r.content)\n",
    "    return open(outputFile, **openkwargs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This function will download, save and open `outputFile` as `outputFile` or just\n",
    "open it if `outputFile` exists. By default `open()` will open the file as read\n",
    "only text with the local encoding, which may cause issues if its not a text\n",
    "file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "try:\n",
    "    d = docx.Document(downloadIfNeeded(example_docx, example_docx_save))\n",
    "except Exception as e:\n",
    "    print(e)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We need to tell `open()` to read in binary mode (`'rb'`), this is why we added\n",
    "`**openkwargs`, this allows us to pass any keyword arguments (kwargs) from\n",
    "`downloadIfNeeded` to `open()`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "d = docx.Document(downloadIfNeeded(example_docx, example_docx_save, mode = 'rb'))\n",
    "for paragraph in d.paragraphs[:7]:\n",
    "    print(paragraph.text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can read the file with `docx.Document` and not have to wait for it to be\n",
    "downloaded every time."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# <font color=\"red\">Exercise 3</font>\n",
    "<font color=\"red\">Construct cells immediately below this that extract and organize textual content from text, PDF or Word into a pandas dataframe.</font>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# <font color=\"red\">Exercise 4</font>\n",
    "\n",
    "<font color=\"red\">In the two cells immediately following, describe a possible project (e.g., it might end up being your final project, but need not be if you are still searching): **WHAT** you will analyze--the texts you will select and the social game, world and actors you intend to learn about through your analysis (<100 words); **WHY** you will analyze these texts to learn about that context--justify the rationale behind your proposed sample design for this project, based on the readings. What is the social game, social work, or social actors about whom you are seeking to make inferences? What are the virtues of your proposed sample with respect to your research questions? What are its limitations? What are alternatives? What would be a reasonable path to \"scale up\" your sample for further analysis (i.e., high-profile publication)? (<150 words)? [**Note**: your individual or collective project will change over the course of the quarter as new data and/or analysis opportunities arise or if old ones fade away.] "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## ***What?*** \n",
    "<100 words"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## ***Why?***\n",
    "<150 words"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Sources:\n",
    "\n",
    "Other popular sources for internet data:\n",
    "\n",
    "[reddit](https://www.reddit.com/) - https://praw.readthedocs.io/en/v2.1.21/\n",
    "\n",
    "[twitter](https://twitter.com/) - https://pypi.org/project/python-twitter/\n",
    "\n",
    "[project gutenburg](https://www.gutenberg.org/) - https://github.com/ageitgey/Gutenberg \n",
    "\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}