notebooks/class7-accessing-apis.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Accessing APIs\n",
    "\n",
    "*I wrote version 1 of this notebook in 2019, based off the tutorial by [Allison Parrish](http://www.decontextualize.com/). It was updated in 2020 by Dan Sinykin and again by me in 2021.* \n",
    "\n",
    "Where we are for today's class is as follows: We all agree that we want to do data science with text. We know we can acquire text from digital libraries, and we've also learned how to acquire text via web scraping. But sometimes our best (by which we mean easiest and most efficient) option is to acquire text through web APIs.\n",
    "\n",
    "Before we go further, what even is an API? \n",
    "\n",
    "[Here](https://medium.com/epfl-extension-school/an-illustrated-introduction-to-apis-10f8000313b9) is a detailed explanation (which you should have read for today's class). But all you really need to know for now is this:\n",
    "\n",
    "**A web API is some collection of data, made available on the web, provided in a format easy for computers to parse.**\n",
    "\n",
    "Note also that the data can be text, but doesn't have to be.\n",
    "\n",
    "Twitter's [Developer APIs](https://developer.twitter.com/) are probably the most commonly used APIs by social scientists and computational linguists seeking to analyze text. You might want to use them for your final project.\n",
    "\n",
    "But let's start with something simple:\n",
    "\n",
    "[Yes or No?](https://yesno.wtf/) (more fun ones on [this list](https://dev.to/mkrl/apis-you-didnt-know-you-needed-38c))\n",
    "\n",
    "**Reload \"Yes or No?\" a few times. What does it do?**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, if we wanted to use this for a project, we *could* scrape the HTML in the way we've just learned. \n",
    "\n",
    "But the site's API gives us an easier way!\n",
    "\n",
    "Take a look:\n",
    "[Yes or No API?](https://yesno.wtf/#api)\n",
    "\n",
    "**What do you think this does?**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Can anyone identify the data format that's being used?**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And since Python has lots of ways for dealing with json, we can use this API in a Very Important Decision-Making Program:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import requests # remember this from last class\n",
    "\n",
    "url = \"https://yesno.wtf/api/\" # note the url is slightly different than the human-facing URL\n",
    "response = requests.get(url)\n",
    "data_dict = response.json() # this turns the json into a dictionary that can be accessed by key/value pairs\n",
    "\n",
    "print(\"This is what 'data' looks like as a string: \\n\" + str(data))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "But because our `data_dict` object is, as suggeted, a dictionary, and because the API documentation (and what's printed above) tells us the name of each key, we can also access each of these keys as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"Key: answer. Value: \" + data_dict['answer'])\n",
    "print(\"Key: forced. Value: \" + data_dict['forced'])\n",
    "print(\"Key: image. Value: \" + data_dict['image'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**This breaks halfway through... why?**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's try again!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"Key: answer. Value: \" + data_dict['answer'])\n",
    "print(\"Key: forced. Value: \" + str(data_dict['forced']))\n",
    "print(\"Key: image. Value: \" + data_dict['image'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**NOTE:** For those of you wanting more information about / practice with dictionaries, I recommend working through [this notebook](dictionaries-sets-tuples.ipynb).\n",
    "\n",
    "But for now, let's get back to our Very Important Decision-Making Program:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# remember that we've already imported the requests library and assigned the \"url\" variable up above\n",
    "# so we can just do this as many times as we want \n",
    "\n",
    "response = requests.get(url) \n",
    "data_dict = response.json() \n",
    "print(\"Answer to your very important question: \" + data_dict['answer'] + \"!\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So you can (hopefully) already start to see how this might be useful in text analysis projects. Especially given the amount of time that we've already spent learning how to scrape the web.\n",
    "\n",
    "But how does it work?\n",
    "\n",
    "Let's slow it down and look at all of the parts. \n",
    "\n",
    "Here again is the code for our Very Important Decision-Making Program:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "url = \"https://yesno.wtf/api/\"\n",
    "response = requests.get(url)\n",
    "data = response.json() # this turns the json into a dictionary that can be accessed by key/value pairs\n",
    "data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice is that it starts with a URL, just like a regular website. \n",
    "\n",
    "And you can [go to the same URL in your web browser](https://yesno.wtf/api/) and see a version of the same thing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So now let's talk a bit more about URLs\n",
    "\n",
    "### URLs\n",
    "\n",
    "A URL (\"uniform resource locator\") uniquely identifies a document on the web, and provides instructions for how to access it. Most of what we do on the web---whether we're using a web browser or writing a program that accesses the web---boils down to manipulating URLs.\n",
    "\n",
    "It's important to understand the structure of URLs so we can take them apart and put them back together (both in our heads and programmatically). URLs have a conventional structure that is specified in Internet standards documentation, and many web APIs assume knowledge of this structure. So let's take the following URL:\n",
    "\n",
    "    http://www.example.com/foo/bar?arg1=baz&arg2=quux\n",
    "    \n",
    "... and break it down into parts, so we have a common vocabulary.\n",
    "\n",
    "| Part | Name |\n",
    "|------|------|\n",
    "| `http` | scheme |\n",
    "| `www.example.com` | host |\n",
    "| `/foo/bar` | path |\n",
    "| `?arg1=baz&arg2=quux` | query string |\n",
    "\n",
    "All of these parts are required, except for the query string, which is optional. Explanations:\n",
    "\n",
    "* The *scheme* determines what *protocol* will be used to access this resource. For our purposes, this will almost always be `http` (HyperText Transfer Protocol) or `https` (HTTP, but over an encrypted connection).\n",
    "* The *host* specifies which server on the Internet we're going to talk to in order to retrieve the document we want.\n",
    "* The *path* names a resource on the server, often using slashes (`/`) to represent hierarchical relationships between resources. (Sometimes this corresponds to actual files on the server, but just as often it does not.)\n",
    "* The *query string* is a means to tell the server *how* we want the document delivered. (More examples of this soon.)\n",
    "\n",
    "Most of the work you'll do in learning how to use a web API is learning how to construct and manipulate URLs. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### HTML, JSON and web APIs\n",
    "\n",
    "The most common format for documents on the web is HTML (HyperText Markup Language). HTML was specifically designed to be a tool for creating web pages, and it excels at that. But it's not so great for describing structured data. \n",
    "\n",
    "But another popular format---and the format we'll be learning how to work with today---is JSON (JavaScript Object Notation). Like HTML, JSON is a format for exchanging structured data between two computer programs. \n",
    "\n",
    "You saw some JSON already when we accessed the reponses to the \"Yes or No\" API via our web browser just a minute ago. So to review:\n",
    "\n",
    "Yes or No has both a human-readable version of its page found at this URL, written in HTML:\n",
    "\n",
    "> https://yesno.wtf/\n",
    "\n",
    "and a version of the same content designed to be easily readable by computers. This is the URL, and it returns a document in JSON format, as we saw:\n",
    "\n",
    "> https://yesno.wtf/api/\n",
    "\n",
    "Every web site makes available a number of URLs that return human-readable documents; and many web sites (like Yes or No, or Twitter, or Genius, which we'll get to next) also make available URLs that return documents intended to be read by computer programs. \n",
    "\n",
    "Often---as is the case with Facebook, for example-- these are just two views into the same data.\n",
    "\n",
    "So, another way to define a web API is as follows: \n",
    "\n",
    "**API: A set of URLs, and rules for manipulating URLs, that a website makes available and that are intended to be read by computer programs.** \n",
    "\n",
    "(API stands for \"application programming interface\"; a \"web API\" is an interface enables you to program applications that use the web site's data.)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### A Note about API Keys\n",
    "\n",
    "Very often, when playing around with APIs via your web browser, you will see this message (or one like it):\n",
    "\n",
    "    {\"message\": \"unauthorized\", \"type\": \"error\"}\n",
    "\n",
    "This message results from the fact that most web APIs (unlike most web pages) require some kind of *authentication*. \n",
    "\n",
    "In many APIs, this takes the form of a \"token\" or \"key\" (also sometimes called a \"client ID\" and/or \"secret\")---usually this informtsion ends up as an extra parameter that you pass on the end of the URL. It identifies the request as having come from a unique user. \n",
    "\n",
    "So how do you get \"keys\" or \"tokens\"? There's usually some kind of sign-up form in or near the developer documentation for the service in question.\n",
    "\n",
    "Different services have different requirements regarding how to include your keys in your request; you'll have to consult the documentation to know for sure. And truth be told, consult the documentation for the particular API you want to use is probably the most important lesson about learning to use APIs (and it's why most of the next Quiz will involve just that). "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The API we'll be using for the next exercise, Wordnik, requires a key. For our purposes, we will use a pre-existing one which you'll see below. (For your next quiz, which involves using the Genius API, you'll need to sign up for a key of your own).  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Making API requests with `requests`\n",
    "\n",
    "All you need to make (most) web API requests is a web browser. But it would be tedious to do these requests in a web browser and then copy over the responses into Python for analysis. We may also want to make *many* requests to a web API (for example, to get the song IDs for *all* of the songs included in our candidate playlist), which is inconvenient to do \"by hand\" with a web browser. \n",
    "\n",
    "Ideally, there would be some way to make web requests *directly inside a Python program* and in fact, we've already used it: the [requests](http://docs.python-requests.org/en/master/) package.\n",
    "\n",
    "Here's an example of how to use `requests` to get the contents of a document intended to be read by computers. In this case, the request is for the definition of the word \"computer\":"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import requests\n",
    "\n",
    "api_key = \"a80a5131f7620c32a8919063dce09d01b6239543e3d0063bf\"\n",
    "url = \"http://api.wordnik.com:80/v4/word.json/API/definitions?api_key=\" + api_key # notice string concatination \n",
    "response = requests.get(url)\n",
    "data = response.json()\n",
    "data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Oh hey wow look at all the deets it gives us about the word \"API\"--that's pretty rad! We'll break down *how* exactly to know what the URL for a particular resource is a bit later (and how to add the API key to the request). But for now, let's just note a few key features.\n",
    "\n",
    "As with the Yes or No example above, the following lines are most important:\n",
    "\n",
    "    response = requests.get(url)\n",
    "    data = response.json()\n",
    "\n",
    "The first line calls the `get()` function in the `requests` package, with one parameter, the URL that you want to fetch (which we previously stored in a variable called `url`). \n",
    "\n",
    "When this function gets called, the `requests` library makes a network request to the specified URL and retrieves its contents, returning a special kind of value called a \"response,\" which contains information about the response generated by the remote server, along with the content of that response. \n",
    "\n",
    "Note that this is the exact same line that we used to request the contents of the NYT page when we were web scraping. \n",
    "\n",
    "The next line is different. With the NYT page, we turned the response into text with the line\n",
    "\n",
    "    html_str = response.text\n",
    "\n",
    "Here, we are using the `.json()` method, which takes data in the response in JSON format (if present) and converts it to the corresponding Python data structure. (Note that [response objects have many other methods and attributes as well](http://docs.python-requests.org/en/master/api/#requests.Response), but the one we're most interested in right now is `.json()`). \n",
    "\n",
    "    data = response.json()\n",
    "\n",
    "If you're familiar looking at Python code, you might observe that the json object is just a list of dictionaries. That's why it starts with a `'['` up above and then seems to have sets of curly braces (`{ ... }`). \n",
    "\n",
    "This is helpful to know, since we can use standard ways of iterating through and accessing lists and dictionaries. For example, we can use a `for` loop to print out the text of each definition of \"hello.\":"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for item in data:\n",
    "    print(\"Definition: \" + item['text'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The combination of JSON, dictionaries, and for-loops is the magic stew that will allow us to automate the gathering of text from web APIs, making it often the most convenient method for developing a corpus for performing data science with text.\n",
    "\n",
    "And believe it or not, you now know (almost) everything you need to get started with APIs.\n",
    "\n",
    "Hurray!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}