4. Accessing Data via APIs.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "!pip install -Uq requests\n",
    "import requests\n",
    "!mkdir -p data"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Using an API\n",
    "\n",
    "Learning outcomes\n",
    "\n",
    "- difference between API & webscraping\n",
    "- what JSON is (and why it's like a Python `dict`)\n",
    "- how to properly handle files in Python\n",
    "- what a REST API is\n",
    "- how to use the `requests` library\n",
    "\n",
    "## API versus web-scraping\n",
    "\n",
    "**Both are ways to sample data from the internet**\n",
    "\n",
    "API\n",
    "- structured\n",
    "- provided as a service (you are talking to a server via a REST API)\n",
    "- limited data / rate limits / paid / require auth (sometimes)\n",
    "- most will give back JSON (maybe XML or CSV)\n",
    "\n",
    "Web scraping\n",
    "- less structure\n",
    "- parsing HTML meant for your browser\n",
    "\n",
    "Neither is better than the other\n",
    "\n",
    "- API developer can limit what data is accessible through the API\n",
    "- API developer can not maintain the API\n",
    "- website page can change HTML structure\n",
    "- website page can have dynamic (Javascript) content that requires execution (usually done by the browser) before the correct HTML is available\n",
    "\n",
    "Much of the work in using an API is figuring out how to properly construct URLs for `GET` requests\n",
    "- requires looking at their documentation (& ideally a Python example!)\n",
    "\n",
    "## Where to find APIs\n",
    "\n",
    "- [ProgrammableWeb](https://www.programmableweb.com/apis/directory) - a collection of available API's\n",
    "- For the *Developer* or *For Developers* documentation on your favourite website\n",
    "- [public-apis/public-apis](https://github.com/public-apis/public-apis)\n",
    "\n",
    "## Using APIs\n",
    "\n",
    "Most APIs require authentication\n",
    "\n",
    "- so they API developer knows who you are\n",
    "- can charge you\n",
    "- can limit access\n",
    "- commonly via key or OAuth (both of which may be free)\n",
    "\n",
    "All the APIs we use here are unauthenticated - this is to avoid the time of you all signing up\n",
    "\n",
    "If your app requires authentication, it's usually done by passing in your credentials into the request (as a header)\n",
    "\n",
    "```python\n",
    "response = requests.get(url, auth=auth)\n",
    "```\n",
    "\n",
    "## JSON strings\n",
    "\n",
    "JSON (JavaScript Object Notation) is a:\n",
    "- lightweight data-interchange format (text)\n",
    "- easy for humans to read and write \n",
    "- easy for machines to parse and generate\n",
    "- based on key, value pairs\n",
    "\n",
    "You can think of the Python `dict` as JSON like:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = {'name': 'alan-turing'}\n",
    "data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "But true JSON is just a string (only text).  We can use `json.dumps` from the standard library to turn the `dict` into a JSON string:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = json.dumps(data)\n",
    "data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "type(data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can then use `json.loads` to turn this string back into a `dict`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "json.loads(data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Opening, reading & writing to files\n",
    "\n",
    "### Reading from a file\n",
    "\n",
    "We open files using the Python `open` builtin function, followed by a `read`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "open('./readme.md', 'r').read()[:100]"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "(Note the `./path` - the `.` refers to the current working directory.  It's not straightforward to know where this is - is it where the notebook is, or where the notebook server is running?\n",
    "\n",
    "If we wanted to read the file as separate lines, we could use `readlines()` (note we would still need to manually strip off the `\\n` characters later)\n",
    "\n",
    "(Note `\\n` is used as a newline indicator in text files - you never see it because your editor interprets it as a line break :)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "open('./readme.md', 'r').readlines()[:8]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Using `open`\n",
    "\n",
    "`open(path, mode)`\n",
    "\n",
    "Common values for the mode:\n",
    "- `r` read\n",
    "- `rb` read binary\n",
    "- `w+` write (`+` to create file if it doesn't exist)\n",
    "- `a` append\n",
    "\n",
    "Note there are options for both reading & writing - we actually use `open` for both reading & writing.\n",
    "\n",
    "We open a file using the Python builtin `open`, which is then followed by either a read or write stage\n",
    "\n",
    "- open the file\n",
    "- read the file OR write to the file\n",
    "\n",
    "Notice that the file is read in as a single string, with the newline character `\\n` separating lines\n",
    "\n",
    "- this is how all text files are structured\n",
    "- your editor does the line splitting for you\n",
    "\n",
    "### Writing to a file (without context management)\n",
    "\n",
    "We can write to a new file using the same `open` builtin\n",
    "\n",
    "- open the file\n",
    "- write to the file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "open('./data/output.data', 'w').write('We make this file to show how not to do it\\n')"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that we can do the same file write by explicitly assigning the file object to a variable (note the `a` to append)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fi = open('./data/output.data', 'a')\n",
    "fi.write('We make this file slightly differently to show how not to do it\\n')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The issue with the code above is that we aren't closing the file - we can fix this by intentionally closing the file.  \n",
    "\n",
    "One way to do this is to use `.close()` when we are done:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "fi = open('./data/output.data', 'a')\n",
    "fi.write('This time we close the file manually\\n')\n",
    "fi.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This requires us to remember to close (also an additional line).\n",
    "\n",
    "### Reading files with context management\n",
    "\n",
    "The Pythonic way of handling opening & closing of files is context management:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open('./readme.md', 'r') as fi:\n",
    "    data = fi.read()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The difference with using context management (which essentially just means using the `with` statement) is that it closes the file automatically when finished writing/reading."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Writing to a file with context management\n",
    "\n",
    "Using the context management syntax, we can save our `data` dict as JSON, using `json.dump` to write the dict to a file:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = {'name': 'alan turing'}\n",
    "with open('./data/output.json', 'w') as fi:\n",
    "    json.dump(data, fi)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's check it worked by loading the file, here using `json.load` to load from the file object:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open('./data/output.json', 'r') as fi:\n",
    "    data = json.load(fi)\n",
    "data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## REST APIs\n",
    "\n",
    "[REST - Wiki](https://en.wikipedia.org/wiki/Representational_state_transfer)\n",
    "\n",
    "REST is a set of constraints that allow **stateless communication of text data on the internet**\n",
    "\n",
    "- REST = REpresentational State Transfer\n",
    "- API = Application Programming Interface\n",
    "\n",
    "REST\n",
    "- communication of resources (located at URLs / URIs)\n",
    "- requests for a resource are responded to with a text payload (HTML, JSON etc)\n",
    "- these requests are made using HTTP (determines how messages are formatted, what actions (methods) can be taken)\n",
    "- common HTTP methods are `GET` and `POST`\n",
    "\n",
    "HTTP methods\n",
    "- GET - retrieve information about the REST API resource\n",
    "- POST - create a REST API resource\n",
    "- PUT - update a REST API resource\n",
    "- DELETE - delete a REST API resource or related component\n",
    "\n",
    "RESTful APIs enable you to develop any kind of web application having all possible CRUD (create, retrieve, update, delete) operations\n",
    "\n",
    "- can do anything we would want to do with a database\n",
    "\n",
    "*Further reading*\n",
    "- [Web Architecture 101](https://medium.com/storyblocks-engineering/web-architecture-101-a3224e126947) for more detail on how the web works"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Example - sunrise API\n",
    "\n",
    "Docs - https://sunrise-sunset.org/api\n",
    "\n",
    "First we need to form the url\n",
    "- use `?` to separate the API server name from the parameters for our request\n",
    "- use `&` to separate the parameters from each other\n",
    "- use `+` instead of space in the parameter"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# getting sunrise & sunset for Berlin today\n",
    "res = requests.get(\"https://api.sunrise-sunset.org/json?lat=52.5200&lng=13.4050\")\n",
    "data = res.json()\n",
    "data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This response is JSON - `requests.json()` turns it into a `dict`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "type(data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It's common to have a top level heirarchy to dig through to get the data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data.keys()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here the interesting stuff is in `results`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data['results']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise: \n",
    "Save the parameter `sunrise` as a string value by itself in a variable called `sunrise_time`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Example - Chronicling America API\n",
    "\n",
    "Docs - https://chroniclingamerica.loc.gov/about/api/ "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "term = \"germany\"\n",
    "fmt = \"json\"\n",
    "url = f\"https://chroniclingamerica.loc.gov/search/pages/results/?proxtext={term}&format={fmt}\"\n",
    "url"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We use the `requests` HTTP library to perform a `GET` request:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "response = requests.get(url)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "response"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## HTTP response\n",
    "\n",
    "What we recieved above is an *HTTP response**\n",
    "\n",
    "[HTTP Response - Wiki](https://en.wikipedia.org/wiki/Hypertext_Transfer_Protocol#Response_message)\n",
    "\n",
    "The response message consists of the following:\n",
    "\n",
    "- a status line which includes the status code and reason message (e.g., HTTP/1.1 200 OK, which indicates that the client's request succeeded)\n",
    "- response header fields (e.g., Content-Type: text/html)\n",
    "- an optional message body\n",
    "\n",
    "## What can we do with this HTTP response in `requests`?\n",
    "\n",
    "The Python builtin `dir` gives us all the attributes & methods of a Python object.\n",
    "\n",
    "This also includes all the `__` dunder (literally double-under) methods) - which we filter out using a list comprehension."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "[item for item in dir(response) if '__' not in item]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can get the HTTP status code (used to communicate things like everything OK (200), stop making requests etc - see [List of HTTP status codes - Wiki](https://en.wikipedia.org/wiki/List_of_HTTP_status_codes)):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "response.status_code"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The HTTP response headers:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "response.headers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And the response body:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "response.text[:1000]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "data = json.loads(response.text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can access the keys of the dictionary:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data.keys()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can access the values using the square bracket indexing with a key:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data['totalItems']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "While JSON is a simple text format, it can become complex due to\n",
    "\n",
    "- nesting (JSON inside JSON)\n",
    "- lists of JSON\n",
    "\n",
    "An example is our `items`, which has been parsed as a Python `list`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "type(data['items'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`items` is a list of dicts:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "item = data['items'][0]\n",
    "item.keys()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can iterate over both the keys and values as a pair using `items()`.\n",
    "\n",
    "Below we use a check  a quick check that the value isn't too long before printing:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from collections.abc import Iterable\n",
    "\n",
    "for k, v in item.items():\n",
    "    if isinstance(v, Iterable) and len(v) < 100:\n",
    "        print(f'{k}: {v}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's finish this exercise by only taking articles that appear between two years, and save those to disk.\n",
    "\n",
    "Normally you would apply this kind of filtering in the API request - we are going to filter in memory.\n",
    "\n",
    "First we need a bit of data cleaning of our date, which is an integer representation of time (but is a `str`):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "item['date']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here use `strptime` to convert the integer into a proper datetime:\n",
    "- ([Python's strftime directives](http://strftime.org/) is very useful!)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from datetime import datetime as dt\n",
    "dt.strptime(item['date'], \"%Y%m%d\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's put this data cleaning & filtering into a pipeline:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "term = \"germany\"\n",
    "fmt = \"json\"\n",
    "url = f\"https://chroniclingamerica.loc.gov/search/pages/results/?proxtext={term}&format={fmt}\"\n",
    "res = requests.get(url)\n",
    "data = res.json()\n",
    "items = data['items']\n",
    "\n",
    "start = 1900\n",
    "extract = []\n",
    "for item in items:\n",
    "    item['date'] = dt.strptime(item['date'], \"%Y%m%d\")\n",
    "    \n",
    "    if item['date'].year > start:\n",
    "        extract.append(item)\n",
    "        \n",
    "len(extract)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have a list of dictionaries, which plays very nice with `pandas`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install -q pandas\n",
    "import pandas as pd\n",
    "df = pd.DataFrame(extract)\n",
    "df.head(2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Example - downloading images\n",
    "\n",
    "We can also `requests` to download things other than text - such as images.\n",
    "\n",
    "Below we do a requests and see we get back a binary string:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "url = 'https://www.google.com/images/branding/googlelogo/2x/googlelogo_color_272x92dp.png'\n",
    "res = requests.get(url)\n",
    "res.text[:100]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can use context management to dump the contents of the binary string into a file:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open('./data/google-logo.png', 'wb') as fi:\n",
    "    fi.write(res.content)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We now have the Google logo locally (you may need to re-run this cell)\n",
    "\n",
    "![](./data/google-logo.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise - earthquake API \n",
    "\n",
    "Let's as a group write a program to get data from the USGS Earthquake Catalog - [documentation](https://earthquake.usgs.gov/fdsnws/event/1/#methods)\n",
    "\n",
    "Steps:\n",
    "- Make a folder (from your current working directory) to hold the earthquake data \n",
    "- Investigate the data from the response \n",
    "- Save each earthquake as its own JSON in the folder you created (hint - you will need to decide on a name for each earthquake JSON) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise - Wikipedia API\n",
    "\n",
    "Now for an open-ended exercise for you! Your task is to:\n",
    "- create a database of countries\n",
    "- in a folder called `countries` (you will need to make the folder)\n",
    "- each country in it's own folder\n",
    "- start with Germany\n",
    "\n",
    "V1 of your program should:\n",
    "- save the url you use to request the data\n",
    "- save the title\n",
    "- save the `line` parameter of each section (`data['parse']['sections']`)\n",
    "- save all in a single JSON\n",
    "\n",
    "V2 of your program should also:\n",
    "- save all '.png' & '.jpg' images as images, with the url as the image name\n",
    "- save all external links as CSV\n",
    "\n",
    "Much of the work will be understanding how the Wikipedia API works - useful resources are below:\n",
    "- [Main API page](https://www.mediawiki.org/wiki/API:Main_page)\n",
    "- [What the actions are](https://www.mediawiki.org/w/api.php)\n",
    "- [Python examples](https://github.com/wikimedia/mediawiki-api-demos/tree/master/python)\n",
    "\n",
    "Please also feel free to work on another API - happy to assist you with this as well :)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "python-classes",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.0 | packaged by conda-forge | (main, Oct 25 2022, 06:21:25) [Clang 14.0.4 ]"
  },
  "vscode": {
   "interpreter": {
    "hash": "33e49de180f7b68adbe0b8c3da4cce2849e5189a09c8fdb059671b22301aa1fe"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}