module-2/day-3-json-api/json-api.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Using APIs and JSON Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Objectives\n",
    "You will be able to:\n",
    "* Access and manipulate data inside a JSON file\n",
    "* Pull data from an API and parse/transform the data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Agenda\n",
    "\n",
    "* Review JSON Schemas\n",
    "* Introduce APIs.\n",
    "* Walk through how to make an API request. \n",
    "* Practice making API requests and Parsing the data.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### What is the difference between a JSON and a python dictionary?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Loading the JSON file\n",
    "\n",
    "As before, we begin by importing the json package, opening a file with python's built in function, and then loading that data in."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "f = open('data/output.json')\n",
    "data = json.load(f)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exploring JSON Schemas  \n",
    "\n",
    "Recall that JSON files have a nested structure. The most granular level of raw data will be individual numbers (float/int) and strings. These in turn will be stored in the equivalent of python lists and dictionaries. Because these can be combined, we'll start exploring by checking the type of our root object, and start mapping out the hierarchy of the json file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "type(data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you can see, in this case, the first level of the hierarchy is a dictionary. Let's explore what keys are within this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data.keys()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this case, there is only a single key, 'albums', so we'll continue on down the pathway exploring and mapping out the hierarchy. Once again, let's start by checking the type of this nested data structure."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "type(data['albums'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Another dictionary! So thus far, we have a dictionary within a dictionary. Once again, let's investigate what's within this dictionary (JSON calls the equivalent of Python dictionaries Objects.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data['albums'].keys()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "At this point, things are starting to look something like this: \n",
    "<img src=\"images\\json_diagram1.JPG\" width=550>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "At this point, if we were to continue checking individual data types, we have a lot to go through. To simplify this, let's use a for loop:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for key in data['albums'].keys():\n",
    "    print(key, type(data['albums'][key]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Adding this to our diagram we now have something like this:\n",
    "<img src=\"images\\json_diagram2.JPG\" width=550>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Normally, you may not draw out the full diagram as done here, but its a useful picture to have in mind, and in complex schemas, can be useful to map out. At this point, you also probably have a good idea of the general structure of the json file. However, there is still the list of items, which we could investigate further:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## What is an API?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Application Program Interfaces**, or APIs, are commonly used to retrieve data from remote websites. Sites like Reddit, Twitter, and Facebook all offer certain data through their APIs. \n",
    "\n",
    "To use an API, you make a request to a remote web server, and retrieve the data you need."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Python with two built-in modules, `urllib` and `urllib2` to handle these requests but these could be very confusing  and the documentation is not clear.\n",
    "\n",
    "To make these things simpler, one easy-to-use third-party library, known as` Requests`, is what most developers prefer to use it instead or urllib/urllib2. With this library, you can access content like web page headers, form data, files, and parameters via simple Python commands. It also allows you to access the response data in a simple way."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](images\\logo.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Below is how you would install and import the requests library before making any requests. \n",
    "```python\n",
    "# Uncomment and install requests if you dont have it already\n",
    "# !pip install requests\n",
    "\n",
    "# Import requests to working environment\n",
    "import requests\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import requests\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "## The `.get()` Method\n",
    "\n",
    "Now we have requests library ready in our working environment, we can start making some requests using the `.get()` method as shown below:\n",
    "\n",
    "\n",
    "We can use a simple GET request to retrieve information from the OpenNotify API.\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "OpenNotify has several API endpoints. An endpoint is a server route that is used to retrieve different data from the API. For example, the /comments endpoint on the Reddit API might retrieve information about comments, whereas the /users endpoint might retrieve data about users. To access them, you would add the endpoint to the base url of the API.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Make a get request to get the latest position of the international space station from the opennotify api.\n",
    "response = requests.get(\"http://api.open-notify.org/iss-now.json\")\n",
    "# Print the status code of the response.\n",
    "print(response.status_code)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "GET is by far the most used HTTP method. We can use GET request to retrieve data from any destination. \n",
    "GET is by far the most used HTTP method. We can use GET request to retrieve data from any destination. \n",
    "\n",
    "## Status Codes\n",
    "The request we make may not be always successful. The best way is to check the status code which gets returned with the response. Here is how you would do this. \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Code here \n",
    "response.status_code == requests.codes.ok"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So this is a good check to see if our request was successful. Depending on the status of the web server, the access rights of the clients and availibility of requested information. A web server may return a number of status codes within the response. Wikipedia has an exhaustive details on all these codes. [Check them out here](https://en.wikipedia.org/wiki/List_of_HTTP_status_codes)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Common status codes\n",
    "\n",
    "* 200 — everything went okay, and the result has been returned (if any)\n",
    "* 301 — the server is redirecting you to a different endpoint. This can happen when a company switches domain names, or an endpoint name is changed.\n",
    "* 401 — the server thinks you’re not authenticated. This happens when you don’t send the right credentials to access an API (we’ll talk about authentication in a later post).\n",
    "* 400 — the server thinks you made a bad request. This can happen when you don’t send along the right data, among other things.\n",
    "* 403 — the resource you’re trying to access is forbidden — you don’t have the right permissions to see it.\n",
    "* 404 — the resource you tried to access wasn’t found on the server."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Hitting the right endpoint\n",
    "iss-pass wasn’t a valid endpoint, so we got a 404 status code in response. We forgot to add `.json` at the end, as the API documentation states.\n",
    "\n",
    "We’ll now make a GET request to http://api.open-notify.org/iss-pass.json."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "## Response Contents\n",
    "Once we know that our request was successful and we have a valid response, we can check the returned information using `.text` property of the response object. \n",
    "```python\n",
    "print (resp.text)\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "response"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "response = requests.get(\"http://api.open-notify.org/iss-pass.json\")\n",
    "print(response.status_code)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Code here \n",
    "print(response.text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Query parameters\n",
    "\n",
    "If you look at the documentation for the OpenNotify API, we see that the ISS Pass endpoint requires two parameters.\n",
    "\n",
    "We can do this by adding an optional keyword argument, params, to our request. In this case, there are two parameters we need to pass:\n",
    "\n",
    "* lat — The latitude of the location we want.\n",
    "* lon — The longitude of the location we want.\n",
    "We can make a dictionary with these parameters, and then pass them into the requests.get function.\n",
    "\n",
    "We can also do the same thing directly by adding the query parameters to the url, like this: http://api.open-notify.org/iss-pass.json?lat=40.71&lon=-74.\n",
    "\n",
    "It’s almost always preferable to setup the parameters as a dictionary, because requests takes care of some things that come up, like properly formatting the query parameters.\n",
    "\n",
    "We’ll make a request using the coordinates of New York City, and see what response we get."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set up the parameters we want to pass to the API.\n",
    "# This is the latitude and longitude of New York City.\n",
    "parameters = {\"lat\": 40.71, \"lon\": -74}\n",
    "\n",
    "\n",
    "# Make a get request with the parameters.\n",
    "response = requests.get(\"http://api.open-notify.org/iss-pass.json\", params=parameters)\n",
    "\n",
    "\n",
    "# Print the content of the response (the data the server returned)\n",
    "print(response.content)\n",
    "# This gets the same data as the command aboveresponse = requests.get(\"http://api.open-notify.org/iss-pass.json?lat=40.71&lon=-74\")\n",
    "print(response.content)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "json.load(response)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So this returns a lot of information which by default is not really human understandable due to data encoding, HTML tags and other styling information that only a web browser can truly translate. In later lessons we shall look at how we can use ** Regular Exprerssions**  to clean this information and extract the required bits and pieces for analysis. \n",
    "\n",
    "## Response Headers\n",
    "The response of an HTTP request can contain many headers that holds different bits of information. We can use `.header` property of the response object to access the header information as shown below:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Code here \n",
    "dict(response.headers)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The content of the headers is our required element. You can see the key-value pairs holding various pieces of  information about the resource and request. Let's try to parse some of these values using the requests library:\n",
    "\n",
    "```python\n",
    "print(resp.headers['Content-Length'])  # length of the response\n",
    "print(resp.headers['Date'])  # Date the response was sent\n",
    "print(resp.headers['server'])   # Server type (google web service - GWS)\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "## HTTP POST method \n",
    "\n",
    "Sometimes we need to send one or more files simultaneously to the server. For example, if a user is submitting a form and the form includes different fields for uploading files, like user profile picture, user resume, etc. Requests can handle multiple files on a single request. This can be achieved by putting the files to a list of tuples in the form (`field_name, file_info)`.\n",
    "\n",
    "\n",
    "```python\n",
    "import requests\n",
    "\n",
    "url = 'http://httpbin.org/post'  \n",
    "file_list = [  \n",
    "    ('image', ('fi.png', open('fi.png', 'rb'), 'image/png')),\n",
    "    ('image', ('fi2.jpeg', open('fi2.jpeg', 'rb'), 'image/png'))\n",
    "]\n",
    "\n",
    "r = requests.post(url, files=file_list)  \n",
    "print(r.text)  \n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](quota.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "## Generating Access Tokens\n",
    "\n",
    "As discussed, in order to use many APIs, one needs to use OAuth which requires an access token. As such, our first step will be to generate this login information so that we can start making some requests.  \n",
    "\n",
    "With that, lets go grab an access token from an API site and make some API calls!\n",
    "Point your browser over to this [yelp page](https://www.yelp.com/developers/v3/manage_app) and start creating an app in order to obtain and api access token:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "![](./images/yelp_app.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can either sign in to an existing Yelp account, or create a new one, if needed.\n",
    "\n",
    "On the page you see above, simply fill out some sample information such as \"Flatiron Edu API Example\" for the app name, or whatever floats your boat. Afterwards, you should be presented with an API key that you can use to make requests!\n",
    "\n",
    "With that, it's time to start making some api calls!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#As a general rule of thumb, don't store passwords in a main file like this!\n",
    "#Instead, you would normally store those passwords under a sub file like passwords.py which you would then import.\n",
    "#Or even better, as an environment variable that could then be imported!\n",
    "#For now, we'll simply hardcode them into our notebook for simplicity.\n",
    "client_id = 'bVX1Jsfp4dkIOqw5HOVplg' #Your client ID goes here (as a string)\n",
    "api_key = 'RTzp-q-TgkJW_NFQogubFvZNRDziXyoR38VbtZMWibDI-FlvB25OE7GmafFEqhTL8_Bk2HlcX24-hRWLMP7Nc6WHO_VXMXldpPBjP0LoPv5EFFELMSI2oll8njhbXHYx' #Your api key goes here (as a string)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## An Example Request with OAuth <a id=\"oauth_request\"></a>\n",
    "https://www.yelp.com/developers/documentation/v3/get_started\n",
    "\n",
    "In the next lesson, we'll further dissect how to read and translate online documentation like the link here. For now, let's simply look at an example request and dissect it into its consituent parts:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "term = 'Mexican'\n",
    "location = 'Astoria NY'\n",
    "SEARCH_LIMIT = 10\n",
    "\n",
    "url = 'https://api.yelp.com/v3/businesses/search'\n",
    "\n",
    "headers = {\n",
    "    'Authorization': 'Bearer {}'.format(api_key),\n",
    "}\n",
    "\n",
    "url_params = {\n",
    "    'term': term.replace(' ', '+'),\n",
    "    'location': location.replace(' ', '+'),\n",
    "    'limit': SEARCH_LIMIT\n",
    "}\n",
    "response = requests.get(url, headers=headers, params=url_params)\n",
    "print(response)\n",
    "print(type(response.text))\n",
    "print(response.text[:1000])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Breaking Down the Request\n",
    "\n",
    "As you can see, there are three main parts to our request.  \n",
    "  \n",
    "They are:\n",
    "* The url\n",
    "* The header\n",
    "* The parameters\n",
    "  \n",
    "The url is fairly straightforward and is simply the base url as described in the documentation (again more details in the upcoming lesson).\n",
    "\n",
    "The header is a dictionary of key-value pairs. In this case, we are using a fairly standard header used by many APIs. It has a strict form where 'Authorization' is the key and 'Bearer YourApiKey' is the value.\n",
    "\n",
    "The parameters are the filters which we wish to pass into the query. These will be embedded into the url when the request is made to the api. Similar to the header, they form key-value pairs. Valid key parameters by which to structure your queries, are described in the API documentation which we'll look at further shortly. A final important note however, is the need to replace spaces with \"+\". This is standard to many requests as URLs cannot contain spaces. (Note that the header itself isn't directly embedded into the url itself and as such, the space between 'Bearer' and YourApiKey is valid.)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "## The Response\n",
    "\n",
    "As before, our response object has both a status code, as well as the data itself. With that, let's start with a little data exploration!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "response.json().keys()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's go a bit further and start to preview what's stored in each of the values for these keys.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for key in response.json().keys():\n",
    "    print(key)\n",
    "    value = response.json()[key] \n",
    "    print(type(value), sep=\"\\n\\n\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's continue to preview these further to get a little better acquainted.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "yelp_data =response.json()\n",
    "yelp_data['businesses'][:2]\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you can see, we're primarily interested in the 'bussinesses' entry. \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Summary <a id=\"sum\"></a>\n",
    "\n",
    "Congratulations! We've covered a lot here! We took some of your previous knowledge with HTTP requests and OAuth in order to leverage an enterprise API! Then we made some requests to retrieve information that came back as a json format. We then transformed this data into a dataframe using the Pandas package. In the next lab, we'll break down how to read API documentation and then put it all together to make a nifty map!"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}