src/API Calls within a Pandas Dataframe API using Pandas Apply.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![https://static.codingforentrepreneurs.com/media/cfe-blog/api-calls-pandas-dataframe-pd-apply/Pandas_x_APIs.jpg](https://static.codingforentrepreneurs.com/media/cfe-blog/api-calls-pandas-dataframe-pd-apply/Pandas_x_APIs.jpg)\n",
    "Original post is [here](https://www.codingforentrepreneurs.com/blog/api-calls-pandas-dataframe-pd-apply)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Whenever you're doing data analysis in Python your most likely using [Pandas](https://pandas.pydata.org/) since it's widely used and, most importantly, one of the best tools available for data analytics. Even better, pandas is 100% open-source.\n",
    "\n",
    "Pandas is only as good as the data it is given. Good data in, good data out. Garbage in, garbage out. That's the ethos of many data scientists around the globe.\n",
    "\n",
    "Since we have this amazing tool in Pandas, it's time we starting using another concept that you've probably heard of: APIs.\n",
    "\n",
    "Many APIs allow applications to automatically talk to one another, exchange data, and do so in a highly scalable and automated way.\n",
    "\n",
    "This post is going to show you a basic example of how you can use the [pandas apply](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.apply.html) method to enrich a dataframe with API data. \n",
    "\n",
    "\n",
    "A few times you might want to use an external API\n",
    "\n",
    "- Calculate shipping costs with Fedex/UPS/etc between your warehouse and your customer\n",
    "- Extract Latitude and Longitude from a street address or zipcode\n",
    "- Trigger an automation in Zapier\n",
    "- Email a customer an order status update\n",
    "\n",
    "Really, the list is endless. We're going to be focusing on integrating 1 API.\n",
    "\n",
    "> Running through this post is **10x** better if you use a [jupyter notebook](https://www.codingforentrepreneurs.com/blog/jupyter-notebook-server-aws-ec2-aws-vpc).\n",
    "\n",
    "### 1. The Google Geocoding API\n",
    "\n",
    "Keep in mind that Google's Geocoding API has a number of free requests but after a certain point, they start to charge you. \n",
    "\n",
    "1. Sign up on [Google Cloud Compute (GCP)](https://cloud.google.com)\n",
    "2. Login to [the console](https://console.developers.google.com).\n",
    "3. Go to the Geocoding API [here](https://console.cloud.google.com/marketplace/details/google/geocoding-backend.googleapis.com) click \"Enable\". If that link fails, do a search for *Geocoding API*, open it, and click \"Enable\"\n",
    "\n",
    "> You might have to create a project and do a few other configuration to get to the point where you can activate the Geocoding API\n",
    "\n",
    "4. Get your Google Cloud API Credentials:\n",
    "    - Do a search for \"Credentials\" within GCP\n",
    "    - Click \"Create Credentials\" > \"API Key\"\n",
    "    - Copy the key value, mine was `AIzaSyBSXMpu6lqd8kViIpy1GNWQ1symTXdMRzw` this is your Google Cloud API key. \n",
    "   \n",
    "> **Google Cloud Compute API Keys** have unrestricted privileges so it's highly recommend that you restrict the API key to the Geocoding API as well as to your local IP address.\n",
    "\n",
    "### 2. Install Requirements\n",
    "```\n",
    "pip install pandas requests\n",
    "```\n",
    "\n",
    "### 3. Let's code"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import datetime\n",
    "import pandas as pd\n",
    "import requests"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# our dataset\n",
    "data = {\"addresses\": ['Newport Beach, California', 'New York City', 'London, England', 10001, 'Sydney, Au']}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Calling DataFrame constructor on addresses list \n",
    "df = pd.DataFrame(data) \n",
    "df "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We only have 1 data point here. But what we're about to do can work for nearly any number of columns in a similar way.\n",
    "\n",
    "Let's take a look at how the [pandas apply](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.apply.html) works with a simple example."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# create a throw-away dataframe\n",
    "df_throwaway = df.copy()\n",
    "\n",
    "def apply_this_function(passed_row):\n",
    "    passed_row['new_col'] = True\n",
    "    passed_row['added'] = datetime.datetime.now()\n",
    "    return passed_row\n",
    "\n",
    "df_throwaway.apply(apply_this_function, axis=1) # axis=1 is important to use the row itself"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we see, the [pandas apply](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.apply.html) function works really well to add additional columns to our current columns. But we can go one step further, we can actually create new columns and row values based on what's within the row itself."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# create another throw-away dataframe\n",
    "df_example_2 = df.copy()\n",
    "\n",
    "def apply_this_other_function(row):\n",
    "    column_name = 'addresses'\n",
    "    address_value = row[column_name]\n",
    "    if isinstance(address_value, int):\n",
    "        row[column_name] = address_value * 2\n",
    "    return row\n",
    "\n",
    "df_example_2.apply(apply_this_other_function, axis=1) # axis=1 is important to use the row itself"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we see two main things about the `.apply` method, in each row we can:\n",
    "\n",
    "1. Add new columns based on other column's values (within that row)\n",
    "2. We can change the any value of any column within a single row)\n",
    "\n",
    "In other words, `.apply` enables us to change rows very dynamically. Now it's time to call our Geocoding API."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# create a working example. I like using a copy of the source data in case we make mistakes\n",
    "rest_api_df = df.copy()\n",
    "GOOGLE_API_KEY = 'your_api_key_from_above' \n",
    "\n",
    "def extract_lat_long_via_address(address_or_zipcode):\n",
    "    lat, lng = None, None\n",
    "    api_key = GOOGLE_API_KEY\n",
    "    base_url = \"https://maps.googleapis.com/maps/api/geocode/json\"\n",
    "    endpoint = f\"{base_url}?address={address_or_zipcode}&key={api_key}\"\n",
    "    # see how our endpoint includes our API key? Yes this is yet another reason to restrict the key\n",
    "    r = requests.get(endpoint)\n",
    "    if r.status_code not in range(200, 299):\n",
    "        return None, None\n",
    "    try:\n",
    "        '''\n",
    "        This try block incase any of our inputs are invalid. This is done instead\n",
    "        of actually writing out handlers for all kinds of responses.\n",
    "        '''\n",
    "        results = r.json()['results'][0]\n",
    "        lat = results['geometry']['location']['lat']\n",
    "        lng = results['geometry']['location']['lng']\n",
    "    except:\n",
    "        pass\n",
    "    return lat, lng\n",
    "    \n",
    "def enrich_with_geocoding_api(row):\n",
    "    column_name = 'addresses'\n",
    "    address_value = row[column_name]\n",
    "    address_lat, address_lng = extract_lat_long_via_address(address_value)\n",
    "    row['lat'] = address_lat\n",
    "    row['lng'] = address_lng\n",
    "    return row\n",
    "\n",
    "rest_api_df.apply(enrich_with_geocoding_api, axis=1) # axis=1 is important to use the row itself"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That's it. Pretty simple right? It's also very useful when you get into doing a lot of different API calls within your pandas workflows."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}