lectures/1 APIs/1c Getting census data.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "coated-diving",
   "metadata": {},
   "source": [
    "# Getting census data"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "familiar-professor",
   "metadata": {},
   "source": [
    "## Lecture objectives\n",
    "\n",
    "1. Provide more experience with APIs and `requests`\n",
    "2. Demonstrate how to access census data"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "sticky-harmony",
   "metadata": {},
   "source": [
    "Traditionally, if you wanted census data, you had to download .csv or other files and decipher them. Recently, the census has introduced an API. [See the documentation here](https://www.census.gov/data/developers/guidance/api-user-guide.Example_API_Queries.html).\n",
    "\n",
    "If you request more than 500 queries a day, you'll need to register for a (free) [API key](https://www.census.gov/data/developers/guidance/api-user-guide.Help_&_Contact_Us.html) from the Census Bureau.\n",
    "\n",
    "Let's download population by county from the 2021 American Community Survey five-year estimates. We see from the documentation that the API call takes the following form:\n",
    "\n",
    "`https://api.census.gov/data/YEAR/acs/DATASET?get=TABLENAME&for=GEOGRAPHY`\n",
    "\n",
    "So getting the population (variable `B01001_001E`) for all counties is:\n",
    "\n",
    "`https://api.census.gov/data/2021/acs/acs5?get=B01001_001E&for=county`\n",
    "\n",
    "Try this in your browser.\n",
    "\n",
    "Now let's get it into Python."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "recovered-marketplace",
   "metadata": {},
   "outputs": [],
   "source": [
    "import requests\n",
    "r = requests.get('https://api.census.gov/data/2021/acs/acs5?get=B01001_001E&for=county')\n",
    "print(type(r.text))\n",
    "# This time, it looks like the data come in a string\n",
    "print(r.text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "behavioral-particular",
   "metadata": {},
   "outputs": [],
   "source": [
    "# But turns out that it's actually a JSON\n",
    "censusdata = r.json()\n",
    "type(censusdata)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "radio-survival",
   "metadata": {},
   "outputs": [],
   "source": [
    "# The JSON format is a list of lists. The first sublist is the column headers\n",
    "censusdata[:5] # show the first five rows"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "surgical-miller",
   "metadata": {},
   "outputs": [],
   "source": [
    "# So we can also convert this to a pandas dataframe, if we use the first list as the column names\n",
    "# Note that the state and county are shown by their FIPS codes\n",
    "import pandas as pd\n",
    "df = pd.DataFrame(censusdata[1:], columns=censusdata[0])\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "43858c10",
   "metadata": {},
   "source": [
    "Let's rename the column to something more meaningful. `pandas` has a helpful `rename` function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "872ea872",
   "metadata": {},
   "outputs": [],
   "source": [
    "df.rename?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "150258e3",
   "metadata": {},
   "outputs": [],
   "source": [
    "# note the inplace keyword changes the dataframe in place, rather than returning a copy\n",
    "df.rename(columns = {'B01001_001E':'population'}, inplace=True)\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ec6f4008",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-block alert-info\">\n",
    "<strong>Exercise:</strong> Explore the Census Bureau API documentation. How would you get, for example, census tract level estimates for population by gender or race?\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "excellent-spank",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-block alert-info\">\n",
    "<h3>Key Takeaways</h3>\n",
    "<ul>\n",
    "  <li>Getting census data is one of the most common tasks we'll do in this course.</li>\n",
    "  <li>The Census Bureau has a well-documented API, that may be useful for more specialized queries.</li>\n",
    "</ul>\n",
    "</div>"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}