examples/notebooks/us_census_data.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "0",
   "metadata": {},
   "source": [
    "<a href=\"https://githubtocolab.com/gee-community/geemap/blob/master/examples/notebooks/us_census_data.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open in Colab\"/></a>\n",
    "\n",
    "Uncomment the following line to install [geemap](https://geemap.org) if needed."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1",
   "metadata": {},
   "source": [
    "# U.S. Census Data \n",
    "\n",
    "The United States Census Bureau Topologically Integrated Geographic Encoding and Referencing (TIGER) dataset contains the 2018 boundaries for the primary governmental divisions of the United States. In addition to the fifty states, the Census Bureau treats the District of Columbia, Puerto Rico, and each of the island areas (American Samoa, the Commonwealth of the Northern Mariana Islands, Guam, and the U.S. Virgin Islands) as the statistical equivalents of States for the purpose of data presentation. Each feature represents a state or state equivalent.\n",
    "\n",
    "For full technical details on all TIGER 2018 products, see the [TIGER technical documentation](https://www2.census.gov/geo/pdfs/maps-data/data/tiger/tgrshp2018/TGRSHP2018_TechDoc.pdf).\n",
    "\n",
    "* [TIGER: US Census States](https://developers.google.com/earth-engine/datasets/catalog/TIGER_2018_States): `ee.FeatureCollection(\"TIGER/2018/States\")`\n",
    "* [TIGER: US Census Counties](https://developers.google.com/earth-engine/datasets/catalog/TIGER_2018_Counties): `ee.FeatureCollection(\"TIGER/2018/Counties\")`\n",
    "* [TIGER: US Census Tracts](https://developers.google.com/earth-engine/datasets/catalog/TIGER_2010_Tracts_DP1): `ee.FeatureCollection(\"TIGER/2010/Tracts_DP1\")`\n",
    "* [TIGER: US Census Blocks](https://developers.google.com/earth-engine/datasets/catalog/TIGER_2010_Blocks): `ee.FeatureCollection(\"TIGER/2010/Blocks\")`\n",
    "* [TIGER: US Census Roads](https://developers.google.com/earth-engine/datasets/catalog/TIGER_2016_Roads): `ee.FeatureCollection(\"TIGER/2016/Roads\")`\n",
    "* [TIGER: US Census 5-digit ZIP Code](https://developers.google.com/earth-engine/datasets/catalog/TIGER_2010_ZCTA5): `ee.FeatureCollection(\"TIGER/2010/ZCTA5\")`"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2",
   "metadata": {},
   "source": [
    "## Install Earth Engine API and geemap\n",
    "Install the [Earth Engine Python API](https://developers.google.com/earth-engine/python_install) and [geemap](https://github.com/gee-community/geemap). The **geemap** Python package is built upon the [ipyleaflet](https://github.com/jupyter-widgets/ipyleaflet) and [folium](https://github.com/python-visualization/folium) packages and implements several methods for interacting with Earth Engine data layers, such as `Map.addLayer()`, `Map.setCenter()`, and `Map.centerObject()`.\n",
    "The following script checks if the geemap package has been installed. If not, it will install geemap, which automatically installs its [dependencies](https://github.com/gee-community/geemap#dependencies), including earthengine-api, folium, and ipyleaflet."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Installs geemap package\n",
    "import subprocess\n",
    "\n",
    "try:\n",
    "    import geemap\n",
    "except ImportError:\n",
    "    print(\"geemap package not installed. Installing ...\")\n",
    "    subprocess.check_call([\"python\", \"-m\", \"pip\", \"install\", \"geemap\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4",
   "metadata": {},
   "outputs": [],
   "source": [
    "import ee\n",
    "import geemap"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5",
   "metadata": {},
   "source": [
    "## TIGER: US Census States\n",
    "\n",
    "https://developers.google.com/earth-engine/datasets/catalog/TIGER_2018_States\n",
    "\n",
    "![](https://i.imgur.com/DAESBK4.png)\n",
    "\n",
    "### Displaying data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6",
   "metadata": {},
   "outputs": [],
   "source": [
    "Map = geemap.Map(center=[40, -100], zoom=4)\n",
    "\n",
    "states = ee.FeatureCollection(\"TIGER/2018/States\")\n",
    "\n",
    "Map.centerObject(states, 4)\n",
    "Map.addLayer(states, {}, \"US States\")\n",
    "\n",
    "Map.addLayerControl()  # This line is not needed for ipyleaflet-based Map\n",
    "Map"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7",
   "metadata": {},
   "source": [
    "### Displaying vector as raster"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8",
   "metadata": {},
   "outputs": [],
   "source": [
    "Map = geemap.Map(center=[40, -100], zoom=4)\n",
    "\n",
    "states = ee.FeatureCollection(\"TIGER/2018/States\")\n",
    "image = ee.Image().paint(states, 0, 2)\n",
    "\n",
    "Map.centerObject(states, 4)\n",
    "Map.addLayer(image, {}, \"US States\")\n",
    "\n",
    "Map.addLayerControl()\n",
    "Map"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9",
   "metadata": {},
   "source": [
    "### Select by attribute\n",
    "\n",
    "#### Select one single state"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "10",
   "metadata": {},
   "outputs": [],
   "source": [
    "Map = geemap.Map(center=[40, -100], zoom=4)\n",
    "\n",
    "tn = ee.FeatureCollection(\"TIGER/2018/States\").filter(ee.Filter.eq(\"NAME\", \"Tennessee\"))\n",
    "\n",
    "Map.centerObject(tn, 6)\n",
    "Map.addLayer(tn, {}, \"Tennessee\")\n",
    "\n",
    "Map.addLayerControl()\n",
    "Map"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "11",
   "metadata": {},
   "outputs": [],
   "source": [
    "tn = (\n",
    "    ee.FeatureCollection(\"TIGER/2018/States\")\n",
    "    .filter(ee.Filter.eq(\"NAME\", \"Tennessee\"))\n",
    "    .first()\n",
    ")\n",
    "\n",
    "props = tn.toDictionary().getInfo()\n",
    "print(props)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "12",
   "metadata": {},
   "source": [
    "#### Select multiple states"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "13",
   "metadata": {},
   "outputs": [],
   "source": [
    "Map = geemap.Map(center=[40, -100], zoom=4)\n",
    "\n",
    "selected = ee.FeatureCollection(\"TIGER/2018/States\").filter(\n",
    "    ee.Filter.inList(\"NAME\", [\"Tennessee\", \"Alabama\", \"Georgia\"])\n",
    ")\n",
    "\n",
    "Map.centerObject(selected, 6)\n",
    "Map.addLayer(selected, {}, \"Selected states\")\n",
    "\n",
    "Map.addLayerControl()\n",
    "Map"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "14",
   "metadata": {},
   "source": [
    "#### Printing all values of a column"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "15",
   "metadata": {},
   "outputs": [],
   "source": [
    "states = ee.FeatureCollection(\"TIGER/2018/States\").sort(\"ALAND\", False)\n",
    "names = states.aggregate_array(\"STUSPS\").getInfo()\n",
    "print(names)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "16",
   "metadata": {},
   "outputs": [],
   "source": [
    "areas = states.aggregate_array(\"ALAND\").getInfo()\n",
    "print(areas)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "17",
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "\n",
    "%matplotlib notebook"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "18",
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.bar(names, areas)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "19",
   "metadata": {},
   "source": [
    "#### Descriptive statistics of a column\n",
    "\n",
    "For example, we can calculate the total land area of all states:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "20",
   "metadata": {},
   "outputs": [],
   "source": [
    "states = ee.FeatureCollection(\"TIGER/2018/States\")\n",
    "area_m2 = states.aggregate_sum(\"ALAND\").getInfo()\n",
    "area_km2 = area_m2 / 1000000\n",
    "print(\"Total land area: \", area_km2, \" km2\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "21",
   "metadata": {},
   "outputs": [],
   "source": [
    "states = ee.FeatureCollection(\"TIGER/2018/States\")\n",
    "stats = states.aggregate_stats(\"ALAND\").getInfo()\n",
    "print(stats)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "22",
   "metadata": {},
   "source": [
    "#### Add a new column to the attribute table"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "23",
   "metadata": {},
   "outputs": [],
   "source": [
    "states = ee.FeatureCollection(\"TIGER/2018/States\").sort(\"ALAND\", False)\n",
    "states = states.map(lambda x: x.set(\"AreaKm2\", x.area().divide(1000000).toLong()))\n",
    "first = states.first().toDictionary().getInfo()\n",
    "print(first)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "24",
   "metadata": {},
   "source": [
    "#### Set symbology based on column  values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "25",
   "metadata": {},
   "outputs": [],
   "source": [
    "Map = geemap.Map(center=[40, -100], zoom=4)\n",
    "\n",
    "states = ee.FeatureCollection(\"TIGER/2018/States\")\n",
    "\n",
    "visParams = {\n",
    "    \"palette\": [\"purple\", \"blue\", \"green\", \"yellow\", \"orange\", \"red\"],\n",
    "    \"min\": 500000000.0,\n",
    "    \"max\": 5e11,\n",
    "    \"opacity\": 0.8,\n",
    "}\n",
    "\n",
    "image = ee.Image().float().paint(states, \"ALAND\")\n",
    "Map.addLayer(image, visParams, \"TIGER/2018/States\")\n",
    "\n",
    "Map.addLayerControl()\n",
    "Map"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "26",
   "metadata": {},
   "source": [
    "#### Download attribute table as a CSV"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "27",
   "metadata": {},
   "outputs": [],
   "source": [
    "states = ee.FeatureCollection(\"TIGER/2018/States\")\n",
    "url = states.getDownloadURL(\n",
    "    filetype=\"csv\",\n",
    "    selectors=[\"NAME\", \"ALAND\", \"REGION\", \"STATEFP\", \"STUSPS\"],\n",
    "    filename=\"states\",\n",
    ")\n",
    "print(url)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "28",
   "metadata": {},
   "source": [
    "#### Formatting the output"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "29",
   "metadata": {},
   "outputs": [],
   "source": [
    "first = states.first()\n",
    "props = first.propertyNames().getInfo()\n",
    "print(props)\n",
    "props = states.first().toDictionary(props).getInfo()\n",
    "print(props)\n",
    "\n",
    "for key, value in props.items():\n",
    "    print(\"{}: {}\".format(key, value))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "30",
   "metadata": {},
   "source": [
    "#### Download data as shapefile to Google Drive"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "31",
   "metadata": {},
   "outputs": [],
   "source": [
    "# function for converting GeometryCollection to Polygon/MultiPolygon\n",
    "def filter_polygons(ftr):\n",
    "    geometries = ftr.geometry().geometries()\n",
    "    geometries = geometries.map(\n",
    "        lambda geo: ee.Feature(ee.Geometry(geo)).set(\"geoType\", ee.Geometry(geo).type())\n",
    "    )\n",
    "\n",
    "    polygons = (\n",
    "        ee.FeatureCollection(geometries)\n",
    "        .filter(ee.Filter.eq(\"geoType\", \"Polygon\"))\n",
    "        .geometry()\n",
    "    )\n",
    "    return ee.Feature(polygons).copyProperties(ftr)\n",
    "\n",
    "\n",
    "states = ee.FeatureCollection(\"TIGER/2018/States\")\n",
    "new_states = states.map(filter_polygons)\n",
    "\n",
    "col_names = states.first().propertyNames().getInfo()\n",
    "print(\"Column names: \", col_names)\n",
    "\n",
    "url = new_states.getDownloadURL(\"shp\", col_names, \"states\")\n",
    "print(url)\n",
    "\n",
    "desc = \"states\"\n",
    "\n",
    "# Set configuration parameters for output vector\n",
    "task_config = {\n",
    "    \"folder\": \"gee-data\",  # output Google Drive folder\n",
    "    \"fileFormat\": \"SHP\",\n",
    "    \"selectors\": col_names,  # a list of properties/attributes to be exported\n",
    "}\n",
    "\n",
    "print(\"Exporting {}\".format(desc))\n",
    "task = ee.batch.Export.table.toDrive(new_states, desc, **task_config)\n",
    "task.start()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "32",
   "metadata": {},
   "source": [
    "## TIGER: US Census Blocks\n",
    "\n",
    "https://developers.google.com/earth-engine/datasets/catalog/TIGER_2010_Blocks\n",
    "\n",
    "![](https://i.imgur.com/KTQqb8v.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "33",
   "metadata": {},
   "outputs": [],
   "source": [
    "Map = geemap.Map(center=[40, -100], zoom=4)\n",
    "\n",
    "dataset = ee.FeatureCollection(\"TIGER/2010/Blocks\").filter(\n",
    "    ee.Filter.eq(\"statefp10\", \"47\")\n",
    ")\n",
    "\n",
    "pop = dataset.aggregate_sum(\"pop10\")\n",
    "\n",
    "print(\"The number of census blocks: \", dataset.size().getInfo())\n",
    "print(\"Total population: \", pop.getInfo())\n",
    "\n",
    "Map.setCenter(-86.79, 35.87, 6)\n",
    "Map.addLayer(dataset, {}, \"Census Block\", False)\n",
    "\n",
    "visParams = {\n",
    "    \"min\": 0.0,\n",
    "    \"max\": 700.0,\n",
    "    \"palette\": [\"black\", \"brown\", \"yellow\", \"orange\", \"red\"],\n",
    "}\n",
    "\n",
    "image = ee.Image().float().paint(dataset, \"pop10\")\n",
    "\n",
    "Map.setCenter(-73.99172, 40.74101, 13)\n",
    "Map.addLayer(image, visParams, \"TIGER/2010/Blocks\")\n",
    "\n",
    "Map.addLayerControl()\n",
    "Map"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "34",
   "metadata": {},
   "source": [
    "## TIGER: US Census Counties 2018\n",
    "\n",
    "https://developers.google.com/earth-engine/datasets/catalog/TIGER_2018_Counties\n",
    "\n",
    "![](https://i.imgur.com/K3j99eA.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "35",
   "metadata": {},
   "outputs": [],
   "source": [
    "Map = geemap.Map(center=[40, -100], zoom=4)\n",
    "\n",
    "Map.setCenter(-110, 40, 5)\n",
    "states = ee.FeatureCollection(\"TIGER/2018/States\")\n",
    "#     .filter(ee.Filter.eq('STUSPS', 'TN'))\n",
    "# // Turn the strings into numbers\n",
    "states = states.map(lambda f: f.set(\"STATEFP\", ee.Number.parse(f.get(\"STATEFP\"))))\n",
    "\n",
    "state_image = ee.Image().float().paint(states, \"STATEFP\")\n",
    "\n",
    "visParams = {\n",
    "    \"palette\": [\"purple\", \"blue\", \"green\", \"yellow\", \"orange\", \"red\"],\n",
    "    \"min\": 0,\n",
    "    \"max\": 50,\n",
    "    \"opacity\": 0.8,\n",
    "}\n",
    "\n",
    "counties = ee.FeatureCollection(\"TIGER/2016/Counties\")\n",
    "# print(counties.first().propertyNames().getInfo())\n",
    "\n",
    "image = ee.Image().paint(states, 0, 2)\n",
    "# Map.setCenter(-99.844, 37.649, 4)\n",
    "# Map.addLayer(image, {'palette': 'FF0000'}, 'TIGER/2018/States')\n",
    "Map.addLayer(state_image, visParams, \"TIGER/2016/States\")\n",
    "Map.addLayer(ee.Image().paint(counties, 0, 1), {}, \"TIGER/2016/Counties\")\n",
    "\n",
    "Map.addLayerControl()\n",
    "Map"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "36",
   "metadata": {},
   "source": [
    "## TIGER: US Census Tracts\n",
    "\n",
    "https://developers.google.com/earth-engine/datasets/catalog/TIGER_2010_Tracts_DP1\n",
    "\n",
    "http://magic.lib.uconn.edu/magic_2/vector/37800/demogprofilehousect_37800_0000_2010_s100_census_1_t.htm\n",
    "\n",
    "![](https://i.imgur.com/WV2XA9b.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "37",
   "metadata": {},
   "outputs": [],
   "source": [
    "Map = geemap.Map(center=[40, -100], zoom=4)\n",
    "\n",
    "dataset = ee.FeatureCollection(\"TIGER/2010/Tracts_DP1\")\n",
    "visParams = {\n",
    "    \"min\": 0,\n",
    "    \"max\": 4000,\n",
    "    \"opacity\": 0.8,\n",
    "    \"palette\": [\n",
    "        \"#ece7f2\",\n",
    "        \"#d0d1e6\",\n",
    "        \"#a6bddb\",\n",
    "        \"#74a9cf\",\n",
    "        \"#3690c0\",\n",
    "        \"#0570b0\",\n",
    "        \"#045a8d\",\n",
    "        \"#023858\",\n",
    "    ],\n",
    "}\n",
    "\n",
    "# print(dataset.first().propertyNames().getInfo())\n",
    "\n",
    "# Turn the strings into numbers\n",
    "dataset = dataset.map(\n",
    "    lambda f: f.set(\"shape_area\", ee.Number.parse(f.get(\"dp0010001\")))\n",
    ")\n",
    "\n",
    "# Map.setCenter(-103.882, 43.036, 8)\n",
    "image = ee.Image().float().paint(dataset, \"dp0010001\")\n",
    "\n",
    "Map.addLayer(image, visParams, \"TIGER/2010/Tracts_DP1\")\n",
    "\n",
    "Map.addLayerControl()\n",
    "Map"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "38",
   "metadata": {},
   "source": [
    "## TIGER: US Census Roads\n",
    "\n",
    "https://developers.google.com/earth-engine/datasets/catalog/TIGER_2016_Roads\n",
    "\n",
    "![](https://i.imgur.com/mguD0IQ.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "39",
   "metadata": {},
   "outputs": [],
   "source": [
    "Map = geemap.Map(center=[40, -100], zoom=4)\n",
    "\n",
    "fc = ee.FeatureCollection(\"TIGER/2016/Roads\")\n",
    "Map.setCenter(-73.9596, 40.7688, 12)\n",
    "Map.addLayer(fc, {}, \"Census roads\")\n",
    "\n",
    "Map.addLayerControl()\n",
    "Map"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "40",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}