examples/notebooks/us_census_data.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<table class=\"ee-notebook-buttons\" align=\"left\">\n",
    "    <td><a target=\"_blank\"  href=\"https://github.com/giswqs/earthengine-py-documentation/tree/master/Template/template.ipynb\"><img width=32px src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" /> View source on GitHub</a></td>\n",
    "    <td><a target=\"_blank\"  href=\"https://nbviewer.jupyter.org/github/giswqs/earthengine-py-documentation/blob/master/Template/template.ipynb\"><img width=26px src=\"https://upload.wikimedia.org/wikipedia/commons/thumb/3/38/Jupyter_logo.svg/883px-Jupyter_logo.svg.png\" />Notebook Viewer</a></td>\n",
    "    <td><a target=\"_blank\"  href=\"https://colab.research.google.com/github/giswqs/earthengine-py-documentation/blob/master/Template/template.ipynb\"><img src=\"https://www.tensorflow.org/images/colab_logo_32px.png\" /> Run in Google Colab</a></td>\n",
    "</table>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# U.S. Census Data \n",
    "\n",
    "The United States Census Bureau Topologically Integrated Geographic Encoding and Referencing (TIGER) dataset contains the 2018 boundaries for the primary governmental divisions of the United States. In addition to the fifty states, the Census Bureau treats the District of Columbia, Puerto Rico, and each of the island areas (American Samoa, the Commonwealth of the Northern Mariana Islands, Guam, and the U.S. Virgin Islands) as the statistical equivalents of States for the purpose of data presentation. Each feature represents a state or state equivalent.\n",
    "\n",
    "For full technical details on all TIGER 2018 products, see the [TIGER technical documentation](https://www2.census.gov/geo/pdfs/maps-data/data/tiger/tgrshp2018/TGRSHP2018_TechDoc.pdf).\n",
    "\n",
    "* [TIGER: US Census States](https://developers.google.com/earth-engine/datasets/catalog/TIGER_2018_States): `ee.FeatureCollection(\"TIGER/2018/States\")`\n",
    "* [TIGER: US Census Counties](https://developers.google.com/earth-engine/datasets/catalog/TIGER_2018_Counties): `ee.FeatureCollection(\"TIGER/2018/Counties\")`\n",
    "* [TIGER: US Census Tracts](https://developers.google.com/earth-engine/datasets/catalog/TIGER_2010_Tracts_DP1): `ee.FeatureCollection(\"TIGER/2010/Tracts_DP1\")`\n",
    "* [TIGER: US Census Blocks](https://developers.google.com/earth-engine/datasets/catalog/TIGER_2010_Blocks): `ee.FeatureCollection(\"TIGER/2010/Blocks\")`\n",
    "* [TIGER: US Census Roads](https://developers.google.com/earth-engine/datasets/catalog/TIGER_2016_Roads): `ee.FeatureCollection(\"TIGER/2016/Roads\")`\n",
    "* [TIGER: US Census 5-digit ZIP Code](https://developers.google.com/earth-engine/datasets/catalog/TIGER_2010_ZCTA5): `ee.FeatureCollection(\"TIGER/2010/ZCTA5\")`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Install Earth Engine API and geemap\n",
    "Install the [Earth Engine Python API](https://developers.google.com/earth-engine/python_install) and [geemap](https://github.com/giswqs/geemap). The **geemap** Python package is built upon the [ipyleaflet](https://github.com/jupyter-widgets/ipyleaflet) and [folium](https://github.com/python-visualization/folium) packages and implements several methods for interacting with Earth Engine data layers, such as `Map.addLayer()`, `Map.setCenter()`, and `Map.centerObject()`.\n",
    "The following script checks if the geemap package has been installed. If not, it will install geemap, which automatically installs its [dependencies](https://github.com/giswqs/geemap#dependencies), including earthengine-api, folium, and ipyleaflet.\n",
    "\n",
    "**Important note**: A key difference between folium and ipyleaflet is that ipyleaflet is built upon ipywidgets and allows bidirectional communication between the front-end and the backend enabling the use of the map to capture user input, while folium is meant for displaying static data only ([source](https://blog.jupyter.org/interactive-gis-in-jupyter-with-ipyleaflet-52f9657fa7a)). Note that [Google Colab](https://colab.research.google.com/) currently does not support ipyleaflet ([source](https://github.com/googlecolab/colabtools/issues/60#issuecomment-596225619)). Therefore, if you are using geemap with Google Colab, you should use [`import geemap.eefolium`](https://github.com/giswqs/geemap/blob/master/geemap/eefolium.py). If you are using geemap with [binder](https://mybinder.org/) or a local Jupyter notebook server, you can use [`import geemap`](https://github.com/giswqs/geemap/blob/master/geemap/geemap.py), which provides more functionalities for capturing user input (e.g., mouse-clicking and moving)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Installs geemap package\n",
    "import subprocess\n",
    "\n",
    "try:\n",
    "    import geemap\n",
    "except ImportError:\n",
    "    print('geemap package not installed. Installing ...')\n",
    "    subprocess.check_call([\"python\", '-m', 'pip', 'install', 'geemap'])\n",
    "\n",
    "# Checks whether this notebook is running on Google Colab\n",
    "try:\n",
    "    import google.colab\n",
    "    import geemap.eefolium as emap\n",
    "except:\n",
    "    import geemap as emap\n",
    "\n",
    "# Authenticates and initializes Earth Engine\n",
    "import ee\n",
    "\n",
    "try:\n",
    "    ee.Initialize()\n",
    "except Exception as e:\n",
    "    ee.Authenticate()\n",
    "    ee.Initialize()  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## TIGER: US Census States\n",
    "\n",
    "https://developers.google.com/earth-engine/datasets/catalog/TIGER_2018_States\n",
    "\n",
    "![](https://i.imgur.com/DAESBK4.png)\n",
    "\n",
    "### Displaying data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "Map = emap.Map(center=[40, -100], zoom=4)\n",
    "\n",
    "states = ee.FeatureCollection('TIGER/2018/States')\n",
    "\n",
    "Map.centerObject(states, 4)\n",
    "Map.addLayer(states, {}, 'US States')\n",
    "\n",
    "Map.addLayerControl()  #This line is not needed for ipyleaflet-based Map\n",
    "Map"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Dispalying vector as raster"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "Map = emap.Map(center=[40, -100], zoom=4)\n",
    "\n",
    "states = ee.FeatureCollection('TIGER/2018/States')\n",
    "image = ee.Image().paint(states, 0, 2)\n",
    "\n",
    "Map.centerObject(states, 4)\n",
    "Map.addLayer(image, {}, 'US States')\n",
    "\n",
    "Map.addLayerControl()\n",
    "Map"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Select by attribute\n",
    "\n",
    "#### Select one single state"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "Map = emap.Map(center=[40, -100], zoom=4)\n",
    "\n",
    "tn = ee.FeatureCollection('TIGER/2018/States') \\\n",
    "    .filter(ee.Filter.eq(\"NAME\", 'Tennessee')) \n",
    "\n",
    "Map.centerObject(tn, 6)\n",
    "Map.addLayer(tn, {}, 'Tennessee')\n",
    "\n",
    "Map.addLayerControl()\n",
    "Map"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tn = ee.FeatureCollection('TIGER/2018/States') \\\n",
    "    .filter(ee.Filter.eq(\"NAME\", 'Tennessee')) \\\n",
    "    .first()\n",
    "\n",
    "props = tn.toDictionary().getInfo()\n",
    "print(props)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Select multiple states"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "Map = emap.Map(center=[40, -100], zoom=4)\n",
    "\n",
    "selected = ee.FeatureCollection('TIGER/2018/States') \\\n",
    "    .filter(ee.Filter.inList(\"NAME\", ['Tennessee', 'Alabama', 'Georgia']))\n",
    "\n",
    "Map.centerObject(selected, 6)\n",
    "Map.addLayer(selected, {}, 'Selected states')\n",
    "\n",
    "Map.addLayerControl()\n",
    "Map"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Printing all values of a column"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "states = ee.FeatureCollection('TIGER/2018/States').sort('ALAND', False)\n",
    "names = states.aggregate_array(\"STUSPS\").getInfo()\n",
    "print(names)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "areas = states.aggregate_array(\"ALAND\").getInfo()\n",
    "print(areas)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "%matplotlib notebook"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.bar(names, areas)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Discriptive statistics of a column\n",
    "\n",
    "For example, we can calcualte the total land area of all states:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "states = ee.FeatureCollection('TIGER/2018/States')\n",
    "area_m2 = states.aggregate_sum(\"ALAND\").getInfo()\n",
    "area_km2 = area_m2 / 1000000\n",
    "print(\"Total land area: \", area_km2, \" km2\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "states = ee.FeatureCollection('TIGER/2018/States')\n",
    "stats = states.aggregate_stats(\"ALAND\").getInfo()\n",
    "print(stats)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Add a new column to the attribute table"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "states = ee.FeatureCollection('TIGER/2018/States').sort('ALAND', False)\n",
    "states = states.map(lambda x: x.set('AreaKm2', x.area().divide(1000000).toLong()))\n",
    "first = states.first().toDictionary().getInfo()\n",
    "print(first)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Set symbology based on column  values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "Map = emap.Map(center=[40, -100], zoom=4)\n",
    "\n",
    "states = ee.FeatureCollection('TIGER/2018/States')\n",
    "\n",
    "visParams = {\n",
    "  'palette': ['purple', 'blue', 'green', 'yellow', 'orange', 'red'],\n",
    "  'min': 500000000.0,\n",
    "  'max': 5e+11,\n",
    "  'opacity': 0.8,\n",
    "}\n",
    "\n",
    "image = ee.Image().float().paint(states, 'ALAND')\n",
    "Map.addLayer(image, visParams, 'TIGER/2018/States')\n",
    "\n",
    "Map.addLayerControl()\n",
    "Map"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Download attribute table as a CSV"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "states = ee.FeatureCollection('TIGER/2018/States')\n",
    "url = states.getDownloadURL(filetype=\"csv\", selectors=['NAME', 'ALAND', 'REGION', 'STATEFP', 'STUSPS'], filename=\"states\")\n",
    "print(url)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Formatting the output"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "first = states.first()\n",
    "props = first.propertyNames().getInfo()\n",
    "print(props)\n",
    "props = states.first().toDictionary(props).getInfo()\n",
    "print(props)\n",
    "\n",
    "for key, value in props.items():\n",
    "    print(\"{}: {}\".format(key, value))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Download data as shapefile to Google Drive"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# function for converting GeometryCollection to Polygon/MultiPolygon\n",
    "def filter_polygons(ftr):\n",
    "    geometries = ftr.geometry().geometries()\n",
    "    geometries = geometries.map(lambda geo: ee.Feature( ee.Geometry(geo)).set('geoType',  ee.Geometry(geo).type()))\n",
    "\n",
    "    polygons = ee.FeatureCollection(geometries).filter(ee.Filter.eq('geoType', 'Polygon')).geometry()\n",
    "    return ee.Feature(polygons).copyProperties(ftr)\n",
    "\n",
    "states = ee.FeatureCollection('TIGER/2018/States')\n",
    "new_states = states.map(filter_polygons)\n",
    "\n",
    "col_names = states.first().propertyNames().getInfo()\n",
    "print(\"Column names: \", col_names)\n",
    "\n",
    "url = new_states.getDownloadURL(\"shp\", col_names, 'states');\n",
    "print(url)\n",
    "\n",
    "desc = 'states'\n",
    "\n",
    "# Set configration parameters for output vector\n",
    "task_config = {\n",
    "    'folder': 'gee-data', # output Google Drive folder\n",
    "    'fileFormat': 'SHP',  \n",
    "    'selectors': col_names # a list of properties/attributes to be exported\n",
    "    }\n",
    "\n",
    "print('Exporting {}'.format(desc))\n",
    "task = ee.batch.Export.table.toDrive(new_states, desc, **task_config)\n",
    "task.start()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## TIGER: US Census Blocks\n",
    "\n",
    "https://developers.google.com/earth-engine/datasets/catalog/TIGER_2010_Blocks\n",
    "\n",
    "![](https://i.imgur.com/KTQqb8v.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "Map = emap.Map(center=[40, -100], zoom=4)\n",
    "\n",
    "dataset = ee.FeatureCollection('TIGER/2010/Blocks') \\\n",
    "    .filter(ee.Filter.eq('statefp10', '47'))\n",
    "    \n",
    "pop = dataset.aggregate_sum('pop10')\n",
    "    \n",
    "print(\"The number of census blocks: \", dataset.size().getInfo())\n",
    "print(\"Total population: \", pop.getInfo())\n",
    "    \n",
    "Map.setCenter(-86.79, 35.87, 6)   \n",
    "Map.addLayer(dataset, {}, \"Census Block\", False)\n",
    "    \n",
    "visParams = {\n",
    "  'min': 0.0,\n",
    "  'max': 700.0,\n",
    "  'palette': ['black', 'brown', 'yellow', 'orange', 'red']\n",
    "}\n",
    "\n",
    "image = ee.Image().float().paint(dataset, 'pop10')\n",
    "\n",
    "Map.setCenter(-73.99172, 40.74101, 13)\n",
    "Map.addLayer(image, visParams, 'TIGER/2010/Blocks')\n",
    "\n",
    "Map.addLayerControl()\n",
    "Map"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## TIGER: US Census Counties 2018\n",
    "\n",
    "https://developers.google.com/earth-engine/datasets/catalog/TIGER_2018_Counties\n",
    "\n",
    "![](https://i.imgur.com/K3j99eA.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "Map = emap.Map(center=[40, -100], zoom=4)\n",
    "\n",
    "Map.setCenter(-110, 40, 5)\n",
    "states = ee.FeatureCollection('TIGER/2018/States') \n",
    "#     .filter(ee.Filter.eq('STUSPS', 'TN'))\n",
    "# // Turn the strings into numbers\n",
    "states = states.map(lambda f: f.set('STATEFP', ee.Number.parse(f.get('STATEFP'))))\n",
    "\n",
    "state_image = ee.Image().float().paint(states, 'STATEFP')\n",
    "\n",
    "visParams = {\n",
    "  'palette': ['purple', 'blue', 'green', 'yellow', 'orange', 'red'],\n",
    "  'min': 0,\n",
    "  'max': 50,\n",
    "  'opacity': 0.8,\n",
    "};\n",
    "\n",
    "counties = ee.FeatureCollection('TIGER/2016/Counties')  \n",
    "# print(counties.first().propertyNames().getInfo())\n",
    "\n",
    "image = ee.Image().paint(states, 0, 2)\n",
    "# Map.setCenter(-99.844, 37.649, 4)\n",
    "# Map.addLayer(image, {'palette': 'FF0000'}, 'TIGER/2018/States')\n",
    "Map.addLayer(state_image, visParams, 'TIGER/2016/States');\n",
    "Map.addLayer(ee.Image().paint(counties, 0, 1), {}, 'TIGER/2016/Counties')\n",
    "\n",
    "Map.addLayerControl()\n",
    "Map"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## TIGER: US Census Tracts\n",
    "\n",
    "https://developers.google.com/earth-engine/datasets/catalog/TIGER_2010_Tracts_DP1\n",
    "\n",
    "http://magic.lib.uconn.edu/magic_2/vector/37800/demogprofilehousect_37800_0000_2010_s100_census_1_t.htm\n",
    "\n",
    "![](https://i.imgur.com/WV2XA9b.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "Map = emap.Map(center=[40, -100], zoom=4)\n",
    "\n",
    "dataset = ee.FeatureCollection('TIGER/2010/Tracts_DP1')\n",
    "visParams = {\n",
    "  'min': 0,\n",
    "  'max': 4000,\n",
    "  'opacity': 0.8,\n",
    "   'palette': ['#ece7f2', '#d0d1e6', '#a6bddb', '#74a9cf', '#3690c0', '#0570b0', '#045a8d', '#023858']\n",
    "}\n",
    "\n",
    "# print(dataset.first().propertyNames().getInfo())\n",
    "\n",
    "# Turn the strings into numbers\n",
    "dataset = dataset.map(lambda f: f.set('shape_area', ee.Number.parse(f.get('dp0010001'))))\n",
    "\n",
    "# Map.setCenter(-103.882, 43.036, 8)\n",
    "image = ee.Image().float().paint(dataset, 'dp0010001')\n",
    "\n",
    "Map.addLayer(image, visParams, 'TIGER/2010/Tracts_DP1')\n",
    "\n",
    "Map.addLayerControl()\n",
    "Map"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## TIGER: US Census Roads\n",
    "\n",
    "https://developers.google.com/earth-engine/datasets/catalog/TIGER_2016_Roads\n",
    "\n",
    "![](https://i.imgur.com/mguD0IQ.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "Map = emap.Map(center=[40, -100], zoom=4)\n",
    "\n",
    "fc = ee.FeatureCollection('TIGER/2016/Roads')\n",
    "Map.setCenter(-73.9596, 40.7688, 12)\n",
    "Map.addLayer(fc, {}, 'Census roads')\n",
    "\n",
    "Map.addLayerControl()\n",
    "Map"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "hide_input": false,
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.3"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": true,
   "title_cell": "Table of Contents",
   "title_sidebar": "Table of Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": true
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}