MMW_API_landproperties_demo.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Model My Watershed (MMW) API Demo\n",
    "[Emilio Mayorga](https://github.com/emiliom/), University of Washington, Seattle. 2018-5-10,17. Demo put together using as a starting point instructions from Azavea  from October 2017."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Introduction\n",
    "\n",
    "The [Model My Watershed API](https://app.wikiwatershed.org/api/docs/) allows you to delineate watersheds and analyze geo-data for watersheds and arbitrary areas. You can read more about the work at [WikiWatershed](http://www.wikiwatershed.org/) or use the [web app](https://www.app.wikiwatershed.org/).\n",
    "\n",
    "MMW users can discover their API keys through the user interface, and test the MMW geoprocessing API on either the live or staging apps. An Account page with the API key is available from either app (live or staging). To see it, go to the app, log in, and click on \"Account\" in the dropdown that appears when you click on your username in the top right. **Your key is different between staging and production.** For testing with the live (production) API and key, go to https://app.wikiwatershed.org/api/docs/\n",
    "\n",
    "The API can be tested from the command line using `curl`. This example uses the production API to test the `watershed` endpoint:\n",
    "```bash\n",
    "curl -H \"Content-Type: application/json\" -H \"Authorization: Token YOUR_API_KEY\" -X POST \n",
    "  -d '{ \"location\": [39.67185,-75.76743] }' https://app.wikiwatershed.org/api/watershed/\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## MMW API: Obtain land properties based on \"analyze\" geoprocessing on AOI (small box around a point)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1. Set up"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "import requests\n",
    "from requests.adapters import HTTPAdapter\n",
    "from requests.packages.urllib3.util.retry import Retry"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "def requests_retry_session(\n",
    "    retries=3,\n",
    "    backoff_factor=0.3,\n",
    "    status_forcelist=(500, 502, 504),\n",
    "    session=None,\n",
    "):\n",
    "    session = session or requests.Session()\n",
    "    retry = Retry(\n",
    "        total=retries,\n",
    "        read=retries,\n",
    "        connect=retries,\n",
    "        backoff_factor=backoff_factor,\n",
    "        status_forcelist=status_forcelist,\n",
    "    )\n",
    "    adapter = HTTPAdapter(max_retries=retry)\n",
    "    session.mount('http://', adapter)\n",
    "    session.mount('https://', adapter)\n",
    "    return session"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "MMW **production** API endpoint base url."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "api_url = \"https://app.wikiwatershed.org/api/\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The job is not completed instantly and the results are not returned directly by the API request that initiated the job. The user must first issue an API request to confirm that the job is complete, then fetch the results. The demo presented here performs automated retries (checks) until the server confirms the job is completed, then requests the JSON results and converts (deserializes) them into a Python dictionary."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_job_result(api_url, s, jobrequest):\n",
    "    url_tmplt = api_url + \"jobs/{job}/\"\n",
    "    get_url = url_tmplt.format\n",
    "    \n",
    "    result = ''\n",
    "    while not result:\n",
    "        get_req = requests_retry_session(session=s).get(get_url(job=jobrequest['job']))\n",
    "        result = json.loads(get_req.content)['result']\n",
    "    \n",
    "    return result"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "s = requests.Session()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "APIToken = 'Token 0501d9a98b8170a41d57df8ce82c000c477c621a'  # HIDE THE API TOKEN\n",
    "\n",
    "s.headers.update({\n",
    "    'Authorization': APIToken,\n",
    "    'Content-Type': 'application/json'\n",
    "})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. Construct AOI GeoJSON for job request"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Parameters passed to the \"analyze\" API requests."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "from shapely.geometry import box, MultiPolygon"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "width = 0.0004  # Looks like using a width smaller than 0.0002 causes a problem with the API?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "# GOOS: (-88.5552, 40.4374) elev 240.93. Agriculture Site—Goose Creek (Corn field) Site (GOOS) at IML CZO\n",
    "# SJER: (-119.7314, 37.1088) elev 403.86. San Joaquin Experimental Reserve Site (SJER) at South Sierra CZO\n",
    "lon, lat = -119.7314, 37.1088"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "bbox = box(lon-0.5*width, lat-0.5*width, lon+0.5*width, lat+0.5*width)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "payload = MultiPolygon([bbox]).__geo_interface__\n",
    "\n",
    "json_payload = json.dumps(payload)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'coordinates': [(((-119.73119999999999, 37.1086),\n",
       "    (-119.73119999999999, 37.109),\n",
       "    (-119.7316, 37.109),\n",
       "    (-119.7316, 37.1086),\n",
       "    (-119.73119999999999, 37.1086)),)],\n",
       " 'type': 'MultiPolygon'}"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "payload"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3. Issue job requests, fetch job results when done, then examine results. Repeat for each request type"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# convenience function, to simplify the request calls, below\n",
    "def analyze_api_request(api_name, s, api_url, json_payload):\n",
    "    post_url = \"{}analyze/{}/\".format(api_url, api_name)\n",
    "    post_req = requests_retry_session(session=s).post(post_url, data=json_payload)\n",
    "    jobrequest_json = json.loads(post_req.content)\n",
    "    # Fetch and examine job result\n",
    "    result = get_job_result(api_url, s, jobrequest_json)\n",
    "    return result"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Issue job request: **analyze/land/**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "result = analyze_api_request('land', s, api_url, json_payload)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Everything below is just exploration of the results. Examine the content of the results (as JSON, and Python dictionaries)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(dict, [u'survey'])"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type(result), result.keys()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`result` is a dictionary with one item, `survey`. This item in turn is a dictionary with 3 items: `displayName`, `name`, `categories`. The first two are just labels. The data are in the `categories` item."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[u'displayName', u'name', u'categories']"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "result['survey'].keys()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "categories = result['survey']['categories']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(16,\n",
       " {u'area': 0.0,\n",
       "  u'code': u'grassland',\n",
       "  u'coverage': 0.0,\n",
       "  u'nlcd': 71,\n",
       "  u'type': u'Grassland/Herbaceous'})"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(categories), categories[1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "land_categories_nonzero = [d for d in categories if d['coverage'] > 0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{u'area': 897.6442935164769,\n",
       "  u'code': u'shrub',\n",
       "  u'coverage': 1.0,\n",
       "  u'nlcd': 52,\n",
       "  u'type': u'Shrub/Scrub'}]"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "land_categories_nonzero"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Issue job request: **analyze/terrain/**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "result = analyze_api_request('terrain', s, api_url, json_payload)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`result` is a dictionary with one item, `survey`. This item in turn is a dictionary with 3 items: `displayName`, `name`, `categories`. The first two are just labels. The data are in the `categories` item."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "categories = result['survey']['categories']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(3,\n",
       " [{u'elevation': 404.43, u'slope': 5.240777969360352, u'type': u'average'},\n",
       "  {u'elevation': 404.43, u'slope': 5.240777969360352, u'type': u'minimum'},\n",
       "  {u'elevation': 404.43, u'slope': 5.240777969360352, u'type': u'maximum'}])"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(categories), categories"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{u'elevation': 404.43, u'slope': 5.240777969360352, u'type': u'average'}]"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "[d for d in categories if d['type'] == 'average']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Issue job request: **analyze/climate/**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "result = analyze_api_request('climate', s, api_url, json_payload)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`result` is a dictionary with one item, `survey`. This item in turn is a dictionary with 3 items: `displayName`, `name`, `categories`. The first two are just labels. The data are in the `categories` item."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "categories = result['survey']['categories']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(12,\n",
       " [{u'month': u'January',\n",
       "   u'monthidx': 1,\n",
       "   u'ppt': 9.198625946044922,\n",
       "   u'tmean': 7.592398643493652},\n",
       "  {u'month': u'February',\n",
       "   u'monthidx': 2,\n",
       "   u'ppt': 7.788037872314454,\n",
       "   u'tmean': 9.737625122070312}])"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(categories), categories[:2]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "ppt = [d['ppt'] for d in categories]\n",
    "tmean = [d['tmean'] for d in categories]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "44.89247835278511"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# ppt is in cm, right?\n",
    "sum(ppt)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "import calendar\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[0, 31, 28, 31, 30, 31, 30, 31, 31, 30, 31, 30, 31]"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "calendar.mdays"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "16.830176646088901"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Annual tmean needs to be weighted by the number of days per month\n",
    "sum(np.asarray(tmean) * np.asarray(calendar.mdays[1:]))/365"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python [conda env:odm2client]",
   "language": "python",
   "name": "conda-env-odm2client-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.14"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}