modis_global_api_access_python.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Submit MODIS Global Subset Tool orders using Python\n",
    "\n",
    "*Author: ORNL DAAC*  \n",
    "*Date: May 29, 2018*  \n",
    "*Contact for the ORNL DAAC: uso@daac.ornl.gov*  \n",
    "\n",
    "### Keywords: MODIS, web service, Python, REST\n",
    "\n",
    "## Overview\n",
    "\n",
    "This notebook will demonstrate how to submit a batch of orders to the [MODIS Global Subset Tool](https://modis.ornl.gov/cgi-bin/MODIS/global/subset.pl) for a list of coordinates in a text file using the MODIS Web Services API maintained by the ORNL DAAC. For a full description and usage examples of the web service, please visit: https://modis.ornl.gov/\n",
    "\n",
    "## Prerequisites:\n",
    "Python 2 or 3 Libraries: requests, pandas, json, datetime\n",
    "\n",
    "## Tutorial:\n",
    "\n",
    "Import libraries and set request URL and headers. Point python to your text file input, formatted in this example with nine columns matching the parameters required by the **subsetOrder** web service function:\n",
    "```\n",
    "site_id,product,latitude,longitude,email,start_date,end_date,kmAboveBelow,kmLeftRight\n",
    "site1,MOD13Q1,35.0,-90.0,mcnelisjj@ornl.gov,2000-01-01,2005-12-31,8,8\n",
    "site2,MOD13Q1,40.0,-95.0,mcnelisjj@ornl.gov,2000-01-01,2005-12-31,8,8\n",
    "site3,MOD13Q1,45.0,-100.0,mcnelisjj@ornl.gov,2000-01-01,2005-12-31,8,8\n",
    "site4,MOD13Q1,50.0,-105.0,mcnelisjj@ornl.gov,2000-01-01,2005-12-31,8,8\n",
    "site5,MOD13Q1,55.0,-110.0,mcnelisjj@ornl.gov,2000-01-01,2005-12-31,8,8\n",
    "```\n",
    "You can of course format your input file however best suits your needs; e.g. the product, email, start_date, end_date, kmAboveBelow, and kmLeftRight are redundant in this example and could be excluded."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import requests\n",
    "import json\n",
    "import pandas as pd\n",
    "from datetime import datetime\n",
    "\n",
    "url = \"https://modis.ornl.gov/rst/api/v1/\"\n",
    "header = {'Accept': 'application/json'} # Use following for a csv response: header = {'Accept': 'text/csv'}\n",
    "\n",
    "csv = \"example_sites.csv\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, read your file into a *pandas* data frame:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>site_id</th>\n",
       "      <th>product</th>\n",
       "      <th>latitude</th>\n",
       "      <th>longitude</th>\n",
       "      <th>email</th>\n",
       "      <th>start_date</th>\n",
       "      <th>end_date</th>\n",
       "      <th>kmAboveBelow</th>\n",
       "      <th>kmLeftRight</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>site1</td>\n",
       "      <td>MOD13Q1</td>\n",
       "      <td>35.0</td>\n",
       "      <td>-90.0</td>\n",
       "      <td>mcnelisjj@ornl.gov</td>\n",
       "      <td>2000-01-01</td>\n",
       "      <td>2005-12-31</td>\n",
       "      <td>8</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>site2</td>\n",
       "      <td>MOD13Q1</td>\n",
       "      <td>40.0</td>\n",
       "      <td>-95.0</td>\n",
       "      <td>mcnelisjj@ornl.gov</td>\n",
       "      <td>2000-01-01</td>\n",
       "      <td>2005-12-31</td>\n",
       "      <td>8</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>site3</td>\n",
       "      <td>MOD13Q1</td>\n",
       "      <td>45.0</td>\n",
       "      <td>-100.0</td>\n",
       "      <td>mcnelisjj@ornl.gov</td>\n",
       "      <td>2000-01-01</td>\n",
       "      <td>2005-12-31</td>\n",
       "      <td>8</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>site4</td>\n",
       "      <td>MOD13Q1</td>\n",
       "      <td>50.0</td>\n",
       "      <td>-105.0</td>\n",
       "      <td>mcnelisjj@ornl.gov</td>\n",
       "      <td>2000-01-01</td>\n",
       "      <td>2005-12-31</td>\n",
       "      <td>8</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>site5</td>\n",
       "      <td>MOD13Q1</td>\n",
       "      <td>55.0</td>\n",
       "      <td>-110.0</td>\n",
       "      <td>mcnelisjj@ornl.gov</td>\n",
       "      <td>2000-01-01</td>\n",
       "      <td>2005-12-31</td>\n",
       "      <td>8</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  site_id  product  latitude  longitude               email  start_date  \\\n",
       "0   site1  MOD13Q1      35.0      -90.0  mcnelisjj@ornl.gov  2000-01-01   \n",
       "1   site2  MOD13Q1      40.0      -95.0  mcnelisjj@ornl.gov  2000-01-01   \n",
       "2   site3  MOD13Q1      45.0     -100.0  mcnelisjj@ornl.gov  2000-01-01   \n",
       "3   site4  MOD13Q1      50.0     -105.0  mcnelisjj@ornl.gov  2000-01-01   \n",
       "4   site5  MOD13Q1      55.0     -110.0  mcnelisjj@ornl.gov  2000-01-01   \n",
       "\n",
       "     end_date  kmAboveBelow  kmLeftRight  \n",
       "0  2005-12-31             8            8  \n",
       "1  2005-12-31             8            8  \n",
       "2  2005-12-31             8            8  \n",
       "3  2005-12-31             8            8  \n",
       "4  2005-12-31             8            8  "
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "coordinates = pd.read_csv(csv)\n",
    "print(coordinates)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before we can submit orders for the global tool, we must find the MODIS dates nearest to the start and end of our desired time series. We will use the **dates** function to get a list of available MODIS dates for each of our input coordinates:\n",
    "\n",
    "    /api/v1/{product}/dates\n",
    "\n",
    "The **dates** function returns a list of available dates for the specified coordinate and MODIS product.\n",
    "\n",
    "| Parameter          | Description                                                   |\n",
    "|:-------------------|:--------------------------------------------------------------|\n",
    "| **product**        | MODIS product code as listed by **products** function         |\n",
    "| **latitude**       | latitude                                                      |\n",
    "| **longitude**      | longitude                                                     |\n",
    "\n",
    "Iterate through the coordinates in the data frame and find the MODIS date nearest to the calendar dates in the start_date and end_date columns. Add them to a new column of the data frame:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>site_id</th>\n",
       "      <th>product</th>\n",
       "      <th>latitude</th>\n",
       "      <th>longitude</th>\n",
       "      <th>email</th>\n",
       "      <th>start_date</th>\n",
       "      <th>end_date</th>\n",
       "      <th>kmAboveBelow</th>\n",
       "      <th>kmLeftRight</th>\n",
       "      <th>start_MODIS_date</th>\n",
       "      <th>end_MODIS_date</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>site1</td>\n",
       "      <td>MOD13Q1</td>\n",
       "      <td>35.0</td>\n",
       "      <td>-90.0</td>\n",
       "      <td>mcnelisjj@ornl.gov</td>\n",
       "      <td>2000-01-01</td>\n",
       "      <td>2005-12-31</td>\n",
       "      <td>8</td>\n",
       "      <td>8</td>\n",
       "      <td>A2000049</td>\n",
       "      <td>A2005353</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>site2</td>\n",
       "      <td>MOD13Q1</td>\n",
       "      <td>40.0</td>\n",
       "      <td>-95.0</td>\n",
       "      <td>mcnelisjj@ornl.gov</td>\n",
       "      <td>2000-01-01</td>\n",
       "      <td>2005-12-31</td>\n",
       "      <td>8</td>\n",
       "      <td>8</td>\n",
       "      <td>A2000049</td>\n",
       "      <td>A2005353</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>site3</td>\n",
       "      <td>MOD13Q1</td>\n",
       "      <td>45.0</td>\n",
       "      <td>-100.0</td>\n",
       "      <td>mcnelisjj@ornl.gov</td>\n",
       "      <td>2000-01-01</td>\n",
       "      <td>2005-12-31</td>\n",
       "      <td>8</td>\n",
       "      <td>8</td>\n",
       "      <td>A2000049</td>\n",
       "      <td>A2005353</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>site4</td>\n",
       "      <td>MOD13Q1</td>\n",
       "      <td>50.0</td>\n",
       "      <td>-105.0</td>\n",
       "      <td>mcnelisjj@ornl.gov</td>\n",
       "      <td>2000-01-01</td>\n",
       "      <td>2005-12-31</td>\n",
       "      <td>8</td>\n",
       "      <td>8</td>\n",
       "      <td>A2000049</td>\n",
       "      <td>A2005353</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>site5</td>\n",
       "      <td>MOD13Q1</td>\n",
       "      <td>55.0</td>\n",
       "      <td>-110.0</td>\n",
       "      <td>mcnelisjj@ornl.gov</td>\n",
       "      <td>2000-01-01</td>\n",
       "      <td>2005-12-31</td>\n",
       "      <td>8</td>\n",
       "      <td>8</td>\n",
       "      <td>A2000049</td>\n",
       "      <td>A2005353</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  site_id  product  latitude  longitude               email start_date  \\\n",
       "0   site1  MOD13Q1      35.0      -90.0  mcnelisjj@ornl.gov 2000-01-01   \n",
       "1   site2  MOD13Q1      40.0      -95.0  mcnelisjj@ornl.gov 2000-01-01   \n",
       "2   site3  MOD13Q1      45.0     -100.0  mcnelisjj@ornl.gov 2000-01-01   \n",
       "3   site4  MOD13Q1      50.0     -105.0  mcnelisjj@ornl.gov 2000-01-01   \n",
       "4   site5  MOD13Q1      55.0     -110.0  mcnelisjj@ornl.gov 2000-01-01   \n",
       "\n",
       "    end_date  kmAboveBelow  kmLeftRight start_MODIS_date end_MODIS_date  \n",
       "0 2005-12-31             8            8         A2000049       A2005353  \n",
       "1 2005-12-31             8            8         A2000049       A2005353  \n",
       "2 2005-12-31             8            8         A2000049       A2005353  \n",
       "3 2005-12-31             8            8         A2000049       A2005353  \n",
       "4 2005-12-31             8            8         A2000049       A2005353  "
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Convert start_date and end_date columns to datetimes\n",
    "coordinates['start_date'] =  pd.to_datetime(coordinates['start_date'])\n",
    "coordinates['end_date'] =  pd.to_datetime(coordinates['end_date'])\n",
    "\n",
    "# Make new columns for MODIS start and end dates \n",
    "coordinates['start_MODIS_date'] = '' \n",
    "coordinates['end_MODIS_date'] = ''\n",
    "\n",
    "for index, row in coordinates.iterrows():\n",
    "    # Submit request\n",
    "    response = requests.get('https://modis.ornl.gov/rst/api/v1/' + row['product'] + '/dates?latitude=' + str(row['latitude']) + '&longitude='+ str(row['longitude']), headers=header)\n",
    "    \n",
    "    # Get dates object as list of python dictionaries\n",
    "    dates = json.loads(response.text)['dates'] \n",
    "    \n",
    "    # Convert to list of tuples; change calendar_date key values to datetimes\n",
    "    dates = [(datetime.strptime(date['calendar_date'], \"%Y-%m-%d\"), date['modis_date']) for date in dates]\n",
    "    \n",
    "    # Get MODIS dates nearest to start_date and end_date and add to new pandas columns\n",
    "    coordinates.loc[index, 'start_MODIS_date'] = min(date[1] for date in dates if date[0] > row['start_date'])\n",
    "    coordinates.loc[index, 'end_MODIS_date'] = max(date[1] for date in dates if date[0] < row['end_date'])\n",
    "\n",
    "print(coordinates)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we are ready to submit our subset orders. We will use the **subsetOrder** function to pass our subset parameters to the ORNL DAAC's [MODIS Global Subset Tool](https://modis.ornl.gov/cgi-bin/MODIS/global/subset.pl) service:\n",
    "\n",
    "    /api/v1/{product}/subsetOrder\n",
    "\n",
    "The **subsetOrder** function returns a unique order identifier (*uid*) that you can use to retreive your order URL. You will also receive an email at the supplied email address once the order has completed processing. Processing times vary based on the size of the order. Most are completed within 30 minutes.\n",
    "\n",
    "| Parameter          | Description                                                   |\n",
    "|:-------------------|:--------------------------------------------------------------|\n",
    "| **product**        | MODIS product code as listed by **products** function         |\n",
    "| **latitude**       | latitude                                                      |\n",
    "| **longitude**      | longitude                                                     |\n",
    "| **email**          | email address for order delivery                              |\n",
    "| **uid**            | unique order identifier                                       |\n",
    "| **start_date**     | MODIS start date as listed by **dates** (\"AYYYYDOY\")          |\n",
    "| **end_date**       | MODIS end date as listed by **dates** (\"AYYYYDOY\")            |\n",
    "| **kmAboveBelow**   | number of kilometers to subset above and below center pixel   |\n",
    "| **kmLeftRight**    | number of kilometers to subset left and right of center pixel |\n",
    "\n",
    "Iterate again through the rows of the dataframe and submit orders using the **subsetOrder** function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[u'29May2018_14:55:35_037543986L35.0L-90.0S65L65_MOD13Q1_site1',\n",
       " u'29May2018_14:55:35_701080305L40.0L-95.0S65L65_MOD13Q1_site2',\n",
       " u'29May2018_14:55:36_310255483L45.0L-100.0S65L65_MOD13Q1_site3',\n",
       " u'29May2018_14:55:36_941348266L50.0L-105.0S65L65_MOD13Q1_site4',\n",
       " u'29May2018_14:55:37_572193963L55.0L-110.0S65L65_MOD13Q1_site5']"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Make list to collect order UIDs\n",
    "order_uids = []\n",
    "\n",
    "for index, row in coordinates.iterrows():\n",
    "    # Build request URL\n",
    "    requestURL = url + row['product'] + \"/subsetOrder?latitude=\" + str(row['latitude']) + \"&longitude=\" + str(row['longitude']) + \"&email=\" + row['email'] + \"&uid=\" + row['site_id'] + \"&startDate=\" + row['start_MODIS_date'] + \"&endDate=\" + row['end_MODIS_date'] + \"&kmAboveBelow=\" + str(row['kmAboveBelow']) + \"&kmLeftRight=\" + str(row['kmLeftRight'])\n",
    "\n",
    "    # Submit request\n",
    "    response = requests.get(requestURL, headers=header)\n",
    "    \n",
    "    # Append UID to list\n",
    "    order_uids.append(json.loads(response.text)['order_id'])\n",
    "    \n",
    "print(order_uids)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you see a list of strings formatted like those above, your orders were received by the ORNL DAAC! \n",
    "\n",
    "As mentioned above, you will receive an email upon completion of your order that will link you to a customized webpage with interactive visualizations and the subset data in CSV and GeoTIFF formats. For more information about the capabilities of the MODIS Global Subset Tool, please visit: https://modis.ornl.gov/\n",
    "\n",
    "You can also link to your orders directly via the order UID:\n",
    "```\n",
    "    https://modis.ornl.gov/subsetdata/<order_uid>\n",
    "    \n",
    "    e.g. \n",
    "    https://modis.ornl.gov/subsetdata/29May2018_14:55:35_037543986L35.0L-90.0S65L65_MOD13Q1_site1\n",
    "    https://modis.ornl.gov/subsetdata/29May2018_14:55:35_701080305L40.0L-95.0S65L65_MOD13Q1_site2\n",
    "    https://modis.ornl.gov/subsetdata/29May2018_14:55:36_310255483L45.0L-100.0S65L65_MOD13Q1_site3\n",
    "    https://modis.ornl.gov/subsetdata/29May2018_14:55:36_941348266L50.0L-105.0S65L65_MOD13Q1_site4\n",
    "    https://modis.ornl.gov/subsetdata/29May2018_14:55:37_572193963L55.0L-110.0S65L65_MOD13Q1_site5\n",
    "```\n",
    "\n",
    "![order_delivery_example](modis_order_delivery_email.png)\n",
    "Figure 1. Order delivery email for 29May2018_14:45:05_170194826L35.0L-90.0S65L65_MOD13Q1_site1"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.14"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}