CUAHSI_HISCentral_AOI_service_tests.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Assessing response times from CUAHSI HIS Central AOI web services\n",
    "- Emilio Mayorga, UW-APL. 2018-4-2\n",
    "- Run with Python 2.7 using `odm2client` conda environment, the one created for the BiGCZ workshop in Nov. 2017\n",
    "- Run time includes issuing service request, waiting for response from server, and processing the SOAP response. It was assessed using the [IPYTHON `time` magic function](http://ipython.readthedocs.io/en/stable/interactive/magics.html#magic-time)\n",
    "- **Each API assessment is conducted by issuing a service request over a 1&deg; x 1&deg; AOI box. The center coordinates of this box is specified in cell 9.**\n",
    "- **For each API assessment, the time to pay attention to is the \"Wall time\". See results at cells 12, 25 and 31.**\n",
    "- This notebook also contains a few other explorations of the catalog API responses."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Code used to calculate the area in $km^2$ of an AOI bounding box\n",
    "This AOI is specified in lat-lon coordinates as `bbox = (ymin, xmin, ymax, xmax)`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "from shapely.geometry import Polygon"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "From https://stackoverflow.com/questions/45733838/how-to-calculate-area-of-a-polygon-with-latitude-and-longitude"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "def to_polygon_vertices(ymin, xmin, ymax, xmax):\n",
    "    # list of bounding box vertices, to form a closed polygon\n",
    "    box_points = [(ymin, xmin), (ymax, xmin), (ymax, xmax), (ymin, xmax), (ymin, xmin)]\n",
    "    return box_points\n",
    "\n",
    "def switch_to_xy_coordinates(coordinates):\n",
    "    # returns x,y coordinates in km\n",
    "    earth_radius = 6371.0  # in km\n",
    "    lat_dist = np.pi * earth_radius / 180.0\n",
    "\n",
    "    latitudes, longitudes = zip(*coordinates)\n",
    "    y = (lat * lat_dist for lat in latitudes)\n",
    "    x = (lon * lat_dist * np.cos(np.radians(lat)) for lat, lon in zip(latitudes, longitudes))\n",
    "    return list(zip(x, y))\n",
    "\n",
    "def compute_polygon_area(coordinates):\n",
    "    # returns area in km2\n",
    "    xy_coordinates = switch_to_xy_coordinates(coordinates)\n",
    "    return Polygon(xy_coordinates).area"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## CUAHSI HIS catalog API testing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "from collections import OrderedDict\n",
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "import suds.client"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Test WDC HIS Central Series Catalog services"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "HIS_Central_URL = \"http://hiscentral.cuahsi.org/webservices/hiscentral.asmx?WSDL\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "client = suds.client.Client(HIS_Central_URL)\n",
    "cserv = client.service"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Gridded services, which should be considered for exclusion\n",
    "# determined in advance, manually (there's no other way to identify them)\n",
    "grid_services = ('GLDAS_NOAH', 'NLDAS_FORA', 'NLDAS_NOAH', 'TRMM_3B42_7')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Set AOI request parameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "keyword = ''\n",
    "networkID = ''\n",
    "start_date = ''\n",
    "end_date   = ''\n",
    "\n",
    "sampleMedium, dataType, valueType, generalCategory = '', '', '', ''"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "# use this scheme to generate a square bounding box centered on the specified lat-lon point\n",
    "# and with the specified width (in degrees)\n",
    "width_deg = 1.0\n",
    "\n",
    "# latctr, lonctr = 40.1, -75.5  # just north of the Schuykil river near Philly\n",
    "latctr, lonctr = 41.1, -75.5  # 1 deg north of the above PA/DRB center point\n",
    "# latctr, lonctr = 46.5, -123.0  # About halfway between Olympia, WA and Portland, OR\n",
    "# latctr, lonctr = 42.0, -93.0  # Iowa\n",
    "# latctr, lonctr = 30.0, -97.5  # Texas, south of Austin"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "bbox = (latctr-0.5*width_deg, lonctr-0.5*width_deg, latctr+0.5*width_deg, lonctr+0.5*width_deg)\n",
    "ymin,xmin,ymax,xmax = bbox"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "BBOX: (40.6, -76.0, 41.6, -75.0).  AOI area: 9317 km2\n"
     ]
    }
   ],
   "source": [
    "print(\"BBOX: {0}.  AOI area: {1:.0f} km2\".format(bbox, compute_polygon_area(to_polygon_vertices(*bbox))))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### TEST 1: GetSeriesCatalogForBox2\n",
    "Includes extended exploration of the resulting set of series."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 50.1 s, sys: 278 ms, total: 50.4 s\n",
      "Wall time: 55.3 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "response = cserv.GetSeriesCatalogForBox2(xmin, xmax, ymin, ymax, \n",
    "                                         keyword, networkID, \n",
    "                                         start_date, end_date)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(suds.sudsobject.ArrayOfSeriesRecord, 1)"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type(response), len(response)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(list, 16744)"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type(response.SeriesRecord), len(response.SeriesRecord)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(suds.sudsobject.SeriesRecord, 18)"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type(response.SeriesRecord[0]), len(response.SeriesRecord[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Import into a DataFrame to facilitate summarization"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "sr_recs = [OrderedDict(sr) for sr in response.SeriesRecord]\n",
    "seriesrec_df = pd.DataFrame.from_records(sr_recs, coerce_float=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ServCode</th>\n",
       "      <th>ServURL</th>\n",
       "      <th>location</th>\n",
       "      <th>VarCode</th>\n",
       "      <th>VarName</th>\n",
       "      <th>beginDate</th>\n",
       "      <th>endDate</th>\n",
       "      <th>ValueCount</th>\n",
       "      <th>Sitename</th>\n",
       "      <th>latitude</th>\n",
       "      <th>longitude</th>\n",
       "      <th>datatype</th>\n",
       "      <th>valuetype</th>\n",
       "      <th>samplemedium</th>\n",
       "      <th>timeunits</th>\n",
       "      <th>conceptKeyword</th>\n",
       "      <th>genCategory</th>\n",
       "      <th>TimeSupport</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>TRMM_3B42_7</td>\n",
       "      <td>https://hydro1.gesdisc.eosdis.nasa.gov/daac-bi...</td>\n",
       "      <td>TRMM_3B42_7:X418-Y364</td>\n",
       "      <td>TRMM_3B42_7:TRMM_3B42.007:precipitation</td>\n",
       "      <td>precipitation</td>\n",
       "      <td>1997-12-31T00:00:00Z</td>\n",
       "      <td>2018-03-19T06:34:23Z</td>\n",
       "      <td>43800</td>\n",
       "      <td>X418-Y364 of TRMM Multi-Satellite Precipitatio...</td>\n",
       "      <td>41.125</td>\n",
       "      <td>-75.375</td>\n",
       "      <td>Incremental</td>\n",
       "      <td>Derived Value</td>\n",
       "      <td>Precipitation</td>\n",
       "      <td>hour</td>\n",
       "      <td>Precipitation</td>\n",
       "      <td>Climate</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>TRMM_3B42_7</td>\n",
       "      <td>https://hydro1.gesdisc.eosdis.nasa.gov/daac-bi...</td>\n",
       "      <td>TRMM_3B42_7:X419-Y362</td>\n",
       "      <td>TRMM_3B42_7:TRMM_3B42.007:precipitation</td>\n",
       "      <td>precipitation</td>\n",
       "      <td>1997-12-31T00:00:00Z</td>\n",
       "      <td>2018-03-19T06:34:23Z</td>\n",
       "      <td>43800</td>\n",
       "      <td>X419-Y362 of TRMM Multi-Satellite Precipitatio...</td>\n",
       "      <td>40.625</td>\n",
       "      <td>-75.125</td>\n",
       "      <td>Incremental</td>\n",
       "      <td>Derived Value</td>\n",
       "      <td>Precipitation</td>\n",
       "      <td>hour</td>\n",
       "      <td>Precipitation</td>\n",
       "      <td>Climate</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>TRMM_3B42_7</td>\n",
       "      <td>https://hydro1.gesdisc.eosdis.nasa.gov/daac-bi...</td>\n",
       "      <td>TRMM_3B42_7:X416-Y364</td>\n",
       "      <td>TRMM_3B42_7:TRMM_3B42.007:precipitation</td>\n",
       "      <td>precipitation</td>\n",
       "      <td>1997-12-31T00:00:00Z</td>\n",
       "      <td>2018-03-19T06:34:23Z</td>\n",
       "      <td>43800</td>\n",
       "      <td>X416-Y364 of TRMM Multi-Satellite Precipitatio...</td>\n",
       "      <td>41.125</td>\n",
       "      <td>-75.875</td>\n",
       "      <td>Incremental</td>\n",
       "      <td>Derived Value</td>\n",
       "      <td>Precipitation</td>\n",
       "      <td>hour</td>\n",
       "      <td>Precipitation</td>\n",
       "      <td>Climate</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>TRMM_3B42_7</td>\n",
       "      <td>https://hydro1.gesdisc.eosdis.nasa.gov/daac-bi...</td>\n",
       "      <td>TRMM_3B42_7:X419-Y363</td>\n",
       "      <td>TRMM_3B42_7:TRMM_3B42.007:precipitation</td>\n",
       "      <td>precipitation</td>\n",
       "      <td>1997-12-31T00:00:00Z</td>\n",
       "      <td>2018-03-19T06:34:23Z</td>\n",
       "      <td>43800</td>\n",
       "      <td>X419-Y363 of TRMM Multi-Satellite Precipitatio...</td>\n",
       "      <td>40.875</td>\n",
       "      <td>-75.125</td>\n",
       "      <td>Incremental</td>\n",
       "      <td>Derived Value</td>\n",
       "      <td>Precipitation</td>\n",
       "      <td>hour</td>\n",
       "      <td>Precipitation</td>\n",
       "      <td>Climate</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>TRMM_3B42_7</td>\n",
       "      <td>https://hydro1.gesdisc.eosdis.nasa.gov/daac-bi...</td>\n",
       "      <td>TRMM_3B42_7:X417-Y364</td>\n",
       "      <td>TRMM_3B42_7:TRMM_3B42.007:precipitation</td>\n",
       "      <td>precipitation</td>\n",
       "      <td>1997-12-31T00:00:00Z</td>\n",
       "      <td>2018-03-19T06:34:23Z</td>\n",
       "      <td>43800</td>\n",
       "      <td>X417-Y364 of TRMM Multi-Satellite Precipitatio...</td>\n",
       "      <td>41.125</td>\n",
       "      <td>-75.625</td>\n",
       "      <td>Incremental</td>\n",
       "      <td>Derived Value</td>\n",
       "      <td>Precipitation</td>\n",
       "      <td>hour</td>\n",
       "      <td>Precipitation</td>\n",
       "      <td>Climate</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      ServCode                                            ServURL  \\\n",
       "0  TRMM_3B42_7  https://hydro1.gesdisc.eosdis.nasa.gov/daac-bi...   \n",
       "1  TRMM_3B42_7  https://hydro1.gesdisc.eosdis.nasa.gov/daac-bi...   \n",
       "2  TRMM_3B42_7  https://hydro1.gesdisc.eosdis.nasa.gov/daac-bi...   \n",
       "3  TRMM_3B42_7  https://hydro1.gesdisc.eosdis.nasa.gov/daac-bi...   \n",
       "4  TRMM_3B42_7  https://hydro1.gesdisc.eosdis.nasa.gov/daac-bi...   \n",
       "\n",
       "                location                                  VarCode  \\\n",
       "0  TRMM_3B42_7:X418-Y364  TRMM_3B42_7:TRMM_3B42.007:precipitation   \n",
       "1  TRMM_3B42_7:X419-Y362  TRMM_3B42_7:TRMM_3B42.007:precipitation   \n",
       "2  TRMM_3B42_7:X416-Y364  TRMM_3B42_7:TRMM_3B42.007:precipitation   \n",
       "3  TRMM_3B42_7:X419-Y363  TRMM_3B42_7:TRMM_3B42.007:precipitation   \n",
       "4  TRMM_3B42_7:X417-Y364  TRMM_3B42_7:TRMM_3B42.007:precipitation   \n",
       "\n",
       "         VarName             beginDate               endDate  ValueCount  \\\n",
       "0  precipitation  1997-12-31T00:00:00Z  2018-03-19T06:34:23Z       43800   \n",
       "1  precipitation  1997-12-31T00:00:00Z  2018-03-19T06:34:23Z       43800   \n",
       "2  precipitation  1997-12-31T00:00:00Z  2018-03-19T06:34:23Z       43800   \n",
       "3  precipitation  1997-12-31T00:00:00Z  2018-03-19T06:34:23Z       43800   \n",
       "4  precipitation  1997-12-31T00:00:00Z  2018-03-19T06:34:23Z       43800   \n",
       "\n",
       "                                            Sitename  latitude  longitude  \\\n",
       "0  X418-Y364 of TRMM Multi-Satellite Precipitatio...    41.125    -75.375   \n",
       "1  X419-Y362 of TRMM Multi-Satellite Precipitatio...    40.625    -75.125   \n",
       "2  X416-Y364 of TRMM Multi-Satellite Precipitatio...    41.125    -75.875   \n",
       "3  X419-Y363 of TRMM Multi-Satellite Precipitatio...    40.875    -75.125   \n",
       "4  X417-Y364 of TRMM Multi-Satellite Precipitatio...    41.125    -75.625   \n",
       "\n",
       "      datatype      valuetype   samplemedium timeunits conceptKeyword  \\\n",
       "0  Incremental  Derived Value  Precipitation      hour  Precipitation   \n",
       "1  Incremental  Derived Value  Precipitation      hour  Precipitation   \n",
       "2  Incremental  Derived Value  Precipitation      hour  Precipitation   \n",
       "3  Incremental  Derived Value  Precipitation      hour  Precipitation   \n",
       "4  Incremental  Derived Value  Precipitation      hour  Precipitation   \n",
       "\n",
       "  genCategory TimeSupport  \n",
       "0     Climate           1  \n",
       "1     Climate           1  \n",
       "2     Climate           1  \n",
       "3     Climate           1  \n",
       "4     Climate           1  "
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "seriesrec_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Number of series by service"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "ShaleNetworkODM    12152\n",
       "NWISGW              2279\n",
       "CocoRaHs             694\n",
       "GHCN                 383\n",
       "NLDAS_FORA           320\n",
       "NLDAS_NOAH           256\n",
       "NWISDV               226\n",
       "GLDAS_NOAH           208\n",
       "NWISUV               123\n",
       "UNH_Snow              50\n",
       "EnviroDIY             32\n",
       "TRMM_3B42_7           16\n",
       "MOPEX                  5\n",
       "Name: ServCode, dtype: int64"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "seriesrec_df.ServCode.value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Field Observation          10432\n",
       "Unknown                     4992\n",
       "Derived Value                607\n",
       "Model Simulation Result      448\n",
       "Sample                       265\n",
       "Name: valuetype, dtype: int64"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "seriesrec_df.valuetype.value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Water Quality      12151\n",
       "Hydrology           3468\n",
       "Climate             1093\n",
       "Water quality         24\n",
       "Instrumentation        8\n",
       "Name: genCategory, dtype: int64"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "seriesrec_df.genCategory.value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "15944"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# I'm filtering out \"*DAS_NOAH* service codes, b/c they're from gridded models\n",
    "# (results represent each cell) and therefore overwhelm the response\n",
    "series_array_nongrid = [series for series in response.SeriesRecord \n",
    "                        if series.ServCode not in grid_services]\n",
    "\n",
    "len(series_array_nongrid)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### TEST 2: GetSeriesMetadataCountOrData"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`GetSeriesMetadataCountOrData` arguments, in order:   \n",
    "getData, getFacetOnCV,   \n",
    "xmin, xmax, ymin, ymax,   \n",
    "sampleMedium, dataType, valueType, generalCategory,   \n",
    "conceptKeyword, networkIDs,   \n",
    "beginDate, endDate"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First use the API to request only the number of series that match the search criteria (`getData = False`)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 1.99 ms, sys: 7.86 ms, total: 9.85 ms\n",
      "Wall time: 183 ms\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "response = cserv.GetSeriesMetadataCountOrData(False, False, \n",
    "                                              xmin, xmax, ymin, ymax, \n",
    "                                              sampleMedium, dataType, valueType, generalCategory, \n",
    "                                              keyword, networkID, \n",
    "                                              start_date, end_date)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(GetSeriesCountOrData){\n",
       "   nseries = 16744\n",
       " }"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "response"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "16744L"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "response.nseries"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now request the actual series records (`getData = True`)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 1min 47s, sys: 659 ms, total: 1min 47s\n",
      "Wall time: 1min 50s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "response = cserv.GetSeriesMetadataCountOrData(True, False, \n",
    "                                              xmin, xmax, ymin, ymax, \n",
    "                                              sampleMedium, dataType, valueType, generalCategory, \n",
    "                                              keyword, networkID, \n",
    "                                              start_date, end_date)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(suds.sudsobject.GetSeriesCountOrData, 2)"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type(response), len(response)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(16744L, 16744L)"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "response.nseries, response[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(suds.sudsobject.ArrayOfSeriesRecordFull2, 1)"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type(response.series), len(response.series)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(list, 16744)"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type(response.series.SeriesRecordFull2), len(response.series.SeriesRecordFull2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(SeriesRecordFull2){\n",
       "   ServCode = \"TRMM_3B42_7\"\n",
       "   ServURL = \"https://hydro1.gesdisc.eosdis.nasa.gov/daac-bin/his/1.0/trmm_3b42.cgi?WSDL\"\n",
       "   location = \"TRMM_3B42_7:X416-Y363\"\n",
       "   VarCode = \"TRMM_3B42_7:TRMM_3B42.007:precipitation\"\n",
       "   VarName = \"precipitation\"\n",
       "   beginDate = \"1997-12-31T00:00:00Z\"\n",
       "   endDate = \"2018-03-19T06:35:17Z\"\n",
       "   ValueCount = \"43800\"\n",
       "   Sitename = \"X416-Y363 of TRMM Multi-Satellite Precipitation Analysis (TMPA-RT)\"\n",
       "   latitude = \"40.875\"\n",
       "   longitude = \"-75.875\"\n",
       "   datatype = \"Incremental\"\n",
       "   valuetype = \"Derived Value\"\n",
       "   samplemedium = \"Precipitation\"\n",
       "   timeunits = \"hour\"\n",
       "   conceptKeyword = \"Precipitation\"\n",
       "   genCategory = \"Climate\"\n",
       "   TimeSupport = \"1\"\n",
       "   QCLID = \"2\"\n",
       "   QCLDesc = \"Derived products\"\n",
       "   Organization = \"NASA Goddard Earth Sciences (GES) Data and Information Services Center (DISC)\"\n",
       "   TimeUnitAbbrev = \"hr\"\n",
       "   IsRegular = \"true\"\n",
       "   Speciation = \"Not Applicable\"\n",
       "   SourceOrg = \"NASA Goddard Earth Sciences (GES) Data and Information Services Center (DISC)\"\n",
       "   VariableUnitsAbbrev = \"mm/hr\"\n",
       "   SourceId = \"1\"\n",
       "   SourceDesc = \"TRMM Multi-Satellite Precipitation Analysis (TMPA-RT)\"\n",
       "   MethodId = \"1\"\n",
       "   MethodDesc = \"TRMM Multi-Satellite Precipitation Analysis (TMPA-RT)\"\n",
       " }"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "response.series.SeriesRecordFull2[8]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### TEST 3: GetSeriesCatalogForBox3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 1min 53s, sys: 596 ms, total: 1min 54s\n",
      "Wall time: 2min\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "# Provides more query parameters, and a much richer response, vs GetSeriesCatalogForBox2\n",
    "response = cserv.GetSeriesCatalogForBox3(xmin, xmax, ymin, ymax, \n",
    "                                         sampleMedium, dataType, valueType,\n",
    "                                         keyword, networkID, \n",
    "                                         start_date, end_date)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(SeriesRecordFull){\n",
       "   ServCode = \"TRMM_3B42_7\"\n",
       "   ServURL = \"https://hydro1.gesdisc.eosdis.nasa.gov/daac-bin/his/1.0/trmm_3b42.cgi?WSDL\"\n",
       "   location = \"TRMM_3B42_7:X416-Y364\"\n",
       "   VarCode = \"TRMM_3B42_7:TRMM_3B42.007:precipitation\"\n",
       "   VarName = \"precipitation\"\n",
       "   beginDate = \"1997-12-31T00:00:00Z\"\n",
       "   endDate = \"2018-03-19T06:37:12Z\"\n",
       "   ValueCount = 43800\n",
       "   Sitename = \"X416-Y364 of TRMM Multi-Satellite Precipitation Analysis (TMPA-RT)\"\n",
       "   latitude = 41.125\n",
       "   longitude = -75.875\n",
       "   datatype = \"Incremental\"\n",
       "   valuetype = \"Derived Value\"\n",
       "   samplemedium = \"Precipitation\"\n",
       "   timeunits = \"hour\"\n",
       "   conceptKeyword = \"Precipitation\"\n",
       "   genCategory = \"Climate\"\n",
       "   TimeSupport = \"1\"\n",
       "   SeriesCode = \"479|X416-Y364|TRMM_3B42.007:precipitation|1|1|2\"\n",
       "   QCLID = \"2\"\n",
       "   QCLDesc = \"Derived products\"\n",
       "   Organization = \"NASA Goddard Earth Sciences (GES) Data and Information Services Center (DISC)\"\n",
       "   TimeUnitAbbrev = \"hr\"\n",
       "   TimeUnits = \"hour\"\n",
       "   IsRegular = \"true\"\n",
       "   Speciation = \"Not Applicable\"\n",
       "   SourceOrg = \"NASA Goddard Earth Sciences (GES) Data and Information Services Center (DISC)\"\n",
       "   VariableUnitsAbbrev = \"mm/hr\"\n",
       "   SourceId = \"1\"\n",
       "   SourceDesc = \"TRMM Multi-Satellite Precipitation Analysis (TMPA-RT)\"\n",
       "   MethodId = \"1\"\n",
       "   MethodDesc = \"TRMM Multi-Satellite Precipitation Analysis (TMPA-RT)\"\n",
       " }"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "response.SeriesRecordFull[2]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python [conda env:odm2client]",
   "language": "python",
   "name": "conda-env-odm2client-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.14"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}