notebooks/How_to_Use_the_Web_Services_API_for_Dataset_Searching.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# How to Use the Web Services API for Dataset Searching\n",
    "***"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Overview:   \n",
    "The NASA Goddard Earth Sciences Data and Information Services Center (GES DISC) has developed an Application Program Interface (API) that is intended for users who prefer to employ our data search and subsetting services using scripts instead of the Web browser interface. The API protocol allows users to find the datasets and data granules they need, and to download any desired subsets. Information is passed back and forth in JavaScript Object Notation (JSON) format. \n",
    "\n",
    "## Example:  \n",
    "This example code demonstrates how to use the API to search GES DISC data collections. The code prompts the user for a search string, submits the search to the GES DISC server, and then retrieves the search results. The response from the server contains a large collection of metadata for each dataset that matches the search string. We provide some examples below that will print out selected pieces of information for each item that might be relevant for a) narrowing down the search, b) finding documentation, or c) setting up another API script to perform a subsetting request. The final example shows how to write out the entire search response to a local file.\n",
    "\n",
    "### Prerequisites:  \n",
    "This example code is written in Python3 and requires these libraries: sys, json, datetime, urllib3, and certifi. \n",
    "\n",
    "### Procedure\n",
    "The first step is to import the required Python libraries. If any of the following import commands fail, check the local Python environment and install any missing packages. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sys \n",
    "import json\n",
    "import datetime\n",
    "import certifi\n",
    "import urllib3"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Initialize the urllib PoolManager and set the base URL for dataset search API requests"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create a PoolManager instance to make requests.\n",
    "http = urllib3.PoolManager(cert_reqs='CERT_REQUIRED',ca_certs=certifi.where())\n",
    "\n",
    "# Set the URL for the GES DISC API endpoint for dataset searches\n",
    "url = 'https://disc.gsfc.nasa.gov/service/datasets/jsonwsp'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The user will be prompted for the desired search parameters -- the syntax is similar to what would be employed when searching with the Web browser interface. The search request is posted to the GES DISC server, and the response is checked for any errors. If there are errors, the user gets another prompt. When the results from a successful search have been retrieved, the script prints \"OK\", followed by the number of items in the search results.\n",
    "\n",
    "Run the following block of code and try these four search strings, noting how the number of results diminishes with each additional keyword:  \n",
    "_merra-2  \n",
    "merra-2 hourly  \n",
    "merra-2 hourly 2d  \n",
    "merra-2 hourly 2d instantaneous_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Prompt for search string keywords\n",
    "# This will keep looping and prompting until search returns an error-free response\n",
    "done = False\n",
    "while done is False :\n",
    "  myString=''\n",
    "  while len(myString) < 1 : \n",
    "    myString = input(\"Enter search keywords: \")\n",
    "\n",
    "  # Set up the JSON WSP request for API method: search\n",
    "  search_request = {\n",
    "    'methodname': 'search',\n",
    "    'type': 'jsonwsp/request',\n",
    "    'version': '1.0',\n",
    "    'args': {'search': myString}\n",
    "  }\n",
    "\n",
    "  # Submit the search request to the GES DISC server\n",
    "  hdrs = {'Content-Type': 'application/json',\n",
    "          'Accept': 'application/json'}\n",
    "  data = json.dumps(search_request)\n",
    "  r = http.request('POST', url, body=data, headers=hdrs)\n",
    "  response = json.loads(r.data)\n",
    "\n",
    "  # Check for errors\n",
    "  if response['type']=='jsonwsp/fault' :\n",
    "    print('ERROR! Faulty request. Please try again.')\n",
    "  else : \n",
    "    done = True\n",
    "print('OK')\n",
    "\n",
    "# Indicate the number of items in the search results\n",
    "total = response['result']['totalResults']\n",
    "if total == 0 :\n",
    "    print('Zero items found')\n",
    "elif total == 1 : \n",
    "    print('1 item found')\n",
    "else :          \n",
    "    print('%d items found' % total)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The response from the server contains <b><em>all</em></b> the metadata for each item in the list of dataset search results. The entire response contains much more information than users will likely need. The four examples below illustrate how to extract and print out a few critical pieces of metadata for each search result item, which could prove helpful for narrowing down the dataset search or developing API subsetting scripts. Please see <em>\"How to Use the Web Services API for Subsetting MERRA-2 Data\"</em> for detailed instructions on subsetting with the API. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The code below prints out the DatasetID (which is needed for any API subset requests) and the Label, a more human-readable description of the dataset. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Report on the results: DatasetID and Label\n",
    "if total > 0 :\n",
    "    for item in response['result']['items']:\n",
    "        print('%-20s  %s' % (item['dataset']['id'], item['dataset']['label']))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The code below prints out the DatasetID along with the starting and ending dates. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Report on the results: DatasetID, StartDate, and EndDate\n",
    "if total > 0 :\n",
    "    for item in response['result']['items']:\n",
    "        start = datetime.datetime.utcfromtimestamp(int(item['startDate']/1000))\n",
    "        end   = datetime.datetime.utcfromtimestamp(int(item['endDate']/1000))\n",
    "        print('%-20s  Start Date = %s    End Date = %s' % (item['dataset']['id'], start, end))                        "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The code below prints out the DatasetID and the Landing Page URL, which is the centralized access point for all the web-based information and services associated with that dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "# Report on the results: DatasetID and Landing Page URL\n",
    "if total > 0 :\n",
    "    for item in response['result']['items']:\n",
    "        print('%-20s  %s' % (item['dataset']['id'], item['link'])) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The code below prints out the DatasetID and a list of variable names and descriptions. The DatasetID and the variable name are pieces of metadata that are used for making subset requests with the API. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "# Report on the results: DatasetID and variable subsetting information\n",
    "varSubset = False\n",
    "for item in response['result']['items']:\n",
    "    # Check for subset services\n",
    "    if item['services']['subset']: \n",
    "        for ss in item['services']['subset']:\n",
    "            # make sure variable subsetting is supported\n",
    "            if 'variables' in ss['capabilities'] and 'dataFields' in ss :\n",
    "                print('The %s service supports variable subsetting for %s' % \n",
    "                      (ss['agentConfig']['agentId'],item['dataset']['id']))\n",
    "                print('Variable names are:')\n",
    "                varSubset = True\n",
    "                # Print a list of variable names and descriptions\n",
    "                for var in ss['dataFields']:\n",
    "                    print(var['value'])\n",
    "                print()\n",
    "if varSubset is False: \n",
    "    print('Variable subsetting is not available for %s' % item['dataset']['id'])\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The complete response can be written to a local text file in case the entire metadata record is needed. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Write out the complete results to a file\n",
    "fname='my_search_results.txt'\n",
    "f = open(fname,'w')\n",
    "f.write(json.dumps(response, indent=2, sort_keys=True))\n",
    "f.close()\n",
    "print('Complete metadata for each search result item has been written out to '+fname)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Additional Info:**  \n",
    "[Complete reference documentation on the GES DISC Dataset Service API](https://disc.gsfc.nasa.gov/service/datasets)  \n",
    "[How to Use the Web Services API for Subsetting](https://disc.gsfc.nasa.gov/information/howto?keywords=API&title=How%20to%20Use%20the%20Web%20Services%20API%20for%20Subsetting)  \n",
    "[How to Use the Web Services API for Subsetting MERRA-2 Data](https://disc.gsfc.nasa.gov/information/howto?keywords=api&title=How%20to%20Use%20the%20Web%20Services%20API%20for%20Subsetting%20MERRA-2%20Data)  \n",
    "[The GES DISC guide to data access](https://disc.gsfc.nasa.gov/data-access)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<font size=\"1\">THE SUBJECT FILE IS PROVIDED \"AS IS\" WITHOUT ANY WARRANTY OF ANY KIND, EITHER EXPRESSED, IMPLIED, OR STATUTORY, INCLUDING, BUT NOT LIMITED TO, ANY WARRANTY THAT THE SUBJECT FILE WILL CONFORM TO SPECIFICATIONS, ANY IMPLIED WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, OR FREEDOM FROM INFRINGEMENT, ANY WARRANTY THAT THE SUBJECT FILE WILL BE ERROR FREE, OR ANY WARRANTY THAT DOCUMENTATION, IF PROVIDED, WILL CONFORM TO THE SUBJECT FILE. THIS AGREEMENT DOES NOT, IN ANY MANNER, CONSTITUTE AN ENDORSEMENT BY GOVERNMENT AGENCY OR ANY PRIOR RECIPIENT OF ANY RESULTS, RESULTING DESIGNS, HARDWARE, SOFTWARE PRODUCTS OR ANY OTHER APPLICATIONS RESULTING FROM USE OF THE SUBJECT FILE. FURTHER, GOVERNMENT AGENCY DISCLAIMS ALL WARRANTIES AND LIABILITIES REGARDING THIRD-PARTY SOFTWARE, IF PRESENT IN THE SUBJECT FILE, AND DISTRIBUTES IT \"AS IS.\"</font>"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}