docs/source/search/collections.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Searching for Collections in MAAP"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "These examples walk through the MAAP API functionality of searching for collections based on specific parameters. Collections are groupings of files that share the same product specification. Searching for collections can be useful for finding individual files, known as granules, which are used for processing.\n",
    "\n",
    "We begin by importing the `MAAP` package and creating a new MAAP class."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# import the MAAP package to handle queries\n",
    "from maap.maap import MAAP\n",
    "\n",
    "# import printing package to help display outputs\n",
    "from pprint import pprint\n",
    "\n",
    "# invoke the MAAP search client\n",
    "maap = MAAP()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can use the `maap.searchCollection` function to return a list of desired collections. Before using this function, let's use the `help` function to view the specific arguments and keywords for `maap.searchCollection`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Help on method searchCollection in module maap.maap:\n",
      "\n",
      "searchCollection(limit=100, **kwargs) method of maap.maap.MAAP instance\n",
      "    Search the CMR collections\n",
      "    :param limit: limit of the number of results\n",
      "    :param kwargs: search parameters\n",
      "    :return: list of results (<Instance of Result>)\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# view help for the searchCollection function\n",
    "help(maap.searchCollection)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The help text is showing that `maap.searchCollection` accepts a limit and search parameters. The limit parameter limits the number of resulting collections returned by `maap.searchCollection`. Note that `limit=100` means that the *default limit* for results from the MAAP API is 100. `maap.searchCollection` accepts any additional search parameters that are included in the CMR. For a list of accepted parameters, please refer to the [CMR Search Collections API reference](https://cmr.earthdata.nasa.gov/search/site/docs/search/api.html#collection-search-by-parameters)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this example we will explore search options including:\n",
    "\n",
    "1. Finding all Collections\n",
    "2. Searching by temporal filter\n",
    "3. Searching by spatial filter\n",
    "4. Using the results from one search as inputs into another\n",
    "5. Searching by additional attributes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Finding all Collections\n",
    "\n",
    "Here we will demonstrate how to create a list containing all of the collections contained within the CMR. To do this, we will use the `maap.searchCollection` function without any additional search parameters. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "'Got 61 results'\n"
     ]
    }
   ],
   "source": [
    "# search all collections\n",
    "results = maap.searchCollection()\n",
    "\n",
    "# print the number of collections\n",
    "pprint(f'Got {len(results)} results')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We were able to get 61 results. The result from the MAAP API is a list of collections where each element in the list is the metadata for that particular collection. Note that as more collections are added to the CMR, it is important to remember that the default limit on results is 100. To change the limit, type `limit=` and then a value within the parentheses after `maap.searchCollection()`.\n",
    "\n",
    "Let's look at the metadata for the first collection in our list of results (`results[0]`) using `pprint`. For formatting purposes, we can use the `depth` parameter to control the number of levels of metadata detail to display. By default, there is no constraint on the depth. By setting a `depth` parameter (in this case `depth=2`), we can ensure that the next contained level is replaced by an ellipsis."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'Collection': {'AdditionalAttributes': {...},\n",
      "                'ArchiveCenter': 'NASA/NSIDC_DAAC',\n",
      "                'Campaigns': {...},\n",
      "                'CollectionState': 'COMPLETE',\n",
      "                'Contacts': {...},\n",
      "                'DOI': {...},\n",
      "                'DataSetId': 'ABoVE LVIS L1B Geolocated Return Energy '\n",
      "                             'Waveforms V001',\n",
      "                'Description': 'This data set contains laser altimetry return '\n",
      "                               'energy waveform measurements over Alaska and '\n",
      "                               'Western Canada taken from the NASA Land, '\n",
      "                               'Vegetation, and Ice Sensor (LVIS). The data '\n",
      "                               \"were collected as part of NASA's Terrestrial \"\n",
      "                               'Ecology Program campaign, the Arctic-Boreal '\n",
      "                               'Vulnerability Experiment (ABoVE).',\n",
      "                'InsertTime': '2020-10-17T20:32:38.639Z',\n",
      "                'LastUpdate': '2020-10-17T20:32:38.639Z',\n",
      "                'LongName': 'Not provided',\n",
      "                'OnlineAccessURLs': {...},\n",
      "                'OnlineResources': {...},\n",
      "                'Orderable': 'false',\n",
      "                'Platforms': {...},\n",
      "                'ProcessingLevelId': '1B',\n",
      "                'RevisionDate': '2019-09-06T19:27:00.000Z',\n",
      "                'ScienceKeywords': {...},\n",
      "                'ShortName': 'ABLVIS1B',\n",
      "                'Spatial': {...},\n",
      "                'SpatialKeywords': {...},\n",
      "                'Temporal': {...},\n",
      "                'VersionId': '001',\n",
      "                'Visible': 'true'},\n",
      " 'concept-id': 'C1200110748-NASA_MAAP',\n",
      " 'format': 'application/echo10+xml',\n",
      " 'revision-id': '11'}\n"
     ]
    }
   ],
   "source": [
    "# print the metadata for the first collection\n",
    "# we use the depth parameter to set the layer of metadata detail in the results, with (1) having the least detail\n",
    "# (1) displays the concept ID, format, and revision ID\n",
    "# adjust the depth to a larger value (6) if you would like to view all of the metadata\n",
    "pprint(results[0],depth=2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `Collection` key has all of the collection information including attributes, the archive center, spatial, and temporal information. The `concept-id` is a unique identifier for this collection. It can be used to further refine search results from the CMR, such as when searching for granule information."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Searching by Temporal Filter\n",
    "\n",
    "Here we use a temporal filter to narrow down our results using the `temporal` keyword in our search. The temporal keyword takes datetime information in a [specific format](https://cmr.earthdata.nasa.gov/search/site/docs/search/api.html#c-temporal). The date format used is `YYYY-MM-DDThh:mm:ssZ` and temporal search criteria may be either a single date or a date range. If one date is provided then it can be inferred as the start or end date. To define a start date and return all collections after the date, put a comma after the date (`YYYY-MM-DDThh:mm:ssZ,`). To define a end date and return all granules prior to the data, put a comma before the date (`,YYYY-MM-DDThh:mm:ssZ`). Lastly, to get a date range, provide the start date and end date separated by a comma (`YYYY-MM-DDThh:mm:ssZ,YYYY-MM-DDThh:mm:ssZ`).\n",
    "\n",
    "In this example we will search for one month of data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "'Got 3 results'\n"
     ]
    }
   ],
   "source": [
    "datetimeRange = '2000-01-01T00:00:00Z,2000-01-31T23:59:59Z' # specify datetime range to search for data in January 2000\n",
    "\n",
    "results = maap.searchCollection(temporal=datetimeRange)\n",
    "pprint(f'Got {len(results)} results')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "'Collection Global_Forest_Change_2000-2017 was acquired starting at 2000-01-01T00:00:00.000Z'\n"
     ]
    }
   ],
   "source": [
    "collectionName = results[0]['Collection']['ShortName'] # get the collection short name\n",
    "collectionDate = results[0]['Collection']['Temporal']['RangeDateTime']['BeginningDateTime'] # get the collection start time\n",
    "\n",
    "pprint(\n",
    "    f'Collection {collectionName} was acquired starting at {collectionDate}', width=100)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It appears the first result correctly matches with the beginning and ending temporal search parameters. Keep in mind that the results are limited to 100 so the final collection returned may not match the end date that was searched for."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Searching by Spatial Filter\n",
    "\n",
    "Here we will illustrate how to search for collections by a spatial filter. There are a couple of [spatial filters available to search by](https://cmr.earthdata.nasa.gov/search/site/docs/search/api.html#c-spatial) in the CMR including point, line, polygon, and bounding box. In this example, we will explore filtering with a bounding box which is a sequence of four latitude and longitude values in the order of `[W,S,E,N]`. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "'Got 17 results'\n"
     ]
    }
   ],
   "source": [
    "collectionDomain = '-42,10,42,20' # specify bounding box to search by\n",
    "\n",
    "results = maap.searchCollection(bounding_box=collectionDomain)\n",
    "pprint(f'Got {len(results)} results')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "'Collection GEDI01_B was acquired within the following geometry: '\n",
      "{'BoundingRectangle': {'EastBoundingCoordinate': '180.0',\n",
      "                       'NorthBoundingCoordinate': '55.0',\n",
      "                       'SouthBoundingCoordinate': '-55.0',\n",
      "                       'WestBoundingCoordinate': '-180.0'},\n",
      " 'CoordinateSystem': 'CARTESIAN'}\n"
     ]
    }
   ],
   "source": [
    "collectionName = results[3]['Collection']['ShortName'] # get a collection short name\n",
    "collectionGeometry = results[3]['Collection']['Spatial']['HorizontalSpatialDomain']['Geometry'] # grab the spatial information from collection\n",
    "\n",
    "pprint(f'Collection {collectionName} was acquired within the following geometry: ', width=100)\n",
    "pprint(collectionGeometry)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see from the first collection that the spatial coordinates of the collection intersect our search box. Also note that spatial filtering yields more refined search results with only 16 collections returned."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Searching by Additional Attributes\n",
    "\n",
    "The MAAP has provided additional metadata, known as additional attributes, to the collection metadata to support the unique search needs of the aboveground terrestrial carbon research community. There are many additional attributes available:\n",
    "\n",
    "| variable name | additional attribute name | data type |\n",
    "| --------------|---------------| -----------|\n",
    "site_name | Site Name | string \n",
    "data_format | Data Format | string \n",
    "track_number | Track Number | float \n",
    "polarization | Polarization | string \n",
    "dataset_status | Dataset Status | string  \n",
    "spat_res | Spatial Resolution | float \n",
    "samp_freq | Sampling Frequency | float  \n",
    "band_ctr_freq | Band Center Frequency | float \n",
    "freq_band_name | Frequency Band Name | string \n",
    "swath_width | Swath Width | float \n",
    "field_view | Field of View | float \n",
    "laser_foot_diam | Laser Footprint Diameter | float \n",
    "pass_number | Pass Number | int \n",
    "revisit_time | Revisit Time | float \n",
    "flt_number | Flight Number | int \n",
    "number_plots | Number of Plots | int \n",
    "plot_area | Plot Area | float  \n",
    "br_ht | Breast Height | float    \n",
    "ret_per_pulse | Returns Per Pulse | string \n",
    "min_diam_meas | Minimum Diameter Measured | float    \n",
    "flt_alt | Flight Altitude | float  \n",
    "hdg | Heading | float \n",
    "swath_slant_rg_st_ang | Swath Slant Range Start Angle | float \n",
    "azm_rg_px_spacing | Azimuth Range Pixel Spacing | float \n",
    "slant_rg_px_spacing | Slant Range Pixel Spacing | float \n",
    "acq_type | Acquisition Type | string \n",
    "orbit_dir | Orbit Direction | string  \n",
    "band_ctr_wavelength | Band Center Wavelength | float \n",
    "swath_slant_rg_end_ang | Swath Slant Range End Angle | float\n",
    "\n",
    "For example, if a user is only interested in using data from the _Lope National Park Gabon_ site, we can use the following query:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "'Got 5 results'\n"
     ]
    }
   ],
   "source": [
    "results = maap.searchCollection(site_name=\"Lope National Park Gabon\")\n",
    "pprint(f'Got {len(results)} results')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'Collection': {'AdditionalAttributes': {...},\n",
      "                'ArchiveCenter': 'MAAP Data Management Team',\n",
      "                'Campaigns': {...},\n",
      "                'CollectionDataType': 'SCIENCE_QUALITY',\n",
      "                'CollectionState': 'COMPLETE',\n",
      "                'Contacts': {...},\n",
      "                'DataSetId': 'AfriSAR UAVSAR Geocoded Covariance Matrix '\n",
      "                             'product Generated Using NISAR Tools',\n",
      "                'Description': 'The Geocoded Covariance Matrix dataset is the '\n",
      "                               '4x4 Native Covariance Matrix geocoded to a '\n",
      "                               'spatial resolution of 25m using cubic '\n",
      "                               'interpolation methods.  These covariance '\n",
      "                               'matrix datasets provides the variability '\n",
      "                               'between retrievals for each co-registered '\n",
      "                               'single-look-complex (SLC) polarization and '\n",
      "                               'provide inferences on the scattering '\n",
      "                               'characteristics of a target.  The SLC data was '\n",
      "                               'collected from repeat-pass flights of the '\n",
      "                               'Uninhabited Aerial Vehicle Synthetic Aperture '\n",
      "                               'Radar (UAVSAR) instrument during the AfriSAR '\n",
      "                               'field campaign over the Gabonese forest in '\n",
      "                               'February-March 2016.  The AfriSAR campaign was '\n",
      "                               'a collaborative airborne mission between the '\n",
      "                               'National Aeronautics and Space Administration, '\n",
      "                               'the European Space Agency and the Gabonese '\n",
      "                               'Space Agency collecting radar, lidar and field '\n",
      "                               'measurements in support of future space borne '\n",
      "                               'missions for biomass research.',\n",
      "                'InsertTime': '2020-10-17T20:32:38.676Z',\n",
      "                'LastUpdate': '2020-10-17T20:32:38.676Z',\n",
      "                'LongName': 'Not provided',\n",
      "                'OnlineAccessURLs': {...},\n",
      "                'OnlineResources': {...},\n",
      "                'Orderable': 'false',\n",
      "                'Platforms': {...},\n",
      "                'ProcessingLevelDescription': 'Geocoded and mapped to uniform '\n",
      "                                              'spatial grid scales',\n",
      "                'ProcessingLevelId': '3',\n",
      "                'RevisionDate': '2019-04-08T21:02:00.000Z',\n",
      "                'ScienceKeywords': {...},\n",
      "                'ShortName': 'AfriSAR_UAVSAR_Geocoded_Covariance',\n",
      "                'Spatial': {...},\n",
      "                'Temporal': {...},\n",
      "                'VersionId': '1',\n",
      "                'Visible': 'true'},\n",
      " 'concept-id': 'C1200109238-NASA_MAAP',\n",
      " 'format': 'application/echo10+xml',\n",
      " 'revision-id': '5'}\n"
     ]
    }
   ],
   "source": [
    "pprint(results[0],depth=2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The returned results will give you only datasets that have been tagged as part of the Lope National Park Gabon research site."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}