python/tutorials/Data_Discovery_CMR_API_Bulk_Query.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Using the CMR API and asyncio for fast CMR Queries  \n",
    "\n",
    "---\n",
    "\n",
    "## Summary  \n",
    "\n",
    "This tutorial demonstrates how to effectively perform queries and extract data download Uniform Resource Locators (URLs) for every Common Metadata Repository (CMR) metadata record within a NASA Earthdata collection. Two examples are shown. The first highlight making sequential requests for data URLs associated with specified collections. The second example demonstrates the how to leverages Python's `asyncio` package to perform bulk parallel requests for the same information and highlights the increase in speed when doing so. The NASA Earthdata collections highlighted here are Harmonized Landsat Sentinel-2 Operational Land Imager Surface Refleactance and TOA Brightness Daily Global 30m ([HLSL30.002](https://doi.org/10.5067/HLS/HLSL30.002)) and\n",
    "Harmonized Landsat Sentinel-2 Multi-spectral Instrument Surface Reflactance Daily Global 30m ([HLSS30.002](https://doi.org/10.5067/HLS/HLSS30.002)).  \n",
    "\n",
    "### What is CMR?  \n",
    "\n",
    "The CMR is a metadata system that catalogs NASA's Earth Observing System Data and Information System (EOSDIS) data and associated metadata. The CMR Application Programming Interface (API) provides programatic search capabilities through CMR's vast metadata holdings using various parameters and keywords. When querying NASA's CMR, there is a limit of 1 million granule matched with only 2000 granules returned per page. This guide shows how to search for CMR records using the CMR API and create a list of download URLs. This guide also shows how to leverage asynchronous, or parallel requests, to increase the speed of this process. The example below leverages the Harmonized Landsat Sentinel-2 collection archived by NASA's LP DAAC to demonstrate how to use Python's `asyncio` to perform large queries again NASA's CMR.  \n",
    "\n",
    "## Objectives  \n",
    "\n",
    "+ Use the CMR API and Python to perform large queries (requests that return more than 2000 granules) against NASA's CMR.  \n",
    "+ Prepare a list of URLs to access or download assets associated with those granules.  \n",
    "+ Utilize asynchronous/parallel requests to increase speed of query and list construction.  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Getting Started  \n",
    "\n",
    "Import the required packages.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import requests\n",
    "import math\n",
    "import aiohttp\n",
    "import asyncio\n",
    "import time"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Searching the CMR\n",
    "\n",
    "Set the CMR API Endpoint. This is the URL that we'll use to search through the CMR."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "CMR_OPS = 'https://cmr.earthdata.nasa.gov/search' # CMR API Endpoint\n",
    "url = f'{CMR_OPS}/{\"granules\"}'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To search the CMR we need to set our parameters. In this example we'll narrow our search using Collection IDs, a range of dates and times, and the number of results we want to show per page. Spatial areas can also be used to narrow searches (example shown in [HLS_Tutorial](https://git.earthdata.nasa.gov/projects/LPDUR/repos/hls-tutorial/browse/HLS_Tutorial.ipynb)). \n",
    "\n",
    "Here, we are interested in both HLS Landsat-8 and Sentinel-2 collections collected from October 17-19, 2021. Specify the `collections` to search, set a `datetime_range` and set the quantity of results to return per page using the `page_size` parameter like below.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "collections = ['C2021957657-LPCLOUD', 'C2021957295-LPCLOUD'] # Collection or concept_id specific to LPDAAC Products (HLS Landsat OLI and HLS Sentinel-2 respectively) \n",
    "datetime_range = '2021-10-17T00:00:00Z,2021-10-19T23:59:59Z'\n",
    "page_size = 2000"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A CMR search can find up to 1 million items or granules, but the number returned per page is limited to 2000, meaning large searches may have several pages of results. By default, `page_size` is set to 10."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Submitting Requests\n",
    "\n",
    "Using the above search criteria we can make a request using the `requests.get()` function. Submit a request and print the `response.status_code`.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "response = requests.get(url, \n",
    "                        params={\n",
    "                            'concept_id': collections,\n",
    "                            'temporal': datetime_range,\n",
    "                            'page_size': page_size,\n",
    "                            },\n",
    "                        headers={\n",
    "                            'Accept': 'application/json'\n",
    "                            }\n",
    "                       )\n",
    "print(response.status_code)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A status code of 200 indicates the request has succeeded. \n",
    "\n",
    "To see the number of results, print the `CMR-Hits` found in the returned header."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(response.headers['CMR-Hits']) # Resulting quantity of granules/items."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Building a List of File URLs\n",
    "\n",
    "We can build a list of URLs to data assets using our search results. Notice this only uses the first page of results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "granules = response.json()['feed']['entry']\n",
    "len(granules) # Resulting quantity of granules on page one."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "file_list = []\n",
    "for g in granules:\n",
    "    file_list.extend([x['href'] for x in g['links'] if 'https' in x['href'] and '.tif' in x['href']])\n",
    "len(file_list) # Total number of assets from page one of granules."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Print part of the URLs list. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "file_list[:25]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This process can be extended to all pages of search results to build a complete list of asset URLs. \n",
    "\n",
    "## Creating a List from Multiple Results Pages\n",
    "\n",
    "To create a list from multiple results pages, we first define a function to build a list of pages based upon the number of results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_page_total(collections, datetime_range, page_size):\n",
    "    hits = requests.get(url, \n",
    "                        params={\n",
    "                            'concept_id': collections,\n",
    "                            'temporal': datetime_range,\n",
    "                            'page_size': page_size,\n",
    "                            },\n",
    "                        headers={\n",
    "                            'Accept': 'application/json'\n",
    "                            }\n",
    "                       ).headers['CMR-Hits']\n",
    "    return math.ceil(int(hits)/page_size)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then we build a list of pages called `page_numbers`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "page_numbers = list(range(1, get_page_total(collections, datetime_range, page_size)+1))\n",
    "page_numbers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After we have a list of pages we can iterate through page by page to make a complete list of assets matching our search."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_urls = [] # empty list\n",
    "start = time.time() # Begin timer\n",
    "for n in page_numbers: # Iterate through requests page by page sequentially\n",
    "    print(f'Page: {n}') # Print Page Number\n",
    "    response = requests.get(url, # Same request function as used previously\n",
    "                            params={\n",
    "                                'concept_id': collections,\n",
    "                                'temporal': datetime_range,\n",
    "                                'page_size': page_size,\n",
    "                                'page_num': n\n",
    "                            },\n",
    "                            headers={\n",
    "                                'Accept': 'application/json'\n",
    "                            }\n",
    "                           )\n",
    "    print(f'Page {n} Resonse Code: {response.status_code}') # Show the response code for each page\n",
    "    \n",
    "    granules = response.json()['feed']['entry']\n",
    "    print(f'Number of Granules: {len(granules)}') # Show the number of granules on each page\n",
    "    \n",
    "    for g in granules:\n",
    "        data_urls.extend([x['href'] for x in g['links'] if 'https' in x['href'] and '.tif' in x['href']])\n",
    "end = time.time()\n",
    "print(f'Total time: {end-start}') # Record the total time taken"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Show the total quantity of assets in our list matching search parameters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "len(data_urls)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also see that the first 25 assets match up from our first page only search results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "file_list[:25]==data_urls[:25]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "\n",
    "## Improve speed using Asynchronous Requests\n",
    "\n",
    "You may have noticed the total time the function above took to run. For searches with a large quantity of results, we can query and build a list of asset URLs more quickly by utilizing asynchronous requests. Asynchronous requests can be run concurrently or in parallel, which typically decreases the total time of operations because a response is not needed for the prior request before a subsequent request is made. This time we'll use a similar approach as before, except we will build a list of page URLs that can be used in asynchronous requests to populate our list of asset URLs more quickly.\n",
    "\n",
    "First we define a new function `get_cmr_pages_urls()` to create a list of results pages URLs, not just the page numbers like we did before, then build that list.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_cmr_pages_urls(collections, datetime_range, page_size): \n",
    "    response = requests.get(url,\n",
    "                       params={\n",
    "                           'concept_id': collections,\n",
    "                           'temporal': datetime_range,\n",
    "                           'page_size': page_size,\n",
    "                       },\n",
    "                       headers={\n",
    "                           'Accept': 'application/json'\n",
    "                       }\n",
    "                      )\n",
    "    hits = int(response.headers['CMR-Hits'])\n",
    "    n_pages = math.ceil(hits/page_size)\n",
    "    cmr_pages_urls = [f'{response.url}&page_num={x}'.replace('granules?', 'granules.json?') for x in list(range(1,n_pages+1))]\n",
    "    return cmr_pages_urls"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "urls = get_cmr_pages_urls(collections, datetime_range, page_size)\n",
    "urls"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we create an empty list to populate with our asset URLs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "results = []"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then we define a function `get_tasks()` to build a list of tasks for each page number URL and a function `get_url()` to make the requests for each page in parallel with one another."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_tasks(session):\n",
    "    tasks = []\n",
    "    for l in urls:\n",
    "        tasks.append(session.get(l))\n",
    "    return tasks"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "async def get_url():\n",
    "    async with aiohttp.ClientSession() as session:\n",
    "        tasks = get_tasks(session)\n",
    "        responses = await asyncio.gather(*tasks)\n",
    "        for response in responses:\n",
    "            res = await response.json()\n",
    "            #print(res)\n",
    "            results.extend([l['href'] for g in res['feed']['entry'] for l in g['links'] if 'https' in l['href'] and '.tif' in l['href']])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Run the functions to submit asynchronous/parallel requests for each page of results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "start = time.time() \n",
    "\n",
    "await get_url()\n",
    "\n",
    "end = time.time()\n",
    "\n",
    "total_time = end - start\n",
    "total_time"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Much faster than before! We can see the same quantity of results and that a subsample of the resulting asset URLs matches what we retrieved before."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "len(results)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_urls[2025:2125] == results[2025:2125]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Additional Resources\n",
    "\n",
    "- [NASA Earthdata CMR Search API Documentation](https://cmr.earthdata.nasa.gov/search/site/docs/search/api.html)  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Contact Information  \n",
    "\n",
    "**Authors:**  LP DAAC¹  \n",
    "**Contact:** LPDAAC@usgs.gov  \n",
    "**Voice:** +1-866-573-3222  \n",
    "**Organization:** Land Processes Distributed Active Archive Center (LP DAAC)  \n",
    "**Website:** [https://lpdaac.usgs.gov/](https://lpdaac.usgs.gov/)  \n",
    "**Date last modified:** 01-25-2024  \n",
    "\n",
    "¹Work performed under USGS contract G15PD00467 for LP DAAC under NASA contract NNG14HH33I.  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  }
 ],
 "metadata": {
  "interpreter": {
   "hash": "2a5a3a7446f46e8e7ea3b3e1c81cb1037d05a65399f584592412a515f77fefb1"
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}