how-tos/Earthdata_Cloud__Single_File__HTTPS_Access_COG_Example.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "fresh-natural",
   "metadata": {},
   "source": [
    "# Accessing Cloud Optimized GeoTIFF (COG) - HTTPS Example"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "persistent-death",
   "metadata": {},
   "source": [
    "## Summary\n",
    "\n",
    "In this notebook, we will access data for the Harmonized Landsat Sentinel-2 (HLS) Operational Land Imager Surface Reflectance and TOA Brightness Daily Global 30m v2.0 (L30) ([10.5067/HLS/HLSL30.002](https://doi.org/10.5067/HLS/HLSL30.002)) data product. These data are archived and distributed as Cloud Optimized GeoTIFF (COG) files, one file for each spectral band.\n",
    "\n",
    "We will access a single COG file, L30 red band (0.64 – 0.67 μm), from inside the AWS cloud (us-west-2 region, specifically) and load it into Python as an `xarray` `dataarray`. This approach leverages S3 native protocols for efficient access to the data.\n",
    "\n",
    "## Requirements\n",
    "\n",
    "### 1. AWS instance running in us-west-2\n",
    "\n",
    "NASA Earthdata Cloud data in S3 can be directly accessed via temporary credentials; this access is limited to requests made within the US West (Oregon) (code: us-west-2) AWS region.\n",
    "\n",
    "### 2. Earthdata Login\n",
    "\n",
    "An Earthdata Login account is required to access data, as well as discover restricted data, from the NASA Earthdata system. Thus, to access NASA data, you need Earthdata Login. Please visit https://urs.earthdata.nasa.gov to register and manage your Earthdata Login account. This account is free to create and only takes a moment to set up.\n",
    "\n",
    "### 3. netrc File\n",
    "\n",
    "You will need a netrc file containing your NASA Earthdata Login credentials in order to execute the notebooks. A netrc file can be created manually within text editor and saved to your home directory. For additional information see: [Authentication for NASA Earthdata](https://nasa-openscapes.github.io/2021-Cloud-Hackathon/tutorials/04_NASA_Earthdata_Authentication.html#authentication-via-netrc-file).\n",
    "\n",
    "## Learning Objectives\n",
    "\n",
    "- how to configure you Python work environment to access Cloud Optimized geoTIFF (COG) files\n",
    "- how to access HLS COG files\n",
    "- how to plot the data"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "completed-creek",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "institutional-given",
   "metadata": {},
   "source": [
    "## Cloud Optimized GeoTIFF (COG)\n",
    "\n",
    "Using Harmonized Landsat Sentinel-2 (HLS) version 2.0"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "usual-disabled",
   "metadata": {},
   "source": [
    "### Import Packages"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "boolean-horse",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "from osgeo import gdal\n",
    "import rasterio as rio\n",
    "import rioxarray\n",
    "import hvplot.xarray\n",
    "import holoviews as hv"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "furnished-orlando",
   "metadata": {},
   "source": [
    "## Workspace Environment Setup\n",
    "\n",
    "For this exercise, we are going to open up a context manager for the notebook using the rasterio.env module to store the required GDAL configurations we need to access the data from Earthdata Cloud. While the context manager is open (rio_env.__enter__()) we will be able to run the open or get data commands that would typically be executed within a with statement, thus allowing us to more freely interact with the data. We’ll close the context (rio_env.__exit__()) at the end of the notebook.\n",
    "\n",
    "GDAL environment variables must be configured to access COGs from Earthdata Cloud. Geospatial data access Python packages like rasterio and rioxarray depend on GDAL, leveraging GDAL’s “Virtual File Systems” to read remote files. GDAL has a lot of environment variables that control it’s behavior. Changing these settings can mean the difference being able to access a file or not. They can also have an impact on the performance."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "developing-danish",
   "metadata": {},
   "outputs": [],
   "source": [
    "rio_env = rio.Env(GDAL_DISABLE_READDIR_ON_OPEN='TRUE',\n",
    "                  GDAL_HTTP_COOKIEFILE=os.path.expanduser('~/cookies.txt'),\n",
    "                  GDAL_HTTP_COOKIEJAR=os.path.expanduser('~/cookies.txt'))\n",
    "rio_env.__enter__()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "expired-crowd",
   "metadata": {},
   "source": [
    "In this example we're interested in the HLS L30 data collection from NASA's LP DAAC in Earthdata Cloud. Below we specify the s3 URL to the data asset in Earthdata Cloud. This URL can be found via Earthdata Search or programmatically through the CMR and CMR-STAC APIs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "gorgeous-relief",
   "metadata": {},
   "outputs": [],
   "source": [
    "https_url = 'https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/HLSL30.020/HLS.L30.T11SQA.2021333T181532.v2.0/HLS.L30.T11SQA.2021333T181532.v2.0.B04.tif'"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fabulous-cornell",
   "metadata": {},
   "source": [
    "## HTTPS Data Access\n",
    "\n",
    "Read in the HLS s3 URL for the L30 red band (0.64 – 0.67 μm) into our workspace using `rioxarray`, an extension of `xarray` used to read geospatial data. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "blocked-municipality",
   "metadata": {},
   "outputs": [],
   "source": [
    "da = rioxarray.open_rasterio(https_url)\n",
    "da"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "african-acting",
   "metadata": {},
   "source": [
    "The file is read into Python as an `xarray` `dataarray` with a **band**, **x**, and **y** dimension. In this example the **band** dimension is meaningless, so we'll use the `squeeze()` function to remove **band** as a dimension. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "vocal-center",
   "metadata": {},
   "outputs": [],
   "source": [
    "da_red = da.squeeze('band', drop=True)\n",
    "da_red"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fdafb279-acbe-473a-aa34-200da33901ce",
   "metadata": {},
   "source": [
    "Plot the `dataarray`, representing the L30 red band, using `hvplot`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "outdoor-handling",
   "metadata": {},
   "outputs": [],
   "source": [
    "da_red.hvplot.image(x='x', y='y', cmap='gray', aspect='equal')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "86efc013-97ad-4af1-9e20-dea661567d04",
   "metadata": {},
   "source": [
    "Exit the context manager."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fitting-liechtenstein",
   "metadata": {},
   "outputs": [],
   "source": [
    "rio_env.__exit__()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}