doc/source/example_notebooks/IS2_data_read-in.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "552e9ef9",
   "metadata": {
    "user_expressions": []
   },
   "source": [
    "# Reading ICESat-2 Data in for Analysis\n",
    "This notebook ({nb-download}`download <IS2_data_read-in.ipynb>`) illustrates the use of icepyx for reading ICESat-2 data files, loading them into a data object.\n",
    "Currently the default data object is an Xarray Dataset, with ongoing work to provide support for other data object types.\n",
    "\n",
    "For more information on how to order and download ICESat-2 data, see the [icepyx data access tutorial](https://icepyx.readthedocs.io/en/latest/example_notebooks/IS2_data_access.html).\n",
    "\n",
    "### Motivation\n",
    "Most often, when you open a data file, you must specify the underlying data structure and how you'd like the information to be read in.\n",
    "A simple example of this, for instance when opening a csv or similarly delimited file, is letting the software know if the data contains a header row, what the data type is (string, double, float, boolean, etc.) for each column, what the delimeter is, and which columns or rows you'd like to be loaded.\n",
    "Many ICESat-2 data readers are quite manual in nature, requiring that you accurately type out a list of string paths to the various data variables.\n",
    "\n",
    "icepyx simplifies this process by relying on its awareness of ICESat-2 specific data file variable storage structure.\n",
    "Instead of needing to manually iterate through the beam pairs, you can provide a few options to the `Read` object and icepyx will do the heavy lifting for you (as detailed in this notebook).\n",
    "\n",
    "### Approach\n",
    "If you're interested in what's happening under the hood: icepyx uses the [xarray](https://docs.xarray.dev/en/stable/) library to read in each of the requested variables of the dataset. icepyx formats each requested variable and then merges the read-in data from each of the variables to create a single data object. The use of xarray is powerful, because the returned data object can be used with relevant xarray processing tools."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0d360de3",
   "metadata": {},
   "source": [
    "Import packages, including icepyx"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2b74b672",
   "metadata": {},
   "outputs": [],
   "source": [
    "import icepyx as ipx"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1ffb9a0c",
   "metadata": {
    "user_expressions": []
   },
   "source": [
    "---------------------------------\n",
    "\n",
    "## Quick-Start Guide\n",
    "For those who might be looking into playing with this (but don't want all the details/explanations)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c4390195",
   "metadata": {},
   "outputs": [],
   "source": [
    "path_root = '/full/path/to/your/ATL06_data/'\n",
    "reader = ipx.Read(path_root)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2f46029d",
   "metadata": {},
   "outputs": [],
   "source": [
    "reader.vars.append(beam_list=['gt1l', 'gt3r'], var_list=['h_li', \"latitude\", \"longitude\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c0439388",
   "metadata": {},
   "outputs": [],
   "source": [
    "ds = reader.load()\n",
    "ds"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fae7768a",
   "metadata": {},
   "outputs": [],
   "source": [
    "ds.plot.scatter(x=\"longitude\", y=\"latitude\", hue=\"h_li\", vmin=-100, vmax=2000)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b8875936",
   "metadata": {
    "user_expressions": []
   },
   "source": [
    "---------------------------------------\n",
    "## Key steps for loading (reading) ICESat-2 data\n",
    "\n",
    "Reading in ICESat-2 data with icepyx happens in a few simple steps:\n",
    "1. Let icepyx know where to find your data (this might be local files or urls to data in cloud storage)\n",
    "2. Create an icepyx `Read` object\n",
    "3. Make a list of the variables you want to read in (does not apply for gridded products)\n",
    "4. Load your data into memory (or read it in lazily, if you're using Dask)\n",
    "\n",
    "We go through each of these steps in more detail in this notebook."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9bf6d38c",
   "metadata": {
    "user_expressions": []
   },
   "source": [
    "### Step 0: Get some data if you haven't already\n",
    "Here are a few lines of code to get you set up with a few data files if you don't already have some on your local system."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "63da2b3c",
   "metadata": {},
   "outputs": [],
   "source": [
    "region_a = ipx.Query('ATL06',[-55, 68, -48, 71],['2019-02-22','2019-02-28'], \\\n",
    "                           start_time='00:00:00', end_time='23:59:59')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e6f7c047",
   "metadata": {},
   "outputs": [],
   "source": [
    "region_a.download_granules(path=path_root)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "04f62f30-b13c-4cfc-95b0-dd1e048f6a85",
   "metadata": {
    "user_expressions": []
   },
   "source": [
    "```{admonition} Important Authentication Update\n",
    "Previously, icepyx required you to explicitly use the `.earthdata_login()` function to login. Running this function is deprecated and will result in an error, as icepyx will call the login function as needed. The user will still need to provide their credentials.\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e8da42c1",
   "metadata": {
    "user_expressions": []
   },
   "source": [
    "### Step 1: Set data source path\n",
    "\n",
    "Provide a full path to the data to be read in (i.e. opened).\n",
    "Currently accepted inputs are:\n",
    "* a string path to directory - all files from the directory will be opened\n",
    "* a string path to single file - one file will be opened\n",
    "* a list of filepaths - all files in the list will be opened\n",
    "* a glob string (see [glob](https://docs.python.org/3/library/glob.html)) - any files matching the glob pattern will be opened"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9cde6679",
   "metadata": {},
   "outputs": [],
   "source": [
    "path_root = '/full/path/to/your/data/'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8b6edf0c",
   "metadata": {},
   "outputs": [],
   "source": [
    "# filepath = path_root + 'ATL06-20181214041627-Sample.h5'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fac636c2-e0eb-4e08-adaa-8f47623e46a1",
   "metadata": {},
   "outputs": [],
   "source": [
    "# list_of_files = ['/my/data/ATL06/processed_ATL06_20190226005526_09100205_006_02.h5', \n",
    "#                  '/my/other/data/ATL06/processed_ATL06_20191202102922_10160505_006_01.h5']"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ba3ebeb0-3091-4712-b0f7-559ddb95ca5a",
   "metadata": {
    "user_expressions": []
   },
   "source": [
    "#### Glob Strings\n",
    "\n",
    "[glob](https://docs.python.org/3/library/glob.html) is a Python library which allows users to list files in their file systems whose paths match a given pattern. Icepyx uses the glob library to give users greater flexibility over their input file lists.\n",
    "\n",
    "glob works using `*` and `?` as wildcard characters, where `*` matches any number of characters and `?` matches a single character. For example:\n",
    "\n",
    "* `/this/path/*.h5`: refers to all `.h5` files in the `/this/path` folder (Example matches: \"/this/path/processed_ATL03_20191130221008_09930503_006_01.h5\" or \"/this/path/myfavoriteicsat-2file.h5\")\n",
    "* `/this/path/*ATL07*.h5`: refers to all `.h5` files in the `/this/path` folder that have ATL07 in the filename. (Example matches: \"/this/path/ATL07-02_20221012220720_03391701_005_01.h5\" or \"/this/path/processed_ATL07.h5\")\n",
    "* `/this/path/ATL??/*.h5`: refers to all `.h5` files that are in a subfolder of `/this/path` and a subdirectory of `ATL` followed by any 2 characters (Example matches: \"/this/path/ATL03/processed_ATL03_20191130221008_09930503_006_01.h5\", \"/this/path/ATL06/myfile.h5\")\n",
    "\n",
    "See the glob documentation or other online explainer tutorials for more in depth explanation, or advanced glob paths such as character classes and ranges."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "20286c76-5632-4420-b2c9-a5a6b1952672",
   "metadata": {
    "user_expressions": []
   },
   "source": [
    "#### Recursive Directory Search"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "632bd1ce-2397-4707-a63f-9d5d2fc02fbc",
   "metadata": {
    "user_expressions": []
   },
   "source": [
    "glob will not by default search all of the subdirectories for matching filepaths, but it has the ability to do so.\n",
    "\n",
    "If you would like to search recursively, you can achieve this by either:\n",
    "1. passing the `recursive` argument into `glob_kwargs` and including `\\**\\` in your filepath\n",
    "2. using glob directly to create a list of filepaths\n",
    "\n",
    "Each of these two methods are shown below."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "da0cacd8-9ddc-4c31-86b6-167d850b989e",
   "metadata": {
    "user_expressions": []
   },
   "source": [
    "Method 1: passing the `recursive` argument into `glob_kwargs`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e276b876-9ec7-4991-8520-05c97824b896",
   "metadata": {},
   "outputs": [],
   "source": [
    "ipx.Read('/path/to/**/folder', glob_kwargs={'recursive': True})"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f5a1e85e-fc4a-405f-9710-0cb61b827f2c",
   "metadata": {
    "user_expressions": []
   },
   "source": [
    "You can use `glob_kwargs` for any additional argument to Python's builtin `glob.glob` that you would like to pass in via icepyx."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "76de9539-710c-49f6-9e9e-238849382c33",
   "metadata": {
    "user_expressions": []
   },
   "source": [
    "Method 2: using glob directly to create a list of filepaths"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "be79b0dd-efcf-4d50-bdb0-8e3ae8e8e38c",
   "metadata": {},
   "outputs": [],
   "source": [
    "import glob"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5d088571-496d-479a-9fb7-833ed7e98676",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "list_of_files = glob.glob('/path/to/**/folder', recursive=True)\n",
    "ipx.Read(list_of_files)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "08df2874-7c54-4670-8f37-9135ea296ff5",
   "metadata": {
    "user_expressions": []
   },
   "source": [
    "```{admonition} Read Module Update\n",
    "Previously, icepyx required two additional conditions: 1) a `product` argument and 2) that your files either matched the default `filename_pattern` or that the user provided their own `filename_pattern`. These two requirements have been removed. `product` is now read directly from the file metadata (the root group's `short_name` attribute). Flexibility to specify multiple files via the `filename_pattern` has been replaced with the [glob string](https://docs.python.org/3/library/glob.html) feature, and by allowing a list of filepaths as an argument.\n",
    "\n",
    "The `product` and `filename_pattern` arguments are now deprecated and were removed in icepyx version 1.0.0.\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4275b04c",
   "metadata": {
    "user_expressions": []
   },
   "source": [
    "### Step 2: Create an icepyx read object\n",
    "\n",
    "Using the `data_source` described in Step 1, we can create our Read object."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "39bd7eb8",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "reader = ipx.Read(data_source=path_root)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7b2acfdb-75eb-4c64-b583-2ab19326aaee",
   "metadata": {
    "user_expressions": []
   },
   "source": [
    "The Read object now contains the list of matching files that will eventually be loaded into Python. You can inspect its properties, such as the files that were located or the identified product, directly on the Read object."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6c9ebc4a",
   "metadata": {},
   "outputs": [],
   "source": [
    "reader.filelist"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7455ee3f-f9ab-486e-b4c7-2fa2314d4084",
   "metadata": {},
   "outputs": [],
   "source": [
    "reader.product"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "da8d8024",
   "metadata": {
    "user_expressions": []
   },
   "source": [
    "### Step 3: Specify variables to be read in\n",
    "\n",
    "To load your data into memory or prepare it for analysis, icepyx needs to know which variables you'd like to read in.\n",
    "If you've used icepyx to download data from NSIDC with variable subsetting (which is the default), then you may already be familiar with the icepyx `Variables` module and how to create and modify lists of variables.\n",
    "We showcase a specific case here, but we encourage you to check out [the icepyx Variables example](https://icepyx.readthedocs.io/en/latest/example_notebooks/IS2_data_variables.html) for a thorough trip through how to create and manipulate lists of ICESat-2 variable paths (examples are provided for multiple data products).\n",
    "\n",
    "If you want to see a \\[likely very long\\] list of all path + variable combinations available to you, this unmutable (unchangeable) list is generated by default from the first file in your list (so not all variables may be contained in all of the files, depending on how you are accessing the data)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "18f65f67",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "reader.vars.avail()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b2449941",
   "metadata": {
    "user_expressions": []
   },
   "source": [
    "To make things easier, you can use icepyx's built-in default list that loads commonly used variables for your non-gridded data product, or create your own list of variables to be read in.\n",
    "icepyx will determine what variables are available for you to read in by creating a list from one of your source files.\n",
    "If you have multiple files that you're reading in, icepyx will automatically generate a list of filenames and take the first one to get the list of available variables.\n",
    "\n",
    "Thus, if you have different variables available across files (even from the same data product), you may run into issues and need to come up with a workaround (we can help you do so!).\n",
    "We anticipate most users will have the minimum set of variables they are seeking to load available across all data files, so we're not currently developing this feature.\n",
    "Please get in touch if it would be a helpful feature for you or if you encounter this problem!\n",
    "\n",
    "You may create a variable list for gridded ICESat-2 products. However, all variables in the file will still be added to your DataSet. (This is an area we're currently exploring on expanding - please let us know if you're working on this and would like to contribute!)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "55092d1b",
   "metadata": {
    "user_expressions": []
   },
   "source": [
    "For a basic case, let's say we want to read in height, latitude, and longitude for all beam pairs.\n",
    "We create our variables list as"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e3734e09",
   "metadata": {},
   "outputs": [],
   "source": [
    "reader.vars.append(var_list=['h_li', \"latitude\", \"longitude\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fff0bb19",
   "metadata": {
    "user_expressions": []
   },
   "source": [
    "Then we can view a dictionary of the variables we'd like to read in."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e5456e36",
   "metadata": {},
   "outputs": [],
   "source": [
    "reader.vars.wanted"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9d5b50b5",
   "metadata": {
    "user_expressions": []
   },
   "source": [
    "Don't forget - if you need to start over, and re-generate your wanted variables list, it's easy!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "69894391",
   "metadata": {},
   "outputs": [],
   "source": [
    "reader.vars.remove(all=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "473de4d7",
   "metadata": {
    "user_expressions": []
   },
   "source": [
    "### Step 4: Loading your data\n",
    "\n",
    "Now that you've set up all the options, you're ready to read your ICESat-2 data into memory!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4a66d889-8d2d-4b9a-821a-96a394ff8d66",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "eaabc976",
   "metadata": {},
   "outputs": [],
   "source": [
    "ds = reader.load()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "db6560f1",
   "metadata": {
    "user_expressions": []
   },
   "source": [
    "Within a Jupyter Notebook, you can get a summary view of your data object.\n",
    "\n",
    "***ATTENTION: icepyx loads your data by creating an Xarray DataSet for each input granule and then merging them. In some cases, the automatic merge fails and needs to be handled manually. In these cases, icepyx will return a warning with the error message from the failed Xarray merge and a list of per-granule DataSets***\n",
    "\n",
    "This can happen if you unintentionally provide the same granule multiple times with different filenames or in segmented products where the rgt+cycle automatically generated `gran_idx` values match. In this latter case, you can simply provide unique `gran_idx` values for each DataSet in `ds` and run `import xarray as xr` and `ds_merged = xr.merge(ds)` to create one merged DataSet."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "723256f7",
   "metadata": {},
   "outputs": [],
   "source": [
    "ds"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b1d7de2d",
   "metadata": {
    "user_expressions": []
   },
   "source": [
    "## On to data analysis!\n",
    "\n",
    "From here, you can begin your analysis.\n",
    "Ultimately, icepyx aims to include an Xarray extension with ICESat-2 aware functions that allow you to do things like easily use only data from strong beams.\n",
    "That functionality is still in development.\n",
    "For fun, we've included a basic plot made with Xarray's built in functionality."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1d6f4560",
   "metadata": {},
   "outputs": [],
   "source": [
    "ds.plot.scatter(x=\"longitude\", y=\"latitude\", hue=\"h_li\", vmin=-100, vmax=2000)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6421f67c",
   "metadata": {
    "user_expressions": []
   },
   "source": [
    "A developer note to users:\n",
    "our next steps will be to create an xarray extension with ICESat-2 aware functions (like \"get_strong_beams\", etc.).\n",
    "Please let us know if you have any ideas or already have functions developed (we can work with you to add them, or add them for you!)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1b0cb477",
   "metadata": {
    "user_expressions": []
   },
   "source": [
    "#### Credits\n",
    "* original notebook by: Jessica Scheick\n",
    "* notebook contributors: Wei Ji and Tian"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "aaf6f5a6-355b-456a-99fd-ce0b51045b58",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8ea1987f-b6bf-44df-a869-949290f498cb",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "icepyx-dev",
   "language": "python",
   "name": "icepyx-dev"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}