docs/notebooks/03_Extracting_forcing_data.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 03 - Extracting forcing data\n",
    "\n",
    "## Extracting meteorological data for a selected watershed\n",
    "Using a GeoJSON file extracted from the HydroSHEDS database or given by the user, meteorological datasets can be extracted inside the watershed's boundaries using the PAVICS-Hydro ERA5 database."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "pycharm": {
     "is_executing": true
    }
   },
   "outputs": [],
   "source": [
    "import datetime as dt\n",
    "import tempfile\n",
    "from pathlib import Path\n",
    "\n",
    "import fsspec  # noqa\n",
    "import intake\n",
    "import numpy as np\n",
    "import s3fs  # noqa\n",
    "import xarray as xr\n",
    "from clisops.core import subset\n",
    "\n",
    "from ravenpy.utilities.testdata import get_file"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we want to extract data for our watershed, we need to know:\n",
    "\n",
    "- The spatial extent (as defined by the watershed boundaries);\n",
    "- The temporal extent (as defined by the start and end days of the period of interest).\n",
    "\n",
    "Let's define those now:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# This will be our input section, where we control what we want to extract.\n",
    "# We know which watershed interests us, it is the input.geojson file that we previously generated!\n",
    "\n",
    "# The contour can be generated using notebook \"01_Delineating watersheds, where it would be placed\n",
    "# in the same folder as the notebooks and available in your workspace. The contour could then be accessed\n",
    "# easily by defining it as follows:\n",
    "\"\"\"\n",
    "basin_contour = \"input.geojson\"\n",
    "\"\"\"\n",
    "# However, to keep things tidy, we have also prepared a version that can be accessed easily for\n",
    "# demonstration purposes:\n",
    "basin_contour = get_file(\"notebook_inputs/input.geojson\")\n",
    "\n",
    "# Also, we can specify which timeframe we want to extract. Here let's focus on a 10-year period\n",
    "reference_start_day = dt.datetime(1985, 12, 31)\n",
    "reference_stop_day = dt.datetime(1987, 1, 1)\n",
    "# Notice we are using one day before and one day after the desired period of 1986-01-01 to 1986-12-31.\n",
    "# This is to account for any UTC shifts that might require getting data in a previous or later time."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We now provide a means to get some data to run our model. Typically, models will require precipitation and temperature data, so let's get that data. We will use a generally reliable dataset that is available everywhere to minimize missing values: the ERA5 Reanalysis.\n",
    "\n",
    "The code block below gathers the required data automatically. If you need other data or want to use another source, this cell will need to be replaced for your customized needs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get the ERA5 data from the Wasabi/Amazon S3 server.\n",
    "catalog_name = \"https://raw.githubusercontent.com/hydrocloudservices/catalogs/main/catalogs/atmosphere.yaml\"\n",
    "cat = intake.open_catalog(catalog_name)\n",
    "ds = cat.era5_reanalysis_single_levels.to_dask()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Get the ERA5 data. We will rechunk it to a single chunk to make it compatible with other codes on the platform, especially bias-correction.\n",
    "We are also taking the daily min and max temperatures as well as the daily total precipitation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "jupyter": {
     "outputs_hidden": false
    }
   },
   "outputs": [],
   "source": [
    "# We will add a wrapper to ensure that the following operations will preserve the original data attributes, such as units and variable names.\n",
    "with xr.set_options(keep_attrs=True):\n",
    "    ERA5_reference = subset.subset_shape(\n",
    "        ds.sel(time=slice(reference_start_day, reference_stop_day)), basin_contour\n",
    "    )\n",
    "    ERA5_tas = ERA5_reference[\"t2m\"].resample(time=\"1D\")\n",
    "    ERA5_tmin = ERA5_tas.min().chunk(-1, -1, -1)\n",
    "    ERA5_tmax = ERA5_tas.max().chunk(-1, -1, -1)\n",
    "    ERA5_pr = ERA5_reference[\"tp\"].resample(time=\"1D\").sum().chunk(-1, -1, -1)\n",
    "\n",
    "ERA5_pr"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can now convert these variables to the desired format and save them to disk in netcdf files to use at a later time (in a future notebook!)\n",
    "\n",
    "First, we will want to make sure that the units we are working with are compatible with the Raven modelling framework. We will want precipitation to be in mm (per time period, here we are working daily so it will be in mm/day), and temperatures will be in °C. Let's check out the current units:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(f\"Tmin units: {ERA5_tmin.units}\")\n",
    "print(f\"Tmax units: {ERA5_tmax.units}\")\n",
    "print(f\"Precipitation units: {ERA5_pr.units}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see that the units are in Kelvin for temperatures and in meters for precipitation. We will want to do some conversions!\n",
    "\n",
    "Let's start by applying offsets for temperatures and a conversion factor for precipitation:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "with xr.set_options(keep_attrs=True):\n",
    "    ERA5_tmin = ERA5_tmin - 273.15  # K to °C\n",
    "    ERA5_tmin.attrs[\"units\"] = \"degC\"\n",
    "\n",
    "    ERA5_tmax = ERA5_tmax - 273.15  # K to °C\n",
    "    ERA5_tmax.attrs[\"units\"] = \"degC\"\n",
    "\n",
    "    ERA5_pr = ERA5_pr * 1000  # m to mm\n",
    "    ERA5_pr.attrs[\"units\"] = \"mm\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see the changes now by re-inspecting the datasets:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(f\"Tmin units: {ERA5_tmin.units}\")\n",
    "print(f\"Tmax units: {ERA5_tmax.units}\")\n",
    "print(f\"Precipitation units: {ERA5_pr.units}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So let's write them to disk for now. We will use the netcdf format as this is what Raven uses for inputs. It is possible you will get some warnings, this is OK and should not cause any problems. Since our model will run in lumped mode, we will average the spatial dimensions of each variable over the domain."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "with xr.set_options(keep_attrs=True):\n",
    "    # Average the variables\n",
    "    ERA5_tmin = ERA5_tmin.mean({\"latitude\", \"longitude\"})\n",
    "    ERA5_tmax = ERA5_tmax.mean({\"latitude\", \"longitude\"})\n",
    "    ERA5_pr = ERA5_pr.mean({\"latitude\", \"longitude\"})\n",
    "\n",
    "    # Ensure that the precipitation is non-negative, which can happen with some reanalysis models.\n",
    "    ERA5_pr = np.maximum(ERA5_pr, 0)\n",
    "\n",
    "    # Transform them to a dataset such that they can be written with attributes to netcdf\n",
    "    ERA5_tmin = ERA5_tmin.to_dataset(name=\"tmin\", promote_attrs=True)\n",
    "    ERA5_tmax = ERA5_tmax.to_dataset(name=\"tmax\", promote_attrs=True)\n",
    "    ERA5_pr = ERA5_pr.to_dataset(name=\"pr\", promote_attrs=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Check and see if the precipitation makes sense:\n",
    "ERA5_pr.pr.plot()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Here we will write the files to disk in a temporary folder since the root folder containing these notebooks is read-only.\n",
    "You can change the path here to your own preferred path in your writable workspace. Alternatively, if you copy this notebook to your writable-workspace as shown in the introduction documentation, you can save just the filename (no absolute path) and the file will appear \"beside\" the notebooks, ready to be read by the next series of notebooks. \n",
    "\n",
    "For this case, you will want to use the second provided option in which all variables are stored in the same netcdf file. This will make the data much easier to find for Raven and prevent some errors, such as if Raven needs to calculate daily average temperature. This will be possible with all variables in the same file, but will cause an error if max and min temperature are in two seperate files."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Option 1, which is not recommended to use in other notebooks but can be really useful in various other workflows:\n",
    "with xr.set_options(keep_attrs=True):\n",
    "    # Write to disk.\n",
    "    tmp = Path(tempfile.mkdtemp())\n",
    "    ERA5_tmin.to_netcdf(tmp / \"ERA5_tmin.nc\")\n",
    "    ERA5_tmax.to_netcdf(tmp / \"ERA5_tmax.nc\")\n",
    "    ERA5_pr.to_netcdf(tmp / \"ERA5_pr.nc\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Option 2, which is recommended, in which we prepare a single file that merges all three variables into one netcdf file:\n",
    "with xr.set_options(keep_attrs=True):\n",
    "    xr.merge([ERA5_tmin, ERA5_tmax, ERA5_pr]).to_netcdf(tmp / \"ERA5_weather_data.nc\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We now have daily precipitation and minimum/maximum temperatures to drive our Raven Model, which we will do in the next notebook!\n",
    "\n",
    "Note that our dataset generated here is very short (1 year) but the same dataset for the period 1980-12-31 to 1991-01-01 has been pre-generated and stored on the server for efficiency.\n"
   ]
  }
 ],
 "metadata": {
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.13"
  },
  "nbdime-conflicts": {
   "local_diff": [
    {
     "diff": [
      {
       "diff": [
        {
         "key": 0,
         "op": "addrange",
         "valuelist": [
          "3.6.7"
         ]
        },
        {
         "key": 0,
         "length": 1,
         "op": "removerange"
        }
       ],
       "key": "version",
       "op": "patch"
      }
     ],
     "key": "language_info",
     "op": "patch"
    }
   ],
   "remote_diff": [
    {
     "diff": [
      {
       "diff": [
        {
         "key": 0,
         "op": "addrange",
         "valuelist": [
          "3.6.10"
         ]
        },
        {
         "key": 0,
         "length": 1,
         "op": "removerange"
        }
       ],
       "key": "version",
       "op": "patch"
      }
     ],
     "key": "language_info",
     "op": "patch"
    }
   ]
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}