docs/notebooks/6_config.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "0",
   "metadata": {},
   "source": [
    "# YAML usage\n",
    "\n",
    "<div class=\"alert alert-info\"> <b>NOTE</b> \n",
    "    \n",
    "This tutorial will mostly remain xscen-specific and, thus, will not go into more advanced YAML functionalities such as anchors. More information on that can be consulted [here](https://support.atlassian.com/bitbucket-cloud/docs/yaml-anchors/), while [this template](https://github.com/Ouranosinc/xscen/blob/main/templates/1-basic_workflow_with_config/config1.yml) makes ample use of them.\n",
    "\n",
    "</div>\n",
    "\n",
    "While parameters can be explicitely given to functions, most support the use of YAML configuration files to automatically pass arguments. This tutorial will go over basic principles on how to write and prepare configuration files, and provide a few examples.\n",
    "\n",
    "An `xscen` function supports YAML parametrisation if it is preceded by the `parse_config` wrapper in the code. Currently supported functions are:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1",
   "metadata": {},
   "outputs": [],
   "source": [
    "from xscen.config import get_configurable\n",
    "\n",
    "list(get_configurable().keys())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2",
   "metadata": {},
   "source": [
    "## Loading an existing YAML config file\n",
    "\n",
    "YAML files are read using `xscen.load_config`. Any number of files can be called, which will be merged together into a single python dictionary accessed through `xscen.CONFIG`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from pathlib import Path\n",
    "\n",
    "import xscen as xs\n",
    "from xscen import CONFIG"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Load configuration\n",
    "xs.load_config(\n",
    "    str(\n",
    "        Path().absolute().parent.parent\n",
    "        / \"templates\"\n",
    "        / \"1-basic_workflow_with_config\"\n",
    "        / \"config1.yml\"\n",
    "    ),\n",
    "    # str(Path().absolute().parent.parent / \"templates\" / \"1-basic_workflow_with_config\" / \"paths1_example.yml\")  We can't actually load this file due to the fake paths, but this would be the format\n",
    ")\n",
    "\n",
    "# Display the dictionary keys\n",
    "print(CONFIG.keys())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5",
   "metadata": {
    "tags": []
   },
   "source": [
    "`xscen.CONFIG` behaves similarly to a python dictionary, but has a custom `__getitem__` that returns a `deepcopy` of the requested item. As such, it is unmutable and thus, reliable and robust."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# A normal python dictionary is mutable, but a CONFIG dictionary is not.\n",
    "pydict = dict(CONFIG[\"project\"])\n",
    "print(CONFIG[\"project\"][\"id\"], \", \", pydict[\"id\"])\n",
    "pydict2 = pydict\n",
    "pydict2[\"id\"] = \"modified id\"\n",
    "print(CONFIG[\"project\"][\"id\"], \", \", pydict[\"id\"], \", \", pydict2[\"id\"])\n",
    "pydict3 = pydict2\n",
    "pydict3[\"id\"] = \"even more modified id\"\n",
    "print(\n",
    "    CONFIG[\"project\"][\"id\"],\n",
    "    \", \",\n",
    "    pydict[\"id\"],\n",
    "    \", \",\n",
    "    pydict2[\"id\"],\n",
    "    \", \",\n",
    "    pydict3[\"id\"],\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7",
   "metadata": {},
   "source": [
    "If one really want to modify the `CONFIG` dictionary from within the workflow itself, its `set` method must be used."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8",
   "metadata": {},
   "outputs": [],
   "source": [
    "CONFIG.set(\"project.id\", \"modified id\")\n",
    "print(CONFIG[\"project\"][\"id\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9",
   "metadata": {},
   "source": [
    "## Building a YAML config file\n",
    "### Generic arguments\n",
    "\n",
    "Since `CONFIG` is a python dictionary, anything can be written in it if it is deemed useful for the execution of the script. A good practice, such as seen in [this template's config1.yml](https://github.com/Ouranosinc/xscen/tree/main/templates/1-basic_workflow_with_config/config1.yml), is for example to use the YAML file to provide a list of tasks to be accomplished, give the general description of the project, or provide a dask configuration:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "10",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(CONFIG[\"tasks\"])\n",
    "print(CONFIG[\"project\"])\n",
    "print(CONFIG[\"regrid\"][\"dask\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "11",
   "metadata": {},
   "source": [
    "These are not linked to any function and will not automatically be called upon by `xscen`, but can be referred to during the execution of the script. Below is an example where `tasks` is used to instruct on which tasks to accomplish and which to skip. Many such example can be seen throughout [the provided templates](https://github.com/Ouranosinc/xscen/tree/main/templates)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "12",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "if \"extract\" in CONFIG[\"tasks\"]:\n",
    "    print(\"This will start the extraction process.\")\n",
    "\n",
    "if \"figures\" in CONFIG[\"tasks\"]:\n",
    "    print(\n",
    "        \"This would start creating figures, but it will be skipped since it is not in the list of tasks.\"\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "13",
   "metadata": {},
   "source": [
    "### Function-specific parameters\n",
    "\n",
    "In addition to generic arguments, a major convenience of YAML files is that parameters can be automatically fed to functions if they are wrapped by `@parse_config` (see above for the list of currently supported functions). The exact following format has to be used:\n",
    "\n",
    "```\n",
    "module:\n",
    "    function:\n",
    "        argument:\n",
    "```\n",
    "\n",
    "The most up-to-date list of modules can be consulted [here](https://xscen.readthedocs.io/en/latest/apidoc/modules.html), as well as at the start of this tutorial. A simple example would be as follows:\n",
    "```\n",
    "aggregate:\n",
    "  compute_deltas:\n",
    "    kind: \"+\"\n",
    "    reference_horizon: \"1991-2020\"\n",
    "    to_level: 'delta'\n",
    "```\n",
    "\n",
    "Some functions have arguments in the form of lists and dictionaries. These are also supported:\n",
    "```\n",
    "extract:\n",
    "    search_data_catalogs:\n",
    "      variables_and_freqs:\n",
    "        tasmax: D\n",
    "        tasmin: D\n",
    "        pr: D\n",
    "        dtr: D\n",
    "      allow_resampling: False\n",
    "      allow_conversion: True\n",
    "      periods: ['1991', '2020']\n",
    "      other_search_criteria:\n",
    "        source:\n",
    "          \"ERA5-Land\"\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "14",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Note that the YAML used here is more complex and separates tasks between 'reconstruction' and 'simulation', which would break the automatic passing of arguments.\n",
    "print(\n",
    "    CONFIG[\"extract\"][\"reconstruction\"][\"search_data_catalogs\"][\"variables_and_freqs\"]\n",
    ")  # Dictionary\n",
    "print(CONFIG[\"extract\"][\"reconstruction\"][\"search_data_catalogs\"][\"periods\"])  # List"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "15",
   "metadata": {},
   "source": [
    "Let's test that it is working, using `climatological_op`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "16",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# We should obtain 30-year means separated in 10-year intervals.\n",
    "CONFIG[\"aggregate\"][\"climatological_op\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "17",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import xarray as xr\n",
    "\n",
    "# Create a dummy dataset\n",
    "time = pd.date_range(\"1951-01-01\", \"2100-01-01\", freq=\"YS-JAN\")\n",
    "da = xr.DataArray([0] * len(time), coords={\"time\": time})\n",
    "da.name = \"test\"\n",
    "ds = da.to_dataset()\n",
    "\n",
    "# Call climatological_op using no argument other than what's in CONFIG\n",
    "print(xs.climatological_op(ds))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "18",
   "metadata": {},
   "source": [
    "### Managing paths\n",
    "\n",
    "As a final note, it should be said that YAML files are a good way to privately provide paths to a script without having to explicitely write them in the code. [An example is provided here](https://github.com/Ouranosinc/xscen/blob/main/templates/1-basic_workflow_with_config/paths1_example.yml). As stated earlier, `xs.load_config` will merge together the provided YAML files into a single dictionary, meaning that the separation will be seamless once the script is running.\n",
    "\n",
    "As an added protection, if the script is to be hosted on Github, `paths.yml` (or whatever it is being called) can then be added to the `.gitignore`.\n",
    "\n",
    "### Configuration of external packages\n",
    "As explained in the `load_config` [documentation](https://xscen.readthedocs.io/en/latest/api.html#special-sections), a few top-level sections can be used to configure packages external to xscen. For example, everything under the `logging` section will be sent to `logging.config.dictConfig(...)`, allowing the [full configuration](https://docs.python.org/3/library/logging.config.html#configuration-dictionary-schema) of python's built-in logging mechanism. The current config does exactly that by configuring a logger for `xscen` that logs to the console, with a sensibility set to the INFO level and a specified record formating :"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "19",
   "metadata": {},
   "outputs": [],
   "source": [
    "CONFIG[\"logging\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "20",
   "metadata": {},
   "source": [
    "## Passing configuration through the command line\n",
    "In order to have a more flexible configuration, it can be interesting to modify it using the command line. This way, the workflow can be started with different values without having to edit and save the YAML file each time. Alternatively, the command line arguments can also be used to determine which configuration file to use, so that the same workflow can be launched with different configurations without needing to duplicate the code. The [second template workflow](https://github.com/Ouranosinc/xscen/blob/main/templates/2-indicators_only/workflow2.py) uses this method.\n",
    "\n",
    "The idea is simply to create an `ArgumentParser` with python's built-in [argparse](https://docs.python.org/3/library/argparse.html) :"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "21",
   "metadata": {},
   "outputs": [],
   "source": [
    "from argparse import ArgumentParser\n",
    "\n",
    "parser = ArgumentParser(description=\"An example CLI arguments parser.\")\n",
    "parser.add_argument(\"-c\", \"--conf\", action=\"append\")\n",
    "\n",
    "# Let's simulate command line arguments\n",
    "example_args = (\n",
    "    \"-c ../../templates/2-indicators_only/config2.yml \"\n",
    "    '-c project.title=\"Title\" '\n",
    "    \"--conf project.id=newID\"\n",
    ")\n",
    "\n",
    "args = parser.parse_args(example_args.split())\n",
    "print(args.conf)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "22",
   "metadata": {},
   "source": [
    "And then we can simply pass this list to `load_config`, which accepts file paths and \"key=value\" pairs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "23",
   "metadata": {},
   "outputs": [],
   "source": [
    "xs.load_config(*args.conf)\n",
    "\n",
    "print(CONFIG[\"project\"][\"title\"])\n",
    "print(CONFIG[\"project\"][\"id\"])"
   ]
  }
 ],
 "metadata": {
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}