tutorials/delta_extraction/picca_delta_extraction_configuration_tutorial.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "toc": true
   },
   "source": [
    "<h1>Table of Contents<span class=\"tocSkip\"></span></h1>\n",
    "<div class=\"toc\"><ul class=\"toc-item\"><li><span><a href=\"#Running-picca_delta_extraction\" data-toc-modified-id=\"Running-picca_delta_extraction-1\"><span class=\"toc-item-num\">1&nbsp;&nbsp;</span>Running picca_delta_extraction</a></span></li><li><span><a href=\"#How-to-build-a-config-file:-General-structure\" data-toc-modified-id=\"How-to-build-a-config-file:-General-structure-2\"><span class=\"toc-item-num\">2&nbsp;&nbsp;</span>How to build a config file: General structure</a></span><ul class=\"toc-item\"><li><span><a href=\"#General-section\" data-toc-modified-id=\"General-section-2.1\"><span class=\"toc-item-num\">2.1&nbsp;&nbsp;</span>General section</a></span></li><li><span><a href=\"#Data-section\" data-toc-modified-id=\"Data-section-2.2\"><span class=\"toc-item-num\">2.2&nbsp;&nbsp;</span>Data section</a></span><ul class=\"toc-item\"><li><span><a href=\"#SdssData\" data-toc-modified-id=\"SdssData-2.2.1\"><span class=\"toc-item-num\">2.2.1&nbsp;&nbsp;</span>SdssData</a></span></li><li><span><a href=\"#DesiHealpix\" data-toc-modified-id=\"DesiHealpix-2.2.2\"><span class=\"toc-item-num\">2.2.2&nbsp;&nbsp;</span>DesiHealpix</a></span></li><li><span><a href=\"#DesiTile\" data-toc-modified-id=\"DesiTile-2.2.3\"><span class=\"toc-item-num\">2.2.3&nbsp;&nbsp;</span>DesiTile</a></span></li><li><span><a href=\"#DesisimMocks\" data-toc-modified-id=\"DesisimMocks-2.2.4\"><span class=\"toc-item-num\">2.2.4&nbsp;&nbsp;</span>DesisimMocks</a></span></li></ul></li><li><span><a href=\"#Corrections-section\" data-toc-modified-id=\"Corrections-section-2.3\"><span class=\"toc-item-num\">2.3&nbsp;&nbsp;</span>Corrections section</a></span><ul class=\"toc-item\"><li><span><a href=\"#SdssCalibrationCorrection\" data-toc-modified-id=\"SdssCalibrationCorrection-2.3.1\"><span class=\"toc-item-num\">2.3.1&nbsp;&nbsp;</span>SdssCalibrationCorrection</a></span></li><li><span><a href=\"#SdssDustCorrection\" data-toc-modified-id=\"SdssDustCorrection-2.3.2\"><span class=\"toc-item-num\">2.3.2&nbsp;&nbsp;</span>SdssDustCorrection</a></span></li><li><span><a href=\"#SdssIvarCorrection\" data-toc-modified-id=\"SdssIvarCorrection-2.3.3\"><span class=\"toc-item-num\">2.3.3&nbsp;&nbsp;</span>SdssIvarCorrection</a></span></li><li><span><a href=\"#SdssOpticalDepthCorrection\" data-toc-modified-id=\"SdssOpticalDepthCorrection-2.3.4\"><span class=\"toc-item-num\">2.3.4&nbsp;&nbsp;</span>SdssOpticalDepthCorrection</a></span></li></ul></li><li><span><a href=\"#Masks-section\" data-toc-modified-id=\"Masks-section-2.4\"><span class=\"toc-item-num\">2.4&nbsp;&nbsp;</span>Masks section</a></span><ul class=\"toc-item\"><li><span><a href=\"#AbsorberMask\" data-toc-modified-id=\"AbsorberMask-2.4.1\"><span class=\"toc-item-num\">2.4.1&nbsp;&nbsp;</span>AbsorberMask</a></span></li><li><span><a href=\"#DlaMask\" data-toc-modified-id=\"DlaMask-2.4.2\"><span class=\"toc-item-num\">2.4.2&nbsp;&nbsp;</span>DlaMask</a></span></li><li><span><a href=\"#BalMask\" data-toc-modified-id=\"BalMask-2.4.3\"><span class=\"toc-item-num\">2.4.3&nbsp;&nbsp;</span>BalMask</a></span></li></ul></li><li><span><a href=\"#Expected-Flux-section\" data-toc-modified-id=\"Expected-Flux-section-2.5\"><span class=\"toc-item-num\">2.5&nbsp;&nbsp;</span>Expected Flux section</a></span><ul class=\"toc-item\"><li><span><a href=\"#Dr16ExpectedFlux\" data-toc-modified-id=\"Dr16ExpectedFlux-2.5.1\"><span class=\"toc-item-num\">2.5.1&nbsp;&nbsp;</span>Dr16ExpectedFlux</a></span></li></ul></li></ul></li><li><span><a href=\"#Examples\" data-toc-modified-id=\"Examples-3\"><span class=\"toc-item-num\">3&nbsp;&nbsp;</span>Examples</a></span></li></ul></div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Running picca_delta_extraction\n",
    "\n",
    "In order to compute the Lyman $\\alpha$ delta field you need to run the code picca_delta_extraction. In this tutorial we are going to see how. If you are running this tutorial, you should have the package picca installed. If you do not have it, then check the INSTALLATION section of the README\n",
    "\n",
    "The usage of the code is very simple. You just need to run from the command line \n",
    "\n",
    "`python picca_delta_extraction.py config.ini`\n",
    "\n",
    "where `config.ini` is a file with the desired configuration. \n",
    "\n",
    "Alternatively, you might want to run it from this jupyter notebook. In that case we do it like this:\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# import main function from delta extraction \n",
    "import sys\n",
    "import os\n",
    "sys.path.append(os.path.expanduser('~/software/picca/bin/')) # this should be changed with your path to picca\n",
    "from picca_delta_extraction import main as delta_extraction"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# run it using a configuration file\n",
    "import argparse\n",
    "parser = argparse.ArgumentParser(\n",
    "        formatter_class=argparse.ArgumentDefaultsHelpFormatter,\n",
    "        description=('Compute the delta field from a list of spectra'))\n",
    "\n",
    "parser.add_argument('config-file', type=str, default=None, help='Configuration file')\n",
    "\n",
    "# this is the bit of code where we specify the config file \n",
    "# it is not going to work now because we are poitning to a non-existant file\n",
    "args = parser.parse_args(\"my_config.ini\") \n",
    "\n",
    "print(\"WARNING: depending on the configuration file used this might take a lot of time\")\n",
    "delta_extraction(args)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Some questions you might have:\n",
    "- What if I want to run without saving the configuration in a file? Unfortunatly no, at least for now. A different initialisation of the Config class would need to be coded.\n",
    "- Can I check the configuration of a previous run? Yes, the configuration for a given run is stored at a file named .config.ini in the output folder \n",
    "- How is the code working? You can take a look at the Workflow and Data Model diagrams under `doc/`\n",
    "- Can I use enviroment variables in my config arguments? Yes, they will be expanded before calling the specified file\n",
    "\n",
    "- How can I create my own config? This is where things start to get tricky. We are now going to review how to build a configuration file, and the different options that are available."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# How to build a config file: General structure\n",
    "\n",
    "The configuration file is nothing but a bunch of arguments that are structured in different sections. They are then parsed into usable format. To create a section simply type its name inside of square braquets:\n",
    "\n",
    "`[new section]`\n",
    "\n",
    "To add an argument to this section specify it simply:\n",
    "\n",
    "`\n",
    "my_string_argument = \"a string\"\n",
    "my_int_argument = 1\n",
    "my_float_argument = 1.4\n",
    "my_bool_argument = True\n",
    "also_a_bool_argument = yes\n",
    "`\n",
    "\n",
    "Note that the arguments and sections are not ordered, but all the arguments in a given section will need to be placed before the next section begins.\n",
    "\n",
    "Now that we have seen a short general introduction on config files, let's move to the more specific task at hand: how to build a config file for `picca_delta_extraction`. In this configuration we need some sections to be present and each section will require some specific parameters. Depending on the options, other sections might be required/used, but we'll come back to this later. These are the mandatory sections:\n",
    "- `[general]`\n",
    "- `[data]`\n",
    "- `[corrections]`\n",
    "- `[masks]`\n",
    "- `[expected flux]`\n",
    "\n",
    "We now revise the arguments of the different sections. Note that if other arguments are passed, they will be ignored."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## General section\n",
    "The general section sets common options that are not really used by any of the classes specifically. The accepted arguments for this section are\n",
    "- `out dir`: This variable specifies where to save the results. **Type: str, Required: yes**\n",
    "- `overwrite`: This variable allows you to overwrite the results from a previous run. Unless this is specified, then the code will crash if the selected output folder already have the results of a previous run. **Type: bool, Required: no, Default: False**\n",
    "- `log`: If a log file is passed, print messages also there **Type: str, Required: no, Default: run.log**\n",
    "- `logging level console`: This variable controls console messages. Must be one of CRITICAL, ERROR, WARNING_OK, WARNING, INFO, PROGRESS, DEBUG, NOTSET **Type: str, Required: no, Default: DEBUG**\n",
    "- `logging level file`: This variable controls log messages. Must be one of CRITICAL, ERROR, WARNING_OK, WARNING, INFO, PROGRESS, DEBUG, NOTSET. Ignored if `log` is `None` **Type: str, Required: no, Default: DEBUG**.\n",
    "- `num processors`: Number of processors to be used for multiprocessed tasks (e.g. data i/o, expected flux). 0 for using half the processes available on the machine (subprocess will take its default value). **Type: int or None, Required: no, Default: 0**\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data section\n",
    "This section controls which type of data we are going to load. For example, we might want to run picca on SDSS data or on DESI data. The main arguments are\n",
    "- `type`: Contains the name of the class used to load your data. For example \"DesiData\" or \"SdssData\". Basically can be any data type stored in the folder `py/picca/delta_extraction/data/`. **Type: str, Required: yes**\n",
    "- `module name`: Name of the file containing the data type. If the file is named following the convention this should normally not be necessary. **Type: str, Required: no**\n",
    "\n",
    "There are some arguments that are passed for any `type`:\n",
    "\n",
    "- `analysis type`: Selected analysis type. Current options are \"BAO 3D\" or \"PK 1D\" **Type: str, Required: no, Default: \"BAO 3D\"\n",
    "- `input directory`: Directory to spectra files. **Type: str, Required: yes**\n",
    "- `lambda abs IGM`: Wavelength of the IGM absorber. Must be one of the keys of `ABSORBER_IGM` in `delta_extraction/utils.py`. Used only if `analysis type` is \"PK 1D\". **Type: str, Required: no, Default: LYA**\n",
    "- `lambda max`: Upper limit on observed wavelength [Angstrom] **Type: float, Required: no, Default: 5500.0**\n",
    "- `lambda max rest frame`: Upper limit on rest frame wavelength [Angstrom] **Type: float, Required: no, Default: 1200.0**\n",
    "- `lambda min`: Lower limit on observed wavelength [Angstrom] **Type: float, Required: no, Default: 3600.0**\n",
    "- `lambda min rest frame`: Lower limit on rest frame wavelength [Angstrom] **Type: float, Required: no, Default: 1040.0**\n",
    "- `minimal snr`: Minimal S/N ratio required for forests to be accepted. **Type: float, Required: no, Default: 0 for `analysis type == BAO 3D` analyses, 1 for `analysis type == Pk 1D`\n",
    "- `minimum number pixels in forest`: Minimum number of pixels in a forest. Forests with less pixels will be dropped. **Type: int, Required: no, Default: 50**\n",
    "- `max num spec`: Maximum number of spectra to read. None for no maximum. **Type: int or None, Required: no, Default: None**\n",
    "- `rejection log file`: Filelame of the rejection log. Must have extension .fits or .fits.gz **Type: str, Required: no, Default: \"rejection_log.fits.gz\"**\n",
    "- `z max`: Maximum redshift. Quasars with redshifts higher than or equal to z_max will be discarded. If not specified will be computed at runtime base on the values of `lambda max` and `lambda min rest frame` as $z_{\\rm max} = \\max\\left(0, \\lambda_{\\rm max} / \\lambda_{\\rm min, rf} - 1\\right)$\n",
    "- `z min`: Minimum redshift. Quasars with redshifts lower than z_min will be discarded. If not specified will be computed at runtime base on the values of `lambda min` and `lambda max rest frame` as $z_{\\rm min} = \\max\\left(0, \\lambda_{\\rm min} / \\lambda_{\\rm max, rf} - 1\\right)$ **Type: float, Required: False**\n",
    "\n",
    "Other arguments will be passed to the constructor of the selected class. Let's review the arguments required by the available Data types\n",
    "\n",
    "###  SdssData\n",
    "\n",
    "will always use `wave solution = log`\n",
    "\n",
    "- `best obs`: If True, reads only the best observation for objects with repeated observations **Type: bool, Required: no, Default: False**\n",
    "- `BI max`: Maximum value allowed for the Balnicity Index to keep the quasar **Type: float or None, Required: no**\n",
    "- `drq catalogue`: Name of the quasar catalogue (in DRQ format) **Type: str, Required: yes** \n",
    "- `keep BAL`: If False, remove the quasars flagged as having a Broad Absorption Line. Ignored if bi_max is not None **Type: str, Required: no, Default: False**\n",
    "- `mode`: Reading mode. Currently supported reading modes are \"spplate\" and \"spec\". **Type: str, Required: no, Default: \"spplate\"**\n",
    "- `rebin`: Rebin wavelength grid by combining this number of adjacent pixels (ivar weight). **Type: int, Required: no, Default: 3**\n",
    "- `rejection log file`: Filelame of the rejection log. Must have extension .fits or .fits.gz **Type: str, Required: no, Default: \"rejection_log.fits.gz\"**\n",
    "- `spAll`: Path to the spAll file required for multiple observations. If not given, it will be looked for using `input directory`. **Type: str, Required: no**\n",
    "- `z max`: Maximum redshift. Quasars with redshifts higher than or equal to z_max will be discarded. If not specified will be computed at runtime base on the values of `lambda max` and `lambda min rest frame` as $z_{\\rm max} = \\max\\left(0, \\lambda_{\\rm max} / \\lambda_{\\rm min, rf} - 1\\right)$ **Type: float, Required: False, Default: 3.5**\n",
    "- `z min`: Minimum redshift. Quasars with redshifts lower than z_min will be discarded. If not specified will be computed at runtime base on the values of `lambda min` and `lambda max rest frame` as $z_{\\rm min} = \\max\\left(0, \\lambda_{\\rm min} / \\lambda_{\\rm max, rf} - 1\\right)$ **Type: float, Required: False**\n",
    "\n",
    "\n",
    "### DesiHealpix\n",
    "\n",
    "- `blinding`: Selected blinding strategy (\"none\" for no blinding). Available strategies listed in `py/picca/delta_extraction/utils.py` in variable `ACCEPTED_BLINDING_STRATEGIES`. For the moment, blinding strategy \"corr_yshift\" is automatically engaged for main survey data, and blinding strategy \"none\" is automatically engaged for SV data or mocks. **Type: str, Required: no, Default: \"corr_yshift\"**\n",
    "- `catalogue`: Name the z_truth catalogue. **Type: str, Required: yes**\n",
    "- `delta lambda`: Variation of the wavelength (in Angs) between two pixels. Ignored if `wave solution` is \"lin\" **Type: float, Required: no, Default: 1.0**\n",
    "- `delta log lambda`: Variation of the logarithm of the wavelength (in Angs) between two pixels. Ignored if `wave solution` is \"log\"  **Type: float, Required: no, Default: 1e-4**\n",
    "- `keep surveys`: Only keep the entries in the catalogue that have a \"SURVEY\" specified in field. Ignored if \"SURVEY\" column is not present in the catalogue. Multiple values can be added spearated by white spaces. Specifying \"all\" is equivalent to specifying \"sv1 sv2 sv3 main\". Accepted values are \"sv1\", \"sv2\", \"sv3\", \"main\", \"all\" and \"special\" **Type: str, Required: no, Default: \"all\"**\n",
    "- `wave solution`: Use linear (lin) or logarithmic wavelength solution. **Type: str, Required: no, Default: \"lin\"**\n",
    "- `delta lambda`: Variation of the wavelength (in Angs) between two pixels. Ignored if `wave solution` is \"lin\" **Type: float, Required: no, Default: 1.0**\n",
    "- `delta log lambda`: Variation of the logarithm of the wavelength (in Angs) between two pixels. Ignored if `wave solution` is \"log\"  **Type: float, Required: no, Default: 1e-4**\n",
    "\n",
    "\n",
    "\n",
    "### DesiTile\n",
    "\n",
    "- `blinding`: Selected blinding strategy (\"none\" for no blinding). Available strategies listed in `py/picca/delta_extraction/utils.py` in variable `ACCEPTED_BLINDING_STRATEGIES`. For the moment, blinding strategy \"corr_yshift\" is automatically engaged for main survey data, and blinding strategy \"none\" is automatically engaged for SV data or mocks. **Type: str, Required: no, Default: \"corr_yshift\"**\n",
    "- `catalogue`: Name the z_truth catalogue. **Type: str, Required: yes**\n",
    "- `delta lambda`: Variation of the wavelength (in Angs) between two pixels. Ignored if `wave solution` is \"lin\" **Type: float, Required: no, Default: 1.0**\n",
    "- `delta log lambda`: Variation of the logarithm of the wavelength (in Angs) between two pixels. Ignored if `wave solution` is \"log\"  **Type: float, Required: no, Default: 1e-4**\n",
    "- `keep surveys`: Only keep the entries in the catalogue that have a \"SURVEY\" specified in field. Ignored if \"SURVEY\" column is not present in the catalogue. Multiple values can be added spearated by white spaces. Specifying \"all\" is equivalent to specifying \"sv1 sv2 sv3 main\". Accepted values are \"sv1\", \"sv2\", \"sv3\", \"main\", \"all\" and \"special\" **Type: str, Required: no, Default: \"all\"**\n",
    "- `rejection log file`: Filelame of the rejection log. Must have extension .fits or .fits.gz **Type: str, Required: no, Default: \"rejection_log.fits.gz\"**\n",
    "- `use all`: Read using the all directory. **Type: bool, Required: no, Default: False**\n",
    "- `use single nights`: Read using only nights specified within the cat **Type: bool, Required: no, Default: False**\n",
    "- `wave solution`: Use linear (lin) or logarithmic wavelength solution. **Type: str, Required: no, Default: \"lin\"**\n",
    "- `z max`: Maximum redshift. Quasars with redshifts higher than or equal to z_max will be discarded. If not specified will be computed at runtime base on the values of `lambda max` and `lambda min rest frame` as $z_{\\rm max} = \\max\\left(0, \\lambda_{\\rm max} / \\lambda_{\\rm min, rf} - 1\\right)$\n",
    "- `z min`: Minimum redshift. Quasars with redshifts lower than z_min will be discarded. If not specified will be computed at runtime base on the values of `lambda min` and `lambda max rest frame` as $z_{\\rm min} = \\max\\left(0, \\lambda_{\\rm min} / \\lambda_{\\rm max, rf} - 1\\right)$ **Type: float, Required: False**\n",
    "\n",
    "### DesisimMocks\n",
    "- `analysis type`: Selected analysis type. Current options are \"BAO 3D\" or \"PK 1D\" **Type: str, Required: no, Default: \"BAO 3D\"\n",
    "- `blinding`: Selected blinding strategy (\"none\" for no blinding). Available strategies listed in `py/picca/delta_extraction/utils.py` in variable `ACCEPTED_BLINDING_STRATEGIES`. For the moment, blinding strategy \"corr_yshift\" is automatically engaged for main survey data, and blinding strategy \"none\" is automatically engaged for SV data or mocks. **Type: str, Required: no, Default: \"corr_yshift\"**\n",
    "- `catalogue`: Name the z_truth catalogue. **Type: str, Required: yes**\n",
    "- `delta lambda`: Variation of the wavelength (in Angs) between two pixels. Ignored if `wave solution` is \"lin\" **Type: float, Required: no, Default: 1.0**\n",
    "- `delta log lambda`: Variation of the logarithm of the wavelength (in Angs) between two pixels. Ignored if `wave solution` is \"log\"  **Type: float, Required: no, Default: 1e-4**\n",
    "- `keep surveys`: Only keep the entries in the catalogue that have a \"SURVEY\" specified in field. Ignored if \"SURVEY\" column is not present in the catalogue. Multiple values can be added spearated by white spaces. Specifying \"all\" is equivalent to specifying \"sv1 sv2 sv3 main\". Accepted values are \"sv1\", \"sv2\", \"sv3\", \"main\", \"all\" and \"special\" **Type: str, Required: no, Default: \"all\"**\n",
    "- `wave solution`: Use linear (lin) or logarithmic wavelength solution. **Type: str, Required: no, Default: \"lin\"**\n",
    "- `z max`: Maximum redshift. Quasars with redshifts higher than or equal to z_max will be discarded. If not specified will be computed at runtime base on the values of `lambda max` and `lambda min rest frame` as $z_{\\rm max} = \\max\\left(0, \\lambda_{\\rm max} / \\lambda_{\\rm min, rf} - 1\\right)$\n",
    "- `z min`: Minimum redshift. Quasars with redshifts lower than z_min will be discarded. If not specified will be computed at runtime base on the values of `lambda min` and `lambda max rest frame` as $z_{\\rm min} = \\max\\left(0, \\lambda_{\\rm min} / \\lambda_{\\rm max, rf} - 1\\right)$ **Type: float, Required: False**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Corrections section\n",
    "This section controls the number and types of corrections that will be applied to data. Unlike the data section, the arguments will be passed in separate specific sections as different corrections might require arguments with the same name but with different values. Let's see how do we do this. At least one argument must be passed:\n",
    "- `num corrections`: Number of corrections. **Type: non-negative int, Required=yes**\n",
    "This variable controls the number of corrections we will load. If the value is 0, no corrections will be loaded. In all other cases, we need the subsequent variables\n",
    "- `type 0`: Contains the name of the first correction. Basically can be any data type stored in the folder `py/picca/delta_extraction/corrections/`. Arguments for this correction should be passed in section `[correction arguments 0]``\n",
    " **Type: str**\n",
    "- `type 1`: Similarly for the second correction\n",
    "- ...\n",
    "- `type 'n-1'`: Similarly for the nth correction. \n",
    "As many arguments as specified in `num corrections` are required. If the Cth correction does not require any argument, then the section `[correction arguments C]` is optional. Optionally we can pass the arguments\n",
    "- `module name 0`: Name of the file containing the correction type specified in `type 0`. If the file is named following the convention this should normally not be necessary. **Type: str**\n",
    "- ...\n",
    "- `module name 'n-1'`: Similarly for the correction type specified in `type 'n-1'`.\n",
    "\n",
    "Let's review the arguments required by the available Corrections\n",
    " \n",
    "### SdssCalibrationCorrection\n",
    "- `filename`: Path to the file containing the multiplicative errors in the pipeline flux calibration. Must contain extension STACK_DELTAS with fields 'LOGLAM' and 'STACK' **Type: str, Required: yes**\n",
    "\n",
    "### SdssDustCorrection\n",
    "- `filename`: Path to DRQ catalog of objects for dust map to apply the Schlegel correction. Must contain extension EXTINCTION with fields THING_ID' and 'EXTINCTION' **Type: str, Required: yes**\n",
    "\n",
    "### SdssIvarCorrection\n",
    "- `filename`: Path to the fits file containing the multiplicative errors in the ivar calibration. Must contain extension VAR_FUNC with fields 'LOGLAM' and 'STACK' **Type: str, Required: yes**\n",
    "\n",
    "### SdssOpticalDepthCorrection\n",
    "- `optical depth tau`: White spaced list of $\\tau$ for each of the optical depth absorbers **Type: list of float, Required: yes**\n",
    "- `optical depth gamma` White spaced list of $\\gamma$ for each of the optical depth absorbers **Type: list of float, Required: yes**\n",
    "- `optical depth absorber` White spaced list of absorbers names for each of the optical depth absorbers. Names should match the species specifie in the dictionary `ABSORBER_IGM` specified in `py/picca/delta_extraction/utils.py` **Type: list of string, Required: yes**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Masks section\n",
    "This section controls the number and types of masks that will be applied. It is constructed in a similar way as the corrections section above. At least one argument must be passed:\n",
    "- `num masks`: Number of masks. **Type: non-negative int, Required=yes**\n",
    "This variable controls the number of masks we will load. If the value is 0, no masks will be loaded. In all other cases, we need the subsequent variables\n",
    "- `type 0`: Contains the name of the first mask. Basically can be any data type stored in the folder `py/picca/delta_extraction/corrections/`. Arguments for this correction should be passed in section [mask arguments 0]\n",
    "- `type 1`: Similarly for the second mask\n",
    "- ...\n",
    "- `type 'n-1'`: Similarly for the nth mask. \n",
    "As many arguments as specified in `num masks` are required. If the Cth correction does not require any argument, then the section `[mask arguments C]` is optional.  Optionally we can pass the arguments\n",
    "- `module name 0`: Name of the file containing the mask type specified in `type 0`. If the file is named following the convention this should normally not be necessary. **Type: str**\n",
    "- ...\n",
    "- `module name 'n-1'`: Similarly for the mask type specified in `type 'n-1'`.\n",
    "\n",
    "Let's review the arguments required by the available Masks\n",
    "\n",
    "### AbsorberMask\n",
    "- `absorber mask width`: Mask width on each side of the absorber central observed wavelength in units of 1e4*dlog10(lambda) **Type: float, Required: no, Default: 2.5**\n",
    "- `filename`: Absorber catalog file **Type: str, Required: yes**\n",
    "- `los_id name`: Name of the line of sight identifier (e.g. THING_ID in SDSS or TARGETID in DESI). **Type: str, Required: no, Default: THING_ID**\n",
    "\n",
    "### DlaMask\n",
    "- `dla mask limit`: Lower limit on the DLA transmission. Transmissions below this number are masked. **Type: float, Required: no, Default: 0.8**\n",
    "- `filename`: DLA catalog file. **Type: str, Required: yes**\n",
    "- `los_id name`: Name of the line of sight identifier (e.g. THING_ID in SDSS or TARGETID in DESI). **Type: str, Required: no, Default: THING_ID**\n",
    "- `mask file`: File with regions to mask in lambda_RF. In file each line is: 'region_name region_min region_max RF_DLA' Wavelenths are in [Angstrom] **Type: str, Required: no**\n",
    "\n",
    "### BalMask\n",
    "- `bal index type`: BAL index type, choose either 'ai' or 'bi'. This will set which velocity the  BAL mask uses. **Type: str, Required: no, Default: 'ai'**\n",
    "- `filename`: BAL catalog file. **Type: str, Required: yes**\n",
    "- `los_id name`: Name of the line of sight identifier (e.g. THING_ID in SDSS or TARGETID in DESI). **Type: str, Required: no, Default: THING_ID**\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Expected Flux section\n",
    "This section controls the extraction of deltas. It specifies the class that will be used to compute the quasar continua and the mean transmission. The main arguments are\n",
    "- `type`: Contains the name of the class used to extract the deltas. For example \"Dr16ExpectedFlux\". Basically can be any data type stored in the folder `py/picca/delta_extraction/expected_fluxes/`. **Type: str, Required: yes**\n",
    "- `module_name`: Name of the file containing the expected flux type. If the file is named following the convention this shoul normally not be necessary. **Type: str, Required: no**\n",
    "- `num processors`: Number of processors to be used to compute the mean continua and to read in data for DesiHealpix based modes. 0 will use half the number of available cores.  **Type: int, Required: no, Default: 0**\n",
    "\n",
    "Other arguments will be passed to the constructor of the selected class. Let's review the arguments required by the available ExpectedFlux types\n",
    "\n",
    "### Dr16ExpectedFlux\n",
    "- `iter out prefix`: Prefix of the iteration files. These file contain the statistical properties of deltas at a given iteration step. Intermediate files will add '_iteration{num}.fits.gz' to the prefix for intermediate steps and '.fits.gz' for the final results. The prefix should not contain any folders. **Type: str, Required: no**\n",
    "- `limit eta`: Limits on function eta. Format is minimum,maximum. **Type: list of float, Required: no: Default: 0.1,1.5**\n",
    "- `limit var lss`: Limits on eta function var_lss. Format is minimum,maximum. **Type: list of float, Required: no: Default: 0.0,0.3**\n",
    "- `num bins variance`: Number of bins to be used to compute variance functions and statistics as a function of wavelength. **Type: int, Required: no, Default: 20**\n",
    "- `num iterations`: Number of iterations to determine the mean continuum shape, LSS variances, etc. **Type: int, Required: no, Default: 5**\n",
    "- `order`: Order of the polynomial for the continuum fit. **Type: int, Required: no, Default: 1**\n",
    "- `use constant weight`: If \"True\", set all the delta weights to one (implemented as eta = 0, sigma_lss = 1, fudge = 0) **Type: bool, Required: no, Default: False**\n",
    "- `use ivar as weight`: If \"True\", use ivar as weights (implemented as eta = 1, sigma_lss = fudge = 0) **Type: bool, Required: no, Default: False**\n",
    "- `force stack delta to zero`: If \"True\" (Default), continuum is corrected by stack_delta. **Type: bool, Required: no, Default: True**\n",
    "\n",
    "### TrueContinuum\n",
    "- `input directory`: Directory to spectra files. **Type: str, Required: yes**\n",
    "- `iter out prefix`: Prefix of the iteration files. These file contain the statistical properties of deltas at a given iteration step. Intermediate files will add '_iteration{num}.fits.gz' to the prefix for intermediate steps and '.fits.gz' for the final results. For now there is only one iteration available, but this might change in the future. The prefix should not contain any folders. **Type: str, Required: no**\n",
    "- `var lss binning`: Binning of the var lss file to be used or var lss file. If binning is input, then it should be 'log', 'lin_2.4' or 'lin3.2'. Otherwise the full file name is required.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Examples\n",
    "To finish let us see a few examples"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "ename": "Exception",
     "evalue": "Under construction",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mException\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-4-e6293ed1ccc3>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0;32mraise\u001b[0m \u001b[0mException\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"Under construction\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;31mException\u001b[0m: Under construction"
     ]
    }
   ],
   "source": [
    "raise Exception(\"Under construction\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.12"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": true,
   "toc_position": {
    "height": "calc(100% - 180px)",
    "left": "10px",
    "top": "150px",
    "width": "288px"
   },
   "toc_section_display": true,
   "toc_window_display": true
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}