docs/development.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "development.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "toc_visible": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "C_OmLFXVkD7y"
      },
      "source": [
        "# Development\n",
        "READ THIS BEFORE YOU CONTRIBUTE CODE!!!  \n",
        "\n",
        "---\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wdIC8NbbkQlT"
      },
      "source": [
        "## Organization\n",
        "The CNGI Prototype is organized into *packages* and *modules* as described below. Base infrastructure, data manipulation and mathematics are contained in the CNGI package. Higher level data reduction functionality is contained in the ngCASA package. Within each package are a set of modules, with each module\n",
        "responsible for a different functional area.\n",
        "\n",
        "*CNGI package*\n",
        "\n",
        "- **conversion** : convert legacy CASA data files to CNGI compatible files\n",
        "- **dio**        : data objects to/from CNGI compatible data files\n",
        "- **vis**        : operations on visibility data objects\n",
        "- **image**      : operations on image data objects\n",
        "- **direct**     : access/initialize the underlying parallel processing framework\n",
        "\n",
        "*ngCASA package*\n",
        "\n",
        "- **flagging**         : Generates flags for visibility data.\n",
        "- **calibration**      : Generates and applies calibration solutions to visibility data.\n",
        "- **imaging**          : Converts visibility data to images and applies antenna primary beam and w-term corrections.\n",
        "- **deconvolution**    : Deconvolves PSF from images and combines images. \n",
        "\n",
        "\n",
        "The [cngi_prototype repository](https://github.com/casangi/cngi_prototype) on GitHub contains both packages along with supporting folders `docs` and `tests`.\n",
        "\n",
        "\n",
        "\n",
        "---\n",
        "\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "O9NjSInKkUbH"
      },
      "source": [
        "## Architecture\n",
        "The CNGI Prototype application programming interface (API) is a set of flat, stateless functions that take an xarray Dataset as an input parameter and return a new xarray Dataset (XDS) as output.  The term **“flat”** means that the functions are not allowed to call each other, and the term **“stateless”** means that they may not access any global data outside the parameter list, nor maintain any persistent internal data.\n",
        "\n",
        "The CNGI Prototype code base is not object oriented, and instead follows a more functional paradigm. Objects are indeed used to hold Visibility\n",
        "and Image data, but they come directly from the underlying xarray/dask framework and are not extended in any way.\n",
        "\n",
        "The data variables in the XDS are associated with task graphs. Compute is only triggered when the user explicitly calls compute on the XDS or instructs a function to save to disk. \n",
        "\n",
        "The CNGI Prototype adheres to a strict design philosophy with the following **RULES**:\n",
        "\n",
        "1.   Each file in a module must have exactly one function exposed to the external API (by docstring and \\_\\_init\\_\\_.py).\n",
        "2.   The exposed function name should match the file name.  \n",
        "3.   Must use stateless functions, not classes. \n",
        "4.   Files in a module cannot import each other.  \n",
        "5.   Files in separate modules cannot import each other.\n",
        "6.   Special _utility modules may exist for internal functions meant to be shared across modules/files. But each\n",
        "module file should be as self contained as possible.\n",
        "7.   A module's utility files may only be imported by that module's API files.\n",
        "8.   A package's utility files may only be imported by other modules in that package.\n",
        "9. No utility functions may be exposed to the external API. \n",
        "\n",
        "\n",
        "```sh\n",
        "cngi_prototype  \n",
        "|-- cngi\n",
        "|    |-- module1\n",
        "|    |     |-- __init__.py  \n",
        "|    |     |-- file1.py    \n",
        "|    |     |-- file2.py  \n",
        "|    |     | ...  \n",
        "|    |-- module2  \n",
        "|    |     |-- __init__.py\n",
        "|    |     |-- file3.py    \n",
        "|    |     |-- file4.py  \n",
        "|    |     | ...  \n",
        "|    |-- _utils\n",
        "|    |     |-- __init__.py\n",
        "|    |     |-- _file5.py    \n",
        "|    |     |-- _file6.py  \n",
        "|    |     | ...  \n",
        "|    | ...\n",
        "|-- ngcasa\n",
        "|    |-- module1\n",
        "|    |     |-- __init__.py  \n",
        "|    |     |-- file1.py    \n",
        "|    |     |-- file2.py  \n",
        "|    |     | ...  \n",
        "|    |     |-- _module1_utils\n",
        "|    |     |     |-- __init__.py\n",
        "|    |     |     |-- _check_module1_parms.py\n",
        "|    |     |     |-- _file5.py\n",
        "|    |     |     |-- _file6.py\n",
        "|    |     |     | ...\n",
        "|    |-- module2  \n",
        "|    |     |-- __init__.py\n",
        "|    |     |-- file3.py    \n",
        "|    |     |-- file4.py  \n",
        "|    |     | ...  \n",
        "|    |     |-- _module2_utils\n",
        "|    |     |     |-- __init__.py\n",
        "|    |     |     |-- _check_module2_parms.py\n",
        "|    |     |     |-- _file7.py\n",
        "|    |     |     |-- _file8.py\n",
        "|    |     |     | ...\n",
        "|    | ...\n",
        "|-- docs  \n",
        "|    | ...  \n",
        "|-- tests  \n",
        "|    | ...  \n",
        "|-- requirements.txt  \n",
        "|-- setup.py  \n",
        "```\n",
        "\n",
        "File1, file2, file3 and file4 **MUST** be documented in the API exactly as they appear. They must **NOT** import each other. _file5 and _file6 are utility files, they must **NOT** be documented in the API. They may be imported by file1-4 (cngi package) and file1-2 (ngcasa package).\n",
        "\n",
        "  \n",
        "There are several important files to be aware of:\n",
        "\n",
        "*   **\\_\\_init\\_\\_.py** : dictates what is seen by the API and importable by other functions\n",
        "*   **requirements.txt** : lists all library dependencies for development, used by IDE during setup\n",
        "*   **setup.py** : defines how to package the code for pip, including version number and library dependencies for installation\n",
        "*   **_check_module_parms.py** : Each module has a _check_module_parms.py file that has functions that check the input parameters of the module's API functions. The parameter defaults are also defined here. \n",
        "*   **_check_parms.py** : Provides the _check_parms and _check_storage_parms functions. The _check_parms is used by all the _check_module_parms.py files to check parameter data types, values and set defaults. The storage_parm is parameter that is common to all API functions is checked by _check_storage_parms.\n",
        "*   **_store.py** : Provides the _store function that stores datasets or appends data variables. All API functions use this function. \n",
        "\n",
        "\n",
        "\n",
        "---\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5IBu0lZcL5gF"
      },
      "source": [
        "### CNGI Function Template\n",
        "```python\n",
        "def myfunction(xds, param_1, ..., param_m):\n",
        "    \"\"\"\n",
        "    Description of function\n",
        "    \n",
        "    Parameters\n",
        "    ----------\n",
        "    xds : xarray.core.dataset.Dataset\n",
        "       input xarray dataset\n",
        "    param_1 : type\n",
        "       description of this parameter\n",
        "    ...\n",
        "    parms_n : type\n",
        "       description of this parameter\n",
        "    \n",
        "    Returns\n",
        "    -------\n",
        "    xarray.core.dataset.Dataset\n",
        "       new output xarray dataset\n",
        "    \"\"\"\n",
        "    \n",
        "    ### Import Statements\n",
        "    import numpy as np\n",
        "    ...\n",
        "    \n",
        "    ### Parameter Checking\n",
        "    assert(param_1 == what_it_shoud, \"ERROR: param_1 is wrong\")\n",
        "    ...\n",
        "\n",
        "    ### Function code\n",
        "    c = xds.a + xds.b\n",
        "    ...\n",
        "\n",
        "    ### Return a new xds, leaving input untouched\n",
        "    nxds = xds.assign({'mynewthing':c}\n",
        "\n",
        "    return nxds\n",
        "    \n",
        "```\n",
        "\n",
        "\n",
        "By default calling an CNGI function will not do computation, but rather build a graph of the computation ([example of a graph](https://ngcasa.readthedocs.io/en/latest/prototypes/cube_imaging_example.html)). Computation can be triggered by using ```dask.compute(dataset)```. \n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_VBdsYBCSoN7"
      },
      "source": [
        "### Data Structures\n",
        "\n",
        "Data is stored in Zarr files with Xarray formatting (which is specified in the metadata). The data is stored as N-dimensional arrays that can be chunked on any axis. The ngCASA/CNGI data formats are:\n",
        "\n",
        "- vis.zarr Visibility data (measurement set).\n",
        "- img.zarr Images (also used for convolution function cache).\n",
        "- cal.zarr Calibration tables.\n",
        "- tel.zar  Telescope layout (used for simulations)\n",
        "- Other formats will be added as needed. \n",
        "\n",
        "CNGI will provide functions to convert between the new Zarr formats and legacy formats (such as the measurement set, FITS files,  ASDM, etc.). The current implementations of the Zarr formats are not fully developed and are only sufficient for prototype development. \n",
        "\n",
        "Zarr data formats rules:\n",
        "\n",
        "- Data variable names are always uppercase and dimension coordinates are lowercase.\n",
        "- Data variable names are not fixed but defaults exist, for example the data variable that contains the uvw data default name is UVW. This flexibility allows for the easy inclusion of more advanced algorithms (for example multi-term deconvolution that produces Taylor term images).\n",
        "- Dimension coordinates and coordinates have fixed names.  \n",
        "- Dimension coordinates are not chunked.\n",
        "- Data variables and coordinates are chunked and should have consistent chunking with each other.\n",
        "- Any number of data variables can be in a dataset but must share a common coordinate and chunking system. \n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VIqjposqObVX"
      },
      "source": [
        "## Framework\n",
        "The abstraction layers of ngCASA and CNGI are unified by the paradigm of functional programming: \n",
        "\n",
        "Build a [directed acyclic graph](https://en.wikipedia.org/wiki/Directed_acyclic_graph#Data_processing_networks) (DAG) of [blocked algorithms](https://dl.acm.org/doi/10.1145/106975.106981) composed from functions (the edges) and data (the vertices) for [lazy evaluation](https://en.wikipedia.org/wiki/Lazy_evaluation) by a scheduler process coordinating a network of (optionally distributed and heterogeneous) machine resources.\n",
        "\n",
        "This approach relies on three compatible packages to form the core of the framework:\n",
        "\n",
        "- [Dask's specificiation](https://docs.dask.org/en/latest/spec.html) of the DAG coordinates processing and network resources\n",
        "- [Zarr's specification](https://zarr.readthedocs.io/en/stable/spec/v2.html) of blocked data structures coordinates input/output and serialization resources\n",
        "- [Xarray's specficiation](http://xarray.pydata.org/en/stable/data-structures.html) of multidimensional indexed arrays serves as the in-memory representation and interface to the other components of the framework\n",
        "\n",
        "The relationship between these libraries can be conceptualized by the following diagram:\n",
        "\n",
        "![singlemachinearchitecture](https://raw.githubusercontent.com/casangi/cngi_prototype/master/docs/_media/ngCASA_design.png)\n",
        "\n",
        "In the framework data is stored in the Zarr format in a file mounted on a local disk or in the cloud. The Dask scheduler manages *N* worker processes identified to the central scheduler by their unique network address and coordinating *M* threads. Each thread applies functions to a set of data chunks. Xarray wraps Dask functions and labels the Dask arrays for ease of interface. Any code that is wrapped with Python can be parallelized with Dask. Therefore, the option to use C++, Numba or other custom high performance computing (HPC) code is retained. The size of the Zarr data chunks (on disk) and Dask data chunks do not have to be the same, this is further explained in the chunking section. Data chunks can either be read from disk as they are needed for computation or the data can be persisted into memory. \n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "sLdufmBhOf5_"
      },
      "source": [
        "### Dask and Dask Distributed\n",
        "\n",
        "[Dask](https://dask.org/) is a flexible library for parallel computing in Python and [Dask Distributed](https://distributed.dask.org/en/latest/)  provides a centrally managed, distributed, dynamic task scheduler. A Dask task graph describes how tasks will be executed in parallel. The nodes in a task graph are made of Dask collections. Different Dask collections can be used in the same task graph. For the CNGI and ngCASA projects Dask `array` and Dask `delayed` collections will predominantly be used. Explanations from the Dask website about Dask arrays and Dask delayed:\n",
        "\n",
        "> [Dask array](https://docs.dask.org/en/latest/array.html) implements a subset of the NumPy ndarray interface using blocked algorithms, cutting up the large array into many small arrays. This lets us compute on arrays larger than memory using all of our cores. We coordinate these blocked algorithms using Dask graphs.\n",
        "\n",
        "> Sometimes problems don’t fit into one of the collections like dask.array or dask.dataframe. In these cases, users can parallelize custom algorithms using the simpler [dask.delayed](https://docs.dask.org/en/latest/delayed.html) interface. This allows one to create graphs directly with a light annotation of normal python code.\n",
        "\n",
        "The following diagram taken from the Dask website illustrates the components of the chosen parallelism framework.\n",
        "\n",
        "![title2](https://docs.dask.org/en/latest/_images/dask-overview.svg)\n",
        "\n",
        "Dask/Dask Distributed Advantages:\n",
        "\n",
        "- Parallelism can be achieved over any dimension, since it is determined by the data chunking. \n",
        "- Data can either be persisted into memory or read from disk as needed. As processing is finished chunks can be saved. This enables the processing of data that is larger than memory.  \n",
        "- Graphs can easily be combined with `dask.compute()` to concurrently execute multiple functions in parallel. For example a cube and continuum image can be created in parallel.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UTUBzACYSFoe"
      },
      "source": [
        "### Xarray\n",
        "[Xarray](http://xarray.pydata.org/en/stable/) provides N-Dimensional labeled arrays and datasets in Python. The Xarray dataset is used to organize and label related arrays. The Xarray website gives the following definition of a dataset:\n",
        "\n",
        "- \"[xarray.Dataset](http://xarray.pydata.org/en/stable/data-structures.html#dataset) is Xarray’s multi-dimensional equivalent of a DataFrame. It is a dict-like container of labeled arrays (DataArray objects) with aligned dimensions.\"\n",
        "\n",
        "The Zarr disk format and Xarray dataset specification are compatible if the Xarray ```to_zarr``` and ```open_zarr``` functions are used. The compatibility is achieved by requiring the Zarr group to have labeling information in the metadata files and a depth of one (see the Zarr section for further explanation).\n",
        "\n",
        "When ```xarray.open_zarr(zarr_group_file)``` is used the array data is not loaded to memory (there is a parameter to force this), rather the metadata is loaded and a lazy loaded Xarray dataset is created. For example a dataset that consists of three dimension coordinates (dim_), two coordinates (coord_) and three data variables (Data_) would have the following structure (this is what is displayed if a print command is used on a lazy loaded dataset):\n",
        "```\n",
        "<xarray.Dataset>\n",
        "Dimensions:        (dim_1: d1, dim_2: d2, dim_3: d3)\n",
        "Coordinates:\n",
        "    coord_1      (dim_1) data_type_coord_1 dask.array<chunksize=(chunk_d1,), meta=np.ndarray>\n",
        "    coord_2      (dim_1, dim_2)  data_type_coord_2 dask.array<chunksize=(chunk_d1,chunk_d2), meta=np.ndarray>\n",
        "  * dim_1        (dim_1) data_type_dim_1 np.array([x_1, x_2, ..., x_d1])\n",
        "  * dim_2        (dim_2) data_type_dim_2 np.array([y_1, y_2, ..., y_d2])\n",
        "  * dim_3        (dim_3) data_type_dim_3 np.array([z_1, z_2, ..., z_d3])\n",
        "  \n",
        "Data variables:\n",
        "    DATA_1       (dim2) data_type_DATA_1 dask.array<chunksize=(chunk_d2,), meta=np.ndarray>\n",
        "    DATA_2       (dim1) data_type_DATA_2 dask.array<chunksize=(chunk_d1,), meta=np.ndarray>\n",
        "    DATA_3       (dim1,dim2,dim3) data_type_DATA_3 dask.array<chunksize=(chunk_d1,chunk_d2,chunk_d3), meta=np.ndarray>\n",
        "    \n",
        "Attributes:\n",
        "    attr_1:        a1\n",
        "    attr_2:        a2\n",
        "```\n",
        "\n",
        "- d1, d2, d3 are integers.\n",
        "- a1, a2 can be any data type that can be stored in a JSON file.\n",
        "- data_type_dim_, data_type_coord_ , data_type_DATA_ can be any acceptable NumPy array data type.\n",
        "\n",
        "Explanations of dimension coordinates, coordinates and data variables from the [Xarray website](http://xarray.pydata.org/en/stable/data-structures.html#coordinates):\n",
        "\n",
        "- dim_ \"Dimension coordinates are one dimensional coordinates with a name equal to their sole dimension (marked by * when printing a dataset or data array). They are used for label based indexing and alignment, like the index found on a pandas DataFrame or Series. Indeed, these dimension coordinates use a pandas. Index internally to store their values.\"\n",
        "\n",
        "- coord_ \"Coordinates (non-dimension) are variables that contain coordinate data, but are not a dimension coordinate. They can be multidimensional (see Working with Multidimensional Coordinates), and there is no relationship between the name of a non-dimension coordinate and the name(s) of its dimension(s). Non-dimension coordinates can be useful for indexing or plotting; otherwise, xarray does not make any direct use of the values associated with them. They are not used for alignment or automatic indexing, nor are they required to match when doing arithmetic.\"\n",
        "\n",
        "- DATA_ The array that dimension coordinates and coordinates label.\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jgx8hnTbPAhB"
      },
      "source": [
        "### Zarr\n",
        "\n",
        "Data is stored in Zarr files with Xarray formatting (which is specified in the metadata). The data is stored as N-dimensional arrays that can be chunked on any axis. The ngCASA/CNGI data formats are:\n",
        "\n",
        "- vis.zarr Visibility data (measurement set).\n",
        "- img.zarr Images (also used for convolution function cache).\n",
        "- cal.zarr Calibration tables.\n",
        "- tel.zar  Telescope layout (used for simulations)\n",
        "- Other formats will be added as needed. \n",
        "\n",
        "CNGI will provide functions to convert between the new Zarr formats and legacy formats (such as the measurement set, FITS files,  ASDM, etc.). The current implementations of the Zarr formats are not fully developed and are only sufficient for prototype development. \n",
        "\n",
        "Zarr data formats rules:\n",
        "\n",
        "- Data variable names are always uppercase and dimension coordinates are lowercase.\n",
        "- Data variable names are not fixed but defaults exist, for example the data variable that contains the uvw data default name is UVW. This flexibility allows for the easy inclusion of more advanced algorithms (for example multi-term deconvolution that produces Taylor term images).\n",
        "- Dimension coordinates and coordinates have fixed names.  \n",
        "- Dimension coordinates are not chunked.\n",
        "- Data variables and coordinates are chunked and should have consistent chunking with each other.\n",
        "- Any number of data variables can be in a dataset but must share a common coordinate and chunking system. \n",
        "\n",
        "\n",
        "[Zarr](https://zarr.readthedocs.io/en/stable/) is the chosen data storage library for the CNGI Prototype. It provides an implementation of chunked, compressed, N-dimensional arrays that can be stored on disk, in memory, in cloud-based object storage services such as [Amazon S3](https://aws.amazon.com/s3/), or [any other collection](https://zarr.readthedocs.io/en/stable/tutorial.html#storage-alternatives) that supports the [MutableMapping](https://docs.python.org/3/library/collections.abc.html#collections.abc.MutableMapping) interface. \n",
        "\n",
        "Compressed arrays can be hierarchically organized into labeled groups. For the CNGI Prototype, the Zarr group hierarchy is structured to be compatible with the [Xarray](http://xarray.pydata.org/en/stable/io.html#zarr) dataset convention (a Zarr group contains all the data for an Xarray dataset):\n",
        "\n",
        "```\n",
        "My/Zarr/Group \n",
        "    |-- .zattrs\n",
        "    |-- .zgroup\n",
        "    |-- .zmetadata\n",
        "\n",
        "    |-- Array_1\n",
        "    |    |-- .zattrs\n",
        "    |    |-- .zarray\n",
        "    |    |-- 0.0.0. ... 0\n",
        "    |    |-- ... \n",
        "    |    |-- C_1.C_2.C_3. ... C_D \n",
        "    |-- ... \n",
        "    |-- Array_N_c\n",
        "    |    |-- ...     \n",
        "```\n",
        "\n",
        "The group is accessed by a logical storage path (in this case, the string `My/Zarr/Group`) and is self-describing; it contains metadata in the form of hidden files (`.zattrs`, `.zgroup` and `.zmetadata`\n",
        ") and a collection of folders. Each folder contains the data for a single array along with two hidden metadata files (`.zattrs`, `.zarray`). The metadata files are used to create the lazily loaded representation of the dataset (only metadata is loaded). The data of each array is chunked and stored in the format `x_1,x_2,x_3, ..., x_D` where D is the number of dimensions and x_i is the chunk index for the ith dimension. For example a three dimensional array with two chunks in the first dimension and three chunks in the second dimension would consist of the following files `0.0.0, 1.0.0, 0.1.0, 1.1.0, 0.2.0, 1.2.0`.\n",
        "\n",
        "Group folder metadata files (encoded using JSON):\n",
        "\n",
        "- ```.zgroup``` contains an integer defining the version of the storage [specification](https://zarr.readthedocs.io/en/stable/spec/v2.html). For example:\n",
        "```\n",
        "{\n",
        "   \"zarr_format\": 2\n",
        "}\n",
        "```\n",
        "- ```.zattrs``` describes data attributes that can not be stored in an array (this file can be empty). For example:\n",
        "```\n",
        "{\n",
        "    \"append_zarr_time\": 1.8485729694366455,\n",
        "    \"auto_correlations\": 0,\n",
        "    \"ddi\": 0,\n",
        "    \"freq_group\": 0,\n",
        "    \"ref_frequency\": 372520022603.63745,\n",
        "    \"total_bandwidth\": 234366781.0546875\n",
        "}\n",
        "```      \n",
        "- `.zmetadata` contains all the metadata from all other metadata files (both in the group directory and array subdirectories). This file does not have to exist, however it can decrease the time it takes to create the lazy loaded representation of the dataset, since each metadata file does not have to be opened and read separately. If any of the files are changed or files are added the `.zmetadata` file must be updated with `zarr.consolidate_metadata(group_folder_name)`.\n",
        "\n",
        "Array folder metadata files (encoded using JSON):\n",
        "\n",
        "- `.zarray` describes the data: how it is chunked, the compression used and array properties. For example the `.zarray` file for a DATA array (contains the visibility data) would contain:\n",
        "```\n",
        "{\n",
        "    \"chunks\": [270,210,12,1],\n",
        "    \"compressor\": {\"blocksize\": 0,\"clevel\": 2,\"cname\": \"zstd\",\"id\": \"blosc\",\"shuffle\": 0},\n",
        "    \"dtype\": \"<c16\",\n",
        "    \"fill_value\": null,\n",
        "    \"filters\": null,\n",
        "    \"order\": \"C\",\n",
        "    \"shape\": [270,210,384,1],\n",
        "    \"zarr_format\": 2\n",
        "}\n",
        "```\n",
        "Zarr supports all the compression algorithms implemented in [numcodecs](https://numcodecs.readthedocs.io/en/stable/blosc.html) (‘zstd’, ‘blosclz’, ‘lz4’, ‘lz4hc’, ‘zlib’, ‘snappy’).\n",
        "- `.zattrs` is used to label the arrays so that an Xarray dataset can be created. The labeling creates three types of arrays:\n",
        "    - Dimension coordinates are one dimensional arrays that are used for label based indexing and alignment of data variable arrays. The array name is the same as its sole dimension. For example the `.zattrs` file in the \"chan\" array would contain:\n",
        "    ```\n",
        "    {\n",
        "    \"_ARRAY_DIMENSIONS\": [\"chan\"]\n",
        "    }\n",
        "    ```\n",
        "    - Coordinates can have any number of dimensions and are a function of dimension coordinates. For example the \"declination\" coordinate is a function of the d0 and d1 dimension coordinates and its `.zattrs` file contains:\n",
        "    ```\n",
        "    {\n",
        "    \"_ARRAY_DIMENSIONS\": [\"d0\",\"d1\"]\n",
        "    }\n",
        "    ```\n",
        "    - Data variables contain the data that dimension coordinates and coordinates label. For example the DATA data variable's `.zattrs` file contains:\n",
        "    ```\n",
        "    {\n",
        "    \"_ARRAY_DIMENSIONS\": [\"time\",\"baseline\",\"chan\",\"pol\"],\n",
        "    \"coordinates\": \"interval scan field state processor observation\"\n",
        "    }\n",
        "    ```\n",
        "\n",
        "Zarr Advantages:\n",
        "\n",
        "- Designed for concurrency, compatible out-of-the-box with chosen parallelism framework (Dask).\n",
        "- Wide variety of compression algorithms supported, see [numcodecs](https://numcodecs.readthedocs.io/en/stable/blosc.html). Each data variable can be compressed using a different compression algorithm. \n",
        "- Supports chunking along any dimension.\n",
        "- Has a defined cloud interface.\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9XxUrDWNPFQf"
      },
      "source": [
        "### Chunking\n",
        "\n",
        "In the zarr and dask model, data is broken up into chunks to improve efficiency by reducing communication and increasing data locality. The zarr chunk size is specified at creation or when converting from CASA6 format datasets.  The Dask chunk size can be specified in the ```cngi.dio.read_vis``` and ```cngi.dio.read_image``` calls using the ```chunks``` parameter. The dask and zarr chunking do not have to be the same. The [zarr chunking](https://zarr.readthedocs.io/en/stable/tutorial.html) is what is used on disk and the [dask chunking ](https://docs.dask.org/en/latest/array-chunks.html) is used during parallel computation. However, it is more efficient for the dask chunk size to be equal to or a multiple of the zarr chunk size (to stop multiple reads of the same data). This hierarchy of chunking allows for flexible and efficient algorithm development. For example cube imaging is more memory efficient if chunking is along the channel axis (the [benchmarking](https://ngcasa.readthedocs.io/en/latest/benchmark.html) example demonstrates this). Note, chunking can be done in any combination of dimensions.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NChLqyttPIZT"
      },
      "source": [
        "### Numba\n",
        "\n",
        "[Numba](http://numba.pydata.org/) is an open source JIT (Just In Time) compiler that uses [LLVM](https://llvm.org/)to translate a subset of Python and NumPy code into fast machine code from a chain of intermediate representations. Numba is used in ngCASA for functions that have long nested for loops (for example the gridder code). Numba can be used by adding the @jit decorator above a function:\n",
        "\n",
        "```python\n",
        "@jit(nopython=True, cache=True, nogil=True)\n",
        "def my_func(input_parms):\n",
        "    does something ...\n",
        "```\n",
        "    \n",
        "Explanation of jit arguments from the [Numba](https://numba.pydata.org/numba-doc/latest/user/jit.html):\n",
        "\n",
        "`nopython`:\n",
        "> The behaviour of the nopython compilation mode is to essentially compile the decorated function so that it will run entirely without the involvement of the Python interpreter. This is the recommended and best-practice way to use the Numba jit decorator as it leads to the best performance.\n",
        "\n",
        "`cache`:\n",
        "> To avoid compilation times each time you invoke a Python program, you can instruct Numba to write the result of function compilation into a file-based cache.\n",
        "\n",
        "`nogil`:\n",
        "> Whenever Numba optimizes Python code to native code that only works on native types and variables (rather than Python objects), it is not necessary anymore to hold Python’s global interpreter lock (GIL)\"\n",
        "    \n",
        "A 5 minute guide to starting with Numba can be found [here](http://numba.pydata.org/numba-doc/latest/user/5minguide.html). \n",
        "\n",
        "Numba also has functionality to run code on [CUDA](http://numba.pydata.org/numba-doc/latest/cuda/index.html) and [AMD ROC](http://numba.pydata.org/numba-doc/latest/roc/index.html)) GPUs. This will be explored in the future."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1elJPyu9P1Te"
      },
      "source": [
        "### Parallel Code with Dask\n",
        "\n",
        "Code can be parallelized in three different ways: \n",
        "\n",
        "- Built in Dask array functions. The list of dask.array functions can be found [here](https://docs.dask.org/en/latest/array-api.html). For example the fast Fourier transform is a built in parallel function:\n",
        "\n",
        "```python\n",
        "uncorrected_dirty_image = dafft.fftshift(dafft.ifft2(dafft.ifftshift(grids_and_sum_weights[0], axes=(0, 1)), \n",
        "                                                     axes=(0,1)), axes=(0, 1))\n",
        "```\n",
        "\n",
        "- Apply a custom function to each Dask data chunk. There are numerous Dask functions, with varying capabilities, that do this: [map_blocks](https://docs.dask.org/en/latest/array-api.html#dask.array.map_blocks), [map_overlap](https://docs.dask.org/en/latest/array-overlap.html#dask.array.map_overlap), [apply_gufunc](https://docs.dask.org/en/latest/array-api.html#dask.array.gufunc.apply_gufunc), [blockwise](https://docs.dask.org/en/latest/array-api.html#dask.array.blockwise). For example the `dask.map_block` function is used to divide each image in a channel with the gridding convolutional kernel:\n",
        "\n",
        "```python\n",
        "def correct_image(uncorrected_dirty_image, sum_weights, correcting_cgk):\n",
        "    sum_weights[sum_weights == 0] = 1\n",
        "    corrected_image = (uncorrected_dirty_image / sum_weights[None, None, :, :]) \n",
        "        / correcting_cgk[:, :, None, None]\n",
        "    return corrected_image\n",
        "\n",
        "corrected_dirty_image = dask.map_blocks(correct_image, uncorrected_dirty_image,                                 \n",
        "                                      grids_and_sum_weights[1],correcting_cgk_image)\n",
        "```\n",
        "\n",
        "- Custom parallel functions can be built using [dask.delayed](https://docs.dask.org/en/latest/delayed.html) objects. Any function or object can be delayed. For example the gridder is implemented using dask.delayed:\n",
        "\n",
        "```python\n",
        "for c_time, c_baseline, c_chan, c_pol in iter_chunks_indx:\n",
        "    sub_grid_and_sum_weights = dask.delayed(_standard_grid_numpy_wrap)(\n",
        "    vis_dataset[grid_parms[\"data_name\"]].data.partitions[c_time, c_baseline, c_chan, c_pol],\n",
        "    vis_dataset[grid_parms[\"uvw_name\"]].data.partitions[c_time, c_baseline, 0],\n",
        "    vis_dataset[grid_parms[\"imaging_weight_name\"]].data.partitions[c_time, c_baseline, c_chan, c_pol],\n",
        "    freq_chan.partitions[c_chan],\n",
        "    dask.delayed(cgk_1D), dask.delayed(grid_parms))\n",
        "    grid_dtype = np.complex128\n",
        "```"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-gDBNsJPpMDE"
      },
      "source": [
        "## Documentation\n",
        "\n",
        "All CNGI documentation is automatically rendered from files placed in the **docs** folder using the Sphinx tool.  A [Readthedocs](https://readthedocs.org/) service scans for updates to the Github repository and automatically calls Sphinx to build new documentation as necessary.  The resulting documentation html is hosted by readthedocs as a CNGI website.\n",
        "\n",
        "![sphinx](https://www.sphinx-doc.org/en/master/_static/sphinxheader.png)\n",
        "\n",
        "Compatible file types in the docs folder that can be rendered by Sphinx include:\n",
        "\n",
        "*   Markdown (.md)\n",
        "*   reStructuredText (.rst)\n",
        "*   Jupyter notebook (.ipynb)\n",
        "\n",
        "Sphinx extension modules are used to automatically crawl the cngi code directories and pull out function definitions. These definitions end up in the API section of the documentation. All CNGI functions must conform to the [numpy docstring format](https://numpydoc.readthedocs.io/en/latest/format.html).\n",
        "\n",
        "The [nbsphinx](https://nbsphinx.readthedocs.io/en/0.6.0/index.html) extension module is used to render Jupyter notebooks to html. \n",
        "\n",
        "\n",
        "\n",
        "---\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GrIBHQMwnT55"
      },
      "source": [
        "## IDE\n",
        "The CNGI team recommends the use of the [PyCharm](https://www.jetbrains.com/pycharm/) IDE for developing CNGI code.  PyCharm provides a simple (relatively) unified environment that includes Github integration, code editor, python shell, system terminal, and venv setup.\n",
        "\n",
        "[![PyCharm](http://img.youtube.com/vi/BPC-bGdBSM8/0.jpg)](http://www.youtube.com/watch?feature=player_embedded&v=BPC-bGdBSM8)\n",
        "\n",
        "CNGI also relies heavily on [Google Colaboratory](https://colab.research.google.com/) for both documentation and code execution examples.  Google colab notebooks integrate with Github and allow markdown-style documentation interleaved with executable python code.  Even in cases where no code is necessary, colab notebooks are the preferred choice for markdown documentation.  This allows other team members to make documentation updates in a simple, direct manner.\n",
        "\n",
        "[![Colab](http://img.youtube.com/vi/inN8seMm7UI/0.jpg)](http://www.youtube.com/watch?feature=player_embedded&v=inN8seMm7UI)\n",
        "\n",
        "\n",
        "\n",
        "---\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jOHGpxP5uyik"
      },
      "source": [
        "## PyPi Packages\n",
        "\n",
        "CNGI is distributed and installed via pip by hosting packages on pypi.  The [pypi test server](https://test.pypi.org/) is available to all authorized CNGI developers to upload an evaluate their code branches.\n",
        "\n",
        "\n",
        "![Pypi](https://pypi.org/static/images/logo-large.72ad8bf1.svg)\n",
        "\n",
        "Typically, the Colab notebook documentation and examples will need a pip installation of CNGI to draw upon. The pypi test server allows notebook documentation to temporarily draw from development branches until everything is finalized in a Github pull request and production pypi distribution.\n",
        "\n",
        "Developers should create a .pypirc file in their home directory for convenient uploading of distributions to the pip test server.  It should look something like:  \n",
        "\n",
        "```\n",
        "[distutils]\n",
        "index-servers = \n",
        "     pypi\n",
        "     pypitest\n",
        "\n",
        "[pypi]\n",
        "username = yourusername\n",
        "password = yourpassword\n",
        "\n",
        "[pypitest]\n",
        "repository = https://test.pypi.org/legacy/\n",
        "username = yourusername\n",
        "password = yourpassword\n",
        "```\n",
        "\n",
        "Production packages are uploaded to the main pypi server by a subset of authorized CNGI developers when a particular version is ready for distribution.\n",
        "\n",
        "\n",
        "\n",
        "---\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FM4EYwDlsvDu"
      },
      "source": [
        "## Step by Step\n",
        "Concise steps for contributing code to CNGI\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7BMXO7Rs97gc"
      },
      "source": [
        "### Install IDE\n",
        "1.   Request that your Github account be added to the contributors of the CNGI repository\n",
        "2.   Make sure Python 3.6 and Git are installed on your machine\n",
        "3.   Download and install the free [PyCharm Community edition](https://www.jetbrains.com/pycharm/download). On Linux, it is just a tar file. Expand it and execute pycharm.sh in the bin folder via something like:\n",
        "```sh\n",
        "$ ./pycharm-community-2020.1/bin/pycharm.sh\n",
        "``` \n",
        "4.   From the welcome screen, click `Get from Version Control`\n",
        "5.   Add your Github account credentials to PyCharm and then you should see a list of all repositories you have access to\n",
        "6.   Select the CNGI repository and set an appropriate folder location/name.  Click \"Clone\". \n",
        "7.   Go to: \n",
        "```\n",
        "File -> Settings -> Project: xyz -> Python Intrepreter\n",
        "``` \n",
        "and click the little cog to add a new Project Interpreter. Make a new **Virtualenv** environment, with the location set to a venv subfolder in the project directory.  Make sure to use Python 3.6.\n",
        "8.   Double click the `requirements.txt` file that was part of the git clone to open it in the editor.  That should prompt PyCharm to ask you if you want to \"Install requirements\" found in this file.  Yes, you do.  You can ignore the stuff about plugins.\n",
        "9.  All necessary supporting Python libraries will now be installed in to the venv created for this project (isolating them from your base system).  Do NOT add any project settings to Git.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-ranmT51-NWK"
      },
      "source": [
        "### Develop stuff\n",
        "1.   Double click on files to open in editor and make changes.\n",
        "2.   Create new files with: \n",
        "```\n",
        "right-click -> New\n",
        "```\n",
        "3.   Move / rename / delete files with: \n",
        "```\n",
        "right-click -> Refactor\n",
        "```\n",
        "4.   Run code interactively by selecting \"Python Console\" from the bottom of the screen. This is your venv enviornment with everything from requirements.txt installed in addition to the cngi package.  You can do things like this:\n",
        "```python\n",
        ">>> from cngi.dio import read_vis  \n",
        ">>> xds = read_vis('path\\to\\data.vis.zarr')  \n",
        "```\n",
        "5.   When you make changes to a module (lets say read_vis for example), close the Python Console and re-open it, then import the module again to see the changes.\n",
        "6.   Commit changes to your local branch with \n",
        "```\n",
        "right-click -> Git -> Commit File\n",
        "```\n",
        "7.   Merge latest version of Github master trunk to your local branch with\n",
        "```\n",
        "right-click -> Git -> Repository -> Pull\n",
        "```\n",
        "8.   Push your local branch up to the Github master trunk with \n",
        "```\n",
        "right-click -> Git -> Repository -> Push\n",
        "```"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dU5c_ir8NcIg"
      },
      "source": [
        "### Make a Pip Package\n",
        "1.   If not already done, create an account on pip (and the test server) and have a CNGI team member grant access to the package.  Then create a `.pypirc` file in your home directory.\n",
        "2.   Set a unique version number in `setup.py` by using the release candidate label, as in: \n",
        "```python \n",
        "version='0.0.48rc1' \n",
        "```\n",
        "3.   Build the source distribution by executing the following commands in the PyCharm Terminal (button at the bottom left):\n",
        "```sh\n",
        "$ rm -fr dist\n",
        "$ python setup.py sdist\n",
        "```\n",
        "4.   call twine to upload the sdist package to pypi-test:\n",
        "```python\n",
        "$ python -m twine upload dist/* -r pypitest\n",
        "```\n",
        "5.   Enjoy your pip package as you would a real production one by pointing to the test server:\n",
        "```sh\n",
        "$ pip install --extra-index-url https://test.pypi.org/simple/ cngi-prototype==0.0.48rc1\n",
        "```"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "P3JF7f1TJZac"
      },
      "source": [
        "### Update the Documentation\n",
        "1.   A bulk of the documentation is in the `docs` folder and in the '.ipynb' format. These files are visible through PyCharm, but should be edited and saved in **Google Colab**. The easiest way to do this is not navigate to the Github [docs folder](https://github.com/casangi/cngi_prototype/tree/master/docs) and click on the .ipynb file you want to edit. There is usually an `open in colab` link at the top.  \n",
        "2.   Alternatively, notebooks can be accessed in Colab by combining a link prefix with the name of the .ipynb file in the repository `docs` folder. For example, this page you are reading now can be edited by combining the colab prefix:\n",
        "```\n",
        "https://colab.research.google.com/github/casangi/cngi_prototype/blob/master/docs/\n",
        "``` \n",
        "with the filename of this notebook:\n",
        "```\n",
        "development.ipynb\n",
        "```\n",
        "producing a link of: https://colab.research.google.com/github/casangi/cngi_prototype/blob/master/docs/development.ipynb\n",
        "  \n",
        "3.   In Colab, make the desired changes and then select \n",
        "```\n",
        "File -> Save a copy in Github\n",
        "```\n",
        "enter you Github credentials if not already stored with Google, and then select the CNGI repository and the appropriate path/filename, i.e. `docs/development.ipynb`\n",
        "4.   Readthedocs will detect changes to the Github master and automatically rebuild the documentation hosted on their server (this page you are reading now, for example).  This can take ~15 minutes\n",
        "\n",
        "\n",
        "In the docs folder, some of the root index files are stored as .md or .rst format and may be edited by double clicking and modifying in the PyCharm editor.  They can then be pushed to the master trunk in the same manner as source code.\n",
        "\n",
        "After modifying an .md or .rst file, double check that it renders correctly by executing the following commands in the PyCharm Terminal\n",
        "```sh\n",
        "$ cd docs/\n",
        "$ rm -fr _api/api\n",
        "$ rm -fr build\n",
        "$ sphinx-build -b html . ./build\n",
        "```\n",
        "Then open up a web browser and navigate to \n",
        "```\n",
        "file:///path/to/project/docs/build/index.html\n",
        "```\n",
        "Do **NOT** add api or build folders to Git, they are intermediate build artifacts.  Note that **_api** is the location of actual documentation files that automatically parse the docstrings in the sourcecode, so that *should* be in Git.  \n",
        "\n",
        "\n",
        "\n",
        "---\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6YdUrtSskcke"
      },
      "source": [
        "## Coding Standards\n",
        "\n",
        "Documentation is generated using Sphinx, with the autodoc and napoleon extensions enabled. Function docstrings should be written in [NumPy style](https://www.sphinx-doc.org/en/master/usage/extensions/napoleon.html#google-vs-numpy). For compatibility with Sphinx, import statements should generally be underneath function definitions, not at the top of the file.\n",
        "\n",
        "A complete set of formal and enforced coding standards have not yet been formally adopted. Some alternatives under consideration are:\n",
        "\n",
        "* Google's [style guide](https://google.github.io/styleguide/pyguide.html)\n",
        "* Python Software Foundation's [style guide](https://www.python.org/dev/peps/pep-008/)\n",
        "* Following conventions established by PyData projects (examples [one](https://docs.dask.org/en/latest/develop/html) and [two](https://xarray.pydata.org/en/stable/contributing.html#code-standards))\n",
        "\n",
        "We are evaluating the adoption of [PEP 484](https://www.python.org/dev/peps/pep-0484/) convention, [mypy](http://mypy-lang.org/), or  [param](https://param.holoviz.org) for type-checking, and [flake8](http://flake8.pycqa.org/en/latest/index.html) or [pylint](https://www.pylint.org/) for enforcement.\n",
        "\n"
      ]
    }
  ]
}