docs/notebooks/cal_library_syntax.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "cal_library_syntax.ipynb",
      "provenance": [],
      "toc_visible": true
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "U3COx9yb0EDx"
      },
      "source": [
        "# Cal Library Syntax \n",
        "\n",
        "<div class=\"alert alert-warning\">\n",
        "**WARNING:** There are [Known Issues](introduction.ipynb#Known-Issues) regarding the Cal Library.\n",
        "</div>\n",
        "\n",
        "The \\\"Cal Library\\\" is a new means of expressing calibration application instructions.  It has nominally been available in **applycal** and the calibration solve tasks since CASA 4.1, via the *docallib=True* parameter, as an alternative to the traditional parameters (e.g., *gaintable*, etc.)  that most users continue to use.  As of CASA 4.5, we have deployed use of the Cal Library for *on-the-fly* calibration in **plotms** and **mstransform**.  In CASA 4.5, our intent is to demonstrate the Cal Library and begin familiarizing users with it.  The capabilities remain limited in some ways, and new features, additional flexibility, and broader deployment in more tasks will be offered in later releases.This page describes basic use of the Cal Library.\n",
        "\n",
        "<div class=\"alert alert-warning\">\n",
        "Please note the section on current (CASA 4.5, 4.6, 4.7, 5.*) limitations.\n",
        "</div>\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9dg99o8r0EDz"
      },
      "source": [
        "## Basic Cal Library usage\n",
        "\n",
        "The Cal Library is a means of specifying calibration instructions in an ascii file, rather than via the traditional *gaintable/gainfield/interp/spwmap/calwt* parameters that often become clumsy when many caltables are involved, and which have rather limited flexibility.  Instead of specifying the traditional parameters, the file name is specified in the *callib* parameter in **applycal** or **plotms** (in **applycal** one must also specifiy *docallib=True*).  For example, to correct an MS called *my.ms*, with a Cal Library file called *mycal.txt*:\n",
        "\n",
        "```\n",
        "applycal(vis='my.ms',docallib=True,callib='mycal.txt')\n",
        "```\n",
        "\n",
        "In a Cal Library file, each row expresses the calibration apply instructions for a particular caltable and (optionally) a specific selection of data in the MS to which it is to be applied.For example, if *mycal.txt* contains:\n",
        "\n",
        "```\n",
        "#mycal.txt cal library file\n",
        "caltable='cal.G' tinterp='linear' calwt=True\n",
        "```\n",
        "\n",
        "this will arrange a caltable called *cal.G* to be applied (with no detailed selection) to all MS data with linear interpolation in time, and with the weights also calibrated.  It corresponds to these settings for the traditional parameters in **applycal**:\n",
        "\n",
        "```\n",
        "applycal(vis='my.ms',gaintable='cal.G',gainfield='',interp='linear',\n",
        "         spwmap=[],calwt=True)\n",
        "```\n",
        "\n",
        "If a bandpass table, *cal.B*, is also available for application, one might use the following Cal Library file:\n",
        "\n",
        "```\n",
        "#mycal.txt cal library file\n",
        "caltable='cal.G' tinterp='linear' calwt=True\n",
        "caltable='cal.B' finterp='linear' calwt=False\n",
        "```\n",
        "\n",
        "This example arranges the same instructions for *cal.G*, and adds a bandpass table that will be interpolated linearly in frequency (the default for time-dependent interpolation is linear, if the bandpass table contains more than one time sample), without weight calibration.  The corresponding form with the traditional parameters is:\n",
        "\n",
        "```\n",
        "applycal(vis='my.ms',gaintable=['cal.G','cal.B'], gainfield=['',''],\n",
        "         interp=['linear','linear,linear'],\n",
        "         spwmap=[],calwt=[True,False])\n",
        "```\n",
        "\n",
        "In general, the Cal LIbrary file should be easier to read and manage than the traditional parameters as the number of specified caltables grows.A more complicated example, involving non-trivial *spwmap* as well as field selection (*fldmap*) in the caltable:\n",
        "\n",
        "```\n",
        "#mycal.txt cal library file\n",
        "caltable='cal.G' tinterp='linear' fldmap='nearest' spwmap=[0,1,1,3] calwt=True\n",
        "caltable='cal.B' finterp='linear' fldmap='3' spwmap=[0,0,0,0] calwt=False\n",
        "```\n",
        "\n",
        "In this case, solutions from *cal.G* will be selected based on directional proximity (*\\'nearest\\'*) for each MS field via the *fldmap* parameter, and spw 2 will be calibrated by spw 1 solutions.  For *cal.B*, solutions from field id 3 will be selected exclusively from the caltable, with spw 0 calibrating all MS spws (of which there are apparently 4).  The corresponding settings for the traditional parameters is as follows:\n",
        "\n",
        "```\n",
        "applycal(vis='my.ms',gaintable=['cal.G','cal.B'], gainfield=['nearest','3'],\n",
        "         interp=['linear','linear,linear'],\n",
        "         spwmap=[[0,1,1,3],[0,0,0,0]],calwt=[True,False])\n",
        "```\n",
        "\n",
        "Comment lines may be included in the cal library file by starting a line with the $#$ character.  (Partial line comments are *not* supported, as yet.)  Existing cal library lines can be turned off (for experimentation purposes) by making those lines comments with $#$.\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "X6wpcy_n0EDz"
      },
      "source": [
        "## More advanced Cal Library Usage\n",
        "\n",
        "The real power of the Cal Library arises from the ability to specify calibration instructions for a caltable *per MS selection*.  This enables consolidating what would be multiple **applycal** executions using the traditional parameters into a single execution.  Extending the example from above, if the MS field *\\'cal\\'* should be calibrated by *cal.G* with *\\'nearest\\'* interpolation in time, and the field *\\'sci\\'* with *\\'linear\\'* interpolation in time, the following Cal Library file will achieve this:\n",
        "\n",
        "```\n",
        "#mycal.txt cal library file\n",
        "caltable='cal.G' field='cal' tinterp='nearest' fldmap='nearest' spwmap=[0,1,1,3] calwt=True\n",
        "caltable='cal.G' field='sci' tinterp='linear' fldmap='nearest' spwmap=[0,1,1,3] calwt=True\n",
        "caltable='cal.B' finterp='linear' fldmap='3' spwmap=[0,0,0,0] calwt=False\n",
        "```\n",
        "\n",
        "Note that the algorithm for selecting solutions from the caltable (*fldmap=\\'nearest\\'*, which may resolve differently for the two MS fields) hasn\\'t been changed, but it could be.  In fact, for each caltable, any of the calibration parameters can be adjusted per MS selection, except *calwt*, which if set to *True* (or *False*) for the first-specified MS selection, will be forced to *True* (or *False*) for all entries for that caltable, to maintain weight consistency within the MS.  In general, it is best to specify *calwt* (or rely on the default, *True*) uniformly for all entries, per caltable, to avoid confusion.   MS selection by spw, intent, and obs id can also be used (see the glossary below).The pair of **applycal** executions corresponding to this Cal Library would be:\n",
        "\n",
        "```\n",
        "applycal(vis='my.ms',field='cal',gaintable=['cal.G','cal.B'], gainfield=['nearest','3'],\n",
        "         interp=['nearest','linear,linear'], spwmap=[[0,1,1,3],[0,0,0,0]],calwt=[True,False])\n",
        "\n",
        "applycal(vis='my.ms',field='sci',gaintable=['cal.G','cal.B'], gainfield=['nearest','3'],\n",
        "         interp=['linear','linear,linear'], spwmap=[[0,1,1,3],[0,0,0,0]],calwt=[True,False])\n",
        "```\n",
        "\n",
        "When there are many fields to which to apply carefully-selected calibration, *fldmap=\\'nearest\\'* may not properly select the correct calibrator fields for each target field.  In this case, the index list style form of *fldmap* (like *spwmap*) can be used (where field ids 1,4,6 are calibators, and 2,5,7 are the corresponding science fields):\n",
        "\n",
        "```\n",
        "#mycal.txt cal library file\n",
        "caltable='cal.G' field='1,2,3,4,5,6,7' tinterp='nearest' fldmap=[0,1,1,3,4,4,6,6] spwmap=[0,1,1,3] calwt=True\n",
        "caltable='cal.B' finterp='linear' fldmap='3' spwmap=[0,0,0,0] calwt=False\n",
        "```\n",
        "\n",
        "In this example, field 1 will calibrate itself and field 2.  Similarly, 4 will calibrate itself and 5, and 6 will calibrate itself and 7.   The bandpass calibrator (3) has been included, too, calibrating itself.  Field indices are specified in the *field* and *fldmap* parameters here, for clarity.   While field names can be used in *field*, the *fldmap* parameter, which in this form is an indexing list, can only interpret indices (note that field 0 is also explicitly included in the *fldmap* to preserve the integrity of the indexing). \n",
        "\n",
        "If multiple calibrators are required for each individual science fields, use the string selection form of *fldmap*, and specify separate entries for each science field:\n",
        "\n",
        "```\n",
        "#mycal.txt cal library file\n",
        "caltable='cal.G' field='1,3,4,6,8,9,10' tinterp='nearest' fldmap='nearest' spwmap=[0,1,1,3] calwt=True\n",
        "caltable='cal.G' field='2' tinterp='linear' fldmap='1,8' spwmap=[0,1,1,3] calwt=True\n",
        "caltable='cal.G' field='5' tinterp='linear' fldmap='4,9' spwmap=[0,1,1,3] calwt=True\n",
        "caltable='cal.G' field='7' tinterp='linear' fldmap='6,10' spwmap=[0,1,1,3] calwt=True\n",
        "caltable='cal.B' finterp='linear' fldmap='3' spwmap=[0,0,0,0] calwt=False\n",
        "```\n",
        "\n",
        "The additional calibrators for science fields 2, 5, and 7 are 8, 9, and 10, respectively.  The first entry for *cal.G* accounts for all calibrators (including field 3, the bandpass calibrator), using *fldmap=\\'nearest\\'* to ensure they are each calibrated solely by themselves.  Then, in separate entries, fields 1 and 8 are selected for field 2, fields 4 and 9 are selected for field 5, and fields 6 and 10 are selecterd for field 7.   When using the string selection style in *fldmap*, field names can be used, if desired.\n",
        "\n",
        "\n",
        "**Exclusivity**\n",
        "\n",
        "Since the Cal Library permits MS-selection-specific calibration specifications, it is even possible to specify different caltables for different MS selections, and take advantage of an implicit *exclusivity* property of the Cal Library.  In the above example, the G calibration for the *\\'cal\\'* and *\\'sci\\'* fields may come from different caltables, *\\'cal.Gcal\\'* and *\\'cal.Gsci\\',* respectively (these caltables may have been solved with different solution intervals, for example).  We would specify the Cal Library as follows:\n",
        "\n",
        "```\n",
        "#mycal.txt cal library file\n",
        "caltable='cal.Gcal' field='cal' tinterp='nearest' fldmap='nearest' spwmap=[0,1,1,3] calwt=True\n",
        "caltable='cal.Gsci' field='sci' tinterp='linear' fldmap='nearest' spwmap=[0,1,1,3] calwt=True\n",
        "caltable='cal.B' finterp='linear' fldmap='3' spwmap=[0,0,0,0] calwt=False\n",
        "```\n",
        "\n",
        "In this case, the *cal.B* table would be applied to both fields as before, but *cal.Gcal* would only be applied to field *\\'cal\\'* and *cal.Gsci* would only be applied to field *\\'sci\\'*.   Both tables would ignore data from the field they weren\\'t intended for.   The corresponding pair of **applycal** calls would be executed as follows:\n",
        "\n",
        "```\n",
        "applycal(vis='my.ms',field='cal',gaintable=['cal.Gcal','cal.B'], gainfield=['nearest','3'],\n",
        "         interp=['nearest','linear,linear'], spwmap=[[0,1,1,3],[0,0,0,0]],calwt=[True,False])\n",
        "\n",
        "applycal(vis='my.ms',field='sci',gaintable=['cal.Gsci','cal.B'], gainfield=['nearest','3'],\n",
        "         interp=['linear','linear,linear'], spwmap=[[0,1,1,3],[0,0,0,0]],calwt=[True,False])\n",
        "```\n",
        "\n",
        "<div class=\"alert alert-warning\">\n",
        "NB: The Cal Library exclusivity property described here only works in CASA version 5.3 and later.  In prior versions, the Cal Library system implicitly assumed that all caltables specifed in the Cal Library were nominally intended for all data selections and would have as much MS-selection-specificty as needed explicitly included.   In that case, missing explicit specifications would result in an error message indicating that the Cal Library was missing an explicit MS-selection-specific entry.\n",
        "</div>\n",
        "\n",
        " \n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "twz71THO0ED0"
      },
      "source": [
        "## General Rules (current, as of CASA 4.5, 4.6, 4.7, 5.\\*)\n",
        "\n",
        "-    Each non-comment line in the Cal Library file must contain a valid (existing) caltable name\n",
        "-    Blank lines (i.e., containing whitespace only) will be ignored\n",
        "-    All parameters (see glossary below) are name/value pairs using an equals sign, delimited with spaces (no commas!)\n",
        "-   Only those parameters (see glossary) for which non-default values are required need be specified\n",
        "-   Each set of coordinated instructions must occur on a single line (there is no line continuation operator, as yet)\n",
        "-   If detailed MS selection is used, care must be exercised to ensure it is mutually exclusive over all MS rows for the same caltable; there is currently no internal checking for redundancy, and only the last calibration instructions for a particular MS selection will be invoked\n",
        "-   Full-line comments are supported by inserting the $#$ character as the first non-whitespace character in the line.  This mechanism can be used to turn off ordinary cal library lines.\n",
        "-   When quoted items within a selection string are used, e.g. field=\\'\\\"B0007+106; J0010+109\\\",GRB021004\\', the string must have double quotation marks enclosing single quotation marks or single quotation marks enclosing double quotation marks.  Parsing will fail with a syntax error if the enclosed marks match the outer marks.  Note: the enclosed quotation marks are not needed; field=\\'B0007+106; J0010+109,GRB021004\\' would work, with the field names separated by commas.\n",
        "\n",
        "**Limitations**\n",
        "\n",
        "-   Application of parallactic angle corrections is not yet supported within the Cal Library file (this only affects use in plotms, where there is no parang parameter)\n",
        "-   Some parametrized calibration tables (*BPOLY*, *GSPLINE*) are not yet supported\n",
        "\n",
        " \n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9tqVCozk0ED0"
      },
      "source": [
        "## Conversion from Existing applycal Scripts\n",
        "\n",
        "To convert exiting **applycal** commands, a simple experimental function, **applycaltocallib** is available.  To access it, type (within CASA):\n",
        "\n",
        "```\n",
        "from callibrary import applycaltocallib\n",
        "```\n",
        "\n",
        "Then, chose a filename for the cal library file, and supply existing settings for **applycal** parameters (*field*, *spw*, *intent*, *gaintable*, *gainfield*, *interp*, *spwmap*, *calwt*) to the **applycaltocallib** function:\n",
        "\n",
        "```\n",
        "callibfile='mycallib.txt'\n",
        "applycaltocallib(filename=callibfile,append=False,\n",
        "                 field,spw,intent,gaintable,gainfield,\n",
        "                 interp,spwmap,calwt)\n",
        "```\n",
        "\n",
        "If *append=False*, the specified *filename* will be overwritten, if it already exists.  If *append=True*, new entries will be appended to the existing *filename*.  Only parameters with non-trivial **applycal** settings need be included.  In general, if *gaintable* is a python list, it is best if *gainfield*, *interp*, *spwmap*, and *calwt* (where non-trivially set) are also lists.For example, if your conventional script contains the following **applycal** executions (duplicated from above):\n",
        "\n",
        "```\n",
        "applycal(vis='my.ms',field='cal',\n",
        "         gaintable=['cal.G','cal.B'], gainfield=['nearest','3'],\n",
        "         interp=['nearest','linear,linear'],\n",
        "         spwmap=[[0,1,1,3],[0,0,0,0]],\n",
        "         calwt=[True,False])\n",
        "applycal(vis='my.ms',field='sci',\n",
        "         gaintable=['cal.G','cal.B'], gainfield=['nearest','3'],\n",
        "         interp=['linear','linear,linear'],\n",
        "         spwmap=[[0,1,1,3],[0,0,0,0]],\n",
        "         calwt=[True,False])\n",
        "```\n",
        "\n",
        "\\...these can be edited to **applycaltocallib** executions as:\n",
        "\n",
        "```\n",
        "callibfile='mycallib.txt'\n",
        "applycaltocallib(filename='mycallib.txt',append=False,\n",
        "                 field='cal',\n",
        "                 gaintable=['cal.G','cal.B'], gainfield=['nearest','3'],\n",
        "                 interp=['nearest','linear,linear'],\n",
        "                 spwmap=[[0,1,1,3],[0,0,0,0]],\n",
        "                 calwt=[True,False])\n",
        "applycaltocallib(filename='mycallib.txt',append=True,\n",
        "                 field='sci',\n",
        "                 gaintable=['cal.G','cal.B'],\n",
        "                 gainfield=['nearest','3'],\n",
        "                 interp=['linear','linear,linear'],\n",
        "                 spwmap=[[0,1,1,3],[0,0,0,0]],\n",
        "                 calwt=[True,False])\n",
        "```\n",
        "\n",
        "After running them, *mycallib.txt* will contain:\n",
        "\n",
        "```\n",
        "caltable='cal.B' calwt=False field='cal' tinterp='linear' finterp='linear' fldmap='3' spwmap=[0, 0, 0, 0]\n",
        "caltable='cal.G' calwt=True field='cal' tinterp='nearest' fldmap='nearest' spwmap=[0, 1, 1, 3]\n",
        "caltable='cal.B' calwt=False field='sci' tinterp='linear' finterp='linear' fldmap='3' spwmap=[0, 0, 0, 0]\n",
        "caltable='cal.G' calwt=True field='sci' tinterp='linear' fldmap='nearest' spwmap=[0, 1, 1, 3]\n",
        "```\n",
        "\n",
        "Note that the *cal.B* table is specified separately for the *\\'cal\\'* and *\\'sci\\'* fields with otherwise the same parameters; thus, those two lines could be manually consolidated to a single line with unified field selection, yielding:\n",
        "\n",
        "```\n",
        "caltable='cal.B' calwt=False field='cal,sci' tinterp='linear' finterp='linear' fldmap='3' spwmap=[0, 0, 0, 0]\n",
        "caltable='cal.G' calwt=True field='cal' tinterp='nearest' fldmap='nearest' spwmap=[0, 1, 1, 3]\n",
        "caltable='cal.G' calwt=True field='sci' tinterp='linear' fldmap='nearest' spwmap=[0, 1, 1, 3]\n",
        "```\n",
        "\n",
        "The field selection for the first row could be removed entirely if *cal.B* will be used uniformly for all fields in the MS (equivalently, *field=\\'\\'*).  This sort of row consolidation is optional, but it may have useful memory efficiency benefits when running **applycal**, and so is recommended.The **applycaltocallib** function should be considered experimental and used with care, and the resulting file examined thoroughly for correctness, since this function will not do any internal duplication checking or other sanity checks.  All other current constraints and limitations on cal libraries (as noted above) will apply.\n",
        "\n",
        " \n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "AM9lPMKm0ED1"
      },
      "source": [
        "## Glossary\n",
        "\n",
        "This is a list of recognized Cal Library parameters.  For each, the default is indicated.  Additional parameters enhancing flexibility will be added in CASA 4.5 and later.\n",
        "\n",
        "**General**\n",
        "\n",
        "-   *caltable* \\-\\-- the name of the caltable for which the instructions on the current line apply; no default; required\n",
        "\n",
        "**MS Selection**\n",
        "\n",
        "Use these parameters to implement calibration instructions specific to particular MS selections (using standard MS Selection syntax, except where noted):\n",
        "\n",
        "-   *field* \\-\\-- the MS field selection for which the calibration instructions on the current line apply; default=*\\'\\'* (all fields)\n",
        "-   *spw* \\-\\-- the MS spw selection for which the calibration instructions on the current line apply; default=*\\'\\'* (all spws) Note that channel selection will be ignored, since the Cal Library does not support variety in calibration application at channel granularity.\n",
        "-   *intent* \\-\\-- the MS intent selection for which the calibration instructions on the current line apply; default=*\\'\\'* (all intents)\n",
        "-   *obs* \\-\\-- the MS observation id selection for which the calibration instructions on the current line apply; default=*\\'\\'* (all obs ids)\n",
        "\n",
        "**Interpolation/application**\n",
        "\n",
        "-   *tinterp* \\-\\-- the time-dependent interpolation mode; default=*\\'linear\\'* options: *\\'linear\\'*, *\\'nearest\\'*\n",
        "-   *finterp* \\-\\-- the chan-dependent interpolation mode (only relevant for channelized caltables); default=\\'linear\\' options: *\\'nearest\\', \\'linear\\', \\'cubic\\', \\'spline\\'*\n",
        "-   *calwt* \\-\\-- weight calibration; default=*True*  options: *True*, *False*\n",
        "\n",
        "**Calibration mapping**\n",
        "\n",
        "The following *\\*map* parameters enable selection on the caltable.  For each *\\*map* parameter, the basic specification is an ordered list indicating the caltable selection indices intended for each MS index on that axis.  E.g., *spwmap=\\[0,1,1,3\\]* means MS spws 0,1,3 will each be be calibrated by the same spw index from the caltable, and MS spw 2 will be calibrated by cal spw 1.  The *\\*map* parameters support other short-hand options as well, as indicated below.  For defaults, \\\"index identity\\\" means that each MS index will be calibrated by the corresponding caltable index, and \\\"no explicit mapping\\\" means that no filter will be applied to that axis, and all available solutions on the axis will be included.\n",
        "\n",
        "-   *spwmap* \\-\\-- spectral window mapping; default=index identity\n",
        "-   *fldmap* \\-\\-- field mapping; default=*\\[\\]* (no explicit mapping); additional options: *\\'nearest\\'* or a string indicating field selection on the caltable (same as traditional *gainfield* options)\n",
        "-   *antmap* \\-\\-- antenna id mapping; default=index identity\n",
        "-   *obsmap* \\-\\-- obs id mapping; default=*\\[\\]* (no explicit mapping)\n",
        "\n",
        "\n",
        "\n"
      ]
    }
  ]
}