docs/notebooks/casa-fundamentals.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "casa-fundamentals.ipynb",
      "provenance": [],
      "toc_visible": true
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EMdqMkmcVczi"
      },
      "source": [
        "# CASA Fundamentals\n",
        "\n",
        "Fundamentals of CASA: Measurement Equation, Science Data Model, and MeasurementSet\n",
        "\n",
        "\n",
        "\n",
        "***\n",
        "\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HQoini5aVczj"
      },
      "source": [
        "## Measurement Equation\n",
        "\n",
        "The visibilities measured by an interferometer must be calibrated before formation of an image. This is because the wavefronts received and processed by the observational hardware have been corrupted by a variety of effects. These include (but are not exclusive to): the effects of transmission through the atmosphere, the imperfect details amplified electronic (digital) signal and transmission through the signal processing system, and the effects of formation of the cross-power spectra by a correlator. Calibration is the process of reversing these effects to arrive at corrected visibilities which resemble as closely as possible the visibilities that would have been measured in vacuum by a perfect system. The subject of this chapter is the determination of these effects by using the visibility data itself.\n",
        "\n",
        "\n",
        "**The HBS Measurement Equation**\n",
        "\n",
        "The relationship between the observed and ideal (desired) visibilities on the baseline between antennas i and j may be expressed by the Hamaker-Bregman-Sault *Measurement Equation* Hamaker, Bregman, & Sault (1996) [\\[1\\]](casa-fundamentals.ipynb#bibliography) and Sault, Hamaker, Bregman (1996) [\\[2\\]](casa-fundamentals.ipynb#bibliography) .\n",
        "\n",
        "$$\\begin{eqnarray} \\vec{V}_{ij}~=~J_{ij}~\\vec{V}_{ij}^{\\mathrm{~IDEAL}} \\end{eqnarray}$$\n",
        "\n",
        "where $\\vec{V}_{ij}$ represents the observed visibility, a complex number representing the amplitude and phase of the correlated data from a pair of antennas in each sample time, per spectral channel. $\\vec{V}_{ij}^{\\mathrm{~IDEAL}}$ represents the corresponding ideal visibilities, and $J_{ij}$ represents the accumulation of all corruptions affecting baseline $ij$. The visibilities are indicated as vectors spanning the four correlation combinations which can be formed from dual-polarization signals. These four correlations are related directly to the Stokes parameters which fully describe the radiation. The $J_{ij}$ term is therefore a $4\\times4$ matrix. Most of the effects contained in $J_{ij}$ (indeed, the most important of them) are antenna-based, i.e., they arise from measurable physical properties of (or above) individual antenna elements in a synthesis array. Thus, adequate calibration of an array of $N_{ant}$ antennas forming $N_{ant} (N_{ant}-1)/2$ baseline visibilities is usually achieved through the determination of only $N_{ant}$ factors, such that $J_{ij} = J_i \\otimes J_j^{*}$.  For the rest of this chapter, we will usually assume that $J_{ij}$ is factorable in this way, unless otherwise noted.\n",
        "\n",
        "As implied above, $J_{ij}$ may also be factored into the sequence of specific corrupting effects, each having their own particular (relative) importance and physical origin, which determines their unique algebra. Including the most commonly considered effects, the Measurement Equation can be written:\n",
        "\n",
        "$$\\begin{eqnarray} \\vec{V}_{ij}~=~M_{ij}~B_{ij}~G_{ij}~D_{ij}~E_{ij}~P_{ij}~T_{ij}~\\vec{V}_{ij}^{\\mathrm{~IDEAL}}    \\end{eqnarray}$$\n",
        "\n",
        "where:\n",
        "\n",
        "-   $T_{ij}~=~$ Polarization-independent multiplicative effects introduced by the troposphere, such as opacity and path-length variation.\n",
        "-   $P_{ij}~=~$ Parallactic angle, which describes the orientation of the polarization coordinates on the plane of the sky. This term varies according to the type of the antenna mount.\n",
        "-   $E_{ij}~=~$ Effects introduced by properties of the optical components of the telescopes, such as the collecting area\\'s dependence on elevation.\n",
        "-   $D_{ij}~=~$ Instrumental polarization response. \\\"D-terms\\\" describe the polarization leakage between feeds (e.g. how much the R-polarized feed picked up L-polarized emission, and vice versa).\n",
        "-   $G_{ij}~=~$ Electronic gain response due to components in the signal path between the feed and the correlator. This complex gain term $G_{ij}$ includes the scale factor for absolute flux density calibration, and may include phase and amplitude corrections due to changes in the atmosphere (in lieu of $T_{ij}$). These gains are polarization-dependent.\n",
        "-   $B_{ij}~=~$ Bandpass (frequency-dependent) response, such as that introduced by spectral filters in the electronic transmission system\n",
        "-   $M_{ij}~=~$ Baseline-based correlator (non-closing) errors. By definition, these are not factorable into antenna-based parts.      \n",
        "Note that the terms are listed in the order in which they affect the incoming wavefront ($G$ and $B$ represent an arbitrary sequence of such terms depending upon the details of the particular electronic system). Note that $M$ differs from all of the rest in that it is not antenna-based, and thus not factorable into terms for each antenna.As written above, the measurement equation is very general; not all observations will require treatment of all effects, depending upon the desired dynamic range. E.g., instrumental polarization calibration can usually be omitted when observing (only) total intensity using circular feeds. Ultimately, however, each of these effects occurs at some level, and a complete treatment will yield the most accurate calibration. Modern high-sensitivity instruments such as ALMA and JVLA will likely require a more general calibration treatment for similar observations with older arrays in order to reach  the advertised dynamic ranges on strong sources.In practice, it is usually far too difficult to adequately measure most calibration effects absolutely (as if in the laboratory) for use in calibration. The effects are usually far too changeable. Instead, the calibration is achieved by making observations of calibrator sources on the appropriate timescales for the relevant effects, and solving the measurement equation for them using the fact that we have $N_{ant}(N_{ant}-1)/2$ measurements and only $N_{ant}$ factors to determine (except for $M$ which is only sparingly used). Note: By partitioning the calibration factors into a series of consecutive effects, it might appear that the number of free parameters is some multiple of $N_{ant}$, but the relative algebra and timescales of the different effects, as well as the  multiplicity of observed polarizations and channels compensate, and it can be shown that the problem remains  well-determined until, perhaps, the effects are direction-dependent within the field of view. Limited solvers for such effects are under study; the **calibrater** tool currently only handles effects which may be assumed constant within the field of view. Corrections for the primary beam are handled in the **imager** tool.  Once determined, these terms are used to correct the visibilities measured for the scientific target. This procedure is known as cross-calibration (when only phase is considered, it is called phase-referencing).\n",
        "\n",
        "The best calibrators are point sources at the phase center (constant visibility amplitude, zero phase), with sufficient flux density to determine the calibration factors with adequate SNR on the relevant timescale. The primary gain calibrator must be sufficiently close to the target on the sky so that its observations sample the same atmospheric effects. A bandpass calibrator usually must be sufficiently strong (or observed with sufficient duration) to provide adequate per-channel sensitivity for a useful calibration. In practice, several calibrators are usually observed, each with properties suitable for one or more of the required calibrations.Synthesis calibration is inherently a bootstrapping process. First, the dominant calibration term is determined, and then, using this result, more subtle effects are solved for, until the full set of required calibration terms is available for application to the target field. The solutions for each successive term are relative to the previous terms. Occasionally, when the several calibration terms are not sufficiently orthogonal, it is useful to re-solve for earlier types using the results for later types, in effect, reducing the effect of the later terms on the solution for earlier ones, and thus better isolating them. This idea is a generalization of the traditional concept of self-calibration, where initial imaging of the target source supplies the visibility model for a re-solve of the gain calibration ($G$ or $T$). Iteration tends toward convergence to a statistically optimal image. In general, the quality of each calibration and of the source model are mutually dependent. In principle, as long as the solution for any calibration component (or the source model itself) is likely to improve substantially through the use of new information (provided by other improved solutions), it is worthwhile to continue this process.In practice, these concepts motivate certain patterns of calibration for different types of observation, and the **calibrater** tool in CASA is designed to accommodate these patterns in a general and flexible manner. For a spectral line total intensity observation, the pattern is usually:\n",
        "\n",
        "1.  Solve for $G$ on the bandpass calibrator\n",
        "2.  Solve for $B$ on the bandpass calibrator, using $G$\n",
        "3.  Solve for $G$ on the primary gain (near-target) and flux density calibrators, using $B$ solutions just obtained\n",
        "4.  Scale $G$ solutions for the primary gain calibrator according to the flux density calibrator solutions\n",
        "5.  Apply $G$ and $B$ solutions to the target data\n",
        "6.  Image the calibrated target data\n",
        "\n",
        "If opacity and gain curve information are relevant and available, these types are incorporated in each of the steps (in future, an actual solve for opacity from appropriate data may be folded into this process):\n",
        "\n",
        "1.  Solve for $G$ on the bandpass calibrator, using $T$ (opacity) and $E$ (gain curve) solutions already derived.\n",
        "2.  Solve for $B$ on the bandpass calibrator, using $G$, $T$ (opacity), and $E$ (gain curve) solutions.\n",
        "3.  Solve for $G$ on primary gain (near-target) and flux density calibrators, using $B$, $T$ (opacity), and $E$ (gain curve) solutions.\n",
        "4.  Scale $G$ solutions for the primary gain calibrator according to the flux density calibrator solutions\n",
        "5.  Apply $T$ (opacity), $E$ (gain curve), $G$, and $B$ solutions to the target data\n",
        "6.  Image the calibrated target data\n",
        "\n",
        "For continuum polarimetry, the typical pattern is:\n",
        "\n",
        "1.  Solve for $G$ on the polarization calibrator, using (analytical) $P$ solutions.\n",
        "2.  Solve for $D$ on the polarization calibrator, using $P$ and $G$ solutions.\n",
        "3.  Solve for $G$ on primary gain and flux density calibrators, using $P$ and $D$ solutions.\n",
        "4.  Scale $G$ solutions for the primary gain calibrator according to the flux density calibrator solutions.\n",
        "5.  Apply $P$, $D$, and $G$ solutions to target data.\n",
        "6.  Image the calibrated target data.\n",
        "\n",
        "For a spectro-polarimetry observation, these two examples would be folded together.In all cases the calibrator model must be adequate at each solve step. At high dynamic range and/or high resolution, many calibrators which are nominally assumed to be point sources become slightly resolved. If this has biased the calibration solutions, the offending calibrator may be imaged at any point in the process and the resulting model used to improve the calibration. Finally, if sufficiently strong, the target may be self-calibrated as well.\n",
        "\n",
        "\n",
        "\n",
        "**General Calibrater Mechanics**\n",
        "\n",
        "The **calibrater** tasks/tool are designed to solve and apply solutions for all of the solution types listed above (and more are in the works). This leads to a single basic sequence of execution for all solves, regardless of type:\n",
        "\n",
        "1.  Set the calibrator model visibilities\n",
        "2.  Select the visibility data which will be used to solve for a calibration type\n",
        "3.  Arrange to apply any already-known calibration types (the first time through, none may yet be available)\n",
        "4.  Arrange to solve for a specific calibration type, including specification of the solution timescale and other specifics\n",
        "5.  Execute the solve process\n",
        "6.  Repeat 1-4 for all required types, using each result, as it becomes available, in step 3, and perhaps repeating for some types to improve the solutions\n",
        "\n",
        "By itself, this sequence doesn\\'t guarantee success; the data provided for the solve must have sufficient SNR on the appropriate timescale, and must provide sufficient leverage for the solution (e.g., D solutions require data taken over a sufficient range of parallactic angle in order to separate the source polarization contribution from the instrumental polarization).\n",
        "\n",
        "\n",
        "***\n",
        "\n",
        "\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mwIWI6mVVczl"
      },
      "source": [
        "## Science Data Model\n",
        "\n",
        "It was decided realtively early in the preparatory phase of ALMA and EVLA that the two projects would:\n",
        "\n",
        "1. use the same data analysis software (CASA) and\n",
        "2. use essentially the same archive data format, the Astronomy Science Data Model (ASDM), also referred to as the ALMA Science Data Model for ALMA data or Science Data Model (SDM) for VLA data.\n",
        "\n",
        "The ASDM was developed to a first prototype by Francois Viallefond (Observatoire de Paris) as an extension and full generalisation of the MeasurementSet. The ASDM is superior to the MS w.r.t. the storage of observatory raw data in that it is capable of capturing the metadata of an interferometic or total-power dataset completely without any compromise including all data relevant for calibration and observatory administration.\n",
        "\n",
        "Just like for the MS, one can think of the ASDM as a relational database. And both databases have in principle a very similar layout. However, while the MS has only 12 required Subtables, the ASDM uses typically 40 Subtables, and there are more optional ones.\n",
        "\n",
        "ASDMs, however, are for data storage and data reduction should be done on the MeasurementSet (although when importing data through **importasdm** with option *lazy=True*  the ASDM is restructured to resemble an MS).\n",
        "\n",
        "For the implementation of the ASDM, (then) novel source code generation techniques were applied which permitted simultaneous implementation in Java and C++. As the actual representation of the data on disk, a hybrid format was chosen: all low-volume metadata is stored as XML files (one per table) while the bulk data is stored in a binary format (MIME) in so-called Binary Large Objects (BLOBs). In particular the Main table is stored as a series of BLOBs of a few GB each with lossless compression. This makes the ASDM more efficient as a bulk data format than the MS which stores the the DATA column of the Main table as one single monolithic file.\n",
        "\n",
        "An up to date description of the tables of the ASDM is [given in this pdf](https://drive.google.com/file/d/16a3g0GQxgcO7N_ZabfdtexQ8r2jRbYIS/view?usp=sharing).\n",
        "\n",
        "This [tar file](https://drive.google.com/file/d/1o_PefqwNubroYSjYzZWv81fquUEFNhWL/view?usp=sharing) contains the XML Schema Definition (xds) files for all of the tables described in the associated ASDM Short Table Description. Use \\\"tar xvfz 0asdmSchematas_v8Dec2020.tgz\\\" to extract its contents.\n",
        "\n",
        "This [tar file](https://drive.google.com/file/d/1B05EB6G_vLqiPUjzKY8cbO5u12pLq4OF/view?usp=sharing) contains the XML Schema Definition (xds) files for all of the enumerations used by the ASDM tables. Use \\\"tar xvfz 0enumerationsSchematas.tgz\\\" to extract its contents.\n",
        "\n",
        "The binary data format is [given in this pdf](https://drive.google.com/file/d/1PMrZFbkrMVfe57K6AAh1dR1FalS35jP2/view?usp=sharing).\n",
        "\n",
        "\n",
        "\n",
        "***\n",
        "\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "aGZSd4hkVczm"
      },
      "source": [
        "## MeasurementSet Basics\n",
        "\n",
        "Data is handled in CASA via the table system. In particular, visibility data are stored in a CASA table known as a MeasurementSet (MS). Details of the physical and logical MS structure are given below, but for our purposes here an MS is just a construct that contains the data. An MS can also store single dish data (as an auto-correlation-only data set), see \"[Single-dish data calibration and reduction](single_dish_calibration.ipynb#Calibration-&-Reduction)\".\n",
        "\n",
        "A full description of the MeasurementSet can be found [here](casa-fundamentals.ipynb#measurementset-v2), and a description of the MS model column can be found in the [Synthesis Calibration](synthesis_calibration.ipynb) section.\n",
        "\n",
        "<div class=\"alert alert-info\">\n",
        "**Inside the Toolkit**: MeasurementSets are handled in the **ms** tool. Import and export methods include **ms.fromfits** and **ms.tofits**.\n",
        "</div>\n",
        "\n",
        "<div class=\"alert alert-info\">\n",
        "**NOTE**: Images are handled through special image tables, although standard FITS I/O is also supported. Images and image data are described in \"[Dealing with Images\"](image_analysis.ipynb#Dealing-with-Images).\n",
        "</div>\n",
        "\n",
        "The headers of any FITS files can be displayed in the logger with the **listfits** task:\n",
        "\n",
        "```\n",
        "#listfits :: List the HDU and typical data rows of a fits file:\n",
        "fitsfile = '' # Name of input fits file\n",
        "```\n",
        "\n",
        "More Information on how to access Visibility Data is provided in the \\\"[Data Examination and Editing](data_examination.ipynb)\\\" chapter.\n",
        "\n",
        "Unless your data was previously processed by CASA, you will need to import it into CASA as an MS. Supported formats include some \"standard\" flavors of UVFITS, the VLA \"Export\" archive format, and most recently, the Astronomy Science Data Model (ASDM) format. These are described in \\\"[UV Data Import](visibilities_import_export.ipynb#UV-Data-Import)\\\".\n",
        "\n",
        "Once in MeasurementSet form, your data can be accessed through various tools and tasks with a common interface. The most important of these is the [data selection](visibility_data_selection.ipynb) interface, which allows you to specify the subset of the data on which the tasks and tools will operate.\n",
        "\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UuceR-HYVczm"
      },
      "source": [
        "**Under the Hood: Structure of the MeasurementSet**\n",
        "\n",
        "<div class=\"alert alert-info\">\n",
        "**Inside the Toolkit**: Generic CASA tables are handled in the **tb** tool. You have direct access to keywords, rows and columns of the tables with the methods of this tool.\n",
        "</div>\n",
        "\n",
        "It is not necessary that a casual CASA user know the specific details on how the data in the MS is stored and the contents of all the sub-tables. However, CASA docs occasionally refers to specific \"columns\" of the MS when describing the actions of various tasks, and thus we provide the following synopsis to familiarize the user with the necessary nomenclature.\n",
        "\n",
        "All CASA data files, including MeasurementSets, are written into the current working directory by default, with each CASA table represented as a separate sub-directory. MS names therefore need only comply with UNIX file or directory naming conventions, and can be referred to from within CASA directly, or via full path names.\n",
        "\n",
        "An MS consists of a *MAIN* table containing the visibility data and associated sub-tables containing auxiliary or secondary information. The tables are logical constructs, with contents located in the physical table.\\* files on disk. The *MAIN* table consists of the table.\\* files in the main directory of the MS-file itself, and the other tables are in the respective subdirectories. The various MS tables and sub-tables can be seen by listing the contents of the MS directory itself (e.g. using Unix ls), or via the **browsetable** task.\n",
        "\n",
        "See figure 1 for an example of the contents of a MS directory. Or, from the casa prompt,\n",
        "\n",
        "```\n",
        "CASA <1>: ls ngc5921.ms #IPython system call: ls -F ngc5921.ms\n",
        "ANTENNA           POLARIZATION     table.f1        table.f3_TSM1  table.f8\n",
        "DATA_DESCRIPTION  PROCESSOR        table.f10       table.f4       table.f8_TSM1\n",
        "FEED              SORTED_TABLE     table.f10_TSM1  table.f5       table.f9\n",
        "FIELD             SOURCE           table.f11       table.f5_TSM1  table.f9_TSM1\n",
        "FLAG_CMD          SPECTRAL_WINDOW  table.f11_TSM1  table.f6       table.info\n",
        "HISTORY           STATE            table.f2        table.f6_TSM0  table.lock\n",
        "OBSERVATION       table.dat        table.f2_TSM1   table.f7\n",
        "POINTING          table.f0         table.f3        table.f7_TSM1\n",
        "```\n",
        "\n",
        "<div class=\"alert alert-info\">\n",
        "**NOTE**: The *MAIN* table information is contained in the table.* files in this directory.\n",
        "</div>\n",
        "\n",
        "Each of the sub-table sub-directories contain their own table.dat and other files, e.g.\n",
        "\n",
        "```\n",
        "CASA <2>: ls ngc5921.ms/SOURCE #IPython system call: ls -F ngc5921.ms/SOURCE\n",
        "table.dat  table.f0  table.f0i  table.info  table.lock\n",
        "```\n",
        "\n",
        "![63d12aa5a921803be758c7bc7488c6c62353d5a4](https://github.com/casangi/casadocs/blob/master/docs/notebooks/media/63d12aa5a921803be758c7bc7488c6c62353d5a4.png?raw=1){.image-inline}\n",
        "\n",
        ">Figure 1: The contents of a MeasurementSet. These tables compose a MeasurementSet named ngc5921.demo.ms on disk. This display is obtained by using the File\\\\:Open menu in browsetable and left double-clicking on the ngc5921.demo.ms directory.\n",
        "\n",
        "Each \"row\" in a table contains entries for a number of specified \"columns\". For example, in the *MAIN* table of the MS, the original visibility data is contained in the *DATA* column --- each \"cell\" contains a matrix of observed complex visibilities for that row at a single time stamp, for a single baseline in a single spectral window. The shape of the data matrix is given by the number of channels and the number of correlations (voltage-products) formed by the correlator for an array.\n",
        "\n",
        "Table 1 lists the non-data columns of the *MAIN* table that are most important during a typical data reduction session. Table 2 at the bottom lists the key data columns of the *MAIN* table of an interferometer MS. The MS produced by fillers for specific instruments may insert special columns, such as *ALMA_PHASE_CORR*, *ALMA_NO_PHAS_CORR* and *ALMA_PHAS_CORR_FLAG_ROW* for ALMA data filled using the **importasdm** filler. These columns are visible in **browsetable** and are accessible from the toolkit in the **ms** tool (e.g. the **ms.getdata** method) and from the **tb** \"table\" tool (e.g. using **tb.getcol**).\n",
        "\n",
        "<div class=\"alert alert-info\">\n",
        "**NOTE**: When you examine table entries for IDs such as *FIELD_ID* or *DATA_DESC_ID*, you will see 0-based numbers.\n",
        "</div>\n",
        "\n",
        "\n",
        "\n",
        "Parameter | Contents\n",
        "--- | ---\n",
        "*ANTENNA1* | First antenna in baseline\n",
        "*ANTENNA2* | Second antenna in baseline\n",
        "*FIELD_ID* | Field (source no.) identification\n",
        "*DATA_DESC_ID* | Spectral window number, polarization identifier pair (IF no.)\n",
        "*ARRAY_ID* | Subarray number\n",
        "*OBSERVATION_ID* | Observation identification\n",
        "*POLARIZATION_ID* | Polarization identification\n",
        "*SCAN_NUMBER* | Scan number\n",
        "*TIME* | Integration midpoint time\n",
        "*UVW* | UVW coordinates\n",
        "\n",
        ">Table 1: Common columns in the *MAIN* table of the MS.\n",
        "  \n",
        "\n",
        "\n",
        "The MS can contain a number of \"scratch\" columns, which are used to hold useful versions of other columns such as the data or weights for further processing. The most common scratch columns are:\n",
        "\n",
        "-   *CORRECTED_DATA* --- used to hold calibrated data for imaging or display;\n",
        "-   *MODEL_DATA* --- holds the Fourier inversion of a particular model image for calibration or imaging. This column is optional.\n",
        "\n",
        "The creation and use of the scratch columns is generally done behind the scenes, but you should be aware that they are there (and when they are used).\n",
        "\n",
        "\n",
        "\n",
        "Column | Format | Contents\n",
        "--- | --- | ---\n",
        "*DATA* | Complex(Nc, Nf) | complex visibility data matrix (= ALMA_PHASE_CORR by default)\n",
        "*FLAG* | Bool(Nc, Nf) | cumulative data flags\n",
        "*WEIGHT* | Float(Nc) | weight for a row\n",
        "*SIGMA* | Float(Nc) | sigma for a row\n",
        "*WEIGHT_SPECTRUM* | Float(Nc, Nf) | individual weights for a data matrix\n",
        "*SIGMA_SPECTRUM* | Float(Nc, Nf) | individual sigmas for a data matrix\n",
        "*ALMA_PHASE_CORR* | Complex(Nc, Nf) | on-line phase corrected data (Not in VLA data)\n",
        "*ALMA_NO_PHAS_CORR* | Bool(Nc, Nf) | data that has not been phase corrected (Not in VLA data)\n",
        "*ALMA_PHAS_CORR_FLAG_ROW* | Bool(Nc, Nf) | flag to use phase-corrected data or not (not in VLA data)\n",
        "*MODEL_DATA* | Complex(Nc, Nf) | Scratch: created by calibrater or imager tools\n",
        "*CORRECTED_DATA* | Complex(Nc, Nf) | Scratch: created by calibrater or imager tools\n",
        "\n",
        ">Table 2: Commonly accessed *MAIN* Table data-related columns. **NOTE**: The columns *ALMA_PHASE_CORR, ALMA_NO_PHAS_CORR and ALMA_PHAS_CORR_FLAG_ROW* are specific to ALMA data filled using the **importasdm** filler.\n",
        "  \n",
        "\n",
        "Data flags can be set in the MS, too. Whenever a flag is set, the data will be ignored in all processing steps but not physically deleted from the MS. The flags are channel-based and stored in the MS *FLAG* subtable. Backups can be stored in the MS.flagversions file that can be accessed via the **flagmanager**.\n",
        "\n",
        "The most recent specification for the MS is [MeasurementSet definition version 2.0](casa-fundamentals.ipynb#measurementset-v2).\n",
        "\n",
        "\n",
        "\n",
        "***\n",
        "\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "etXlveJYVczn"
      },
      "source": [
        "## MeasurementSet v2\n",
        "\n",
        "\n",
        "The MeasurementSet version 2 [[3](casa-fundamentals.ipynb#bibliography)], is a database designed to hold radioastronomical data to be calibrated following the MeasurementEquation approach by Hamaker, Bregman, and Sault (1996).\n",
        "\n",
        "Since its publication, the MeasurementSet (MS) design has been implemented by several software development groups, among them the CASA team and, e.g., the European VLBI Network team. CASA has also adopted the MeasurementEquation as its fundamental calibration scheme and has thus embraced the MS as its native way to store radio observations. With CASA becoming the designated analysis package for ALMA and the VLA, this means that the MS is now the default way of storing ALMA and VLA data during the actual analysis.\n",
        "\n",
        "The ALMA and VLA raw data format, however, is not the MS but the so-called Astronomy Science Data Model (ASDM), also referred to as the ALMA Science Data Model for ALMA, and the Science Data Model (SDM) for the VLA. The ALMA and VLA archives hence do not store data in MS format but in ASDM format, and when a CASA user starts to work with this data, the first step has to be the import of the ASDM into the CASA MS format.\n",
        "\n",
        "The MS is effectively a relational database which on the one hand tries to permit the storage of all imaginable radio (interferometric, single-dish) data with corresponding metadata, and on the other hand ventures to be storage-space and data-maintenance efficient by avoiding data redundancy.\n",
        "\n",
        "The universality is achieved by offering many optional parts in the format which cover most imaginable use cases in radio astronomy. So a simple, few-antenna interferometer observing a simple object with time-independent position at just a single frequency can store its data using a small sub-set of the format while a large interferometer with antennas on time-dependent locations, observing many objects in rapid succession with time-dependent source positions using a complex, time-dependent spectral setup etc., can equally use the MS to store its data albeit using a larger subset of the possibilities of the MS.\n",
        "\n",
        "The non-redundance of the format is achieved by simply following the standard approach of relational databases which is to put repeating pieces of information into separate database tables, the Subtables, and replacing them in the main body of the data base, the Main table, by references to the Subtables. In the case of the MS this happens in two layers of Subtables with the first layer being referenced by the Main table and the second layer being referenced by the first layer. I.e., there are some Subtables which reference other Subtables.\n",
        "\n",
        "The Subtable referencing mechanism is defined in the original design. It works either via the *line numbers* of the individual Subtable ,this implies that the reference is a zero-based integer and that the removal of a line in such a Subtable requires reindexing in the referencing table(s), or via explicit references to an index column in the Subtable ,the latter is much less common.\n",
        "\n",
        "These design principles lead to a format which puts the bulk of the data ,the interferometric visibilities and/or the single-dish total-power measurements with their timestamps, into a Main Table , and most of the metadata in the two layers of Subtables.\n",
        "\n",
        "In the CASA MS implementation, the individual Tables are all stored in the CASA Table format, i.e. they are actually not single files on disk but directories containing several files, essentially one for each column of the table. So the entire MS is also not a single file (like, e.g., in the FITS IDI format) but a whole directory tree. For transport, the MS typically has to be turned into a single file by using the command \\\"tar\\\".\n",
        "\n",
        "The Main Table contains the radio data initially in a column called *DATA* (interferometric data) or *FLOAT_DATA* (pure single-dish data). One of these two columns always has to be present.\n",
        "\n",
        "When a calibration is applied to the *DATA* column, a *CORRECTED_DATA* column is created to contain the calibrated data leaving the original data untouched. Furthermore, a *MODEL_DATA* column can be required to store expectation values for the emission of calibration sources.\n",
        "\n",
        "For large datasets these bulk data columns can require large amounts of disk space and access to them may be slow. To mitigate these problems, the CASA team is working on making the columns \\\"virtual\\\" as much as possible, i.e. replacing the *CORRECTED_DATA* and *MODEL_DATA* columns by parameterised versions calculated on-the-fly.\n",
        "\n",
        "In the case of the virtual *MODEL_DATA* column, this is essentially a model image which is stored with the MS and converted on-the-fly to visibilities.\n",
        "\n",
        "In the case of the virtual *CORRECTED_DATA* column, this is a so-called \\\"Cal Library\\\" which permits to calibrate the data in the *DATA* column on-the-fly and make the results available as if they were stored in a standard table column.\n",
        "\n",
        "Finally, a major case of data redundance for ALMA and VLA data is of course the fact that the raw data arrive at the user in ASDM format but then have to be translated into MS format which creates a completely redundant copy of all raw data without any gain for the user. This problem was addressed by introducing the so-called \\\"lazy\\\" import of ASDM data. The development is not yet completely finished but is already available for ALMA interferometric data. The idea here is to also make the *DATA* column virtual and perform the translation from the ASDM format on-the-fly. This typically shrinks the MS by a factor 30 in data volume. Of course the ASDM raw data has to be kept on disk for access. Access speeds to a virtual *DATA* column are essentially the same as to a non-virtual one. They may even be a little faster since the ASDM data is better compressed.\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rKBg4xXBVczo"
      },
      "source": [
        "### MS v2.0 Layout\n",
        "\n",
        "CASA uses the MeasurementSet Version 2  (A.J. Kemball and M.H. Wieringa, eds., 2000) as the internal working data format. The MeasurementSet set was orignially defined in AIPS++ Note 191 (Wieringa and Cornwell 1996).  Reproduced below is the table structrue for the MeasurementSet as used by CASA.\n",
        "\n",
        "There is a MAIN table containing a number of data columns and keys into various subtables. There is at most one of each subtable. The subtables are stored as keywords of the MS, and all defined sub-tables are tabulated below. Optional sub-tables are shown in italics and in parentheses.\n",
        "\n",
        "**Subtables**\n",
        "\n",
        "Table | Contents | Keys\n",
        "--- | --- | ---\n",
        "ANTENNA | Antenna characteristics | ANTENNA_ID\n",
        "DATA_DESCRIPTION | Data description | DATA_DESC_ID\n",
        "(*DOPPLER*) | Doppler tracking | DOPPLER_ID, SOURCE_ID\n",
        "FEED | Feed characteristics | FEED_ID, ANTENNA_ID, TIME, SPECTRAL_WINDOW_ID\n",
        "FIELD | Field position | FIELD_ID\n",
        "FLAG_CMD | Flag commands | TIME\n",
        "(*FREQ_OFFSET*) | Frequency offset information | FEED_ID, ANTENNA*n*, FEED_ID, TIME, SPECTRAL_WINDOW_ID\n",
        "HISTORY | History information | OBSERVATION_ID, TIME\n",
        "OBSERVATION | Observer, Schedule, etc | OBSERVATION_ID\n",
        "POINTING | Pointing information | ANTENNA_ID, TIME\n",
        "POLARIZATION | Polarization setup | POLARIZATION_ID\n",
        "PROCESSOR | Processor information | PROCESSOR_ID\n",
        "(*SOURCE*) | Source information | SOURCE_ID, SPECTRAL_WINDOW_ID, TIME\n",
        "SPECTRAL_WINDOW | Spectral window setups | SPECTRAL_WINDOW_ID\n",
        "STATE | State information | STATE_ID\n",
        "(*SYSCAL*) | System calibration characteristics | FEED_ID, ANTENNA_ID, TIME, SPECTRAL_WINDOW_ID\n",
        "(*WEATHER*) | Weather info for each antenna | ANTENNA_ID, TIME\n",
        "\n",
        "Note that there are two types of subtables. For the first, simpler type, the key (ID) is the row number in the subtable. Examples are FIELD*,* SPECTRAL_WINDOW*,* OBSERVATION and PROCESSOR. For the second, the key is a collection of parameters, usually including TIME. Examples are FEED*, (SOURCE),* (SYSCAL), and *(WEATHER)*.\n",
        "\n",
        "<div class=\"alert alert-info\">\n",
        "Note that all optional columns are indicated in italics and in parentheses.\n",
        "</div>\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PVSBtcG6Vczp"
      },
      "source": [
        "### MAIN table: Data, Coordinates and Flags\n",
        "\n",
        "+----------------------+------------------------------------+-------+---------+----------------------------------------------+\n",
        "| Name                 | Format                             | Units | Measure | Comments                                     |\n",
        "+======================+====================================+=======+=========+==============================================+\n",
        "| **Columns**          |                                    |       |         |                                              |\n",
        "+----------------------+------------------------------------+-------+---------+----------------------------------------------+\n",
        "| *Keywords*           |                                    |       |         |                                              |\n",
        "+----------------------+------------------------------------+-------+---------+----------------------------------------------+\n",
        "| MS_VERSION           | Float                              |       |         | MS format version                            |\n",
        "+----------------------+------------------------------------+-------+---------+----------------------------------------------+\n",
        "| *(SORT_COLUMNS)*     | String                             |       |         | Sort columns                                 |\n",
        "+----------------------+------------------------------------+-------+---------+----------------------------------------------+\n",
        "| *(SORT_ORDER)*       | String                             |       |         | Sort order                                   |\n",
        "+----------------------+------------------------------------+-------+---------+----------------------------------------------+\n",
        "| *Key*                |                                    |       |         |                                              |\n",
        "+----------------------+------------------------------------+-------+---------+----------------------------------------------+\n",
        "| TIME                 | Double                             | s     | EPOCH   | Integration midpoint                         |\n",
        "+----------------------+------------------------------------+-------+---------+----------------------------------------------+\n",
        "| *(TIME_EXTRA_PREC)*  | Double                             | s     |         | extra TIME precision                         |\n",
        "+----------------------+------------------------------------+-------+---------+----------------------------------------------+\n",
        "| ANTENNA1             | Int                                |       |         | First antenna                                |\n",
        "+----------------------+------------------------------------+-------+---------+----------------------------------------------+\n",
        "| ANTENNA2             | Int                                |       |         | Second antenna                               |\n",
        "+----------------------+------------------------------------+-------+---------+----------------------------------------------+\n",
        "| *(ANTENNA3)*         | Int                                |       |         | Third antenna                                |\n",
        "+----------------------+------------------------------------+-------+---------+----------------------------------------------+\n",
        "| FEED1                | Int                                |       |         | Feed on ANTENNA1                             |\n",
        "+----------------------+------------------------------------+-------+---------+----------------------------------------------+\n",
        "| FEED2                | Int                                |       |         | Feed on ANTENNA2                             |\n",
        "+----------------------+------------------------------------+-------+---------+----------------------------------------------+\n",
        "| *(FEED3)*            | Int                                |       |         | Feed on ANTENNA3                             |\n",
        "+----------------------+------------------------------------+-------+---------+----------------------------------------------+\n",
        "| DATA_DESC_ID         | Int                                |       |         | Data desc. id.                               |\n",
        "+----------------------+------------------------------------+-------+---------+----------------------------------------------+\n",
        "| PROCESSOR_ID         | Int                                |       |         | Processor id.                                |\n",
        "+----------------------+------------------------------------+-------+---------+----------------------------------------------+\n",
        "| *(PHASE_ID)*         | Int                                |       |         | Phase id.                                    |\n",
        "+----------------------+------------------------------------+-------+---------+----------------------------------------------+\n",
        "| FIELD_ID             | Int                                |       |         | Field id.                                    |\n",
        "+----------------------+------------------------------------+-------+---------+----------------------------------------------+\n",
        "| *Non-key attributes* |                                    |       |         |                                              |\n",
        "+----------------------+------------------------------------+-------+---------+----------------------------------------------+\n",
        "| INTERVAL             | Double                             | s     |         | Sampling interval                            |\n",
        "+----------------------+------------------------------------+-------+---------+----------------------------------------------+\n",
        "| EXPOSURE             | Double                             | s     |         | The effective integration time               |\n",
        "+----------------------+------------------------------------+-------+---------+----------------------------------------------+\n",
        "| TIME_CENTROID        | Double                             | s     | EPOCH   | Time centroid                                |\n",
        "+----------------------+------------------------------------+-------+---------+----------------------------------------------+\n",
        "| *(PULSAR_BIN)*       | Int                                |       |         | Pulsar bin number                            |\n",
        "+----------------------+------------------------------------+-------+---------+----------------------------------------------+\n",
        "| *(PULSAR_GATE_ID)*   | Int                                |       |         | Pulsar gate id.                              |\n",
        "+----------------------+------------------------------------+-------+---------+----------------------------------------------+\n",
        "| SCAN_NUMBER          | Int                                |       |         | Scan number                                  |\n",
        "+----------------------+------------------------------------+-------+---------+----------------------------------------------+\n",
        "| ARRAY_ID             | Int                                |       |         | Subarray number                              |\n",
        "+----------------------+------------------------------------+-------+---------+----------------------------------------------+\n",
        "| OBSERVATION_ID       | Int                                |       |         | Observation id.                              |\n",
        "+----------------------+------------------------------------+-------+---------+----------------------------------------------+\n",
        "| STATE_ID             | Int                                |       |         | State id.                                    |\n",
        "+----------------------+------------------------------------+-------+---------+----------------------------------------------+\n",
        "| *(BASELINE_REF)*     | Bool                               |       |         | Reference antenna                            |\n",
        "+----------------------+------------------------------------+-------+---------+----------------------------------------------+\n",
        "| UVW                  | Double(3)                          | m     | UVW     | UVW coordinates                              |\n",
        "+----------------------+------------------------------------+-------+---------+----------------------------------------------+\n",
        "| *(UVW2)*             | Double(3)                          | m     | UVW     | UVW (baseline 2)                             |\n",
        "+----------------------+------------------------------------+-------+---------+----------------------------------------------+\n",
        "| *Data*               |                                    |       |         |                                              |\n",
        "+----------------------+------------------------------------+-------+---------+----------------------------------------------+\n",
        "| *(DATA)*             | Complex(*N*~c~, *N*~f~)            |       |         | Complex visibility matrix (synthesis arrays) |\n",
        "+----------------------+------------------------------------+-------+---------+----------------------------------------------+\n",
        "| *(FLOAT_DATA)*       | Float(*N*~c~, *N*~f~)              |       |         | Float data matrix (single dish)              |\n",
        "+----------------------+------------------------------------+-------+---------+----------------------------------------------+\n",
        "| *(VIDEO_POINT)*      | Complex(*N*~c~)                    |       |         | Video point                                  |\n",
        "+----------------------+------------------------------------+-------+---------+----------------------------------------------+\n",
        "| *(LAG_DATA)*         | Complex(*N*~c~, *N*~l~)            |       |         | Correlation function                         |\n",
        "+----------------------+------------------------------------+-------+---------+----------------------------------------------+\n",
        "| SIGMA                | Float(*N*~c~)                      |       |         | Estimated rms noise for single channel       |\n",
        "+----------------------+------------------------------------+-------+---------+----------------------------------------------+\n",
        "| *(SIGMA_SPECTRUM)*   | Float(*N*~c~, *N*~f~^\\*^)          |       |         | Estimated rms noise                          |\n",
        "+----------------------+------------------------------------+-------+---------+----------------------------------------------+\n",
        "| WEIGHT               | Float(*N*~c~)                      |       |         | Weight for whole data matrix                 |\n",
        "+----------------------+------------------------------------+-------+---------+----------------------------------------------+\n",
        "| *(WEIGHT_SPECTRUM)*  | Float(*N*~c~, *N*~f~^\\*^)          |       |         | Weight for each channel                      |\n",
        "+----------------------+------------------------------------+-------+---------+----------------------------------------------+\n",
        "| *Flag information*   |                                    |       |         |                                              |\n",
        "+----------------------+------------------------------------+-------+---------+----------------------------------------------+\n",
        "| FLAG                 | Bool(*N*~c~, *N*~f~^\\*^)           |       |         | Cumulative data flags                        |\n",
        "+----------------------+------------------------------------+-------+---------+----------------------------------------------+\n",
        "| FLAG_CATEGORY        | Bool(*N*~c~, *N*~f~^\\*^, *N*~cat~) |       |         | Flag categories                              |\n",
        "+----------------------+------------------------------------+-------+---------+----------------------------------------------+\n",
        "| FLAG_ROW             | Bool                               |       |         | The row flag                                 |\n",
        "+----------------------+------------------------------------+-------+---------+----------------------------------------------+\n",
        "\n",
        "**Notes:**   \n",
        "Note that *N*~l~= number of lags, *N*~c~= number of correlators, *N*~f~= number of frequency channels, and *N*~cat~= number of flag categories.\n",
        "\n",
        "- **MS_VERSION** - The MeasurementSet format revision number, expressed as ${major}_{revision}$ ${minor}_{revision}$. This version is 2.0.\n",
        "- **SORT_COLUMNS** - Sort indices, in the form ${index}_1$ ${index}_2$ $\\cdots$, for the underlying MS. A string containing \\\"NONE\\\" reflects no sort order. An example might be *SORT_COLUMNS=\\\"TIME ANTENNA1 ANTENNA2\\\"*, to indicate sorting in in time-baseline order.\n",
        "- **SORT_ORDER** - Sort order as either \\\"ASCENDING\\\" or \\\"DESCENDING\\\".\n",
        "- **TIME** - Mid-point (not centroid) of data interval. Time is provided in Modified Julian Date. The CASA/casacore reference epoch (0 time) for timestamps in MeasurementSets is the MJD epoch: 1858/11/17.\n",
        "- **TIME_EXTRA_PREC** - Extra time precision.\n",
        "- **ANTENNA n*** - Antenna number (≥ 0), and a direct index into the *ANTENNA* sub-table *rownr*. For *n* \\> 2, triple-product data are implied.\n",
        "- **FEED n*** - Feed number ≥0). For *n*\\> 2, triple-product data are implied.\n",
        "- **DATA_DESC_ID** - Data description identifier (≥0), and a direct index into the *DATA_DESCRIPTION* sub-table *rownr*.\n",
        "- **PROCESSOR_ID** - Processor indentifier (≥0), and a direct index into the *PROCESSOR* sub-table *rownr*.\n",
        "- **PHASE_ID** - Switching phase identifier (≥0)\n",
        "- **FIELD_ID** - Field identifier (≥0).\n",
        "- **INTERVAL** - Data sampling interval. This is the nominal data interval and does not include the effects of bad data or partial integration.\n",
        "- **EXPOSURE** - Effective data interval, including bad data and partial averaging.\n",
        "- **PULSAR_BIN** - Pulsar bin number for the data record. Pulsar data may be measured for a limited number of pulse phase bins. The pulse phase bins are described in the *PULSAR* sub-table and indexed by this bin number.\n",
        "- **PULSAR_GATE_ID** - Pulsar gate identifier (≥0), and a direct index into the *PULSAR_GATE* sub-table *rownr*.\n",
        "- **SCAN_NUMBER** - Arbitrary scan number to identify data taken in the same logical scan. Not required to be unique.\n",
        "- **ARRAY_ID** - Subarray identifier (≥0), which identifies data in separate subarrays.\n",
        "- **OBSERVATION_ID** - Observation identifier (≥0), which identifies data from separate observations.\n",
        "- **STATE_ID** - State identifier (≥0), which identifies information relating to active reference signals or loads.\n",
        "**BASELINE_REF** - Flag to indicate the original correlator reference antenna for baseline-based correlators (True for *ANTENNA1*; False for *ANTENNA2*).\n",
        "- **UVW** - *uvw* coordinates for the baseline from *ANTENNE2* to *ANTENNA1*, i.e. the baseline is equal to the difference POSITION2 - POSITION1. The UVW given are for the *TIME_CENTROID*, and correspond in general to the reference type for the *PHASE_DIR* of the relevant field. I.e. J2000 if the phase reference direction is given in J2000 coordinates. However, any known reference is valid. Note that the choice of baseline direction and UVW definition (*W* towards source direction; *V* in plane through source and system\\'s pole; *U* in direction of increasing longitude coordinate) also determines the sign of the phase of the recorded data.\n",
        "- **UVW2** - *uvw* coordinates for the baseline from *ANTENNE3* to *ANTENNA1* (triple-product data only), i.e. the baseline is equal to the difference POSITION3 - POSITION1. The UVW given are for the *TIME_CENTROID*, and correspond in general to the reference type for the *PHASE_DIR* of the relevant field. I.e. J2000 if the phase reference direction is given in J2000 coordinates. However, any known reference is valid. Note that the choice of baseline direction and UVW definition (*W* towards source direction; *V* in plane through source and system\\'s pole; *U* in direction of increasing longitude coordinate) also determines the sign of the phase of the recorded data.\n",
        "- **DATA, FLOAT_DATA, LAG_DATA** - At least one of these columns should be present in a given MeasurementSet. In special cases one or more could be present (e.g., single dish data used in synthesis imaging or a mix of auto and crosscorrelations on a multi-feed single dish). If only correlation functions are stored in the MS, then *N*~f~^\\*^ is the maximum number of lags (*N*~l~) specified in the LAG table for this LAG_ID. If both correlation functions and frequency spectra are stored in the same MS, then *N*~f~^\\*^ is the number of frequency channels, and the weight information refers to the frequency spectra only. The units for these columns (eg. \\'Jy\\') specify whether the data are in flux density units or correlation coefficients.\n",
        "- **VIDEO_POINT** - The video point for the spectrum, to allow the full reverse transform.\n",
        "- **SIGMA** - The estimated rms noise for a single channel, for each correlator.\n",
        "- **SIGMA_SPECTRUM** - The estimated rms noise for each channel.\n",
        "- **WEIGHT** - The weight for the whole data matrix for each correlator, as assigned by the correlator or processor.\n",
        "- **WEIGHT_SPECTRUM** - The weight for each channel in the data matrix, as assigned by the correlator or processor. The weight spectrum should be used in preference to the *WEIGHT*, when available.\n",
        "- **FLAG** - An array of Boolean values with the same shape as DATA (see the DATA item above) representing the cumulative flags applying to this data matrix, as specified in *FLAG_CATEGORY*. Data are flagged bad if the FLAG array element is True.\n",
        "- **FLAG_CATEGORY** - An array of flag matrices with the same shape as DATA, but indexed by category. The category identifiers are specified by a keyword CATEGORY, containing an array of string identifiers, attached to the FLAG_CATEGORY column and thus shared by all rows in the MeasurementSet. The cumulative effect of these flags is reflected in column FLAG. Data are flagged bad if the FLAG array element is True. See Section 3.1.8 for further details.\n",
        "- **FLAG_ROW** - True if the entire row is flagged.\n",
        "\n",
        "\n",
        "***\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vwgRuHI1Vczp"
      },
      "source": [
        "### ANTENNA: Antenna Characteristics\n",
        "\n",
        "+---------------------+-----------+-------+------------------------------------------------------------------------------------------------------+----------------------------------------------+\n",
        "| Name                | Format    | Units | Measure                                                                                              | Comments                                     |\n",
        "+=====================+===========+=======+======================================================================================================+==============================================+\n",
        "| **Columns**         |           |       |                                                                                                      |                                              |\n",
        "+---------------------+-----------+-------+------------------------------------------------------------------------------------------------------+----------------------------------------------+\n",
        "| *Data*              |           |       |                                                                                                      |                                              |\n",
        "+---------------------+-----------+-------+------------------------------------------------------------------------------------------------------+----------------------------------------------+\n",
        "| NAME                | String    |       |                                                                                                      | Antenna name                                 |\n",
        "+---------------------+-----------+-------+------------------------------------------------------------------------------------------------------+----------------------------------------------+\n",
        "| STATION             | String    |       |                                                                                                      | Station name                                 |\n",
        "+---------------------+-----------+-------+------------------------------------------------------------------------------------------------------+----------------------------------------------+\n",
        "| TYPE                | String    |       |                                                                                                      | Antenna type                                 |\n",
        "+---------------------+-----------+-------+------------------------------------------------------------------------------------------------------+----------------------------------------------+\n",
        "| MOUNT               | String    |       | Mount type:alt-az, equatorial, X-Y, orbiting, bizarre                                                |                                              |\n",
        "+---------------------+-----------+-------+------------------------------------------------------------------------------------------------------+----------------------------------------------+\n",
        "| POSITION            | Double(3) | m     | POSITION                                                                                             | Antenna X,Y,Z phase reference positions      |\n",
        "+---------------------+-----------+-------+------------------------------------------------------------------------------------------------------+----------------------------------------------+\n",
        "| OFFSET              | Double(3) | m     | POSITION                                                                                             | Axes offset of mount to FEED REFERENCE point |\n",
        "+---------------------+-----------+-------+------------------------------------------------------------------------------------------------------+----------------------------------------------+\n",
        "| DISH_DIAMETER       | Double    | m     |                                                                                                      | Diameter of dish                             |\n",
        "+---------------------+-----------+-------+------------------------------------------------------------------------------------------------------+----------------------------------------------+\n",
        "| *(ORBIT_ID)*        | Int       |       |                                                                                                      | Orbit id.                                    |\n",
        "+---------------------+-----------+-------+------------------------------------------------------------------------------------------------------+----------------------------------------------+\n",
        "| *(MEAN_ORBIT)*      | Double(6) |       |                                                                                                      | Mean Keplerian elements                      |\n",
        "+---------------------+-----------+-------+------------------------------------------------------------------------------------------------------+----------------------------------------------+\n",
        "| *(PHASED_ARRAY_ID)* | Int       |       |                                                                                                      | Phased array id.                             |\n",
        "+---------------------+-----------+-------+------------------------------------------------------------------------------------------------------+----------------------------------------------+\n",
        "| Flag information    |           |       |                                                                                                      |                                              |\n",
        "+---------------------+-----------+-------+------------------------------------------------------------------------------------------------------+----------------------------------------------+\n",
        "| FLAG_ROW            | Bool      |       |                                                                                                      | Row flag                                     |\n",
        "+---------------------+-----------+-------+------------------------------------------------------------------------------------------------------+----------------------------------------------+\n",
        "\n",
        "**Notes:**   \n",
        "This sub-table contains the global antenna properties for each antenna in the MS. It is indexed directly from MAIN via ANTENNA*n*.\n",
        "  \n",
        "- **NAME** - Antenna name (e.g. \\\"NRAO_140\\\")\n",
        "- **STATION** - Station name (e.g. \\\"GREENBANK\\\")\n",
        "- **TYPE** - Antenna type. Reserved keywords include: (\\\"GROUND-BASED\\\" - conventional antennas; \\\"SPACE-BASED\\\" - orbiting antennas; \\\"TRACKING-STN\\\" - tracking stations).\n",
        "- **MOUNT** - Mount type of the antenna. Reserved keywords include: (\\\"EQUATORIAL\\\" - equatorial mount; \\\"ALT-AZ\\\" - azimuth-elevation mount; \\\"X-Y\\\" - x-y mount; \\\"SPACE-HALCA\\\" - specific orientation model.)\n",
        "- **POSITION** - In a right-handed frame, X towards the intersection of the equator and the Greenwich meridian, Z towards the pole. The exact frame should be specified in the MEASURE_REFERENCE keyword (ITRF or WGS84). The reference point is the point on the az or ha axis closest to the el or dec axis.\n",
        "- **OFFSET** - Axes offset of mount to feed reference point.\n",
        "- **DISH_DIAMETER** - Nominal diameter of dish, as opposed to the effective diameter.\n",
        "- **ORBIT_ID** - Orbit identifier. Index used in ORBIT sub-table if ANTENNA_TYPE is \\\"SPACE_BASED\\\".\n",
        "- **MEAN_ORBIT** - Mean Keplerian orbital elements, using the standard convention (Flatters 1998):\n",
        "    -   **0:** Semi-major axis of orbit (*a*) in *m*.\n",
        "    -   **1:** Ellipticity of orbit (*e*).\n",
        "    -   **2:** Inclination of orbit to the celestial equator (*i*) in *deg*.\n",
        "    -   **3:** Right ascension of the ascending node (Ω) in *deg*.\n",
        "    -   **4:** Argument of perigee (ω ) in *deg*.\n",
        "    -   **5:** Mean anomaly (*M*) in *deg*.\n",
        "\n",
        "- **PHASED_ARRAY_ID** - Phased array identifier. Points to a *PHASED_ARRAY* sub-table which points back to multiple entries in the *ANTENNA* sub-table and contains information on how they are combined.\n",
        "- **FLAG_ROW** - Boolean flag to indicate the validity of this entry. Set to True for an invalid row. This does not imply any flagging of the data in MAIN, but is necessary as the *ANTENNA* index in MAIN points directly into the *ANTENNA* sub-table. Thus *FLAG_ROW* can be used to delete an antenna entry without re-ordering the ANTENNA indices throughout the MS.\n",
        "\n",
        "\n",
        "\n",
        "***\n",
        "\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oTMTJtbkVczq"
      },
      "source": [
        "### DATA_DESCRIPTION: Data Description Table\n",
        "\n",
        "+--------------------+--------+-------+---------+---------------------+\n",
        "| Name               | Format | Units | Measure | Comments            |\n",
        "+====================+========+=======+=========+=====================+\n",
        "| **Columns**        |        |       |         |                     |\n",
        "+--------------------+--------+-------+---------+---------------------+\n",
        "| *Data*             |        |       |         |                     |\n",
        "+--------------------+--------+-------+---------+---------------------+\n",
        "| SPECTRAL_WINDOW_ID | Int    |       |         | Spectral window id. |\n",
        "+--------------------+--------+-------+---------+---------------------+\n",
        "| POLARIZATION_ID    | Int    |       |         | Polarization id.    |\n",
        "+--------------------+--------+-------+---------+---------------------+\n",
        "| *(LAG_ID)*         | Int    |       |         | Lag fn. id.         |\n",
        "+--------------------+--------+-------+---------+---------------------+\n",
        "| *Flags*            |        |       |         |                     |\n",
        "+--------------------+--------+-------+---------+---------------------+\n",
        "| FLAG_ROW           | Bool   |       |         | Row flag.           |\n",
        "+--------------------+--------+-------+---------+---------------------+\n",
        "\n",
        "**Notes:**   \n",
        "This table define the shape of the associated DATA array in MAIN, and in indexed directly by DATA_DESC_ID.\n",
        "\n",
        "- **SPECTRAL_WINDOW_ID** - Spectral window identifier.\n",
        "- **POLARIZATION_ID** - Polarization identifier (≥0); direct index into the *POLARIZATION* sub-table.\n",
        "- **LAG_ID** - Lag function identifier (≥0), and a direct index into the *LAG* sub-table *rownr*.\n",
        "- **FLAG_ROW** - True if the row does not contain valid data; does not imply flagging in MAIN.\n",
        "\n",
        "***\n",
        "\n",
        "\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "10ONxI5TVczq"
      },
      "source": [
        "### DOPPLER: Doppler Tracking Information\n",
        "\n",
        "+---------------+--------+-------+---------+---------------------------------------+\n",
        "| Name          | Format | Units | Measure | Comments                              |\n",
        "+===============+========+=======+=========+=======================================+\n",
        "| **Columns**   |        |       |         |                                       |\n",
        "+---------------+--------+-------+---------+---------------------------------------+\n",
        "| *Key*         |        |       |         |                                       |\n",
        "+---------------+--------+-------+---------+---------------------------------------+\n",
        "| DOPPLER_ID    | Int    |       |         | Doppler tracking id.                  |\n",
        "+---------------+--------+-------+---------+---------------------------------------+\n",
        "| SOURCE_ID     | Int    |       |         | Source id.                            |\n",
        "+---------------+--------+-------+---------+---------------------------------------+\n",
        "| *Data*        |        |       |         |                                       |\n",
        "+---------------+--------+-------+---------+---------------------------------------+\n",
        "| TRANSITION_ID | Int    |       |         | Transition id.                        |\n",
        "+---------------+--------+-------+---------+---------------------------------------+\n",
        "| VELDEF        | Double | m/s   | Doppler | Velocity definition of Doppler shift. |\n",
        "+---------------+--------+-------+---------+---------------------------------------+\n",
        "\n",
        "**Notes:**   \n",
        "This sub-table contains frame information for different Doppler tracking modes. It is indexed from the SPECTRAL_WINDOW_ID sub-table (with SOURCE_ID as a secondary index) and thus allows the specification of a source-dependent Doppler tracking reference for each SPECTRAL_WINDOW. This model allows multiple possible transitions per source per spectral window, but only one reference at any given time.\n",
        "\n",
        "- **DOPPLER_ID** - Doppler identifier, as used in the *SPECTRAL_WINDOW* sub-table.\n",
        "- **SOURCE_ID** - Source identifier (as used in the *SOURCE* sub-table).\n",
        "- **TRANSITION_ID** - This index selects the appropriate line from the list of transitions stored for each SOURCE_ID in the *SOURCE* table.\n",
        "- **VELDEF** - Velocity definition of the Doppler shift, e.g., RADIO or OPTICAL velocity in m/s.\n",
        "\n",
        "\n",
        "***\n",
        "\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qchJAautVczr"
      },
      "source": [
        "### FEED: Feed Characteristics\n",
        "\n",
        "+--------------------+----------------------------------------+-------+-----------+-----------------------------------------------------------------------+\n",
        "| Name               | Format                                 | Units | Measure   | Comments                                                              |\n",
        "+====================+========================================+=======+===========+=======================================================================+\n",
        "| **Columns**        |                                        |       |           |                                                                       |\n",
        "+--------------------+----------------------------------------+-------+-----------+-----------------------------------------------------------------------+\n",
        "| *Key*              |                                        |       |           |                                                                       |\n",
        "+--------------------+----------------------------------------+-------+-----------+-----------------------------------------------------------------------+\n",
        "| ANTENNA_ID         | Int                                    |       |           | Antenna id                                                            |\n",
        "+--------------------+----------------------------------------+-------+-----------+-----------------------------------------------------------------------+\n",
        "| FEED_ID            | Int                                    |       |           | Feed id                                                               |\n",
        "+--------------------+----------------------------------------+-------+-----------+-----------------------------------------------------------------------+\n",
        "| SPECTRAL_WINDOW_ID | Int                                    |       |           | Spectral window id.                                                   |\n",
        "+--------------------+----------------------------------------+-------+-----------+-----------------------------------------------------------------------+\n",
        "| TIME               | Double                                 | s     | EPOCH     | Interval midpoint                                                     |\n",
        "+--------------------+----------------------------------------+-------+-----------+-----------------------------------------------------------------------+\n",
        "| INTERVAL           | Double                                 | s     |           | Time interval                                                         |\n",
        "+--------------------+----------------------------------------+-------+-----------+-----------------------------------------------------------------------+\n",
        "| *Data description* |                                        |       |           |                                                                       |\n",
        "+--------------------+----------------------------------------+-------+-----------+-----------------------------------------------------------------------+\n",
        "| NUM_RECEPTORS      | Int                                    |       |           | receptors on this feed                                                |\n",
        "+--------------------+----------------------------------------+-------+-----------+-----------------------------------------------------------------------+\n",
        "| *Data*             |                                        |       |           |                                                                       |\n",
        "+--------------------+----------------------------------------+-------+-----------+-----------------------------------------------------------------------+\n",
        "| BEAM_ID            | Int                                    |       |           | Beam model                                                            |\n",
        "+--------------------+----------------------------------------+-------+-----------+-----------------------------------------------------------------------+\n",
        "| BEAM_OFFSET        | Double(2, NUM_RECEPTORS)               | rad   | DIRECTION | Beam position offset (on sky but in antenna reference frame).         |\n",
        "+--------------------+----------------------------------------+-------+-----------+-----------------------------------------------------------------------+\n",
        "| *(FOCUS_LENGTH)*   | Double                                 | m     |           | Focus length                                                          |\n",
        "+--------------------+----------------------------------------+-------+-----------+-----------------------------------------------------------------------+\n",
        "| *(PHASED_FEED_ID)* | Int                                    |       |           | Phased feed                                                           |\n",
        "+--------------------+----------------------------------------+-------+-----------+-----------------------------------------------------------------------+\n",
        "| POLARIZATION_TYPE  | String (NUM_RECEPTORS)                 |       |           | Type of polarization to which a given RECEPTOR responds.              |\n",
        "+--------------------+----------------------------------------+-------+-----------+-----------------------------------------------------------------------+\n",
        "| POL_RESPONSE       | Complex (NUM_RECEPTORS, NUM_RECEPTORS) |       |           | Feed polzn. response                                                  |\n",
        "+--------------------+----------------------------------------+-------+-----------+-----------------------------------------------------------------------+\n",
        "| POSITION           | Double(3)                              | m     | POSITION  | Position of feed relative to feed reference position for this antenna |\n",
        "+--------------------+----------------------------------------+-------+-----------+-----------------------------------------------------------------------+\n",
        "| RECEPTOR_ANGLE     | Double (NUM_RECEPTORS)                 | rad   |           | The reference angle for polarization.                                 |\n",
        "+--------------------+----------------------------------------+-------+-----------+-----------------------------------------------------------------------+\n",
        "\n",
        "**Notes:**\n",
        "A feed is a collecting element on an antenna, such as a single horn, that shares joint physical properties and makes sense to calibrate as a single entity. It is an abstraction of a generic antenna feed and is considered to have one or more RECEPTORs that respond to different polarization states. A FEED may have a time-variable beam and polarization response. Feeds are numbered from 0 on each separate antenna for each SPECTRAL_WINDOW_ID. Consequently, FEED_ID should be non-zero only in the case of feed arrays, i.e. multiple, simultaneous beams on the sky at the same frequency and polarization.\n",
        "\n",
        "- **ANTENNA_ID** - Antenna number, as indexed from ANTENNA*n* in MAIN.\n",
        "- **FEED_ID** - Feed identifier, as indexed from FEED*n* in MAIN.\n",
        "- **SPECTRAL_WINDOW_ID** - Spectral window identifier. A value of -1 indicates the row is valid for all spectral windows.\n",
        "- **TIME** - Mid-point of time interval for which the feed parameters in this row are valid. The same Measure reference used for the TIME column in MAIN must be used.\n",
        "- **INTERVAL** - Time interval.\n",
        "- **NUM_RECEPTORS** - Number of receptors on this feed. See *POLARIZATION_TYPE* for further information.\n",
        "- **BEAM_ID** - Beam identifier. Points to an optional BEAM sub-table defining the primary beam and polarization response for this *FEED*. A value of -1 indicates that no associated beam response is defined.\n",
        "- **BEAM_OFFSET** - Beam position offset, as defined on the sky but in the antenna reference frame.\n",
        "- **FOCUS_LENGTH** - Focus length. As defined along the optical axis of the antenna.\n",
        "- **PHASED_FEED_ID** - Phased feed identifier. Points to a *PHASED_FEED* sub-table which in turn points back to multiple entries in the *FEED* table, and specifies the manner in which they are combined.\n",
        "- **POLARIZATION_TYPE** - Polarization type to which each receptor responds (e.g. \\\"R\\\",\\\"L\\\",\\\"X\\\" or \\\"Y\\\"). This is the receptor polarization type as recorded in the final correlated data (e.g. \\\"RR\\\"); i.e. as measured after all polarization combiners.\n",
        "- **POL_RESPONSE** - Polarization response at the center of the beam for this feed. Expressed in a linearly polarized basis ($ \\textbf{\\vec e_x}$, $ \\textbf{\\vec e_y}$) using the IEEE convention.\n",
        "- **POSITION** - Offset of feed relative to the feed reference position for this antenna (see ANTENNA sub-table).\n",
        "- **RECEPTOR_ANGLE** - Polarization reference angle. Converts into parallactic angle in the sky domain.\n",
        "\n",
        "\n",
        "***\n",
        "\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "X8WIvFFnVczr"
      },
      "source": [
        "### FIELD: Field Positions for Each Source\n",
        "\n",
        "+------------------+-----------------------+-------+-----------+------------------------------------------+\n",
        "| Name             | Format                | Units | Measure   | Comments                                 |\n",
        "+==================+=======================+=======+===========+==========================================+\n",
        "| **Columns**      |                       |       |           |                                          |\n",
        "+------------------+-----------------------+-------+-----------+------------------------------------------+\n",
        "| *Key*            |                       |       |           |                                          |\n",
        "+------------------+-----------------------+-------+-----------+------------------------------------------+\n",
        "| *Data*           |                       |       |           |                                          |\n",
        "+------------------+-----------------------+-------+-----------+------------------------------------------+\n",
        "| NAME             | String                |       |           | Name of field                            |\n",
        "+------------------+-----------------------+-------+-----------+------------------------------------------+\n",
        "| CODE             | String                |       |           | Special characteristics of field         |\n",
        "+------------------+-----------------------+-------+-----------+------------------------------------------+\n",
        "| TIME             | Double                | s     | EPOCH     | Time origin for the directions and rates |\n",
        "+------------------+-----------------------+-------+-----------+------------------------------------------+\n",
        "| NUM_POLY         | Int                   |       |           | Series order                             |\n",
        "+------------------+-----------------------+-------+-----------+------------------------------------------+\n",
        "| DELAY_DIR        | Double(2, NUM_POLY+1) | rad   | DIRECTION | Direction of delay center.               |\n",
        "+------------------+-----------------------+-------+-----------+------------------------------------------+\n",
        "| PHASE_DIR        | Double(2, NUM_POLY+1) | rad   | DIRECTION | Phase center.                            |\n",
        "+------------------+-----------------------+-------+-----------+------------------------------------------+\n",
        "| REFERENCE_DIR    | Double(2, NUM_POLY+1) | rad   | DIRECTION | Reference center                         |\n",
        "+------------------+-----------------------+-------+-----------+------------------------------------------+\n",
        "| SOURCE_ID        | Int                   |       |           | Index in Source table                    |\n",
        "+------------------+-----------------------+-------+-----------+------------------------------------------+\n",
        "| *(EPHEMERIS_ID)* | Int                   |       |           | Ephemeris id.                            |\n",
        "+------------------+-----------------------+-------+-----------+------------------------------------------+\n",
        "| Flags            |                       |       |           |                                          |\n",
        "+------------------+-----------------------+-------+-----------+------------------------------------------+\n",
        "| FLAG_ROW         | Bool                  |       |           | Row flag                                 |\n",
        "+------------------+-----------------------+-------+-----------+------------------------------------------+\n",
        "\n",
        "**Notes:**   \n",
        "The *FIELD* table defines a field position on the sky. For interferometers, this is the correlated field position. For single dishes, this is the nominal pointing direction.\n",
        "\n",
        "- **NAME** - Field name; user specified.\n",
        "- **CODE** - Field code indicating special characteristics of the field; user specified.\n",
        "- **TIME** - Time reference for the directions and rates. Required to use the same TIME Measure reference as in MAIN.\n",
        "- **NUM_POLY** - Series order for the \\*\\_DIR columns.\n",
        "- **DELAY_DIR** - Direction of delay center; can be expressed as a polynomial in time. Final result converted to the defined Direction Measure type.\n",
        "- **PHASE_DIR** - Direction of phase center; can be expressed as a polynomial in time. Final result converted to the defined Direction Measure type.\n",
        "- **REFERENCE_DIR** - Reference center; can be expressed as a polynomial in time. Final result converted to the defined Direction Measure type. Used in single-dish to record the associated reference direction if position-switching has already been applied. For interferometric data, this is the original correlated field center, and may equal *DELAY_DIR* or *PHASE_DIR*.\n",
        "- **SOURCE_ID** - Points to an entry in the optional *SOURCE* subtable, a value of -1 indicates there is no corresponding source defined.\n",
        "- **EPHEMERIS_ID** - Points to an entry in the *EPHEMERIS* sub-table, which defines the ephemeris used to compute the field position. Useful for moving, near-field objects, where the ephemeris may be revised over time.\n",
        "- **FLAG_ROW** - True if data in this row are invalid, else False. Does not imply flagging in MAIN.\n",
        "\n",
        "***\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9K0eLPHTVczs"
      },
      "source": [
        "### FLAG_CMD: Flag Commands\n",
        "\n",
        "+-------------+--------+-------+---------+-------------------------+\n",
        "| Name        | Format | Units | Measure | Comments                |\n",
        "+=============+========+=======+=========+=========================+\n",
        "| **Columns** |        |       |         |                         |\n",
        "+-------------+--------+-------+---------+-------------------------+\n",
        "| *Key*       |        |       |         |                         |\n",
        "+-------------+--------+-------+---------+-------------------------+\n",
        "| TIME        | Double | s     | EPOCH   | Mid-point of interval   |\n",
        "+-------------+--------+-------+---------+-------------------------+\n",
        "| INTERVAL    | Double | s     |         | Time interval           |\n",
        "+-------------+--------+-------+---------+-------------------------+\n",
        "| *Data*      |        |       |         |                         |\n",
        "+-------------+--------+-------+---------+-------------------------+\n",
        "| TYPE        | String |       |         | FLAG or UNFLAG          |\n",
        "+-------------+--------+-------+---------+-------------------------+\n",
        "| REASON      | String |       |         | Flag reason             |\n",
        "+-------------+--------+-------+---------+-------------------------+\n",
        "| LEVEL       | Int    |       |         | Flag level              |\n",
        "+-------------+--------+-------+---------+-------------------------+\n",
        "| SEVERITY    | Int    |       |         | Severity code           |\n",
        "+-------------+--------+-------+---------+-------------------------+\n",
        "| APPLIED     | Bool   |       |         | True if applied in MAIN |\n",
        "+-------------+--------+-------+---------+-------------------------+\n",
        "| COMMAND     | String |       |         | Flag command            |\n",
        "+-------------+--------+-------+---------+-------------------------+\n",
        "\n",
        "**Notes:**   \n",
        "The *FLAG_CMD* sub-table defines global flagging commands which apply to the data in MAIN, as described in Section 3.1.8.\n",
        "\n",
        "- **TIME** - Mid-point of the time interval to which this flagging command applies. Required to use the same TIME Measure reference as used in *MAIN*.\n",
        "- **INTERVAL** - Time interval.\n",
        "- **TYPE** - Type of flag command, representing either a flagging (\\\"FLAG\\\") or un-flagging (\\\"UNFLAG\\\") operation.\n",
        "- **REASON** - Flag reason; user specified.\n",
        "- **LEVEL** - Flag level (≥0); reflects different revisions of flags which have the same *REASON*.\n",
        "- **SEVERITY** - Severity code for the flag, on a scale of 0-10 in order of increasing severity; user specified.\n",
        "- **APPLIED** - True if this flag has been applied to *MAIN*, and update in *FLAG_CATEGORY* and *FLAG*. False if this flag has not been applied to *MAIN*.\n",
        "- **COMMAND** - Global flag command, expressed in the standard syntax for data selection, as adopted within the project as a whole.\n",
        "\n",
        "\n",
        "***\n",
        "\n",
        "\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RQXHZnNyVczs"
      },
      "source": [
        "### FREQ_OFFSET: Frequency Offset Information\n",
        "\n",
        "+--------------------+--------+-------+---------+---------------------+\n",
        "| Name               | Format | Units | Measure | Comments            |\n",
        "+====================+========+=======+=========+=====================+\n",
        "| **Columns**        |        |       |         |                     |\n",
        "+--------------------+--------+-------+---------+---------------------+\n",
        "| *Key*              |        |       |         |                     |\n",
        "+--------------------+--------+-------+---------+---------------------+\n",
        "| ANTENNA1           | Int    |       |         | Antenna 1.          |\n",
        "+--------------------+--------+-------+---------+---------------------+\n",
        "| ANTENNA2           | Int    |       |         | Antenna 2.          |\n",
        "+--------------------+--------+-------+---------+---------------------+\n",
        "| FEED_ID            | Int    |       |         | Feed id.            |\n",
        "+--------------------+--------+-------+---------+---------------------+\n",
        "| SPECTRAL_WINDOW_ID | Int    |       |         | Spectral window id. |\n",
        "+--------------------+--------+-------+---------+---------------------+\n",
        "| TIME               | Double | s     | EPOCH   | Interval midpoint   |\n",
        "+--------------------+--------+-------+---------+---------------------+\n",
        "| INTERVAL           | Double | s     |         | Time interval       |\n",
        "+--------------------+--------+-------+---------+---------------------+\n",
        "| *Data*             |        |       |         |                     |\n",
        "+--------------------+--------+-------+---------+---------------------+\n",
        "| OFFSET             | Double | Hz    |         | Frequency offset    |\n",
        "+--------------------+--------+-------+---------+---------------------+\n",
        "\n",
        "**Notes:**   \n",
        "The table contains frequency offset information, to be added directly to the defined frequency labeling in the *SPECTRAL_WINDOW* sub-table as a Measure offset. This allows bands with small, time-variable, ad hoc frequency offsets to be labeled as the same *SPECTRAL_WINDOW_ID*, and calibrated together if required.\n",
        "\n",
        "- **ANTENNA n*** - Antenna identifier, as indexed from *ANTENNAn* in *MAIN*.\n",
        "- **FEED_ID** - Antenna identifier, as indexed from *FEEDn* in *MAIN*.\n",
        "- **SPECTRAL_WINDOW_ID** - Spectral window identifier.\n",
        "- **TIME** - Mid-point of the time interval for which this offset is valid. Required to use the same TIME Measure reference as used in *MAIN*.\n",
        "- **INTERVAL** - Time interval.\n",
        "- **OFFSET** - Frequency offset to be added to the frequency axis for this spectral window, as defined in the SPECTRAL_WINDOW sub-table. Required to have the same Frequency Measure reference as CHAN_FREQ in that table.\n",
        "\n",
        "***\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Nb4KEyDAVczt"
      },
      "source": [
        "### HISTORY: History Information\n",
        "\n",
        "+----------------+------------+-------+---------+-----------------------------+\n",
        "| Name           | Format     | Units | Measure | Comments                    |\n",
        "+================+============+=======+=========+=============================+\n",
        "| **Columns**    |            |       |         |                             |\n",
        "+----------------+------------+-------+---------+-----------------------------+\n",
        "| *Key*          |            |       |         |                             |\n",
        "+----------------+------------+-------+---------+-----------------------------+\n",
        "| TIME           | Double     | s     | EPOCH   | Time-stamp for message      |\n",
        "+----------------+------------+-------+---------+-----------------------------+\n",
        "| OBSERVATION_ID | Int        |       |         | Points to OBSERVATION table |\n",
        "+----------------+------------+-------+---------+-----------------------------+\n",
        "| *Data*         |            |       |         |                             |\n",
        "+----------------+------------+-------+---------+-----------------------------+\n",
        "| MESSAGE        | String     |       |         | Log message                 |\n",
        "+----------------+------------+-------+---------+-----------------------------+\n",
        "| PRIORITY       | String     |       |         | Message priority            |\n",
        "+----------------+------------+-------+---------+-----------------------------+\n",
        "| ORIGIN         | String     |       |         | Code origin                 |\n",
        "+----------------+------------+-------+---------+-----------------------------+\n",
        "| OBJECT_ID      | String     |       |         | Originating ObjectID        |\n",
        "+----------------+------------+-------+---------+-----------------------------+\n",
        "| APPLICATION    | String     |       |         | Application name            |\n",
        "+----------------+------------+-------+---------+-----------------------------+\n",
        "| CLI_COMMAND    | String(\\*) |       |         | CLI command sequence        |\n",
        "+----------------+------------+-------+---------+-----------------------------+\n",
        "| APP_PARAMS     | String(\\*) |       |         | Application paramters       |\n",
        "+----------------+------------+-------+---------+-----------------------------+\n",
        "\n",
        "**Notes:**   \n",
        "This sub-table contains associated history information for the MS.\n",
        "\n",
        "- **TIME** - Time-stamp for the history record. Required to have the same TIME Measure reference as used in *MAIN*.\n",
        "- **OBSERVATION_ID** - Observation identifier (see the *OBSERVATION* table)\n",
        "- **MESSAGE** - Log message.\n",
        "- **PRIORITY** - Message priority, with allowed types: (\\\"DEBUGGING\\\", \\\"WARN\\\", \\\"NORMAL\\\", or \\\"SEVERE\\\").\n",
        "- **ORIGIN** - Source code origin from which message originated.\n",
        "- **OBJECT_ID** - Originating ObjectID, if available, else blank.\n",
        "- **APPLICATION** - Application name.\n",
        "- **CLI_COMMAND** - CLI command sequence invoking the application.\n",
        "- **APP_PARAMS** - Application parameter values, in the adopted project-wide format.\n",
        "\n",
        "\n",
        "***\n",
        "\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0j3oeyWNVczt"
      },
      "source": [
        "### OBSERVATION: Observation Information\n",
        "\n",
        "+----------------+------------+-------+---------+--------------------------------+\n",
        "| Name           | Format     | Units | Measure | Comments                       |\n",
        "+================+============+=======+=========+================================+\n",
        "| **Columns**    |            |       |         |                                |\n",
        "+----------------+------------+-------+---------+--------------------------------+\n",
        "| *Data*         |            |       |         |                                |\n",
        "+----------------+------------+-------+---------+--------------------------------+\n",
        "| TELESCOPE_NAME | String     |       |         | Telescope name                 |\n",
        "+----------------+------------+-------+---------+--------------------------------+\n",
        "| TIME_RANGE     | Double(2)  | s     | EPOCH   | Start, end times               |\n",
        "+----------------+------------+-------+---------+--------------------------------+\n",
        "| OBSERVER       | String     |       |         | Name of observer(s)            |\n",
        "+----------------+------------+-------+---------+--------------------------------+\n",
        "| LOG            | String(\\*) |       |         | Observing log                  |\n",
        "+----------------+------------+-------+---------+--------------------------------+\n",
        "| SCHEDULE_TYPE  | String     |       |         | Schedule type                  |\n",
        "+----------------+------------+-------+---------+--------------------------------+\n",
        "| SCHEDULE       | String(\\*) |       |         | Project schedule               |\n",
        "+----------------+------------+-------+---------+--------------------------------+\n",
        "| PROJECT        | String     |       |         | Project identification string. |\n",
        "+----------------+------------+-------+---------+--------------------------------+\n",
        "| RELEASE_DATE   | Double     | s     | EPOCH   | Target release date            |\n",
        "+----------------+------------+-------+---------+--------------------------------+\n",
        "| *Flags*        |            |       |         |                                |\n",
        "+----------------+------------+-------+---------+--------------------------------+\n",
        "| FLAG_ROW       | Bool       |       |         | Row flag.                      |\n",
        "+----------------+------------+-------+---------+--------------------------------+\n",
        "\n",
        "**Notes:**   \n",
        "This table contains information specifying the observing instrument or epoch. See the discussion in Section 3.3 for details. It is indexed directly from *MAIN* via *OBSERVATION_ID*.\n",
        "\n",
        "- **TELESCOPE_NAME** - Telescope name (e.g. \\\"WSRT\\\" or \\\"VLBA\\\").\n",
        "- **TIME_RANGE** - The start and end times of the overall observing period spanned by the actual recorded data in *MAIN*. Required to use the same TIME Measure reference as in *MAIN*. Time is provided in Modified Julian Date. The CASA/casacore reference epoch (0 time) for timestamps in MeasurementSets is the MJD epoch: 1858/11/17.\n",
        "- **OBSERVER** - The name(s) of the observer(s).\n",
        "- **LOG** - The observing log, as supplied by the telescope or instrument.\n",
        "- **SCHEDULE_TYPE** - The schedule type, with current reserved types (\\\"VLBA-CRD\\\", \\\"VEX\\\", \\\"WSRT\\\", \\\"ATNF\\\").\n",
        "- **SCHEDULE** - Unmodified schedule file, of the type specified, and as used by the instrument.\n",
        "- **PROJECT** - Project code (e.g. \\\"BD46\\\")\n",
        "- **RELEASE_DATE** - Project release date. This is the date on which the data may become public.\n",
        "- **FLAG_ROW** - Row flag. True if data in this row is invalid, but does not imply any flagging in *MAIN*.\n",
        "\n",
        "\n",
        "***\n",
        "\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DcxfNX7eVczu"
      },
      "source": [
        "### POINTING: Antenna Pointing Information\n",
        "\n",
        "+-----------------------+-----------------------+-------+-----------+------------------------------+\n",
        "| Name                  | Format                | Units | Measure   | Comments                     |\n",
        "+=======================+=======================+=======+===========+==============================+\n",
        "| **Columns**           |                       |       |           |                              |\n",
        "+-----------------------+-----------------------+-------+-----------+------------------------------+\n",
        "| *Key*                 |                       |       |           |                              |\n",
        "+-----------------------+-----------------------+-------+-----------+------------------------------+\n",
        "| ANTENNA_ID            | Int                   |       |           | Antenna id.                  |\n",
        "+-----------------------+-----------------------+-------+-----------+------------------------------+\n",
        "| TIME                  | Double                | s     | EPOCH     | Interval midpoint            |\n",
        "+-----------------------+-----------------------+-------+-----------+------------------------------+\n",
        "| INTERVAL              | Double                | s     |           | Time interval                |\n",
        "+-----------------------+-----------------------+-------+-----------+------------------------------+\n",
        "| *Data*                |                       |       |           |                              |\n",
        "+-----------------------+-----------------------+-------+-----------+------------------------------+\n",
        "| NAME                  | String                |       |           | Pointing position desc.      |\n",
        "+-----------------------+-----------------------+-------+-----------+------------------------------+\n",
        "| NUM_POLY              | Int                   |       |           | Series order                 |\n",
        "+-----------------------+-----------------------+-------+-----------+------------------------------+\n",
        "| TIME_ORIGIN           | Double                | s     | EPOCH     | Origin for the polynomial    |\n",
        "+-----------------------+-----------------------+-------+-----------+------------------------------+\n",
        "| DIRECTION             | Double(2, NUM_POLY+1) | rad   | DIRECTION | Antenna pointing direction   |\n",
        "+-----------------------+-----------------------+-------+-----------+------------------------------+\n",
        "| TARGET                | Double(2, NUM_POLY+1) | rad   | DIRECTION | Target direction             |\n",
        "+-----------------------+-----------------------+-------+-----------+------------------------------+\n",
        "| *(POINTING_OFFSET)*   | Double(2, NUM_POLY+1) | rad   | DIRECTION | A priori pointing correction |\n",
        "+-----------------------+-----------------------+-------+-----------+------------------------------+\n",
        "| *(SOURCE_OFFSET)*     | Double(2, NUM_POLY+1) | rad   | DIRECTION | Offset from source           |\n",
        "+-----------------------+-----------------------+-------+-----------+------------------------------+\n",
        "| *(ENCODER)*           | Double(2)             | rad   | DIRECTION | Encoder values               |\n",
        "+-----------------------+-----------------------+-------+-----------+------------------------------+\n",
        "| *(POINTING_MODEL_ID)* | Int                   |       |           | Pointing model id.           |\n",
        "+-----------------------+-----------------------+-------+-----------+------------------------------+\n",
        "| TRACKING              | Bool                  |       |           | True if on-position          |\n",
        "+-----------------------+-----------------------+-------+-----------+------------------------------+\n",
        "| *(ON_SOURCE)*         | Bool                  |       |           | True if on-source            |\n",
        "+-----------------------+-----------------------+-------+-----------+------------------------------+\n",
        "| *(OVER_THE_TOP)*      | Bool                  |       |           | True if over the top         |\n",
        "+-----------------------+-----------------------+-------+-----------+------------------------------+\n",
        "\n",
        "**Notes:**   \n",
        "This table contains information concerning the primary pointing direction of each antenna as a function of time. Note that the pointing offsets for inidividual feeds on a given antenna are specified in the *FEED* sub-table with respect to this pointing direction.\n",
        "\n",
        "- **ANTENNA_ID** - Antenna identifier, as specified by *ANTENNAn* in *MAIN*.\n",
        "- **TIME** - Mid-point of the time interval for which the information in this row is valid. Required to use the same TIME Measure reference as in *MAIN*.\n",
        "- **INTERVAL** - Time interval.\n",
        "- **NAME** - Pointing direction name; user specified.\n",
        "- **NUM_POLY** - Series order for the polynomial expressions in *DIRECTION* and *POINTING_OFFSET*.\n",
        "- **TIME_ORIGIN** - Time origin for the polynomial expansions.\n",
        "- **DIRECTION** - Antenna pointing direction, optionally expressed as polynomial coefficients. The final result is interpreted as a Direction Measure using the specified Measure reference.\n",
        "- **TARGET** - Target pointing direction, optionally expressed as polynomial coefficients. The final result is interpreted as a Direction Measure using the specified Measure reference. This is the true expected position of the source, including all coordinate corrections such as precession, nutation etc.\n",
        "- **POINTING_OFFSET** - The a priori pointing corrections applied by the telescope in pointing to the *DIRECTION* position, optionally expressed as polynomial coefficients. The final result is interpreted as a Direction Measure using the specified Measure reference.\n",
        "- **SOURCE_OFFSET** - The commanded offset from the source position, if offset pointing is being used.\n",
        "- **ENCODER** - The current encoder values on the primary axes of the mount type for the antenna, expressed as a Direction Measure.\n",
        "- **TRACKING** - True if tracking the nominal pointing position.\n",
        "- **ON-SOURCE** - True if the nominal pointing direction coincides with the source, i.e. offset-pointing is not being used.\n",
        "- **OVER-THE-TOP** - True if the antenna was driven to this position \\\"over the top\\\" (az-el mount).\n",
        "\n",
        "\n",
        "***\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0lRRHvneVczu"
      },
      "source": [
        "### POLARIZATION: Polarization Setup Information\n",
        "\n",
        "+----------------------------+------------------+-------+---------+------------------------------+\n",
        "| Name                       | Format           | Units | Measure | Comments                     |\n",
        "+============================+==================+=======+=========+==============================+\n",
        "| **Columns**                |                  |       |         |                              |\n",
        "+----------------------------+------------------+-------+---------+------------------------------+\n",
        "| *Data description columns* |                  |       |         |                              |\n",
        "+----------------------------+------------------+-------+---------+------------------------------+\n",
        "| NUM_CORR                   | Int              |       |         | correlations                 |\n",
        "+----------------------------+------------------+-------+---------+------------------------------+\n",
        "| *Data*                     |                  |       |         |                              |\n",
        "+----------------------------+------------------+-------+---------+------------------------------+\n",
        "| CORR_TYPE                  | Int(NUM_CORR)    |       |         | Polarization of correlation  |\n",
        "+----------------------------+------------------+-------+---------+------------------------------+\n",
        "| CORR_PRODUCT               | Int(2, NUM_CORR) |       |         | Receptor cross-products      |\n",
        "+----------------------------+------------------+-------+---------+------------------------------+\n",
        "| *Flags*                    |                  |       |         |                              |\n",
        "+----------------------------+------------------+-------+---------+------------------------------+\n",
        "| FLAG_ROW                   | Bool             |       |         | Row flag                     |\n",
        "+----------------------------+------------------+-------+---------+------------------------------+\n",
        "\n",
        "**Notes:**   \n",
        "This table defines the polarization labeling of the *DATA* array in *MAIN*, and is directly indexed from the *DATA_DESCRIPTION* table via *POLARIZATION_ID*.\n",
        "\n",
        "- **NUM_CORR**- The number of correlation polarization products. For example, for (RR) this value would be 1, for (RR, LL) it would be 2, and for (XX,YY,XY,YX) it would be 4, etc.\n",
        "- **CORR_TYPE** - An integer for each correlation product indicating the Stokes type as defined in the Stokes class enumeration.\n",
        "- **CORR_PRODUCT** - Pair of integers for each correlation product, specifying the receptors from which the signal originated. The receptor polarization is defined in the *POLARIZATION_TYPE* column in the *FEED* table. An example would be (0,0), (0,1), (1,0), (1,1) to specify all correlations between two receptors.\n",
        "- **FLAG_ROW** - Row flag. True is the data in this row are not valid, but does not imply the flagging of any *DATA* in *MAIN*.\n",
        "\n",
        "***\n",
        "\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dtUa4cj_Vczv"
      },
      "source": [
        "### PROCESSOR: Processor Information\n",
        "\n",
        "+-------------+--------+-------+---------+-----------------------+\n",
        "| Name        | Format | Units | Measure | Comments              |\n",
        "+=============+========+=======+=========+=======================+\n",
        "| **Columns** |        |       |         |                       |\n",
        "+-------------+--------+-------+---------+-----------------------+\n",
        "| *Data*      |        |       |         |                       |\n",
        "+-------------+--------+-------+---------+-----------------------+\n",
        "| TYPE        | String |       |         | Processor type        |\n",
        "+-------------+--------+-------+---------+-----------------------+\n",
        "| SUB_TYPE    | String |       |         | Processor sub-type    |\n",
        "+-------------+--------+-------+---------+-----------------------+\n",
        "| TYPE_ID     | Int    |       |         | Processor type id.    |\n",
        "+-------------+--------+-------+---------+-----------------------+\n",
        "| MODE_ID     | Int    |       |         | Processor mode id.    |\n",
        "+-------------+--------+-------+---------+-----------------------+\n",
        "| *(PASS_ID)* | Int    |       |         | Processor pass number |\n",
        "+-------------+--------+-------+---------+-----------------------+\n",
        "| *Flags*     |        |       |         |                       |\n",
        "+-------------+--------+-------+---------+-----------------------+\n",
        "| FLAG_ROW    | Bool   |       |         | Row flag              |\n",
        "+-------------+--------+-------+---------+-----------------------+\n",
        "\n",
        "**Notes:**   \n",
        "This table holds summary information for the back-end processing device used to generate the basic data in the MAIN table. Such devices include correlators, radiometers, spectrometers, pulsar-timers, amongst others. See Section 4.0.4 for further details.\n",
        "\n",
        "- **TYPE** - Processor type; reserved keywords include (\\\"CORRELATOR\\\" - interferometric correlator; \\\"SPECTROMETER\\\" - single-dish correlator; \\\"RADIOMETER\\\" - generic detector/integrator; \\\"PULSAR-TIMER\\\" - pulsar timing device).\n",
        "- **SUB_TYPE** - Processor sub-type, e.g. \\\"GBT\\\" or \\\"JIVE\\\".\n",
        "- **TYPE_ID** - Index used in a specialized sub-table named as *subtype_type*, which contains time-independent processor information applicable to the current data record (e.g. a JIVE_CORRELATOR sub-table). Time-dependent information for each device family is contained in other tables, dependent on the device type.\n",
        "- **MODE_ID** - Index used in a specialized sub-table named as *subtype_type_mode*, containing information on the processor mode applicable to the current data record. (e.g. a *GBT_SPECTROMETER_MODE* sub-table).\n",
        "- **PASS_ID** - Pass identifier; this is used to distinguish data records produced by multiple passes through the same device, where this is possible (e.g. VLBI correlators). Used as an index into the associated table containing pass information.\n",
        "- **FLAG_ROW** - Row flag. True if data in the row is not valid, but does not imply flagging in *MAIN*.\n",
        "\n",
        "\n",
        "***\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "214QedT0Vczv"
      },
      "source": [
        "### SOURCE: Source Information\n",
        "\n",
        "+--------------------+-------------------+-------+-----------------+---------------------------------------------------------------+\n",
        "| Name               | Format            | Units | Measure         | Comments                                                      |\n",
        "+====================+===================+=======+=================+===============================================================+\n",
        "| **Columns**        |                   |       |                 |                                                               |\n",
        "+--------------------+-------------------+-------+-----------------+---------------------------------------------------------------+\n",
        "| *Key*              |                   |       |                 |                                                               |\n",
        "+--------------------+-------------------+-------+-----------------+---------------------------------------------------------------+\n",
        "| SOURCE_ID          | Int               |       |                 | Source id                                                     |\n",
        "+--------------------+-------------------+-------+-----------------+---------------------------------------------------------------+\n",
        "| TIME               | Double            | s     | EPOCH           | Midpoint of time for which this set of parameters is accurate |\n",
        "+--------------------+-------------------+-------+-----------------+---------------------------------------------------------------+\n",
        "| INTERVAL           | Double            | s     |                 | Interval                                                      |\n",
        "+--------------------+-------------------+-------+-----------------+---------------------------------------------------------------+\n",
        "| SPECTRAL_WINDOW_ID | Int               |       |                 | Spectral Window id                                            |\n",
        "+--------------------+-------------------+-------+-----------------+---------------------------------------------------------------+\n",
        "| *Data description* |                   |       |                 |                                                               |\n",
        "+--------------------+-------------------+-------+-----------------+---------------------------------------------------------------+\n",
        "| NUM_LINES          | Int               |       |                 | Number of spectral lines                                      |\n",
        "+--------------------+-------------------+-------+-----------------+---------------------------------------------------------------+\n",
        "| *Data*             |                   |       |                 |                                                               |\n",
        "+--------------------+-------------------+-------+-----------------+---------------------------------------------------------------+\n",
        "| NAME               | String            |       |                 | Name of source as given during observations                   |\n",
        "+--------------------+-------------------+-------+-----------------+---------------------------------------------------------------+\n",
        "| CALIBRATION_GROUP  | Int               |       |                 | grouping for calibration purpose                              |\n",
        "+--------------------+-------------------+-------+-----------------+---------------------------------------------------------------+\n",
        "| CODE               | String            |       |                 | Special characteristics of source, e.g. Bandpass calibrator   |\n",
        "+--------------------+-------------------+-------+-----------------+---------------------------------------------------------------+\n",
        "| DIRECTION          | Double(2)         | rad   | DIRECTION       | Direction (e.g. RA, DEC)                                      |\n",
        "+--------------------+-------------------+-------+-----------------+---------------------------------------------------------------+\n",
        "| *(POSITION)*       | Double(3)         | m     | POSITION        | Position (e.g. for solar system objects)                      |\n",
        "+--------------------+-------------------+-------+-----------------+---------------------------------------------------------------+\n",
        "| PROPER_MOTION      | Double(2)         | rad/s |                 | Proper motion                                                 |\n",
        "+--------------------+-------------------+-------+-----------------+---------------------------------------------------------------+\n",
        "| *(TRANSITION)*     | String(NUM_LINES) |       |                 | Transition name                                               |\n",
        "+--------------------+-------------------+-------+-----------------+---------------------------------------------------------------+\n",
        "| *(REST_FREQUENCY)* | Double(NUM_LINES) | Hz    | FREQUENCY       | Line rest frequency                                           |\n",
        "+--------------------+-------------------+-------+-----------------+---------------------------------------------------------------+\n",
        "| *(SYSVEL)*         | Double(NUM_LINES) | m/s   | RADIAL VELOCITY | Systemic velocity at reference                                |\n",
        "+--------------------+-------------------+-------+-----------------+---------------------------------------------------------------+\n",
        "| *(SOURCE_MODEL)*   | TableRecord       |       |                 | Default csm                                                   |\n",
        "+--------------------+-------------------+-------+-----------------+---------------------------------------------------------------+\n",
        "| *(PULSAR_ID)*      | Int               |       |                 | Pulsar id.                                                    |\n",
        "+--------------------+-------------------+-------+-----------------+---------------------------------------------------------------+\n",
        "\n",
        "**Notes:**   \n",
        "This table contains time-variable source information, optionally associated with a given FIELD_ID.    \n",
        "\n",
        "- **SOURCE_ID** - Source identifier (≥ 0), as specified in the *FIELD* sub-table.\n",
        "- **TIME** - Mid-point of the time interval for which the data in this row is valid. Required to use the same TIME Measure reference as in *MAIN*.\n",
        "- **INTERVAL** - Time interval.\n",
        "- **SPECTRAL_WINDOW_ID** - Spectral window identifier. A -1 indicates that the row is valid for all spectral windows.\n",
        "- **NUM_LINES** - Number of spectral line transitions associated with this source and spectral window id. combination.\n",
        "- **NAME** - Source name; user specified.\n",
        "- **CALIBRATION_GROUP** - Calibration group number to which this source belongs; user specified.\n",
        "- **CODE** - Source code, used to describe any special characteristics f the source, such as the nature of a calibrator. Reserved keyword, including (\"BANDPASS CAL\").\n",
        "- **DIRECTION** - Source direction at this TIME.\n",
        "- **POSITION** - Source position (*x*, *y*, *z*) at this TIME (for near-field objects).\n",
        "- **PROPER_MOTION** - Source proper motion at this TIME.\n",
        "- **TRANSITION** - Transition names applicable for this spectral window (e.g. \\\"v=1, J=1-0, SiO\\\").\n",
        "- **REST_FREQUENCY** - Rest frequencies for the transitions.\n",
        "- **SYSVEL** - Systemic velocity for each transition.\n",
        "- **SOURCE_MODEL** - Reference to an assigned component source model table.\n",
        "- **PULSAR_ID** - An index used in the *PULSAR* sub-table to define further pulsar-specific properties if the source is a pulsar.\n",
        "\n",
        " ***\n",
        "\n",
        "\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "teMOgfJ1Vczw"
      },
      "source": [
        "### SPECTRAL_WINDOW: Spectral Window Description\n",
        "\n",
        "+----------------------------+------------------+-------+-----------+---------------------------------------------------------+\n",
        "| Name                       | Format           | Units | Measure   | Comments                                                |\n",
        "+============================+==================+=======+===========+=========================================================+\n",
        "| **Columns**                |                  |       |           |                                                         |\n",
        "+----------------------------+------------------+-------+-----------+---------------------------------------------------------+\n",
        "| *Data description columns* |                  |       |           |                                                         |\n",
        "+----------------------------+------------------+-------+-----------+---------------------------------------------------------+\n",
        "| NUM_CHAN                   | Int              |       |           | spectral channels                                       |\n",
        "+----------------------------+------------------+-------+-----------+---------------------------------------------------------+\n",
        "| *Data*                     |                  |       |           |                                                         |\n",
        "+----------------------------+------------------+-------+-----------+---------------------------------------------------------+\n",
        "| NAME                       | String           |       |           | Spectral window name                                    |\n",
        "+----------------------------+------------------+-------+-----------+---------------------------------------------------------+\n",
        "| REF_FREQUENCY              | Double           | Hz    | FREQUENCY | The reference frequency.                                |\n",
        "+----------------------------+------------------+-------+-----------+---------------------------------------------------------+\n",
        "| CHAN_FREQ                  | Double(NUM_CHAN) | Hz    | FREQUENCY | Center frequencies for each channel in the data matrix. |\n",
        "+----------------------------+------------------+-------+-----------+---------------------------------------------------------+\n",
        "| CHAN_WIDTH                 | Double(NUM_CHAN) | Hz    |           | Channel width for each channel in the data matrix.      |\n",
        "+----------------------------+------------------+-------+-----------+---------------------------------------------------------+\n",
        "| MEAS_FREQ_REF              | Int              |       |           | FREQUENCY Measure ref.                                  |\n",
        "+----------------------------+------------------+-------+-----------+---------------------------------------------------------+\n",
        "| EFFECTIVE_BW               | Double(NUM_CHAN) | Hz    |           | The effective noise bandwidth of each spectral channel  |\n",
        "+----------------------------+------------------+-------+-----------+---------------------------------------------------------+\n",
        "| RESOLUTION                 | Double(NUM_CHAN) | Hz    |           | The effective spectral resolution of each channel       |\n",
        "+----------------------------+------------------+-------+-----------+---------------------------------------------------------+\n",
        "| TOTAL_BANDWIDTH            | Double           | Hz    |           | total bandwidth for this window                         |\n",
        "+----------------------------+------------------+-------+-----------+---------------------------------------------------------+\n",
        "| NET_SIDEBAND               | Int              |       |           | Net sideband                                            |\n",
        "+----------------------------+------------------+-------+-----------+---------------------------------------------------------+\n",
        "| *(BBC_NO)*                 | Int              |       |           | Baseband converter no.                                  |\n",
        "+----------------------------+------------------+-------+-----------+---------------------------------------------------------+\n",
        "| *(BBC_SIDEBAND)*           | Int              |       |           | BBC sideband                                            |\n",
        "+----------------------------+------------------+-------+-----------+---------------------------------------------------------+\n",
        "| IF_CONV_CHAIN              | Int              |       |           | The IF conversion chain                                 |\n",
        "+----------------------------+------------------+-------+-----------+---------------------------------------------------------+\n",
        "| *(RECEIVER_ID)*            | Int              |       |           | Receiver id.                                            |\n",
        "+----------------------------+------------------+-------+-----------+---------------------------------------------------------+\n",
        "| FREQ_GROUP                 | Int              |       |           | Frequency group                                         |\n",
        "+----------------------------+------------------+-------+-----------+---------------------------------------------------------+\n",
        "| FREQ_GROUP_NAME            | String           |       |           | Freq. group name                                        |\n",
        "+----------------------------+------------------+-------+-----------+---------------------------------------------------------+\n",
        "| *(DOPPLER_ID)*             | Int              |       |           | Doppler id.                                             |\n",
        "+----------------------------+------------------+-------+-----------+---------------------------------------------------------+\n",
        "| *(ASSOC_SPW_ID)*           | Int(\\*)          |       |           | Associated spw_id.                                      |\n",
        "+----------------------------+------------------+-------+-----------+---------------------------------------------------------+\n",
        "| *(ASSOC_NATURE)*           | String(\\*)       |       |           | Nature of association                                   |\n",
        "+----------------------------+------------------+-------+-----------+---------------------------------------------------------+\n",
        "| Flags                      |                  |       |           |                                                         |\n",
        "+----------------------------+------------------+-------+-----------+---------------------------------------------------------+\n",
        "| FLAG_ROW                   | Bool             |       |           |                                                         |\n",
        "+----------------------------+------------------+-------+-----------+---------------------------------------------------------+\n",
        "\n",
        "\n",
        "**Notes:**   \n",
        "This table describes properties for each defined spectral window. A spectral window is both a frequency label for the associated DATA array in MAIN, but also represents a generic frequency conversion chain that shares joint physical properties and makes sense to calibrate as a single entity.\n",
        "\n",
        "- **NUM_CHAN** - Number of spectral channels.\n",
        "- **NAME** - Spectral window name; user specified.\n",
        "- **REF_FREQUENCY** - The reference frequency. A frequency representative of this spectral window, usually the sky frequency corresponding to the DC edge of the baseband. Used by the calibration system if a fixed scaling frequency is required or in algorithms to identify the observing band.\n",
        "- **CHAN_FREQ** - Center frequencies for each channel in the data matrix. These can be frequency-dependent, to accommodate instruments such as acousto-optical spectrometers. Note that the channel frequencies may be in ascending or descending frequency order.\n",
        "- **CHAN_WIDTH** - Nomical channel width of each spectral channel. Although these can be derived from *CHAN_FREQ* by differencing, it is more efficient to keep a separate reference to this information.\n",
        "- **MEAS_FREQ_REF** - Frequency Measure reference for *CHAN_FREQ*. This allows a row-based reference for this column in order to optimize the choice of Measure reference when Doppler tracking is used. Modified only by the MS access code.\n",
        "- **EFFECTIVE_BW** - The effective noise bandwidth of each spectral channel.\n",
        "- **RESOLUTION** - The effective spectral resolution of each channel.\n",
        "- **TOTAL_BANDWIDTH** - The total bandwidth for this spectral window.\n",
        "- **NET_SIDEBAND** - The net sideband for this spectral window.\n",
        "- **BBC_NO** - The baseband converter number, if applicable.\n",
        "- **BBC_SIDEBAND** - The baseband converter sideband, is applicable.\n",
        "- **IF_CONV_CHAIN** - Identification of the electronic signal path for the case of multiple (simultaneous) IFs. (e.g. VLA: AC=0, BD=1, ATCA: Freq1=0, Freq2=1)\n",
        "- **RECEIVER_ID** - Index used to identify the receiver associated with the spectral window. Further state information is planned to be stored in a RECEIVER sub-table.\n",
        "- **FREQ_GROUP** - The frequency group to which the spectral window belongs. This is used to associate spectral windows for joint calibration purposes.\n",
        "- **FREQ_GROUP_NAME** - The frequency group name; user specified.\n",
        "- **DOPPLER_ID** - The Doppler identifier defining frame information for this spectral window.\n",
        "- **ASSOC_SPW_ID** - Associated spectral windows, which are related in some fashion (e.g. \\\"channel-zero\\\").\n",
        "- **ASSOC_NATURE** - Nature of the association for ASSOC_SPW_ID; reserved keywords are (\\\"CHANNEL-ZERO\\\" - channel zero; \\\"EQUAL-FREQUENCY\\\" - same frequency labels; \\\"SUBSET\\\" - narrow-band subset).\n",
        "- **FLAG_ROW** - True if the row does not contain valid data.\n",
        "\n",
        "***\n",
        "\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7tqyeVS5Vczw"
      },
      "source": [
        "### STATE: State Information\n",
        "\n",
        "+-------------+--------+-------+------------+--------------------+\n",
        "| Name        | Format | Units | Measure    | Comments           |\n",
        "+=============+========+=======+============+====================+\n",
        "| **Columns** |        |       |            |                    |\n",
        "+-------------+--------+-------+------------+--------------------+\n",
        "| *Data*      |        |       |            |                    |\n",
        "+-------------+--------+-------+------------+--------------------+\n",
        "| SIG         | Bool   |       |            | Signal             |\n",
        "+-------------+--------+-------+------------+--------------------+\n",
        "| REF         | Bool   |       |            | Reference          |\n",
        "+-------------+--------+-------+------------+--------------------+\n",
        "| CAL         | Double | K     |            | Noise calibration  |\n",
        "+-------------+--------+-------+------------+--------------------+\n",
        "| LOAD        | Double | K     |            | Load temperature   |\n",
        "+-------------+--------+-------+------------+--------------------+\n",
        "| SUB_SCAN    | Int    |       |            | Sub-scan number    |\n",
        "+-------------+--------+-------+------------+--------------------+\n",
        "| OBS_MODE    | String |       |            | Observing mode     |\n",
        "+-------------+--------+-------+------------+--------------------+\n",
        "| *Flags*     |        |       |            |                    |\n",
        "+-------------+--------+-------+------------+--------------------+\n",
        "| FLAG_ROW    | Bool   |       |            | Row flag           |\n",
        "+-------------+--------+-------+------------+--------------------+\n",
        "\n",
        "**Notes:**   \n",
        "This table defines the state parameters for a particular data record as they refer to external loads, calibration sources or references, and also characterizes the observing mode of the data record, as an aid to defining the scheduling heuristics. It is indexed directly via STATE_ID in *MAIN*.\n",
        "\n",
        "- **SIG** - True if the source signal is being observed.\n",
        "- **REF** - True for a reference phase.\n",
        "- **CAL** - Noise calibration temperature (zero if not added).\n",
        "- **LOAD** - Load temperature (zero if no load).\n",
        "- **SUB_SCAN** - Sub-scan number (≥ 0), relative to the *SCAN_NUMBER* in MAIN. Used to identify observing sequences.\n",
        "- **OBS_MODE** - Observing mode; defined by a set of reserved keywords characterizing the current observing mode (e.g. \\\"OFF-SPECTRUM\\\"). Used to define the schedule strategy.\n",
        "- **FLAG_ROW** - True if the row does not contain valid data. Does not imply flagging in *MAIN*.\n",
        "\n",
        "***\n",
        "\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "a6ypqg-pxIPa"
      },
      "source": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "M5EomAHwVczx"
      },
      "source": [
        "### SYSCAL: System Calibration\n",
        "\n",
        "+------------------------+------------------------+-------+---------+---------------------------------------------------------------+\n",
        "| Name                   | Format                 | Units | Measure | Comments                                                      |\n",
        "+========================+========================+=======+=========+===============================================================+\n",
        "| **Columns**            |                        |       |         |                                                               |\n",
        "+------------------------+------------------------+-------+---------+---------------------------------------------------------------+\n",
        "| *Key*                  |                        |       |         |                                                               |\n",
        "+------------------------+------------------------+-------+---------+---------------------------------------------------------------+\n",
        "| ANTENNA_ID             | Int                    |       |         | Antenna id                                                    |\n",
        "+------------------------+------------------------+-------+---------+---------------------------------------------------------------+\n",
        "| FEED_ID                | Int                    |       |         | Feed id                                                       |\n",
        "+------------------------+------------------------+-------+---------+---------------------------------------------------------------+\n",
        "| SPECTRAL_WINDOW_ID     | Int                    |       |         | Spectral window id                                            |\n",
        "+------------------------+------------------------+-------+---------+---------------------------------------------------------------+\n",
        "| TIME                   | Double                 | s     | EPOCH   | Midpoint of time for which this set of parameters is accurate |\n",
        "+------------------------+------------------------+-------+---------+---------------------------------------------------------------+\n",
        "| INTERVAL               | Double                 | s     |         | Interval                                                      |\n",
        "+------------------------+------------------------+-------+---------+---------------------------------------------------------------+\n",
        "| *Data*                 |                        |       |         |                                                               |\n",
        "+------------------------+------------------------+-------+---------+---------------------------------------------------------------+\n",
        "| *(PHASE_DIFF)*         | Float                  | rad   |         | Phase difference between receptor 0 and receptor 1            |\n",
        "+------------------------+------------------------+-------+---------+---------------------------------------------------------------+\n",
        "| *(TCAL)*               | Float (*N*~r~)         | K     |         | Calibration temp                                              |\n",
        "+------------------------+------------------------+-------+---------+---------------------------------------------------------------+\n",
        "| *(TRX)*                | Float (*N*~r~)         | K     |         | Receiver temperature                                          |\n",
        "+------------------------+------------------------+-------+---------+---------------------------------------------------------------+\n",
        "| *(TSKY)*               | Float (*N*~r~)         | K     |         | Sky temperature                                               |\n",
        "+------------------------+------------------------+-------+---------+---------------------------------------------------------------+\n",
        "| *(TSYS)*               | Float (*N*~r~)         | K     |         | System temp                                                   |\n",
        "+------------------------+------------------------+-------+---------+---------------------------------------------------------------+\n",
        "| *(TANT)*               | Float (*N*~r~)         | K     |         | Antenna temperature                                           |\n",
        "+------------------------+------------------------+-------+---------+---------------------------------------------------------------+\n",
        "| *(TANT_TSYS)*          | Float(*N*~r~)          |       |         | $ {{T_{ant}}\\over{T_{sys}}}$                                  |\n",
        "+------------------------+------------------------+-------+---------+---------------------------------------------------------------+\n",
        "| *(TCAL_SPECTRUM)*      | Float (*N*~r~, *N*~f~) | K     |         | Calibration temp                                              |\n",
        "+------------------------+------------------------+-------+---------+---------------------------------------------------------------+\n",
        "| *(TRX_SPECTRUM)*       | Float (*N*~r~, *N*~f~) | K     |         | Receiver temperature                                          |\n",
        "+------------------------+------------------------+-------+---------+---------------------------------------------------------------+\n",
        "| *(TSKY_SPECTRUM)*      | Float (*N*~r~, *N*~f~) | K     |         | Sky temperature spectrum                                      |\n",
        "+------------------------+------------------------+-------+---------+---------------------------------------------------------------+\n",
        "| *(TSYS_SPECTRUM)*      | Float (*N*~r~, *N*~f~) | K     |         | System temp                                                   |\n",
        "+------------------------+------------------------+-------+---------+---------------------------------------------------------------+\n",
        "| *(TANT_SPECTRUM)*      | Float (*N*~r~, *N*~f~) | K     |         | Antenna temperature spectrum                                  |\n",
        "+------------------------+------------------------+-------+---------+---------------------------------------------------------------+\n",
        "| *(TANT_TSYS_SPECTRUM)* | Float (*N*~r~,*N*~f~)  |       |         | $ {{T_{ant}}\\over{T_{sys}}}$ spectrum                         |\n",
        "+------------------------+------------------------+-------+---------+---------------------------------------------------------------+\n",
        "| *Flags*                |                        |       |         |                                                               |\n",
        "+------------------------+------------------------+-------+---------+---------------------------------------------------------------+\n",
        "| *(PHASE_DIFF_FLAG)*    | Bool                   |       |         | Flag for PHASE_DIFF                                           |\n",
        "+------------------------+------------------------+-------+---------+---------------------------------------------------------------+\n",
        "| *(TCAL_FLAG)*          | Bool                   |       |         | Flag for TCAL                                                 |\n",
        "+------------------------+------------------------+-------+---------+---------------------------------------------------------------+\n",
        "| *(TRX_FLAG)*           | Bool                   |       |         | Flag for TRX                                                  |\n",
        "+------------------------+------------------------+-------+---------+---------------------------------------------------------------+\n",
        "| *(TSKY_FLAG)*          | Bool                   |       |         | Flag for TSKY                                                 |\n",
        "+------------------------+------------------------+-------+---------+---------------------------------------------------------------+\n",
        "| *(TSYS_FLAG)*          | Bool                   |       |         | Flag for TSYS                                                 |\n",
        "+------------------------+------------------------+-------+---------+---------------------------------------------------------------+\n",
        "| *(TANT_FLAG)*          | Bool                   |       |         | Flag for TANT                                                 |\n",
        "+------------------------+------------------------+-------+---------+---------------------------------------------------------------+\n",
        "| *(TANT_TSYS_FLAG)*     | Bool                   |       |         | Flag for ${{T_{ant}}\\over{T_{sys}}}$                          |\n",
        "+------------------------+------------------------+-------+---------+---------------------------------------------------------------+\n",
        "\n",
        "**Notes:**   \n",
        "This table contains time-variable calibration measurements for each antenna, as indexed on feed and spectral window. Note that *N*~r~= number of receptors, and *N*~f~= number of frequency channels.\n",
        "\n",
        "- **ANTENNA_ID** - Antenna identifier, as indexed by *ANTENNAn* in *MAIN*.\n",
        "- **FEED_ID** - Feed identifier, as indexed by *FEEDn* in *MAIN*.\n",
        "- **SPECTRAL_WINDOW_ID** - Spectral window identifier.\n",
        "- **TIME** - Mid-point of the time interval for which the data in this row are valid. Required to use the same TIME Measure reference as that in *MAIN*.\n",
        "- **INTERVAL** - Time interval.\n",
        "- **PHASE_DIFF** - Phase difference between receptor 0 and receptor 1.\n",
        "- **TCAL** - Calibration temperature.\n",
        "- **TRX** - Receiver temperature.\n",
        "- **TSKY** - Sky temperature.\n",
        "- **TSYS** - System temperature.\n",
        "- **TANT** - Antenna temperature.\n",
        "- **TANT_TSYS** - Antenna temperature over system temperature.\n",
        "- **TCAL_SPECTRUM** - Calibration temperature spectrum.\n",
        "- **TRX_SPECTRUM** - Receiver temperature spectrum.\n",
        "- **TSKY_SPECTRUM** - Sky temperature spectrum.\n",
        "- **TSYS_SPECTRUM** - System temperature spectrum.\n",
        "- **TANT_SPECTRUM** - Antenna temperature spectrum.\n",
        "- **TANT_TSYS_SPECTRUM** - Antenna temperature over system temperature spectrum.\n",
        "- **PHASE_DIFF_FLAG** - True if *PHASE_DIFF* flagged.\n",
        "- **TCAL_FLAG** - True if *TCAL* flagged.\n",
        "- **TRX_FLAG** - True if *TRX* flagged.\n",
        "- **TSKY_FLAG** - True if *TSKY* flagged.\n",
        "- **TSYS_FLAG** - True if *TSYS* flagged.\n",
        "- **TANT_FLAG** - True if *TANT* flagged.\n",
        "- **TANT_TSYS_FLAG** - True if *TANT_TSYS* flagged.\n",
        "\n",
        "\n",
        "***\n",
        "\n",
        "\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bUVgMzF_Vczx"
      },
      "source": [
        "### WEATHER: Weather Station Information\n",
        "\n",
        "+-------------------------+--------+---------+------------+----------------------------------------+\n",
        "| Name                    | Format | Units   | Measure    | Comments                               |\n",
        "+=========================+========+=========+============+========================================+\n",
        "| **Columns**             |        |         |            |                                        |\n",
        "+-------------------------+--------+---------+------------+----------------------------------------+\n",
        "| *Key*                   |        |         |            |                                        |\n",
        "+-------------------------+--------+---------+------------+----------------------------------------+\n",
        "| ANTENNA_ID              | Int    |         |            | Antenna number                         |\n",
        "+-------------------------+--------+---------+------------+----------------------------------------+\n",
        "| TIME                    | Double | s       | EPOCH      | Mid-point of interval                  |\n",
        "+-------------------------+--------+---------+------------+----------------------------------------+\n",
        "| INTERVAL                | Double | s       |            | Interval over which data is relevant   |\n",
        "+-------------------------+--------+---------+------------+----------------------------------------+\n",
        "| *Data*                  |        |         |            |                                        |\n",
        "+-------------------------+--------+---------+------------+----------------------------------------+\n",
        "| *(H2O)*                 | Float  | *m*^-2^ |            | Average column density of water        |\n",
        "+-------------------------+--------+---------+------------+----------------------------------------+\n",
        "| *(IONOS_ELECTRON)*      | Float  | *m*^-2^ |            | Average column density of electrons    |\n",
        "+-------------------------+--------+---------+------------+----------------------------------------+\n",
        "| *(PRESSURE)*            | Float  | hPa     |            | Ambient atmospheric pressure           |\n",
        "+-------------------------+--------+---------+------------+----------------------------------------+\n",
        "| *(REL_HUMIDITY)*        | Float  |         |            | Ambient relative humidity              |\n",
        "+-------------------------+--------+---------+------------+----------------------------------------+\n",
        "| *(TEMPERATURE)*         | Float  | K       |            | Ambient air temperature for an antenna |\n",
        "+-------------------------+--------+---------+------------+----------------------------------------+\n",
        "| *(DEW_POINT)*           | Float  | K       |            | Dew point                              |\n",
        "+-------------------------+--------+---------+------------+----------------------------------------+\n",
        "| *(WIND_DIRECTION)*      | Float  | rad     |            | Average wind direction                 |\n",
        "+-------------------------+--------+---------+------------+----------------------------------------+\n",
        "| *(WIND_SPEED)*          | Float  | m/s     |            | Average wind speed                     |\n",
        "+-------------------------+--------+---------+------------+----------------------------------------+\n",
        "| *Flags*                 |        |         |            |                                        |\n",
        "+-------------------------+--------+---------+------------+----------------------------------------+\n",
        "| *(H2O_FLAG)*            | Bool   |         |            | Flag for H2O                           |\n",
        "+-------------------------+--------+---------+------------+----------------------------------------+\n",
        "| *(IONOS_ELECTRON_FLAG)* | Bool   |         |            | Flag for IONOS_ELECTRON                |\n",
        "+-------------------------+--------+---------+------------+----------------------------------------+\n",
        "| *(PRESSURE_FLAG)*       | Bool   |         |            | Flag for PRESSURE                      |\n",
        "+-------------------------+--------+---------+------------+----------------------------------------+\n",
        "| *(REL_HUMIDITY_FLAG)*   | Bool   |         |            | Flag for REL_HUMIDITY                  |\n",
        "+-------------------------+--------+---------+------------+----------------------------------------+\n",
        "| *(TEMPERATURE_FLAG)*    | Bool   |         |            | Flag for TEMPERATURE                   |\n",
        "+-------------------------+--------+---------+------------+----------------------------------------+\n",
        "| *(DEW_POINT_FLAG)*      | Bool   |         |            | Flag for DEW_POINT                     |\n",
        "+-------------------------+--------+---------+------------+----------------------------------------+\n",
        "| *(WIND_DIRECTION_FLAG)* | Bool   |         |            | Flag for WIND_DIRECTION                |\n",
        "+-------------------------+--------+---------+------------+----------------------------------------+\n",
        "| *(WIND_SPEED_FLAG)*     | Bool   |         |            | Flag for WIND_SPEED                    |\n",
        "+-------------------------+--------+---------+------------+----------------------------------------+\n",
        "\n",
        "**Notes:**   \n",
        "This table contains mean external atmosphere and weather information.\n",
        "\n",
        "- **ANTENNA_ID** - Antenna identifier, as indexed by *ANTENNAn* from *MAIN*.\n",
        "- **TIME** - Mid-point of the time interval over which the data in the row are valid. Required to use the same TIME Measure reference as in *MAIN*.\n",
        "- **INTERVAL** - Time interval.\n",
        "- **H2O** - Average column density of water.\n",
        "- **IONOS_ELECTRON** - Average column density of electrons.\n",
        "- **PRESSURE** - Ambient atmospheric pressure.\n",
        "- **REL_HUMIDITY** - Ambient relative humidity.\n",
        "- **TEMPERATURE** - Ambient air temperature.\n",
        "- **DEW_POINT** - Dew point temperature.\n",
        "- **WIND_DIRECTION** - Average wind direction.\n",
        "- **WIND_SPEED** - Average wind speed.\n",
        "- **H2O_FLAG** - Flag for H2O.\n",
        "- **IONOS_ELECTRON_FLAG** - Flag for IONOS_ELECTRON.\n",
        "- **PRESSURE_FLAG** - Flag for PRESSURE.\n",
        "- **REL_HUMIDITY_FLAG** - Flag for REL_HUMIDITY.\n",
        "- **TEMPERATURE_FLAG** - Flag for TEMPERATURE.\n",
        "- **DEW_POINT_FLAG** - Flag for DEW_POINT.\n",
        "- **WIND_DIRECTION_FLAG** - Flag for DEW_POINT.\n",
        "- **WIND_SPEED_FLAG** - Flag for DEW_POINT.\n",
        "\n",
        "\n",
        "***\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "F81tUDUDVczy"
      },
      "source": [
        "## Definition Synthesized Beam\n",
        "\n",
        "CASA uses the following zero-centered two dimensional elliptical Gaussian function or Gaussian beam:\n",
        "\n",
        "$$\\begin{align} f(x,y) &= A \\exp\\left\\lbrace -\\left(\\frac{4 \\ln(2)}{d_1^2} (\\cos(\\theta) x + \\sin(\\theta) y)^2 + \\frac{4 \\ln(2)}{d_2^2} (-\\sin(\\theta) x + \\cos(\\theta) y)^2 \\right)\\right\\rbrace, \\label{eq:1} \\end{align}$$\n",
        "\n",
        "where A is the amplitude (usually set to unity) and $\\theta$ is the anti-clockwise angle from the x axis to the line that lies along the greatest width of $f(x,y)$ (the line and the x axis must be coplanar). The factors $d_1$ and $d_2$ are respectively the semi-major and semi-minor axis of the ellipse, which is formed by the cross-section that lies parallel to the $x, y$ plane, at a height so that $d_1$ is equal to the FWHM (full width at half maximum) distance of the one dimensional Gaussian which lies on the plane formed by the $z$ axis and $d_1$. Note that $d_1 \\geqslant d_2 > 0$, since $d_1$ is the semi-major axis.\n",
        "\n",
        "![bd6ce30313654fc00a6d88e248e1ae0e864f1edf](https://github.com/casangi/casadocs/blob/master/docs/notebooks/media/bd6ce30313654fc00a6d88e248e1ae0e864f1edf.png?raw=1){.image-inline width=\"535\" height=\"436\"}\n",
        "\n",
        ">Figure 1: Surface plot of a two dimensional elliptical Gaussian with $A = 1$, $d_1 = 3$, $d_2=1$ and $\\theta = 30^\\circ$.\n",
        "\n",
        "![5cd15ffb3f39e942040ef35a229b1e5d3f11a9e1](https://github.com/casangi/casadocs/blob/master/docs/notebooks/media/5cd15ffb3f39e942040ef35a229b1e5d3f11a9e1.png?raw=1){.image-inline width=\"383\" height=\"312\"}\n",
        "\n",
        ">Figure 2: Cross-section parallel to the $x, y$  plane of the two dimensional elliptical Gaussian from Fig. 1, where the resulting ellipse has a semi-major and semi-minor axis equal to $d_1$ and $d_2$, respectively.\n",
        "\n",
        "![79bf91cfc2367dde815f4b998c57847420f15018](https://github.com/casangi/casadocs/blob/master/docs/notebooks/media/79bf91cfc2367dde815f4b998c57847420f15018.png?raw=1){.image-inline width=\"360\" height=\"294\"}\n",
        "\n",
        ">Figure 3: One dimensional Gaussian plot for $A = 1$, $y = 0$, $\\theta  = 0$ and $d_1 = 1 = FWHM$.\n",
        "\n",
        "For calculating the Fourier transform of the two dimensional elliptical Gaussian, the above Equation can be re-written by grouping the $x$ and $y$ terms:$$\\begin{align} f(x,y) &= A \\exp\\left[-\\left(\\alpha x^2 + \\beta y x + \\gamma y^2\\right)\\right], \\label{eq:eg_2} \\end{align}$$where$$\\begin{align} \\alpha &= 4 \\ln(2) \\left[ \\frac{\\cos^2(\\theta)}{d_1^2} +\\frac{ \\sin^2(\\theta)}{d_2^2} \\right], \\label{eq:a} \\\\ \\beta &= 8 \\ln(2) \\left[ \\frac{1}{d_1^2} - \\frac{1}{d_2^2} \\right] \\sin(\\theta) \\cos(\\theta) ,\\\\ \\gamma &= 4 \\ln(2) \\left[ \\frac{\\sin^2(\\theta)}{d_1^2} +\\frac{ \\cos^2(\\theta)}{d_2^2} \\right]. \\label{eq:g} \\end{align}$$\n",
        "\n",
        "Converting from $\\alpha$, $\\beta$, $\\gamma$ to $d_1$, $d_2$, $\\theta$ can be done using the following set of equations:$$\\begin{align} d_1 &= \\sqrt{\\frac{ 8 \\ln(2) }{ (\\alpha + \\gamma) - \\sqrt{\\alpha^2 - 2\\alpha\\gamma + \\gamma^2 + \\beta^2} }}, \\label{eq:d1} \\\\ d_2 &= \\sqrt{\\frac{ 8 \\ln(2) }{ (\\alpha + \\gamma) + \\sqrt{\\alpha^2 - 2\\alpha\\gamma + \\gamma^2 + \\beta^2} }}, \\label{eq:d2}\\\\ \\theta &= 0.5 {\\rm arctan2}(-\\beta,\\gamma-\\alpha). \\label{eq:t} \\end{align}$$\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "***\n",
        "\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Foyvuta5Vczz"
      },
      "source": [
        "## Working with MS Data\n",
        "\n",
        "The ALMA and VLA raw data are stored in their respective archives in the Astronomy Science Data Model (ASDM) format.  The definition of the format can be found [here](casa-fundamentals.ipynb#science-data-model).\n",
        "\n",
        "To bring them into CASA, the ASDMs are filled into a so-called [MeasurementSet](casa-fundamentals.ipynb#measurementset-v2) (or MS) (format description can be found [here](https://casa.nrao.edu/../Memos/229.html)). In its logical structure, the MS looks like a generalized description of data from any interferometric or single dish telescope. Physically, the MS consists of several tables in a directory on disk, in XML format.\n",
        "\n",
        "Tables in CASA are actually directories containing files that are the sub-tables. For example, when you create a MS called AM675.ms, then the name of the directory where all the tables are stored will be called AM675.ms/. See chapter \\\"[Visibility Data Import Export](visibilities_import_export.ipynb)\\\" for more information on MeasurementSet and Data Handling in CASA.\n",
        "\n",
        "The data that you originally get from a telescope can be put in any directory that is convenient to you. Once you \\\"fill\\\" the data into a MeasurementSet that can be accessed by CASA, it is generally best to keep that MS in the same directory where you started CASA so you can get access to it easily (rather than constantly having to specify a full path name).\n",
        "\n",
        "When you generate calibration solutions or images (again these are in table format), these will also be written to disk. It is a good idea to keep them in the directory in which you started CASA, too.\n",
        "\n",
        "**How do I get rid of my data in CASA?**\n",
        "\n",
        "Note that when you delete a MeasurementSet, calibration table, or image, which are in fact directories, you must delete this and all underlying directories and files. If you are not running CASA, this is most simply done by using the file delete method of the operating system from which you started CASA. For example, when running CASA on a Linux system, in order to delete the MeasurementSet named AM675.ms type\n",
        "\n",
        "```\n",
        "CASA <5>: !rm -r AM675.ms\n",
        "```\n",
        "\n",
        "from within CASA. The ! tells CASA that a system command follows, and the -r makes sure that all subdirectories are deleted recursively.\n",
        "\n",
        "It is convenient to prefix all MS, calibration tables, and output files produced in a run with a common string. For example, one might prefix all files from VLA project AM675 with AM675, e.g. AM675.ms, AM675.cal, AM675.clean. Then,\n",
        "\n",
        "```\n",
        "CASA <6>: !rm -r AM675*\n",
        "```\n",
        "\n",
        " will clean up all of these.\n",
        "\n",
        "In scripts, the ! escape to the OS will not work. Instead, use the **os.system**() function to do the same thing:\n",
        "\n",
        "```\n",
        "os.system('rm -r AM675*')\n",
        "```\n",
        "\n",
        "If you are within CASA, then the CASA system is keeping a cache of tables that you have been using and using the OS to delete them will confuse things. For example, running a script that contains *rm* commands multiple times will often not run or crash the second time as the cache gets confused. The clean way of removing CASA tables (MS, caltables, images) inside CASA is to use the **rmtables** task:\n",
        "\n",
        "```\n",
        "rmtables('AM675.ms')\n",
        "```\n",
        "\n",
        "and this can also be wildcarded (though you may get warnings if it tries to delete files or directories that fit the name wildcard that are not CASA tables).\n",
        "\n",
        "<div class=\"alert alert-warning\">\n",
        "**ALERT**: **rmtables** is the preferred way to remove data. **clean** is a good example where frequently data are left in the cache after deleting the output files via \"!rm -r\". Restarting **clean** then sometimes claims that the files still exist, even though they are not present on disk anymore. **rmtables** will completely remove the files on disks and all cached versions and restarting **clean** will work as intended.  \n",
        "</div>\n",
        "\n",
        "<div class=\"alert alert-warning\">\n",
        "**ALERT:** Some CASA processes lock the file and forget to give it up when they are done. You will get *WARNING* messages from **rmtables** and your script will probably crash second time around as the file isn't removed. The safest thing is still to exit CASA and start a new session for multiple runs.\n",
        "</div>\n",
        "\n",
        "\n",
        "\n",
        "**What's in my data?**\n",
        "\n",
        "The actual data is in a large *MAIN* table that is organized in such a way that you can access different parts of the data easily. This table contains a number of \"rows\", which are effectively a single timestamp for a single spectral window (like an IF from the VLA) and a single baseline (for an interferometer).\n",
        "\n",
        "There are a number of \"columns\" in the MS, the most important of which for our purposes is the DATA column --- this contains the original visibility data from when the MS was created or filled. There are other helpful \"scratch\" columns which hold useful versions of the data or weights for further processing: the CORRECTED_DATA column, which is used to hold calibrated data and an optional MODEL_DATA column, which may hold the Fourier inversion of a particular model image. The creation and use of the scratch columns is generally done behind the scenes, but you should be aware that they are there (and when they are used). We will occasionally refer to the rows and columns in the MS.\n",
        "\n",
        "\n",
        "\n",
        "***\n",
        "\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "O9mFc2F0Ip1D"
      },
      "source": [
        "## Combining Datasets\n",
        "Several tasks can work or are intended to work with the multiple MSes. These can be divided into two broad categories based on functionality: one processes multiple input MSes to produce a combined MS and one uses multiple MSes together as input for further processing such as imaging. For the former, **concat**, **virtualconcat**, and **testconcat** are available. For the latter, **tclean**, **sdintimaging**, **tsdimaging**, and **sdimaging** are the tasks support such input. These tasks accept the vis parameter as a list of strings to point to each input MS. The level of conformance that the input MSs need to be in for these tasks to handle them properly varies among the tasks. The table summarizes required conformance among the input MS datasets. While providing more uniform behavior across CASA tasks in handling the multiple MSs is on-going effort, currently each task mentioned above calls a common helper function, **check_mslist** to check any differences in all the columns in the MS tables and the tasks will report users through warning or error messages in the log when incompatitiblity is encountered. Also, whenever the chronological order of the list of the MS matters, sorting the list by chrological order is applied by calling **sort_mslist**.\n",
        "\n",
        "\n",
        "The table below summarizes how each of these tasks handles the list of MSes currently.\n",
        "\n",
        "\n",
        "\n",
        "task | WEGHT_SPECTRUM |\tSIGMA_SPECTRUM | MODEL |\tVirtual model |\tDATA/CORRECTED_DATA/FLOAT_DATA |\tSorting in time |\tSorting in frequency\n",
        "--- | --- | --- | --- | --- | --- | --- | ---\n",
        "**concat** | Warn about absence of the column. After sorting the input MSs by start time, the first MS determines <br>if the output MS has the column or not.  <br>No data will be filled in the cells corresponding to MSs with no WEIGHT_SPECTRUM (i.e. no initialization) | As for WEIGHT_SPECTRUM |If MODEL_DATA is present only in some MSes, <br>it fills with 1 for other MSes |No warning if all MSes have virtual model only. <br> The SOURCE_MODEL column will be ignored.<br> If at least one has MODEL_DATA, <br>there is warning about the virtual model in <br>source table will be lost. |If CORRECTED_DATA is present only in some MSs,<br> it fills with DATA for other MSes.<br> | The list of input MSs is always<br>sorted by start time before processing.\t| |\n",
        "**virtualconcat** | Warn about absence of the column. The behavior is same as concat. | | Warn about absence of the column. |\t|\t | Same as concat |\n",
        "**testconcat** | Warn about absence of the column. Th behavior is the same as concat | | Warn about absence of the column | | | No sorting |\n",
        "**tclean** | Warn about absence of the column. <br>For MSes without WEIGHT_SPECTRUM, <br> the column will be added and filled with <br>uniform values across channels according to <br> WEIGHT using <br>cb.initweights(wtmode=’weight’, dowtsp=True).<br> If some of MSes have WEIGHT_SPECTRUM<br> but no data, WEIGHT will be used for such MSes. | No warning. Since tclean does not use the column, no action will be taken to make the column consistent among list of the MSs | No warning. <br>But when the column is created for the relevant MSes <br>the message appears( as in a single MS case). <br> Since this is updated in the input MSes, <br> the same rules apply for a single MS and list of MSes. <br>MODEL_DATA column will be created before <br>writing of the model<br> if the column does not exist. | No warning. <br> SOURCE_MODEL column created on the fly. | If CORRECTED_DATA is requested <br>but some MSes or all of them do not have the column,<br> it falls back to DATA.\t|The list of MSs given in vis parameter is sorted <br>in chronogical order so that the first MS in the list <br>has earliest time. <br> Other data selection parameters given as <br>lists are also sorted according to the reordered <br>vis list. |\tNo sorting of the list.\n",
        "**sdintimaging** | Same as tclean | Same as tclean | Same as tclean |\tSame as tclean | Same as tclean |\tSame as tclean\t|\n",
        "**tsdimaging** | Warn about absece and existence of <br>the column for all MSs. <br> For MSs with WEIGHT_SPECTRUM,<br> backup-copies will be made and <br>the column will be removed and WEIGHT <br>will be used.| |\tN/A\t| N/A\t| Same as tclean except that <br> the parameter \"datacolumn\" is not exposed to user and <br>it is always set to \"corrected\" |\tYes, via sort_mslist |\tNo sorting\n",
        "**sdimaging**\t| same as tsdimaging\t| | N/A\t| N/A\t|Warn about absence of the column. <br> For MSs without CORRECTED_DATA, <br>backup-copies will be made and the column will be <br>added to those MSs |Yes, via sort_mslist | No sorting\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FAp3GD2tVcz5"
      },
      "source": [
        "## Loading Data to Images\n",
        "\n",
        "The subsections below provide a brief overview of the steps you will need to load data into CASA and obtain a final, calibrated image. Each subject is covered in more detail in other chapters.\n",
        "\n",
        "An end-to-end workflow diagram for CASA data reduction for interferometry data is shown in the Figure below. This might help you chart your course through the package. In the following sub-sections, we will chart a rough course through this process, with the later chapters filling in the individual boxes.\n",
        "\n",
        "![ab58d87341f6d1d313c318373ddcda0f04b4d077](https://github.com/casangi/casadocs/blob/master/docs/notebooks/media/ab58d87341f6d1d313c318373ddcda0f04b4d077.png?raw=1)\n",
        "\n",
        "> Flow chart of the data processing operations that a general user will carry out in an end-to-end CASA reduction session.\n",
        "\n",
        "Note that single-dish data reduction (for example with the ALMA single-dish system) follows a similar course. This is detailed in the corresponding chapters.\n",
        "\n",
        "**Loading Data into CASA**\n",
        "\n",
        "The key data and image import tasks are (see \\\"[Visibility Data Import Export](visibilities_import_export.ipynb)\\\"):\n",
        "\n",
        "-   **importuvfits** --- import visibility data in UVFITS format\n",
        "-   **importvla** --- import data from VLA that is in export format\n",
        "-   **importasdm** --- import data in ASDM format\n",
        "-   **importfits** --- import a FITS image into a CASA image format table\n",
        "\n",
        "These are used to bring in your interferometer data, to be stored as a CASA MeasurementSet (MS), and any previously made images or models (to be stored as CASA image tables).\n",
        "\n",
        "The data import tasks will create a MS with a path and name specified by the vis parameter.  The [MeasurementSet](casa-fundamentals.ipynb#measurementset-v2) is the internal data format used by CASA, and conversion from any other native format is necessary for most of the data reduction tasks.\n",
        "\n",
        "Once data is imported, there are other operations you can use to manipulate the datasets:\n",
        "\n",
        "-   **concat** --- concatenate multiple MSs into a given or a new MS\n",
        "\n",
        "**VLA: Filling data from VLA archive format**   \n",
        "Jansky VLA data in \\\"archive SDM format are read into CASA via **importasdm**.  Historic VLA data can be filled with the tasl **importvla**.\n",
        "\n",
        "**Filling data from Scantable format**   \n",
        "CASA can import data from the Scantable format, since the development of Single-Dish started with that format (based on [ASAP](http://svn.atnf.csiro.au/trac/asap) format ). Currently, CASA tasks in Scantable format is no longer supported, but Scantable format can be converted into MeasurementSet format, with importASAP.\n",
        "\n",
        "**Filling data from UVFITS format**   \n",
        "For UVFITS format, use the **importuvfits** task. A subset of popular flavors of UVFITS (in particular UVFITS as written by AIPS) is supported by the CASA filler. FITSIDI (frequently used for VLBI data) can be read by **importfitsidi**. See \\\"[Visibility Data Import Export](visibilities_import_export.ipynb)\\\" for details.\n",
        "\n",
        "**Loading FITS images**   \n",
        "For FITS format images, such as those to be used as calibration models, use the **importfits** task. Most, though not all, types of FITS images written by astronomical software packages can be read in. See \\\"[Image Analysis](image_analysis.ipynb)\\\" for more information.\n",
        "\n",
        "**Concatenation of multiple MS**   \n",
        "Once you have loaded data into MeasurementSets on disk, you can use the tasks **concat** or **virtualconcat** to combine them.\n",
        "\n",
        "\n",
        "***\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YfPqLJLXVcz8"
      },
      "source": [
        "## Data Examination, Editing, and Flagging\n",
        "\n",
        "The main data examination and flagging tasks are:\n",
        "\n",
        "-   **listobs** --- summarize the contents of a MS\n",
        "-   **flagmanager** --- save and manage versions of the flagging entries in the MeasurementSet\n",
        "-   **plotms** --- interactive X-Y plotting and flagging of visibility data\n",
        "-   **flagdata** --- flagging (and unflagging) of specified data\n",
        "-   **msview** --- the CASA msview task can display (as a raster image) MS data, with some editing capabilities\n",
        "\n",
        "These tasks allow you to list, plot, and/or flag data in a CASA MS.\n",
        "\n",
        "**Interactive X-Y Plotting and Flagging**   \n",
        "The principal tool for making X-Y plots of visibility data is **plotms** (see \\\"[Data Examination and Editing](data_examination.ipynb)\\\"). Amplitudes and phases (among other things) can be plotted against several x-axis options.\n",
        "\n",
        "Interactive flagging (i.e., \"see it -- flag it\") is possible on the **plotms** X-Y displays of the data. Since flags are inserted into the MeasurementSet, it is useful to backup (or make a copy) of the current flags before further flagging is done, using **flagmanager**. Copies of the flag table can also be restored to the MS in this way.\n",
        "\n",
        "**plotms** can also be invoked without starting CASA. Launch it from the terminal with:\n",
        "\n",
        "```\n",
        "$ casaplotms &\n",
        "```\n",
        "\n",
        "**Flag the Data Non-interactively**   \n",
        "The **flagdata** task (\\\"[Data Examination and Editing](data_examination.ipynb)\\\") will flag the visibility data set based on the specified data selections. The **listobs** task may be run (e.g. with *verbose=True*) to provide some of the information needed to specify the flagging scope. **flagdata** also contains autoflagging routines.\n",
        "\n",
        "\n",
        "**Viewing and Flagging the MS**   \n",
        "The CASA task **msview** can be used to display the data in the MS as a (grayscale or color) raster image. The MS can also be edited (\\\"[Data Examination and Editing](data_examination.ipynb)\\\").\n",
        "\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "D_pyEGF2p0GN"
      },
      "source": [
        "### Calibration\n",
        "\n",
        "The major calibration tasks are:\n",
        "\n",
        "-   **setjy** --- Computes the model visibilities for a specified source given a flux density or model image, knows about standard calibrator sources\n",
        "-   **initweights** --- if necessary, supports (re-)initialization of the data weights, including an option for enabling spectral weight accounting\n",
        "-   **gencal** --- Creates a calibration table for known delay and antenna position offsets, opacities, and requantizer gains\n",
        "-   **bandpass** --- Solves for frequency-dependent (bandpass) complex gains\n",
        "-   **gaincal** --- Solves for time-dependent (frequency-independent) complex gains\n",
        "-   **fluxscale** --- Bootstraps the flux density scale from standard calibrators\n",
        "-   **polcal** --- polarization calibration\n",
        "-   **applycal** --- Applies calculated calibration solutions\n",
        "-   **clearcal** --- Re-initializes calibrated visibility data in a given MeasurementSet\n",
        "-   **listcal** --- Lists calibration solutions\n",
        "-   **plotms** --- Plots (and optionally flags) calibration solutions\n",
        "-   **uvcontsub** --- carry out uv-plane continuum subtraction for spectral-line data\n",
        "-   **split** --- write out a new (calibrated) MS for specified sources\n",
        "-   **cvel** --- Regrid a spectral MS onto a new frequency channel system\n",
        "\n",
        "During the course of calibration, the user will specify a set of calibrations to pre-apply before solving for a particular type of effect, for example gain or bandpass or polarization. The solutions are stored in a calibration *table (*subdirectory) which is specified by the user, not by the task: care must be taken in naming the table for future use. The user then has the option, as the calibration process proceeds, to accumulate the current state of calibration in a new cumulative table. Finally, the calibration can be applied to the dataset.\n",
        "\n",
        "See \\\"[Synthesis Calibration](synthesis_calibration.ipynb)\\\" for more information.\n",
        "\n",
        "**Prior Calibration**\n",
        "\n",
        "The **setjy** task calculates absolute fluxes for MeasurementSet base on known calibrator sources. This can then be used in later calibration tasks. Currently, **setjy** knows the flux density as a function of frequency for several standard VLA flux calibrators and solar system objects, and the value of the flux density can be manually inserted for any other source. If the source is not well-modeled as a point source, then a model image of that source structure can be used (with the total flux density scaled by the values given or calculated above for the flux density). Models are provided for the standard VLA calibrators and calculated for solar system objects.\n",
        "\n",
        "Antenna gain-elevation curves (e.g. for the VLA antennas), gain curves, requantizer gains, and atmospheric optical depth corrections (applied as an elevation-dependent function) may be pre-applied before solving for the **bandpass** and gains. The task **gencal** will generate those to be applied for further calibration.\n",
        "\n",
        "See \\\"[Synthesis Calibration](synthesis_calibration.ipynb)\\\" for more information.\n",
        "\n",
        "**Delay Calibration**\n",
        "\n",
        "A delay for each antenna can be calculated using **gaincal** with option \\\"K\\\". The delay calibration will remove delay errors that cause systematic slopes in the phases as a function opf time. In particular phase wraps will be removed.\n",
        "\n",
        "**Bandpass Calibration**\n",
        "\n",
        "The **bandpass** task calculates a bandpass calibration solution: that is, it solves for gain variations in frequency as well as in time. Since the bandpass (relative gain as a function of frequency) generally varies much more slowly than the changes in overall (mean) gain solved for by gaincal, one generally uses a long time scale when solving for the bandpass. The default *'B'* solution mode solves for the gains in frequency slots consisting of channels or averages of channels.\n",
        "\n",
        "A polynomial fit for the solution (solution type '*BPOLY')* may be carried out instead of the default frequency-slot based *'B'* solutions. This single solution will span (combine) multiple spectral windows.\n",
        "\n",
        "Bandpass calibration is discussed in detail in \\\"[Synthesis Calibration](synthesis_calibration.ipynb)\\\".\n",
        "\n",
        "If the gains of the system are changing over the time that the bandpass calibrator is observed, then you may need to do an initial gain calibration (see next step).\n",
        "\n",
        "**Gain Calibration**\n",
        "\n",
        "The **gaincal** task determines solutions for the time-based complex antenna gains, for each spectral window, from the specified calibration sources. A solution interval may be specified. The default *'G'* solution mode solves for antenna-based gains in each polarization in specified time solution intervals. The *'T'* solution mode is the same as *'G'* except that it solves for a single solution shared by both polarizations.\n",
        "\n",
        "A spline fit for the solution (solution type *'GSPLINE'*) may be carried out instead of the default time-slot based *'G'* solutions.\n",
        "\n",
        "Gain calibration is discussed in detail in \\\"[Synthesis Calibration](synthesis_calibration.ipynb)\\\".\n",
        "\n",
        "**Polarization Calibration**\n",
        "\n",
        "The **polcal** task will solve for any unknown polarization leakage and cross-hand phase terms (*'D'* and *'X'* solutions). The *'D'* leakage solutions will work on sources with no polarization and sources with known (and supplied, e.g., using smodel) polarization. For sources with unknown polarization tracked through a range in parallactic angle on the sky, using poltype *'D+QU'*, which will first estimate the calibrator polarization for you.\n",
        "\n",
        "The solution for the unknown cross-hand polarization phase difference *'X'* term requires a polarized source with known linear polarization (Q,U).\n",
        "\n",
        "Frequency-dependent (i.e., per channel) versions of all of these modes are also supported (poltypes *'Df', 'Df+QU'*, and *'Xf'*.\n",
        "\n",
        "**Examining Calibration Solutions**\n",
        "\n",
        "The **plotms** task can plot the solutions in a calibration table. The *xaxis* choices include time (for gaincal solutions) and channel (e.g. for bandpass calibration).\n",
        "\n",
        "The **listcal** task will print out the calibration solutions in a specified table.\n",
        "\n",
        "**Bootstrapping Flux Calibration**\n",
        "\n",
        "The **fluxscale** task bootstraps the flux density scale from \"primary\" standard calibrators to the \"secondary\" calibration sources. Note that the flux density scale must have been previously established on the \"primary\" calibrator(s) using **setjy**, and of course a calibration table containing valid solutions for all calibrators must be available.\n",
        "\n",
        "**Correcting the Data**\n",
        "\n",
        "The final step in the calibration process, **applycal** may be used to apply several calibration tables (e.g., from **gaincal** or **bandpass**, along with prior calibration tables). The corrections are applied to the *DATA* column of the visibility, writing the *CORRECTED_DATA* column which can then be plotted in **plotms**, **split** out as the *DATA* column of a new MS, or imaged (e.g. using **clean**). Any existing corrected data are overwritten.\n",
        "\n",
        "**Splitting the Data**\n",
        "\n",
        "After a suitable calibration is achieved, it may be desirable to create one or more new MeasurementSets containing the data for selected sources. This can be done using the **split** task (see \\\"[UV Manipulation](uv_manipulation.ipynb)\\\").\n",
        "\n",
        "Further imaging and calibration (e.g. self-calibration) can be carried out on these split MeasurementSets.\n",
        "\n",
        "**UV Continuum subtraction**\n",
        "\n",
        "For spectral line data, continuum subtraction can be performed in the image domain (**imcontsub**) or in the uv domain. For the latter, **uvcontsub** subtracts polynomial of desired order from each baseline, defined by line-free channels.\n",
        "\n",
        "**Transforming the Data to a new frame**\n",
        "\n",
        "If you want to transform your dataset to a different frequency and velocity frame than the one it was observed in, then you can use the **cvel** task (\\\"[UV Manipulation](uv_manipulation.ipynb)\\\"). Alternatively, you can do the regridding during the imaging process in **clean** without running **cvel** before.\n",
        "\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "W6iEhVf8Vcz_"
      },
      "source": [
        "### Synthesis Imaging\n",
        "\n",
        "The key synthesis imaging tasks are:\n",
        "\n",
        "- **tclean** - Calculates a deconvolved image based on the visibility data, using one of several clean algorithms\n",
        "- **feather** - Combines a single dish and synthesis image in the Fourier plane\n",
        "\n",
        "Most of these tasks are used to take calibrated interferometer data, with the possible addition of a single-dish image, and reconstruct a model image of the sky.\n",
        "\n",
        "See Chapter \\\"[Synthesis Imaging](synthesis_calibration.ipynb)\\\" and \\\"[Image Combination](image_combination.ipynb)\\\" for more information.\n",
        "\n",
        "**Cleaning a single-field image or a mosaic**\n",
        "\n",
        "The CLEAN algorithm is the most popular and widely-studied method for reconstructing a model image based on interferometer data. It iteratively removes at each step a fraction of the flux in the brightest pixel in a defined region of the current \"dirty\" image, and places this in the model image. The clean task implements the CLEAN algorithm for single-field data. The user can choose from a number of options for the particular flavor of CLEAN to use.\n",
        "\n",
        "Often, the first step in imaging is to make a simple gridded Fourier inversion of the calibrated data to make a \"dirty\" image. This can then be examined to look for the presence of noticeable emission above the noise, and to assess the quality of the calibration by searching for artifacts in the image. This is done using **clean** with *niter=0*.\n",
        "\n",
        "The **clean** task can jointly deconvolve mosaics as well as single fields, and also has options to do wide-field and wide-band multi-frequency synthesis imaging.\n",
        "\n",
        "See \\\"[Synthesis Imaging](synthesis_calibration.ipynb)\\\" for an in-depth discussion of the **clean** task.\n",
        "\n",
        "**Feathering in a Single-Dish image**\n",
        "\n",
        "If you have a single-dish image of the large-scale emission in the field, this can be \"feathered\" in to the image obtained from the interferometer data. This is carried out using the **feather** task as the weighted sum in the uv-plane of the gridded transforms of these two images. While not as accurate as a true joint reconstruction of an image from the synthesis and single-dish data together, it is sufficient for most purposes. A graphical version of **feather** is provided by **casafeather**.\n",
        "\n",
        "See \\\"[Image Combination](image_combination.ipynb)\\\" for an in-depth discussion of the **feather** task.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CnXzQUNsVcz_"
      },
      "source": [
        "### Self Calibration\n",
        "\n",
        "Once a calibrated dataset is obtained, and a first deconvolved model image is computed, a \"self-calibration\" loop can be performed. Effectively, the model (not restored) image is passed back to another calibration process (on the target data). This refines the calibration of the target source, which up to this point has had (usually) only external calibration applied. This process follows the regular calibration procedure outlined above.\n",
        "\n",
        "Any number of self-calibration loops can be performed. As long as the images are improving, it is usually prudent to continue the self-calibration iterations.\n",
        "\n",
        "This process is described in \\\"[Synthesis Calibration](synthesis_calibration.ipynb)\\\".\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lWWPFQmlVc0A"
      },
      "source": [
        "### Data and Image Analysis\n",
        "\n",
        "The key data and image analysis tasks are:\n",
        "\n",
        "-   **imhead** --- summarize and manipulate the \"header\" information in a CASA image\n",
        "-   **imcontsub** --- perform continuum subtraction on a spectral-line image cube\n",
        "-   **immath** --- perform mathematical operations on or between images\n",
        "-   **immoments** --- compute the moments of an image cube\n",
        "-   **imstat** --- calculate statistics on an image or part of an image\n",
        "-   **imval** --- extract values of one or more pixels, as a spectrum for cubes, from an image\n",
        "-   **imfit** --- simple 2D Gaussian fitting of single components to a region of an image\n",
        "-   **imregrid** --- regrid an image onto the coordinate system of another image\n",
        "-   **imview** --- there are useful region statistics and image cube plotting capabilities in imview.\n",
        "\n",
        "**What's in an image?**\n",
        "\n",
        "The **imhead** task will print out a summary of image \"header\" keywords and values. This task can also be used to retrieve and change the header values.\n",
        "\n",
        "See \\\"[Image Analysis](image_analysis.ipynb)\\\" for more.\n",
        "\n",
        "**Image statistics**\n",
        "\n",
        "The **imstat** task will print image statistics. There are options to restrict this to a box region, and to specified channels and Stokes of the cube. This task will return the statistics in a Python dictionary return variable.\n",
        "\n",
        "**Image values**\n",
        "\n",
        "The **imval** task will return values from an image. There are options to restrict this to a box region, and to return specified channels and Stokes of the cube as a spectrum. This task will return these values in a Python dictionary return variable which can then be operated on in the CASA environment.\n",
        "\n",
        "**Moments of an image cube**\n",
        "\n",
        "The **immoments** task will compute a \"moments\" image of an input image cube. A number of options are available, from the traditional true moments (zero, first, second) and variations thereof, to other images such as median, minimum, or maximum along the moment axis.\n",
        "\n",
        "**Image math**\n",
        "\n",
        "The **immath** task will allow you to form a new image by mathematical combinations of other images (or parts of images). This is a powerful task to use.\n",
        "\n",
        "**Regridding an Image**\n",
        "\n",
        "It is occasionally necessary to regrid an image onto a new coordinate system. The **imregrid** task can be used to regrid an input image onto the coordinate system of an existing template image, creating a new output image.\n",
        "\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6xKaYZSeVc0A"
      },
      "source": [
        "### Displaying Images\n",
        "\n",
        "To display an image use the task **imview**. The **imview** task will display images in raster, contour, or vector form. Blinking and movies are available for spectral-line image cubes. To start imview, type:\n",
        "\n",
        "```\n",
        "imview\n",
        "```\n",
        "\n",
        "within CASA.\n",
        "\n",
        "Executing the **imview** task will bring up two windows: a imview screen showing the data or image, and a file catalog list. Click on an image or MS from the file catalog list, choose the proper display, and the image should pop up on the screen. Clicking on the wrench tool (second from left on upper left) will obtain the data display options. Most functions are self-documenting.\n",
        "\n",
        "See \\\"[Image / Cube Visualization](carta.ipynb)\\\" for more details.\n",
        "\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UDKQH8PHVc0B"
      },
      "source": [
        "### Getting data and images out of CASA\n",
        "\n",
        "The key data and image export tasks are:\n",
        "\n",
        "-   **exportuvfits** --- export a CASA MS in UVFITS format\n",
        "-   **exportfits** --- export a CASA image table as FITS\n",
        "\n",
        "These tasks can be used to export a CASA MS or image to UVFITS or FITS respectively. See the individual sections referred to above for more on each.\n",
        "\n",
        "***\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3fTmQM4LbUff"
      },
      "source": [
        "## Bibliography\n",
        "\n",
        "1. Hamaker, J.P.; Bregman, J.D.; Sault, R.J. 1996, A&AS 117, 137 ([ADS](http://adsabs.harvard.edu/abs/1996A%26AS..117..137H))\n",
        "2. Sault, R.J.; Hamaker, J.P.; Bregman, J.D. 1996, A&AS, 117, 149 ([ADS](http://adsabs.harvard.edu/abs/1996A%26AS..117..149S))\n",
        "3. [Kemball & Wieringa 2000](https://drive.google.com/file/d/1bWXRDx-EkCfdnY8fWFRtBu1EvncrJFhA/view?usp=sharing)\n",
        "\n",
        "\n",
        "\n"
      ]
    }
  ]
}