docs/notebooks/visibilities_import_export.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "visibilities_import_export.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "toc_visible": true
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mucRwpac0ECv"
      },
      "source": [
        "# Visibilities Import Export \n",
        "\n",
        "\n",
        "To use CASA to process your data, you first will need to get it into a form that is understood by the package. These are \"MeasurementSets\" for synthesis and single dish data, which is the purpose of this chapter. Importing images, or \"image tables\" as understood by CASA, is explained [here](image_analysis.ipynb#Image-Import/Export).\n",
        "\n",
        "There are a number of tasks used to fill telescope-specific data and to import/export standard formats. These are:\n",
        "\n",
        "-   **importasdm** --- import of ALMA data in ASDM format\n",
        "-   **importuvfits** --- import visibility data in UVFITS format\n",
        "-   **importfitsidi** --- import visibility data in the FITS-IDI format\n",
        "-   **importvla** --- import data from VLA that is in export format\n",
        "-   **importmiriad** --- import data from MIRIAD visibilities\n",
        "-   **importatca** --- import ATCA data that is in the RPFITS (archive) format\n",
        "-   **importgmrt** --- import GMRT data\n",
        "-   **importasap** --- convert ASAP (ATNF Spectral Analysis Package) into a CASA visibility data format\n",
        "-   **importnro** --- import NRO 45m data \n",
        "-   **exportasdm** --- convert a CASA MS into an ASDM\n",
        "-   **exportuvfits** --- export a CASA MS in UVFITS format\n",
        "\n",
        "\n",
        "\n",
        "***\n",
        "\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Sf8a4cr90ECx"
      },
      "source": [
        "## UV Data Import \n",
        "\n",
        "Converting Telescope UV Data to a MeasurementSet\n",
        "\n",
        "There are a number of tasks available to bring data in various forms into CASA as a MeasurementSet:\n",
        "\n",
        "-   ALMA and VLA Science Data Model format via **importasdm** and **importevla**\n",
        "-   historic VLA Archive format data via **importvla**\n",
        "-   ATCA Data via **importatca**\n",
        "-   MIRIAD Data from various telescopes via **importmiriad**\n",
        "-   GMRT Data via **importgmrt**\n",
        "-   UVFITS format can be imported into and exported from CASA (**importuvfits**, **importfitsidi**, and **exportuvfits**)\n",
        "\n",
        "\n",
        "*ALMA and VLA Filling of Science Data Model (ASDM) data*\n",
        "\n",
        "The ALMA and JVLA projects have agreed upon a common archival science data model (ASDM; sometimes also called SDM) format, and have jointly developed the software to fill this data into CASA. In the ASDM format, the bulk of the data is contained in large binary data format (BDF) tables, with the meta-data and ancillary information in XML tables. This is structured as a directory, like the MS, and was designed to be similar to the MS to facilitate conversion.\n",
        "\n",
        "The content of an ASDM can be listed with the task **asdmsummary**:\n",
        "\n",
        "```\n",
        "#asdmsummary :: Summarized description of an ASDM dataset.\n",
        "asdm = '' #Name of input ASDM directory\n",
        "```\n",
        "\n",
        "with an output that contains the list and positions of the antennas, followed by the metadata of each scan like observation time, source name, frequency and polarization setup:\n",
        "\n",
        "```python\n",
        "Input ASDM dataset : TDEM0008.sb3373760.eb3580330.55661.22790537037\n",
        "========================================================================================\n",
        "ASDM dataset :TDEM0008.sb3373760.eb3580330.55661.22790537037\n",
        "========================================================================================\n",
        "\n",
        "Exec Block : ExecBlock_0\n",
        "Telescope : JVLA\n",
        "Configuration name : B\n",
        "Observer name : Dr. Juergen Ott\n",
        "The exec block started on 2011-04-10T05:28:13.200000000 and ended on 2011-04-10T10:27:12.300000256\n",
        "\n",
        "27 antennas have been used in this exec block.\n",
        "        Id     Name         Make Station    Diameter         X              Y             Z\n",
        "     Antenna_0  ea01    UNDEFINED   W36        25    -1606841.96   -5042279.689    3551913.017\n",
        "     Antenna_1  ea02    UNDEFINED   E20        25     -1599340.8   -5043150.965    3554065.219\n",
        "     Antenna_2  ea03    UNDEFINED   E36        25   -1596127.728   -5045193.751    3552652.421\n",
        "     Antenna_3  ea04    UNDEFINED   W28        25   -1604865.649    -5042190.04    3552962.365\n",
        "     Antenna_4  ea05    UNDEFINED   W08        25   -1601614.091   -5042001.653    3554652.509\n",
        "     Antenna_5  ea06    UNDEFINED   N24        25    -1600930.06   -5040316.397    3557330.397\n",
        "     Antenna_6  ea07    UNDEFINED   E32        25   -1597053.116   -5044604.687    3553058.987\n",
        "     Antenna_7  ea08    UNDEFINED   N28        25   -1600863.684   -5039885.318    3557965.319\n",
        "     Antenna_8  ea09    UNDEFINED   E24        25    -1598663.09   -5043581.392    3553767.029\n",
        "     Antenna_9  ea10    UNDEFINED   N32        25   -1600781.039   -5039347.456    3558761.542\n",
        "     Antenna_10  ea11    UNDEFINED   E04        25    -1601068.79    -5042051.91    3554824.835\n",
        "     Antenna_11  ea12    UNDEFINED   E08        25   -1600801.926   -5042219.366    3554706.448\n",
        "     Antenna_12  ea14    UNDEFINED   W12        25   -1602044.903   -5042025.824    3554427.832\n",
        "     Antenna_13  ea15    UNDEFINED   W24        25   -1604008.742   -5042135.828    3553403.707\n",
        "     Antenna_14  ea16    UNDEFINED   N12        25   -1601110.052   -5041488.079    3555597.439\n",
        "     Antenna_15  ea17    UNDEFINED   W32        25   -1605808.656   -5042230.082    3552459.202\n",
        "     Antenna_16  ea18    UNDEFINED   N16        25   -1601061.961    -5041175.88    3556058.022\n",
        "     Antenna_17  ea19    UNDEFINED   W04        25   -1601315.893    -5041985.32    3554808.305\n",
        "     Antenna_18  ea20    UNDEFINED   N36        25   -1600690.606   -5038758.734    3559632.061\n",
        "     Antenna_19  ea21    UNDEFINED   E12        25    -1600416.51    -5042462.45    3554536.041\n",
        "     Antenna_20  ea22    UNDEFINED   N04        25   -1601173.979   -5041902.658    3554987.518\n",
        "     Antenna_21  ea23    UNDEFINED   E16        25   -1599926.104   -5042772.967    3554319.789\n",
        "     Antenna_22  ea24    UNDEFINED   W16        25   -1602592.854   -5042054.997      3554140.7\n",
        "     Antenna_23  ea25    UNDEFINED   N20        25   -1601004.709   -5040802.809    3556610.133\n",
        "     Antenna_24  ea26    UNDEFINED   W20        25   -1603249.685   -5042091.404    3553797.803\n",
        "     Antenna_25  ea27    UNDEFINED   E28        25   -1597899.903   -5044068.676    3553432.445\n",
        "     Antenna_26  ea28    UNDEFINED   N08        25    -1601147.94   -5041733.837    3555235.956\n",
        "\n",
        "Number of scans in this exec Block : 234\n",
        "\n",
        "scan #1 from 2011-04-10T05:28:13.200000000 to 2011-04-10T05:33:35.500000256\n",
        "        Intents : OBSERVE_TARGET\n",
        "        Sources : 1331+305=3C286\n",
        "        Subscan #1 from 2011-04-10T05:28:13.200000000 to 2011-04-10T05:33:35.500000256\n",
        "                Intent : UNSPECIFIED\n",
        "                Number of integrations : 322\n",
        "\n",
        "                 Binary data in uid:///evla/bdf/1302413292901\n",
        "                 Number of integrations : 322\n",
        "                 Time sampling : INTEGRATION\n",
        "                 Correlation Mode : CROSS_AND_AUTO\n",
        "                 Spectral resolution type : FULL_RESOLUTION\n",
        "                 Atmospheric phase correction : AP_UNCORRECTED\n",
        "                 SpectralWindow_0 : numChan = 256, frame = TOPO,\n",
        "                 firstChan = 8484000000, chandWidth = 125000 x Polarization_0 : corr = RR,LL\n",
        "\n",
        "scan #2 from 2011-04-10T05:33:35.500000256 to 2011-04-10T05:35:35.200000000\n",
        "        Intents : OBSERVE_TARGET\n",
        "        Sources : 1331+305=3C286\n",
        "        Subscan #1 from 2011-04-10T05:33:35.500000256 to 2011-04-10T05:35:35.200000000\n",
        "                Intent : UNSPECIFIED\n",
        "                Number of integrations : 119\n",
        "\n",
        "                 Binary data in uid:///evla/bdf/1302413293280\n",
        "                 Number of integrations : 119\n",
        "                 Time sampling : INTEGRATION\n",
        "                 Correlation Mode : CROSS_AND_AUTO\n",
        "                 Spectral resolution type : FULL_RESOLUTION\n",
        "                 Atmospheric phase correction : AP_UNCORRECTED\n",
        "                 SpectralWindow_0 : numChan = 256, frame = TOPO,\n",
        "                 firstChan = 8484000000, chandWidth = 125000 x Polarization_0 : corr = RR,LL\n",
        "\n",
        "scan #3 from 2011-04-10T05:35:35.200000000 to 2011-04-10T05:36:34.999999488\n",
        "        Intents : OBSERVE_TARGET\n",
        "        Sources : 1331+305=3C286\n",
        "        Subscan #1 from 2011-04-10T05:35:35.200000000 to 2011-04-10T05:36:34.999999488\n",
        "...\n",
        "```\n",
        "\n",
        "The **importasdm** task will fill SDM format data from ALMA and the JVLA into a CASA visibility data set (MS). All SDM formats from version 2 to the current version can be filled by **importasdm**. \n",
        "\n",
        "The default inputs of **importasdm** are:\n",
        "\n",
        "```\n",
        "# importasdm -- Convert an ALMA Science Data Model observation into a\n",
        "# CASA visibility file (MS)\n",
        "asdm                =         ''        # Name of input asdm directory (on\n",
        "                                        # disk)\n",
        "vis                 =         ''        # Root name of the MS to be created.\n",
        "                                        # Note the .ms is NOT added\n",
        "createmms           =      False        # Create a multi-MS output\n",
        "corr_mode           =      'all'        # specifies the correlation mode to be\n",
        "                                        # considered on input. A quoted string\n",
        "                                        # containing a sequence of ao, co,\n",
        "                                        # ac,or all separated by whitespaces\n",
        "                                        # is expected\n",
        "srt                 =      'all'        # specifies the spectral resolution\n",
        "                                        # type to be considered on input. A\n",
        "                                        # quoted string containing a sequence\n",
        "                                        # of fr, ca, bw, or all separated by\n",
        "                                        # whitespaces is expected\n",
        "time_sampling       =      'all'        # specifies the time sampling\n",
        "                                        # (INTEGRATION and/or SUBINTEGRATION)\n",
        "                                        # to be considered on input. A quoted\n",
        "                                        # string containing a sequence of i,\n",
        "                                        # si, or all separated by whitespaces\n",
        "                                        # is expected\n",
        "ocorr_mode          =       'ca'        # output data for correlation mode\n",
        "                                        # AUTO_ONLY (ao) or CROSS_ONLY (co) or\n",
        "                                        # CROSS_AND_AUTO (ca)\n",
        "compression         =      False        # Flag for turning on data compression\n",
        "lazy                =      False        # Make the MS DATA column read the ASDM\n",
        "                                        # Binary data directly (faster import,\n",
        "                                        # smaller MS)\n",
        "asis                =         ''        # Creates verbatim copies of the\n",
        "                                        # ASDMtables in the ouput MeasurementSet.\n",
        "                                        # Value given must be a string\n",
        "                                        # of table names separated by spaces;\n",
        "                                        # A * wildcard is allowed.\n",
        "wvr_corrected_data  =       'no'        # Specifies which values are considerd\n",
        "                                        # in the SDM binary data to fill the\n",
        "                                        # DATA column in the MAIN table of the\n",
        "                                        # MS; yes for corrected, no for \n",
        "                                        # uncorrected, both for corrected and \n",
        "                                        # uncorrected (resulting in two MSs) \n",
        "scans               =         ''        # Processes only the specified scans.\n",
        "                                        # A scan specification consists of an \n",
        "                                        # exec block index followed by the :\n",
        "                                        # character, followed by a comma\n",
        "                                        # separated list of scan indexes or\n",
        "                                        # scan index ranges.\n",
        "                                        # (e.g. 0:1;1:2~6,8;2:,3:24~30), \n",
        "ignore_time         =      False        # All the rows of the tables Feed,\n",
        "                                        # History, Pointing, Source, SysCal,\n",
        "                                        # CalDevice, SysPower, and Weather are\n",
        "                                        # processed independently of the time\n",
        "                                        # range of the selected exec\n",
        "                                        # block / scan.\n",
        "process_syspower    =       True        # Process the SysPower table? \n",
        "process_caldevice   =       True        # Process the CalDevice table? \n",
        "process_pointing    =       True        # Process the Pointing table? \n",
        "process_flags       =       True        # Create online flags in the FLAG_CMD\n",
        "                                        # sub-table? \n",
        "     tbuff          =        0.0        # Time padding buffer (seconds)\n",
        "     applyflags     =      False        # Apply the flags to the MS.\n",
        "     savecmds       =      False        # Save flag commands to an ASCII file\n",
        "     outfile        =         ''        # Name of ASCII file to save flag\n",
        "                                        # commands\n",
        "flagbackup          =       True        # Back up flag column before applying\n",
        "                                        # flags.\n",
        "verbose             =      False        # Output lots of information while the\n",
        "                                        # filler is working\n",
        "overwrite           =      False        # Over write an existing MS(s)\n",
        "bdfflags            =      False        # Set the MS FLAG column according to\n",
        "                                        # the ASDM _binary_ flags\n",
        "with_pointing_correction = False        # Add (ASDM::Pointing::encoder -\n",
        "                                        # ASDM::Pointing::pointingDirection)\n",
        "                                        # to the value to be written in\n",
        "                                        # MS::Pointing::direction\n",
        "convert_ephem2geo   =       True        # if True, convert any attached\n",
        "                                        # ephemerides to the GEO reference\n",
        "                                        # frame (time-spacing not changed)\n",
        "polyephem_tabtimestep =      0.0        # Timestep (days) for the tabulation \n",
        "                                        # of polynomial ephemerides. \n",
        "                                        # A value <= 0 disables tabulation. \n",
        "```\n",
        "\n",
        "If *scans* is set, then **importasdm** processes only the scans specified in the option's value. This value is a semicolon separated list of scan specifications. A scan specification consists of an exec block index followed by the character ':' followed by a comma separated list of scan indexes or scan index ranges. A scan index is relative to the exec block it belongs to. Scan indexes are 1-based while exec blocks are 0-based. The expressions\n",
        "\n",
        "```\n",
        " \"0:1\"\n",
        " \"2:2~6\"\n",
        " \"0:1,1:2~6,8;2:,3:24~30\"\n",
        " \"1,2\"\n",
        " \"3:\"\n",
        "```\n",
        "\n",
        "are all valid values for the [selection](visibility_data_selection.ipynb). The \\\"3:\\\" selector will be interpreted as 'all the scans of the exec block 3'. A scan index or a scan index range not preceded by an exec block index will be interpreted as 'all the scans with such indexes in all the exec blocks'. By default all the scans are considered.\n",
        "\n",
        "When *process_flags=True* the task will create online flags based on the Flag.xml, Antenna.xml and SpectralWindow.xml files and copy them to the *FLAG_CMD* sub-table of the MS. The flags will NOT be applied unless the parameter *applyflags* is set to *True*. Optionally, the flags can also be saved to an external ASCII file if savecmds is set to *True*. The flags can later be applied to the MS using task **flagdata** in *list* mode.\n",
        "\n",
        "When *bdfflags=True* the task will apply online flags contained in the ASDM BDF data by calling the executable bdflags2MS which the user can also do from the OS prompt. This is recommended for ALMA data.\n",
        "\n",
        "The option *createmms* prepares the output file for [parallel processing](parallel-processing.ipynb) and creates a [multi-MS](parallel-processing.ipynb#the-multi-ms).\n",
        "\n",
        " \n",
        "\n",
        "**Specifics on importing Janksy VLA data with importasdm**\n",
        "\n",
        "As of CASA 5.4, the task importevla is no longer available to import JVLA data, but a lot of its functionality is replaced by importasdm. However, several additional steps are required to duplicate the behaviour of importevla when using importasdm, involving a difference in default parameters and the fact that some of the on-the-go flagging cannot be performed by importasdm.\n",
        "\n",
        "To mimic the behaviour of importevla, change the following parameters in **importasdm** from their default settings:\n",
        "\n",
        "-   *ocorr_mode = \\'co\\'* to import cross-correlations only (discarding auto-correlations)*    *\n",
        "-   *with_pointing_correction = True* to add pointing corrections*    *\n",
        "-   *process_flags = True* (default) to read in the online flags, then *applyflags = True* to apply the online flags and/or *savecmd = True* to save flag commands to an ascii table.\n",
        "-   For ephemeris objects: convert_ephem2geo = False\n",
        "\n",
        "While online flags can thus be created by leaving the parameter *process_flags = True* by default, additional flagging steps need to be performed after **importasdm** to flag zero values and shadowing of antennas:\n",
        "\n",
        "-   **Shadow flags:** use task **flagdata**, with *mode = \\'shadow\\'* (and optionally *reason = \\'shadow\\'*). The parameters *tolerance* and *addantenna* can be specified in flagdata in the same way they were used in importevla. *    *\n",
        "-   **Zero clipping flags:** use task **flagdata**, with ***mode = \\'clip\\',*** *correlation = \\'ABS_ALL\\',* and *clipzeros = True* (and optionally *reason = \\'clip\\'*)*.* Note that the non-default case in importevla where *flagpol = False c*an be replicated by setting *correlation=\\\"ABS_RR, ABS_LL\\\".*\n",
        "\n",
        "Like **importasdm**, the task **flagdata** can also save the flagging commands to an ascii table by setting *savepars = True.* To NOT apply the flags (*applyflags=False* in importevla) add *action=\\'calculate\\'* to flagdata. You may also chose to add a reason using the cmdreason argument, e.g. *cmdreason=\\\"CLIP_ZERO_ALL\\\".*\n",
        "\n",
        "<div class=\"alert alert-warning\">\n",
        "**WARNING***:* The task **flagdata** can only write out the flag commands for that invocation of flagdata. The default *overwrite=True* must be used to overwrite an existing file. In order to save the commands from all 3 possible flagging steps (importasdm, zero, and shadow) each step must be saved to a separate file, which must then be concatenated into a single file to be used to flag the data.\n",
        "</div>\n",
        "\n",
        " \n",
        "\n",
        "**Import of ASDM data with option lazy=True**\n",
        "\n",
        "With *lazy=False,* **importasdm** will fill the visibilities into a newly created *DATA* column of the MS converting them from their binary format in the ASDM to the CASA Table format.\n",
        "\n",
        "If, however, *lazy* is set to *True*, the task will create the *DATA* column with an ALMA data-specific storage manager, the asdmstman, which enables CASA to directly read the binary data from the ASDM with on-the-fly conversion. No redundant copy of the raw data is created.\n",
        "\n",
        "This procedure has the advantage that it saves more than 60% disk space and at least in some cases makes the access to the *DATA* column ≥ 10% faster because the data I/O volume is decreased. For the same reason, it also accelerates the import itself by ca. a factor 2. The acceleration is particularly large in the **applycal** task and here particularly on standard SATA disks.\n",
        "\n",
        "E.g., if your ASDM has a size of 36 GB, the import with default parameters will turn this into an MS of 73 GB size (total disk space consumption = 36 GB + 73 GB = 109 GB). With *lazy=True*, the imported MS has a size of only 2 GB (total disk space consumption = 36 GB + 2 GB = 38 GB). I.e. your total disk space savings are ca. 65%. Even when you compare to the case where you delete the ASDM after normal import, the solution with lazy import and keeping the ASDM will save you ca. 48% disk space (in the example above 38 GB compared to 73 GB).\n",
        "\n",
        "The only caveats are the following:\n",
        "\n",
        "1.  You must not delete your ASDM. You can, however, move it but you have to update the reference stored in the MS. Symbolic links will work. See below on how to use the tool method **ms.asdmref()** to manipulate the ASDM reference.\n",
        "2.  The lazily imported *DATA* column is read-only. But in any normal data reduction, the *DATA* column (as opposed to *CORRECTED_DATA*) is treated as read-only anyway.\n",
        "\n",
        "The lazily imported MS is numerically identical with the traditionally imported MS and so are all results derived from the MSs.\n",
        "\n",
        "An important additional tool to manipulate lazily imported MSs is the new method **ms.asdmref()** in the **ms** tool. If the MS is imported from an ASDM with option *lazy=True*, the DATA column of the MS is virtual and directly reads the visibilities from the ASDM. A reference to the original ASDM is stored with the MS. If the ASDM needs to be moved to a different path, the reference to it in the MS needs to be updated. This can be achieved with **ms.asdmref()**.\n",
        "\n",
        "The method takes one argument: *abspath*. When called with *abspath* equal to an empty string (default), the method just reports the currently set ASDM path or an empty string if the ASDM path was not set, i.e. the MS was not lazily imported.\n",
        "\n",
        "If you want to move the referenced ASDM to a different path, you can set the new absolute path by providing it as the value of *abspath* to the method.\n",
        "\n",
        "```\n",
        " ms.open('uid___A12345_X678_X910.ms',False)\n",
        " ms.asdmref('/home/alma/myanalysis/uid___A12345_X678_X910')\n",
        " ms.close()\n",
        "```\n",
        "\n",
        "will set the new location of the referenced ASDM to /home/alma/myanalysis/uid\\_\\_\\_A12345_X678_X910. Contrary to what one would expect from the parameter name, you can also provide a *relative* path as *abspath*. This path will be interpreted as relative to the location of the MS.\n",
        "\n",
        "<div class=\"alert alert-info\">\n",
        "**Info**: the lazily imported MS itself can be moved without any restrictions independently from the referenced ASDM as long as the path to the ASDM remains accessible, even across file systems.\n",
        "</div>\n",
        "\n",
        "\n",
        "**VLA: Filling data from archive format (importvla)**\n",
        "\n",
        "VLA data in archive format (i.e., as downloaded from the historic VLA data archive) are read into CASA from disk using the **importvla** task. The inputs are:\n",
        "\n",
        "```\n",
        "#importvla :: import VLA archive file(s) to a MeasurementSet:\n",
        "\n",
        "archivefiles  =         ''   #Name of input VLA archive file(s)\n",
        "vis           =         ''   #Name of output visibility file\n",
        "bandname      =         ''   #VLA frequency band name:''=>obtain all bands in archive files\n",
        "frequencytol  =   150000.0   #Frequency shift to define a unique spectral window (Hz)\n",
        "project       =         ''   #Project name:  '' => all projects in file\n",
        "starttime     =         ''   #start time to search for data\n",
        "stoptime      =         ''   #end time to search for data\n",
        "applytsys     =       True   #apply nominal sensitivity scaling to data & weights\n",
        "autocorr      =      False   #import autocorrelations to ms, if set to True\n",
        "antnamescheme =      'new'   #'old' or 'new'; 'VA04' or '4' for ant 4\n",
        "keepblanks    =      False   #Fill scans with empty source names (e.g. tipping scans)?\n",
        "evlabands     =      False   #Use updated eVLA frequencies and bandwidths\n",
        "```\n",
        "\n",
        "The main parameters are *archivefiles* to specify the input VLA Archive format file names, and *vis* to specify the output MS name.\n",
        "\n",
        "<div class=\"alert alert-info\">\n",
        "**Info:** The scaling of VLA data both before and after the June 2007 Modcomp-turnoff is fully supported, based on the value of *applytsys*.\n",
        "</div>\n",
        "\n",
        "Note that *archivefiles* takes a string or list of strings, as there are often multiple files for a project in the archive.\n",
        "\n",
        "For example:\n",
        "\n",
        "```\n",
        "archivefiles = ['AP314_A950519.xp1','AP314_A950519.xp2']\n",
        "   vis = 'NGC7538.ms'\n",
        "```\n",
        "\n",
        "The **importvla** task allows selection on the frequency band. Suppose that you have 1.3 cm line observations in K-band and you have copied the archive data files AP314_A95019.xp\\* to your working directory and started casa. Then,\n",
        "\n",
        "```\n",
        "  default('importvla')\n",
        "   archivefiles = ['AP314_A950519.xp1','AP314_A950519.xp2','AP314_A950519.xp3']\n",
        "   vis = 'ngc7538.ms'\n",
        "   bandname = 'K'\n",
        "   frequencytol = 10e6\n",
        "   importvla()\n",
        "```\n",
        "\n",
        "If the data is located in a different directory on disk, then use the full path name to specify each archive file, e.g.:\n",
        "\n",
        "```\n",
        "archivefiles=['/home/rohir2/jmcmulli/ALMATST1/Data/N7538/AP314_A950519.xp1',\n",
        "     '/home/rohir2/jmcmulli/ALMATST1/Data/N7538/AP314_A950519.xp2',\n",
        "     '/home/rohir2/jmcmulli/ALMATST1/Data/N7538/AP314_A950519.xp3']\n",
        "```\n",
        "\n",
        "<div class=\"alert alert-info\">\n",
        "**Info:** **importvla** will import the on-line flags (from the VLA system) along with the data. Shadowed antennas will also be flagged. The flags will be put in the *MAIN* table and thus available to subsequent tasks and tools. If you wish to revert to unflagged data, use **flagmanager** to save the flags (if you wish), and then use **flagdata** with *mode='manual'* and *unflag=True* to toggle off the flags.\n",
        "</div>\n",
        "\n",
        " \n",
        "\n",
        "The other parameters are:\n",
        "\n",
        "- Parameter *applytsys* \n",
        "    - The *applytsys* parameter controls whether the nominal sensitivity scaling (based on the measured *TSYS*, with the weights scaled accordingly using the integration time) is applied to the visibility amplitudes or not. If *True*, then it will be scaled so as to be the same as AIPS **FILLM** (i.e. approximately in deciJanskys). Note that post-Modcomp data is in raw correlation coefficient and will be scaled using the *TSYS* values, while Modcomp-era data had this applied online. In all cases **importvla** will do the correct thing to data and weights based on an internal flag in the VLA Archive file, either scaling it or unscaling based on your choice for *applytsys*.\n",
        "    - If *applytsys=True* and you see strange behavior in data amplitudes, it may be due to erroneous *TSYS* values from the online system. You might want to then fill with *applytsys=False* and look at the correlation coefficients to see if the behavior is as expected.\n",
        "\n",
        "- Parameter *bandname*\n",
        "    - The *bandname* indicates the VLA Frequency band(s) to load, using the traditional bandname codes. These are:\n",
        "        -   '4' = 48-96 MHz\n",
        "        -   'P' = 298-345 MHz\n",
        "        -   'L' = 1.15-1.75 GHz\n",
        "        -   'C' = 4.2-5.1 GHz\n",
        "        -   'X' = 6.8-9.6 GHz\n",
        "        -   'U' = 13.5-16.3 GHz\n",
        "        -   'K' = 20.8-25.8 GHz\n",
        "        -   'Q' = 38-51 GHz\n",
        "        -   '' = all bands (default)\n",
        "    - Note that as the transition from the VLA to JVLA progressed, the actual frequency ranges covered by the bands expanded, and additional bands were added (namely 'S' from 2-4 GHz and 'A' from 26.4-40 GHz).\n",
        "\n",
        "- Parameter *frequencytol*\n",
        "   - The *frequencytol* parameter specifies the frequency separation tolerated when assigning data to spectral windows. The default is *frequencytol=150000* (Hz). For Doppler tracked data, where the sky frequency changes with time, a *frequencytol* \\< 10000 Hz may produce too many unnecessary spectral windows.\n",
        "\n",
        "- Parameter *project*\n",
        "    - You can specify a specific *project* name to import from archive files. The default '' will import data from all projects in file(s) archivefiles.\n",
        "    - For example for VLA Project AL519:\n",
        "    \n",
        "    ```\n",
        "    project = 'AL519'    #this will work\n",
        "    project = 'al519'    #this will also work\n",
        "    ```\n",
        "    \n",
        "    while *project='AL0519'* will NOT work (even though that is what queries to the VLA Archive will print it as - sorry!).\n",
        "\n",
        "- Parameters *starttime* and *stoptime* \n",
        "    - You can specify start and stop times for the data, e.g.:\n",
        "    \n",
        "    ```\n",
        "    starttime = '1970/1/31/00:00:00'\n",
        "    stoptime = '2199/1/31/23:59:59'\n",
        "    ```\n",
        "    \n",
        "    Note that the blank defaults will load all data fitting other criteria.\n",
        "\n",
        "- Parameter *autocorr* \n",
        "    - Note that autocorrelations are filled into the data set if *autocorr=True*. Generally for the VLA, autocorrelation data is not useful, and furthermore the imaging routine will try to image the autocorrelation data (it assumes it is single dish data) which will swamp any real signal. Thus, if you do fill the autocorrelations, you will have to flag them before imaging.\n",
        "\n",
        "- Parameter *antnamescheme* \n",
        "    - The *antnamescheme* parameter controls whether **importvla** will try to use a naming scheme where JVLA antennas are prefixed with EA (e.g. 'EA16') and old VLA antennas have names prefixed with VA (e.g. 'VA11'). Our method to detect whether an antenna is JVLA is not yet perfected, and thus unless you require this feature, simply use *antnamescheme='old'*.\n",
        "\n",
        "- Parameter *evlabands* \n",
        "    - The *evlabands=True* option is provided to allow users to access JVLA frequencies outside the standard VLA tunings (e.g. the extended C-band above 6 GHz).\n",
        "  \n",
        "<div class=\"alert alert-warning\">\n",
        "**ALERT:** use of this option for standard VLA data will cause unexpected associations, such as X-band data below 8 GHz being extracted to C-band (as the JVLA C-band is 4--8 GHz). Use with care.\n",
        "</div>\n",
        "\n",
        "\n",
        "**Import ATCA and CARMA data**\n",
        "\n",
        "There are several ways to import data from ATCA and CARMA into CASA. The data from these arrays has historically been processed in MIRIAD. For simple cases (single source and frequency) exporting from MIRIAD to UVFITS format and importing using **importuvfits** often works ok, although some fixes to the resulting MeasurementSet may be needed.\n",
        "\n",
        "The **importmiriad** task reads MIRIAD visibility data and can handle multiple frequencies and sources in the input. Since it does not apply any calibration, make sure to apply it beforehand in MIRIAD.\n",
        "\n",
        "The **importatca** task reads the ATCA archive format (RPFITS) directly, avoiding the need to go through MIRIAD to load the data. It can handle ATCA data from both the old and new (CABB) correlator.\n",
        "\n",
        " \n",
        "\n",
        "**Import MIRIAD visibilities (importmiriad)**\n",
        "\n",
        "The task **importmiriad** allows one to import visibilities in the MIRIAD data format to be converted to a MS. The task has mainly been tested on data from the ATCA and CARMA telescopes and the inputs are:\n",
        "\n",
        "```\n",
        "#importmiriad :: Convert a Miriad visibility file into a CASA MeasurementSet\n",
        "mirfile             =         ''        #Name of input Miriad visibility file\n",
        "vis                 =         ''        #Name of output MeasurementSet\n",
        "tsys                =      False        #Use the Tsys to set the visibility weights\n",
        "spw                 =      'all'        #Select spectral windows\n",
        "vel                 =         ''        #Select velocity reference (TOPO,LSRK,LSRD)\n",
        "linecal             =      False        #(CARMA) Apply line calibration\n",
        "wide                =      'all'        #(CARMA) Select wide window averages\n",
        "debug               =          0        #Display increasingly verbose debug messages\n",
        "```\n",
        "\n",
        "The *mirfile* parameter specifies a single MIRIAD visibility file which should have any calibration done in MIRIAD already applied to it.\n",
        "\n",
        "Set the *tsys* parameter to *True* to change the visibility weights from the MIRIAD default (usually the integration time) to the inverse of the noise variance using the recorded system temperature.\n",
        "\n",
        "The *spw* parameter can be used to select all or some of the simultaneous spectral windows from the input file. Use the default of 'all' for all the data or use e.g., *spw='0,2'* to select the first and third window.\n",
        "\n",
        "The *vel* parameter can be used to set the output velocity frame reference. For ATCA this defaults to '*TOPO*' and for CARMA it defaults to '*LSRK*'. Only change this if your data comes out with the incorrect velocity.\n",
        "\n",
        "The *linecal* parameter is only useful for CARMA data and can apply the line calibration if it is stored with the MIRIAD data.\n",
        "\n",
        "The *wide* parameter is only useful for CARMA data and can select which of the wide-band channels should be loaded.\n",
        "\n",
        " \n",
        "\n",
        "**Import ATCA RPFITS data (importatca)**\n",
        "\n",
        "The data from the ATCA is available from the archive in files in the RPFITS format. These files can be imported into CASA with the **importatca** task.\n",
        "\n",
        "```\n",
        "#importatca :: Import ATCA RPFITS file(s) to a MeasurementSet\n",
        "files               =['*.C1234']        #Name of input ATCA RPFits file(s)\n",
        "vis                 = 'c1234.ms'        #Name of output visibility file\n",
        "                                        #(MeasurementSet)\n",
        "options             =         ''        #Processing options: birdie, reweight,\n",
        "                                        #noxycorr, fastmosaic, hires, noac\n",
        "                                        #(comma separated list)\n",
        "spw                 =       [-1]        #Specify the spectral windows to use,\n",
        "                                        #default=all\n",
        "nscans              =     [0, 0]        #Number of scans to skip followed by\n",
        "                                        #number of scans to read\n",
        "lowfreq             =   '0.1GHz'        #Lowest reference frequency to select\n",
        "highfreq            =   '999GHz'        #Highest reference frequency to select\n",
        "fields              =       ['']        #List of field names to select\n",
        "edge                =          8        #Percentage of edge channels to flag.\n",
        "                                        #For combined zooms, this specifies\n",
        "                                        #the percentage for a single zoom\n",
        "                                        #window\n",
        "```\n",
        "\n",
        "The files parameter can take a string or a list of strings as input and also allows the use of wildcards as shown in the example above.\n",
        "\n",
        "For older ATCA continuum data (before the CABB correlator, April 2009) use *options='birdie,reweight'* to suppress internally generated RFI.\n",
        "\n",
        "The options parameter:\n",
        "\n",
        "-   *birdie* - (pre-CABB data only) Discard edge channels and channels affected by internal RFI.\n",
        "-   *reweight* - (pre-CABB data only) Suppress ringing of RFI spikes by reweighting of the lag spectrum\n",
        "-   *noxycorr* - do not apply the xy phase correction as derived from the switched noise calibration, by default this is applied during loading of the data.\n",
        "-   *fastmosaic* - use this option if you are loading mosaic data with many pointings and only one or two integrations per pointing. This option changes the tiling of the data to avoid excessive I/O.\n",
        "-   *hires* - use this option if you have data in time binning mode (as used for pulsars) but you want to make it look like data with very short integration time (no bins).\n",
        "-   *noac* - discard the auto-correlation data\n",
        "\n",
        "The *spw* parameter takes a list of integers and can be used to select one or more of the simultaneous frequencies. With CABB there can be up to 34 spectra. The order of the frequency bands in the RPFITS file is: the two continuum bands (0 and 1), followed by the zoom bands for the first frequency and then the zoom bands for the second frequency. Note that this *spw* parameter does not take a string with wildcards. Use *spw=-1* to get all the data.\n",
        "\n",
        "The *nscans* parameter can be used to select part of a file, e.g., to retrieve a few test scans for a quick look.\n",
        "\n",
        "The *lowfreq* and *highfreq* parameters select data based on the reference frequency.\n",
        "\n",
        "The *fields* parameter selects data based on the field/source name.\n",
        "\n",
        "The *edge* parameter specifies how many edge channels to discard as a percentage of the number of channels in each band. E.g., the default value of 8 will discard 8 channels from the top and bottom of a 2048 channel spectrum.\n",
        "\n",
        "\n",
        "**UVFITS Import**\n",
        "\n",
        "The UVFITS format is not exactly a standard, but is a popular archive and transport format nonetheless. CASA supports UVFITS files written by the AIPS FITTP task, and others.\n",
        "\n",
        "UVFITS is supported for both import and export.\n",
        "\n",
        "**Import using importuvfits** \n",
        "\n",
        "To import UVFITS format data into CASA, use the **importuvfits** task:\n",
        "\n",
        "```\n",
        "#In CASA: inp(importuvfits)\n",
        "fitsfile            =         ''  #Name of input UVFITS file\n",
        "vis                 =         ''  #Name of output visibility file (MS)\n",
        "antnamescheme       =      'old'  #For VLA only; 'new' or 'old'; 'VA04' or '04' for VLA ant 4\n",
        "```\n",
        "\n",
        "This is straightforward, since all it does is read in a UVFITS file and convert it as best it can into a MS.\n",
        "\n",
        "For example:\n",
        "\n",
        "```\n",
        "importuvfits(fitsfile='NGC5921.fits',vis='ngc5921.ms')\n",
        "```\n",
        "\n",
        "Here is a hint for handling CARMA data loaded into CASA using importuvfits:\n",
        "\n",
        "```python\n",
        "tb.open(\"c0104I/ANTENNA\",nomodify=False)\n",
        "namelist=tb.getcol(\"NAME\").tolist()\n",
        "for i in range(len(namelist)):\n",
        " name = 'CA'+namelist[i]\n",
        " print ' Changing '+namelist[i]+' to '+name\n",
        " namelist[i]=name\n",
        " \n",
        "tb.putcol(\"NAME\",namelist)\n",
        "tb.close()\n",
        "```\n",
        "\n",
        "\n",
        "**Import using importfitsidi**\n",
        "\n",
        "Some **uvfits** data is written in the FITS-IDI standard. Those files can be imported into CASA with the **importfitsidi** task:\n",
        "\n",
        "```\n",
        "#importfitsidi :: Convert a FITS-IDI file to a CASA visibility data set\n",
        "fitsidifile         =       ['']       #Name(s) of input FITS-IDI file(s)\n",
        "vis                 =          ''       #Name of output visibility file (MS)\n",
        "constobsid          =      False        #If True, give constant obs ID==0 to\n",
        "                                        #the data from all input fitsidi\n",
        "                                        #files (False = separate obs id for\n",
        "                                        #each file)\n",
        "scanreindexgap_s    =        0.0        #min time gap (seconds) between\n",
        "                                        #integrations to start a new scan\n",
        "```\n",
        "\n",
        "The *constobs* parameter can be used to give all visibilities the same observation id of 0. *scanreindexgap_s* controls the gap that defines different scans.\n",
        "\n",
        "Example:\n",
        "\n",
        "```\n",
        "importfitsidi(fitsidifile='NGC1300.fits',vis='NGC1300.ms')\n",
        "```\n",
        "\n",
        "\n",
        "\n",
        "***\n",
        "\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VjM7a1jo0ECy"
      },
      "source": [
        "## MeasurementSet Export \n",
        "\n",
        "Convert a MeasurementSet to UVFITS\n",
        "\n",
        "**Export using exportuvfits**\n",
        "\n",
        "The *exportuvfits* task will take a MS and write it out in UVFITS format. The defaults are:\n",
        "\n",
        "```\n",
        "#exportuvfits :: Convert a CASA visibility data set to a UVFITS file:\n",
        "vis                 =         ''        #Name of input visibility file\n",
        "fitsfile            =         ''        #Name of output UV FITS file\n",
        "datacolumn          = 'corrected'       #Visibility file data column\n",
        "field               =         ''        #Select field using field id(s) or field name(s)\n",
        "spw                 =         ''        #Select spectral window/channels\n",
        "antenna             =         ''        #Select data based on antenna/baseline\n",
        "timerange           =         ''        #Select data based on time range\n",
        "avgchan             =          1        #Channel averaging width (value > 1 indicates averaging)\n",
        "writesyscal         =      False        #Write GC and TY tables, (Not yet available)\n",
        "multisource         =       True        #Write in multi-source format\n",
        "combinespw          =       True        #Export the spectral windows as IFs\n",
        "     padwithflags   =       True        #Fill in missing data with flags to fit IFs\n",
        "\n",
        "writestation        =       True        #Write station name instead of antenna name\n",
        "overwrite           =      False        #Overwrite output file if it exists?\n",
        "```\n",
        "\n",
        "For example:\n",
        "\n",
        "```\n",
        "exportuvfits(vis='ngc5921.split.ms',\n",
        "fitsfile='NGC5921.split.fits',\n",
        "multisource=False)\n",
        "```\n",
        "\n",
        "The MS selection parameters *field, spw, antenna*, and *timerange* follow the [standard selection syntax](visibility_data_selection.ipynb).\n",
        "\n",
        "The *datacolumn* parameter chooses which data-containing column of the MS is to be written out to the UV FITS file. Choices are: '*data*', '*corrected*', and '*model*'.\n",
        "\n",
        "There are a number of special parameters that control what is written out. These are mostly here for compatibility with AIPS.\n",
        "\n",
        "The *writesyscal* parameter toggles whether GC and TY extension tables are written. These are important for VLBA data, and for JVLA data.\n",
        "\n",
        "<div class=\"alert alert-warning\">\n",
        "**ALERT:** The *writesyscal* option is not yet available.\n",
        "</div>\n",
        "\n",
        "The *multisource* parameter determines whether the UV FITS file is a multi-source file or a single-source file, if you have a single-source MS or choose only a single source. Note: the difference between a single-source and multi-source UVFITS file here is whether it has a source (SU) table and the source ID in the random parameters. Some programs (e.g. difmap) only accept single-source files. If you select more than one source in fields, then the *multisource* parameter will be overridden to be *True* regardless.\n",
        "\n",
        "The *combinespw* parameter allows, if some conditions are met, exporting of all spectral windows (SpW) as a set of \\\"IF\\\"s in a single \\\"FREQID\\\" setup instead of giving each SpW its own FREQID in the FITS file. In this context an IF (Intermediate Frequency) is a specialization of an SpW, where each IF in a UV FITS file must have the same number of channels and polarizations, each channel must have the same width, and each IF must be present (even if flagged) throughout the entire observation. If these conditions are not met the data must be exported using multiple FREQIDs, the UV FITS equivalent of a general SpW. This matters since many (sub)programs will work with multiple IFs, but not multiple FREQIDs. For example, a UV FITS file with multiple FREQIDs can be read by AIPS, but you may find that you have to separate the FREQIDs with SPLIT before you can do very much with them. Therefore *combinespw* should be *True* if possible. Typically MSes where each band was observed simultaneously can be exported with *combinespw=True*. MSes where the tuning changed with time, e.g. 10 minutes at 4.8 GHz followed by 15 minutes at 8.4 GHz, should be exported to multiple UV FITS files using *spw* to select one tuning (set of simultaneous SpWs) per file.\n",
        "\n",
        "The *writestation* parameter toggles the writing of the station name instead of antenna name.\n",
        "\n",
        "\n",
        "\n",
        "\n"
      ]
    }
  ]
}