Specifications

Radio Astronomy Gains and Visibility Inspector [ragavi]

A python based software to aid in the visualisation of radio astronomy data reduction by-products: gains and visibilities that can be used within a notebook environment (Jupyter) or as a command line tool.

Visibilities

Axes supported by the visibility plotter are as listed. In brackets are the arguments to input.

For x axis:

• Antenna1 (antenna1)
• Antenna2 (antenna2)
• Frequency (frequency)
• Real (real)
• Scan (scan)
• Time (time)
• Uvdistance metres (uvdistance)
• Uvdistance lambda (uvwave)

For y-axis:

• Amplitude (amplitude)
• Phase (phase)
• Real (real)
• Imaginary (imaginary)

With iterations over

• Correlations (corr)
• Scan (scan)
• Spectral windows (spw)

Specialised Plots*

• Antenna1 vs Antenna2 grids
• Scan vs frequency chunk grids

Visibility plotter

Currently only run in command line mode and currently does not require the installation of ragavi as they are not yet integrated. You must however install the requirements from the requirements.txt file in some virtual environment.

Running python visibilities.py -h brings up a help output that you may use also You do however need the script which can be found by cloning the branch here.

Required arguments are:

• table
• field
• xaxis
• yaxis

The rest are optional.

As an example run in the terminal

$ python visibilities.py --table /path/to/ms --field field_name_or_field_id --xaxis time --yaxis real --cmap seismic --iterate scan

Specialised Plots*

Ragavi aims to support the visualisation of some plots which will provide as summary of information on the progression of the pipeline. They include:

• Antenna1 vs Antenna2 grid whose summary plot show the mean and standard deviation of each baseline as a function of time in one diagonal and a function of frequency in the other. On clicking any item on this grid, a waterfall plot, showing variations in in both time and quantity of the measured datum is displayed. This is currently limited to amplitude and phase data.

• Scan vs Frequency chunk grid. In this plot, the summary is represented by 2 circles; one showing the variance of the data in that frequency chunk and scan while the other shows the percentage of data that is flagged in that particular chunk and scan. On clicking any element in this grid, and amplitude vs Phase plot is displayed. This plot is useful as it provides some insight on the calibration process by showing how much amplitude varies with phase. An ideal calibration process should result in a plot that is mostly centered around 0.

Currently, there are two separate, rudimentary and experimental scripts that are used to generate these plots.

1. bk_ant1_vs_ant2.py
2. bk_scan_vs_chunk.py

These however, require the use the bokeh server to achieve synchronisation between the plot and what is displayed and allow the generation of a new plot on demand. These plots have been separated from the visibility plotter because they require slightly more work to set up and require bespoke layouts and thus integration has not been accomplished yet meaning that in order to run them, one must edit the script itself as it does not take in any command line arguments yet, a feature that is due to be introduced because of the nature of use of the tools.

For Scan vs Frequency chunk the following should be edited to get script working:

• bin_width: Size of bins within which frequencies will be chunked and averaged together. Determines the number of chunks that are generated.
• field_id: The field whose data is to be processed.
• ms_name: path to the ms you want to visualise.
• corr: Correlation index that needs to be selected.

Antenna1 vs Antenna2:

• data_col: column from which data will be extracted.
• ddid: data description id/ spectral window.
• doplot: whether amplitude/ phase or real/imaginary
• fid: the field whose data is to be processed.
• ms_name: path to the ms you want to visualise.
• corr: Correlation index that needs to be selected.
• yaxis: Actual quantity to display i.e either amplitude or phase or real or imaginary.

An example to launch one of the scripts using the bokeh server:

bokeh serve bk_ant1_vs_ant2.py --port 8888

This will startup the server but will not open a browser for you.

• To open a page in a browser automatically then add --show to the previous command.

• To serve the file in development mode, i.e change and serve the file without having to restart the server, then add --dev flag to the command

Specifications

---

Scan vs Frequency Chunk Plots

• Create Grid of number_of_scans x number_of_frequency_bins
• On click generates amplitude vs Phase plot for that element (scan, frequency chunk) in the grid.
• Represent variance as a circle of radius==variance
• Represent percentage deviation from central mean with a color + a colour bar
• Represent percent of data flagged, with circle overlaid on that of variance.
• On tooltip, display:
  • Percentage of data flagged
  • Mean on current grid element

Antenna1 vs Antenna2

• Create a grid matrix of n_antennas x n_antennas
• Each element of that grid to have a plot showing mean and sigma.
• Lower diagonal to display mean and sigma over time.
• Upper diagonal to display mean and sigma over frequency.
• Display the means using scatter plots
• Display sigma using a filled band area.
• On click, display a waterfall plot of amplitude / phase showing variation over time and frequency.
• Plots for amplitude and phase are separated

Implementation

---

For a general description on how the scripts were implemented.

To generate scan vs frequency chunks:

• Using xarray-ms, group the data based on DATA_DESC_ID and FIELD_ID. This outputs a list of length N_unique_data_desc_ids x N_unique_field_ids.
• While Iterating over this list:
  • Check if that field_id == field id in current chunk. If so:
  • Preprocess the data, calculate amplitude, phase etc. and store the data in an xarray data set
    • Chunk the data into the desired chunk sizes
    • Convert the xarray data set into a dask data frame. This will result in a data frame containing the number of partitions that result from the chunking process.
    • Create a new column that contains assigned chunk numbers to each of the partitions. This can be done easily now because once data is chunked in the xarray data set, the number of partitions created there, is propagated to the dask data frame.
    • Group the data frame based on the chunk number and SCAN_NUMBER. This should result in a data frame with number_of_scans x number_of_frequency_chunks groups.
    • Apply reduction functions to the resulting groups. This results in the calculation of the mean and sigma per scan per frequency chunk meaning that there will be redundant values for mean and sigma in each groups.
    • Drop the rows with duplicate values in the mean data column. We then end up with the number of rows in the column corresponding to the exact number of groups described previously. We use this column because the mean of an entire group will be the same.
    • Iterate over these rows plotting the required parameters.

To generate antenna1 vs antenna2:

• Using xarray-ms, chunk the data based on DATA_DESC_ID, FIELD_ID, ANTENNA1 and ANTENNA2. This outputs a list of length N_baselines x N_unique_field_ids x N_unique_data_desc_ids.

• Iterate over this list once to extract items and store from for the specified field_id and data_desc_id.

• Create a numpy grid array of n_antennas x n_antennas to allow for easier organisation of the grid elements

• While iterating over the newly created list:

  • Process the data for amplitude / phase / real / imaginary.
  • Perform and compute reductions for mean, standard deviation over both time and frequency on this data.
  • Setup plot figure for current baseline
  • Create a plot of the mean, mean $\pm$ sigma over time.
  • Insert this plot in the grid position [current_antenna1, current_antenna2].
  • Create a plot of the mean, mean $\pm$ sigma over frequency.
  • Insert this plot in the grid position [current_antenna2, current_antenna1].
  • Attach "on click" callback functions to these plots

• Convert the grid array to a list

• Pass the list to a bokeh layout component for display.