Yet Another Radio data reduction Pipeline (yarp) is a modular CASA 6.0 based radio data reduction module. The current focus of this pipeline is to provide a modular environment for reduction raw interferometric radio data, mainly for HI imaging of nearby galaxies. This is mainly possible due to the purely Pythonic framework provided by CASA 6.0. This is a development version with several attempts being made to make the pipeline as modular as possible.
Currently there is no installation required for running this pipeline. Simply clone the repository and start using it. However, the following packages are required before running yarp:
casa 6.0
astropy
PyYAML
This pipeline is mainly tested on Ubuntu 18.04. Use the latest pip
:
python3 -m pip install -U pip
Then install casatools
and casatasks
following the casa 6.0 installation guide. It is also recommended to install casampi
which can speed things up.
First make sure you have openmpi. If not, install:
apt-get install -y openmpi-bin openmpi-docs libopenmpi-dev mpi
Then pip
install casampi
.
Sometimes casatools
may not install in one go. In this manually download the casatools
wheel and then install it using pip
.
A detailed documentation for yarp is under construction. Here I will give a brief overview on the usage.
Before starting anything we need to convert the LTA files to FITS using ltabin and gvfits using the following commands:
listscan <filename>_gwb.lta
After this edit the FITS column in the .log
file to a suitable name for the output filts file.
gvfits <filename>.log
If you first want to run Jayaram's flagcal
do:
flagcal fits_in=input.fits
Then finally convert the .FITS
file to .MS
using the importgmrt.py
script. First simply change the locations of the input .FITS
file and the output .ms
files and then run:
python3 importgmrt.py
Before running the yarp pipeline the most important step is to fill in the parameters.yaml
file. Here initially only fill in the msfile
and outdir
variables in the general
section. Then run:
python3 initial_inspection.py
This will create a listobs.txt
file in the outdir/listscan/
folder. Use this to finally complete the general
section in the parameters.yaml
file. Next open the yarp_planner.ipynb
to follow all the steps there. This will finally help you in filling all the sections related to the imaging, uvsubtraction and cubeimaging in the parameters.yaml
file.
yarp consists of major steps:
- initial flagging. In this step several bad antennas are flagged and the data is quacked, edge channels flagged and the entire data is tfcroped.
- flagcal. This is the main intial flagging and calibration
Each script consists of several recipes (python modules).
This script does the following steps:
- intial_inspection.py:
Right now this scripts plots the various antenna locations and saves it in the ants/ folder. It then runs the listobs and saves the output in a text files in the listobs/ folder. It then plots the Amp vs. channel data for one of the fluxcal. This can be used to select a good channel for initial flagging and calibrations. It then plots the visibilities as a function of time iterating over all antennas for each of the fluxcal and saves the output in the ants/ folder. It then asks for a user input of all the bad antennas, which it then flags.
-
flagcal.py: Done !
-
Make a test cube image or a average spec to identify line channels
-
imagecal (imaging and self-cal) Done. Several things can be changed here.
-
Keep the niter to 10000 and only use threshold to reach the required rms.
-
uvsub --> rflag --> seflcal and repeat
Play with different robust values during the selfcal loop and a different during the final image.
IMP: Check for convergence in selfcal.
Flag after applying selfcal. rflag the line free channels to 6 sigma, and the line channels to 10 sigma.
-
continuum_sub: subtracting the model using the ft task. Then do uvcontsub.
-
cube_image.py: Use sofia for masking.
The following flowchart provides a broad overview of the various steps followed by yarp.
The workflow is ready in the form of a pipeline. Use ruffus to fuse together all the scripts in a workflow.
Some of the initial developement of this pipeline took some inspiration from the CAPTURE pipeline designed for the uGMRT continuum data reduction. The extensive VLA data reduction tutorials and the Radio Astronomy School 2019 tutorials came out to be very useful in writing this pipeline. Nissim's AIPS.INFO also came out to be handy. I would also like to thank Sushma Kurapati and Biny Sebastian for useful dicussions which helped in improving this pipeline.