Home

Welcome to the dataqa wiki!

Scripts for data QA

Wrapper scripts for specific data QA should live at the top level. There are separate directories for scripts/code related to each type of data QA. E.g.

run_crosscalQA.py
crosscal
->ccal_plots.py

Procedure

At any time during the process you can run create_report.py <obs_id> to create the QA overview

1. Initial setup

   1. Log-in to Happili-01
   2. Open a screen session (optional, but recommended)
      • Example: screen -S dataqa
   3. Set environment for apercal (if not already done)
      • See https://github.com/apertif/apercal/wiki/Running-apercal-on-happili

2. Run crosscal qa

   • Example: python run_ccal_plots.py <scannumber> <fluxcal> [-p --/path/to/write/images/too]

3. Run selfcal qa

   • Example: python run_scal_plots.py <obs_id> <target>

4. Run continuum qa

   • Example: python run_continuum_validation.py <obs_id>
   • The script creates a directory for each beam in /data/apertif/<TASKID>/qa/continuum/<beam>/. Otherwise, there would be too many files in a single directory.

5. Run line qa

   • Example: python run_cube_stats.py <obs_id>

6. Subtract continuum from line cube

   • Example: python subtract_continuum.py <obs_id>
   • It creates a new fits file: /data*/apertif/<obs_id>/<beam>/line/cubes/HI_image_cube_contsub.fits
   • contsub.src is the bash version of subtract_continuum.py

7. Run mosaic qa

   1. Run mosaic

      • python make_mosaic_image.py <obs_id>
      • This script will create a directory: /data/apertif/<obs_id>/mosaic/ and place the mosaic image there (miriad image + fits image).
      • Good to know:
      • If not all of the beams have a continuum image there will be an error message, but the script will still run and create a mosaic from the beams that have a continuum image (fits file).
      • The script breaks if there are no beams at all to mosaic.
      • Other issues could be if the obs_id is too long. eg. more then 15 characters (regular obs_ids are 9). In this case miriad protests and the script breaks.

   2. Run validation

      • Example: `python run_continuum_validation.py <obs_id> --for_mosaic

8. Put relevant information into Google-Doc