Flagging the data

Initial flagging of the data (init_flag)

This step runs a plethora of initial flagging routines that are usually adopted for VLBI observations. These include flagging of RFI (through the aoflagger software), flagging of edge channels, quacking of data (where antennae are not on source yet) and the option of manual flagging.

Parameters

    "init_flag":{
        "AO_flag_strategy"  :  ["VLBI_pipeline/aoflag_strategies/4C.rfis","VLBI_pipeline/aoflag_strategies/faint_sources.rfis"], #Aoflagger rfi strategies to be used
        "AO_flag_fields"    : [["4C39.25","J1011+0106"],["COSMOS-POINT"]], #Fields to be flagged with strategies (in this case '4C39.25" & "J1011+0106" flagged by 4C.rfis and "COSMOS-POINT" with faint_sources.rfis strategy)
        "flag_edge_chans":{
            "run"           :  false, #if true, flag the edge channels
            "edge_chan_flag":  "5%" #percentage of the edge channels to be flagged
        },
        "flag_autocorrs"    :  false, #if true, flag autocorrelations (if you want to run step `scalar_bandpass` keep set to false
        "quack_data":{
            "run"           :  false, #if true, remove start/end of scanes
            "quack_time"    :  {"*":5.0}, #dictionary corresponding to {"field":time}. "*" = all fields. Time is in seconds.
            "quack_mode"    :  "beg" #which part of the scan to flag ("beg" = beginning, "end" = end or "both" = both start and end)
        },
        "manual_flagging":{
            "run"           :  false, #if true, use "flag_file" to manually flag data
            "flag_file"     :  "manual_flagging_bd228a.txt" #flagging file name (CASA flagdata list format)
        },
        "hpc_options":{
            "partition"     :  "default",
            "walltime"      :  "default",
            "nodes"         :   -1,
            "cpus"          :   -1,
            "mpiprocs"      :   -1,
            "nodetype"      :  "default"
        }

Recommended use: This step is a little more involved that the others so the following sub-sections will advise you on the recommended parameters to set:

Removing radio frequency interference with AOFlagger

The pipeline uses the auto-flagger, AOFlagger (Offringa+07; https://sourceforge.net/p/aoflagger/wiki/Home/), to flag these data. At the moment, AOFlagger is only supported for versions < 3 but will be updated soon.

This flagger uses automated strategies to flag RFI interference and you may want to develop your own strategies for your own data sets. To do this, look at the AOFlagger documentation (link is above). Once a strategy file has been made, you can input these into the parameters (through AO_flag_strategy) to execute the strategy on the data. There are a couple of pre-made strategies within the pipeline (within aoflag_strategies/) which should generally be used for the following situations (but are not-optimal!):

• 4C.rfis - bright sources (such as fringe-finders / >100 mJy> flux densities)
• faint_sources.rfis - faint sources (<20 mJy)

These strategies can be mapped to specific fields through the use of the AO_flag_fields parameter. For example in the following situation,

"AO_flag_strategy"  :  ["VLBI_pipeline/aoflag_strategies/4C.rfis","VLBI_pipeline/aoflag_strategies/faint_sources.rfis"], 
"AO_flag_fields"    : [["4C39.25","J1011+0106"],["COSMOS-POINT"]]

The strategy 4C.rfis will execute on the fields 4C39.25 and J1011+0106, and faint_sources.rfis will execute on the field COSMOS-POINT.

Flagging edge channels

Due to the various filters, VLBI arrays often require the very edge channels of each spectral window to be flagged as they can become massively mis-scaled when the bandpass correction is applied. The pipeline can do this automatically through the "edge_chan_flag" parameter by either taking a string of the format XX% which will flag a percentage of channels off each end (so if 5% is specified, this will flag both ends by 5%, so 10% in total); or it can take an integer which will flag the number of channels from each end.

This will flag all telescopes by the same amount so it may not be useful if you want to flag different amounts on each antenna. E.g., the e-MERLIN telescopes have flat bandpasses across the bandwidth so flagging the end channels will be flagging good data. In this case, you probably should not run this step and instead include the variable flagging of the end channels in the manual flagging list.

Quacking data

The hilariously named 'quacking' of data regards the times where the antennae are still coming / leaving the source but data is still recording. This produces a period where the data goes from 0 to the source flux density and this period should be flagged. This can be run in the pipeline by setting the quack_time:{"X":XX.X} and "quack_mode":'XXX' parameters. The dictionary in quack_time states what fields must be flagged and by how many seconds from either the start ("quack_mode":'XXX'), end ("quack_mode":'end'), or both ends ("quack_mode":'both') of the scan.

To see how much time needs to be flagged, plot a few scans of the bright phase calibrator and see whether there is low amplitudes at the start or end of the scan. Again, this will flag the telescopes all the same, but some telescopes have longer slewing times and so may come onto source later. In this case, you should input the quacking commands in the manual flagging list.

Making a manual flag file

All of the previous steps are not perfect at finding and removing the bad data and/or RFI. In this case, it may be worth putting together a manual flag file which can remove the remaining bad data. You will have to identify the bad data yourself (typically using a visualisation software such as plotms or shadems) and then produce a text file containing the commands. The way to put together these lists can be found on the CASA flagdata page under the mode='list' part.

Expected inputs

• <project_code>.ms
• AOflagger strategies (some located in VLBI_pipeline/aoflagger_strategies)

Expected outputs

• <project_code>.ms with adjusted FLAG column