FlagHandling
ImageLink(catflag.jpg,height=64)
The aips++ MS definition specifies a boolean FLAG column containing data flags. This provides one boolean flag per correlation per channel per "row" (baseline+timeslot). There is also a FLAG_ROW column that is meant as a shortcut for flagging an entire row. Note that these flags are common to all data columns (DATA, MODEL_DATA, CORRECTED_DATA, etc.)
During reduction, having only a single set of flags can become restrictive. It is often desirable to have several distinct sets of flags, with the ability to enable or disable whole sets. For example, when experimenting with residual flagging, it is easy to go overboard and flag too much. By storing residual flags in a separate "flagset", one can easily roll back from this situation simply by removing the "overzealous" flagset.
Aips++ made a nod towards multiple flagsets by providing a FLAG_CATEGORY column, but in reality tools to manipulate this are non-existent. Tools such as the imager honour the FLAG+FLAG_ROW columns, but are not aware of anything else.
MeqTrees works with "bitflags", a-la NEWSTAR. That is, flags are represented by a 32-bit "flagword". Each bit can in principle be treated as a separate flagset, and trees may be told to only honour specific bits in the flagword (via a "flagmask").
Bitflags may also be stored in an MS, in a column called BITFLAG (and BITFLAG_ROW). This is a non-standard extension: this column is not recognized by any aips++ tools, but neither does it get in their way. When processing an MS with Calico, you have the option of copying the existing FLAG column (called "legacy flags") into a bit of the flagword. This will initialize a BITFLAG column, if one doesn't already exist. When reading an MS, you have the option of using both legacy flags, and bitflags (if a BITFLAG column exists.)
Of course, the imager will only use the "legacy" FLAG column. If you have some bitflags that you would like to be applied during imaging, you need to copy the bitflags (or only designated bits of these, via a flagmask) back to the FLAG column first.
Calico takes the bitflag concept one step further, by forcing the user to assign names to individual bits of the flagword. In fact, it completely hides the concept of "bits" -- the user works only in terms of named flagsets. A flagset corrsponds to some bit of the flagword, but you don't need to know which, since Calico keeps track of the name-to-bit-number mapping automatically (via keywords in the MS BITFLAG column.)
All Calico scripts have a common "Data selection & flag handling" menu in their runtime options screen. This provides the following options:
- "Read flags from MS" enables the reading of flags. This has a number of sub-options:
- "Include legacy FLAG column" will include the "legacy" flags, if any.
- "Include flagset XXX" will apply the flags contained in that flagset. If no bitflags/flagsets have been defined, these sub-options do not show up.
- "Write output flagset" enables the writing of flags. This option only shows up if a script actually generates new flags (currently, this only happens in calico-view-ms, when converting legacy flags to a bitflag). This has a number of options:
* "Output flagset": you may write the new flags to an existing flagset, or create a new flagset by specifying a new name (a new bit in the flagword will be allocated automatically)
* "Output flagset policy" determines what happens when writing new flags to an existing flagset. "Add to set" means flags will be logically-ORed with the flagset. "Replace set" means the entire flagset will be replaced. "Replace all sets" means, in addition, that all other flagsets will be cleared.
- "Fill legacy FLAG column", if enabled, overwrites the legacy FLAG column with a combination of bitflags. It has a number of sub-options, one per each known flagset. Output FLAG will be True if it is set in ANY of the selected flagsets, or False if it is clear in ALL selected flagsets.
The "Quick Look" calico-view-ms.py script also doubles as a flag management tool. A number of runtime options is provided for this. There's a number of flag-related activities that you might use this script for, these will be listed below.
This is something you do when you start with a new MS that has flags, or every time you generate new flags with, e.g., the aips++ autoflag tool.
- Enable the "Convert legacy FLAG column into output flagset" option
- Enable "Read flags from MS" and, inside that, "Include legacy FLAG column".
- Enable "Write output flagset". Under "Output flagset", enter a new name for the flagset (or select an existing one, if you want to replace it.)
Now select the "View MS" job. Once the job has executed, you're done.
This is something you do when you want to apply one or more flagsets to, e.g., the imaging step. Remember that the imager only recognizes the standard FLAG column. If you want to try imaging with flagsets in effect, you must convert these flagsets into the FLAG column:
- Disable the "Convert legacy FLAG column into output flagset" option
- Enable the "Fill legacy FLAG column" option. Within the submenu for this option, make sure that the flagsets you want converted are enabled. If multiple flagsets are enabled, the result is a logical-OR (i.e. union) of the flagsets. Now select the "View MS" job. Once the job has executed, you're done.
Typically, you would do this after an extra round of flagging. For example, one you've run aips++ autoflag, you probably want to see the effects (before you enshrine the flags into a new flagset). To view the effect of flags, you will need to open a few bookmarks (see above). Now,
- Disable the "Convert legacy FLAG column into output flagset" option
- Disable the "Fill legacy FLAG column" option
- Enable the "Read flags from MS" option, and within it, enable the flagsets that you want to see. For example, to see standard FLAGs only, enable "legacy FLAG column" and disable all flagsets. Now select the "View MS" job, and watch the visualizations. By default, flagged data is omitted from the time track plots (and shown as black boxes on the time-freq images). You can change this via the plot's right-click menu.
Currently, no facility for deleting a flagset is provided. You can always cause a flagset to be ignored by un-clicking it within the "Read flags from MS" menu.