List additional info, including MWA-specific details

[cjordan]

At present, most (all?) of our uvfits files require an accompanying metafits file to provide enough information for calibration. We can write new HDUs to our uvfits files to provide this information, which also won't break the AIPS 117 standard and can address issues #13 and #2. In case anyone in the community already uses obvious names like "TIME" for uvfits HDUs, we should prefix our HDUs with "MWA_", i.e. "MWA_TIME".

Desired additional information:

• Unique timestamps, GPS format preferred for precision
• The time resolution of each integration. This can be supplied by the INTTIM key in the main table, but supplying this key for every row is excessive. As we don't have variable integration times in MWA data, a single key that applies to all data is useful.
• MWA tile information, including
  • whether a tile should be considered flagged or not
  • a tile's dipole delays, including 32s for dead dipoles (as per the metafits)
  • cable lengths.
• MWA coarse channel information
• Fine channel flags. At present, a channel may be flagged in an integration with a negative weight, but it is impossible to know if this is due to RFI or it's an edge channel etc. It's useful to display what channels are flagged across all integrations (e.g. because they're edge/centre channels) as most users won't want to read them, and thus hyperdrive can do the same work with less memory.

Suggested HDU names:

• MWA_TIME
• MWA_TILES
• MWA_CHANS

[d3v-null]

Just to expand on this a bit, the fields that exist in the metafits for each antenna that we don't currently store in uvfits are:

PRIMARY HDU:

• EXPOSURE, NSCANS, INTTIME - integration time, number of scans and length of the observation
• CHANNELS - coarse channel selection info
• DELAYMOD, CABLEDEL, GEODEL, CALIBDEL, DELDESC - MWAX delay mode - do we want to do anything with these?
• GRIDNAME, GRIDNUM, DELAYS - sweet-spot grid being used - is this needed?

TILEDATA HDU:

• Length - cable length (ms ANTENNA:MWA_CABLE_LENGTH)
• Rx - mwa receiver number (ms ANTENNA:MWA_RECEIVER)
• Tile - global MWA antenna identifier, not specific to obs (ms ANTENNA:MWA_TILE_NR)
• Flag - antenna flag (ms ANTENNA:FLAG_ROW)
• Slot - input slot within receiver (ms ANTENNA:MWA_SLOT)
• Gains - digital gains for each coarse channel (ms MWA_SUBBAND:GAIN)
• Flavors / BFTemps - cable flavor & beamformer temps - is this needed?
• Delays - beamformer delays - (ms MWA_TILE_POINTING:DELAYS)
• VCSOrder - Index for that correlator input in VCS data from the old correlator. - is this needed?
• Calib_Delay, Calib_Gains - rarely populated - is this needed?

having a look through aips 117, there are a couple of places where some of these values could sit:

• §4.1 AIPS AN antenna table
  • arguably the global MWA tile identifier should go in the AIPS AN:NOSTA uvfits column, which is currently populated with an antenna index that is only unique to this observation. This information is redundant, as it could be obtained from the row number + 1
  • otherwise, some of the other missing tiledata HDUs could be added as extra fields on the AN table. particularly cable length, mwa receiver number, mwa tile identifier, mwa receiver slot number
• §4.5 AIPS FG flag table:
  • this could be used to specify timestep, antenna, baseline, coarse channel, and fine channel flags
  • each row specifies a pair of antennas, a start and end time, and a start and end channel.
  • the memo says if one antenna is zero, then that row specifies flags for all baselines for that antenna, and if both are zero, that means all baselines
  • we could make use of the REASON field to make filtering entries in the table easier, like using certain values to differentiate between rows that specify channel flags and antenna flags
  • we should use a convention for the start and end times to mean "all times in this observation", like start time=0, end time= floating point max, or just use the start and end values from the DATE column.
  • we could specify the flagged edge channel bandwidth in the header, and whether the center channel is flagged on each coarse channel.
• §4.11 AIPS NX index table
  • the visibility index number of the start and end of the timesteps for each visibility could be written into this column, which has the potential to speed up read times (for software that implements this table), and could easily be used to extract a list of timesteps.
  • an additional column could be used to store the GPS time as well.

Storing coarse channel info with frequency setup ids

at some point we'll need to deal with picket fences, birli#13 , mwa_hyperdrive#13, and I can see three options:

1. multiple files for each contiguous group of coarse channels
2. multiple spectral windows for each contiguous group of coarse channels using uvfits "frequency setup IDs" aka FREQID, aka FREQSEL, aka IFs
3. one spectral window for each coarse channel, as first discussed here

There are several tables in AIPS117 that store information for alongside each frequency setup id. If we solve picket fences with (3), then we can use these tables to store coarse channel information along with the frequency setup ids, which we can't do with (3). Looking at these tables, it seems like the MWA concept of coarse channels maps very neatly onto the aips concept of spectral windows.

• §4.7 AIPS FQ frequency table:
  • this table is required to implement multiple spectral windows properly, with either (2) or (3).
  • hypothetically, if we implemented (3), we could have the option to not write edge channels to disk at all, saving about 12.5% in file size for obs with 80kHz edges.
  • doing this gives each coarse channel a "frequency setup ID number", which would allow us to specify information about each coarse channel in other tables like the ones below.
  • we could add information like mwa coarse channel number to this table
• §4.8 AIPS GC gain curve
  • each row specifies an antenna number, freq id and gain for each feed. Probably suitable for Calib_Gains
• §4.15 AIPS PC phase cal table
  • there is a cable delay field in this table for each antenna, probably suitable for Calib_Delay
• §4.18 AIPS SY sys power table
  • there is space for a separate gain for each freq id, antenna number and polarization that could be used to store digital gains, although MWA digital gains really deserves its own separate table, as there isn't a clear mapping for it in AIPS world.
• §4.19 AIPS TY system temperature table
  • bftemps could be stored in TSYS

We don't have to solve spectral windows with (3), and with (2) we could still store some of the coarse channel info using overlapping spectral windows (a few spectral windows that we actually use, then one for each coarse channel with the mwa info) but I think (3) is a pretty neat way of solving multiple problems at once.

[cjordan]

SDC3 measurement sets include a ARRAY_CENTER column in the OBSERVATION table, which give the "array position" as a geocentric coordinate. This would be useful to include in our own MSs.

[d3v-null]

I'd really like to get fine channel flag info from Birli into hyperdrive, but none of the options available in AIPS117 are particularly straightforward:

• AIPS FQ table: specifying the edge width of each coarse channel where each coarse channel is a separate spectral window.
• AIPS FG table:
  • a start and end time, and a start and end channel. for each fine channel flag.
  • specify the flagged edge channel bandwidth in the AIPS FG header, and whether the center channels are flagged

Any ideas?

[cjordan]

Looking at the standard, I agree that your proposal is probably the best approach if we were to stick to the standard. However, I suspect you're on the same page as me in that having a separate spectral window for each coarse channel adds a lot of complexity for little gain. I'm not opposed to multiple spectral windows, but I'd prefer to see it for our picket fence obs.

Personally, I think the easiest and best solution would be to have our own HDUs (e.g. MWA_CHAN which is a table containing flagged channel indices). Any of these non-standard HDUs would strictly be optional; we'd just take advantage of their info if it's available.