Updates for 18.11

[weaverba137]

This PR (which is for 18.11 despite the name of the branch):

• Fixes update datamodel for software release 18.11 #64.
• Allows the use of 'maggie'-based units in FITS files without excessive complaining.
• Incorporates changes from the 'mockobserving' branch.

Note the 'WIP' label!

[weaverba137]

@sbailey, please look over this initial assessment of the files in 18.11. I'm taking care of the straightforward things like adding new columns, but there are some structural problems in here too that need examination.

• DESI_SPECTRO_DATA
  • NIGHT/EXPID/fibermap
    • File has no EXTNAME in HDU 0.
    • File is missing extension TARGETS.
    • Due to an astropy bug 10**6 arcsec-2 yr2 is not properly parsed. It can be changed to 10**+6 arcsec-2 yr2 as a work-around.
• DESI_SPECTRO_REDUX
  • SPECPROD/zcatalog
    • File has no EXTNAME in HDU 0.
    • Is TEMVER00 supposed to be a float? String seems more appropriate.
  • SPECPROD/exposures/NIGHT/EXPID/cframe
    • In the SCORES HDU the names of the columns depend on the name of the file (BRZ). This is kind of annoying.
  • SPECPROD/spectra-NSIDE/PIXGROUP/PIXNUM/spectra
    • File has no EXTNAME in HDU 0.
    • I thought we got rid of compressed MASK HDUs. This mismatch actually causes the check_model program to crash!
• DESI_SPECTRO_SIM
  • PIXPROD/NIGHT/EXPID/simspec
    • File contains units in the form 1e-17 erg / (Angstrom cm2 s). Should be 10**-17 erg/(s cm2 Angstrom).
    • Add descriptions of new tables.
    • Do the WAVE_[BRZ] HDUs have units?
• DESI_TARGET
  • mtl
    • File has no EXTNAME in HDU 0.
    • Check HDU 1 columns.
  • sky
    • File has no EXTNAME in HDU 0.
    • File sky.fits no longer has a data model!
  • stdstar
    • File has no EXTNAME in HDU 0.
    • File standards-(bright|dark).fits no longer has a data model!
  • targets
    • File has no EXTNAME in HDU 0.
    • Check HDU 1 columns.
  • truth
    • File has no EXTNAME in HDU 0.
    • Check number of HDUs.
  • fiberassign/tile
    • I can't remember: which of the three tile files in DESI_TARGET/fiberassign are we supposed to be using?
    • File has no EXTNAME in HDU 0.
    • Check number of HDUs.

[weaverba137]

This is particularly annoying: many files have fibermap tables, but they are not all the same! For example zbest and spectra files add NIGHT, EXPID and TILEID to the fibermap table.

[sbailey]

Thanks for the detailed list. Many of these are in the category of prescriptive (what the files should have) vs. descriptive (what the 18.11 reference run files do have). I've tried to add comments with the most egregious cases (missing HDUs, missing columns).

Since we're imminent on the 18.12 release, I'm going to call this good enough directionally correct progress for 18.11. Test pass on my laptop but not yet on Travis. After resolving that I'll merge and tag for 18.11 and then continue with cleanup for 18.12 and beyond (including opening a few tickets for some specific cases).

[sbailey]

Good enough for 18.11, moving on.