Merge pull request #52 from desihub/18.6

updates for 18.6 software release

doc/DESI_SPECTRO_DATA/NIGHT/fibermap-EXPID.rst → doc/DESI_SPECTRO_DATA/NIGHT/EXPID/fibermap-EXPID.rst

@@ -9,9 +9,9 @@ fibermap-EXPID.fits
 :File Type: FITS, 1 MB
 
 This table is also propagated forward to the
-:doc:`frame <../../DESI_SPECTRO_REDUX/SPECPROD/exposures/NIGHT/EXPID/frame-CAMERA-EXPID>`,
-:doc:`cframe <../../DESI_SPECTRO_REDUX/SPECPROD/exposures/NIGHT/EXPID/cframe-CAMERA-EXPID>`, and
-:doc:`spectra <../../DESI_SPECTRO_REDUX/SPECPROD/spectra-NSIDE/PIXGROUP/PIXNUM/spectra-NSIDE-PIXNUM>`
+:doc:`frame <../../../DESI_SPECTRO_REDUX/SPECPROD/exposures/NIGHT/EXPID/frame-CAMERA-EXPID>`,
+:doc:`cframe <../../../DESI_SPECTRO_REDUX/SPECPROD/exposures/NIGHT/EXPID/cframe-CAMERA-EXPID>`, and
+:doc:`spectra <../../../DESI_SPECTRO_REDUX/SPECPROD/spectra-NSIDE/PIXGROUP/PIXNUM/spectra-NSIDE-PIXNUM>`
 files.
 
 Contents
@@ -115,6 +115,6 @@ propagated from the tractor / sweep files.
 *e.g.* ``MAG[5] -> FLUX_G, FLUX_R, FLUX_Z, FLUX_W1, FLUX_W2``.
 
 This format is out of sync with the
-:doc:`tile <../../DESI_TARGET/fiberassign/tile>` datamodel, and will be updated to
+:doc:`tile <../../../DESI_TARGET/fiberassign/tile>` datamodel, and will be updated to
 be consistent for values that are propagated forward from fiber assignment.
 *e.g.* ``POSITIONER -> LOCATION``.

doc/DESI_SPECTRO_DATA/NIGHT/EXPID/focus-EXPID.rst

@@ -0,0 +1,41 @@
+===========
+focus-EXPID
+===========
+
+:Summary: Placeholder datamodel for the focus GFA raw data
+:Naming Convention: ``focus-EXPID.fits.fz``, where EXPID is the zero-padded
+    8-digit exposure ID.
+:Regex: ``focus-[0-9]{8}\.fits.fz``
+:File Type: FITS, 600 MB
+
+Contents
+========
+
+Details are TBD, but the nominal format is:
+
+====== ========== ================= =============================
+Number EXTNAME    Type              Contents
+====== ========== ================= =============================
+HDU0   FOCUS      IMAGE             Header keywords only
+HDU1   FOCUS1     Compressed IMAGE  Focus GFA frames from petal 1
+HDU2   FOCUS1T    BINTABLE          Metadata about FOCUS1 frames
+HDU3   FOCUS3     Compressed IMAGE  Focus GFA frames from petal 3
+HDU4   FOCUS3T    BINTABLE          Metadata about FOCUS3 frames
+HDU5   FOCUS6     Compressed IMAGE  Focus GFA frames from petal 6
+HDU6   FOCUS6T    BINTABLE          Metadata about FOCUS6 frames
+HDU7   FOCUS8     Compressed IMAGE  Focus GFA frames from petal 8
+HDU8   FOCUS8T    BINTABLE          Metadata about FOCUS8 frames
+====== ========== ================= =============================
+
+The FOCUSn data will be 3D[nframes, ny, nx] such that
+`data[i]` is the 2D GFA frame number `i`.  Row `i` of the
+FOCUSnT table will contain the metadata about that frame, e.g. the
+DATE-OBS and EXPTIME.
+
+Note that other than the blank data primary HDU, the order of the other
+HDUs is arbitrary and some FOCUSn(T) HDUs may even be missing.  The
+nominal set (1,3,6,8) is the plan for full DESI, but particularly during
+commissioning other combinations will appear in the data.
+
+Other than the name and number of the HDUs, the structure of this format
+is identical to the guide GFA raw data.

doc/DESI_SPECTRO_DATA/NIGHT/EXPID/guide-EXPID.rst

@@ -0,0 +1,45 @@
+===========
+guide-EXPID
+===========
+
+:Summary: Placeholder datamodel for the guider GFA raw data
+:Naming Convention: ``guide-EXPID.fits.fz``, where EXPID is the zero-padded
+    8-digit exposure ID.
+:Regex: ``guide-[0-9]{8}\.fits.fz``
+:File Type: FITS, 1 GB
+
+Contents
+========
+
+Details are TBD, but the nominal format is:
+
+====== ========== ================= =============================
+Number EXTNAME    Type              Contents
+====== ========== ================= =============================
+HDU0   GUIDE      IMAGE             Header keywords only
+HDU1   GUIDE0     Compressed IMAGE  Focus GFA frames from petal 0
+HDU2   GUIDE0T    BINTABLE          Metadata about FOCUS0 frames
+HDU3   GUIDE2     Compressed IMAGE  Focus GFA frames from petal 2
+HDU4   GUIDE2T    BINTABLE          Metadata about FOCUS2 frames
+HDU5   GUIDE4     Compressed IMAGE  Focus GFA frames from petal 4
+HDU6   GUIDE4T    BINTABLE          Metadata about FOCUS4 frames
+HDU7   GUIDE5     Compressed IMAGE  Focus GFA frames from petal 5
+HDU8   GUIDE5T    BINTABLE          Metadata about FOCUS5 frames
+HDU9   GUIDE7     Compressed IMAGE  Focus GFA frames from petal 7
+HDU10  GUIDE7T    BINTABLE          Metadata about FOCUS7 frames
+HDU11  GUIDE9     Compressed IMAGE  Focus GFA frames from petal 9
+HDU12  GUIDE9T    BINTABLE          Metadata about FOCUS9 frames
+====== ========== ================= =============================
+
+The GUIDEn data will be 3D[nframes, ny, nx] such that
+`data[i]` is the 2D GFA frame number `i`.  Row `i` of the
+GUIDEnT table will contain the metadata about that frame, e.g. the
+DATE-OBS and EXPTIME.
+
+Note that other than the blank data primary HDU, the order of the other
+HDUs is arbitrary and some GUIDEn(T) HDUs may even be missing.  The
+nominal set (0,2,4,5,7,9) is the plan for full DESI, but particularly during
+commissioning other combinations will appear in the data.
+
+Other than the name and number of the HDUs, the structure of this format
+is identical to the focus GFA raw data.

doc/DESI_SPECTRO_DATA/NIGHT/EXPID/index.rst

@@ -0,0 +1,16 @@
+=====
+EXPID
+=====
+
+``EXPID`` is the 8-digit zero-padded exposure ID.
+
+Each exposure id (expid) generates multiple files:
+
+.. toctree::
+   :maxdepth: 1
+
+   desi-{expid}.fits.fz : Raw data <desi-EXPID>
+   fibermap-{expid}.fits : Table of which fibers were assigned to which targets <fibermap-EXPID>
+   focus-{expid}.fits.fz : Placeholder for focus GFA raw data <focus-EXPID>
+   guide-{expid}.fits.fz : Placeholder for guide GFA raw data <guide-EXPID>
+   sky-{expid}.fits.fz : Placeholder for guide GFA raw data <sky-EXPID>

doc/DESI_SPECTRO_DATA/NIGHT/EXPID/sky-EXPID.rst

@@ -0,0 +1,27 @@
+=========
+sky-EXPID
+=========
+
+:Summary: Placeholder datamodel for the sky monitor raw data
+:Naming Convention: ``sky-EXPID.fits.fz``, where EXPID is the zero-padded
+    8-digit exposure ID.
+:Regex: ``sky-[0-9]{8}\.fits.fz``
+:File Type: FITS, 350 MB
+
+Contents
+========
+
+Details are TBD, but the nominal format is:
+
+====== ========== ================= =========================
+Number EXTNAME    Type              Contents
+====== ========== ================= =========================
+HDU0   SKY        IMAGE             Header keywords only
+HDU1   SKYCAMERA  Compressed IMAGE  Sky camera frames
+HDU2   SKYCAMERAT BINTABLE          Metadata about each frame
+====== ========== ================= =========================
+
+The SKYCAMERA data will be 3D[nframes, ny, nx] such that
+`data[i]` is the 2D sky camera frame number `i`.  Row `i` of the
+SKYCAMERAT table will contain the metadata about that frame, e.g. the
+DATE-OBS and EXPTIME.

doc/DESI_SPECTRO_DATA/NIGHT/index.rst

@@ -3,13 +3,11 @@ NIGHT
 =====
 
 ``NIGHT`` is the night of observation in YYYYMMDD format.  The "night" roles
-over at noon local time, so all data taken between sunset and sunrise belong to the same night (i.e. the date of the sunset).
-
-Each exposure id (expid) has two files:
+over at noon local time, so all data taken between sunset and sunrise
+belong to the same night (i.e. the date of the sunset). Under each night, data
+are grouped in subdirectories by exposure ID (zero-padded 8-digit).
 
 .. toctree::
    :maxdepth: 1
 
-   desi-{expid}.fits.fz : Raw data <desi-EXPID>
-   fibermap-{expid}.fits : Table of which fibers were assigned to which targets <fibermap-EXPID>
-   gfa-{expid}.fits : Placeholder for GFA data <gfa-EXPID>
+   EXPID/index.rst

doc/DESI_SPECTRO_DATA/index.rst

@@ -6,7 +6,9 @@ DESI_SPECTRO_DATA
 The canonical location is ``$DESI_ROOT/spectro/data``, but one can set the
 environment variable :envvar:`DESI_SPECTRO_DATA` to point anywhere.
 The exposures are grouped by night as a YEARMMDD string.  The "night" roles
-over at noon local time, so all data taken between sunset and sunrise belong to the same night (i.e. the date of the sunset).
+over at noon local time, so all data taken between sunset and sunrise belong
+to the same night (i.e. the date of the sunset).  Under each night, data
+are grouped in subdirectories by exposure ID (zero-padded 8-digit).
 
 Subdirectories:
 

doc/DESI_SPECTRO_REDUX/SPECPROD/exposures/NIGHT/EXPID/cframe-CAMERA-EXPID.rst

@@ -22,7 +22,7 @@ HDU1_  IVAR       IMAGE    Inverse variance, ``(erg/s/cm2/A)^-2``
 HDU2_  MASK       IMAGE    Mask (0 = good)
 HDU3_  WAVELENGTH IMAGE    wavelength in Angstrom
 HDU4_  RESOLUTION IMAGE    Resolution Matrix
-HDU5_  FIBERMAP   BINTABLE Fibermap details propagated from :doc:`fibermap-EXPID.fits <../../../../../DESI_SPECTRO_DATA/NIGHT/fibermap-EXPID>`
+HDU5_  FIBERMAP   BINTABLE Fibermap details propagated from :doc:`fibermap-EXPID.fits <../../../../../DESI_SPECTRO_DATA/NIGHT/EXPID/fibermap-EXPID>`
 HDU6_  SCORES     BINTABLE Quality Assurance scores
 ====== ========== ======== ===================
 

doc/DESI_SPECTRO_REDUX/SPECPROD/exposures/NIGHT/EXPID/frame-CAMERA-EXPID.rst

@@ -195,7 +195,7 @@ HDU5
 EXTNAME = FIBERMAP
 
 Fibermap propagated from the raw data inputs; see
-:doc:`fibermap file <../../../../../DESI_SPECTRO_DATA/NIGHT/fibermap-EXPID>`.
+:doc:`fibermap file <../../../../../DESI_SPECTRO_DATA/NIGHT/EXPID/fibermap-EXPID>`.
 
 Required Header Keywords
 ~~~~~~~~~~~~~~~~~~~~~~~~

doc/DESI_SPECTRO_SIM/PIXPROD/NIGHT/EXPID/fibermap-EXPID.rst

@@ -0,0 +1,5 @@
+===================
+fibermap-EXPID.fits
+===================
+
+See :doc:`DESI_SPECTRO_DATA/NIGHT/EXPID/fibermap-EXPID <../../../../DESI_SPECTRO_DATA/NIGHT/EXPID/fibermap-EXPID>`.

doc/DESI_SPECTRO_SIM/PIXPROD/NIGHT/EXPID/index.rst

@@ -0,0 +1,24 @@
+=====
+EXPID
+=====
+
+$DESI_SPECTRO_SIM contains simulated raw data, with the canonical location
+of $DESI_ROOT/spectro/sim.  Simulated files are additionally grouped under
+a $PIXPROD subdirectory to isolate different simulation runs.
+
+The intention is that $DESI_SPECTRO_SIM/$PIXPROD will contain a superset of
+the files in $DESI_SPECTRO_DATA, such that one could set
+DESI_SPECTRO_DATA=$DESI_SPECTRO_SIM/$PIXPROD and run the pipeline.
+The simulator does not yet output truly raw data (only preprocessed data)
+so this is not yet possible.
+
+Within each NIGHT (YEARMMDD), individual exposures are grouped in subdirectories
+by EXPID.
+
+.. toctree::
+   :maxdepth: 1
+
+   fibermap-EXPID.rst
+   pix-CAMERA-EXPID.rst
+   simpix-CAMERA-EXPID.rst
+   simspec-EXPID.rst

doc/DESI_SPECTRO_SIM/PIXPROD/NIGHT/index.rst

@@ -9,14 +9,11 @@ a $PIXPROD subdirectory to isolate different simulation runs.
 The intention is that $DESI_SPECTRO_SIM/$PIXPROD will contain a superset of
 the files in $DESI_SPECTRO_DATA, such that one could set
 DESI_SPECTRO_DATA=$DESI_SPECTRO_SIM/$PIXPROD and run the pipeline.
-The simulator does not yet output truly raw data (only preprocessed data)
-so this is not yet possible.
 
+Within each NIGHT (YEARMMDD), individual exposures are grouped in subdirectories
+by EXPID.
 
 .. toctree::
    :maxdepth: 1
 
-   fibermap-EXPID.rst
-   pix-CAMERA-EXPID.rst
-   simpix-CAMERA-EXPID.rst
-   simspec-EXPID.rst
+   EXPID/index

doc/DESI_SPECTRO_SIM/PIXPROD/index.rst

@@ -3,7 +3,7 @@ PIXPROD
 =======
 
 Simulation runs are grouped into $PIXPROD subdirectories, which are
-further subdivided by NIGHT = YYYYMMDD.
+further subdivided by NIGHT = YYYYMMDD and 8-digit zero-padded EXPID.
 
 .. toctree::
    :maxdepth: 1

doc/DESI_TARGET/fiberassign/tile.rst

@@ -9,17 +9,26 @@ tile_TILEID
 :Regex: ``tile_[0-9]{8}\.fits``
 :File Type: FITS, 800 kB
 
+NOTE: this data model describes the format of the fiberassign output files
+for the 18.6 DESI software release (fiberassign/0.9.0), but this is not yet
+the format that we intend to use for real observing.  See the mockobserving
+branch of desidatamodel for that file.
+
 Contents
 ========
 
-====== ===================== ======== ===================
-Number EXTNAME               Type     Contents
-====== ===================== ======== ===================
-HDU0_  PRIMARY               IMAGE    Blank
-HDU1_  FIBER_ASSIGNMENTS     BINTABLE Target assignment for each fiber
-HDU2_  POTENTIAL_ASSIGNMENTS BINTABLE All targets that could have been assigned
-====== ===================== ======== ===================
+====== ============= ======== ===================
+Number EXTNAME       Type     Contents
+====== ============= ======== ===================
+HDU0_  PRIMARY       IMAGE    Blank
+HDU1_  FIBERASSIGN   BINTABLE Target assignment for each fiber
+HDU2_  POTENTIAL     BINTABLE All targets that could have been assigned
+HDU3_  SKYETC        BINTABLE Fiber assignments for the sky monitor fibers
+====== ============= ======== ===================
 
+The final format will also include HDUs for `TARGETS` (from target selection)
+and `GFA_TARGETS` (for guiding/focus).  `SKYETC` will likely be renamed
+to `SKY_MONITOR`.
 
 FITS Header Units
 =================
@@ -36,7 +45,7 @@ Empty HDU.
 HDU1
 ----
 
-EXTNAME = FIBER_ASSIGNMENTS
+EXTNAME = FIBERASSIGN
 
 The target assignments for each fiber of this tile.
 
@@ -72,6 +81,7 @@ DEC           float64 deg      Declination of target
 XFOCAL_DESIGN float32 mm       Designed X location on focal plane
 YFOCAL_DESIGN float32 mm       Designed Y location on focal plane
 BRICKNAME     char[8]          Brick name from tractor input
+FIBERMASK     int32            mask of known fiber problems; 0=good
 ============= ======= ======== ===================
 
 Notes:
@@ -80,7 +90,7 @@ Notes:
   be; this is non-authoritative and more detailed downstream code will have
   a refined answer for each actual observation of this tile.
 * This table defines the *requested* fiber assignments.  See
-  :doc:`fiberassign <../../DESI_SPECTRO_DATA/NIGHT/fibermap-EXPID>` for the
+  :doc:`fiberassign <../../DESI_SPECTRO_DATA/NIGHT/EXPID/fibermap-EXPID>` for the
   actual observed assignments.
 
 Expected changes:
@@ -119,6 +129,49 @@ Name              Type  Units Description
 POTENTIALTARGETID int64       label for field   1
 ================= ===== ===== ===================
 
+HDU3
+----
+
+EXTNAME = SKYETC
+
+Fiber assignments for the 20 sky monitor fibers used by the
+Exposure Time Calculator (ETC).  This has the same structure
+as the FIBERASSIGN HDU, but it is kept separate since these fibers
+go to the sky monitor camera instead of the spectrographs.
+
+Required Header Keywords
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+======= ============= ==== ===================================
+KEY     Example Value Type Comment
+======= ============= ==== ===================================
+NAXIS1  82            int  width of table in bytes
+NAXIS2  20            int  number of rows in table
+EXTNAME SKYETC        str  name of this binary table extension
+======= ============= ==== ===================================
+
+Required Data Table Columns
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+============= ======= ======== ===================
+Name          Type    Units    Description
+============= ======= ======== ===================
+FIBER         int32            Fiber ID [0-19, TBD]
+LOCATION      int32            Location on the focal plane PETAL_LOC*1000 + DEVICE_LOC
+NUMTARGET     int16            number of targets that this fiber covers
+PRIORITY      int32            priority that was used while placing this target
+TARGETID      int64            Selected target ID
+DESI_TARGET   int64            Dark survey + calibration targeting bits
+BGS_TARGET    int64            Bright Galaxy Survey targeting bits
+MWS_TARGET    int64            Milky Way Survey targeting bits
+RA            float64 deg      Right ascension of target
+DEC           float64 deg      Declination of target
+XFOCAL_DESIGN float32 mm       Designed X location on focal plane
+YFOCAL_DESIGN float32 mm       Designed Y location on focal plane
+BRICKNAME     char[8]          Brick name from tractor input
+FIBERMASK     int32            mask of known fiber problems; 0=good
+============= ======= ======== ===================
+
 Notes and Examples
 ==================