fiberassign input for newexp-desi

[sbailey]

Refactor newexp-desi to optionally take fiberassign output + a truth file to generate the targets for the exposure of that tile.

Background: newexp-desi currently works in isolation, generating a new exposure with the right target densities, but it doesn't have any connection to fiber assignment or other exposures of overlapping tiles. The main difficulty is simulating the same target on different exposures on different tiles if it gets re-observed: newexp-desi needs some way of handing off the truth spectrum from one exposure to another.

Suggestion: Keep a cache of templates for every target on each brick. When simulating a new tile, look up all the bricks covered by that tile. If a template cache for a brick already exists, use that. If not, generate a template for every target on that brick (whether or not it was observed by the current tile) and add that to the cache.

newexp-desi outputs simspec files to $DESI_SPECTRO_SIM/$PIXPROD/{night}/simspec-{expid}.fits. I suggest the brick cache should be at $DESI_SPECTRO_SIM/$PIXPROD/bricks/{brickname}-templates.fits.

Template generation currently doesn't let us specify the redshift distribution (see #104), which we need for this feature to be fully functional. But even if we ignore the redshifts from the input truth, it would be useful to get the rest of this infrastructure going and have the ability to simulate multiple observations of the same target based upon fiber assignment output.

Example fiber assignment files at NERSC in /project/projectdirs/desi/target/fiberassign/examples/0.2.3:

• truth-lite.fits : a truth table
• mtl-lite.fits : the "merged target list" that was input to fiber assignment
• tile_{tileid}.fits : fiberassign output

The new options would be something like

newexp-desi --fiberassign tile_28598.fits --truth truth-lite.fits [other options...]

[sbailey]

Just had a chat with Lado and Jaime; updating the ticket for the record. We'll start with a tool for generating spectra for a brick from an ra,dec,z,truetype truth file, and then separately update newexp-desi to use that tool and/or the pre-cached results from that tool.

The catalog truth file format is documented at http://desidatamodel.readthedocs.io/en/latest/DESI_TARGET/truth.html

The output "simbrick" (?) file with spectra per brick would be similar to the FLUX and METADATA HDUs of the simspec files described at http://desidatamodel.readthedocs.io/en/latest/DESI_SPECTRO_SIM/PIXPROD/NIGHT/simspec-EXPID.html . They should not include the sky or photon-level information that the simspec files have. i.e. simbrick files just describe the true objects, while the simspec files contain additional information about a particular observation of those objects.