standardize spectro file names/locations

[sbailey]

Opening this PR to get it into consideration for the 18.3 software release, even though it needs a bit more vetting / work before merging. This PR does some cleanup of spectro pipeline file names and locations for better consistency:

• exposures/{night}/{expid}/pix*.fits -> preproc/{night}/{expid}/preproc*.fits
  • Reason: separate large pix files from other per-exposure files for easier backup and deletion
  • clearer name than "pix"
• spectra-{nside}/{group}/{pix}/rrdetails_redshift_{nside}_{pix}.h5 ->
  spectra-{nside}/{group}/{pix}/redrock-{nside}-{pix}.h5
  • Reason: dash instead of underscore delimiter like others; shorter name
  • Note: data challenge prods so far have bypassed the pipeline anyway and have used "rr-{nside}-{pix}.h5".
  • TODO: the only place where I found the "rrdetails_redshift" name mentioned was in py/desispec/pipeline/attic/workers.py which doesn't smell like the place that it actually would be. Did I really fix this or not?
  • TODO (longer term): reformat hdf5 -> fits
• calib2d -> calibnight
  • Reason: rescoped to be per-night calibs; includes fiberflats which aren't 2d
• calib2d/psf/{night}/psfnight-{camera}.fits -> calibnight/{night}/psfnight-{camera}-{night}.fits
  • Reason: make filenames unique without relying upon directory structure too
• calib2d/{night}/fiberflat-{camera}-{expid}.fits ->
  exposures/{night}/{expid}/fiberflat-{camera}-{expid}.fits
  • Reason: move per-exposure quantities back into per-exposure directories
• calib2d/{night}/fiberflatnight-{camera}.fits ->
  calibnight/{night}/fiberflatnight-{camera}-{night}.fits
  • Reason: like psfnight, make filenames unique without relying upon directory structure too
• calib2d/psf/{night}/psfboot-{camera}.fits ->
  exposures/{night}/{expid}/psfboot-{camera}-{expid}.fits
  • Reason: like PSF, grouping per-exposure outputs together
  • TODO: check usage of psfboot; it is no longer used by offline pipeline
• TODO (later): QA usage of calib2d
• TODO (later): io.findfile has lots of quicklook files mixed in with other directories instead of isolated into a quicklook area. Review that.

[sbailey]

@julienguy please review and test; I've likely broken something for the pipeline...

[julienguy]

Tested on sims after a few bug fixes.