Improvement to sky subtraction with a PCA

[julienguy]

In this PR is implemented a principal component analysis of the wavelength and line spread function corrections to the emission lines in the sky spectra. The PCA components are based the analysis of many dark time exposures. For each science exposure, a linear combination of the components is fit using the measured corrections to the sky lines of the sky fibers only, and then the corrections are applied to all fibers. Compared to the Everest data release, this method reduces the residual scatter on sky lines by 10 to 20% depending on the camera.
See more details and plots in https://desi.lbl.gov/DocDB/cgi-bin/private/ShowDocument?docid=6437 .

Changes to the code

• Changes to desi_compute_sky
  • add option --adjust-with-more-fibers use more fibers than just the sky fibers for the adjustments of the sky line wavelength and width (used only to build the PCA components)
  • add option --save-adjustments SAVE_ADJUSTMENTS save adjustments of wavelength calibration and LSF width in fits images (fibers x wavelength) (used only to build the PCA components)
  • add option --pca-corr PCA_CORR use this PCA frames file for interpolation of wavelength and/or LSF adjustment (used by default for sky subtraction in desi_proc)
• Changes to desi_proc
  • add options --adjust-sky-with-more-fibers and --save-sky-adjustments, only used to build the PCA components, which saves the adjustments as fits files 'skycorr-camera-expid.fits' is the reduced exposure directory.
  • uses by default desi_compute_sky with option --pca-corr FILENAME , where the FILENAME is found with the calibfinder.
• New script desi_compute_skycorr_pca to compute the PCA components from a large list of 'skycorr*.fits' files.

Changes to the calibration SVN

• Add files $DESI_SPECTRO_CALIB/spec/sm*/skycorr-pca-sm*-*.fits (one per camera)
• Add keyword SKYCORR spec/sm*/skycorr-pca-sm*-*.fits in the yaml files.

[coveralls]

Coverage increased (+0.2%) to 26.819% when pulling a810249 on skysub into 2777a0d on master.

[sbailey]

Overall looks good though I'd like to poke at running it a bit more after cori is back, before merging. I put some maintenance/maintainability comments inline for consideration.

In desi_spectro_calib, the skycorr files are now the biggest thing in the spec subdirectory. Let's change those to single precision float to save 2x in size. gzipping doesn't help much beyond that.

[sbailey]

This breaks our usual rule of separating I/O from algorithms. Consider reading that in with a desispec.io.read_skycorr routine and passing the correction model into compute_sky -> compute_uniform_sky .

[julienguy]

added I/O

[julienguy]

I/O tested in isolation but not in production

[julienguy]

Changes tested in small prod with desi_proc. All comments so far have been addressed. Unless there are more, this PR could be merged once the impact on redshifts is evaluated.

[sbailey]

Discussed with Julien and we won't wait for redshift tests on this PR specifically, but rather bundle those tests with other recent algorithmic updates, so I'll merge this now.