additional testing and speed-ups of v2.0.0

[moustakas]

This PR builds on #95 and #96 with an eye toward addressing #98, i.e., speed, while also fixing various bugs discovered in testing.

Although the algorithms are essentially identical to those in v2.0.0, this PR is a major refactor of how data and methods are passed between multiprocessing cores. In essence, in the v2.0.0 refactor of the code (compared to v1.0.1), I was relying on the idea that I could read the (~few hundred MB) templates and instantiate a one-stop-shop Class (FastFit) for all the fitting, and pass both the data and this Class to each individual process without any copying. However, for reasons I still don't fully understand, it appears that the individual processes were making their own local copy of everything (via an inefficient pickling process), which led to significant (and unacceptable) slowdowns.

This PR represents a major refactor of how the data and methods are handled, which leads to an enormous speed-up (a factor of 10 or more in some cases!) compared to v2.0.0.

For example, the partial log below illustrates fitting a single healpixel with 1008 targets in approximately 270 s (=4 m 30 s) with 128 cores on perlmutter. (With v2.0.0, the fitting would have taken well over 45 m with the same number of cores.) Of this time, roughly 25 s is spent on I/O, 100 s is spent pre-processing the spectra to determine initial line-widths (which could probably be sped up in a future PR), and 145 s is spent on the fitting. If I'm doing the math correctly, this comes out to an average of 18 s/core/object (145/1008*128=18) for the fitting, which is roughly consistent with #98.

After a little more testing of this PR, I'm planning to tag, run on a largish sample, and then (hopefully) process all of fuji, guadalupe, and then iron.

% time fastspec /global/cfs/cdirs/desi/spectro/redux/fuji/healpix/sv3/bright/260/26001/redrock-sv3-bright-26001.fits \
  -o fastspec-sv3-bright-26001-v2.1.fits.gz --mp 128

INFO:fastspecfit.py:118:parse: /global/homes/i/ioannis/code/desihub/fastspecfit/bin/fastspec /global/cfs/cdirs/desi/spectro/redux/fuji/healpix/sv3/bright/260/26001/redrock-sv3-bright-26001.fits -o fastspec-sv3-bright-26001-v2.1.fits.gz --mp 128
INFO:io.py:500:select: Reading and parsing 1 unique redrockfile(s).
INFO:io.py:559:select: specprod=fuji, coadd_type=healpix, survey=sv3, program=bright, healpix=26001
INFO:io.py:674:select: Updating QSO redshifts using a QN threshold of 0.95.
INFO:io.py:805:select: Gathered photometric metadata for 1008 objects in 4.66 sec
INFO:io.py:895:read_and_unpack: Reading 1008 spectra from /global/cfs/cdirs/desi/spectro/redux/fuji/healpix/sv3/bright/260/26001/coadd-sv3-bright-26001.fits
INFO:spectra.py:291:read_spectra: iotime 1.469 sec to read coadd-sv3-bright-26001.fits at 2023-02-04T12:50:56.975720
INFO:io.py:922:read_and_unpack: Coadding across cameras took 2.62 seconds.
INFO:continuum.py:421:_get_linesigma: Forbidden masking sigma=116 km/s and S/N=15
[snip]
INFO:emlines.py:2415:emline_specfit: Dropping broad-line model: S/N in either of the two reddest broad lines < 3.0.
INFO:emlines.py:2521:emline_specfit: Final line-fitting with 32 free parameters took 0.53 seconds [niter=3, rchi2=0.9561].
INFO:emlines.py:2521:emline_specfit: Final line-fitting with 34 free parameters took 0.88 seconds [niter=5, rchi2=0.9595].
INFO:emlines.py:2612:emline_specfit: Dn(4000)=0.753 in the emission-line subtracted spectrum.
INFO:emlines.py:2653:emline_specfit: Emission-line fitting took 3.21 seconds.
INFO:emlines.py:2612:emline_specfit: Dn(4000)=-1.599 in the emission-line subtracted spectrum.
INFO:emlines.py:2653:emline_specfit: Emission-line fitting took 6.43 seconds.
INFO:fastspecfit.py:205:fastspec: Fitting 1008 object(s) took 142.74 seconds.
INFO:io.py:1290:write_fastspecfit: Writing results for 1,008 objects to fastspec-sv3-bright-26001-v2.1.fits.gz
INFO:io.py:1345:write_fastspecfit: Writing out took 5.82 seconds.

real	4m32.046s
user	197m10.902s
sys	7m21.463s

[coveralls]

Pull Request Test Coverage Report for Build 4145738449

• 1420 of 2666 (53.26%) changed or added relevant lines in 6 files are covered.
• 4 unchanged lines in 1 file lost coverage.
• Overall coverage decreased (-9.5%) to 53.435%

---

Changes Missing Coverage	Covered Lines	Changed/Added Lines	%
py/fastspecfit/util.py	6	8	75.0%
py/fastspecfit/io.py	289	367	78.75%
py/fastspecfit/continuum.py	472	671	70.34%
py/fastspecfit/fastspecfit.py	35	310	11.29%
py/fastspecfit/emlines.py	615	1307	47.05%

Files with Coverage Reduction	New Missed Lines	%
py/fastspecfit/fastspecfit.py	4	13.62%

Totals
Change from base Build 4014896532:	-9.5%
Covered Lines:	2201
Relevant Lines:	4119

---

💛 - Coveralls