b5 PSF fits failing May 2021

[sbailey]

b5 PSF fits are regularly failing for May 2021 data, e.g. on nights 20210507, 0508, and 0509 every b5 arc exposures fails, thus losing the spectrograph for the night unless we do by-hand fallback to other calibs.

Curiously, these did not fail for the Everest production. Known changes since then:

	everest	test-21.12
desi_spectro_calib	0.2.7	trunk
specex	0.8.0	0.8.3
desispec	0.45.3	0.48.0
ZEROs -> nightly bias	no	yes
DARK -> badcol mask	no	yes

Example failure run from an interactive batch node:

source $CFS/desi/software/desi_environment.sh test-21.12
srun -N 1 -n 21 -c 2 desi_proc --mpi --traceshift -n 20210507 -e 87539 --cameras b5
...
WARNING cholesky_solve failed with status 351
FATAL ERROR (other std) FitSeveralSpots failed for FLUX+TRACE (at line 2562 of file /tmp/pip-req-build-lryb6298/src/specex_psf_fitter.cc)
...
ERROR:specex.py:194:main: desi_psf_fit on process 18 failed with return value 1 running desi_psf_fit -a /global/cfs/cdirs/desi/users/sjbailey/spectro/redux/f3x/preproc/20210507/00087539/preproc-b5-00087539.fits --in-psf /global/cfs/cdirs/desi/users/sjbailey/spectro/redux/f3x/exposures/20210507/00087539/shifted-input-psf-b5-00087539.fits --out-psf /global/cfs/cdirs/desi/users/sjbailey/spectro/redux/f3x/exposures/20210507/00087539/fit-psf-b5-00087539_18.fits --lamp-lines /global/common/software/desi/cori/desiconda/20200801-1.4.0-spec/code/specex/0.8.3/lib/python3.8/site-packages/specex/data/specex_linelist_desi.txt --first-bundle 18 --last-bundle 18 --first-fiber 450 --last-fiber 474 --legendre-deg-wave 1 --broken-fibers 342
...

Note: desihub/desispec#1528 reports a bug about the desi_spec batch script generation when running specex on a single camera, thus the "by hand" command run from an interactive batch node above.

@julienguy @marcelo-alvarez please track this down to get these working again; this is a blocking factor for Fuji.

[julienguy]

I do not have the FLUX+TRACE after masking fiber 2900 which is on a dead column (I am committing the change to DESI_SPECTRO_CALIB). But for the specific example given, the readnoise on amp B is such that the code (rightfully) discards the PSF.

ERROR:proc.py:490:main: 25% of b5 input pixels have bad readnoise; don't use this PSF

[julienguy]

For those nights in May where b5 was very noisy, we can use the default PSF $DESI_SPECTRO_CALIB/spec/sm9/psf-sm9-b5-20210912.fits. The difference is typically of 1-2% with the daily PSF (from a comparison with the PSF $DESI_SPECTRO_REDUX/daily/calibnight/20210507/psfnight-b5-20210507.fits which was completed despite the high readnoise, and now masked out with the recent software). It is not optimal but is definitely better than discarding nights of data.

[julienguy]

• Part of this issue is addressed in desispec PR Readnoise estimation whensubtracting the overscan per row desispec#1564 where we compute a single readnoise value per amplifier instead of having one value per row when we are doing the overscan subtraction per row.
• The other aspect is which readnoise threshold should we use for the PSF.

[julienguy]

The read noise threshold issue is addressed in desispec issue desihub/desispec#1559 so we may close this one (specex 59) when both desispec PR #1564 issue #1559 are merged/addressed.

[sbailey]

another contributing factor to the b5 failures is a huge number of incorrectly identified bad columns (see desihub/desispec#1567). I think all of the relevant pieces are being tracked in other tickets so I'm going to close this one.