evaluate b5 readnoise impact

[sbailey]

Evaluate the impact of b5 amp B readnoise in May 2021 data to decide what should be done for Fuji.

PSF

In the f4 production, b5 PSFs failed for every exposure on nights 20210507, 20210508, and 20210509, and 4/5 exposures on 20210510. Although PR #1543 updated the OSTEP calculation so that the entire CCD isn't masked, b5 amp B is running just under the 10 e- masking threshold such that many rows get flagged as BADREADNOISE and so much data is masked that PSFs fail.

Is it really so bad that we need to flag these exposures as bad? If so, are the default PSFs in desi_spectro_calib good enough or do we need a different fall back? Or do we need an update to PSF fitting, e.g. to ignore the BADREADNOISE mask bit?

Flats

Test production /global/cfs/cdirs/desi/users/sjbailey/spectro/redux/b5readnoise night 20210509 exposure 87782 is an example flat. Many rows of b5B are flagged as bad, but this wasn't fatal for the flats. Evaulate whether they are actually ok.

Science

Also in the sjbailey b5readnoise test prod, night 20210509 exposure 87817 is a science exposure. The fibers/wavelengths covering b5B are clearly readnoise dominated instead of sky/object shotnoise dominated, leading to a different ivar structure (stripes by CCD row instead of stripes in wavelength), but it isn't obviously catastrophically wrong (vs. just noisier).

In general

The readnoise is really bad on those nights, ~10 e-, but the pipeline basically models and propagates that and it should be reflected in ivar, TSNR2, redshifts, etc. Currently >10 e- triggers the CCD pixel-level BADREADNOISE mask. This is not one of the bits in desispec.maskbits.extractmaskval so it doesn't force ivar=0 for the extractions, but it looks like that might be happening for PSFs. I think the extraction behavior is correct: treat BADREADNOISE as informative, but still use the pixels with whatever ivar they have.

@julienguy can you evaluate this, especially for the failing PSFs?

[julienguy]

• PSF

  • the read noise of amp B for those exposures varies from 9.3 to 10.5 elec
  • increasing the read threshold to 20 elec recovers all PSFs except for 20210509/00087767 and 00087769 which have OSTEP > 5 elec
  • comparing the first and last of each series we get variations of PSF width<0.5% and
    an emission line bias < 0.2% at 4000A
  • the conclusion is that we can use PSFs with read noise ~10 elec.
  • one suggestion is simply to increase the threshold to 15 elec (larger than 10 but not too large because untested regime)

• FLAT

  • I computed the fiberflat for night 20210508 with typical read noise close to 10 elec.
  • I then looked at the ratio of 2 fiberflats and found the flatfield noise was almost the same for fibers [0:250] and [250:500], so again we can safely use flats with read noise of ~10 elec.
  • We can set the threshold to 15 elec.

• SCIENCE

  • We should avoid setting ivar=0 for amplifiers with high read noise. We should use the data in redrock for single exposures.
  • The difficulty is with coadds, should we or should we not mask the pixels when there is another exposure with good read noise ? If this read noise were Gaussian white noise, then the inverse variance weight used in the coadd would be the natural solution, but this read noise (or actually bias fluctuations) has some 'stripy' structure and is not Gaussian. We may want to set a different CCD mask bit for the specific stripy bias fluctuations measured with the parameter 'OSTEP' .

[julienguy]

A proposal is

• do not set ivar=0 (and do not mask in redrock) spectral values with BADREADNOISE in single exposure spectra
• do not set ivar=0 (and do not mask in redrock) spectral values of coadd if for all exposures, the BADREADNOISE bit=1
• set weight=ivar=0 to spectral values with BADREADNOISE contributing to a coadd if there is at least one other exposure where this bit is not set for this wavelength and fiber.

[sbailey]

badamp issues are flagged better via PR #1605 and #1610 . Closing