Failing PSFs that succeeded in everest

[sbailey]

The following PSFs succeeded in everest but failed in the pre-Fuji "f4" test run:

20210130 73663 z3
20210205 74755 z3
20210214 75767 z3
20210215 76065 z3
20210218 76704 z3
20210314 80419 z3
20210315 80541 z3
20210316 80628 z3
20210322 81428 z3
20210326 82283 z3
20210327 82428 z3
20210331 82753 r0
20210402 83092 z6
20210428 86448 z3
20210523 89828 r0
20210523 89829 r0
20210613 93532 z3

@julienguy please check whether these are genuinely bad and should be discarded or whether they are worth recovering.

Note that there are additional PSF failures on 20210506/87422, 20210509/87766, and 20210509/87768 covered by desihub/specex#60; and many b5 PSF failures during May 2021 covered by #1559.

Context: originally the pipeline would proceed anyway if some PSFs failed but we repeatedly regretted that decision so now the pipeline stops on failure. As a result, these PSF failures are blocking processing those nights until we either proactively flag these frames as bad or fix them (either algorithmically or with calib update).

[julienguy]

• z3 : all cases are caused by a masked amplifier because OSTEP>2. We should not even try to fit the PSF in those cases.
• r0 : intermittent bad column x=1168 (starting at 0)
• z6 : (one event) not sure, probably a rare occurence of a cosmic on a spot

I don't think they are worth recovering.

[julienguy]

One correction to the previous message: all problematic z3 exposures but one (20210613/00093532) have OSTEPD>5 and the amplifier D is masked. The PSF failure for the remaining one (20210613/00093532) is caused by another issue.

[julienguy]

I confirm for 20210613/00093532 we have a fit convergence ERROR in bundle 11. It is something that can happen because of cosmic rays, masked pixels, and this is one of the reasons we take 5 exposures. I would not spend more time on this. The other issue of having one amplifier fully masked because of BADREADNOISE causing PSF failures is expressed more explicitly in issue #1559 so I am closing this one.