Ignore overscan rows with large charge deposit detected in preceeding active CCD pixels

[julienguy]

• Define a CCD band next to overscan columns but in active region
• Measure charge per row
• If above a threshold of 50,000 ADUs, ignore values in overscan rows (replace by interpolated values from good overscan rows)

This solves desihub/desisurveyops#28

This change to the preprocessing still needs to be validated on a larger set of exposures/cameras (including ones with noisy amps and large OSTEP values, and other instances of cosmics near the overscan region).

[julienguy]

Example (from issue desihub/desisurveyops#28 )

desi_preproc -i /global/cfs/cdirs/desi/spectro/data/20220221/00123247/desi-00123247.fits.fz --cam b1 -o $SCRATCH/preproc-b1-00123247-new.fits

WARNING:preproc.py:940:preproc: Camera b1 amp C, ignore overscan rows = [1592 1593 1594 1595 1596] because of large charge deposit = [ 64176.76540536  71701.28887749 146165.43177915 154903.11496449 106286.49147892] ADUs
INFO:preproc.py:948:preproc: Camera b1 amp C overscan max-min per row (OSTEP) = 1.181718 ADU
INFO:preproc.py:950:preproc: Camera b1 amp C subtracting average overscan

left: daily right: this PR

Other example (from issue desihub/desisurveyops#20 )

desi_preproc -i /global/cfs/cdirs/desi/spectro/data/20220205/00121487/desi-00121487.fits.fz --cam r8 -o $SCRATCH/preproc-r8-00121487-new.fits

• Daily version is prior to the dark trail correction in overscan PR :

INFO:preproc.py:927:preproc: Camera r8 amp D overscan max-min per row (OSTEP) = 5.712583 ADU
INFO:preproc.py:937:preproc: Camera r8 amp D subtracting overscan per row
WARNING:preproc.py:945:preproc: Camera r8 amp D OSTEP=5.71 is too large, set ccdmask.BADREADNOISE bit mask

• Current master version does the dark trail correction in overscan, but it's not good enough because the cosmic ray hit is very bright :

INFO:preproc.py:897:preproc: Perform a dark trail correction before fitting the overscan region
INFO:preproc.py:927:preproc: Camera r8 amp D overscan max-min per row (OSTEP) = 5.840006 ADU
INFO:preproc.py:937:preproc: Camera r8 amp D subtracting overscan per row
WARNING:preproc.py:945:preproc: Camera r8 amp D OSTEP=5.84 is too large, set ccdmask.BADREADNOISE bit mask

• Version in this PR detects the cosmic ray and mask the impacted overscan area:

WARNING:preproc.py:940:preproc: Camera r8 amp D, ignore overscan rows = [1586 1587 1588 1589 1590 1591] because of large charge deposit = [218204.82608698 260394.30434773 191296.30434796 205352.78260871
INFO:preproc.py:948:preproc: Camera r8 amp D overscan max-min per row (OSTEP) = 0.752226 ADU
INFO:preproc.py:950:preproc: Camera r8 amp D subtracting average overscan

left: daily center: master right: this PR

[schlafly]

If you want another example case, P6 on tile 2838 (124474) looks like it got hit with this too.
https://data.desi.lbl.gov/desi/spectro/redux/daily/tiles/cumulative/2838/20220301/tile-qa-2838-thru20220301.png

[julienguy]

The current version of this PR does not fix that case, where the cosmic is farther from the overscan region.

[sbailey]

Because this algorithm is applied before dark subtraction, it is sensitive to unsubtracted dark current hot columns, e.g. it is triggered near the upper edge of z7D for most exposures on 20220221, and often z7C as well. I think that's harmless because only a few edge rows are impacted in this case. Other non z7D/z7C cases from that night for followup spot checking:

20220221/00123247/desi-00123247.fits.fz b1 C [3705 3706 3707 3708 3709]
20220221/00123247/desi-00123247.fits.fz r2 C [3485 3486 3487 3488 3489 3490 3491 3492 3493]
20220221/00123255/desi-00123255.fits.fz z8 B [1635]
20220221/00123258/desi-00123258.fits.fz b5 C [2732 2733 2734 2735 2736]
20220221/00123287/desi-00123287.fits.fz z8 B [1192]

(the arrays are the raw data rows, not the preproc rows, and not the row-within-the-amp reported by the warning message...)

That's a large enough number of cases from a single night that I'd like to check more nights spanning more time and CCD conditions, looking for (and hopefully not finding) runaway false-positives.

[sbailey]

Brain dump for followup before I forget:

I scanned 1/3 of the exposures for Fuji, looking for runaway making cases with more than 20 rows masked. The following came up for closer inspection:

20210327/00082505/desi-00082505.fits.fz z2 C rows [2199 2201 2203 2205 2443 2444 2445 2446 2476 2483 2484 2486 2494 2496 2498 2504 2506 2508 2509 2510 3916 4013 4191])

20210412/00084535/desi-00084535.fits.fz z6 C rows [2913 2915 2921 2925 2927 2929 2931 2933 3401 3405 3406 3409 3418 3419 3420 3421 3422 3423 3425 3427 3536 3538 4191 4192])

20210420/00085636/desi-00085636.fits.fz z6 C rows [2909 2910 2911 2913 2914 2921 2925 3404 3408 3418 3419 3420 3421 3422 3423 3424 3425 3427 3531 3533 4191 4192])

that's not very many out of ~850 exposures scanned.

There certainly are cases that are flagging rows that had a nearby cosmic but didn't have a charge trail, but it seems relatively harmless since only a few rows are impacted (albeit in a way that introduces a bias-subtraction correlation with neighboring rows, though I think that is minor relative to other "features" in the data).

[sbailey]

All 3 of the cases with >20 rows masked turned out to be zcam saturation events anyway, i.e. I did not find any significant problems on any of the thousands of CCD frames scanned. Merging.