Readout hysteresis on z8-D triggering rejection of that quarter of CCD

[djschlegel]

At first, I thought there was a trap on z8-D that caused a large OSTEP value (bias jump) in the bias region on exposure 19664 on 2022-01-21. First attachment below shows the raw image in the vicinity of a CR at (2194,3637).

Upon inspection of more exposures, I'd conjecture this is a mild readout hysteresis problem on that amplifier that started around 2020-11-13. The charge is depressed after reading out lots of charge. Bias images look normal. Arc images always show approx -5 ADU after reading bright arc lines. Science images can show that depression after reading bright sky lines. The first example above shows this depression after a bright CR next to the overscan region.

Second attachment below shows the max bias jump in the overscan region for z8-D for bias images (white), arc images (red), and science images (blue).

[djschlegel]

Another CCD where this could potentially occur is z3-D. That had its readout parameters modified on 2021-12-02 to mitigate nonlinear CTE, but it still shows CTE/hysteresis that can result in OSTEP > 5 ADU in recent data. An example would be in the exposure desi/spectro/data/20220111/00118181/desi-00118181.fits.fz[Z3]. Cutout of the center of that image attached below.

[djschlegel]

Finally, the last CCD where I see examples of OSTEP > 5 ADU in some science exposures is Z5-A. Example image is desi/spectro/data/20220109/00117641/desi-00117641.fits.fz[Z5], caused by a bright CR at (1989,618).

[djschlegel]

Another example of this happening on Z8-D is tile 10184 on 2022-01-06 exp 116955, when a bright cosmic lands right next to the overscan at (2201,2619).

[julienguy]

This can be fixed with the dark trail correction already in place in the code, see PR #798 and issue #796.

[djschlegel]

Adding another Z8 exposure to look at: tile 25275, night 20220126, expid 120334. It has OSTEP=5.4 ADU, discards half the CCD, discards all but 1 standard, then the whole petal gets rejected. But I don't see any features in the overscan region that would have triggered the large OSTEP, so this is a mystery.

[julienguy]

It is the same situation as for r8, a cosmic ray on the edge of the D amplifier causes a negative trail in the overscan region (x=2200 y=2200 (not a typo)) in /global/cfs/cdirs/desi/spectro/data/20220126/00120334/desi-00120334.fits.fz hdu Z8.

[julienguy]

It's a very small effect ~10^-4 but sufficient to cause trouble for cosmic rays that leave traces with ~50000 electrons/pixel. The software fix exists, it just has to be calibrated for this CCD amp.

[julienguy]

@djschlegel , for z8 , I can fit a (very small) correction on the recent data, and I expect it applies to all data, because I have found the dark trails to be stable for other CCDs, but you mention above a starting date of 2020-11-13. do you confirm the effect is not present in earlier data?

[djschlegel]

The z8-D CTE does indeed extend before 2020-11-13, there's just very little data from before then (this was the start of SV1).

[julienguy]

Thanks. I have made changes to the code and updated the calib of r8,z8,z3 to correct the dark trails in the overscan region.

[sbailey]

Fixed in PR #1636 + desi_spectro_calib updates.

[djschlegel]

Re-opening for tile 4402 on 2022-02-05 expid 121487 which rejects fibers 4250-4499 due to a cosmic appearing at location (2197,3718) of the r8 CCD. This has OSTEPD=5.7 in the frame file and sets a mask value of 512. Was this exposure reduced with the fix described in issue #1636 ?

[schlafly]

@julienguy , is it useful to have more of these?
https://data.desi.lbl.gov/desi/spectro/redux/daily/tiles/cumulative/22337/20220321/tile-qa-22337-thru20220321.png
b6 is another such case. Just including in case we thought these should disappear entirely.

[schlafly]

And another: https://data.desi.lbl.gov/desi/spectro/redux/daily/tiles/cumulative/4623/20220403/tile-qa-4623-thru20220403.png

[geordie666]

And another: https://data.desi.lbl.gov/desi/spectro/redux/daily/tiles/cumulative/10481/20220404/tile-qa-10481-thru20220404.png

[schlafly]

Reopening this issue since @julienguy was surprised during the telecon yesterday.

[julienguy]

PR #1748 fixes all cases described above since the issue was reopened. It's not perfect, because it is still a "recipe" to flag impactful cosmics, with several thresholds tuned to trigger on the above cases while minimizing the false positives. But hopefully this is sufficient of lower the rate of occurrence of masking entire amplifiers because of cosmics.

• for tile 4402 on 2022-02-05 expid 121487 which rejects fibers 4250-4499 (r8)

desi_preproc -i ~/data/20220205/00121487/desi-00121487.fits.fz |& tee preproc-00121487.log | grep 'ignore overscan'

WARNING:preproc.py:965:preproc: Camera r8 amp D, ignore overscan rows = [1587 1588 1589 1590] because of large charge deposit = [49381.24       48383.45999999 48020.78       48423.62      ] ADUs

• tile 22230 on 2022-02-11 (expid 122156) rejects half of R4 (fibers 2250-2499)

desi_preproc -i ~/data/20220211/00122156/desi-00122156.fits.fz |& tee preproc-00122156.log | grep 'ignore overscan'

WARNING:preproc.py:965:preproc: Camera r4 amp B, ignore overscan rows = [1032 1033 1034 1035 1036 1037 1038] because of large charge deposit = [39970.04347825 40576.47826087 44794.56521741 42787.39130435
WARNING:preproc.py:965:preproc: Camera z7 amp C, ignore overscan rows = [2063] because of large charge deposit = [52945.04347825] ADUs

• https://data.desi.lbl.gov/desi/spectro/redux/daily/tiles/cumulative/22337/20220321/tile-qa-22337-thru20220321.png

desi_preproc -i ~/data/20220321/00127032/desi-00127032.fits.fz |& tee preproc-00127032.log | grep 'ignore overscan rows'

WARNING:preproc.py:972:preproc: Camera b6 amp C, ignore overscan rows = [383 384 385 386] because of large charge deposit = [64125.72000003 64126.10000002 64125.39999998 64125.5       ] ADUs

• https://data.desi.lbl.gov/desi/spectro/redux/daily/tiles/cumulative/4623/20220403/tile-qa-4623-thru20220403.png

desi_preproc -i ~/data/20220403/00128560/desi-00128560.fits.fz |& tee preproc-00128560.log | grep 'ignore overscan'

WARNING:preproc.py:972:preproc: Camera b6 amp D, ignore overscan rows = [1929 1930 1931 1932 1933] because of large charge deposit = [64345.68000007 64345.90539551 64346.0999999  64346.11999989

• https://data.desi.lbl.gov/desi/spectro/redux/daily/tiles/cumulative/10481/20220404/tile-qa-10481-thru20220404.png

desi_preproc -i ~/data/20220404/00128680/desi-00128680.fits.fz |& tee preproc-00128680.log | grep 'ignore overscan'

WARNING:preproc.py:972:preproc: Camera b1 amp C, ignore overscan rows = [891 892 893] because of large charge deposit = [64284.68000007 64284.5        64283.74000001] ADUs
WARNING:preproc.py:972:preproc: Camera z7 amp C, ignore overscan rows = [2063] because of large charge deposit = [53796.45999998] ADUs

[sbailey]

Closing since all known cases are fixed, but please re-reopen if it occurs again and we'll iteratively tune parameters for the new cases.

[schlafly]

Continuing to record cases here:
https://nightwatch.desi.lbl.gov/20220424/00131715/preproc-b6-00131715-4x.html

[sybenzvi]

New case from 20220516, BRIGHT exposure 135121, tile 22350:
https://nightwatch.desi.lbl.gov/20220516/00135121/preproc-b1-00135121-4x.html