N(z) K-S test to identify bad fibers in jura

[rongpu]

Repeat the exercise in issue #1946, I have identified the following 15 fibers in jura with K-S test p-value less than 1e-4 for either BGS (14 fibers) or LRG (10 fibers):
466, 551, 552, 553, 725, 1008, 1098, 3234, 3235, 3250, 3504, 3969, 3994, 4720, 4891

Below are redshift vs date plots for these fibers:

FIBER 466
BGS

FIBER 551
BGS

FIBER 552
BGS

LRG

FIBER 553
BGS

LRG

FIBER 725
LRG

FIBER 1008
BGS

LRG

FIBER 1098
BGS

FIBER 3234
BGS

LRG

FIBER 3235
BGS

LRG

FIBER 3250
BGS

LRG

FIBER 3504
BGS

LRG

FIBER 3969
BGS

LRG

FIBER 3994
BGS

LRG

FIBER 4720
BGS

FIBER 4891
BGS

[rongpu]

If I run the same test on the ELGs without the OII quality cuts, the K-S test finds 129 fibers with p-value below the (more restrictive) threshold of 2e-6. This large number of "bad" fibers are perhaps expected since 20-30% of the ELG targets are low-S/N featureless spectra, and their redshifts can be sensitive to subtle CCD defects (which can otherwise be ignored for any spectrum with reasonable S/N). Still there might be useful information in their redshift vs date plots, and they can be found here:
https://data.desi.lbl.gov/desi/users/rongpu/redshift_qa/nz_ks_test/bad_fibers_from_nz-elg-zcatalog.html

Alternatively we can simply look at ELG spectra that pass the OII cuts (here instead of zcatalog, I use Ashley's LSS catalog which includes the OII columns). This flags 10 fibers with p-value<1e-4:
551, 1261, 1269, 1295, 1659, 1971, 1972, 2246, 3234, 3359
Since the OII cuts already removed most of the spurious redshifts, this list is almost certainly an incomplete list of fibers with low ELG redshift efficiency. The redshift vs date plots can be found here: https://data.desi.lbl.gov/desi/users/rongpu/redshift_qa/nz_ks_test/bad_fibers_from_nz-elg-lss_catalog.html

[rongpu]

Julien and I have looked into some of the fibers. Here are the preliminary conclusions:

466: already flagged in Y1. Nothing obvious. ~20% low flat field.
551-553: The beginning of a CTE charge trap (tracked by this issue).
725: z1 CTE in CCD_X range 543:2057 and fiber at X=1878 (should have affected other fibers if caused by CTE). No redshift outliers in 2024 after we replaced the CCD. This issue is not understood yet.
3234-3254: r6 CTE first detected on May 21, 2023; we switched to 2-amp mode on March 25, 2024. Software correction for CTE in place with simple sector offset; this correction might be not sufficient. CCD_X range of CTE correction: 1905-2057; this corresponds to fibers 3234-3253(?). We also see the CTE pattern in the dark.
3504: appears to be a bad column in the dark at CCD_X=177:178 (the fiber is centered at 174) in r7; the two bad columns have no signal in the flats (see below). The resulting spurious redshifts first appeared after summer 2022. Action item: mask the two bad columns.
Below: Left: dark; right: flat

3969: Also at the beginning of a CTE charge trap (similar to fibers 551-553) in z7 amp B (as has already been documented). CCD_X=2057:3724 (the fiber is centered at 3719).

Summary: all the above issues were related to charge traps, except for fibers 725 (unknown issue) and 3504 (due to unmasked bad columns).

Remaining fibers to look at (all of them were also identified in the Y1 iron): 466, 1008, 1098, 3994, 4720, 4891. Need to understand fiber 725 too.

[rongpu]

Julien and I have gone through all the fibers. Here are our findings (and the fixes that we have implemented for some of the fibers). Some of the information might be redundant with my previous post.

466: already flagged in Y1. Nothing obvious. ~20% low flat field.
Good PSF, spectra trace, nothing wrong in bias, dark, pixel flat, pixel mask, fiber flat
Can’t be variation of fiber tip alignment because it would be mostly achromatic and we see a rather strong chromatic effect in the blue. This fiber is on the edge of the focal plane but the neighboring fiber 465 does not have any issue.
Example BGS spectrum:

(jura/tiles/cumulative/20946/20221205/coadd-0-20946-thru20221205.fits)

551-553: The beginning of a CTE charge trap (tracked by #2286).

725: the fiber trace lies on top of a vertical band of excess charge (~15 elec) in z1 amplifier A that we see in master bias frames (probably induced by the parallel clocking at the transfer gate?). This excess charge is fluctuating a bit during the night causing offsets in the spectra. We don’t think it requires masking out the CCD column. We suggest only flagging the fiber for the LSS catalog.

1008: charge trap along the parallel at CCD_X=190, CCD_Y=1901 that was not sufficiently masked. We have now fixed this by extending the size of the mask. This fix has been pushed to SVN.

1098: symptom: discontinuity between blue and red caused by excess flux in blue (see example spectrum below). No signal of any problem in bias,dark,flat,psf. Related to TPCORR parameter in BLUE which could be the cause or another symptom.
We could test this by rerunning the sky modeling without the tpcorr.
Example BGS spectrum:

(jura/tiles/cumulative/21315/20231125/coadd-2-21315-thru20231125.fits)

Seen as outlier in TPCORR blue only (see below for other cases)

3234-3254: r6 CTE first detected on May 21, 2023; we switched to 2-amp mode on March 25, 2024. Software correction for CTE in place with simple sector offset; this correction might be not sufficient. CCD_X range of CTE correction: 1905-2057; this corresponds to fibers 3234-3254. We also see the CTE pattern in the dark.
Possible correction using the complex CTE model from Eddie (or we simply flag the fibers in LSS catalog).

3504: appears to be a bad column in the dark at CCD_X=177:178 (the fiber is centered at 174) in r7; the two bad columns have no signal in the flats (see below). The resulting spurious redshifts first appeared after summer 2022. We masked the two bad columns and committed in SVN.

3969: Also at the beginning of a CTE charge trap (similar to 551-553) in z7 amp B (as has already been documented). CCD_X=2057:3724 (the fiber is centered at 3719).

3994: we suspect that it's a problem with tpcorr, because this fiber is an outlier in tpcorr parameters in R camera but not in B camera (and we expect them to be consistent as the tpcorr correction should be achromatic).

We could rerun sky subtraction without tpcorr.
Example BGS spectrum:

(jura/tiles/cumulative/24983/20220313/coadd-7-24983-thru20220313.fits)

4720, 4891: We are also finding large TPCORR values for those fibers. For 4891, the values are mostly identical for all cameras. For 4720, we find a large TPCORR value only in r band.

• For those 3 fibers (including 3994), a possible cause is that the tip the fiber is such that the amount of light collected in the fiber varies from exposure to exposure.
• Or, the tpcorr parameters have some noise that we could probably fix.
  
  Note that he steps we see as a function of fiber correspond to a smooth radial variation in the focal plane:
  

[sbailey]

Summary from @rongpu's 2024-07-23 data telecon presentation, slightly augmented from discussion

• CTE-related and not-yet-fixed
  • 551-553, 3969: Edge effects; add a new FIBERSTATUS bit
  • 3234-3254: Try turning on Simplified Regnault model, OFFCOLS -> CTECOLS
• Suspected TPCORR issue: 1098, 3994, 4720, 4891 - add new FIBERSTATUS bit
• Fixed since jura: 3504 (bad column; now masked); 1008 (insufficient masking of CTE in jura)
• Cannot be fixed: 725 (hot column)
• Causes unknown: 466

[julienguy]

Fiber 466 is among the fibers with the largest spatial throughput correction 'TPCORR' ( achromatic throughput correction as a function of positioner move x,y). I see the correction is exactly the same in B,R,Z. But maybe it shouldn't?

[rongpu]

@schlafly pointed out that there are problems with fibers 918-921 that were identified from Tertiary 37. It's also causing spurious LRGs redshifts starting ~Jan 2024 (see below). They were not flagged by the K-S test presumably because only a small portion of the data was affected.

FIBER 918

FIBER 919

FIBER 920

FIBER 921

[araichoor]

I ve not followed all the details in this ticket, but if useful / complementary:
I ve generated for all jura exposures these diagnosis plots using the sky spectra, and looking for offsets between amplifiers.
(presented here: #2193 (comment))

those 918-921 fibers likely are the "spike" in red dots here, for 20240304 (reaching the y=1 line):

@rongpu: it may be worth it to check few fibers around those values, as this "918-921" comes from tertiary37, where the stats are not great.

pdf files here (those are heavy, 100-400M each!):
https://data.desi.lbl.gov/desi/users/raichoor/spectro/jura/peramp-medians/peramp-median-jura-sv.pdf
https://data.desi.lbl.gov/desi/users/raichoor/spectro/jura/peramp-medians/peramp-median-jura-main-y1.pdf
https://data.desi.lbl.gov/desi/users/raichoor/spectro/jura/peramp-medians/peramp-median-jura-main-y2.pdf
https://data.desi.lbl.gov/desi/users/raichoor/spectro/jura/peramp-medians/peramp-median-jura-main-y3.pdf

[forero]

@rongpu

Building upon your previous work on fiber outlier detection, I've identified three additional fibers that exhibit atypical behavior in their LRG redshift distributions. These fibers were not previously mentioned in this issue, so I'm reporting them for further investigation.

Newly Identified Atypical Fibers:

• 650
• 675
• 1754

I've attached plots for each fiber showing:

• Percentile curves compared to the mean
• Time series of redshift measurements
• Redshift distribution histograms

The full analysis notebook is available here.

Do you think these fibers warrant further investigation?

Are there any known issues or characteristics of these specific fibers that might explain their behavior?

[schlafly]

650, 675 are hit by the z1 CTE. I would guess that 700, 725, and potentially 750 are also affected. Because they're the first fiber in a block of 25 they are more sensitive to the trap modeling than the others; apparently the fit isn't doing a good job there. z3C has a serious CTE which vaguely matches that date range. That looks like it might be roughly the last fiber affected by the problematic region, which I wouldn't have thought was as special of a case (i.e., it lies right on the amp boundary).

[rongpu]

Thanks @forero! These fibers have somewhat low p-values in my K-S test but still above the 1e-4 threshold.
These fibers also have low overall success rates (see #2297).

Is fiber 1754 also a CTE issue? I spot checked a few of its z=~1.7 spectra (see an example below), and there's nothing obviously wrong –– it seems that this fiber simply have more featureless spectra than the other fibers.

[rongpu]

For convenience I have created PDFs that includes the per-fiber redshift QA plots for all 5000 fibers:
https://data.desi.lbl.gov/desi/users/rongpu/redshift_qa/per_fiber_qa/jura/per_fiber_redshifts_bgs_any_zcatalog.pdf
https://data.desi.lbl.gov/desi/users/rongpu/redshift_qa/per_fiber_qa/jura/per_fiber_redshifts_lrg_zcatalog.pdf

[sbailey]

Tracking has been moved to spreadsheet (don't post link here since this is too public).