Fibers yielding unusual redshift distributions in Y1 himalayas data #1946
Comments
|
These are very useful, @rongpu, thanks! Please try making them as redshift vs. time to see if the pileup of problematic redshifts occurs on individual non-contiguous nights, or time periods of mostly contiguous nights, or is spread out over the entire survey. |
|
Thanks @sbailey. The occurrence of bad redshifts vs time would be one of the things that we will examine when we look into each individual case. I'm hoping that for many of those cases we can identify the problems and find fixes for them in iron, rather than having to throw out these fibers (either altogether or in some time window). |
|
I looked at a few cases vs. time. While I see some time dependence, they all look like outliers for most of Y1. |
|
Fiber 1008
With corrected pixmask (bottom left image):
Correction committed in calib SVN. |
|
Fibers 2675-2680 |
|
@julienguy and @rongpu beat me to it on the diagnosis of fiber 1008, but FYI this is the kind of redshift vs. time plot that I was thinking: from datetime import datetime
import numpy as np
import fitsio
import matplotlib.pylab as plt
zcatalog = fitsio.read('ztile-main-dark-cumulative.fits', 'ZCATALOG',
columns=('Z', 'ZWARN', 'TILEID', 'FIBER', 'LASTNIGHT', 'DESI_TARGET'))
fiber = 1251
zcat = zcatalog[zcatalog['FIBER'] == fiber]
dates = np.asarray([datetime.strptime(str(night), '%Y%m%d') for night in zcat['LASTNIGHT']])
isLRG = (zcat['DESI_TARGET'] & 1) != 0
plt.plot(dates[isLRG], zcat['Z'][isLRG], '.', alpha=0.2)
plt.xlabel('date'); plt.ylabel('Redshift')
plt.title(f'Fiber {fiber} LRGs')
plt.show()
In all of the cases I've spot checked, the problem appears after the summer 2021 shutdown as isn't isolated to a few nights or a small time range. Himalayas preprocessing was run before we had @Waelthus new monthly darks, so if the problems are in the dark model itself they may already be fixed, but testing is a non-trivial amount of processing so its good to chase down individual cases and fix things like the pixmask (fiber 1008 issue) first. |
|
Fiber 3994-3995 Bad (parallel) charge transfer in CCD row x=3922 (starting at 0) in amplifier D of CCD r7. See the trails in the 7th fiber from the left in the following plot
|
|
Fiber 4349 Dead pixel at x=2752 y=3472 and for all y>3472 in amplifier D of b8.
|
|
Thanks @ashleyjross, @sbailey. Here I'm plotting redshift vs date for all the flagged fibers using Stephen's script. None issues are isolated to a few nights. For some fibers (including 1008, 1251, 1219, 3038, 4349), there are fewer pileups before the 2021 summer shutdown, although it might not necessarily mean that the pre-shutdown data is glitch-free or that their redshifts are correct. FIBER 1008 FIBER 1098 FIBER 1219 FIBER 1251 FIBER 2675 FIBER 2676 FIBER 2678 FIBER 2679 FIBER 2680 FIBER 3994 FIBER 3995 FIBER 4349 FIBER 4720 Below are fibers flagged only in BGS: FIBER 2250 FIBER 2251 FIBER 2252 FIBER 2253 FIBER 3038 |
|
To keep track of progress, here is the status of the investigation into the problematic fibers. I will keep editing this post with updates. Fixed or not fixable: No fix yet: |
|
FIBER 4720 Left: science exposure; right: arc exposure. |
|
Fiber 466 |
|
Fibers 2250-2253 Edit: forgot to mention, this is r4. Left: sframe in himalayas; right: sframe in daily; EXPID= 95297 |
|
@schlafly is the same issue as the r4 one we've been discussing on Slack for the data since r4 got fixed? |
|
FIBER 1219 and FIBER 1251
I have not updated the pixmask yet. Arc exposure EXPID=113356 |
|
About Fibers 2250-2253, it seems that the dominant effect is on the 'tpcorr' correction.
|
|
Fiber 1098 Again a tpcorr issue.
Recomputing the sky model without tpcorr
Rerunning with the tpcorr option gives some systematic offset, but smaller than the results from himalayas. So something else has changed since then.
|
|
Fiber 3038 We see a dip in the sky subtracted frames for bright time exposures. From left to right, preproc of one arc lamp exposure ( /global/cfs/cdirs/desi/spectro/redux/himalayas/preproc/20220211/00122115/preproc-r6-00122115.fits.gz) (left) , extracted spectra from that exposure with fiber 3038 in red (right), extracted spectra from a continuum lamp exposure (center) during the same day.
The excess flux is consistent with the ratio of exposure times (5sec and 120sec). We don't see anything in bias and dark.
|
|
Fiber 3038 is connected to robot M01480 and located in hole 434 of petal_loc 6 (petal_id 11). Its neighbors are M01382,1106,1926,8347,6300,6602 (2 disabled robots in bold italics). |
|
We see this spurious signal also in the blue camera because the wavelength of 5880A is in the dichroic transition range so it is indeed light that goes through the fiber. |
|
On the focal plane, fiber 3038 is near the edge of the focal plane. Its 6 neighbors are fibers [3011, 3027, 3030, 3379, 3383, 3394]. |
|
The signal is seen only in fiber 3038. |
|
Now that we know what we are looking for, at other dates, it's visible in other isolated fibers. And one can see this pollution in nightwatch. See for instance https://nightwatch.desi.lbl.gov/20220530/00137320/preproc-r8-00137320-4x.html . |
|
I have rerun this N(z) test on the iron zcatalogs. The results for the previously flagged fibers (still flagged fibers are in bold): 466: still flagged in BGS N(z) (expected as no fix was implemented) Two fibers that appeared normal in himalayas are flagged in iron: 4321: newly flagged fiber by LRG N(z) (many with z=~0.5 and they only appeared between 2021-10 and 2022-01) Based on the N(z) test, we may need to mask the following fibers from the iron LSS catalog:
Below are redshift vs date plots for some of the still flagged fibers in iron: Fiber 466 Fiber 1008 Fiber 2676 Fiber 3038 Fiber 3994 Fiber 4321 Fiber 4891 I have also updated the redshift vs fiber plots for iron, and many improvements (not just those implemented here) are visible compared to himalayas. |
Edit: the status of the fixes is listed here: #1946 (comment)
Previously I had identified fibers that yield redshift distributions that are not consistent with the overall redshift distribution (based on K-S tests). I posted the details here on the #spectroscopic-systematics slack channel.
To help identify the causes of the bad redshifts, here I'm going through each problematic fiber, providing an example spectrum and the corresponding coadd file path.
So far I have only gone through the bad fibers identified from LRG and BGS observations. The LRG bad fibers are:
1008 1098 1219 1251 2675 2676 2678 2679 2680 3994 3995 4349 4720
A few more fibers are not flagged in LRG observations but are flagged in BGS; they are
466 2250 2251 2252 2253 3038
Starting with the LRG bad fibers:
FIBER 1008 p-value = 2.2558e-139 n_tot = 564 (n_tot is the total number of LRG observations by that fiber)
/global/cfs/cdirs/desi/spectro/redux/himalayas/tiles/cumulative/9195/20211128/coadd-2-9195-thru20211128.fits
FIBER 1098 p-value = 0.00014002 n_tot = 593
/global/cfs/cdirs/desi/spectro/redux/himalayas/tiles/cumulative/1517/20211031/coadd-2-1517-thru20211031.fits
FIBER 1219 p-value = 4.7286e-25 n_tot = 591
/global/cfs/cdirs/desi/spectro/redux/himalayas/tiles/cumulative/10791/20220429/coadd-2-10791-thru20220429.fits
FIBER 1251 p-value = 2.534e-44 n_tot = 652
/global/cfs/cdirs/desi/spectro/redux/himalayas/tiles/cumulative/10376/20211212/coadd-2-10376-thru20211212.fits
Fibers 2675-2680 appear to be cause by the same problem. Here I'm only giving an example for FIBER 2675.
FIBER 2675 p-value = 3.2445e-12 n_tot = 139
/global/cfs/cdirs/desi/spectro/redux/himalayas/tiles/cumulative/9360/20211006/coadd-5-9360-thru20211006.fits
FIBER 3994 p-value = 4.331e-42 n_tot = 639
/global/cfs/cdirs/desi/spectro/redux/himalayas/tiles/cumulative/4703/20220125/coadd-7-4703-thru20220125.fits
FIBER 3995 p-value = 0.0005322 n_tot = 595
/global/cfs/cdirs/desi/spectro/redux/himalayas/tiles/cumulative/3028/20220207/coadd-7-3028-thru20220207.fits
FIBER 4349 p-value = 2.9092e-30 n_tot = 611
/global/cfs/cdirs/desi/spectro/redux/himalayas/tiles/cumulative/2338/20211029/coadd-8-2338-thru20211029.fits
FIBER 4720 p-value = 8.8502e-05 n_tot = 442
/global/cfs/cdirs/desi/spectro/redux/himalayas/tiles/cumulative/1650/20220219/coadd-9-1650-thru20220219.fits
Problematic fibers that are only identified from BGS redshift distributions:
FIBER 466 p-value = 1.3478e-09 n_tot = 969
/global/cfs/cdirs/desi/spectro/redux/himalayas/tiles/cumulative/26253/20211020/coadd-0-26253-thru20211020.fits
Fibers 2250-2253 appear to have the same issue.
FIBER 2250 p-value = 0.00013892 n_tot = 503
/global/cfs/cdirs/desi/spectro/redux/himalayas/tiles/cumulative/21308/20211022/coadd-4-21308-thru20211022.fits
FIBER 2251 p-value = 0.00054949 n_tot = 520
/global/cfs/cdirs/desi/spectro/redux/himalayas/tiles/cumulative/20040/20210628/coadd-4-20040-thru20210628.fits
FIBER 2252 p-value = 2.3389e-06 n_tot = 505
/global/cfs/cdirs/desi/spectro/redux/himalayas/tiles/cumulative/20915/20210523/coadd-4-20915-thru20210523.fits
FIBER 2253 p-value = 0.00016885 n_tot = 521
/global/cfs/cdirs/desi/spectro/redux/himalayas/tiles/cumulative/24451/20211022/coadd-4-24451-thru20211022.fits
FIBER 3038 p-value = 6.2344e-06 n_tot = 1524
/global/cfs/cdirs/desi/spectro/redux/himalayas/tiles/cumulative/26213/20220211/coadd-6-26213-thru20220211.fits
The text was updated successfully, but these errors were encountered: