Fiber crosstalk

[julienguy]

Code to compute and correct for fiber cross-talk at the spectral level

Fiber cross-talk (or fiber to fiber contamination) is induced by extended Lorentzian PSF tails on the spectrographs CCDs. It is a small effect in most of the wavelength range but becomes large (reaching 1%) in the NIR for wavelength > 9000A. It causes a visible spurious increase at the largest NIR wavelength in the spectra of faint targets which fiber is next to a bright one in the spectrographs.

2D modeling

The PSF tails can be modeled accurately using data from the parse test fiber slit. The following figures show the PSF tails measured with continuum lamp data from SM1 NIR camera (solid curves), along with the model (dashed curves) obtained by adding a Lorentzian tail to the 2D PSF model used for extraction ( in specter ).

A precise measurement shows that the PSF tails are asymmetric and depend on the position of the fiber in the CCD.
On the following figure the specter PSF tails (dashed curve) are also asymmetric,

The 2D PSF tail model fitted on the sparse fiber data is found consistent with the signal measured on the edge fibers for exposures with the continuum lamps on the Mayall dome white spot.

1D correction

Despite having a rather good modeling of the 2D PSF tails, we resorted not to use this approach to correct for the fiber cross-talk.
Some cross-talk is visible on fiber F+2 (next to next fiber). Correcting for this with a 2D PSF would require to extend its size/footprint to more than 2x15 pixels on each dimension which would make the extraction prohibitively slow and would also make the extraction even more sensitive to unmasked cosmic ray pixels.

For this PR we resort to use an empirical 1D correction, meaning at the spectral level, after extraction.

• desi_compute_fiber_crosstalk , measure the crosstalk
  • runs on a very large set of sframe files for a given camera (or multiple cameras of the same arm (b,r, or z) but different spectrographs)
  • computes for each fiber bundle (group of 25, to detect variations across the CCD) the contamination from fiber F-2,-1,+1+2, as a function of wavelength, after convolving the contaminant spectra in 1D by the correct kernel derived from the known 2D PSF tails (and taking into account the fiber separation in the CCD), accounting for the offset in the wavelength solution between the neighboring fibers.
  • use only sky fibers for the contaminated set, and bright fibers with SNR>3 (per spectral pix) for the contaminant (to avoid bias in the cross talk measurement).
  • solve a linear system with 4 parameters for each wavelength and each bundle to compute the crosstalk values at -2,-1,+1,+2.
  • returns a table with 20 * 4 spectra of contamination

Example, for all z cameras (z0...z9 combined), for all exposures with sky subtraction from Dec 2020.
Mean cross-talk over all 20 bundles (z cameras):

Cross-talk at 9600+-50 A as a function of fiber number (one value per bundle, z cameras):

Mean cross-talk for the b cameras (much smaller ~0.001;

Mean cross-talk for the r cameras (<0.001);

No variation with fiber number is detected for the blue and red cameras.

• desi _fit_fiber_crosstalk , fit the crosstalk as a function of fiber, delta-fiber, and wavelength
  • fit the output of desi_compute_fiber_crosstalk with a model that varies from camera to camera

Example for NIR (dots are the measurements and the curves the model)

• save the output fit parameters in a yaml file. see https://github.com/desihub/desispec/blob/fiber-crosstalk/py/desispec/data/fiber-crosstalk.yaml

• correction function

  • implemented in py/desispec/fibercrosstalk.py (using same kernel convolution etc ...)
  • applied in desi_process_exposure with option --xtalk.
  • option set by default in desi_proc and disabled with --no-xtalk

Results presented in another post.

[coveralls]

Coverage decreased (-0.003%) to 28.846% when pulling 06490bd on fiber-crosstalk into c883f71 on master.

[julienguy]

Results improved after fitting the cross-talk parameters for each of the 30 cameras independently.
Below are examples compared to the blanc released for coadded cframe files of tile 80613 (Lynx, BGS+MWS),
focusing on some problematic fibers reported by Boris and Ting-Wen (I excluded fibers where there is another problem like a bad amplifier or a bad column).

based on desi_proc runs for all the exposures of tile 80613.

[sbailey]

NOTE: original post of this comment had a bookkeeping error caught by @julienguy; I'm updating the numbers and text below.

The latest version of this branch is distinctly better than previous versions for their impact on redshift success. Testing on:

TILEID TARGETS  NIGHT     EXPIDS
80608  ELG      20201216  68317,68327,68328
80609  QSO+LRG  20201216  68334,68336,68337,68338,68339,68340
80608  ELG      20201221  69249,69251,69252,69253
80613  BGS      20201221  69225,69226,69227,69228,69229,69230

the BGS tile 80613 in particular has 22 (0.9%) targets move from bad in Blanc to good in this PR, with none going from good to bad. 80609 was mostly unchanged: 6 from bad to good and 4 from good to bad. 80608 was similar, with 2-3 targets changing state per night but no big win or loss.

Results in /global/cfs/cdirs/desi/users/sjbailey/spectro/redux/xtalk3, with 3 summary files:

• bad2good.csv : targets that were bad in Blanc but good in xtalk3 (this PR)
• good2bad.csv : targets that were good in Blanc but became bad in xtalk3
• stillbad.csv : targets that were bad in both Blanc and xtalk3

where "bad" = |deltav| > 1000 km/s and ZWARN=0.

So: good win for BGS, neutral for other target classes, i.e. an overall improvement. Thanks.

[julienguy]

Added minor modifications. Saved a new version of the parameter file from a run on all of the recent data. This PR is ready for review and could be merged.

[sbailey]

Thanks; I will re-rerun and evaluate with latest params as a cross check (hopefully without bookkeeping errors this time). Since we're done with the Cascades high priority deep tiles and substantially done with other nights, I'd like to hold off on merging this until Cascades is completely finished, and then fast-track a Denali run using Cascades calibs + extractions plus this and potentially a few other high priority fixes.

[julienguy]

A few updated plots for the record.

[sbailey]

There is a problem with the ivar correction to fibers that neighbor fibers masked as BADFIBER (fibermap.FIBERSTATUS bit 16), resulting poor redshift performance for those neighboring fibers. Examples from exposures/20201215/00067968/cframe-z4-00067968.fits are fibers that neighbor fibers 2036, 2051, 2132, 2156.

[julienguy]

The code was computing the crosstalk from all bad fibers instead of considering only the ones with bad positioning and no other issue.

[sbailey]

The latest push definitely improved things, but still has ~1% worse performance on ELG tile 80606 compared to Cascades. The dominant problem is targets that went from ZWARN=0 to ZWARN=4 while retaining the same redshift. This appears to come from a scaling of ivar leading to a scaling of chi2 leading to shrinking deltachi2. Examples below; note the clustering of FIBERs e.g. 1733,1735 neighboring masked fiber 1734.

TARGETID         FIBER  Z_VI    Z1      Z2    ZW1  ZW2 dCHI2_1 dCHI2_2
39627640570644285  318 1.5156  1.5157  1.5157   0  4     9.6     8.9
39627652578938846  969 0.8602  0.8602  0.8061   0  4   255.2     4.6
39627664608206027 1218 1.1560  1.1560  1.1560   0  4    16.7     7.4
39627670631226497 1227 1.2635  1.2634  0.4151   0  4   111.4     1.1
39627664616589876 1495 0.9613  0.9613  0.9614   0  4   120.1     7.0
39627664616594294 1498 0.7754  0.7754  0.7754   0  4   289.0     0.3
39627694744279867 1566 1.4097  1.4098  1.4096   0  4   121.3     0.1
39627688721255460 1733 1.2187  1.2187  1.2187   0  4   115.6     0.0
39627676671018589 1735 0.8372  0.8372  0.8372   0  4  2940.5     0.7
39627688708677379 1774 0.7379  0.7380  0.7381   0  4    88.3     0.1
39627676658441102 1869 0.7835  0.7835  0.7835   0  4   834.4     0.7
39627700784072035 2035 0.8114  0.8114  0.8114   0  4  2248.1     4.5
39627706811289017 2037 0.8708  0.8707  0.8707   0  4  1115.4     1.3
39627682698240106 2133 1.0220  1.0219  1.0218   0  4    64.9     0.3
39627688729646901 2134 0.8462  0.8462  0.8462   0  4   248.0     7.4
39627706811288852 2157 0.7930  0.7929  0.5454   0  4   114.5     0.5
39627712863669368 2551 1.3793  1.3794  0.7720   0  4    43.7     0.7
39627688738029727 2840 0.1161  0.1161  0.1161   0  4   195.6     7.8
39627688733843868 2841 1.4758  1.4757  1.4757   0  4    37.9     0.0
39627706840650721 3167 0.0698  0.0699  0.0699   0  4    64.5     8.6
39627694773635534 3397 1.2988  1.2987  1.2987   0  4    85.9     0.1
39627700813438998 3399 1.0072  1.0066  1.0073   0  4   208.8     0.3
39627676700379440 3586 0.1413  0.1413  0.1413   0  4  2103.4     1.4
39627688759003012 3628 1.4823  1.4823  1.4823   0  4    10.4     5.7
39627682727593822 3916 1.0244  1.0242  1.0241   0  4    24.6     0.2

[julienguy]

Some ZWARN!=0 were due to artificially large variance added to the contaminated spectra. I have chosen to ignore the variance of the contaminating spectrum because accounting for it would add extra complexity and possibly more fragility. I only add to the variance of the contaminated spectrum a term reflecting a systematic error of 10% on the contamination. This is still not perfect because it is considered uncorrelated across wavelength, but we do not have yet the data model for correlated errors. Could you try rerunning your test with this correction?

[sbailey]

With this version, the changes are an improvement for all major target classes (yay!). Using the nightly redshift completeness and purity for visual inspection tiles 80605, 80606, 80607, 80608, 80609, 80610, 80613 on nights 20201214, 20201216, 20201218, 20201221, 20210205, 20201215, 20201217, 20201219, 20210130, 20210208, the % improvement relative to Cascades is:

TYPE     NUM    dComp   dPurity
 ELG   40900    +0.0    +0.1
 LRG   13564    +0.4    +0.2
 QSO   12664    +0.1    +0.1
 BGS    5186    +1.4    +1.4

I think (but haven't verified) that BGS gets the most improvement because they are most likely to be neighboring a bright MWS target, while on ELG/LRG/QSO the primary contamination is from rare standard stars.

Now that this is vetted, I changed the desi_process_exposures --xtalk option to --no-xtalk to match the desi_proc --no-xtalk option (and inverted the desi_proc logic for whether to add --xtalk or --no-xtalk). Now both scripts have the default to do the recommended thing (apply crosstalk).

Heads up: while testing this I was sidetracked by chasing "lower" efficiency due to desi_process_exposure (as called by desi_proc -> runcmd) exiting with no error messages and return code=0 but without having written its output files (!). The coadds/redshifts picked up whatever was there and ran redshifts missing some exposures, resulting in lower S/N and lower efficiency. I was unable to reproduce the problem in multiple reruns today so I think it was a transient NERSC problem and not a feature of this branch, but we should keep our eye out for this.

Merging now (after fixing up the conflicts).