desi_zcatalog patch missing sv1 FLUX_IVAR_W1/W2

[sbailey]

Candidate for including in final fuji redshift catalogs, if we can get review by Monday
~noon Pacific Time; I think this is simple enough to be viable for that (despite the long PR description...)

This PR fixes the missing sv1 FLUX_IVAR_W1/W2 values in the stacked zcatalog files (but not in the fibermap through redrock files; too late for Fuji). These values are missing in fiberassign tiles designed prior to 20201212. desi_zcatalog --patch-missing-ivar-w12 ... identifies the rows with missing FLUX_IVAR_W1/W2 values, looks up the original values in the sv1 targeting files (i.e. what would have been propagated if that column had been included by fiberassign at the time), and patches the zcatalog columns before writing out the files.

Originally I thought I would use external catalogs curated by @stephjuneau, @Ragadeepika-Pucha, and/or @moustakas (e.g. PR #1716), but the scope of that work is growing beyond simple quick review due to the messiness of secondary targets, and also has the chicken-and-the-egg problem of using the zcatalog files to identify which TARGETIDs to curate, while I want to patch the zcatalog files while making them in the first place. By limiting the scope of this PR to just sv1 FLUX_IVAR_W1/W2, the target lookup problem becomes much simpler.

Impacted catalogs in fuji/zcatalog (see code snippet below):

• Fixed by this PR:
  • zpix-sv1-[dark,bright].fits
  • ztile-sv1-[dark,bright]-*.fits
  • ztile-[1x_depth,4x_depth,lowspeed].fits
• NOT fixed by this PR:
  • zpix-cmx-other.fits
  • ztile-cmx-other-*.fits

Thankfully it did not impact sv1-other or sv1-backup tiles, or anything post-sv1, thus simplifying the target lookup.

The code looks in $DESI_TARGET/catalogs/dr9/*/targets/sv1/resolve/*/sv1targets-*-hp-{hpix}.fits where the first * in the glob is a desitarget version number (different tiles were designed with different desitarget) and the second * is for dark vs. bright. It's a small enough number of files that I didn't try to optimize which to read. If a TARGETID appears in more than one file that's fine, since they all derive from dr9 tractor photometry and have the same FLUX_IVAR_W1/W2 values (i.e. although the desitarget version could impact whether the target was selected, it doesn't impact what the WISE photometry is).

Test files

Example output files are in /global/cfs/cdirs/desi/users/sjbailey/dev/fuji/zcat; exact commands below:

• Should be the same as fuji/zcatalog, except for where FLUX_IVAR_W1/W2<0:
  • *-sv1-bright*.fits
  • *-sv1-dark*.fits
  • ztile-1x_depth.fits
  • (there are other cases that need to be re-run, but I included only a subset of them testing cumulative, pernight, healpix, and custom coadds)
• Should be unchanged compared to fuji/zcatalog because there was nothing to fix:
  • ztile-sv2-dark-cumulative.fits
• Still has problems, but I'm not wading into CMX-targeting land for this PR, but it at least demonstrates that the code doesn't crash or otherwise make things worse if it can't find a TARGETID in the input targeting:
• ztile-cmx-other-cumulative.fits

Full disclosure: I haven't fully checked these myself, so I would appreciate an independent check with fresh eyes on this, e.g. by comparing to original targets yourself, or to one of the catalogs from @stephjuneau / @Ragadeepika-Pucha / @moustakas . Mentioning other candidate reviewers @geordie666 @schlafly @djschlegel @dstndstn @araichoor @akremin .

Code for identifying catalogs with missing FLUX_IVAR_W1/W2

import glob
import numpy as np
import fitsio

for filename in sorted(glob.glob('z*.fits')):
    columns = ['TARGETID', 'OBJTYPE', 'FLUX_IVAR_W1', 'FLUX_IVAR_W2']
    zcat = fitsio.read(filename, 1, columns=columns)
    
    bad = (zcat['FLUX_IVAR_W1']<0) | (zcat['FLUX_IVAR_W2']<0)
    bad &= (zcat['TARGETID']>0) & (zcat['OBJTYPE'] == 'TGT')
    if np.any(bad):
        nbad = np.sum(bad)
        print(f'{filename} - {nbad} missing FLUX_IVAR_W1/2 entries')
    else:
        print(f'{filename} - ok')

Commands run for test files

rx=$DESI_SPECTRO_REDUX
desi_zcatalog -i $rx/fuji/tiles/cumulative -o ztile-sv1-dark-cumulative.fits -t ~desi/labeled_proc_run_files/fuji/zcatalog/sv1-dark-tiles.txt -g cumulative --header SURVEY=sv1 PROGRAM=dark SPGRP=cumulative --patch-missing-ivar-w12
desi_zcatalog -i $rx/fuji/tiles/cumulative -o ztile-sv1-bright-cumulative.fits -t ~desi/labeled_proc_run_files/fuji/zcatalog/sv1-bright-tiles.txt -g cumulative --header SURVEY=sv1 PROGRAM=bright SPGRP=cumulative --patch-missing-ivar-w12
desi_zcatalog -i $rx/fuji/tiles/cumulative -o ztile-sv2-dark-cumulative.fits -t ~desi/labeled_proc_run_files/fuji/zcatalog/sv2-dark-tiles.txt -g cumulative --header SURVEY=sv2 PROGRAM=dark SPGRP=cumulative --patch-missing-ivar-w12
desi_zcatalog -i $rx/fuji/healpix/sv1/dark -o zpix-sv1-dark.fits -g healpix --header SURVEY=sv1 PROGRAM=dark SPGRP=healpix --patch-missing-ivar-w12
desi_zcatalog -i $rx/fuji/healpix/sv1/bright -o zpix-sv1-bright.fits -g healpix --header SURVEY=sv1 PROGRAM=bright SPGRP=healpix --patch-missing-ivar-w12
desi_zcatalog -i $rx/fuji/tiles/pernight -o ztile-sv1-bright-pernight.fits -t ~desi/labeled_proc_run_files/fuji/zcatalog/sv1-bright-tiles.txt -g pernight --header SURVEY=sv1 PROGRAM=bright SPGRP=pernight --patch-missing-ivar-w12
desi_zcatalog -i $rx/fuji/tiles/1x_depth -o ztile-1x_depth.fits --header SPGRP=1x_depth --patch-missing-ivar-w12
desi_zcatalog -i $rx/fuji/tiles/cumulative -o ztile-cmx-other-cumulative.fits -t ~desi/labeled_proc_run_files/fuji/zcatalog/cmx-other-tiles.txt -g cumulative --header SURVEY=cmx PROGRAM=other SPGRP=cumulative --patch-missing-ivar-w12

[araichoor]

sorry for the late feedback.

I did have a look at the code, looks good overall, just few comments:

• hpix8: wouldn t it be safer to also explicitly setnest? (I think it s nest=True);
• targetid2idx: I m not sure if that happens for the restricted set of OBJTYPE=="TGT" rows, but what if a TARGETID appears multiple times? (I think that s properly handled in the code, but just mentioning in case)

[sbailey]

Thanks for the review. Comments:

hpix8: wouldn t it be safer to also explicitly setnest? (I think it s nest=True);

This uses desimodel.footprint.radec2pix which wraps healpy.ang2pix and enforces nest=True (i.e. radec2pix purposefully doesn't give you that choice to ensure that we always use nest=True)

targetid2idx: I m not sure if that happens for the restricted set of OBJTYPE=="TGT" rows, but what if a TARGETID appears multiple times? (I think that s properly handled in the code, but just mentioning in case)

Good question, but the code handles that case:

When a TARGETID appears in multiple target catalogs (different desitarget versions), the assumption (cross checked with Adam) is that the WISE photometry is the same for every version since they all trace back to the same DR9 release (that's partially why this PR doesn't try to fix CMX, where it gets messier with DR8 vs. DR9).

It is also robust to the case where the TARGETID appears in more than one row of the input redshift catalogs (e.g. perexp and pernight) due to the for i in np.where(hpixmiss)[0]: loop tracking every row with missing FLUX_IVAR_W1/W2, not caring if the same target is corrected on multiple rows.

I also cross checked that my patch values match the patch values found by @moustakas in his tractorphot files in PR #1716, and that the FLUX_W1/W2 (not IVAR) values match (e.g. to check for an indexing bug).