tilepix.json identifies healpixels with no corresponding files in Everest

[akremin]

Another issue brought to my attention by @araichoor:

tilepix.json in everest maps tiles-petals to healpixels that don't have files on disk.

One example is tile 80606, petal 8:

import json
with open('/global/cfs/cdirs/desi/spectro/redux/everest/healpix/tilepix.json', 'r') as fil: 
     tilepix = json.load(fil)  
print(tilepix['80606']['8'] )  

Output:

[17684, 17685, 17686, 17687, 17692]

Among these, only 17684 exists:

kremin@cori03:/global/cfs/cdirs/desi/spectro/redux/everest/healpix> ls ./*/*/176* 
./sv1/bright/176:
17665  17666  17667  17668  17669  17670  17671  17672  17673  17674  17675  17676  17677  17678  17680  17681  17682  17683  17688  17696  17697  17698  17699

./sv1/dark/176:
17680  17681  17682  17683  17684

[araichoor]

if that helps: it s not a "corner case", i.e. most of tileid=80606 is missing:

>>> import json
>>> with open('/global/cfs/cdirs/desi/spectro/redux/everest/healpix/tilepix.json', 'r') as fil: 
...      tilepix = json.load(fil)  
>>> healpixs = []
>>> for petal in tilepix["80606"]:
...     healpixs = np.unique(healpixs + tilepix["80606"][petal]).tolist()
>>> healpixs
[17680, 17681, 17682, 17683, 17684, 17685, 17686, 17687, 17688, 17689, 17690, 17691, 17692, 17693, 17694, 17695, 17716]
>>> for healpix in healpixs:
...     fn = "/global/cfs/cdirs/desi/spectro/redux/everest/healpix/sv1/dark/{}/{}/redrock-sv1-dark-{}.fits".format(healpix//100, healpix, healpix)
...     if not os.path.isfile(fn):
...             print("no {}".format(fn))
... 
no /global/cfs/cdirs/desi/spectro/redux/everest/healpix/sv1/dark/176/17685/redrock-sv1-dark-17685.fits
no /global/cfs/cdirs/desi/spectro/redux/everest/healpix/sv1/dark/176/17686/redrock-sv1-dark-17686.fits
no /global/cfs/cdirs/desi/spectro/redux/everest/healpix/sv1/dark/176/17687/redrock-sv1-dark-17687.fits
no /global/cfs/cdirs/desi/spectro/redux/everest/healpix/sv1/dark/176/17688/redrock-sv1-dark-17688.fits
no /global/cfs/cdirs/desi/spectro/redux/everest/healpix/sv1/dark/176/17689/redrock-sv1-dark-17689.fits
no /global/cfs/cdirs/desi/spectro/redux/everest/healpix/sv1/dark/176/17690/redrock-sv1-dark-17690.fits
no /global/cfs/cdirs/desi/spectro/redux/everest/healpix/sv1/dark/176/17691/redrock-sv1-dark-17691.fits
no /global/cfs/cdirs/desi/spectro/redux/everest/healpix/sv1/dark/176/17692/redrock-sv1-dark-17692.fits
no /global/cfs/cdirs/desi/spectro/redux/everest/healpix/sv1/dark/176/17693/redrock-sv1-dark-17693.fits
no /global/cfs/cdirs/desi/spectro/redux/everest/healpix/sv1/dark/176/17694/redrock-sv1-dark-17694.fits
no /global/cfs/cdirs/desi/spectro/redux/everest/healpix/sv1/dark/176/17695/redrock-sv1-dark-17695.fits
no /global/cfs/cdirs/desi/spectro/redux/everest/healpix/sv1/dark/177/17716/redrock-sv1-dark-17716.fits

[araichoor]

and for sv3-dark, it also looks like those 10 healpix reductions are missings:

/global/cfs/cdirs/desi/spectro/redux/everest/healpix/sv3/dark/98/9837/redrock-sv3-dark-9837.fits
/global/cfs/cdirs/desi/spectro/redux/everest/healpix/sv3/dark/98/9839/redrock-sv3-dark-9839.fits
/global/cfs/cdirs/desi/spectro/redux/everest/healpix/sv3/dark/98/9842/redrock-sv3-dark-9842.fits
/global/cfs/cdirs/desi/spectro/redux/everest/healpix/sv3/dark/98/9848/redrock-sv3-dark-9848.fits
/global/cfs/cdirs/desi/spectro/redux/everest/healpix/sv3/dark/153/15359/redrock-sv3-dark-15359.fits
/global/cfs/cdirs/desi/spectro/redux/everest/healpix/sv3/dark/160/16042/redrock-sv3-dark-16042.fits
/global/cfs/cdirs/desi/spectro/redux/everest/healpix/sv3/dark/259/25955/redrock-sv3-dark-25955.fits
/global/cfs/cdirs/desi/spectro/redux/everest/healpix/sv3/dark/259/25958/redrock-sv3-dark-25958.fits
/global/cfs/cdirs/desi/spectro/redux/everest/healpix/sv3/dark/259/25961/redrock-sv3-dark-25961.fits
/global/cfs/cdirs/desi/spectro/redux/everest/healpix/sv3/dark/259/25964/redrock-sv3-dark-25964.fits

note that those are "patched".

[araichoor]

if it helps, for completeness, here is the list + sky position of the 13 main-dark pixels I see as missing:

/global/cfs/cdirs/desi/spectro/redux/everest/healpix/main/dark/150/15051/redrock-main-dark-15051.fits
/global/cfs/cdirs/desi/spectro/redux/everest/healpix/main/dark/260/26003/redrock-main-dark-26003.fits
/global/cfs/cdirs/desi/spectro/redux/everest/healpix/main/dark/260/26004/redrock-main-dark-26004.fits
/global/cfs/cdirs/desi/spectro/redux/everest/healpix/main/dark/260/26005/redrock-main-dark-26005.fits
/global/cfs/cdirs/desi/spectro/redux/everest/healpix/main/dark/260/26006/redrock-main-dark-26006.fits
/global/cfs/cdirs/desi/spectro/redux/everest/healpix/main/dark/450/45036/redrock-main-dark-45036.fits
/global/cfs/cdirs/desi/spectro/redux/everest/healpix/main/dark/450/45037/redrock-main-dark-45037.fits
/global/cfs/cdirs/desi/spectro/redux/everest/healpix/main/dark/450/45038/redrock-main-dark-45038.fits
/global/cfs/cdirs/desi/spectro/redux/everest/healpix/main/dark/450/45039/redrock-main-dark-45039.fits
/global/cfs/cdirs/desi/spectro/redux/everest/healpix/main/dark/450/45040/redrock-main-dark-45040.fits
/global/cfs/cdirs/desi/spectro/redux/everest/healpix/main/dark/450/45041/redrock-main-dark-45041.fits
/global/cfs/cdirs/desi/spectro/redux/everest/healpix/main/dark/450/45042/redrock-main-dark-45042.fits
/global/cfs/cdirs/desi/spectro/redux/everest/healpix/main/dark/450/45043/redrock-main-dark-45043.fits

[moustakas]

This looks like a fairly straightforward fix which I can take on (unless @sbailey or someone else has a fixed started in a branch already).

[moustakas]

Fixed in #1614.

[moustakas]

For the record, I re-generated the Everest tilepix.json and tilepix.fits files using #1614, which took just under 38 minutes with a Cori login node:

time desi_map_tilepix --reduxdir /global/cfs/cdirs/desi/spectro/redux/everest --nside 64 -o new-tilepix.fits

I then confirmed that all the issues discussed here by @araichoor and @akremin are addressed. In case anyone is inspired to check the new files, I've put them here:
https://data.desi.lbl.gov/desi/users/ioannis/tmp/new-tilepix.fits
https://data.desi.lbl.gov/desi/users/ioannis/tmp/new-tilepix.json