90prime images with mucked up FWHM values -- lots of noise

[moustakas]

Creating this issue as I track down the 90prime images in DR8 without -photom.fits files. Lots of complaining that we should be catching much earlier.

ioannis@cori08:~/dr8c% python ~/repos/git/legacypipe/py/legacyzpts/legacy_zeropoints.py --image 90prime/BOK_CP/CP20180419/ksb_180420_110216_ooi_r_v1.fits.fz --outdir . --camera 90prime --calibdir calib --threads 1
Reading /global/homes/i/ioannis/repos/git/legacypipe/py/legacyzpts/data/90prime-bad_expid.txt
Working on image 1/1: 90prime/BOK_CP/CP20180419/ksb_180420_110216_ooi_r_v1.fits.fz
CP Header: EXPNUM =  82280092
CP Header: PROCDATE =  2018-04-21T06:40:34
CP Header: PLVER =  V2.1
CP Header: PLPROCID =  915ddf1
File not found ./90prime/CP20180419/ksb_180420_110216_ooi_r_v1-survey.fits
File not found ./90prime/CP20180419/ksb_180420_110216_ooi_r_v1-annotated.fits
File not found calib/90prime/psfex-merged/82280/90prime-82280092.fits
File not found calib/90prime/splinesky-merged/82280/90prime-82280092.fits
File not found ./90prime/CP20180419/ksb_180420_110216_ooi_r_v1-photom.fits
TIMING:before-run  Wall: 0.17 s, CPU: 0.03 s
CP Header: EXPNUM =  82280092
CP Header: PROCDATE =  2018-04-21T06:40:34
CP Header: PLVER =  V2.1
CP Header: PLPROCID =  915ddf1
File not found calib/90prime/splinesky-merged/82280/90prime-82280092.fits
File not found calib/90prime/psfex-merged/82280/90prime-82280092.fits
Fpack quantization method: SUBTRACTIVE_DITHER_2
Traceback (most recent call last):
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacyzpts/legacy_zeropoints.py", line 430, in read_weight
    fixed = fix_weight_quantization(wt, fn, self.ext, self.slc)
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/image.py", line 1533, in fix_weight_quantization
    raise ValueError('fix_weight_quantization: ZZERO is not all zero: [%.g, %.g]!' % (np.min(zzero), np.max(zzero)))
ValueError: fix_weight_quantization: ZZERO is not all zero: [4e+03, 5e+03]!

> WARNING: No source with appropriate FWHM found!!


> WARNING: No appropriate source found!!


> WARNING: 1st context group removed (not enough samples)


> WARNING: No source with appropriate FWHM found!!


> WARNING: No appropriate source found!!

Fpack quantization method: SUBTRACTIVE_DITHER_2
Traceback (most recent call last):
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/image.py", line 786, in read_invvar
    fixed = fix_weight_quantization(invvar, self.wtfn, self.hdu, slice)
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/image.py", line 1533, in fix_weight_quantization
    raise ValueError('fix_weight_quantization: ZZERO is not all zero: [%.g, %.g]!' % (np.min(zzero), np.max(zzero)))
ValueError: fix_weight_quantization: ZZERO is not all zero: [4e+03, 5e+03]!
/global/common/software/desi/cori/desiconda/20190311-1.2.7-img/aux/lib/python3.6/site-packages/astrometry/util/miscutils.py:54: FutureWarning: `rcond` parameter will change to the default of machine precision times ``max(M, N)`` where M and N are the input matrix dimensions.
To use the future default and silence this warning we advise to pass `rcond=None`, to keep using the old, explicitly pass `rcond=-1`.
  res = np.linalg.lstsq(A, b)
Fpack quantization method: SUBTRACTIVE_DITHER_2
Traceback (most recent call last):
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacyzpts/legacy_zeropoints.py", line 430, in read_weight
    fixed = fix_weight_quantization(wt, fn, self.ext, self.slc)
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/image.py", line 1533, in fix_weight_quantization
    raise ValueError('fix_weight_quantization: ZZERO is not all zero: [%.g, %.g]!' % (np.min(zzero), np.max(zzero)))
ValueError: fix_weight_quantization: ZZERO is not all zero: [3e+03, 5e+03]!

> WARNING: No source with appropriate FWHM found!!


> WARNING: No appropriate source found!!
[snip]

[dstndstn]

Okay, the complaint about ZZERO is from my recent attempt to find the fpack quantization level. In this case, I don't think it's even required, since this image is using SUBTRACTIVE_DITHER_2, the one where it keeps zeros as zeros after compression.

The "No source with appropriate FWHM found" would be coming from PsfEx, I expect.

[moustakas]

Yes, the FWHM complaint is the bug I'm actually trying to track down (it's what leads to not generating the -photom files.)

Separately, maybe we can capture the compression Traceback stuff.

[dstndstn]

I explicitly raise that exception because the test case I was looking at had ZZERO all zero and I didn't want to think too hard about it... Well look at this one tomorrow.

[moustakas]

For the record and thanks to @mlandriau, these are the 29 90prime images without -photom files.

ksb_180420_110216_ooi_r_v1                                                                        
ksb_180420_094712_ooi_r_v1                                                                        
ksb_180420_093738_ooi_r_v1                                                                        
ksb_180420_110703_ooi_r_v1                                                                        
ksb_160206_022011_ooi_g_v1                                                                        
ksb_160206_021614_ooi_g_v1                                                                        
ksb_170321_054024_ooi_r_v1
ksb_170429_070946_ooi_r_v1
ksb_170429_065644_ooi_r_v1
ksb_170429_084036_ooi_r_v1
ksb_170429_071921_ooi_r_v1
ksb_170429_032009_ooi_g_v1
ksb_170429_104053_ooi_r_v1
ksb_170429_060558_ooi_r_v1
ksb_170429_083548_ooi_r_v1
ksb_170429_052346_ooi_r_v1
ksb_170429_070458_ooi_r_v1
ksb_170429_084632_ooi_r_v1
ksb_170429_050958_ooi_g_v1
ksb_160706_045718_ooi_g_v1
ksb_170128_124934_ooi_g_v1
ksb_170128_130450_ooi_g_v1
ksb_170128_123919_ooi_g_v1
ksb_170128_123404_ooi_g_v1
ksb_170128_124429_ooi_g_v1
ksb_170128_122901_ooi_g_v1
ksb_170128_125945_ooi_g_v1
ksb_160606_084503_ooi_r_v1
ksb_160606_085059_ooi_r_v1

[moustakas]

I've been drilling down into the first exposure, CP20180419/ksb_180420_110216_ooi_r_v1, and I can confirm that the reason we don't get photom files is that PSFEx fails with the warning

> WARNING: No source with appropriate FWHM found!!
> WARNING: No appropriate source found!!

The reason seems to be that the seeing is absolutely atrocious. Below is the standard diagnostic diagram that PSFEx uses to find stars from which to generate the PSF for this exposure (blue) compared to a more typical "good" exposure (orange). And indeed, the CP header claims a mean seeing of 4.4 arcsec FWHM.

Assuming all of these are comparably bad (I'll double-check) I think we should add all 29 of these images to the bad exposure list. We should also have the code not generate the down-stream files if/when this happens -- who knows what's in those files.

[moustakas]

Here are the 29 files with their CP "seeing" values (arcsec, presumably FWHM):

images/90prime/BOK_CP/CP20180419/ksb_180420_110216_ooi_r_v1.fits.fz 4.38
images/90prime/BOK_CP/CP20180419/ksb_180420_094712_ooi_r_v1.fits.fz 3.62
images/90prime/BOK_CP/CP20180419/ksb_180420_093738_ooi_r_v1.fits.fz 4.99
images/90prime/BOK_CP/CP20180419/ksb_180420_110703_ooi_r_v1.fits.fz 4.11
images/90prime/BOK_CP/CP20160205/ksb_160206_022011_ooi_g_v1.fits.fz 3.72
images/90prime/BOK_CP/CP20160205/ksb_160206_021614_ooi_g_v1.fits.fz 4.12
images/90prime/BOK_CP/CP20170320/ksb_170321_054024_ooi_r_v1.fits.fz 1.59
images/90prime/BOK_CP/CP20170428/ksb_170429_070946_ooi_r_v1.fits.fz 4.87
images/90prime/BOK_CP/CP20170428/ksb_170429_065644_ooi_r_v1.fits.fz 4.17
images/90prime/BOK_CP/CP20170428/ksb_170429_084036_ooi_r_v1.fits.fz 4.16
images/90prime/BOK_CP/CP20170428/ksb_170429_071921_ooi_r_v1.fits.fz 4.55
images/90prime/BOK_CP/CP20170428/ksb_170429_032009_ooi_g_v1.fits.fz 4.29
images/90prime/BOK_CP/CP20170428/ksb_170429_104053_ooi_r_v1.fits.fz 4.19
images/90prime/BOK_CP/CP20170428/ksb_170429_060558_ooi_r_v1.fits.fz 4.33
images/90prime/BOK_CP/CP20170428/ksb_170429_083548_ooi_r_v1.fits.fz 6.94
images/90prime/BOK_CP/CP20170428/ksb_170429_052346_ooi_r_v1.fits.fz 5.08
images/90prime/BOK_CP/CP20170428/ksb_170429_070458_ooi_r_v1.fits.fz 4.59
images/90prime/BOK_CP/CP20170428/ksb_170429_084632_ooi_r_v1.fits.fz 4.96
images/90prime/BOK_CP/CP20170428/ksb_170429_050958_ooi_g_v1.fits.fz 4.11
images/90prime/BOK_CP/CP20160705/ksb_160706_045718_ooi_g_v1.fits.fz 3.63
images/90prime/BOK_CP/CP20170127/ksb_170128_124934_ooi_g_v1.fits.fz 4.12
images/90prime/BOK_CP/CP20170127/ksb_170128_130450_ooi_g_v1.fits.fz 4.28
images/90prime/BOK_CP/CP20170127/ksb_170128_123919_ooi_g_v1.fits.fz 4.36
images/90prime/BOK_CP/CP20170127/ksb_170128_123404_ooi_g_v1.fits.fz 3.84
images/90prime/BOK_CP/CP20170127/ksb_170128_124429_ooi_g_v1.fits.fz 4.1
images/90prime/BOK_CP/CP20170127/ksb_170128_122901_ooi_g_v1.fits.fz 3.93
images/90prime/BOK_CP/CP20170127/ksb_170128_125945_ooi_g_v1.fits.fz 4.73
images/90prime/BOK_CP/CP20160605/ksb_160606_084503_ooi_r_v1.fits.fz 5.52
images/90prime/BOK_CP/CP20160605/ksb_160606_085059_ooi_r_v1.fits.fz 4.55

[dstndstn]

FWIW, just after this step in the calibration we have a psfzpt_cuts for seeing > 3".

…

On Fri, Apr 12, 2019 at 10:14 AM Moustakas ***@***.***> wrote: Here are the 29 files with their CP "seeing" values (arcsec, presumably FWHM): images/90prime/BOK_CP/CP20180419/ksb_180420_110216_ooi_r_v1.fits.fz 4.38 images/90prime/BOK_CP/CP20180419/ksb_180420_094712_ooi_r_v1.fits.fz 3.62 images/90prime/BOK_CP/CP20180419/ksb_180420_093738_ooi_r_v1.fits.fz 4.99 images/90prime/BOK_CP/CP20180419/ksb_180420_110703_ooi_r_v1.fits.fz 4.11 images/90prime/BOK_CP/CP20160205/ksb_160206_022011_ooi_g_v1.fits.fz 3.72 images/90prime/BOK_CP/CP20160205/ksb_160206_021614_ooi_g_v1.fits.fz 4.12 images/90prime/BOK_CP/CP20170320/ksb_170321_054024_ooi_r_v1.fits.fz 1.59 images/90prime/BOK_CP/CP20170428/ksb_170429_070946_ooi_r_v1.fits.fz 4.87 images/90prime/BOK_CP/CP20170428/ksb_170429_065644_ooi_r_v1.fits.fz 4.17 images/90prime/BOK_CP/CP20170428/ksb_170429_084036_ooi_r_v1.fits.fz 4.16 images/90prime/BOK_CP/CP20170428/ksb_170429_071921_ooi_r_v1.fits.fz 4.55 images/90prime/BOK_CP/CP20170428/ksb_170429_032009_ooi_g_v1.fits.fz 4.29 images/90prime/BOK_CP/CP20170428/ksb_170429_104053_ooi_r_v1.fits.fz 4.19 images/90prime/BOK_CP/CP20170428/ksb_170429_060558_ooi_r_v1.fits.fz 4.33 images/90prime/BOK_CP/CP20170428/ksb_170429_083548_ooi_r_v1.fits.fz 6.94 images/90prime/BOK_CP/CP20170428/ksb_170429_052346_ooi_r_v1.fits.fz 5.08 images/90prime/BOK_CP/CP20170428/ksb_170429_070458_ooi_r_v1.fits.fz 4.59 images/90prime/BOK_CP/CP20170428/ksb_170429_084632_ooi_r_v1.fits.fz 4.96 images/90prime/BOK_CP/CP20170428/ksb_170429_050958_ooi_g_v1.fits.fz 4.11 images/90prime/BOK_CP/CP20160705/ksb_160706_045718_ooi_g_v1.fits.fz 3.63 images/90prime/BOK_CP/CP20170127/ksb_170128_124934_ooi_g_v1.fits.fz 4.12 images/90prime/BOK_CP/CP20170127/ksb_170128_130450_ooi_g_v1.fits.fz 4.28 images/90prime/BOK_CP/CP20170127/ksb_170128_123919_ooi_g_v1.fits.fz 4.36 images/90prime/BOK_CP/CP20170127/ksb_170128_123404_ooi_g_v1.fits.fz 3.84 images/90prime/BOK_CP/CP20170127/ksb_170128_124429_ooi_g_v1.fits.fz 4.1 images/90prime/BOK_CP/CP20170127/ksb_170128_122901_ooi_g_v1.fits.fz 3.93 images/90prime/BOK_CP/CP20170127/ksb_170128_125945_ooi_g_v1.fits.fz 4.73 images/90prime/BOK_CP/CP20160605/ksb_160606_084503_ooi_r_v1.fits.fz 5.52 images/90prime/BOK_CP/CP20160605/ksb_160606_085059_ooi_r_v1.fits.fz 4.55 — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#349 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABBD_Vo_cvG7aPzTW1t-J8xAwLyeWzJJks5vgJStgaJpZM4crC2w> .

[moustakas]

FWIW, just after this step in the calibration we have a psfzpt_cuts for seeing > 3".

True, but this bit wasn't getting set because the FWHM column in the CCDs table wasn't being populated because the PSF modeling failed. In #350 (sloppy, I know), I just pushed a change where I initialize with the CP value of the FWHM, so now these CCDs get

In [45]: tt['ccd_cuts']
Out[45]: 
<Column name='ccd_cuts' dtype='int32' length=4>
4176
4176
4176
4176

(i.e., ccdnmatch (it's zero!), zpt_small (also zero!), and seeing_bad).

However, when trying to build the annotated CCDs file we now get:

Failed to get_tractor_image
Traceback (most recent call last):
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacyzpts/annotate_ccds.py", line 162, in annotate_one_ccd
    tim = im.get_tractor_image(**kwargs)
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/image.py", line 545, in get_tractor_image
    tim.galnorm = self.galaxy_norm(tim)
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/image.py", line 678, in galaxy_norm
    galmod = gal.getModelPatch(tim, modelMask=mm).patch
  File "/usr/local/lib/python/tractor/pointsource.py", line 52, in getModelPatch
    assert(np.all(np.isfinite(upatch.patch)))
AssertionError

So, what do we want to do? Presumably we still want the -annotated.fits and -survey.fits files to get generated, even though the -photom.fits file does not. Or do we want to require a -photom.fits file for every exposure and not generate the other files at all?

[moustakas]

I just added these 29 exposures to the 90prime bad exposure list (in 8f75c39).

[moustakas]

Fixed in #350.