New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

MJD_MIN,MAX NaNs In DR4: centroids outside the image edges #154

Closed
kaylanb opened this Issue May 25, 2017 · 2 comments

Comments

Projects
None yet
2 participants
@kaylanb
Contributor

kaylanb commented May 25, 2017

There are 3,024 DR4 bricks (out of 65,543) that have MJD_{MIN,MAX} = NaN. I inspected four of these bricks
0917p652 - 16 occurrences
1787p607 - 1 occurrences
2221p460 - 5 occurrences
2754p645 - 2 occurrences
chosen randomly for the 3,024.

Observations

  • MJD_MIN = NaN occurs only where MJD_MAX = NaN, and vice-versa

  • In nearly all cases (23/24), sources with MJD_{MIN,MAX} = NaN have a centroid position that is at least 1 pixel from the edge of the image (e.g. if the image is zero'd out over x = 1,100 y=1,100, then the centroid might be at x=99,y=99 or x=90,y=95). I double checked that these x,y positions match the corresponding ra,dec in the catalogues, and these off-image locations really are correct

  • A particularly strange case is brick 2221p460, where all MJD_{MIN,MAX} NaNs have a centroid with x pixel location < 0 (values of [-20, -200]). e.g. the catalogue says x = -20 and the image doesn't exist there... Here's the ipython print

In [31]: a.bx[i],a.by[i]
Out[31]:
(array([ -27.98950577, -21.94595909, -48.53701019, -112.6945343 ,
-221.56390381], dtype=float32),
array([ 3354.54638672, 2990.68579102, 3307.671875 , 3334.58154297,
3677.11547852], dtype=float32))

Conclusions (extrapolating from these bricks)

  • The majority of sources with MJD_{MIN,MAX} = NaN are not astrophysical, are not even on the image, and should be ignored. This should be the case for all MJD_{MIN,MAX} = NaN where either the image is 0 at the source centroid OR the centroid has a negative x,y pixel value.

  • Since these sources should be ignored, how do we convey this in the tractor catalogues? Do these sources show up without glitch in the neighboring/overlapping brick? Is there a bug in how overlapping brick regions are treated?

Examples (look for the square cursor in the upper right cutout to see the exact location)
0917p652
image
0917p652_image
invvar
0917p652_invvar
model
0917p652_model
another example in the same image as above
0917p652

2754p645
2754p645

Only Example of source being real and on the image
1787p607
1787p607

@dstndstn

This comment has been minimized.

Show comment
Hide comment
@dstndstn

dstndstn May 25, 2017

Member

Fun. In your first example, these are and should be BRICK_PRIMARY; in brick space there's nothing special about them. But in CCD space they're at the edge. I was hoping to see that the NOBS_{GRZ} fields would help, but indeed, for tractor-0917p652.fits, of the 16 sources with NaNs for the MJDs, three of them have NOBS_Z = 1 (NOBS_{G,R} are all zero), and one has NOBS_Z = 1 AND BRICK_PRIMARY. That pixel is, however, masked with ALLMASK_Z = ANYMASK_Z = 3 = BADPIX | SATUR.

It has FRACMASKED_Z = 0.998 -- most of the pixels in the profile are masked, but FRACIN_Z=0.433487 -- a good fraction of the pixels in the profile are actually within the image.

It looks like this requires some careful work on the consistency between the NOBS and the MJDMIN/MAX values.

Looking at the code, we get NOBS from looking up the nearest pixel in the map resampled to brick space, while for the MJD_MIN/MAX we compute the pixel position within each overlapping image. I bet this is a situation where it's on the very edge of the image, so these pixel-rounding effects are relevant.

Nobs is compute here:
https://github.com/legacysurvey/legacypipe/blob/master/py/legacypipe/coadds.py#L185

And MJD_MIN/MAX here:
https://github.com/legacysurvey/legacypipe/blob/master/py/legacypipe/runbrick.py#L1366

So one option would be to make it so that NOBS and MJD_MIN/MAX are computed consistently; we could drop any sources with sum(NOBS_*) = 0, and thereby drop any with MJD_MIN/MAX. This seems like a reasonable solution.

Member

dstndstn commented May 25, 2017

Fun. In your first example, these are and should be BRICK_PRIMARY; in brick space there's nothing special about them. But in CCD space they're at the edge. I was hoping to see that the NOBS_{GRZ} fields would help, but indeed, for tractor-0917p652.fits, of the 16 sources with NaNs for the MJDs, three of them have NOBS_Z = 1 (NOBS_{G,R} are all zero), and one has NOBS_Z = 1 AND BRICK_PRIMARY. That pixel is, however, masked with ALLMASK_Z = ANYMASK_Z = 3 = BADPIX | SATUR.

It has FRACMASKED_Z = 0.998 -- most of the pixels in the profile are masked, but FRACIN_Z=0.433487 -- a good fraction of the pixels in the profile are actually within the image.

It looks like this requires some careful work on the consistency between the NOBS and the MJDMIN/MAX values.

Looking at the code, we get NOBS from looking up the nearest pixel in the map resampled to brick space, while for the MJD_MIN/MAX we compute the pixel position within each overlapping image. I bet this is a situation where it's on the very edge of the image, so these pixel-rounding effects are relevant.

Nobs is compute here:
https://github.com/legacysurvey/legacypipe/blob/master/py/legacypipe/coadds.py#L185

And MJD_MIN/MAX here:
https://github.com/legacysurvey/legacypipe/blob/master/py/legacypipe/runbrick.py#L1366

So one option would be to make it so that NOBS and MJD_MIN/MAX are computed consistently; we could drop any sources with sum(NOBS_*) = 0, and thereby drop any with MJD_MIN/MAX. This seems like a reasonable solution.

@dstndstn

This comment has been minimized.

Show comment
Hide comment
@dstndstn

dstndstn Jul 25, 2017

Member

done in 541b6e1

Member

dstndstn commented Jul 25, 2017

done in 541b6e1

@dstndstn dstndstn closed this Jul 25, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment