DM catalog validation #124
Conversation
|
An example of the plots that this test generates (using
(Since the galaxies in these catalogs are unrelated we can expect that we do not find any correlation between positions nor magnitudes). |
|
Success! Thanks @yymao:
|
|
I was comparing with PSF mag, now comparing with cModel mag :)
|
|
Full summary here: https://portal.nersc.gov/project/lsst/descqa/v2/?run=2018-07-27_15 |
|
@fjaviersanchez - do you understand why your red points in the cmodel mag plot are offset by Could there be a few outliers that are throwing the mean off, in which case the median (or a mean determined with outlier rejection) would be more stable? |
|
@rmandelb I think those are bad matches. I am only using nearest-neighbor matching (because it's the fastest) and I think that some sources are being wrongly associated to a fainter source creating pretty big outliers. I am including the median here:
I think that I'll include both mean and median (the more information the better I guess). |
yymao
added
validation: post-DM
and removed
developing
validation: extragalactic
labels
There was a problem hiding this comment.
@fjaviersanchez sorry for being late on reviewing this. I directly pushed a few changes to make the PR works with the CI test. And made a few comments for code review.
descqa/CheckAstroPhoto.py
Outdated
| try: | ||
| assert(len(self.ra.keys())==2) # Making sure that we have *just* two catalogs | ||
| except AssertionError: | ||
| print('The test can compare two catalogs only!') |
There was a problem hiding this comment.
raise an error if the test shouldn't work
if len(self.ra) != 2: # Making sure that we have *just* two catalogs
raise ValueError('The test can compare two catalogs only!')
descqa/CheckAstroPhoto.py
Outdated
| except AssertionError: | ||
| print('The test can compare two catalogs only!') | ||
|
|
||
| cat_names = list(self.ra.keys()) # This is an auxiliary list to easily get the catalogs |
There was a problem hiding this comment.
unnecessary .keys()
cat_names = list(self.ra)
descqa/CheckAstroPhoto.py
Outdated
| cat_names = list(self.ra.keys()) # This is an auxiliary list to easily get the catalogs | ||
| print(cat_names) | ||
| cat_len = [len(self.ra[cat_names[0]]),len(self.ra[cat_names[1]])] | ||
| cat_names = np.array(cat_names)[np.argsort(cat_len)].tolist() # We find the catalog with less objects to build the tree |
There was a problem hiding this comment.
Lines 104-105, simplify the logic
cat_names = sorted(cat_names, key=lambda name: len(self.ra[name]))
descqa/CheckAstroPhoto.py
Outdated
| -1, 1, self.nbins, '%s' % band, r'$\Delta %s$' % band, photo_savename, bin_stat=True) | ||
| n_true, _ = np.histogram(self.magnitude[(cat_names[1], band)], bins=50, range=(10, 30)) | ||
| n_meas, bin_edges = np.histogram(self.magnitude[(cat_names[0], band)], bins=50, range=(10, 30)) | ||
| plt.figure() |
There was a problem hiding this comment.
Store the figure object so that we can close it later.
fig = plt.figure()
descqa/CheckAstroPhoto.py
Outdated
| plt.ylim(0,1) | ||
| plt.tight_layout() | ||
| photo_savename = os.path.join(output_dir, 'mag_ratio_%s_%s_%s.png' % (cat_names[0], cat_names[1], band)) | ||
| plt.savefig(photo_savename) |
There was a problem hiding this comment.
Add after Line 139
plt.close(fig)so that matplotlib will not waste memory on open figures.
descqa/CheckAstroPhoto.py
Outdated
| n_true, _ = np.histogram(self.magnitude[(cat_names[1], band)], bins=50, range=(10, 30)) | ||
| n_meas, bin_edges = np.histogram(self.magnitude[(cat_names[0], band)], bins=50, range=(10, 30)) | ||
| plt.figure() | ||
| plt.plot(0.5*(bin_edges[1:]+bin_edges[:-1]), 1.0*n_meas/n_true,'o') |
There was a problem hiding this comment.
Instead of 1.0*n_meas/n_true, do n_meas.astype(float) / n_true to be explicit what the code does.
descqa/CheckAstroPhoto.py
Outdated
| rect_histx = [left, bottom_h, width, 0.2] | ||
| rect_histy = [left_h, bottom, 0.2, height] | ||
|
|
||
| plt.figure() |
There was a problem hiding this comment.
store the figure object so that we can close it later
fig = plt.figure()
| axHisty.autoscale(tight=True) | ||
| axHistx.set_xlim(axScatter.get_xlim()) | ||
| axHisty.set_ylim(axScatter.get_ylim()) | ||
| plt.savefig(savename) |
With this PR I am creating a test that matches two catalogs and plots the difference in magnitude and position. @yymao I tried to follow your advice and set up the tests in conclude_test. Right now, I am reading the magnitudes from mag_true_$BAND_lsst but I think it would be useful to use something like mag_$BAND as an alias for the magnitudes that we want to compare.
I am using the simple KDTree nearest neighbor matcher (spatial_closest) from https://github.com/LSSTDESC/CatalogMatcher for speed purposes but I am open to use any other matching technique.
For DM catalogs we also need to set up the selection in the config file (and this selection cuts should be transparent somehow to the truth catalogs). Right now, I have as a placeholder ra < 500.
So, as to-do list:
Generalize magnitude reading.
Generalize selection.