color distribution test

[yymao]

• code to reduce mock data
• code that works within DESCQA framework
• validation data
• validation criteria

[yymao]

@rongpu made a test in DESCQA v1 but we need an updated version for v2 (probably means lots of tidying up and refactoring code). So far we haven't made any progress.

[rmandelb]

@yymao - might this test be superceded by #40 and #41 ?

[yymao]

It might be. I put a "not required" label on this now, but @janewman-pitt-edu can confirm.

Also, this test is inherited from DESCQA v1 and I think @rongpu is working on it. We can check with @rongpu and see if he's interested in helping out on #40 and #41.

[janewman-pitt-edu]

I would say definitely not superseded by #41 (which only tests a narrow range of SEDs); and probably not superseded by #40 (because it should be equivalent in conclusions, but is simpler to implement).

[yymao]

Per our discussion in #40, I relabeled this test to be "required".

@rongpu are you still working on this? Can you give us a status update?

[rongpu]

Sorry for the late response. I have just finished a first implementation of the color distribution (very similar to the color test in DESCQA1), and I have created a pull request. The results can be found here: https://portal.nersc.gov/project/lsst/descqa/v2/?run=2018-03-04_11

Main changes from DESCQA1 color test:

• Since we are now dealing with light-cone catalogs, K correction is no longer applied. All comparisons are done in the observed magnitudes, with apparent magnitude and redshift cuts.
• Since the filters of mock catalogs and observed catalogs do not always match, color transformations are applied to the mock catalogs to match the filters of the observed catalogs. This is far from ideal, but that's the best I can do with the available datasets.

[yymao]

Implemented in #89. Still awaits validation criteria.

[evevkovacs]

@rongpu Can we rerun this test on the new protoDC2 v3 catalog ASAP? Is the test available in the master branch yet?

[yymao]

@evevkovacs it's on master already. See a test run here:
https://portal.nersc.gov/project/lsst/descqa/v2/?run=2018-03-26_3

[rongpu]

@evevkovacs For a comparison with the protoDC2 v2 catalog, here are the v2 results: https://portal.nersc.gov/project/lsst/descqa/v2/?run=2018-03-04_11
Apparently the v3 catalog has produced many more very blue galaxies in the DEEP2 magnitude range.

[yymao]

@evevkovacs here's a run that has v2, v3, and buzzard: https://portal.nersc.gov/project/lsst/descqa/v2/?run=2018-03-27

[evevkovacs]

Thanks that's useful. We have solved the discreteness problems in the color distributions. The agreement with SDSS is much better in v3. We will look into improving the agreement with DEEP2.

[evevkovacs]

@rongpu Can you give some details about the data sets used in the tests? We would like to know what catalogs you started with and what selections you made to get the final samples that are used for the DESCQA comparisons. If you have some notes about what you did, that would be great! Thanks, Eve

[rongpu]

@evevkovacs There are two validation catalogs: SDSS and DEEP2+CFHTLS.

I obtained the SDSS catalog using this SQL query on DR13, and then I selected the "main galaxy sample" and applied extinction correction with this script.

The DEEP2+CFHTLS catalog test includes redshifts from DEEP2 Field 1 and ugriz photometry from cross-matched CFHTLS Deep field (The catalog also contains Subaru Y-band photometry, although in the current version it's not used for color test). Only objects with DEEP2 ZQUALITY>=3 (secure redshifts) are used. DEEP2 selection weights are used in the color distribution calculation -- objects are weighted by 1/(DEEP2 selection weight).

The current DEEP2 test uses the "MAG_APERCOR" columns in the catalog, which is a matched aperture magnitude that I created from the original CFHTLS data, and it should have smaller error than the default MAG_AUTO photometry.

I just realized that I forgot to remove the much shallower CFHTLS-Wide objects from the dataset (~30% of all objects), so I will fix that and I expect a small change in the color distribution.

[rongpu]

Here is the latest run of color test with the bug fixed (now only using CFHTLS-Deep photometry): https://portal.nersc.gov/project/lsst/descqa/v2/?run=2018-03-27_1&test=Color_Dist_DEEP2
There is very small change in the color distribution.

I should note that both SDSS and DEEP2 color tests apply redshift cuts in addition to the r-band apparent magnitude cut. The redshift cuts applied in the runs are 0.05 < z < 0.1 for the SDSS test and 0.85 < z < 0.95 for the DEEP2 test. These cuts can be changed in the yaml files under the configs directory.

And I should also mention that more information of the DEEP2+3 dataset can be found here: https://confluence.slac.stanford.edu/display/LSSTDESC/Testbed+data

[yymao]

This test has been implemented in #89 and so far people are happy with visually inspecting the test results. If a formal validation criterion is needed, we should open a separate issue.