color-color diagram

[yymao]

As discussed in #40, we open this issue to discuss implementing a color-color diagram test in DESCQA. According to @janewman-pitt-edu this would not be a required test but it would be nice to have for visual inspection. (cc'ing @morriscb @sschmidt23)

@nsevilla, it seems to me that you have implemented this test to some extent. Will you be able to port it into DESCQA? Let me know if you need any help.

• code to reduce mock data
• code that works within DESCQA framework

[nsevilla]

Sounds good to get me started @yymao . It runs under the descqa already in my version at NERSC as I said in the hack day. Will tidy it up and make a PR.

Concerning a possible test (I know it's not required any more), a possibility would be binning the SDSS 2D data, creating some jackknife errors in each bin, and check statistical compatibility... I think it is going to be far from a good chi2 in general, but nonetheless, maybe to check quantitatively the impact of any changes in the catalogs (this process can be a bit computationally expensive though but could be optional).

But, if only plots are required, wouldn't this be better part of the general suite of tests done by readiness_test.py for instance?

[yymao]

@nsevilla great, looking forward to your PR.

I think it would be better that this test is a standalone test in DESCQA. For one, as you mentioned, there are potential validation data sets and criteria that we can implement, if at a later time we deem that it is important that the catalogs have certain reasonable color-color distribution. In addition, the DESCQA framework allows running several tests at once, so there's no need to put many tests into one.

[janewman-pitt-edu]

@nsevilla : I would say that you're guaranteed a horrible chi-squared. The errors in the distribution should be dominated by counting statistics and N is very large.

Your proposed statistic would also be very sensitive to small shifts in the overall colors of objects in data vs. simulations (as that will shift objects from bin-to-bin) as well as the choice of binning. I would suggest that we would want to do something that is unbinned and that applies shifts to align distributions before comparison. In 1D, that is all well-defined, but that is not true in 2D (e.g. there is not a true 2D analog of the K-S test).

[yymao]

closed by #88