shear test #54

patricialarsen · 2018-01-04T22:10:04Z

Shear-Shear correlation function comparison between theory code using pyCAMB and simulation using TreeCorr. This does not yet output a test result or plot and there is some repeated code, but it computes both theory and simulation correlation functions within the DESCQA framework.

yymao · 2018-01-05T03:13:36Z

@patricialarsen thank you for submitting this PR! This is great progress and what needs to be done the next is to migrate the content in your notebook into the DESCQA framework.

Basically you'll need to copy the content in cells In [11] and In [15] to a python file under the descqa directory. And then create a corresponding yaml file under descqa/configs.

Once you are done, you can then try running your test (follow instruction here) on NERSC and see your test appear on the web interface!

Please don't hesitate to contact me if you encounter any issue and/or need more detail instruction.

Thanks again! This is very nice!

yymao

See the inline comments. I haven't done a full review but start with ones that breaks the test.

Please also remove the jupyter notebook.

yymao · 2018-01-05T16:32:30Z

descqa/shear_test.py

+
+
+import sys
+sys.path.insert(0, '/global/common/software/lsst/common/miniconda/py3-4.2.12/lib/python3.6/site-packages')


remove both lines with sys.path.insert (this will be taken care by the framework)

yymao · 2018-01-05T16:32:59Z

descqa/shear_test.py

+import yaml
+import numpy as np
+import healpy as hp
+from descqa.plotting import plt


remove this line as it's already imported above

yymao · 2018-01-05T16:33:12Z

descqa/shear_test.py

+import healpy as hp
+from descqa.plotting import plt
+
+from descqa import BaseValidationTest, TestResult


remove this line as it's already imported above

yymao · 2018-01-05T16:34:15Z

descqa/shear_test.py

+#from __future__ import unicode_literals, absolute_import, division
+import os
+import numpy as np
+from .base import BaseValidationTest, TestResult


reorder all imports --- (1) built-in modules, (2) installed modules, (3) this module (those start with from .)

yymao · 2018-01-05T16:35:33Z

descqa/shear_test.py

+            ax_this.plot(data, label=catalog_name)
+
+        #self.post_process_plot(ax)
+        #fig.savefig(os.path.join(output_dir, 'plot.png'))


uncomment these so it'll save the figure to be viewed on the web interface

yymao · 2018-01-05T16:44:38Z

sorry, I accidentally clicked approve...

yymao · 2018-01-05T16:47:07Z

descqa/shear_test.py

+
+
+from GCR import GCRQuery
+import GCRCatalogs


remove this line as this will be taken care by the framework

patricialarsen · 2018-01-05T16:52:58Z

Thanks @yymao for the notes - should I be testing this on the forked repository with:
./run_master.sh -v -c proto-dc2_v2.0_test -t shear
?

yymao · 2018-01-05T16:57:23Z

@patricialarsen yes. you can run the script in your cloned forked repo. You may want to use -c protoDC2 for the up-to-date version.

patricialarsen · 2018-01-05T17:52:58Z

@yymao everything is working now, the reason this failed the build is because in the run_master script it uses "/global/common/cori/contrib/lsst/apps/anaconda/py3-envs/DESCQA/bin/python" when the camb and treecorr modules are located in
"/global/common/software/lsst/common/miniconda/py3-4.2.12/bin/python"

I didn't want to make changes to the run_master script without first confirming with you, but when I swap the python environment it works and uploads the results to the website as expected.

yymao · 2018-01-05T17:59:01Z

@patricialarsen thanks! I'll review the code soon.

The python env in run_master.sh in the master branch has been already updated (See
https://github.com/LSSTDESC/descqa/blob/master/run_master.sh#L10)

…r_corr Necessary as updates have been made to master branch that update python directory in run_master.sh file

patricialarsen · 2018-01-05T22:47:28Z

Here's a preview of the test within the framework. It's for one bin around redshift 0.5-0.6 for protoDC2. Just the theory v simulations plots for now (x-axis is separation in arcminutes and y-axis is real-space correlation function), there's still no pass-fail criteria.

https://portal.nersc.gov/project/lsst/descqa/v2/?run=2018-01-05_48&test=shear&catalog=protoDC2

rmandelb · 2018-01-07T21:25:14Z

Hi @patricialarsen - I looked at the test preview and have a few questions and comments:

Are these the standard xi+/xi- shear correlation functions, or something else? They are labeled w+ and w- but I think they must be the projected xi+/xi-, e.g. as in equation 32 of http://adsabs.harvard.edu/abs/2015RPPh...78h6901K ?
What is your vision for how this test will be incorporated eventually? Right now you've just shows a narrow redshift slice, I gather (maybe for speed when debugging?). Will you eventually use a few broader bins spanning a range of redshifts?
I think it might be useful to put vertical lines indicating the min/max scales for which shear-shear measurements are typically carried out (e.g., from KiDS or DES). We would probably not care too much about discrepancies that are well outside that range of scales.
The vertical axis/tick labels are too wide so they need some tweaking (esp once you add the axis labels).

Thanks for your work on this!

patricialarsen · 2018-01-08T16:45:33Z

Hi @rmandelb,

Thanks for your feedback. Yes, these should be xi_plus and xi_minus. Sorry for the labelling confusion. And thanks for your suggestions on the plot - I'll incorporate those in the next version.

For the extent of this test, it really depends how much testing we want on the catalogues at this verification stage. I can see it either being an initial validation test to make sure the catalogue is roughly okay (i.e. shears don't have any major bugs), or a test to see if the required levels of accuracy of the shear correlation functions are met. The current narrow redshift bin is just for testing and as the limits are input parameters in the .yaml file it's straightforward to broaden these or to use multiple bins.

After I've determined some pass/fail criterion, tidied up the output and fixed the current version conflicts, it would be easy enough to make this incorporate:

cross-correlations between redshift bins
a check for shear sign convention mismatch
consistency with convergence autocorrelations
Longer term we could also update the theory to the WLPipe theory code, once we have the necessary modules in the DESCQA environment. I'm very open to suggestions about what would be useful to incorporate (or what wouldn't be useful) if you have any thoughts on this.

evevkovacs · 2018-01-08T22:59:16Z

@patricialarsen Thanks very much. I have one small cosmetic comment. It is difficult to distinguish the blue and green colors on the w+/xi+ plots. Perhaps a darker value or a different choice for one of the colors would help.

yymao · 2018-02-14T20:44:49Z

@patricialarsen Just checking in --- what's the current status of this test? Is this ready to be reviewed?

…into shear_corr Update master

yymao

Large chunks of commented-out code should be removed.

Line 20 to 29 --- these settings have cosmology dependence, shouldn't they be set by the cosmology of the catalog?

patricialarsen · 2018-02-22T20:41:49Z

Yes, the cosmology ideally should be set to that of the catalog. I can update that assuming there's some way to query each catalog's cosmology? In the meantime since it's currently a rough check I think it's still useful to compare to some realistic cosmology.

I'll remove commented out code - but will wait until you're done with changes so we don't conflict.

yymao · 2018-02-23T17:20:16Z

I think this PR is ready to be merged. Before merging I'll run the latest version on NERSC again to make sure there's no issue.

patricialarsen · 2018-02-23T20:46:51Z

Example output of current version:

https://portal.nersc.gov/project/lsst/descqa/v2/?run=2018-02-23_3
(note that default has no jack-knifing)

…into shear_corr It says I need this

Patricia Larsen and others added 2 commits January 4, 2018 13:28

first commit of shear test

d0f25c1

Commented out unnecessary code

90fcdc4

yymao added developing validation: extragalactic labels Jan 5, 2018

Patricia Larsen added 2 commits January 5, 2018 07:56

added config file and python file

fd18501

fixed Jupyter to python copying issues

f7260dd

yymao self-requested a review January 5, 2018 16:23

yymao removed the developing label Jan 5, 2018

further minor fixes

6599e36

yymao requested changes Jan 5, 2018

View reviewed changes

adding path - still tracking down build issue

ee8974a

yymao approved these changes Jan 5, 2018

View reviewed changes

yymao requested changes Jan 5, 2018

View reviewed changes

descqa/shear_test.py Outdated

from GCR import GCRQuery

import GCRCatalogs

Copy link

Member

yymao Jan 5, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

remove this line as this will be taken care by the framework

minor fixes

b239e0f

Patricia Larsen added 2 commits January 5, 2018 10:06

Merge branch 'master' of https://github.com/LSSTDESC/descqa into shea…

1adc9de

…r_corr Necessary as updates have been made to master branch that update python directory in run_master.sh file

commented out errant GCR load

f032ddb

added plots, fixed bug in theory separation values

029be49

patricialarsen mentioned this pull request Jan 8, 2018

shear-shear correlation #35

Closed

4 tasks

Patricia Larsen added 2 commits January 10, 2018 11:49

added jack-knife errors, tidied up plots

0c191eb

added (diagonal) chi2 based on jack-knife errors, and fail criteria

03f8e3f

EiffL mentioned this pull request Feb 1, 2018

2pt correlation (e.g. ellipticity-direction) for IA model testing #42

Open

4 tasks

yymao mentioned this pull request Feb 13, 2018

Suite of validation tests on DC2 extragalactic catalogs #50

Open

yymao and others added 5 commits February 19, 2018 20:07

Merge branch 'master' into shear_corr

1e784cc

recent updates

deb9ccf

Merge branch 'shear_corr' of https://github.com/patricialarsen/descqa …

bd12a82

…into shear_corr Update master

minor fixes

f621acc

travis build fixes

50b9b36

yymao changed the title ~~first commit of shear test~~ shear test Feb 22, 2018

yymao added 2 commits February 22, 2018 14:04

add camb

44b4826

fix imports; fix pylint; reformat

25af573

yymao requested changes Feb 22, 2018

View reviewed changes

yymao added 5 commits February 22, 2018 14:31

install gfortran for camb in travis

a8e7ab0

need gfortran 4.9 in travis

dd07485

i don't know why gfortran is not found...

e704639

another try to get gfortran 4.9

cf96226

alias is not enough...

a4d0c1a

yymao and others added 2 commits February 22, 2018 15:42

also needs scikit-learn

16b4550

updated cosmology

f30b3a5

yymao removed the developing label Feb 22, 2018

Patricia Larsen and others added 2 commits February 23, 2018 08:28

syntax and removal of old cosmology

98c42f4

Merge branch 'master' into shear_corr

e0321d9

yymao approved these changes Feb 23, 2018

View reviewed changes

Patricia Larsen and others added 3 commits February 23, 2018 13:16

note on scaling factor

eaf8d0c

Merge branch 'shear_corr' of https://github.com/patricialarsen/descqa …

99eedaf

…into shear_corr It says I need this

Merge branch 'master' into shear_corr

180d76b

yymao merged commit 0a8ba73 into LSSTDESC:master Feb 24, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

shear test #54

shear test #54

patricialarsen commented Jan 4, 2018

yymao commented Jan 5, 2018

yymao left a comment •

edited

yymao Jan 5, 2018

yymao Jan 5, 2018

yymao Jan 5, 2018

yymao Jan 5, 2018

yymao Jan 5, 2018

yymao commented Jan 5, 2018

yymao Jan 5, 2018

patricialarsen commented Jan 5, 2018

yymao commented Jan 5, 2018

patricialarsen commented Jan 5, 2018

yymao commented Jan 5, 2018

patricialarsen commented Jan 5, 2018

rmandelb commented Jan 7, 2018 •

edited

patricialarsen commented Jan 8, 2018

evevkovacs commented Jan 8, 2018

yymao commented Feb 14, 2018 •

edited

yymao left a comment

patricialarsen commented Feb 22, 2018

yymao commented Feb 23, 2018

patricialarsen commented Feb 23, 2018



		import sys
		sys.path.insert(0, '/global/common/software/lsst/common/miniconda/py3-4.2.12/lib/python3.6/site-packages')

shear test #54

shear test #54

Conversation

patricialarsen commented Jan 4, 2018

yymao commented Jan 5, 2018

yymao left a comment • edited

Choose a reason for hiding this comment

yymao Jan 5, 2018

Choose a reason for hiding this comment

yymao Jan 5, 2018

Choose a reason for hiding this comment

yymao Jan 5, 2018

Choose a reason for hiding this comment

yymao Jan 5, 2018

Choose a reason for hiding this comment

yymao Jan 5, 2018

Choose a reason for hiding this comment

yymao commented Jan 5, 2018

yymao Jan 5, 2018

Choose a reason for hiding this comment

patricialarsen commented Jan 5, 2018

yymao commented Jan 5, 2018

patricialarsen commented Jan 5, 2018

yymao commented Jan 5, 2018

patricialarsen commented Jan 5, 2018

rmandelb commented Jan 7, 2018 • edited

patricialarsen commented Jan 8, 2018

evevkovacs commented Jan 8, 2018

yymao commented Feb 14, 2018 • edited

yymao left a comment

Choose a reason for hiding this comment

patricialarsen commented Feb 22, 2018

yymao commented Feb 23, 2018

patricialarsen commented Feb 23, 2018

yymao left a comment •

edited

rmandelb commented Jan 7, 2018 •

edited

yymao commented Feb 14, 2018 •

edited