DM-20382: Add description of RC2 ap_pipe repos #12
Conversation
yalsayyad
changed the title
|
Includes all and "best seeing" visit lists for tract 9813. |
false_positives/ap_pipe_HSC_RC2.md
Outdated
| @@ -0,0 +1,84 @@ | |||
| # Running ap_pipe on HSC RC2 data from PDR1 | |||
|
|
|||
| When PDR2 hits lsst-dev we'll recreate an AP-style repo using PDR1 templates to difference new visits in PDR2. Until then, this repo will demonstrate running ap_pipe on a dataset more like DRP mode, where the templates are the best seeing images within a data release and all visits in that data release are differenced. | |||
There was a problem hiding this comment.
I don't know what a PDR is, and other readers might not, either.
false_positives/ap_pipe_HSC_RC2.md
Outdated
| * `/datasets/hsc/repo/rerun/private/yusra/RC2/ap_pipe_PDR1_bestcase` (HSC-I band) | ||
| * `/datasets/hsc/repo/rerun/private/yusra/RC2/ap_pipe_PDR1_bestcase_noSkyCorr` (HSC-G, HSC-R, HSC-I, HSC-Z) | ||
|
|
||
| These instructions go into detail on the second was generated. |
false_positives/ap_pipe_HSC_RC2.md
Outdated
|
|
||
| These instructions go into detail on the second was generated. | ||
|
|
||
| To remember the name, the dataset only contails "PDR1", is "bestcase" in that the templates are chosen as the best seeing images, the best case scenario. The only config difference was that the HSC-specific skyCorrection was turned off. |
There was a problem hiding this comment.
"contains", plus the grammar makes it hard to follow. Consider rewording.
false_positives/ap_pipe_HSC_RC2.md
Outdated
| export COSMOS_Z_SEEING=17928^17908^17906^17926^17904^17902^17930^17962^17946^17944 | ||
| export COSMOS_Y_SEEING=22602^22604^22644^22632^22628^22642^22626^22630^22664^11730^22662^22606^11736^22646^22660^11718^11732 | ||
| ``` | ||
| LSST has no plans for a narrowband filter, so let's put that data aside. |
There was a problem hiding this comment.
I'm confused. What narrowband filter? Why bring it up in the first place?
false_positives/ap_pipe_HSC_RC2.md
Outdated
|
|
||
|
|
||
|
|
||
| Choose the 1/3 of visits with the best seeing. These visit lists are sorted by ascending visit FHWM, so reducing the number of visits can be accomplished by removing visits from the end: |
There was a problem hiding this comment.
At first I was confused because I assumed this would be done with a BestSeeingSelector. Maybe say "Manually choose" to make it more explicit?
false_positives/ap_pipe_HSC_RC2.md
Outdated
|
|
||
| ``` | ||
| mkdir /datasets/hsc/repo/rerun/private/yusra/RC2/ap_pipe_PDR1_bestcase_noSkyCorr | ||
| make_ppdb.py --config ppdb.isolation_level=READ_UNCOMMITTED ppdb.db_url="sqlite:////datasets/hsc/repo/rerun/private/yusra/RC2/ap_pipe_PDR1_bestcase_noSkyCorr/association.db |
There was a problem hiding this comment.
I suggest using the same syntax as for the ap_pipe.py call:
| make_ppdb.py --config ppdb.isolation_level=READ_UNCOMMITTED ppdb.db_url="sqlite:////datasets/hsc/repo/rerun/private/yusra/RC2/ap_pipe_PDR1_bestcase_noSkyCorr/association.db | |
| make_ppdb.py --configfile ./configfile.py |
Writing the calls in the same way helps ensure the database setup matches what ap_pipe.py expects. The config is safe; make_ppdb.py will simply ignore any non-PPDB fields.
There was a problem hiding this comment.
(Though, looking closer, I do find it a bit odd that the output repo is specified both in the config file and on the ap_pipe.py command line...)
false_positives/ap_pipe_HSC_RC2.md
Outdated
| config.ppdb.db_url="sqlite:////datasets/hsc/repo/rerun/private/yusra/RC2/ap_pipe_PDR1_bestcase_noSkyCorr/association.db" | ||
| config.ppdb.isolation_level="READ_UNCOMMITTED" | ||
| config.ppdb.connection_timeout=240 | ||
| config.differencer.doWriteMatchedExp=True |
There was a problem hiding this comment.
I'm a bit confused by this config. I think you told me at one point not to use doSelectSources with coadd templates, but you seem to be doing so here. What's the difference?
There was a problem hiding this comment.
You're right that the filterMap configs are a no-op if doSelectSources is False. I'll clarify.
There was a problem hiding this comment.
Now I'm more confused. Is doSelectSources False from some other config?
There was a problem hiding this comment.
It's False by default in ap_pipe: https://github.com/lsst/ap_pipe/blob/master/python/lsst/ap/pipe/ap_pipe.py#L89
false_positives/ap_pipe_HSC_RC2.md
Outdated
| config.differencer.refObjLoader.retarget(LoadIndexedReferenceObjectsTask) | ||
| config.differencer.refObjLoader.load(os.path.join(getPackageDir("obs_subaru"), "config", "filterMap.py")) | ||
|
|
||
| config.differencer.refObjLoader.ref_dataset_name = "ps1_pv3_3pi_20170110" |
There was a problem hiding this comment.
After DM-18394 I don't think you need differencer.refObjLoader anymore, because kernelSourcesFromRef is False.
yalsayyad
force-pushed
the
tickets/DM-20382
branch
from
4e264ae
to
dccd4fe
Compare
|
Didn't force push after rebasing master, so GitHub is not automatically registering the merge. |
No description provided.