DM-32430: Create DECam DRP Processing Pipeline #208

leeskelvin · 2021-11-03T17:41:47Z

No description provided.

parejkoj

Every one of the config lines in these files needs a comment describing why it is different from the default, please!

parejkoj · 2021-11-22T17:42:37Z

config/jointcal.py

+# HSC needs a higher order polynomial to track the steepness of the optical
+# distortions along the edge of the field. Emperically, this provides a
+# measurably better fit than the default order=5.
+config.astrometryVisitOrder = 7


I think you shouldn't set this for DECam unless we actually test how much it's better.

parejkoj · 2021-11-22T17:43:16Z

config/jointcal.py

+# For the deep fields with hundreds of visits, outlier rejection may take many
+# weeks of runtime. By stopping outlier rejection when it ceases to have a
+# significant effect on the model, we can bring compute time down to a few days.
+config.astrometryOutlierRelativeTolerance = 0.002


Lets leave this alone until we run jointcal on enough visits where the runtime matters: jointcal on DECam should be significantly faster than on HSC (less stars/visit and less detectors).

parejkoj · 2021-11-22T17:44:26Z

config/filterMap.py

+        ('r', 'r'),
+        ('i', 'i'),
+        ('z', 'z'),
+        ('y', 'y'),


Remove all of the above filterMaps: redundant filtermaps like these are unnecessary and can result in schema errors.

leeskelvin · 2021-11-22T18:07:52Z

Every one of the config lines in these files needs a comment describing why it is different from the default, please!

Many are copied over from obs_subaru, with some adapted for DECam, for example, pixel scale. This seems like a good place to start, to begin testing this pipeline against results output in HSC processing.

taranu

Some minor requests for clarification.

taranu · 2021-12-01T01:21:58Z

config/calibrate.py

 config.photoCal.photoCatName = "ps1_pv3_3pi_20170110"
+colors = config.photoCal.match.referenceSelection.colorLimits
+# The following two color limits are adopted from obs_subaru for the HSC SSP survey


Are they still suitable for DECam?

These limits are taken from Song Huang's original PR: #206. Song is one of the group leading the Merian data reduction effort, so I adopt them as per his request.

taranu · 2021-12-01T01:22:25Z

config/calibrate.py

+config.photoCal.match.referenceSelection.magLimit.maximum = 22.0
+config.photoCal.colorterms.load(os.path.join(obsConfigDir, 'colorterms.py'))
+
+# Better astrometry matching


This comment could be more informative.

Edit: Comment changed to: Number of bright stars to use. Sets the max number of patterns that can be tested.

taranu · 2021-12-01T01:25:06Z

config/calibrate.py

+        matchConfig.matcher.maxMatchDistArcSec = 2.0
+        matchConfig.sourceSelector.active.excludePixelFlags = False
+
+# Set to match defaults curretnly used in HSC production runs (e.g. S15B)


currently; also, has this not changed since S15B? I guess not.

That's right, this config change is set to mimic current HSC production runs.

taranu · 2021-12-01T02:03:15Z

config/extinctionCoeffs.py

@@ -0,0 +1,15 @@
+# Extinction coefficients for DECam filters for conversion from E(B-V) to extinction, A_filter.


I'll note that these are all slightly lower than the HSC values in faro (ignore u band, I made that number up) - are we fine with that? Is it reflecting an actual physical difference in the filter set (no pun intended)?

Yes, these are different filters, so I'd expect these values to be somewhat different to HSC (albeit, close).

taranu · 2021-12-01T02:55:59Z

policy/Object.yaml

@@ -0,0 +1,712 @@
+funcs:


I gather these are copy-pasted from HSC; is there any way to keep them in sync?

Yes - ultimately, DM-28862 will be the single go-to for all policy files. Following that change, all surveys should be using the same policy files. However, in the short term, I've had to place these policy files inside obs_decam to allow for this ticket to function correctly.

mrawls

Most of these changes are great, and a big thank you for adding color terms! Per our Slack chat, I would really like to ensure each config override line (a) ideally differs from the task default and (b) has some documented motivation for existing, even if that is as simple as "obs_subaru does this so obs_decam probably ought to also."

mrawls · 2021-12-01T21:58:50Z

config/colorterms.py

+# Average color terms for the DECam filters are calculated using the Pickles stellar spectra atlas.
+# The following values are provided by Song Huang (Tsinghua University; dr.guangtou@gmail.com)
+# Color terms for u, Y, N419 and N964 bands are not currently available.


You're my hero for effectively doing DM-19284 for me 😅

PS - all thanks to Song Huang (formerly here at Princeton) who actually did the hard work on this in his original PR - I'm just implementing it for us on this ticket!

config/transformObjectCatalog.py

mrawls · 2021-12-06T22:15:38Z

config/measureCoaddSources.py

-config.match.refObjLoader.filterMap['u'] = 'g'
-config.match.refObjLoader.filterMap['Y'] = 'y'
-config.match.refObjLoader.filterMap['N419'] = 'g'
-config.match.refObjLoader.filterMap['N540'] = 'g'
-config.match.refObjLoader.filterMap['N708'] = 'i'
-config.match.refObjLoader.filterMap['N964'] = 'z'


Are you sure these aren't needed anymore? It is probably moot due to Gen2 going away, but the mapping is for non-ugrizy filters that need some "close enough" lookup refcat lookup filter.

I moved the filter definitions to the file config/filterMap.py, as in obs_subaru, which seemed to be the most sensible thing to do. Broad band filters were removed because they were causing schema errors (see this comment from John P above, and this Slack thread for my original report of this error).

config/apertures.py

config/assembleCoadd.py

pipelines/DRP.yaml

mrawls

Very nearly set! I'd prefer to see some of the coaddBase (makeWarp) and assembleCoadd configs in DRP.yaml itself instead of as obs_decam defaults, since we make less fancy templates in AP. Otherwise I think this is good to go.

mrawls · 2021-12-08T06:18:36Z

config/coaddBase.py

@@ -0,0 +1,10 @@
+# Configs shared between makeWarp and assembleCoadd


The settings in this config file assumes warps and coadds are being built for DRP reasons and not for AP template reasons; I suggest setting these in DRP.yaml instead of here (presumably in the makeWarp task).

All configs in the files coaddBase.py, makeWarp.py, compareWarpAssembleCoadd.py and assemblyCoadd.py have been moved to config sections in the main DRP.yaml pipeline file. This requires some amount of config duplication, as several of these configs are shared between quanta. I ran before/after diffs on the config outputs to confirm that the configs are identical.

mrawls · 2021-12-08T06:19:56Z

config/compareWarpAssembleCoadd.py

+config.assembleStaticSkyModel.doApplyExternalPhotoCalib = True
+config.assembleStaticSkyModel.externalPhotoCalibName = 'jointcal'
+config.assembleStaticSkyModel.doApplyExternalSkyWcs = True
+config.doFilterMorphological = True


As in coaddBase.py, these configs are DRP specific and not always desired for AP template building. I'd prefer to have them set in DRP.yaml instead of for all DECam things.

See above - this file no longer exists.

mrawls · 2021-12-08T06:24:21Z

config/transformObjectCatalog.py

+
+# Always produce these output bands in the Parquet coadd tables, no matter
+# what bands are in the input.
+# config.outputBands = ["g", "r", "i", "z", "y"]


This is commented out on purpose?

Good spot! Removed.

pipelines/DRP.yaml

mrawls · 2021-12-08T19:00:07Z

pipelines/README.md

@@ -1,8 +1,7 @@
-# This is not a good place for pipelines
+# Temporary DECam Pipelines


I suppose I'll allow it 😜

I forgot I made this edit! Seemed like a good idea... but sad to see the old message go!

leeskelvin force-pushed the tickets/DM-32430 branch from 89e314a to ef7e260 Compare November 3, 2021 17:45

leeskelvin force-pushed the tickets/DM-32430 branch 10 times, most recently from 3ffc2ff to b59a192 Compare November 22, 2021 17:26

parejkoj reviewed Nov 22, 2021

View reviewed changes

parejkoj requested changes Nov 22, 2021

View reviewed changes

leeskelvin force-pushed the tickets/DM-32430 branch 2 times, most recently from 7dea1b7 to bb34265 Compare November 22, 2021 18:02

leeskelvin force-pushed the tickets/DM-32430 branch 7 times, most recently from 1f97abd to 6ee6a46 Compare November 29, 2021 18:50

parejkoj approved these changes Nov 30, 2021

View reviewed changes

taranu reviewed Dec 1, 2021

View reviewed changes

leeskelvin force-pushed the tickets/DM-32430 branch from 6ee6a46 to 6020f26 Compare December 1, 2021 18:26

mrawls requested changes Dec 6, 2021

View reviewed changes

leeskelvin force-pushed the tickets/DM-32430 branch 2 times, most recently from 94597fb to 4f3fee0 Compare December 7, 2021 21:45

mrawls approved these changes Dec 8, 2021

View reviewed changes

leeskelvin force-pushed the tickets/DM-32430 branch from 4f3fee0 to 5cec38c Compare December 8, 2021 12:57

mrawls reviewed Dec 8, 2021

View reviewed changes

leeskelvin force-pushed the tickets/DM-32430 branch from 5cec38c to f746155 Compare December 9, 2021 03:28

Create DECam DRP Processing Pipeline

588ff63

leeskelvin force-pushed the tickets/DM-32430 branch from f746155 to 588ff63 Compare December 9, 2021 03:29

leeskelvin merged commit 46ac641 into main Dec 9, 2021

leeskelvin deleted the tickets/DM-32430 branch December 9, 2021 17:40

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

DM-32430: Create DECam DRP Processing Pipeline #208

DM-32430: Create DECam DRP Processing Pipeline #208

leeskelvin commented Nov 3, 2021

parejkoj left a comment

parejkoj Nov 22, 2021

parejkoj Nov 22, 2021

parejkoj Nov 22, 2021

leeskelvin commented Nov 22, 2021

taranu left a comment

taranu Dec 1, 2021

leeskelvin Dec 1, 2021

taranu Dec 1, 2021

leeskelvin Dec 1, 2021 •

edited

taranu Dec 1, 2021

leeskelvin Dec 1, 2021

taranu Dec 1, 2021

leeskelvin Dec 1, 2021

taranu Dec 1, 2021

leeskelvin Dec 1, 2021

mrawls left a comment

mrawls Dec 1, 2021

leeskelvin Dec 7, 2021

mrawls Dec 6, 2021

leeskelvin Dec 7, 2021

mrawls left a comment

mrawls Dec 8, 2021

leeskelvin Dec 8, 2021

mrawls Dec 8, 2021

leeskelvin Dec 8, 2021

mrawls Dec 8, 2021

leeskelvin Dec 8, 2021

mrawls Dec 8, 2021

leeskelvin Dec 8, 2021

		@@ -0,0 +1,15 @@
		# Extinction coefficients for DECam filters for conversion from E(B-V) to extinction, A_filter.

		@@ -0,0 +1,10 @@
		# Configs shared between makeWarp and assembleCoadd

		@@ -1,8 +1,7 @@
		# This is not a good place for pipelines
		# Temporary DECam Pipelines

DM-32430: Create DECam DRP Processing Pipeline #208

DM-32430: Create DECam DRP Processing Pipeline #208

Conversation

leeskelvin commented Nov 3, 2021

parejkoj left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

leeskelvin commented Nov 22, 2021

taranu left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

leeskelvin Dec 1, 2021 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mrawls left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mrawls left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

leeskelvin Dec 1, 2021 •

edited