Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DM-31324: add step4 and step5 to DRP #336

Merged
merged 2 commits into from
Aug 5, 2021
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
44 changes: 43 additions & 1 deletion pipelines/imsim/DRP.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -49,6 +49,8 @@ subsets:
subset:
- consolidateSourceTable
- consolidateVisitSummary
- makeCcdVisitTable
- makeVisitTable
description: >
Per-visit tasks that can be run together, but only after the 'step1'.

Expand All @@ -68,12 +70,13 @@ subsets:
- deblend
- measure
- mergeMeasurements
- forcedPhotCcd
- forcedPhotCoadd
- transformObjectTable
- writeObjectTable
- consolidateObjectTable
- healSparsePropertyMaps
- selectGoodSeeingVisits
- templateGen
description: >
Tasks that can be run together, but only after the 'step1' and 'step2'
subsets.
Expand All @@ -90,3 +93,42 @@ subsets:

This subset is considered a workaround for missing middleware and task
functionality. It may be removed in the future.
step4:
subset:
- forcedPhotCcd
- forcedPhotDiffim
- imageDifference
- transformDiaSourceCat
description: >
Tasks that can be run together, but only after the 'step1', 'step2'
and 'step3' subsets

These detector-level tasks should not be run with
'tract' or 'patch' as part of the data ID expression if all
reference catalogs or diffIm templates that cover these
detector-level quanta are desired.
step5:
subset:
- consolidateDiaSourceTable
- drpAssociation
- drpDiaCalculation
description: >
Tasks that can be run together, but only after the 'step1', 'step2',
'step3', and 'step4' subsets

These DIA aggregation steps include both a visit-level aggregation
step consolidateDiaSourceTable and tract-level aggregation steps
drpAssociation and drpDiaCalculation.

While processing DC2 test-med-1, running without tract or patch in
the data query is recommended. If processing many contiguous tracts
running with tract in the dataId will be unavoidable.
If run with tract or patch in the data query, drpAssociation and
drpDiaCalculation will be complete, but the outputs of
consolidateDiaSourceTable will not contain complete visits.

This subset is separate from step4 to signal to operators to pause
to assess unexpected image differencing failures before these
aggregation steps. Otherwise, if run in the same quantum graph,
aggregated data products (e.g. diaObjects) would not be created if
one or more of the expected inputs is missing.