DM-31580: Move consolidateDiaSourceTable to step6 #340
Conversation
pipelines/imsim/DRP.yaml
Outdated
| - drpAssociation | ||
| - drpDiaCalculation | ||
| description: > | ||
| Tasks that can be run together, but only after the 'step1', 'step2', | ||
| 'step3', and 'step4' subsets | ||
|
|
||
| These DIA aggregation steps include both a visit-level aggregation | ||
| step consolidateDiaSourceTable and tract-level aggregation steps | ||
| These DIA aggregation steps include tract-level aggregation steps | ||
| drpAssociation and drpDiaCalculation. |
There was a problem hiding this comment.
Now that the description isn't responsible for explaining which are tract-level and which are visit level, we don't have to call out all the tasks in this step. We should also include a recommendation on the data query: include tract. (It's less important here that people include tract in the data query because diaSources will only exist where a template was found that covers the detector, and "tract" was specified in step 3, but you'll still end up with a ragged edge and bad dia stats.
"This step includes tract-level aggregation Tasks. These should be run with explicit 'tract' constraints because otherwise quanta will be created for jobs with only partial visit coverage."
| - consolidateDiaSourceTable | ||
| description: > | ||
| Tasks that can be run together, but only after the 'step1', 'step2', | ||
| 'step3', and 'step4' subsets |
There was a problem hiding this comment.
Does consolidateDiaSourceTable depend on drpAssociation and drpDiaCalculation (step5)?
I thought these new step5 and step6 aren't blocking either other?
There was a problem hiding this comment.
You're right. They're not blocking each other at the moment.
I was thinking ahead to running forced phot on the diaObject centroids associated in step5.
pipelines/imsim/DRP.yaml
Outdated
| Tasks that can be run together, but only after the 'step1', 'step2', | ||
| 'step3', and 'step4' subsets | ||
|
|
||
| This DIA aggregation step includes a visit-level aggregation |
There was a problem hiding this comment.
Can also simplify now that all tasks here will be visit level:
"This step includes visit-level, aggregation tasks. Running without..."
This makes step5 a clear per-tract step and step6 a per-visit step.
hsinfang
force-pushed
the
tickets/DM-31580
branch
from
7980e9b
to
111446d
Compare
No description provided.