DM-35551: restructure and clean up pipelines #30
Conversation
TallJimbo
force-pushed
the
tickets/DM-35551
branch
from
cfd0da6
to
613a1e2
Compare
There was a problem hiding this comment.
Some minor comments on the fgcmcal defaults. I'm trusting that the changes on pipe_tasks have allowed you to look at the output pipelines/configs and confirm that this reorg leaves things the same (where they are supposed to be the same, fixing preSource, etc.)
pipelines/HSC/DRP-Prod.yaml
Outdated
| fgcmFitCycle: | ||
| class: lsst.fgcmcal.fgcmFitCycle.FgcmFitCycleTask | ||
| config: | ||
| doMultipleCycles: true |
There was a problem hiding this comment.
I think doMultipleCycles should be in ingrediences/HSC/DRP.yaml.
There was a problem hiding this comment.
For prod we should set the number of cycles multipleCyclesFinalCycleNumber: 6 probably for a production run.
There was a problem hiding this comment.
Will do, especially since 5 was just what you wanted for the test run, and that's now done. I'm assuming that change goes hand-in-hand with fgcmOutputProducts.cycleNumber=6.
| class: lsst.fgcmcal.fgcmFitCycle.FgcmFitCycleTask | ||
| config: | ||
| doMultipleCycles: true | ||
| useExposureReferenceOffset: true |
There was a problem hiding this comment.
We can commit this for now, but I want to see the test output from the HSC processing before we fully sign off.
This addresses a lot of "backwards" imports, in which the very-much concrete and complete HSC/DRP-RC2 pipeline was imported by other pipelines, by moving its contents to new pipelines in ingredients. This also reduces duplication between HSC and DECam. While it's mostly a refactor, it does change the expanded pipelines in a few ways: - HSC/DRP-Prod now has the correct (full-SSP) configuration for the FGCM tasks, mostly by avoiding RC2-specific overrides. - Two ill-defined subsets (singleFrame and multiVisit) have been removed from all pipelines. - Superfluous "step" subsets have been removed (including all steps in the ci_hsc pipelines). - faro tasks have been removed from HSC/DRP-Prod and ci_hsc.
TallJimbo
force-pushed
the
tickets/DM-35551
branch
from
613a1e2
to
8b06b48
Compare
No description provided.