DM-24638: Add Source Table generation tasks to DRP pipeline #341
Conversation
pipelines/DRP.yaml
Outdated
| - writeSourceTable | ||
| - transformSourceTable | ||
| - consolidateSourceTable | ||
| description: Set of tasks to generate parquet Source Tables from output of processCcd subset. |
There was a problem hiding this comment.
So are these tasks eventually going to be used in all the incarnations of the "main" DRP, and not just be an HSC thing? If so I would suggest that instead of adding theses here and then handling moving them around later, you add these to the DRP.yaml in pipe_tasks, and if this is not working for decam, you edit the DRP.yaml in obs_decam to exclude these:
description: DRP specialized for DECAM
instrument: lsst.obs.decam.DarkEnergyCamera
imports:
- location: $PIPE_TASK_DIR/pipelines/DRP.yaml
exclude:
- sourceTable
This will make sure those tasks are not included for DECAM, and any subset they appear in is excluded. You can make sure of this with pipetask build -p $OBS_DECAM_DIR/pipelines/DRP.yaml --show pipeline
I notice there is not already a pipeline in obs_decam, and this would be a good time to add it. Its not best practice to run against the one in pipe_tasks any way (for centralization reasons).
There was a problem hiding this comment.
Yeah, I opened DM-28862 for that.
We could either add specialized pipelines obs_decam, obs_cfht (and any other obs package we can find) today. and delete them on DM-28862, but that seems like more work than moving sourceTable from obs_subaru to pipe_tasks on DM-28862.
There was a problem hiding this comment.
Oh.... but I see that you think it's bad to run the pipe_tasks pipeline and the obs_packages DRP.yamls will be there to stay. pair coding time.
yalsayyad
force-pushed
the
tickets/DM-24638
branch
3 times, most recently
from
11570d7
to
4ad81e8
Compare
in preparation for running on other cameras:
- Move gen3 specific configs to pipe_tasks
- Make override files in config/ for tasks that changed
DefaultName
- Remove configs redundant with those in config/ now that
they are picked up by pipetask run
yalsayyad
force-pushed
the
tickets/DM-24638
branch
from
4ad81e8
to
60a6269
Compare
No description provided.