Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

DM-33453: Update pipeline subdirectories in response to RFC-775. #150

Merged
merged 1 commit into from
Jan 30, 2024
Merged

DM-33453: Update pipeline subdirectories in response to RFC-775. #150

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
6 changes: 3 additions & 3 deletions pipelines/DarkEnergyCamera/ApPipe.yaml → pipelines/DECam/ApPipe.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,15 +1,15 @@
description: End to end AP pipeline specialized for DECam
# (1) Run $AP_PIPE_DIR/pipelines/DarkEnergyCamera/RunIsrForCrosstalkSources.yaml
# (1) Run $AP_PIPE_DIR/pipelines/DECam/RunIsrForCrosstalkSources.yaml
# (2) Execute `make_apdb.py`, e.g.,
# make_apdb.py -c db_url="sqlite:////project/user/association.db"
# (3) Run this pipeline, setting appropriate diaPipe configs
# (diaPipe configs should match the make_apdb.py configs)

instrument: lsst.obs.decam.DarkEnergyCamera
imports:
- location: $AP_PIPE_DIR/pipelines/DarkEnergyCamera/ProcessCcd.yaml
- location: $AP_PIPE_DIR/pipelines/DECam/ProcessCcd.yaml
- location: $AP_PIPE_DIR/pipelines/_ingredients/ApPipe.yaml
exclude: # These tasks come from DarkEnergyCamera/ProcessCcd.yaml instead
exclude: # These tasks come from DECam/ProcessCcd.yaml instead
- processCcd
subsets:
# apPipe must be redefined from $AP_PIPE_DIR/pipelines/_ingredients/ApPipe.yaml
Expand Down
6 changes: 3 additions & 3 deletions ...arkEnergyCamera/ApPipeCalibrateImage.yaml → pipelines/DECam/ApPipeCalibrateImage.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,16 +1,16 @@
# TODO DM-40388: This file will replace ApPipe.yaml, once the new pipeline is vetted.
description: End to end AP pipeline specialized for DECam
# (1) Run $AP_PIPE_DIR/pipelines/DarkEnergyCamera/RunIsrForCrosstalkSources.yaml
# (1) Run $AP_PIPE_DIR/pipelines/DECam/RunIsrForCrosstalkSources.yaml
# (2) Execute `make_apdb.py`, e.g.,
# make_apdb.py -c db_url="sqlite:////project/user/association.db"
# (3) Run this pipeline, setting appropriate diaPipe configs
# (diaPipe configs should match the make_apdb.py configs)

instrument: lsst.obs.decam.DarkEnergyCamera
imports:
- location: $AP_PIPE_DIR/pipelines/DarkEnergyCamera/ProcessCcdCalibrateImage.yaml
- location: $AP_PIPE_DIR/pipelines/DECam/ProcessCcdCalibrateImage.yaml
- location: $AP_PIPE_DIR/pipelines/_ingredients/ApPipeCalibrateImage.yaml
exclude: # These tasks come from DarkEnergyCamera/ProcessCcdCalibrateImage.yaml instead
exclude: # These tasks come from DECam/ProcessCcdCalibrateImage.yaml instead
- processCcd
subsets:
# apPipe must be redefined from $AP_PIPE_DIR/pipelines/_ingredients/ApPipeCalibrateImage.yaml
Expand Down
4 changes: 2 additions & 2 deletions ...nes/DarkEnergyCamera/ApPipeWithFakes.yaml → pipelines/DECam/ApPipeWithFakes.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
description: DECam AP Pipeline with synthetic/fake sources. Templates are inputs.
# Remember:
# (0) Ensure median calibration products and template coadds exist for the data being processed
# (1) Run $AP_PIPE_DIR/pipelines/DarkEnergyCamera/RunIsrForCrosstalkSources.yaml
# (1) Run $AP_PIPE_DIR/pipelines/DECam/RunIsrForCrosstalkSources.yaml
# (2) Execute `make_apdb.py`, e.g.,
# make_apdb.py -c db_url="sqlite:////project/user/association.db"
# (3) Run this pipeline, setting appropriate diaPipe configs
Expand All @@ -12,7 +12,7 @@ imports:
- location: $AP_PIPE_DIR/pipelines/_ingredients/ApPipeWithFakes.yaml
exclude: # These tasks frome from DECam's ApPipe.yaml instead
- processCcd
- location: $AP_PIPE_DIR/pipelines/DarkEnergyCamera/ApPipe.yaml
- location: $AP_PIPE_DIR/pipelines/DECam/ApPipe.yaml
exclude: # These tasks come from ApPipeWithFakes.yaml instead
- retrieveTemplate
- subtractImages
Expand Down
4 changes: 2 additions & 2 deletions pipelines/DarkEnergyCamera/ApTemplate.yaml → pipelines/DECam/ApTemplate.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,9 +4,9 @@ description: The AP template building pipeline specialized for DECam

instrument: lsst.obs.decam.DarkEnergyCamera
imports:
- location: $AP_PIPE_DIR/pipelines/DarkEnergyCamera/ProcessCcd.yaml
- location: $AP_PIPE_DIR/pipelines/DECam/ProcessCcd.yaml
- location: $AP_PIPE_DIR/pipelines/_ingredients/ApTemplate.yaml
exclude: # These tasks come from DarkEnergyCamera/ProcessCcd.yaml instead
exclude: # These tasks come from DECam/ProcessCcd.yaml instead
- processCcd

subsets:
Expand Down
4 changes: 2 additions & 2 deletions pipelines/DarkEnergyCamera/ProcessCcd.yaml → pipelines/DECam/ProcessCcd.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
description: A set of tasks to run when processing raw images, specialized for DECam
# You must run $AP_PIPE_DIR/pipelines/DarkEnergyCamera/RunIsrForCrosstalkSources.yaml first!
# You must run $AP_PIPE_DIR/pipelines/DECam/RunIsrForCrosstalkSources.yaml first!

instrument: lsst.obs.decam.DarkEnergyCamera
imports:
- location: $AP_PIPE_DIR/pipelines/DarkEnergyCamera/RunIsrWithCrosstalk.yaml
- location: $AP_PIPE_DIR/pipelines/DECam/RunIsrWithCrosstalk.yaml
- location: $AP_PIPE_DIR/pipelines/_ingredients/ProcessCcd.yaml
exclude:
- isr
Expand Down
4 changes: 2 additions & 2 deletions ...nergyCamera/ProcessCcdCalibrateImage.yaml → ...lines/DECam/ProcessCcdCalibrateImage.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,10 +1,10 @@
# TODO DM-40388: This file will replace ProcessCcd.yaml, once the new pipeline is vetted.
description: A set of tasks to run when processing raw images, specialized for DECam
# You must run $AP_PIPE_DIR/pipelines/DarkEnergyCamera/RunIsrForCrosstalkSources.yaml first!
# You must run $AP_PIPE_DIR/pipelines/DECam/RunIsrForCrosstalkSources.yaml first!

instrument: lsst.obs.decam.DarkEnergyCamera
imports:
- location: $AP_PIPE_DIR/pipelines/DarkEnergyCamera/RunIsrWithCrosstalk.yaml
- location: $AP_PIPE_DIR/pipelines/DECam/RunIsrWithCrosstalk.yaml
- location: $AP_PIPE_DIR/pipelines/_ingredients/ProcessCcdCalibrateImage.yaml
exclude:
- isr
Expand Down
0 ...ergyCamera/RunIsrForCrosstalkSources.yaml → ...ines/DECam/RunIsrForCrosstalkSources.yaml
View file
Open in desktop
File renamed without changes.
0 ...DarkEnergyCamera/RunIsrWithCrosstalk.yaml → pipelines/DECam/RunIsrWithCrosstalk.yaml
View file
Open in desktop
File renamed without changes.
0 ...mera/RunIsrWithoutInterChipCrosstalk.yaml → ...ECam/RunIsrWithoutInterChipCrosstalk.yaml
View file
Open in desktop
File renamed without changes.
4 changes: 2 additions & 2 deletions pipelines/HyperSuprimeCam/ApPipe.yaml → pipelines/HSC/ApPipe.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,9 +6,9 @@ description: End to end AP pipeline specialized for HSC

instrument: lsst.obs.subaru.HyperSuprimeCam
imports:
- location: $AP_PIPE_DIR/pipelines/HyperSuprimeCam/ProcessCcd.yaml
- location: $AP_PIPE_DIR/pipelines/HSC/ProcessCcd.yaml
- location: $AP_PIPE_DIR/pipelines/_ingredients/ApPipe.yaml
exclude: # These tasks come from HyperSuprimeCam/ProcessCcd.yaml instead
exclude: # These tasks come from HSC/ProcessCcd.yaml instead
- processCcd
subsets:
# apPipe must be redefined from $AP_PIPE_DIR/pipelines/_ingredients/ApPipe.yaml
Expand Down
0 ...HyperSuprimeCam/ApPipeCalibrateImage.yaml → pipelines/HSC/ApPipeCalibrateImage.yaml
View file
Open in desktop
File renamed without changes.
2 changes: 1 addition & 1 deletion ...ines/HyperSuprimeCam/ApPipeWithFakes.yaml → pipelines/HSC/ApPipeWithFakes.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ imports:
- location: $AP_PIPE_DIR/pipelines/_ingredients/ApPipeWithFakes.yaml
exclude: # These tasks frome from HSC's ApPipe.yaml instead
- processCcd
- location: $AP_PIPE_DIR/pipelines/HyperSuprimeCam/ApPipe.yaml
- location: $AP_PIPE_DIR/pipelines/HSC/ApPipe.yaml
exclude: # These tasks come from _ingredients/ApPipeWithFakes.yaml instead
- retrieveTemplate
- subtractImages
Expand Down
4 changes: 2 additions & 2 deletions pipelines/HyperSuprimeCam/ApTemplate.yaml → pipelines/HSC/ApTemplate.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,9 +3,9 @@ description: The AP template building pipeline specialized for HSC

instrument: lsst.obs.subaru.HyperSuprimeCam
imports:
- location: $AP_PIPE_DIR/pipelines/HyperSuprimeCam/ProcessCcd.yaml
- location: $AP_PIPE_DIR/pipelines/HSC/ProcessCcd.yaml
- location: $AP_PIPE_DIR/pipelines/_ingredients/ApTemplate.yaml
exclude: # These tasks come from HyperSuprimeCam/ProcessCcd.yaml instead
exclude: # These tasks come from HSC/ProcessCcd.yaml instead
- processCcd

tasks:
Expand Down
0 pipelines/HyperSuprimeCam/ProcessCcd.yaml → pipelines/HSC/ProcessCcd.yaml
View file
Open in desktop
File renamed without changes.
4 changes: 2 additions & 2 deletions pipelines/LsstCamImSim/ApPipe.yaml → pipelines/LSSTCam-imSim/ApPipe.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,9 +6,9 @@ description: End to end AP pipeline specialized for ImSim

instrument: lsst.obs.lsst.LsstCamImSim
imports:
- location: $AP_PIPE_DIR/pipelines/LsstCamImSim/ProcessCcd.yaml
- location: $AP_PIPE_DIR/pipelines/LSSTCam-imSim/ProcessCcd.yaml
- location: $AP_PIPE_DIR/pipelines/_ingredients/ApPipe.yaml
exclude: # These tasks come from LsstCamImSim/ProcessCcd.yaml instead
exclude: # These tasks come from LSSTCam-imSim/ProcessCcd.yaml instead
- processCcd
subsets:
# apPipe must be redefined from $AP_PIPE_DIR/pipelines/_ingredients/ApPipe.yaml
Expand Down
0 ...es/LsstCamImSim/ApPipeCalibrateImage.yaml → ...s/LSSTCam-imSim/ApPipeCalibrateImage.yaml
View file
Open in desktop
File renamed without changes.
4 changes: 2 additions & 2 deletions pipelines/LsstCamImSim/ApTemplate.yaml → pipelines/LSSTCam-imSim/ApTemplate.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,9 +3,9 @@ description: The AP template building pipeline specialized for ImSim

instrument: lsst.obs.lsst.LsstCamImSim
imports:
- location: $AP_PIPE_DIR/pipelines/LsstCamImSim/ProcessCcd.yaml
- location: $AP_PIPE_DIR/pipelines/LSSTCam-imSim/ProcessCcd.yaml
- location: $AP_PIPE_DIR/pipelines/_ingredients/ApTemplate.yaml
exclude: # These tasks come from LsstCamImSim/ProcessCcd.yaml instead
exclude: # These tasks come from LSSTCam-imSim/ProcessCcd.yaml instead
- processCcd

tasks:
Expand Down
0 pipelines/LsstCamImSim/ProcessCcd.yaml → pipelines/LSSTCam-imSim/ProcessCcd.yaml
View file
Open in desktop
File renamed without changes.
4 changes: 2 additions & 2 deletions pipelines/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,10 +6,10 @@ The pipelines defined here come in three flavors: camera-specific (within named
Pipelines within the ingredients directory are meant to be imported by other pipelines, and are not intended to be used directly by end-users.

The `pipetask build` command can be used to expand a pipeline YAML and resolve any imports for the purposes of visualizing it.
For example, to visualize the `apPipe` subset from the [LSSTCam-imSim ApPipe pipeline](https://github.com/lsst/ap_pipe/blob/main/pipelines/LsstCamImSim/ApPipe.yaml) pipeline, run:
For example, to visualize the `apPipe` subset from the [LSSTCam-imSim ApPipe pipeline](https://github.com/lsst/ap_pipe/blob/main/pipelines/LSSTCam-imSim/ApPipe.yaml) pipeline, run:

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oops, found a problem -- the link on line 9 needs to be updated as well.

```bash
pipetask build \
-p $AP_PIPE_DIR/pipelines/LsstCamImSim/ApPipe.yaml#apPipe \
-p $AP_PIPE_DIR/pipelines/LSSTCam-imSim/ApPipe.yaml#apPipe \
--show pipeline
```