Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

DM-43049: Add more plots to LsstComCamSim #109

Merged
merged 8 commits into from
Feb 29, 2024
Merged

DM-43049: Add more plots to LsstComCamSim #109

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
82ee62a
Add analyzeObjectTableExtended to non-CI pipelines
taranu Feb 27, 2024
4040c5e
Sort tasks alphabetically in analysis_tools subset
taranu Feb 27, 2024
24a0b98
Add analyzeDiaSourceTableTract to ops-rehearsal-3 pipeline
mrawls Feb 27, 2024
adbb045
Add diaTractQualityCore to DRP-ci_imsim.yaml
taranu Feb 27, 2024
8a12a5e
Clarify comments in DRP-ci_hsc.yaml
taranu Feb 27, 2024
48ffc4c
Add photometric catalog match/diff tasks to ComCamSim
taranu Feb 28, 2024
3713c55
Disable YPerp for ComCamSim
taranu Feb 28, 2024
78ee970
Add plotPropertyMapTract to DRP-ops-rehearsal-3.yaml
taranu Feb 28, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
7 changes: 4 additions & 3 deletions pipelines/HSC/DRP-RC2.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -236,15 +236,16 @@ subsets:
in the data query.
step8:
subset:
- analyzeMatchedVisitCore
- analyzeObjectTableCore
- analyzeObjectTableExtended
- analyzeObjectTableSurveyCore
- catalogMatchTract
- refCatObjectTract
- validateObjectTableCore
- analyzeMatchedVisitCore
- photometricCatalogMatch
- photometricRefCatObjectTract
- plotPropertyMapTract
- refCatObjectTract
- validateObjectTableCore
description: |
Core (most important), coadd-level plots/metrics from analysis_tools.

Expand Down
7 changes: 4 additions & 3 deletions pipelines/HSC/DRP-RC2_subset.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -408,15 +408,16 @@ subsets:
for bands with no columns in the object table.
nightlyStep8:
subset:
- validateObjectTableCore
- analyzeMatchedVisitCore
- analyzeObjectTableCore
- refCatObjectTract
- analyzeObjectTableExtended
- analyzeObjectTableSurveyCore
- catalogMatchTract
- analyzeMatchedVisitCore
- photometricCatalogMatch
- photometricRefCatObjectTract
- plotPropertyMapTract
- refCatObjectTract
- validateObjectTableCore
description: |
Core (most important), coadd-level plots/metrics from analysis_tools.

Expand Down
13 changes: 5 additions & 8 deletions pipelines/HSC/DRP-ci_hsc.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,11 +10,8 @@ imports:
- faro_visit
- faro_matched
- faro_tract
# Some of these plots fail with ci_hsc data
# - analyzeObjectTableCore
# - refCatObjectTract
# We generally don't want all visit-level plots for non-CI runs
- location: $ANALYSIS_TOOLS_DIR/pipelines/visitQualityCore.yaml
- location: $ANALYSIS_TOOLS_DIR/pipelines/coaddQualityExtended.yaml

tasks:
# Do expect jointcal's photoCalibs to be present even though we didn't
Expand Down Expand Up @@ -56,14 +53,14 @@ tasks:
subsets:
analysis_tools:
subset:
- analyzeMatchedVisitCore
- analyzeObjectTableCore
- analyzeObjectTableSurveyCore
- analyzeObjectTableExtended
- analyzeObjectTableSurveyCore
- analyzeSourceTableCore
- catalogMatchTract
- refCatObjectTract
- validateObjectTableCore
- analyzeMatchedVisitCore
- photometricCatalogMatch
- photometricRefCatObjectTract
- plotPropertyMapTract
- refCatObjectTract
- validateObjectTableCore
6 changes: 3 additions & 3 deletions pipelines/LSSTCam-imSim/DRP-ci_imsim.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ instrument: lsst.obs.lsst.LsstCamImSim
imports:
- $DRP_PIPE_DIR/pipelines/_ingredients/LSSTCam-imSim/DRP.yaml
- $ANALYSIS_DRP_DIR/pipelines/LSSTCam-imSim/analysis_drp_plots.yaml
- $ANALYSIS_TOOLS_DIR/pipelines/coaddQualityExtended.yaml
- $ANALYSIS_TOOLS_DIR/pipelines/diaTractQualityCore.yaml
- $ANALYSIS_TOOLS_DIR/pipelines/visitQualityCore.yaml

tasks:
Expand Down Expand Up @@ -44,12 +44,12 @@ tasks:
subsets:
analysis_tools:
subset:
- analyzeMatchedVisitCore
- analyzeObjectTableCore
- analyzeObjectTableSurveyCore
- analyzeObjectTableExtended
- analyzeObjectTableSurveyCore
- analyzeSourceTableCore
- catalogMatchTract
- diff_matched_analysis
- refCatObjectTract
- validateObjectTableCore
- analyzeMatchedVisitCore
22 changes: 19 additions & 3 deletions pipelines/LSSTComCamSim/DRP-ops-rehearsal-3.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,9 +2,11 @@ description: DRP Pipeline for Simulated ComCam Ops Rehearsal 3
instrument: lsst.obs.lsst.LsstComCamSim
imports:
- $DRP_PIPE_DIR/pipelines/_ingredients/DRP-full.yaml
- $ANALYSIS_TOOLS_DIR/pipelines/coaddColumnValidate.yaml
- $ANALYSIS_TOOLS_DIR/pipelines/coaddQualityCore.yaml
- $ANALYSIS_TOOLS_DIR/pipelines/matchedVisitQualityCore.yaml
- $ANALYSIS_TOOLS_DIR/pipelines/coaddQualityExtended.yaml
- $ANALYSIS_TOOLS_DIR/pipelines/diaTractQualityCore.yaml
- $ANALYSIS_TOOLS_DIR/pipelines/matchedVisitQualityCore.yaml
- $ANALYSIS_TOOLS_DIR/pipelines/visitQualityCore.yaml
tasks:
selectGoodSeeingVisits:
Expand All @@ -24,6 +26,13 @@ tasks:
wcs_provider: "input_summary"
photo_calib_provider: "input_summary"
background_provider: "input_summary"
analyzeObjectTableCore:
class: lsst.analysis.tools.tasks.ObjectTableTractAnalysisTask
config:
python: |
# no z-band in comCamSim
del config.atools.yPerpPSF
del config.atools.yPerpCModel
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I didn't know it was this easy to remove a tool from a task. Cool!

I understand why this has to be here and not obs_lsst too: the pipeline yaml overrides get applied after the obs package ones.

subsets:
step1:
subset:
Expand Down Expand Up @@ -109,10 +118,16 @@ subsets:
- healSparsePropertyMaps
- selectGoodSeeingVisits
- templateGen
- catalogMatchTract
- refCatObjectTract
# analysis_tools tasks
- analyzeMatchedVisitCore
- analyzeObjectTableCore
- analyzeObjectTableExtended
- catalogMatchTract
- photometricCatalogMatch
- photometricRefCatObjectTract
- plotPropertyMapTract
- refCatObjectTract
- validateObjectTableCore
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we also add the plotPropertyMapTract task please.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Its already a member of coaddQualityCore

description: |
Tasks that can be run together, but only after the 'step1' and 'step2d'
subsets.
Expand Down Expand Up @@ -150,6 +165,7 @@ subsets:
- consolidateFullDiaObjectTable
- writeForcedSourceOnDiaObjectTable
- transformForcedSourceOnDiaObjectTable
- analyzeDiaSourceTableTract
description: |
Tasks that can be run together, but only after the 'step1', 'step2',
'step3', and 'step4' subsets
Expand Down
1 change: 1 addition & 0 deletions pipelines/_ingredients/HSC/DRP.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@ imports:
- $ANALYSIS_DRP_DIR/pipelines/analysis_drp_plots.yaml
- $ANALYSIS_TOOLS_DIR/pipelines/coaddColumnValidate.yaml
- $ANALYSIS_TOOLS_DIR/pipelines/coaddQualityCore.yaml
- $ANALYSIS_TOOLS_DIR/pipelines/coaddQualityExtended.yaml
- $ANALYSIS_TOOLS_DIR/pipelines/matchedVisitQualityCore.yaml
tasks:
isolatedStarAssociation:
Expand Down
7 changes: 5 additions & 2 deletions pipelines/_ingredients/LSSTCam-imSim/DRP.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@ imports:
- $ANALYSIS_TOOLS_DIR/pipelines/coaddColumnValidate.yaml
- $ANALYSIS_TOOLS_DIR/pipelines/coaddDiffMatchedQualityExtended.yaml
- $ANALYSIS_TOOLS_DIR/pipelines/coaddQualityCore.yaml
- $ANALYSIS_TOOLS_DIR/pipelines/coaddQualityExtended.yaml
- $ANALYSIS_TOOLS_DIR/pipelines/matchedVisitQualityCore.yaml
parameters:
filterMap: {
Expand Down Expand Up @@ -247,13 +248,15 @@ subsets:
in the data query.
step8:
subset:
- analyzeMatchedVisitCore
- analyzeObjectTableCore
- analyzeObjectTableExtended
- analyzeObjectTableSurveyCore
- catalogMatchTract
- refCatObjectTract
- diff_matched_analysis
- refCatObjectTract
- validateObjectTableCore
- analyzeMatchedVisitCore

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We don't usually incl whitespace between task lists and descriptions

description: |
Core (most important), coadd-level plots/metrics from analysis_tools.

Expand Down