-
Notifications
You must be signed in to change notification settings - Fork 0
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Reorganize pipelines import structure and fix configs.
This addresses a lot of "backwards" imports, in which the very-much concrete and complete HSC/DRP-RC2 pipeline was imported by other pipelines, by moving its contents to new pipelines in ingredients. This also reduces duplication between HSC and DECam. While it's mostly a refactor, it does change the expanded pipelines in a few ways: - HSC/DRP-Prod now has the correct (full-SSP) configuration for the FGCM tasks, mostly by avoiding RC2-specific overrides. - Two ill-defined subsets (singleFrame and multiVisit) have been removed from all pipelines. - Superfluous "step" subsets have been removed (including all steps in the ci_hsc pipelines). - faro tasks have been removed from HSC/DRP-Prod and ci_hsc.
- Loading branch information
Showing
13 changed files
with
379 additions
and
339 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,78 @@ | ||
| description: | ||
| Base Pipeline for Data Release Production process, with full multi-epoch | ||
| calibration and characterizations. | ||
|
|
||
| Never run this pipeline or any of its subsets directly; always use those in | ||
| the 'pipelines' directory (or create a custom one) instead. | ||
| imports: | ||
| - location: $DRP_PIPE_DIR/ingredients/DRP-minimal-calibration.yaml | ||
| exclude: | ||
| - writeSourceTable # cannot reconfigure because different dimensions. | ||
| tasks: | ||
| skyCorr: | ||
| class: lsst.pipe.drivers.skyCorrection.SkyCorrectionTask | ||
| isolatedStarAssociation: | ||
| class: lsst.pipe.tasks.isolatedStarAssociation.IsolatedStarAssociationTask | ||
| config: | ||
| connections.source_table_visit: preSourceTable_visit | ||
| jointcal: | ||
| class: lsst.jointcal.JointcalTask | ||
| config: | ||
| connections.inputSourceTableVisit: preSourceTable_visit | ||
| fgcmBuildStarsTable: | ||
| class: lsst.fgcmcal.fgcmBuildStarsTable.FgcmBuildStarsTableTask | ||
| config: | ||
| connections.sourceTable_visit: preSourceTable_visit | ||
| fgcmFitCycle: | ||
| class: lsst.fgcmcal.fgcmFitCycle.FgcmFitCycleTask | ||
| fgcmOutputProducts: | ||
| class: lsst.fgcmcal.fgcmOutputProducts.FgcmOutputProductsTask | ||
| writePreSourceTable: | ||
| class: lsst.pipe.tasks.postprocess.WriteSourceTableTask | ||
| config: | ||
| connections.outputCatalog: preSource | ||
| transformPreSourceTable: | ||
| class: lsst.pipe.tasks.postprocess.TransformSourceTableTask | ||
| config: | ||
| connections.inputCatalog: preSource | ||
| connections.outputCatalog: preSourceTable | ||
| consolidatePreSourceTable: | ||
| class: lsst.pipe.tasks.postprocess.ConsolidateSourceTableTask | ||
| config: | ||
| connections.inputCatalogs: preSourceTable | ||
| connections.outputCatalog: preSourceTable_visit | ||
| writeRecalibratedSourceTable: | ||
| class: lsst.pipe.tasks.postprocess.WriteRecalibratedSourceTableTask | ||
| config: | ||
| connections.outputCatalog: source | ||
| measure: | ||
| class: lsst.pipe.tasks.multiBand.MeasureMergedCoaddSourcesTask | ||
| config: | ||
| connections.sourceTableHandles: preSourceTable_visit | ||
| subsets: | ||
| # This pipeline file only defines subsets that have a consistent or | ||
| # near-consistent definition as a set of tasks, for use by downstream | ||
| # pipelines as way to include or exclude those tasks. It does not (and | ||
| # should not) contain "step" subsets aimed at splitting up a processing | ||
| # campaign. | ||
| fgcm: | ||
| subset: | ||
| - fgcmBuildStarsTable | ||
| - fgcmFitCycle | ||
| - fgcmOutputProducts | ||
| description: | | ||
| Subset that includes all FGCM tasks. | ||
| preSourceTable: | ||
| subset: | ||
| - writePreSourceTable | ||
| - transformPreSourceTable | ||
| - consolidatePreSourceTable | ||
| description: > | ||
| Set of tasks to generate parquet PreSource Tables, from single-visit | ||
| processing steps only. | ||
| sourceTable: | ||
| subset: | ||
| - writeRecalibratedSourceTable | ||
| description: > | ||
| Set of tasks to generate parquet Source Tables, using final | ||
| calibrations. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,145 @@ | ||
| description: The DRP Fakes pipeline specalised for the HSC RC2 dataset. | ||
| instrument: lsst.obs.subaru.HyperSuprimeCam | ||
| imports: | ||
| location: "$DRP_PIPE_DIR/ingredients/HSC/DRP.yaml" | ||
| exclude: | ||
| - skyCorr | ||
| - jointcal | ||
| - fgcm | ||
| - fgcmBuildStarsTable | ||
| - fgcmFitCycle | ||
| - fgcmOutputProducts | ||
| - isr | ||
| - characterizeImage | ||
| - calibrate | ||
| - selectGoodSeeingVisits | ||
| - templateGen | ||
| - imageDifference | ||
| - transformDiaSourceCat | ||
| - consolidateDiaSourceTable | ||
| - drpAssociation | ||
| - drpDiaCalculation | ||
| - forcedPhotDiffim | ||
| - forcedPhotCcdOnDiaObjects | ||
| - forcedPhotDiffOnDiaObjects | ||
| - transformForcedSourceTable | ||
| - consolidateForcedSourceTable | ||
| - getTemplate | ||
| - writeForcedSourceTable | ||
| - consolidateAssocDiaSourceTable | ||
| - consolidateFullDiaObjectTable | ||
| - writeForcedSourceOnDiaObjectTable | ||
| - transformForcedSourceOnDiaObjectTable | ||
| - consolidateForcedSourceOnDiaObjectTable | ||
| - faro_all | ||
| tasks: | ||
| singleFrameWithFakes: | ||
| class: lsst.pipe.tasks.processCcdWithFakes.ProcessCcdWithFakesTask | ||
| config: | ||
| insertFakes.doProcessAllDataIds: true | ||
| connections.photoCalibName: "fgcm" | ||
| doApplyExternalGlobalPhotoCalib: true | ||
| doApplyExternalTractSkyWcs: true | ||
| writePreSourceTable: | ||
| class: lsst.pipe.tasks.postprocess.WriteSourceTableTask | ||
| config: | ||
| connections.catalogType: "fakes_" | ||
| connections.outputCatalog: fakes_preSource | ||
| transformPreSourceTable: | ||
| class: lsst.pipe.tasks.postprocess.TransformSourceTableTask | ||
| config: | ||
| connections.inputCatalog: fakes_preSource | ||
| connections.outputCatalog: fakes_preSourceTable | ||
| consolidatePreSourceTable: | ||
| class: lsst.pipe.tasks.postprocess.ConsolidateSourceTableTask | ||
| config: | ||
| connections.inputCatalogs: fakes_preSourceTable | ||
| connections.outputCatalog: fakes_preSourceTable_visit | ||
| writeRecalibratedSourceTable: | ||
| class: lsst.pipe.tasks.postprocess.WriteRecalibratedSourceTableTask | ||
| config: | ||
| connections.outputCatalog: fakes_source | ||
| connections.catalogType: "fakes_" | ||
| transformSourceTable: | ||
| class: lsst.pipe.tasks.postprocess.TransformSourceTableTask | ||
| config: | ||
| connections.catalogType: "fakes_" | ||
| consolidateSourceTable: | ||
| class: lsst.pipe.tasks.postprocess.ConsolidateSourceTableTask | ||
| config: | ||
| connections.catalogType: "fakes_" | ||
| consolidateVisitSummary: | ||
| class: lsst.pipe.tasks.postprocess.ConsolidateVisitSummaryTask | ||
| config: | ||
| connections.calexpType: "fakes_" | ||
| makeCcdVisitTable: | ||
| class: lsst.pipe.tasks.postprocess.MakeCcdVisitTableTask | ||
| config: | ||
| connections.calexpType: "fakes_" | ||
| makeVisitTable: | ||
| class: lsst.pipe.tasks.postprocess.MakeVisitTableTask | ||
| config: | ||
| connections.calexpType: "fakes_" | ||
| assembleCoadd: | ||
| class: lsst.pipe.tasks.assembleCoadd.CompareWarpAssembleCoaddTask | ||
| config: | ||
| connections.inputCoaddName: "fakes_deep" | ||
| connections.outputCoaddName: "fakes_deep" | ||
| makeWarp: | ||
| class: lsst.pipe.tasks.makeCoaddTempExp.MakeWarpTask | ||
| config: | ||
| connections.photoCalibName: "fgcm" | ||
| useGlobalExternalPhotoCalib: true | ||
| connections.coaddName: "fakes_deep" | ||
| connections.calexpType: "fakes_" | ||
| detection: | ||
| class: lsst.pipe.tasks.multiBand.DetectCoaddSourcesTask | ||
| config: | ||
| connections.inputCoaddName: "fakes_deep" | ||
| connections.outputCoaddName: "fakes_deep" | ||
| mergeDetections: | ||
| class: lsst.pipe.tasks.mergeDetections.MergeDetectionsTask | ||
| config: | ||
| connections.inputCoaddName: "fakes_deep" | ||
| connections.outputCoaddName: "fakes_deep" | ||
| deblend: | ||
| class: lsst.pipe.tasks.deblendCoaddSourcesPipeline.DeblendCoaddSourcesMultiTask | ||
| config: | ||
| connections.inputCoaddName: "fakes_deep" | ||
| connections.outputCoaddName: "fakes_deep" | ||
| measure: | ||
| class: lsst.pipe.tasks.multiBand.MeasureMergedCoaddSourcesTask | ||
| config: | ||
| connections.inputCoaddName: "fakes_deep" | ||
| connections.outputCoaddName: "fakes_deep" | ||
| mergeMeasurements: | ||
| class: lsst.pipe.tasks.mergeMeasurements.MergeMeasurementsTask | ||
| config: | ||
| connections.inputCoaddName: "fakes_deep" | ||
| connections.outputCoaddName: "fakes_deep" | ||
| writeObjectTable: | ||
| class: lsst.pipe.tasks.postprocess.WriteObjectTableTask | ||
| config: | ||
| connections.coaddName: "fakes_deep" | ||
| transformObjectTable: | ||
| class: lsst.pipe.tasks.postprocess.TransformObjectCatalogTask | ||
| config: | ||
| connections.coaddName: "fakes_deep" | ||
| forcedPhotCcd: | ||
| class: lsst.meas.base.forcedPhotCcd.ForcedPhotCcdTask | ||
| config: | ||
| connections.inputCoaddName: "fakes_deep" | ||
| forcedPhotCoadd: | ||
| class: lsst.meas.base.forcedPhotCoadd.ForcedPhotCoaddTask | ||
| config: | ||
| connections.inputCoaddName: "fakes_deep" | ||
| connections.outputCoaddName: "fakes_deep" | ||
| healSparsePropertyMaps: | ||
| class: lsst.pipe.tasks.healSparseMapping.HealSparsePropertyMapTask | ||
| config: | ||
| connections.coaddName: "fakes_deep" | ||
| connections.calexpType: "fakes_" | ||
| consolidateHealSparsePropertyMaps: | ||
| class: lsst.pipe.tasks.healSparseMapping.ConsolidateHealSparsePropertyMapTask | ||
| config: | ||
| connections.coaddName: "fakes_deep" |
Oops, something went wrong.