Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

DM-30563: ap_verify failing to find dataset type fakes_deepDiff_warpedExp #130

Merged
merged 3 commits into from
Jun 9, 2021
Merged

DM-30563: ap_verify failing to find dataset type fakes_deepDiff_warpedExp #130

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
ea530c3
Clarify reasons for ApVerify.yaml configs.
kfindeisen Jun 8, 2021
68fb45f
Override all coadd-dependent dataset types in ApVerify.yaml.
kfindeisen Jun 8, 2021
5eb6fb1
Add explicit overrides for "fakes_" datasets.
kfindeisen Jun 8, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
21 changes: 18 additions & 3 deletions pipelines/ApVerify.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,29 +10,44 @@ tasks:
imageDifference:
class: lsst.pipe.tasks.imageDifference.ImageDifferenceTask
config:
coaddName: deep
getTemplate.coaddName: deep
# All extant datasets use deep templates
# TODO: move to dataset configs in DM-26140
coaddName: deep # Can be removed once ImageDifference no longer supports Gen 2
getTemplate.coaddName: deep # Can be removed once ImageDifference no longer supports Gen 2
connections.coaddName: deep
# TODO: redundant connection definitions workaround for DM-30210
kfindeisen marked this conversation as resolved.
Show resolved Hide resolved
connections.coaddExposures: deepCoadd
connections.outputSchema: deepDiff_diaSrc_schema
connections.subtractedExposure: deepDiff_differenceExp
connections.scoreExposure: deepDiff_scoreExp
connections.warpedExposure: deepDiff_warpedExp
connections.matchedExposure: deepDiff_matchedExp
connections.diaSources: deepDiff_diaSrc
# TODO: end DM-30210 workaround
transformDiaSrcCat:
class: lsst.ap.association.TransformDiaSourceCatalogTask
config:
# TODO: move to dataset configs in DM-26140
connections.coaddName: deep
# TODO: redundant connection definitions workaround for DM-30210
connections.diaSourceSchema: deepDiff_diaSrc_schema
connections.diaSourceCat: deepDiff_diaSrc
connections.diffIm: deepDiff_differenceExp
connections.diaSourceTable: deepDiff_diaSrcTable
# TODO: end DM-30210 workaround
diaPipe:
# TODO: how to prevent duplication with ApPipe definition?
class: lsst.ap.association.DiaPipelineTask
config:
apdb.isolation_level: READ_UNCOMMITTED
doPackageAlerts: True
# TODO: move to dataset configs in DM-26140
connections.coaddName: deep
# TODO: redundant connection definitions workaround for DM-30210
connections.diaSourceTable: deepDiff_diaSrcTable
connections.diffIm: deepDiff_differenceExp
connections.warpedExposure: deepDiff_warpedExp
connections.associatedDiaSources: deepDiff_assocDiaSrcTable
# TODO: end DM-30210 workaround
contracts:
# Metric inputs must match pipeline outputs
- imageDifference.connections.coaddName == fracDiaSourcesToSciSources.connections.coaddName
Expand Down
30 changes: 29 additions & 1 deletion pipelines/ApVerifyWithFakes.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,16 +27,44 @@ tasks:
class: lsst.pipe.tasks.imageDifference.ImageDifferenceTask
config:
connections.fakesType: "fakes_"
# TODO: redundant connection definitions workaround for DM-30210
# TODO: move hardcoding of "deep" to dataset configs in DM-26140
connections.coaddName: deep # For file-level consistency; hardcoded values force "deep" anyway
connections.exposure: fakes_calexp
connections.coaddExposures: fakes_deepCoadd
connections.dcrCoadds: fakes_dcrCoadd
connections.outputSchema: fakes_deepDiff_diaSrc_schema
connections.subtractedExposure: fakes_deepDiff_differenceExp
connections.scoreExposure: fakes_deepDiff_scoreExp
connections.warpedExposure: fakes_deepDiff_warpedExp
connections.matchedExposure: fakes_deepDiff_matchedExp
connections.diaSources: fakes_deepDiff_diaSrc
# TODO: end DM-30210 workaround
transformDiaSrcCat:
class: lsst.ap.association.TransformDiaSourceCatalogTask
config:
connections.fakesType: "fakes_"
# TODO: redundant connection definitions workaround for DM-30210
# TODO: move hardcoding of "deep" to dataset configs in DM-26140
connections.coaddName: deep # For file-level consistency; hardcoded values force "deep" anyway
connections.diaSourceSchema: fakes_deepDiff_diaSrc_schema
connections.diaSourceCat: fakes_deepDiff_diaSrc
connections.diffIm: fakes_deepDiff_differenceExp
connections.diaSourceTable: fakes_deepDiff_diaSrcTable
# TODO: end DM-30210 workaround
diaPipe:
# TODO: how to prevent duplication with ApPipe definition?
class: lsst.ap.association.DiaPipelineTask
config:
doWriteAssociatedSources: True
connections.fakesType: "fakes_"
# TODO: redundant connection definitions workaround for DM-30210
# TODO: move hardcoding of "deep" to dataset configs in DM-26140
connections.coaddName: deep # For file-level consistency; hardcoded values force "deep" anyway
connections.diaSourceTable: fakes_deepDiff_diaSrcTable
connections.diffIm: fakes_deepDiff_differenceExp
connections.warpedExposure: fakes_deepDiff_warpedExp
connections.associatedDiaSources: fakes_deepDiff_assocDiaSrc
# TODO: end DM-30210 workaround
matchFakes:
class: lsst.ap.pipe.matchApFakes.MatchApFakesTask
config:
Expand Down