DM-26593: Create an integrated pipeline for AP fake analysis #118
Conversation
Modify ap_verify pipeline
Debug connections.
Move fakes to specific pipelines.
Pull latest pipeline. Debug pipeline.
morriscb
force-pushed
the
tickets/DM-26593
branch
from
1c352c3
to
29a4afd
Compare
pipelines/ApVerifyWithFakes.yaml
Outdated
| - location: $AP_PIPE_DIR/pipelines/ApPipe.yaml | ||
| - location: $AP_VERIFY_DIR/pipelines/MetricsRuntime.yaml | ||
| - location: $AP_VERIFY_DIR/pipelines/MetricsMiscWithFakes.yaml |
There was a problem hiding this comment.
It might be more future-proof to import ApVerify.yaml here, though I see below that you end up overriding a lot of configs anyway.
pipelines/ApVerifyWithFakes.yaml
Outdated
| # Always prefer decorrelation; may eventually become ImageDifferenceTask default | ||
| doDecorrelation: True | ||
| detection.thresholdValue: 5.0 # needed with doDecorrelation | ||
| # Don't have source catalogs for templates | ||
| doSelectSources: False |
There was a problem hiding this comment.
You can leave these lines out; they're redundant with the import.
pipelines/ApVerifyWithFakes.yaml
Outdated
| apdb.isolation_level: READ_UNCOMMITTED | ||
| doPackageAlerts: True |
There was a problem hiding this comment.
These lines are redundant if you import ApVerify.yaml, but not if you use ApPipe.yaml. (Putting the isolation level in the original pipeline was arguably a mistake, but that's out of scope for this ticket.)
pipelines/ApVerifyWithFakes.yaml
Outdated
| # Metric inputs must match pipeline outputs | ||
| - imageDifference.connections.coaddName == fracDiaSourcesToSciSources.connections.coaddName | ||
| - imageDifference.connections.fakesType == fracDiaSourcesToSciSources.connections.fakesType |
There was a problem hiding this comment.
Redundant if you import ApVerify.yaml, but do you need any extra contracts for the fakes* tasks?
pipelines/MetricsMiscWithFakes.yaml
Outdated
| numNewDiaObjects: | ||
| class: lsst.ap.association.metrics.NumberNewDiaObjectsMetricTask | ||
| config: | ||
| connections.labelName: diaPipe # partial name of metadata dataset | ||
| numUnassociatedDiaObjects: | ||
| class: lsst.ap.association.metrics.NumberUnassociatedDiaObjectsMetricTask | ||
| config: | ||
| connections.labelName: diaPipe | ||
| fracUpdatedDiaObjects: | ||
| class: lsst.ap.association.metrics.FractionUpdatedDiaObjectsMetricTask | ||
| config: | ||
| connections.labelName: diaPipe | ||
| totalUnassociatedDiaObjects: | ||
| class: lsst.ap.association.metrics.TotalUnassociatedDiaObjectsMetricTask | ||
| numSciSources: | ||
| class: lsst.ip.diffim.metrics.NumberSciSourcesMetricTask |
There was a problem hiding this comment.
Why not replace these with an import? Or (given my advice to import ApVerify.yaml above) why not create a pipeline file that only has the fakes-specific metrics, then import it and MetricsMisc.yaml separately? As it stands, the files need to be kept in sync if somebody creates a new metric.
Inherit pipelines. Add contracts or all fakes tasks.
Debug Pipelines
morriscb
force-pushed
the
tickets/DM-26593
branch
from
838ff12
to
a33ffcc
Compare
No description provided.