DM-53634: Add daytime AP pipeline and scripts #250
Conversation
Prompt Processing does not save `dia_source_unstandardized` so switch to an output catalog that will be available to the afterburner
isullivan
force-pushed
the
tickets/DM-53634
branch
2 times, most recently
from
7aed524 to
7de4ad0
Compare
|
It could be useful to enable dumping of APDB metrics to the log. It needs one envvar set in your job: |
isullivan
force-pushed
the
tickets/DM-53634
branch
from
11d6216 to
dbbcd98
Compare
| AP daytime pipeline to process images that were missed by Prompt Processing and generate additional plots and metrics. | ||
| instrument: lsst.obs.lsst.LsstCam | ||
| imports: | ||
| - location: $AP_PIPE_DIR/pipelines/_ingredients/ApPipeWithIsrTaskLSST.yaml |
There was a problem hiding this comment.
Should this be $AP_PIPE_DIR/pipelines/LSSTCam/ApPipe.yaml instead? I assume the filterDiaSourcePostReliability config needs to match.
There was a problem hiding this comment.
Even better, I can move the release_id config setting to the bps yaml, and remove ApPipe-daytime entirely.
| imports: | ||
| - location: $AP_PIPE_DIR/pipelines/_ingredients/ApPipeWithIsrTaskLSST.yaml | ||
| exclude: | ||
| - getRegionTimeFromVisit |
There was a problem hiding this comment.
I'm still not sure why this exclusion is needed. getRegionTimeFromVisit was specifically intended for batch processing (since it uses the visitinfo, it's more precise and more accurate than PP's estimate from the nextVisit message).
Unfortunately, it turns out that ap_verify removed it when preprocessing was removed on lsst/ap_verify#238, so a bug may have creeped in without our noticing. 😬
There was a problem hiding this comment.
I am putting getRegionTimeFromVisit back in the daytime pipeline, and will work around the visit-ordering issues.
isullivan
force-pushed
the
tickets/DM-53634
branch
10 times, most recently
from
e41f016 to
9cc5752
Compare
erinleighh
left a comment
erinleighh
left a comment
There was a problem hiding this comment.
Everything looks good to me! I just have a small management question that I'll put on the Jira ticket.
isullivan
force-pushed
the
tickets/DM-53634
branch
from
9cc5752 to
df67dd1
Compare
No description provided.