Skip to content

DM-35053: Implement syncing of output products for Prompt Processing #24

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 20 commits into from
Oct 11, 2022
Merged

DM-35053: Implement syncing of output products for Prompt Processing #24

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
20 commits
Select commit Hold shift + click to select a range
cafe9f3
Use correct types in Visit objects.
kfindeisen Aug 24, 2022
63b215b
Use dataclasses.replace to update Visit objects.
kfindeisen Aug 24, 2022
a3706e0
Clarify input type of run_pipeline.
kfindeisen Aug 15, 2022
b625c35
Remove DM-34202 workaround from MiddlewareInterface.
kfindeisen Aug 15, 2022
cdad18f
Merge internal attributes in MiddlewareInterface.
kfindeisen Aug 17, 2022
f97cb94
Standardize subtask setup in MiddlewareInterface.
kfindeisen Aug 17, 2022
6f40a58
Move Butler collection setup to MiddlewareInterface.__init__.
kfindeisen Aug 17, 2022
f719dc0
Make explicit the pre- and postconditions of the main methods on Midd…
kfindeisen Sep 1, 2022
73f9bf8
Move run definition to run_pipeline.
kfindeisen Aug 17, 2022
2590269
Fix overwriting bug in MiddlewareInterface output chain.
kfindeisen Aug 17, 2022
42fd4ca
Move pipeline definition to run_pipeline.
kfindeisen Aug 18, 2022
b5e32f9
Ingest raws into group-specific runs.
kfindeisen Aug 24, 2022
92ad273
Add MiddlewareInterface test class for writeable tests.
kfindeisen Aug 22, 2022
fac49b4
Implement MiddlewareInterface.export_outputs.
kfindeisen Aug 23, 2022
14e89ad
Call export_outputs in main pipeline sequence.
kfindeisen Aug 24, 2022
0e59a93
Clarify use of timestamped collections for output runs.
kfindeisen Aug 24, 2022
5201378
Fix deprecation warning from refObjLoader.
kfindeisen Sep 1, 2022
fea4d44
Use a single Butler object in MiddlewareInterface.
kfindeisen Sep 6, 2022
14a0e53
Remove internal use of Registry.refresh.
kfindeisen Sep 6, 2022
507b924
Use daily instead of weekly in main container builds.
kfindeisen Oct 10, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion Dockerfile.main
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
FROM lsstsqre/centos:w_latest
FROM lsstsqre/centos:d_latest
ENV PYTHONUNBUFFERED True
RUN source /opt/lsst/software/stack/loadLSST.bash \
&& mamba install -y \
Expand Down
2 changes: 0 additions & 2 deletions pipelines/calibrate.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,10 +6,8 @@
# band with the most depth).

config.connections.astromRefCat = "gaia"
config.astromRefObjLoader.ref_dataset_name = config.connections.astromRefCat
config.astromRefObjLoader.anyFilterMapsToThis = "phot_g_mean"
config.astromRefObjLoader.filterMap = {}

# Use panstarrs for photometry (grizy filters).
config.connections.photoRefCat = "panstarrs"
config.photoRefObjLoader.ref_dataset_name = config.connections.photoRefCat
7 changes: 5 additions & 2 deletions python/activator/activator.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -84,7 +84,7 @@
# the central repo is located, either, so perhaps we need a new module.
central_butler = Butler(calib_repo,
collections=[active_instrument.makeCollectionName("defaults")],
writeable=False,
writeable=True,
inferDefaults=False)
repo = f"/tmp/butler-{os.getpid()}"
butler = Butler(Butler.makeRepo(repo), writeable=True)
Expand Down Expand Up @@ -176,7 +176,7 @@ def next_visit_handler() -> Tuple[str, int]:
)
if oid:
m = re.match(RAW_REGEXP, oid)
mwi.ingest_image(oid)
mwi.ingest_image(expected_visit, oid)
expid_set.add(m.group('expid'))

_log.debug(f"Waiting for snaps from {expected_visit}.")
Expand Down Expand Up @@ -227,6 +227,9 @@ def next_visit_handler() -> Tuple[str, int]:
_log.warning(f"Processing {len(expid_set)} snaps, expected {expected_visit.snaps}.")
_log.info(f"Running pipeline on {expected_visit}.")
mwi.run_pipeline(expected_visit, expid_set)
# TODO: broadcast alerts here
# TODO: call export_outputs on success or permanent failure in DM-34141
mwi.export_outputs(expected_visit, expid_set)
return "Pipeline executed", 200
else:
_log.error(f"Timed out waiting for images for {expected_visit}.")
Expand Down
Loading