DM-38269: For LSST cameras, get group/snap from sidecar JSON. #55
Conversation
ktlim
force-pushed
the
tickets/DM-38269
branch
19 times, most recently
from
0f43b24 to
61c0398
Compare
ktlim
force-pushed
the
tickets/DM-38269
branch
2 times, most recently
from
aaea135 to
244802f
Compare
There was a problem hiding this comment.
I'm a bit concerned about the way this has been implemented -- the new Snap.is_consistent method breaks down a lot of the careful separation of functionality you yourself had insisted on in the beginning, and I don't understand why the new code looks for raws in the central repo. I think I'd at least like to understand the motivation for these changes a little better.
| """ | ||
| datasets = set(self.central_butler.registry.queryDatasets( | ||
| datasetType="raw", | ||
| collections=f"{instrument}/raw/all", |
There was a problem hiding this comment.
This is an unsafe operation for two reasons:
- As I understand it, raws are supposed to be ingested to the central repo independently of the PP workflow, and the prototype code will no longer do so on or after DM-36051. That makes this check unreliable for deciding if raws are available.
- Because the repo is shared among all workers,
self.central_butleris prone to concurrency problems. I'd recommend keeping the number ofcentral_butleroperations to the absolute minimum; if it really is necessary to query it here, then please callself.central_butler.registry.refresh()first.
There was a problem hiding this comment.
I'm no longer doing this — but I'm no longer doing anything. Since the Kafka consumer is subscribed at the beginning of next_visit_handler, I think it's only cold starts that could have an image show up before the subscription. Otherwise, all image notifications should be picked up by consumer.consume().
ktlim
force-pushed
the
tickets/DM-38269
branch
2 times, most recently
from
d20fdfb to
8b1c929
Compare
|
I've redone this with far fewer changes to activator.py but totally getting rid of the I have a bunch of LSST-specific code that recalculates certain things like detector numbers and exposure ids. I was reluctant to use |
ktlim
force-pushed
the
tickets/DM-38269
branch
3 times, most recently
from
e4f159b to
94bd0e6
Compare
This allows LSST cameras to use different path patterns from other cameras, and it allows LSST cameras to get their group information from a "sidecar" JSON file rather than the path. Determining based on a group id whether the images associated with it have already arrived works for non-LSST cameras but is left for a future ticket for the LSST cameras.
4 participants
No description provided.