New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
DM-36395: Support incremental redefinition of multi-snap visits #447
Conversation
a5df38b
to
50f4b4a
Compare
instrument=first.instrument, | ||
) | ||
) | ||
records.difference_update(set(exposures_in_group)) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Seems kind of a shame to have this by group, then merge it all into one big list, then group again. But I guess that keeps the interfaces simple.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, it does, although I think there is a proposal to remove this subclass completely in #453 so it's not worth trying to make it more efficient.
da4a605
to
b5f370b
Compare
We have to close the Pool after we are done with it.
Previously the multi-snap visit was only defined when the second snap arrived. Now we can define the multi-snap visit even if we only have one snap (so we get two visits defined). When the second snap arrives there will be three visits and the multi-snap visit will be updated if --update-records is enabled.
…ries Implies update-records and so should be used when exposures for multi-snap visits are being ingested incrementally whilst continually updating the visit definition.
We had not been updating it as new options were added.
This is the true incremental mode where we ingest a new file and immediately run visit definition.
Since we know that it's a single dataId we can make it more efficiently with DataCoordinate rather than querying registry for a single value.
Rather than asking for every exposure in the visit, first work out which exposures we have and then determine the missing sequence numbers and query for those.
Checklist
doc/changes