Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

DM-36395: Support incremental redefinition of multi-snap visits #447

Merged
merged 11 commits into from
May 4, 2023
Merged

DM-36395: Support incremental redefinition of multi-snap visits #447

merged 11 commits into from
May 4, 2023

Commits on May 3, 2023

  1. Fix multiprocessing pool ResourceWarning in ingest 

    We have to close the Pool after we are done with it.
    @timj
    timj committed May 3, 2023
    Configuration menu
    Copy the full SHA
    d2a14e4 View commit details
    Browse the repository at this point in the history

  2. Add update-records option to butler define-visits

    @timj
    timj committed May 3, 2023
    Configuration menu
    Copy the full SHA
    54b9ca2 View commit details
    Browse the repository at this point in the history

  3. Allow multi-snap visits to be redefined incrementally 

    Previously the multi-snap visit was only defined when the second
snap arrived. Now we can define the multi-snap visit even if
we only have one snap (so we get two visits defined). When the
second snap arrives there will be three visits and the multi-snap
visit will be updated if --update-records is enabled.
    @timj
    timj committed May 3, 2023
    Configuration menu
    Copy the full SHA
    c4cd6e0 View commit details
    Browse the repository at this point in the history

  4. Add --incremental update option that will update visit_definition ent… 

    …ries

Implies update-records and so should be used when exposures for
multi-snap visits are being ingested incrementally whilst
continually updating the visit definition.
    @timj
    timj committed May 3, 2023
    Configuration menu
    Copy the full SHA
    d647bcf View commit details
    Browse the repository at this point in the history

  5. Change the test_all test to test all 

    We had not been updating it as new options were added.
    @timj
    timj committed May 3, 2023
    Configuration menu
    Copy the full SHA
    c67d8de View commit details
    Browse the repository at this point in the history

  6. Add tests for incremental visit definition

    @timj
    timj committed May 3, 2023
    Configuration menu
    Copy the full SHA
    82c13e1 View commit details
    Browse the repository at this point in the history

  7. Add news fragment

    @timj
    timj committed May 3, 2023
    Configuration menu
    Copy the full SHA
    ee1505e View commit details
    Browse the repository at this point in the history

  8. Allow visit definition to work with a single exposure in multi-snap 

    This is the true incremental mode where we ingest a new file
and immediately run visit definition.
    @timj
    timj committed May 3, 2023
    Configuration menu
    Copy the full SHA
    1c7d142 View commit details
    Browse the repository at this point in the history

  9. Tweak the docstrings for incremental mode

    @timj
    timj committed May 3, 2023
    Configuration menu
    Copy the full SHA
    1594078 View commit details
    Browse the repository at this point in the history

  10. Create DataCoordinate directly rather than querying dataIDs 

    Since we know that it's a single dataId we can make it more
efficiently with DataCoordinate rather than querying registry
for a single value.
    @timj
    timj committed May 3, 2023
    Configuration menu
    Copy the full SHA
    491c13c View commit details
    Browse the repository at this point in the history

  11. Optimize missing query to only ask for exposures that we don't have 

    Rather than asking for every exposure in the visit, first
work out which exposures we have and then determine the
missing sequence numbers and query for those.
    @timj
    timj committed May 3, 2023
    Configuration menu
    Copy the full SHA
    112cb9b View commit details
    Browse the repository at this point in the history