Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DM-41117: Improvements to caching and Butler initialization #904

Merged
merged 11 commits into from
Nov 14, 2023

Commits on Nov 8, 2023

  1. Remove caching of collection summaries.

    Pre-fetching of collection summaries was quite expensive and we did not use
    those summaries very often. Removing the cache completely, now we query
    summaries each time but only for the collections that are actually used.
    andy-slac committed Nov 8, 2023
    Configuration menu
    Copy the full SHA
    6bba713 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    780e66e View commit details
    Browse the repository at this point in the history
  3. Fetch all chained collection definitions in one query.

    `DefaultCollectionManager.refresh` now runs a single query to fetch
    full contents of collection_chain table. This removes update logic from
    collection record classes which became simple data classes now.
    andy-slac committed Nov 8, 2023
    Configuration menu
    Copy the full SHA
    d8c3aed View commit details
    Browse the repository at this point in the history
  4. Reduce need for schema reflection for registry tables.

    Static tables do not really need schema verification because we rely
    on version numbers in butler_attributes. Verification and reflection
    may still be usefule for dynamic tables (tags/calibs) but we now delay
    it until the tables are actually used.
    andy-slac committed Nov 8, 2023
    Configuration menu
    Copy the full SHA
    0d87929 View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    8481829 View commit details
    Browse the repository at this point in the history
  6. Configuration menu
    Copy the full SHA
    b1d4cb8 View commit details
    Browse the repository at this point in the history

Commits on Nov 9, 2023

  1. Configuration menu
    Copy the full SHA
    5833c91 View commit details
    Browse the repository at this point in the history

Commits on Nov 14, 2023

  1. Add special context manager to enable caching in registry.

    Adds special cache classes for collection and summary records and
    an additional structure that holds caches. New registry method is
    a context manager that enables caches temporarily for the duration
    of that context.
    andy-slac committed Nov 14, 2023
    Configuration menu
    Copy the full SHA
    d174d63 View commit details
    Browse the repository at this point in the history
  2. Re-enable dataset type caching using new class DatasetTypeCache.

    Unlike collection caches, dataset type cache is always on, this
    helps to reduce number of queries in `pipetask run` without the need
    to explicitly enable caching in multiple places.
    andy-slac committed Nov 14, 2023
    Configuration menu
    Copy the full SHA
    254b87a View commit details
    Browse the repository at this point in the history
  3. Typing fixes for mypy 1.7

    andy-slac committed Nov 14, 2023
    Configuration menu
    Copy the full SHA
    eaa968d View commit details
    Browse the repository at this point in the history
  4. Add Butler._caching_context method.

    Registry shiim now redirects to this method instead of Registry
    method. RemoteButler raises NotImplementedError but may do something
    non-trivial later when we know how caching is going to work with
    client/server.
    andy-slac committed Nov 14, 2023
    Configuration menu
    Copy the full SHA
    922359c View commit details
    Browse the repository at this point in the history