-
Notifications
You must be signed in to change notification settings - Fork 12
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
DM-41117: Improvements to caching and Butler initialization #904
Commits on Nov 8, 2023
-
Remove caching of collection summaries.
Pre-fetching of collection summaries was quite expensive and we did not use those summaries very often. Removing the cache completely, now we query summaries each time but only for the collections that are actually used.
Configuration menu - View commit details
-
Copy full SHA for 6bba713 - Browse repository at this point
Copy the full SHA 6bba713View commit details -
Configuration menu - View commit details
-
Copy full SHA for 780e66e - Browse repository at this point
Copy the full SHA 780e66eView commit details -
Fetch all chained collection definitions in one query.
`DefaultCollectionManager.refresh` now runs a single query to fetch full contents of collection_chain table. This removes update logic from collection record classes which became simple data classes now.
Configuration menu - View commit details
-
Copy full SHA for d8c3aed - Browse repository at this point
Copy the full SHA d8c3aedView commit details -
Reduce need for schema reflection for registry tables.
Static tables do not really need schema verification because we rely on version numbers in butler_attributes. Verification and reflection may still be usefule for dynamic tables (tags/calibs) but we now delay it until the tables are actually used.
Configuration menu - View commit details
-
Copy full SHA for 0d87929 - Browse repository at this point
Copy the full SHA 0d87929View commit details -
Configuration menu - View commit details
-
Copy full SHA for 8481829 - Browse repository at this point
Copy the full SHA 8481829View commit details -
Configuration menu - View commit details
-
Copy full SHA for b1d4cb8 - Browse repository at this point
Copy the full SHA b1d4cb8View commit details
Commits on Nov 9, 2023
-
Configuration menu - View commit details
-
Copy full SHA for 5833c91 - Browse repository at this point
Copy the full SHA 5833c91View commit details
Commits on Nov 14, 2023
-
Add special context manager to enable caching in registry.
Adds special cache classes for collection and summary records and an additional structure that holds caches. New registry method is a context manager that enables caches temporarily for the duration of that context.
Configuration menu - View commit details
-
Copy full SHA for d174d63 - Browse repository at this point
Copy the full SHA d174d63View commit details -
Re-enable dataset type caching using new class DatasetTypeCache.
Unlike collection caches, dataset type cache is always on, this helps to reduce number of queries in `pipetask run` without the need to explicitly enable caching in multiple places.
Configuration menu - View commit details
-
Copy full SHA for 254b87a - Browse repository at this point
Copy the full SHA 254b87aView commit details -
Configuration menu - View commit details
-
Copy full SHA for eaa968d - Browse repository at this point
Copy the full SHA eaa968dView commit details -
Add
Butler._caching_context
method.Registry shiim now redirects to this method instead of Registry method. RemoteButler raises NotImplementedError but may do something non-trivial later when we know how caching is going to work with client/server.
Configuration menu - View commit details
-
Copy full SHA for 922359c - Browse repository at this point
Copy the full SHA 922359cView commit details