Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DM-42825: Prompt Processing does slow calib queries against /repo/embargo #153

Merged
merged 5 commits into from
Apr 15, 2024

Commits on Apr 10, 2024

  1. Prefer concrete data IDs in Butler queries.

    Dataset queries done using fixed IDs are 2-3 times faster than queries
    done using the where keyword, a significant savings when querying the
    central repo. However, queries against a set of IDs cannot be converted
    to this form and must continue using where.
    kfindeisen committed Apr 10, 2024
    Configuration menu
    Copy the full SHA
    70babed View commit details
    Browse the repository at this point in the history

Commits on Apr 15, 2024

  1. Remove outdated comment.

    The comment justifies why we can't filter by validity range, but we
    have been doing so for a while.
    kfindeisen committed Apr 15, 2024
    Configuration menu
    Copy the full SHA
    b645f18 View commit details
    Browse the repository at this point in the history
  2. Avoid queries against ... dataset types.

    Butler registry queries against all dataset types are an order of
    magnitude less efficient than an equivalent set of queries that
    enumerates all possible types.
    kfindeisen committed Apr 15, 2024
    Configuration menu
    Copy the full SHA
    749167b View commit details
    Browse the repository at this point in the history
  3. Factor out enumeration of refcat dataset types.

    Creating a function that returns the types makes it possible to replace
    the hardcoded list with something more flexible later.
    kfindeisen committed Apr 15, 2024
    Configuration menu
    Copy the full SHA
    3efcb2d View commit details
    Browse the repository at this point in the history
  4. Replace hardcoded refcat names with a query.

    This still forces us to load refcats the pipelines don't use, but at
    least prevents us from having to manually curate a list of known types.
    kfindeisen committed Apr 15, 2024
    Configuration menu
    Copy the full SHA
    e886784 View commit details
    Browse the repository at this point in the history