Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Process for identifying datastores that need column indexes #3651

Open
grugnog opened this issue Sep 1, 2021 · 2 comments
Open

Process for identifying datastores that need column indexes #3651

grugnog opened this issue Sep 1, 2021 · 2 comments

Comments

@grugnog
Copy link
Member

grugnog commented Sep 1, 2021

User Story

Datastore queries (at least common ones that have previously been slow) are fast and efficient on server resources

Acceptance Criteria

  • Datastore indexes are added (and re-added if dataset re-uploads occur)
  • Datastore indexes account for real life usage (since it is impractical to anticipate every possible API query)
@grugnog grugnog self-assigned this Sep 1, 2021
@github-actions github-actions bot added this to Incoming/Triage in DKAN 2 Issue Triage Sep 1, 2021
@grugnog grugnog added this to To do in DKAN 2 Development via automation Sep 1, 2021
@grugnog grugnog removed this from Incoming/Triage in DKAN 2 Issue Triage Sep 1, 2021
@grugnog
Copy link
Member Author

grugnog commented Sep 4, 2021

Working is ongoing at https://git.drupalcode.org/project/indexer - I have code to parse slow query log data and identify compound indexes with the most value (most rows read in slow queries and highest cardinality) as well as identify existing indexes (so we don't duplicate those). Next I need to parse the tree and actually add or remove indexes as needed.

@grugnog
Copy link
Member Author

grugnog commented Sep 14, 2021

So the main task of adding/removing indexes based on this analysis is complete. The remaining task is to (optionally) store these in config and automatically add them to new tables that match provided column name combinations.

@grugnog grugnog moved this from To do to Needs review in DKAN 2 Development Sep 17, 2021
@dafeder dafeder moved this from Needs review to To do in DKAN 2 Development Sep 23, 2021
@dafeder dafeder moved this from To do to In progress in DKAN 2 Development Sep 23, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
DKAN 2 Development
  
In progress
Development

No branches or pull requests

1 participant