Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Create pg_trgm index on boundary_huc* tables to speed up search #3207

Merged
merged 2 commits into from Dec 16, 2019

Conversation

@lossyrob
Copy link
Member

lossyrob commented Dec 10, 2019

Overview

This PR adds the pg_trgm extension and creates indexes on the name column of the boundary_huc* tables on database setup.

See this blog post about speeding up the LIKE operator with the pg_trgm extension.

Connects #2833

Demo

mmw-search

Notes

With the index, I still have seen some slow-ish database query times - but this seems to be on startup, and after a bit of warm up time the queries come back quite fast.

Testing Instructions

  • Re-run the setupdb.sh script as explained in the README changes as part of this PR.
  • Perform the duplication steps of the related issue
  • Notice it's not as slow as it once was.

Checklist

  • All JavaScript tests pass ./scripts/testem.sh (no javascript changes)
lossyrob added 2 commits Dec 10, 2019
This speeds up the _do_boundary_search search logic that uses a LIKE
filter on the name column of these boundary tables.

Refs #2833
@lossyrob lossyrob requested a review from rajadain Dec 10, 2019
@lossyrob lossyrob changed the title Creat Create pg_trgm index on boundary_huc* tables to speed up search Dec 11, 2019
@rajadain rajadain added the PA DEP label Dec 13, 2019
Copy link
Member

rajadain left a comment

+1 tested. Much faster after the warm-up. Do you have any ideas / recommendations for what can be done about the cold start? Do you know if it is the database query that's the bottleneck or something else (like establishing a database connection) holding it back for the first request?

@rajadain rajadain assigned lossyrob and unassigned rajadain Dec 13, 2019
@lossyrob

This comment has been minimized.

Copy link
Member Author

lossyrob commented Dec 16, 2019

I was seeing longer query times on cold startup, so it seems like potentially an index caching issue? I don't have a recommendation for a workaround - we could perform a few queries against the database to warm it up on startup, but that would introduce some deployment and maintenance complexity that might not be worth it.

@lossyrob lossyrob merged commit 355def2 into develop Dec 16, 2019
2 checks passed
2 checks passed
default Build finished.
Details
model-my-watershed-pull-requests Build #4115 succeeded in 9 min 28 sec
Details
@lossyrob lossyrob deleted the rde/fix/slow-huc branch Dec 16, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

2 participants
You can’t perform that action at this time.