bug 729679 and bug 746609: admin search index browser and unified doctype #582
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This adds an elasticsearch index browser to the admin. This will make it **way** easier to debug certain classes of problems that require answers to questions such as: * "Did xyz get updated in the index?" * "What the hell did we put in the index and does it bite?"
This is kind of intense. It reworks all the indexing and searching code so that we're using a single ES doctype in the index. Documents can be distinguished by their 'model' field which indicates which model they derive from. This pulls in a change to elasticutils that adds a get_doctype() to S. Previously the doctype was based on cls._meta.db_table and that made a unified doctype difficult. I added a .search() method to SearchMixin. This is the class method to use to get an S that is tuned for that model. I changed a bunch of tests to use the new .search() rather than building their own S or Sphilastic. Additionally, this does some minor cleanup of related code. One thing this doesn't do is rework how the AAQ suggestions work. That's still a single function that does its thing. Need to keep an eye on that when we make future changes so we make sure it doesn't break.
| 'id': {'type': 'integer'}, | ||
| 'id': {'type': 'long'}, | ||
| 'model': {'type': 'string', 'index': 'not_analyzed', | ||
| 'store': 'yes'}, |
There was a problem hiding this comment.
'model' holds the name of the model. Instead of searching by doctype, we're going to filter on the model. The model value is cls._meta.db_table.
This fixes get_doctype_stats so it doesn't use Model.search() which always looks at the READ_INDEX. We need get_doctype_stats to look at either the READ_INDEX or the WRITE_INDEX.
| @@ -18,7 +18,8 @@ | |||
| from dashboards import LAST_30_DAYS, PERIODS | |||
| from sumo.urlresolvers import reverse | |||
| from sumo.redis_utils import redis_client, RedisError | |||
| from wiki.models import (Document, MEDIUM_SIGNIFICANCE, MAJOR_SIGNIFICANCE, | |||
| from wiki.models import Document | |||
| from wiki.config import (MEDIUM_SIGNIFICANCE, MAJOR_SIGNIFICANCE, | |||
There was a problem hiding this comment.
Oh, thanks for cleaning all these up.
|
All looks good and tests pass. Reindexing is almost done so I am going to go ahead and r+ |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Oh man... This is pretty intense. I tried to touch as many files as I could. The commits are well-contained so if you want to browse it commit-by-commit, that might be easier.
First, this covers bug 729679 which adds the admin search index browser. That was in another pull request, so that's probably fine. Sorry for including it again here, but it would have been tricky to extract.
Second, this covers bug 746609 which reworks all the ES bits so that we use a single doctype for everything and filter on the 'model' field when we want results from a specific bucket. We're still doing bucketed search, but we're now in a good position to write a unified search results view hidden by a waffle flag.
I also cleaned up some code that I accidentally touched and then had to finish cleaning. I added a note to a test that fails periodically. I fixed import statements across the codebase to pull things from wiki.config rather than wiki.models.
r?