Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Already on GitHub? Sign in to your account

Elasticsearch 1.0 prep #1781

Closed
wants to merge 1 commit into
from

Conversation

Projects
None yet
2 participants
Member

robhudson commented Feb 19, 2014

These are changes that prepare us for Elasticsearch 1.0 and also work on < 1.0.

@diox diox and 1 other commented on an outdated diff Feb 20, 2014

mkt/reviewers/tests/test_views_themes.py
@@ -625,16 +625,16 @@ def search(self, q, flagged=False, rereview=False):
return json.loads(themes_search(request).content)['objects']
def test_pending(self):
- eq_(self.search('theme')[0]['id'], self.addon.id)
+ eq_(self.search('theme')[0]['id'], [self.addon.id])
@diox

diox Feb 20, 2014

Member

id is a list ? why ? doesn't it break stuff ?

@robhudson

robhudson Feb 20, 2014

Member

I was too quick to change this test. I didn't realize this was testing the whole view. We shouldn't change the result of the API. I'll fix this at the view level and not change the API.

Elasticsearch 1.0 returns a list whenever we use fields in a query, which is what values_dict does. It's because ES can support both the fields native type and a list of that type so they are removing ambiguity in the result.

@diox diox commented on an outdated diff Feb 20, 2014

media/js/zamboni/themes_review.js
@@ -467,7 +467,7 @@ function buildThemeResultRow(theme, review_url, statuses) {
// Rather resolve URLs in backend, infer from slug.
theme.review_url = review_url.replace(
- '__slug__', theme.slug);
- theme.status = statuses[theme.status];
+ '__slug__', theme.slug[0]);
@diox

diox Feb 20, 2014

Member

Same question as the id below, why are slug and status lists now ?

Member

robhudson commented Feb 20, 2014

Updated.

Member

robhudson commented Feb 26, 2014

re-r? @diox

@diox diox and 1 other commented on an outdated diff Feb 26, 2014

apps/amo/search.py
@@ -310,18 +310,42 @@ def __len__(self):
class DictSearchResults(SearchResults):
def set_objects(self, hits):
- key = 'fields' if self.fields else '_source'
- self.objects = [r[key] for r in hits]
+ if self.fields:
@diox

diox Feb 26, 2014

Member

I'd refactor this to avoid the 2 very similar branches.

@robhudson

robhudson Feb 26, 2014

Member

refactored

@diox diox commented on an outdated diff Feb 26, 2014

apps/amo/search.py
class ListSearchResults(SearchResults):
def set_objects(self, hits):
if self.fields:
- getter = itemgetter(*self.fields)
- objs = [getter(r['fields']) for r in hits]
+ objs = []
+ for hit in hits:
+ objs.append(tuple([v] if type(v) != list else v
+ for v in hit['fields'].values()))
else:
@diox

diox Feb 26, 2014

Member

again very similar I'd refactor that too

Member

diox commented Feb 26, 2014

As discussed on IRC: I really don't like the unnecessary lists all over the place. If we could find a way to avoid them I think it'd be much better.

@robhudson robhudson commented on the diff Feb 26, 2014

apps/editors/views_themes.py
@@ -224,6 +224,10 @@ def themes_search(request):
themes = list(themes.values_dict('name', 'slug', 'status'))
for theme, reviewer in zip(themes, reviewers):
+ # Collapse single value fields from a list.
+ theme['id'] = theme['id'][0]
+ theme['slug'] = theme['slug'][0]
+ theme['status'] = theme['status'][0]
@robhudson

robhudson Feb 26, 2014

Member

Changing this to not use values_dict means the theme becomes an Addon model object. I could probably update how we add the reviewer to this but this change seems pretty harmless here.

@robhudson robhudson commented on an outdated diff Feb 26, 2014

apps/zadmin/views.py
# Get all overrides for this add-on.
- obj['overrides'] = CompatOverride.objects.filter(addon__id=obj['id'])
+ obj['overrides'] = CompatOverride.objects.filter(
+ addon__id=obj['id'][0])
@robhudson

robhudson Feb 26, 2014

Member

I looked at removing values_dict here but I get AttributeError: Manager isn't available; AppCompat is abstract. I'd say this is probably one of the ugliest changes. The AppCompat model isn't a real model and is intended to be used with values_dict so only the _source is returned -- not try to turn it into a django model.

Make S's value_dict Elasticsearch 1.0 compatible
With Elasticsearch 1.0, any time you query with `fields` the values
always come back as lists. This assumes this everywhere and forces it
for Elasticsearch < 1.0 to be consistent.
Member

robhudson commented Feb 26, 2014

Ok. This is cleaned up now.

I'm only coercing to list if values or values_dict specifies a subset of fields. Otherwise I'm returning the _source as it is in the index. This matches what ES will do and is less impactful. It was incorrect before b/c values_dict without fields should just return the _source and not try to convert it to a django model object.

Member

diox commented Mar 3, 2014

r+

Member

robhudson commented Mar 4, 2014

Thanks, merged in 0fab353

@robhudson robhudson closed this Mar 4, 2014

@robhudson robhudson deleted the robhudson:es-1.0-text-to-match branch Jul 14, 2014

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment