first pass with some ES #1

Merged
merged 3 commits into from Dec 28, 2012

Conversation

Projects
None yet
2 participants
Owner

andymckay commented Dec 28, 2012

No description provided.

andymckay added a commit that referenced this pull request Dec 28, 2012

@andymckay andymckay merged commit dfc798d into mozilla:master Dec 28, 2012

+ 'date': today - timedelta(days=x)
+ }
+ obj = Metric.objects.create(**data)
+ obj.index()
@robhudson

robhudson Dec 29, 2012

Member

Do you want a es.refresh() here after you've indexed?

@andymckay

andymckay Dec 29, 2012

Owner

maybe, its going to happen automagically, the refresh just forces it to happen faster right?

@robhudson

robhudson Dec 29, 2012

Member

Yeah. ES docs say "periodically".

+ query = json.loads(request.body)
+ result = es.search(query, 'monolith')
+ return [Hit(**data.get('_source', data.get('fields', {})))
+ for data in result['hits']['hits']]
@robhudson

robhudson Dec 29, 2012

Member

Thought elasticutils had something like this already. Or you're using raw pyes? E.g. I thought it had a .result_dict or something.

@andymckay

andymckay Dec 29, 2012

Owner

this is elasticutils, will look

+ fixtures = ['metrics.json']
+
+ def test_metrics(self):
+ eq_(len(Metric.objects.filter(name='addons.downloads.count')), 1)
@robhudson

robhudson Dec 29, 2012

Member

Would Metric.objects.filter(name='addons.downloads.count').count() be faster? Maybe not since it's only a single record but it's more to the point to me than doing len.

@andymckay

andymckay Dec 29, 2012

Owner

yeah prolly, but not if it goes over 80 chars..

+ def test_delete(self, delete):
+ self.api_client.delete(self.url)
+ assert delete.called
+ assert Metric.objects.filter(pk=1).exists()
@robhudson

robhudson Dec 29, 2012

Member

Why does it exists() if you deleted it?

@andymckay

andymckay Dec 29, 2012

Owner

because i've deleted the document in ES, not the record in MySQL, that's what the test is checking

+ eq_(result['date'], None)
+ eq_(result['value'], 2681)
+
+
@robhudson

robhudson Dec 29, 2012

Member

Extra lines

+ {u'_score': 1.0, u'_source':
+ {u'date': u'2012-12-27', u'key': u'',
+ u'name': u'some.name', u'value': 2681}, u'_index':
+ u'monolith'}]}}
@robhudson

robhudson Dec 29, 2012

Member

That's some ugly wrapping. Why is key empty?

@andymckay

andymckay Dec 29, 2012

Owner

just some sample data, key is optional for some records

+ super(TestESResource, self).setUp()
+ self.api_client = MetricsApiClient()
+
+ def test_get_no_json(self, search):
@robhudson

robhudson Dec 29, 2012

Member

not quite seeing where this is mocked so that it returns an empty list?

@andymckay

andymckay Dec 29, 2012

Owner

mock will return an empty iterator by default, the class is mocked

+ KEY `metric_45544485` (`key`),
+ KEY `metric_52094d6e` (`name`),
+ KEY `metric_40858fbd` (`value`)
+) ENGINE=InnoDB AUTO_INCREMENT=1003 DEFAULT CHARSET=utf8;
@robhudson

robhudson Dec 29, 2012

Member

Why AUTO_INCREMENT=1003?

@andymckay

andymckay Dec 29, 2012

Owner

thats the result of a dump, let me clean up

@@ -0,0 +1,12 @@
+CREATE TABLE `metric` (
+ `id` int(11) NOT NULL AUTO_INCREMENT,
@robhudson

robhudson Dec 29, 2012

Member

unsigned?

+ def obj_get_list(self, request, **kwargs):
+ es = get_es()
+ query = json.loads(request.body)
+ result = es.search(query, 'monolith')
@robhudson

robhudson Dec 29, 2012

Member

Do the max_limit and limit filters happen automatically somewhere?

@andymckay

andymckay Dec 29, 2012

Owner

you send through a size limit in the JSON, that's the amount ES will return, but tastypie kindly reformats the data applying the Meta.limits for you. rather than futzing i pushed tastypie up to a nice limit, in later versions we should do smart things with the data (eg applying size from the limit)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment