Skip to content


Subversion checkout URL

You can clone with
Download ZIP
Branch: master
Fetching contributors…

Cannot retrieve contributors at this time

434 lines (264 sloc) 12.3 KB
What's new in ElasticUtils
.. contents::
Version 0.7: in development
.. Note::
This is a *big* change. We switched from pyes to pyelasticsearch. In
doing that, we changed a handful of signatures, nixed some
functionality that didn't make any sense any more, and cleaned a
bunch of things up.
If this terrifies you, read through these notes carefully and/or
stay with v0.6.
**API-breaking changes:**
* **pyelasticsearch v0.3 or later now required.**
ElasticUtils now requires pyelasticsearch v0.3 or later and its
* **elasticutils.PYES_VERSION is removed.**
Since we're not using pyes, we removed `elasticutils.PYES_VERSION`.
* **ElasticUtils no longer supports thrift.**
Pretty sure we did a lousy job of supporting it before---it was all
in the pyes code and we had no tests for it.
* **get_es() signatures have changed.**
* takes urls now instead of hosts
* dump_curl argument is now gone
* default_indexes argument is gone
The arguments correspond with pyelasticsearch `ElasticSearch`
ElasticUtils uses HTTP urls for connecting to ElasticSearch now.
Previously, you'd do::
get_es(hosts=['localhost:9200']) # Old way
Now you do::
get_es(urls=['http://localhost:9200']) # New way
The dump_curl argument was helpful for debugging, but we don't
really need it anymore. See the :ref:`debugging-chapter` for better
debugging methods.
Will now raise a `DeprecationWarning` if you pass in `hosts`
* **S searches all indexes and doctypes by default.**
Previously, if you did::
it'd search an index named "default" for doctypes "document". That
was dumb. Now it searches all indexes and all doctypes by default.
* **S.es_builder is gone.**
``es_builder()`` was there to get around problems with pyes' ES
class. The pyelasticsearch `ElasticSearch` class is more
straightforward, so we don't need to do circus shenanigans.
You can probably do what you need to with either the ``es()``
transform or by subclassing `S` and overriding the ``get_es()``
* **MLT arguments changed.**
The `fields` argument in the constructor was renamed to `mlt_fields`
to be in line with ElasticSearch API names.
Will now raise a `DeprecationWarning` if you pass in `fields`
* **Django: changed settings.**
Changed ES_HOSTS setting to ES_URLS. This is both a name and a value
change. ES_URLS takes a list of strings each is an http url. You'll
neex to update your settings files from::
ES_HOSTS = ['localhost:9200'] # Old way
ES_URLS = ['http://localhost:9200'] # New way
ES_DUMP_CURL is gone.
* **Django: removed the statsd code.**
* **Django: ESTestCase was renamed to ElasticTestCase.**
* **Django: Indexable.index() method no longer has bulk argument.**
The `Indexable.index()` method no longer does bulk indexing. The
way pyes did this was kind of squirrely and caused issues if you
didn't have the order of operations correct.
Now `Indexable.index()` only indexes a single document.
But wait...
* **Django: Indexable now has bulk_index() .**
pyes would keep track of all the things you wanted to bulk index
and then at some point push them all. Instead of doing it under the
hood, we added a separate `bulk_index()` method and now you control
how many items get indexed in bulk in one pass.
* **Django: Indexable.refresh_index no longer takes a timeout argument.**
pyelasticsearch `ElasticSearch.refresh` doesn't take a timesleep
argument, so we don't need that anymore.
* **Django: Indexable `es` argument defaults to Indexable.get_es() now.**
Previously it defaulted to `elasticsearch.contrib.django.get_es()`. Now
it defaults to `Indexable.get_es()` class method making it more flexible.
* **pyes -> pyelasticsearch changes.**
If you called ``.get_es()`` and got a pyes `ES` object and did
things with that (create index, create mappings, delete indexes,
indexing, cluster health, ...), you're going to need to make
some changes.
You can either:
1. rewrite that code to use pyelasticsearch `ElasticSearch`
equivalents, or
2. write and use your own ``get_es()`` function that returns
a pyes `ES` object
Rewriting shouldn't be too hard. The `pyelasticsearch documentation
<>`_ is pretty good
and for most things, there's a 1-to-1 translation.
* **pyes is no longer a requirement.**
We no longer use pyes so you can remove it from your requirements.
* **Django: es_required_or_50x handles different exceptions.**
Previously it handled:
* pyes.urllib3.MaxRetryError
* pyes.exceptions.IndexMissingException
* pyes.exceptions.ElasticSearchException
We're not using pyes anymore, so now it handles:
* pyelasticsearch.exceptions.ConnectionError
* pyelasticsearch.exceptions.ElasticHttpNotFoundError
* pyelasticsearch.exceptions.Timeout
You probably don't need to do anything about this, but it's good to
* **Django: celery tasks rewritten.**
The celery tasks were rewritten, docs were updated, and tests were
added so they work now.
* **elasticutils.PYELASTICSEARCH_VERSION is added.**
You can see which version of pyelasticsearch is in use by doing::
from elasticutils import PYELASTICSEARCH_VERSION
* **S.execute added**
This allows you to explicitly execute a search and get back a
`SearchResults` instance.
* **S.all added**
Allows you to get **all** the search results possible. This is
dangerous, so it's been documented with lots of warnings.
Version 0.6: Released January 17th, 2013
**API-breaking changes:**
* **S.values_dict no longer always includes id.**
``values_dict`` no longer always includes an 'id' field in the
fields list if you don't specify it.
Specifying no fields now returns **all** fields::
Specifying fields now returns only those fields::
S().values_dict('name', 'number')
* **S.values_list no longer always includes id.**
``values_list`` no longer always includes an 'id' field in the
fields list if you don't specify it.
Specifying no fields now returns **all** fields::
Specifying fields now returns data for those fields in the order
the fields are specified::
S().values_list('name', 'number')
* **Types have changed.**
This is a big change.
Up through ElasticUtils v0.5, `S` could take a type and that type was
a model. This is now completely different.
In ElasticUtils v0.6 and later, `S` takes a `MappingType`. A
`MappingType` can be related to a model, but it itself should not be
a model. This allows us to return search results as a list of
`MappingType` instances which can do things rather than forcing you
to do a db hit to get back instances that can do things.
This is similar to how django-haystack works with the SearchIndex
class, except ElasticUtils doesn't yet support declarative mapping
See documentation for more details.
* **By default, results are now DefaultMappingType.**
In ElasticUtils v0.4 and v0.5, if the `S` was untyped and you didn't
specify either ``values_dict`` or ``values_list``, then the results
would come back as a list of dicts.
In ElasticUtils v0.5, if the `S` is untyped and you didn't specify
either ``values_dict`` or ``values_list``, then the results would
come back as a list of `DefaultMappingType`.
See documentation for more details.
* **elasticutils.contrib.django.models.SearchMixin is no more.**
The `SearchMixin` class is replaced by `DjangoMappingType` which
relates ElasticSearch mapping types to Django ORM models and
`Indexable` which is a mixin that adds a bunch of index-related
* Added ``_source`` and ``_id`` to the metadata decorated on the
search results.
See documentation for more details.
* Fixed ``elasticutils.contrib.django.es_required_or_50x``.
It works better now.
* prefix filter support.
ElasticUtils supports prefix filters. You can do this now::
Version 0.5: Released September 4th, 2012
**API-breaking changes:**
* Added ``demote`` transform: it adds boosting query support allowing
you to do a negative query which reduces scores for documents that
* The elasticutils version is now available in ``elasticutils.__version__``
as well as ``elasticutils._version.__version__``.
* Added ``__in`` support for queries. Doing::
S().query(foo__in=['a', 'b', 'c'])
does a terms query now.
* Added `MLT` class which does morelikethis.
* Added API documentation for S, an index, ``order_by`` docs, fixed some
icky bugs, and generally improved everything at least a little bit.
Version 0.4: Released July 31st, 2012
**API-breaking changes:**
* **ElasticUtils no longer requires Django.**
If you're using Django, you should change your import statements
from things like::
from elasticutils import get_es, S, F
from elasticutils.contrib.django import get_es, S, F
Further, Django helper modules like ``cron``, ``tasks``, and
``models`` were all moved to ``elasticutils.contrib.django``.
We moved `ESTestCase` from ``elasticutils.tests`` to
If you don't use Django, ElasticUtils is easier to use!
* **S no longer requires a type.**
If you're not using Django, `S` no longer requires a type. If you
don't specify a type, then ElasticUtils will return results as
* **Values and values_list changed.**
``values()`` was renamed to ``values_list()``.
``values_list()`` (was ``values()``) now always returns a list of
tuples even if you only requested a single field. Previously, doing
something like::
searcher = S().values_list('id')
would return something like::
[1, 2, 3, 4, 5]
Now it returns::
[(1,), (2,), (3,), (4,), (5,)]
* **Facet functionality was rewritten.**
Changed ``.facet()`` to be arg-driven and allow for `filtered` and
`global_` flags.
Changed ``.facets()`` to ``.facet_counts()`` to match Django
Added ``.facet_raw()`` which allows you to do more complicated
facets including scripting. This is similar to the original
``.facet()`` implementation.
* Overhauled and cleaned up ElasticUtils tests. Running tests can be done
DJANGO_SETTINGS_MODULE=es_settings nosetests
* Default timeout was changed from 1 second to 5 seconds.
* Added ``es`` transform: it allows you to specify the settings with which
to create an ES when the search is executed.
* Added ``es_builder`` transform: it allows you to specify a function
that builds an ES which will be executed to create an ES when the
search is executed.
* Added ``indexes`` transform: it allows you to specify the indexes to
use for the search.
* Added ``doctypes`` transform: it allows you to specify the doctypes
to use for the search.
* Added ``explain`` transform: it allows you to set the "explain" flag
which gives you an explanation of how the score was calculated.
I also added ``elasticutils.utils.format_elasticutils`` which formats
the resulting explanation text into something slightly more
readable. But it's likely this will change in the future.
* Added ``boost`` transform: it allows you to do query-time field
* Added support for ``prefix``. It's the same as ``startswith``, but
it uses the same word that ElasticSearch uses. At some point, we'll
remove support for ``startswith``.
* Added support for ``text_phrase`` and ``query_string`` queries.
* Added ``highlight`` transform: generates highlighted fragments of
content that matched the query.
* Removed requirement for nuggets.
* Continued to improve documentation.
Version 0.3: Released June 1st, 2012
* Add documentation for debugging, project details and other things.
* Minor project cleanup to make it easier to maintain and use
* Make ``get_es()`` more useful. It now takes overrides that allow you to
configure multiple kinds of ES objects for different purposes.
Jump to Line
Something went wrong with that request. Please try again.