Skip to content

Commit

Permalink
Added an experimental Elasticsearch backend.
Browse files Browse the repository at this point in the history
Tests are not yet passing but it works in basic hand-testing. Passing test coverage coming soon.
  • Loading branch information
toastdriven committed Jan 1, 2012
1 parent 6e52a66 commit 6d21579
Show file tree
Hide file tree
Showing 14 changed files with 2,358 additions and 16 deletions.
7 changes: 4 additions & 3 deletions README.rst
Expand Up @@ -3,19 +3,20 @@ Haystack
========

:author: Daniel Lindsley
:date: 2011/09/18
:date: 2011/12/31

Haystack provides modular search for Django. It features a unified, familiar
API that allows you to plug in different search backends (such as Solr_,
Whoosh_, Xapian_, etc.) without having to modify your code.
Elasticsearch_, Whoosh_, Xapian_, etc.) without having to modify your code.

.. _Solr: http://lucene.apache.org/solr/
.. _Elasticsearch: http://elasticsearch.org/
.. _Whoosh: http://whoosh.ca/
.. _Xapian: http://xapian.org/

Haystack is BSD licensed, plays nicely with third-party app without needing to
modify the source and supports advanced features like faceting, More Like This,
highlighting and spelling suggestions.
highlighting, spatial search and spelling suggestions.

You can find more information at http://haystacksearch.org/.

Expand Down
40 changes: 30 additions & 10 deletions docs/backend_support.rst
Expand Up @@ -9,10 +9,12 @@ Supported Backends
==================

* Solr_
* Elasticsearch_
* Whoosh_
* Xapian_

.. _Solr: http://lucene.apache.org/solr/
.. _Elasticsearch: http://elasticsearch.org/
.. _Whoosh: http://whoosh.ca/
.. _Xapian: http://xapian.org/

Expand All @@ -32,7 +34,23 @@ Solr
* Faceting
* Stored (non-indexed) fields
* Highlighting
* Requires: pysolr (2.0.13+) & Solr 1.3+
* Spatial search
* Requires: pysolr (2.0.13+) & Solr 3.5+

Elasticsearch
----

**Complete & included with Haystack.**

* Full SearchQuerySet support
* Automatic query building
* "More Like This" functionality
* Term Boosting
* Faceting
* Stored (non-indexed) fields
* Highlighting
* Spatial search
* Requires: pyelasticsearch (toastdriven git master) & Elasticsearch 0.17.7+

Whoosh
------
Expand Down Expand Up @@ -62,15 +80,17 @@ Xapian
* Backend can be downloaded here: `xapian-haystack <http://github.com/notanumber/xapian-haystack/>`_


+----------------+------------------------+---------------------+----------------+------------+----------+---------------+--------------+
| Backend | SearchQuerySet Support | Auto Query Building | More Like This | Term Boost | Faceting | Stored Fields | Highlighting |
+================+========================+=====================+================+============+==========+===============+==============+
| Solr | Yes | Yes | Yes | Yes | Yes | Yes | Yes |
+----------------+------------------------+---------------------+----------------+------------+----------+---------------+--------------+
| Whoosh | Yes | Yes | No | Yes | No | Yes | Yes |
+----------------+------------------------+---------------------+----------------+------------+----------+---------------+--------------+
| Xapian | Yes | Yes | Yes | Yes | Yes | Yes | Yes (plugin) |
+----------------+------------------------+---------------------+----------------+------------+----------+---------------+--------------+
+----------------+------------------------+---------------------+----------------+------------+----------+---------------+--------------+---------+
| Backend | SearchQuerySet Support | Auto Query Building | More Like This | Term Boost | Faceting | Stored Fields | Highlighting | Spatial |
+================+========================+=====================+================+============+==========+===============+==============+=========+
| Solr | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes |
+----------------+------------------------+---------------------+----------------+------------+----------+---------------+--------------+---------+
| Solr | Yes | Yes | Yes | Yes | Yes | Yes | Yes | Yes |

This comment has been minimized.

Copy link
@robhudson

robhudson Jan 1, 2012

Contributor

I think one of these should say "Elasticsearch".

+----------------+------------------------+---------------------+----------------+------------+----------+---------------+--------------+---------+
| Whoosh | Yes | Yes | No | Yes | No | Yes | Yes | No |
+----------------+------------------------+---------------------+----------------+------------+----------+---------------+--------------+---------+
| Xapian | Yes | Yes | Yes | Yes | Yes | Yes | Yes (plugin) | No |
+----------------+------------------------+---------------------+----------------+------------+----------+---------------+--------------+---------+


Wishlist
Expand Down
56 changes: 55 additions & 1 deletion docs/installing_search_engines.rst
Expand Up @@ -9,7 +9,7 @@ Solr

Official Download Location: http://www.apache.org/dyn/closer.cgi/lucene/solr/

Solr is Java but comes in a pre=packaged form that requires very little other
Solr is Java but comes in a pre-packaged form that requires very little other
than the JRE and Jetty. It's very performant and has an advanced featureset.
Haystack suggests using Solr 3.5+, though it's possible to get it working on
Solr 1.4 with a little effort. Installation is relatively simple::
Expand Down Expand Up @@ -99,6 +99,60 @@ your ``SearchIndex`` classes (in this case, assuming the main field is called
``text``).


Elasticsearch
=============

Official Download Location: http://www.elasticsearch.org/download/

Elasticsearch is Java but comes in a pre-packaged form that requires very
little other than the JRE. It's also very performant, scales easily and has
an advanced featureset. Haystack requires at least version 0.17.7 (0.18.6 is
current as of writing). Installation is best done using a package manager::

# On Mac OS X...
brew install elasticsearch

# On Ubuntu...
apt-get install elasticsearch

# Then start via:
elasticsearch -f -D es.config=<path to YAML config>

# Example:
elasticsearch -f -D es.config=/usr/local/Cellar/elasticsearch/0.17.7/config/elasticsearch.yml

You may have to alter the configuration to run on ``localhost`` when developing
locally. Modifications should be done in a YAML file, the stock one being
``config/elasticsearch.yml``::

# Unicast Discovery (disable multicast)
discovery.zen.ping.multicast.enabled: false
discovery.zen.ping.unicast.hosts: ["127.0.0.1"]

# Name your cluster here to whatever.
# My machine is called "Venus", so...
cluster:
name: venus

network:
host: 127.0.0.1

path:
logs: /usr/local/var/log
data: /usr/local/var/data

You'll also need an Elasticsearch binding, ``pyelasticsearch`` (**NOT ``pyes``).
The unofficial, Haystack-compatible ``pyelasticsearch`` package,
hosted on GitHub_, is the best version to use. Place ``pyelasticsearch.py``
somewhere on your ``PYTHONPATH`` (usually ``python setup.py install``).
.. note::

``pyelasticsearch`` has it's own dependencies that aren't covered by
Haystack. You'll also need ``requests`` & ``simplejson`` for speedier
JSON construction/parsing.


Whoosh
======

Expand Down
6 changes: 4 additions & 2 deletions docs/spatial.rst
Expand Up @@ -39,7 +39,8 @@ For Solr_, you'll need at least **v3.5+**. In addition, if you have an existing
install of Haystack & Solr, you'll need to upgrade the schema & reindex your
data. If you're adding geospatial data, you would have to reindex anyhow.

For Elasticsearch, you'll need...
For Elasticsearch, you'll need at least v0.17.7, preferably v0.18.6 or better.
If you're adding geospatial data, you'll have to reindex as well.

.. _Solr: http://lucene.apache.org/solr/

Expand All @@ -50,12 +51,13 @@ Lookup Type Solr Elasticsearch Whoosh Xapian Simple
`dwithin` X X
`distance` X X
`order_by('distance')` X X
`polygon` X
`polygon` X
====================== ====== =============== ======== ======== ======

For more details, you can inspect http://wiki.apache.org/solr/SpatialSearch
or http://www.elasticsearch.org/guide/reference/query-dsl/geo-bounding-box-filter.html.


Geospatial Assumptions
======================

Expand Down
21 changes: 21 additions & 0 deletions docs/tutorial.rst
Expand Up @@ -108,6 +108,20 @@ Example::
}


Elasticsearch
~~~~~~~~~~~~~

Example::

HAYSTACK_CONNECTIONS = {
'default': {
'ENGINE': 'haystack.backends.elasticsearch_backend.ElasticsearchSearchEngine',
'URL': 'http://127.0.0.1:9200/',
'INDEX_NAME': 'haystack',
},
}


Whoosh
~~~~~~

Expand Down Expand Up @@ -154,6 +168,13 @@ Simple
The ``simple`` backend using very basic matching via the database itself. It's
not recommended for production use but it will return results.

.. warning::

This backend does *NOT* work like the other backends do. Data preparation
does nothing & advanced filtering calls do not work. You really probably
don't want this unless you're in an environment where you just want to
silence Haystack.

Example::

HAYSTACK_CONNECTIONS = {
Expand Down

0 comments on commit 6d21579

Please sign in to comment.