Test refactor - merge all the tests into one test suite (closes #951)

Major refactor by @honzakral which stabilized the test suite, makes it easier to run and add new tests and
somewhat faster, too.

* Merged all the tests
* Mark tests as skipped when a backend is not available (e.g. no ElasticSearch or Solr connection)
* Massively simplified test runner (``python setup.py test``)

Minor updates:
* Travis:
    - Test Python 3.4
    - Use Solr 4.6.1
* Simplified legacy test code which can now be replaced by the test utilities in newer versions of Django
* Update ElasticSearch client & tests for ES 1.0+
* Add option for SearchModelAdmin to specify the haystack connection to use
* Fixed a bug with RelatedSearchQuerySet caching using multiple instances (429d234)

.gitignore

@@ -10,6 +10,8 @@ MANIFEST
 .tox
 env
 env3
+*.egg
+.coverage
 
-# Because I'm ghetto like that.
-tests/pyelasticsearch.py
+test_haystack/solr_tests/server/solr-4.6.0.tgz
+test_haystack/solr_tests/server/solr4/

.travis.yml

@@ -4,8 +4,8 @@ python:
 
 before_install:
   - sudo apt-get update
-  - sudo apt-get install wajig binutils gdal-bin libproj-dev libgeos-c1 libxapian22 python-xapian default-jdk
-  - curl https://raw.github.com/moliware/travis-solr/master/travis-solr.sh | SOLR_CONFS="tests/schema.xml tests/solrconfig.xml" SOLR_VERSION=4.4.0 bash
+  - sudo apt-get install wajig binutils gdal-bin libproj-dev libgeos-c1 libgdal1h libxapian22 python-xapian default-jdk
+  - curl https://raw.github.com/moliware/travis-solr/master/travis-solr.sh | SOLR_CONFS="test_haystack/solr_tests/server/schema.xml test_haystack/solr_tests/server/solrconfig.xml" SOLR_VERSION=4.6.1 bash
 
 install:
   - pip install tox

docs/running_tests.rst

@@ -4,81 +4,61 @@
 Running Tests
 =============
 
-Dependencies
-============
-
 Everything
-----------
+==========
 
 The simplest way to get up and running with Haystack's tests is to run::
 
-    pip install -r tests/requirements.txt
+    python setup.py test
 
 This installs all of the backend libraries & all dependencies for getting the
-tests going. You will still have to setup search servers (for running Solr
-tests, the spatial Solr tests & the Elasticsearch tests).
+tests going and runs the tests. You will still have to setup search servers
+(for running Solr tests, the spatial Solr tests & the Elasticsearch tests).
 
 
 Cherry-Picked
--------------
-
-If you'd rather not run all the tests, install only the backends you need.
-Additionally, ``Haystack`` uses the Mock_ library for testing. You will need
-to install it before running the tests::
-
-    pip install mock
-
-.. _Mock: http://pypi.python.org/pypi/mock
-
-
-Core Haystack Functionality
-===========================
-
-In order to test Haystack with the minimum amount of unnecessary mocking and to
-stay as close to real-world use as possible, ``Haystack`` ships with a test
-app (called ``core``) within the ``django-haystack/tests`` directory.
-
-In the event you need to run ``Haystack``'s tests (such as testing
-bugfixes/modifications), here are the steps to getting them running::
+=============
 
-    cd django-haystack/tests
-    export PYTHONPATH=`pwd`/..:`pwd`
-    django-admin.py test core --settings=settings
+If you'd rather not run all the tests, run only the backends you need since
+tests for backends that are not running will be skipped.
 
 ``Haystack`` is maintained with all tests passing at all times, so if you
 receive any errors during testing, please check your setup and file a report if
 the errors persist.
 
-Backends
-========
+To run just a portion of the tests you can use the script ``run_tests.py`` and
+just specify the files or directories you wish to run, for example::
 
-If you want to test a backend, the steps are the same with the exception of
-the settings module and the app to test. To test an engine, use the
-``engine_settings`` module within the ``tests`` directory, substituting the
-``engine`` for the name of the proper backend. You'll also need to specify the
-app for that engine. For instance, to run the Solr backend's tests::
+    cd test_haystack
+    ./run_tests.py whoosh_tests test_loading.py
 
-    cd django-haystack/tests
-    export PYTHONPATH=`pwd`/..:`pwd`
-    django-admin.py test solr_tests --settings=solr_settings
+The ``run_tests.py`` script is just a tiny wrapper around the nose_ library and
+any options you pass to it will be passed on; including ``--help`` to get a
+list of possible options::
 
-Or, to run the Whoosh backend's tests::
+    cd test_haystack
+    ./run_tests.py --help
 
-    cd django-haystack/tests
-    export PYTHONPATH=`pwd`/..:`pwd`
-    django-admin.py test whoosh_tests --settings=whoosh_settings
+.. _nose: https://nose.readthedocs.org/en/latest/
 
-Or, to run the Elasticsearch backend's tests::
-
-    cd django-haystack/tests
-    export PYTHONPATH=`pwd`/..:`pwd`
-    django-admin.py test elasticsearch_tests --settings=elasticsearch_settings
 
 Configuring Solr
-----------------
+================
+
+Haystack assumes that you have a Solr server running on port ``9001`` which
+uses the schema and configuration provided in the
+``test_haystack/solr_tests/server/`` directory. For convenience, a script is
+provided which will download, configure and start a test Solr server::
+
+    test_haystack/solr_tests/server/start-solr-test-server.sh
+
+If no server is found all solr-related tests will be skipped.
+
+Configuring Elasticsearch
+=========================
 
-Haystack assumes that you have a Solr server running on port ``9001`` which uses the schema and
-configuration provided in the ``tests/`` directory. For convenience, a script is provided which
-will download, configure and start a test Solr server::
+The test suite will try to connect to Elasticsearch on port ``9200``. If no
+server is found all elasticsearch tests will be skipped. Note that the tests
+are destructive - during the teardown phase they will wipe the cluster clean so
+make sure you don't run them against an instance with data you wish to keep.
 
-    tests/start-solr-test-server.sh

haystack/admin.py

@@ -31,17 +31,21 @@ def list_max_show_all(changelist):
 
 
 class SearchChangeList(ChangeList):
+    def __init__(self, **kwargs):
+        self.haystack_connection = kwargs.pop('haystack_connection', 'default')
+        super(SearchChangeList, self).__init__(**kwargs)
+
     def get_results(self, request):
         if not SEARCH_VAR in request.GET:
             return super(SearchChangeList, self).get_results(request)
 
         # Note that pagination is 0-based, not 1-based.
-        sqs = SearchQuerySet().models(self.model).auto_query(request.GET[SEARCH_VAR]).load_all()
+        sqs = SearchQuerySet(self.haystack_connection).models(self.model).auto_query(request.GET[SEARCH_VAR]).load_all()
 
         paginator = Paginator(sqs, self.list_per_page)
         # Get the number of objects, with admin filters applied.
         result_count = paginator.count
-        full_result_count = SearchQuerySet().models(self.model).all().count()
+        full_result_count = SearchQuerySet(self.haystack_connection).models(self.model).all().count()
 
         can_show_all = result_count <= list_max_show_all(self)
         multi_page = result_count > self.list_per_page
@@ -64,6 +68,9 @@ def get_results(self, request):
 
 
 class SearchModelAdmin(ModelAdmin):
+    # haystack connection to use for searching
+    haystack_connection = 'default'
+
     @csrf_protect_m
     def changelist_view(self, request, extra_context=None):
         if not self.has_change_permission(request, None):
@@ -75,7 +82,7 @@ def changelist_view(self, request, extra_context=None):
 
         # Do a search of just this model and populate a Changelist with the
         # returned bits.
-        if not self.model in connections['default'].get_unified_index().get_indexed_models():
+        if not self.model in connections[self.haystack_connection].get_unified_index().get_indexed_models():
             # Oops. That model isn't being indexed. Return the usual
             # behavior instead.
             return super(SearchModelAdmin, self).changelist_view(request, extra_context)
@@ -85,6 +92,7 @@ def changelist_view(self, request, extra_context=None):
         list_display = list(self.list_display)
 
         kwargs = {
+            'haystack_connection': self.haystack_connection,
             'request': request,
             'model': self.model,
             'list_display': list_display,

haystack/backends/simple_backend.py

@@ -2,9 +2,13 @@
 A very basic, ORM-based backend for simple search during tests.
 """
 from __future__ import unicode_literals
+
+from warnings import warn
+
 from django.conf import settings
 from django.db.models import Q
 from django.utils import six
+
 from haystack import connections
 from haystack.backends import BaseEngine, BaseSearchBackend, BaseSearchQuery, SearchNode, log_query
 from haystack.inputs import PythonData
@@ -32,16 +36,13 @@ def emit(self, record):
 
 class SimpleSearchBackend(BaseSearchBackend):
     def update(self, indexer, iterable, commit=True):
-        if logger is not None:
-            logger.warning('update is not implemented in this backend')
+        warn('update is not implemented in this backend')
 
     def remove(self, obj, commit=True):
-        if logger is not None:
-            logger.warning('remove is not implemented in this backend')
+        warn('remove is not implemented in this backend')
 
     def clear(self, models=[], commit=True):
-        if logger is not None:
-            logger.warning('clear is not implemented in this backend')
+        warn('clear is not implemented in this backend')
 
     @log_query
     def search(self, query_string, **kwargs):
@@ -73,13 +74,14 @@ def search(self, query_string, **kwargs):
 
                             queries.append(Q(**{'%s__icontains' % field.name: term}))
 
-                        qs = model.objects.filter(six.moves.reduce(lambda x, y: x|y, queries))
+                        qs = model.objects.filter(six.moves.reduce(lambda x, y: x | y, queries))
 
                 hits += len(qs)
 
                 for match in qs:
                     match.__dict__.pop('score', None)
-                    result = result_class(match._meta.app_label, match._meta.module_name, match.pk, 0, **match.__dict__)
+                    result = result_class(match._meta.app_label, match._meta.module_name, match.pk, 0,
+                                          **match.__dict__)
                     # For efficiency.
                     result._model = match.__class__
                     result._object = match

haystack/backends/whoosh_backend.py

@@ -44,6 +44,7 @@
 from whoosh.filedb.filestore import FileStorage, RamStorage
 from whoosh.searching import ResultsPage
 from whoosh.writing import AsyncWriter
+from whoosh.highlight import HtmlFormatter, highlight as whoosh_highlight, ContextFragmenter
 
 # Handle minimum requirement.
 if not hasattr(whoosh, '__version__') or whoosh.__version__ < (2, 5, 0):
@@ -55,6 +56,15 @@
 LOCALS.RAM_STORE = None
 
 
+class WhooshHtmlFormatter(HtmlFormatter):
+    """
+    This is a HtmlFormatter simpler than the whoosh.HtmlFormatter.
+    We use it to have consistent results across backends. Specifically,
+    Solr, Xapian and Elasticsearch are using this formatting.
+    """
+    template = '<%(tag)s>%(t)s</%(tag)s>'
+
+
 class WhooshSearchBackend(BaseSearchBackend):
     # Word reserved by Whoosh for special use.
     RESERVED_WORDS = (
@@ -613,13 +623,19 @@ def _process_results(self, raw_page, highlight=False, query_string='', spelling_
                 del(additional_fields[DJANGO_ID])
 
                 if highlight:
-                    from whoosh import analysis
-                    from whoosh.highlight import highlight, ContextFragmenter, UppercaseFormatter
-                    sa = analysis.StemmingAnalyzer()
-                    terms = [term.replace('*', '') for term in query_string.split()]
-
+                    sa = StemmingAnalyzer()
+                    formatter = WhooshHtmlFormatter('em')
+                    terms = [token.text for token in sa(query_string)]
+
+                    whoosh_result = whoosh_highlight(
+                        additional_fields.get(self.content_field_name),
+                        terms,
+                        sa,
+                        ContextFragmenter(),
+                        formatter
+                    )
                     additional_fields['highlighted'] = {
-                        self.content_field_name: [highlight(additional_fields.get(self.content_field_name), terms, sa, ContextFragmenter(terms), UppercaseFormatter())],
+                        self.content_field_name: [whoosh_result],
                     }
 
                 result = result_class(app_label, model_name, raw_result[DJANGO_ID], score, **additional_fields)

haystack/management/commands/build_solr_schema.py

@@ -7,15 +7,15 @@
 from django.core.management.base import BaseCommand
 from django.template import loader, Context
 from haystack.backends.solr_backend import SolrSearchBackend
-from haystack.constants import ID, DJANGO_CT, DJANGO_ID, DEFAULT_OPERATOR, DEFAULT_ALIAS
+from haystack import constants
 
 
 class Command(BaseCommand):
     help = "Generates a Solr schema that reflects the indexes."
     base_options = (
         make_option("-f", "--filename", action="store", type="string", dest="filename",
                     help='If provided, directs output to a file instead of stdout.'),
-        make_option("-u", "--using", action="store", type="string", dest="using", default=DEFAULT_ALIAS,
+        make_option("-u", "--using", action="store", type="string", dest="using", default=constants.DEFAULT_ALIAS,
                     help='If provided, chooses a connection to work with.'),
     )
     option_list = BaseCommand.option_list + base_options
@@ -41,10 +41,10 @@ def build_context(self, using):
         return Context({
             'content_field_name': content_field_name,
             'fields': fields,
-            'default_operator': DEFAULT_OPERATOR,
-            'ID': ID,
-            'DJANGO_CT': DJANGO_CT,
-            'DJANGO_ID': DJANGO_ID,
+            'default_operator': constants.DEFAULT_OPERATOR,
+            'ID': constants.ID,
+            'DJANGO_CT': constants.DJANGO_CT,
+            'DJANGO_ID': constants.DJANGO_ID,
         })
 
     def build_template(self, using):

setup.py

@@ -7,6 +7,23 @@
     use_setuptools()
     from setuptools import setup
 
+install_requires = [
+    'Django',
+]
+
+tests_require = [
+    'elasticsearch==0.4.5',
+    'pysolr>=3.2.0',
+    'whoosh==2.5.4',
+    'lxml==3.2.3',
+    'python-dateutil',
+    'geopy==0.95.1',
+    'httplib2==0.8',
+
+    'nose',
+    'mock',
+    'coverage',
+]
 
 setup(
     name='django-haystack',
@@ -43,4 +60,7 @@
         'Topic :: Utilities',
     ],
     zip_safe=False,
+    install_requires=install_requires,
+    tests_require=tests_require,
+    test_suite="test_haystack.run_tests.run_all",
 )

test_haystack/__init__.py

@@ -0,0 +1,21 @@
+import os
+test_runner = None
+old_config = None
+
+os.environ['DJANGO_SETTINGS_MODULE'] = 'test_haystack.settings'
+
+
+def setup():
+    global test_runner
+    global old_config
+    from django.test.simple import DjangoTestSuiteRunner
+    test_runner = DjangoTestSuiteRunner()
+    test_runner.setup_test_environment()
+    old_config = test_runner.setup_databases()
+
+
+def teardown():
+    test_runner.teardown_databases(old_config)
+    test_runner.teardown_test_environment()
+
+

tests/solr_tests/admin.py → test_haystack/core/admin.py

@@ -1,9 +1,10 @@
 from django.contrib import admin
 from haystack.admin import SearchModelAdmin
-from core.models import MockModel
+from .models import MockModel
 
 
 class MockModelAdmin(SearchModelAdmin):
+    haystack_connection = 'solr'
     date_hierarchy = 'pub_date'
     list_display = ('author', 'pub_date')
 

tests/discovery/search_indexes.py → test_haystack/discovery/search_indexes.py

@@ -1,5 +1,5 @@
 from haystack import indexes
-from discovery.models import Foo, Bar
+from test_haystack.discovery.models import Foo, Bar
 
 
 class FooIndex(indexes.SearchIndex, indexes.Indexable):