ElasticSearch 2.0 support

[PedroAquilino]

Gives haystack support for Elasticsearch 2.x

• Adds a new backend haystack.backends.elasticsearch2_backend.Elasticsearch2SearchEngine
• Reuses the code and tests for ES 1.x
• Changes facets (aggregations on ES 2.x)
• Changes delete_by_query (scroll API and bulk actions)
• "more like this" (More Like This Query)
• Modifies the dependecy on elasticsearch library version

Issue:
#1247

[acdha]

We need to keep testing with 1.x. I'm not sure whether it'd be easier to run the tests with an alternate port or to use a build matrix to pick one of the two. It looks like travis-ci/apt-source-safelist@189b706 was merged awhile back so I think we could use a matrix like what they're doing here:

insomnia-lab/libreant@49fc5d0

[PedroAquilino]

using a build matrix seems the best solution, maybe with only elasticsearch-1.7 and elasticsearch-2.1 versions

[acdha]

1.7 and 2.x seems great to me - the main thing I was thinking is e.g. AWS with the hosted 1.7 used by many people

[acdha]

This is generally looking quite good — testing is always tedious with Haystack but I think you're very close.

[PedroAquilino]

Finally elasticsearch in travis is installed using tar. The apt sources for es 2.x are broken, the url used is not longer valid.

W: Failed to fetch http://packages.elastic.co/elasticsearch/2.1/debian/dists/stable/main/binary-amd64/Packages  404  Not Found [IP: 184.73.186.87 80]
W: Failed to fetch http://packages.elastic.co/elasticsearch/2.1/debian/dists/stable/main/binary-i386/Packages  404  Not Found [IP: 184.73.186.87 80]

The versions are:
elasticsearch 1.7.5
elasticsearch 2.2.1

[PedroAquilino]

Rebuilding on travis, I see than sometimes
test_haystack.solr_tests.test_management_commands.ManagementCommandTestCase#test_multiprocessing fails

[acdha]

It would probably be a good idea to rebase this on 10bcc76 so to pick up Django 1.9.

re:the tests failing on multiprocessing, that should probably go into a separate ticket – there's a race condition which I have thought was fixed at least twice and it deserves a deeper investigation.

[Jyrno42]

Hey,

i wanted to point out that any query against elasticsearch>=2.1 will fail if one uses a slice where start_offset + end_offset is bigger than index.max_result_window which defaults to 10000. This is generally not an issue since standard use cases will not do this.

However, update_index --remove uses slices of SearchQuerySet(using=backend.connection_alias).models(model) to figure out which items in the index are stale which causes elasticsearch to return a 500 resulting in update index breaking

elasticsearch.exceptions.TransportError: TransportError(
    500, 
    'search_phase_execution_exception', 
    'Result window is too large, from + size must be less than or '
      'equal to: [10000] but was [10010]. See the scroll api for a '
      'more efficient way to request large data sets. This limit can be set by  '
      'changing the [index.max_result_window] index level parameter.')

[yeago]

Does anyone have a sense of what django versions will be still be supported by this changeset?

[PedroAquilino]

The pull request is tested on django 1.8 and 1.9

[anmolghosh]

Any ETA for when these changes will be merged and released ?

[acdha]

@anmolghosh if you have time to contribute, please update this ticket with the results of your testing.

[frague59]

Add the AttachmentField from:

https://gist.github.com/frague59/8ab2470ed133754a6327

should be great, to support attachments : files indexed using apache Tika to explore documents.

[Alkalit]

I use this branch on production. What I notice:
highlight doesn't work against autocomplete on Index class. Only against SearchQuerySet.

AdIndex.objects.autocomplete(text=request.GET['q']).highlight() # no highlight

SearchQuerySet().models(Ad).autocomplete(text_auto=request.GET['q']).highlight() # will highlight

highlight method doesn't accept arguments. And highlighted attribute contains list of highlighted snippets, not a dict.

[Alkalit]

For arguments of highlighted - it would be enough to replace with this snippet

        if highlight:

            kwargs['highlight'] = {
                'fields': {
                    content_field: {'store': 'yes'},
                }
            }

            if isinstance(highlight, dict):
                kwargs['highlight'].update(highlight)

just here
https://github.com/PedroAquilino/django-haystack/blob/es2/haystack/backends/elasticsearch2_backend.py#L350

[PedroAquilino]

Before adding new funcionality. (arguments of highlighted, AttachmentField) @acdha What we need to merge the changes?

[nherriot]

Hi Guys,

just to let you know - testing against elastic search 2.3 still gives errors with 'facets'. I'm just fresh into this and started looking at the commits.

Can someone get me started and point me to where the construction of the JSON object is done that gets passed to Elastic ? I'd like to just add some debug on my machine.

Kind regards, Nicholas.

[JanMalte]

@nherriot add simple logging to django maybe helps

LOGGING = {
    'handlers': {
        'console': {
            'class': 'logging.StreamHandler',
            'stream': sys.stdout,
        }
    },
    'loggers': {
        'haystack': {
            'handlers': ['console'],
            'level': 'DEBUG',
            'propagate': False,
        },
        'elasticsearch': {
            'handlers': ['console'],
            'level': 'DEBUG',
            'propagate': False,
        },
    },
}

the JSON objects seems to be build in https://github.com/elastic/elasticsearch-py/

[nherriot]

Hi all,

OK this may be obvious to many of you. Just trying to checkout this branch. How do I do this?
I'm on origin/Head -> origin/master.

When I do a /> git branch -r . I can't see branch 'es2' ?
Is this private?
At the moment I can see that master does not have many of the v2 files - so I guess I'm doing something silly!

Thanks in advance to anyone who responds.

Kind regards, Nicholas.

[Alkalit]

@nherriot I do this

git clone https://github.com/PedroAquilino/django-haystack.git
cd django-haystack
git checkout es2

[nherriot]

Hi all,

anything in particular that has to be done for Haystack to start using the new elasticsearch2_backend.py?
Looks like to me it still uses {'facet' .....bla } rather than {'aggs' .......bla}.

By the way if anyone is interested I have a hack that just really changes the string in the json request to ES to use the new 'aggs' parm. It works for us and gets us moving again. But I still see I might have some work to do on parsing the new response.
I take it the thinking is that Haystack is going to keep it's interface static and change the protocol to suit ES - is that correct? Is there anyone controlling this or managing our direction? Should this sort of chat be better on IRC or user groups?

Kind regards, Nicholas

[joaojunior]

@PedroAquilino I see that exist some conflicts to merge, you have time to fix this?
I would like to refactoring this code because the code is very similar with version 1, What do you think about this?

[PedroAquilino]

I see that exists some conflicts to merge, but I don't have time to fix it now...
About the refactoring... Go for it. My code is the elasticsearch_backend with little changes (deprecated ES APIs)

[hanneman]

Is there already an idea about a potential delivery...? And will it be production-ready? We have tested the code but are already experiencing issues while getting it running... We really need the ES2.x support (and the announced version 5 after summer). We use ES for GEO-queries and facets. #just-asking

Is there any other way we can contribute?

[acdha]

@hanneman The main thing we're missing is someone who actually uses ES2 helping get a release-quality branch ready to merge. If memory serves, #1391 was the closest and could use more testing / review.

[seocam]

This has been replaced by #1460.