Support for Elasticsearch 5.x

[ElSaico]

Based on #1460 - will be properly rebased once it's accepted.

So far only two features seem to be failing, according to the test suite:

• Faceted search (ES seems to throw an error about fielddata)
• Autocomplete

Any help to solve any of those would be appreciated.

[seocam]

This PR addresses #1383

[Alkalit]

Hi! I also work on elasticsearch 5 support for haystack. My project lies in this repo.
I think it would be better if are new backends (and also are some old) will be placed in separate repos.
The reason for this is that testing for all these backends complicates the testing process. And it imposes an additional burden
at its core developers, since in addition to the main project code must still follow trends for each search engine.

What do you say if I offer you to work together at this project? Or at least borrow the idea.

[acdha]

I just merged #1460 so it might be good to see how small this request can be

[seocam]

@ElSaico travis has whitelisted elasticsearch 5.x repo. We can update/simplify travis.yml ;)

[ElSaico]

@Alkalit just managed to fix the last of the issues, but thank you for your offer!

@acdha the major source of shared code is build_search_kwargs, which is true on 1.x/2.x as well. I've refactored all three to address this, which can be seen here: https://github.com/CraveFood/django-haystack-elasticsearch
Perhaps a PR to do this on the current backends would be a good preceding step?

[acdha]

@ElSaico That seems reasonable – I've been meaning to take a look at that in general and seeing how much we could refactor into shared utilities or methods on BaseSearchBackend so things like input parameter validation & type conversion would be consistent across all of the backends.

Things like https://github.com/django-haystack/django-haystack/blob/master/haystack/backends/elasticsearch_backend.py#L309 or

django-haystack/haystack/backends/elasticsearch_backend.py

Lines 413 to 420 in f8921b6

	if models and len(models):
	model_choices = sorted(get_model_ct(model) for model in models)
	elif limit_to_registered_models:
	# Using narrow queries, limit the results to only models handled
	# with the current routers.
	model_choices = self.build_models_list()
	else:
	model_choices = []

come to mind.

[bqumsiyeh]

What is the timeframe for merging this into a stable release?

[seocam]

We have been using ES5 with haystack in our production for about 20 days now. For our needs it seems reasonably stable.

[acdha]

@seocam thanks - I can do a release soon but since I don't use it I'm not entirely comfortable saying it's been well tested

[seocam]

@acdha I totally understand! Even I would be a lot more comfortable with more people using it from the branch before having a new version out.

[acdha]

Maybe I'll merge it and publish a -pre release to make it easier for people to test

[mintusah25]

@acdha Any update on pre release for ES 5.X??.

[rrmerugu]

@acdha any update on pre release for ES 5.x !

[SalahAdDin]

@acdha Please.

[adamchainz]

@SalahAdDin typo? I don't have anything to do with django-haystack

[Ameriks]

ES 5.x support branch doesn't work for me.

File "/src/django-haystack/haystack/urls.py", line 7, in <module>
from haystack.views import SearchView

File "/src/django-haystack/haystack/views.py", line 16, in <module>
class SearchView(object):

File "/src/django-haystack/haystack/views.py", line 20, in SearchView
results = EmptySearchQuerySet()

File "/src/django-haystack/haystack/query.py", line 29, in __init__
self._determine_backend()

File "/src/django-haystack/haystack/query.py", line 62, in _determine_backend
self.query = connections[backend_alias].get_query()

File "/src/django-haystack/haystack/utils/loading.py", line 108, in __getitem__
self.thread_local.connections[key] = load_backend(self.connections_info[key]['ENGINE'])(using=key)

TypeError: __init__() missing 1 required positional argument: 'connection_alias'

Sorry, false issue. I included wrong class in settings.py. Included Elasticsearch5SearchBackend instead of Elasticsearch5SearchEngine.

Well done!

[alanjds]

Is there anything holding it to be merged yet?

[acdha]

@alanjds More test results from people using ES would be helpful

[alanjds]

@acdha I see. Looks like I would have the 1st cobaia project using this... But the good news is that nobody said "It have a problem. Be never merged. Beware and run to the hills!".

Thanks.

[acdha]

@alanjds the time taken to merge it isn't high but I don't really have that much time for an in-depth review. If you have some positive test results that'd save me the time there.

[millerf]

Could it be possible to merge it?

[acdha]

@millerf If you have time to work on this, test results are appreciated

[millerf]

I have very little experience with all that ES, so I am a bit perplexed...
From what I see, it is missing the sort= option on facetting.
Also, the repo ask to remove faceted=True options on SearchIndex. Is that because of ES 5.x? I suppose haystack docs should be updated for that as well, no?

[erikdao]

When will this be official?

[lokeshpahal]

ES 5.x got issue while executing rebuild_index command

Traceback (most recent call last):
File "manage.py", line 15, in
execute_from_command_line(sys.argv)
File "/var/www/MSDev/env/lib/python3.6/site-packages/django/core/management/init.py", line 371, in execute_from_command_line
utility.execute()
File "/var/www/MSDev/env/lib/python3.6/site-packages/django/core/management/init.py", line 365, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/var/www/MSDev/env/lib/python3.6/site-packages/django/core/management/base.py", line 288, in run_from_argv
self.execute(*args, **cmd_options)
File "/var/www/MSDev/env/lib/python3.6/site-packages/django/core/management/base.py", line 335, in execute
output = self.handle(*args, **options)
File "/var/www/MSDev/env/lib/python3.6/site-packages/haystack/management/commands/rebuild_index.py", line 36, in handle
call_command('clear_index', **options)
File "/var/www/MSDev/env/lib/python3.6/site-packages/django/core/management/init.py", line 133, in call_command
', '.join(sorted(valid_options)),
TypeError: Unknown option(s) for clear_index command: batchsize, workers. Valid options are: commit, help, interactive, no_color, nocommit, noinput, pythonpath, settings, skip_checks, stderr, stdout, traceback, using, verbosity, version.

[erikdao]

Currently haystack doesn’t support ES 5.x.

…

On Fri, 8 Jun 2018 at 15:07 Lokesh Pahal ***@***.***> wrote: ES 5.x got issue while executing rebuild_index command Traceback (most recent call last): File "manage.py", line 15, in execute_from_command_line(sys.argv) File "/var/www/MSDev/env/lib/python3.6/site-packages/django/core/management/ *init*.py", line 371, in execute_from_command_line utility.execute() File "/var/www/MSDev/env/lib/python3.6/site-packages/django/core/management/ *init*.py", line 365, in execute self.fetch_command(subcommand).run_from_argv(self.argv) File "/var/www/MSDev/env/lib/python3.6/site-packages/django/core/management/base.py", line 288, in run_from_argv self.execute(*args, **cmd_options) File "/var/www/MSDev/env/lib/python3.6/site-packages/django/core/management/base.py", line 335, in execute output = self.handle(*args, **options) File "/var/www/MSDev/env/lib/python3.6/site-packages/haystack/management/commands/rebuild_index.py", line 36, in handle call_command('clear_index', **options) File "/var/www/MSDev/env/lib/python3.6/site-packages/django/core/management/ *init*.py", line 133, in call_command ', '.join(sorted(valid_options)), TypeError: Unknown option(s) for clear_index command: batchsize, workers. Valid options are: commit, help, interactive, no_color, nocommit, noinput, pythonpath, settings, skip_checks, stderr, stdout, traceback, using, verbosity, version. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#1461 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AZuKazm_QnoMb3LkKce3x617qYXmXHWlks5t6jDAgaJpZM4LLugB> .

[chocobn69]

All tests are passing on my project. I will use it on production and tell you results in a few days.

[chocobn69]

Working on production system since 72h, no issues.

[acdha]

Thanks for testing - I’ll merge it later today

[angvp]

This might need a rebase before merge, we are waiting for you @acdha :-)

[acdha]

Indeed - I didn’t have time for that last week since it was more involved than I thought

[acdha]

I merged this in my working branch (see acdha@e366f49) but we still need to fix the GIS code to work with Django 2:

https://travis-ci.org/acdha/django-haystack/jobs/393751311#L2309-L2319

[acdha]

Rebased and merged with 1fda20e to fix the get_coords calls

[UmanShahzad]

Can this be pushed to pypi?

[intellisense]

Hi, When this will be released? Thanks!

[randomlock]

I get this exception when searching data
elasticsearch.exceptions.RequestError: TransportError(400, 'parsing_exception', 'no [query] registered for [filtered]')