Autosuggest endpoint with self hosted ES : documentation

[clemorphy]

I have a self hosted ES server, and I'm trying to use the autosuggest feature of ElasticPress.
I can't find any documentation about how to do this.

I found this plugin : https://github.com/grossherr/elasticpress-autosuggest-endpoint
But it doesn't work with the last versions of ES and EP.

Where can I start ?

[clemorphy]

I tried with ES 6.0.1 and it works. I think it's the same problem as #1625

[brandwaffle]

@clemorphy we have tested with ES 7.5 and EP 3.3 on Docker-based Elasticsearch, including Autosuggest, and it works as of the current plugin release. To use autosuggest, you simply need to put in your _search endpoint in the appropriate WP Dashboard field.

If it's not working, and you have username/password auth set up (recommended if you are exposing your ES server directly), you'll need to come up with a solution for how to handle that. The lack of effective, granular authorization in ES is one of the reasons we built ElasticPress.io.

[tamara-m]

@brandwaffle

I can't find how to relate this to the connection, as I do receive a successful, authenticated response from ElasticSearch. It just has "0" hits.

To recap for context:

With ElasticPress 3.3 and ElasticSearch 5.6 + custom Autosuggest endpoint: Works correctly, hits are returned

With ElasticPress 3.3 and ElasticSearch 6.8 + custom Autosuggest endpoint: Works correctly, hits are returned

With ElasticPress 3.3 and ElasticSearch 7.xx + custom Autosuggest endpoint: Doesn't work any more. A response is returned, but always with 0 hits.

The Autosuggest endpoint and all custom code involved is exactly the same, and works when using up to ElasticSearch 6.8

It breaks when I start using as from ElasticSearch 7.x

How can this be related to the _search endpoint?

You suggested this on my issue too. This is likely a stupid question, but could you please clarify what you mean by "you simply need to put in your _search endpoint in the appropriate WP Dashboard field." I have to be missing something obvious here...

[brandwaffle]

@tamara-m What I mean is that the query generated for Autosuggest search is the same JSON query generated for regular search, so if you are using Autosuggest directly connected to your Elasticsearch instance, you would simply use the same URL for Autosuggest that you use to connect ElasticPress for normal searches. To get that path, simply look at an EP query that's run in the Debug Bar ElasticPress plugin and note the full URL that used there (it will end in _search).

To be clear: the path that the plugin is designed to support is that the JSON Autosuggest request travels directly to the ES server, and never touches WordPress. As I mentioned, this does leave the site potentially exposed, as your ES URL will be shown in JS in the browser, which means you need to address security. Since routing through PHP/WordPress slows the process down to the point where Autosuggest is, in our opinion, mostly unusable, we don't support any sort of passthrough implementation, though there are certainly repos out there which attempt to handle this. Depending on how they do so, updates to the plugin to handle changes in Elasticsearch syntax associated with later Elasticsearch versions could easily break things. Since we don't control those repos, your best bet is to request that the authors update their solution to support 7.x.

Overall, to connect your self-hosted instance to Autosuggest, there are three options:

1. Create a custom endpoint in WordPress/PHP and route your queries through there, to avoid exposing your ES URL. This will work if you properly forward the request, but is not part of the plugin and not something we can support. It's also not recommended since you lose 90% of the request turnaround speed (WordPress AJAX or even a PHP wrapper script is much slower than ES JSON/HTTP via a direct connection).
2. Build something similar to what we offer on ElasticPress.io, which creates a custom endpoint /autosuggest instead of /_search that bypasses username/password authentication but only allows requests with the same format as your typical search request to pass through. This will allow you to skip WordPress/PHP and get good Autosuggest performance with adequate security.
3. Leave authentication off of your ES index, and simply allow anyone to send requests from the client side directly to your server. To do this, as I mentioned above, you would use the exact URL for your ES setup's _search endpoint--no additional changes are necessary. At this point, you will be using the same query for Autosuggest and backend searches, and this is the scenario we test to ensure compatibility with a generic Elasticsearch cluster.

[brandwaffle]

Adding some screenshots here for a future potential explainer post about how Autosuggest functions. This example uses WP Local Docker, which creates an ES container named elasticsearch and makes it available via the following URL pattern for HTTP requests: http://<site.domain>/__elasticsearch/<indexname>-post-1/_search . Because of this approach, we need to manually set the Autosuggest endpoint to a different custom value, though it will normally be the exact same as the normal _search endpoint.

First we set our ES server, in this case the elasticsearch container via its HTTP endpoint

Then we simply re-paste this endpoint URL with the index name and _search suffix into the appropriate field in the Autosuggest Feature:

As we can see here, the Autosuggest endpoint is simply the same as the generated index URL for any normal search.

[brandwaffle]

@clemorphy @tamara-m I've also updated the bug report on that Autosuggest plugin grossherr/elasticpress-autosuggest-endpoint#12. From what I can tell, since the plugin uses a method that no longer exists to set the index name, it will always fail. Again, this is just based on a quick review, since we don't officially support any third-party integrations.

[tamara-m]

@brandwaffle

Thank you for your time on this, and the explanations.

I will try playing with the URL. So far I had been using the URL of the endpoint as indicated in that Autosuggest plugin (* This is the endpoint you have to specify in the admin like this: http(s)://domain.com/wp-json/elasticpress/autosuggest/) works correctly when using ES 5.6 through 6.8. From your past messages I had already tried some stuff with it there but nothing worked.

I had mentioned in the issue I opened that we were using a modified version of this plugin, as it doesn't work as it is there as from ElasticPress 3 when it was refactored.

The exact code I'm using that works up until ES 6.8 is here:
https://gist.github.com/tamara-m/6b8bdb61aa9cf9b2a59a63ffa9e0d4f7

You can see in lines 39 and 45 I had already updated them to work on EP 3.x . ep_get_index_name is not the issue, this still does not work on ES 7.x.

To recap:

With ElasticPress 3.3 and ElasticSearch 5.6 + the modified Autosuggest endpoint above: This is the response with results as shown here https://gist.github.com/tamara-m/e4903308081a6eae8994d3d5a591ab86
This is the actual response, I just edited the data to share

With ElasticPress 3.3 and ElasticSearch 6.8 + the modified Autosuggest endpoint above: Same. Still works. This is the response with results as shown here https://gist.github.com/tamara-m/e4903308081a6eae8994d3d5a591ab86
This is the actual response, I just edited the data to share

With ElasticPress 3.3 and ElasticSearch 7 + the modified Autosuggest endpoint above: Stops working. These is the response with hits always in 0: https://gist.github.com/tamara-m/fdb770d0685acb3d0ff609f68b3bdcc3
This is the exact response.

Note I am always using ElasticPress 3.3 in all cases.

I understand you don't provide support for this. I also understand this is a complicated thing, and I think overall you guys provide a fantastic solution with ElasticPress.

However, Autosuggest is a bit of a thorn, and as a user it makes you feel you're thrown into a loop logic. ElasticPress highlights Autosuggest as one of the supported features.
However, you can't use it out-of-the-box. As a user, you are then forced to look in outside repos, custom code and outside solutions to use it in any way. When support is requested, the answer is "we don't control those repos". It's a strange experience from this side.

Even with this, I'm still I'm an ElasticPress fan. I also value very much that you reply here and are trying to help, also posted this in ElasticSearch forums and it's been only crickets.

I'll ask the Autosuggest plugin dev if he wants to update his version with the fixes for EP 3 in case it helps others and also in case you want to use it to point at when someone asks.

[tamara-m]

Setting aside security and the exposure of the endpoint for the sake of troubleshooting, setting the endpoint URL with the index name and _search suffix into the appropriate field in the Autosuggest Feature as indicated gives this error:

Setting instead https://domain.com/wp-json/elasticpress/yourendpoint/ works correctly when using ES 5.6 through 6.8.

[brandwaffle]

@tamara-m yes, CORS is a consideration that needs to be addressed when implementing a cross-resource web request from the browser. As I've said multiple times, this is not a simple process, but the plugin does 100% support Autosuggest for a properly-configured Elasticsearch instance, if you follow these steps:

1. install the plugin
2. activate Autosuggest
3. connect to a properly-configured ES server
4. paste the URL from that server's search endpoint into the Autosuggest field

Most of your examples indicate you're still trying to route through WordPress. Again, that involves custom solutions and is really not a great way to handle Autosuggest. I understand your frustration, but I also ask that you consider that the multiple moving parts and complexity of providing a secure and performant Autosuggest component is why we built a service that addresses these concerns from end to end. While I understand that you are not willing to use the service we offer, connecting multiple open source tools does often come at the price of spending time troubleshooting and configuring tools (such as Elasticsearch) that are intended for generic and multiple use-cases rather than purpose-built for a specific need.

At this point, when you connect directly, you're getting a CORS error (which can be resolved by updating your headers), and there is an issue in the script connecting to ES through WordPress. The latter I can confirm since I'm able to test directly using my WP Local Docker instance and have also confirmed the query for Autosuggest matches the regular search query, as expected. This implies that the query, when it reaches your ES server, should behave 100% the same way (aka return hits) assuming the equivalent normal search query also returns hits.

All that said, I spent some time testing your script, and if you remove the 'type' => 'post' parameter in your query (not supported since ES 7.x), things seem to work again. Can you let me know if this also works for you?

[tamara-m]

@brandwaffle You got it. Removing 'type' => 'post' from the query makes it work in ES 7.5 and you debugged code you shouldn't have had to fix. I didn't think the issue would be on my code, I am sorry.

If it was for me, I would be more than willing to use your service, you know how these decisions many times don't fall into one's hands.

To double check, I understand your are referring to these service plans: https://www.elasticpress.io/#pricing. Would like to confirm that using this service makes Autosuggest plug-and-play out of the box, and the performance issue of the custom endpoint in WordPress/PHP and routing queries through there.

Can I compensate for your help by contributing anything here?

This thread, for instance, has very valuable explanations I did not find anywhere else. Would be happy to organize it into something you can add to docs.
Or help with any Spanish translations, or with issue 13 above?