CompletionSuggester can return partial information (no _id and _score)

[nomoa]

We are seeing this behavior from times to times, we see a spike on invalid responses returned by the completion suggester: the options returned may be "partial".
By partial response I mean that the individual hits are not provided with the usual information retrieved during the fetch_phase.

In this case only text seems to be available, _id and _score are not populated.

I'm almost sure that this happens only when a node was dropped from the cluster or during a rolling restart.

Looking at the code this behavior seems to line-up with a TODO comment in SearchPhaseController.java where a failure during the fetch phase may produce this inconsistency.

[elasticmachine]

Pinging @elastic/es-search-aggs

[jimczi]

The TODO comment is about adding a verification that the fetch phase really failed on the node (e.g. checking the shard failures). If the fetch phase failed on one node we don't have many options especially if the node is gone. When the same situation happens in normal search we simply remove the hit from the response. We should probably do the same for suggestions. Note that the failure is reported in the response inside the _shards section so this is something that you can/need to handle in the client side. We also added a new search option called allow_partial_search_results that when set to false will fail a request if there is a shard failure so you could also use this option to detect these bad queries. How often do you see this behavior and how many requests are impacted ? We cannot redirect the fetch phase to another node/replica so the best solution so far is just to retry the query with the failure.
I am marking this issue with the discuss label to get feedbacks on whether we should hide the partial suggestion or not, thanks for reporting @nomoa !

[nomoa]

Thanks for looking into this!

For us it happens during short spikes where we can see few thousands of responses broken like that (we serve between 500 and 1000 qps with this endpoint).
Thanks for reminding me that the _shards section may contain the failures, I probably overlooked this section.
About allow_partial_search_results, yes we noticed this new option but we're still running elastic 5.5.2 which I think behaves like if it was always true.

As for the possible solutions to fix this, I think that the main problem here is that a client may be confused to not see the usual data when using the _search endpoint and what you suggest about removing partial suggestions sounds good to me, it'd be coherent with the "normal" search I think.

For now we'll deploy a workaround by just inspecting the hits array and discard the response if it does not contain all the necessary info. This bug is not really a big deal per se, just confusing.

Thanks!

[polyfractal]

Chatted about this in Fixit Thursday, although we weren't quite sure of the technical details.

@jimczi is it correct to say that these "partial" results can happen because we identify a suggestion during the search phase, merge all the shard results but a failure in the fetch phase means we can't enrich it with extra metadata?

If that's roughly correct, we agreed that hiding the partial suggestions would be the most consistent thing to do. 👍 Seems very trappy for clients to get back partially-formed results... would be similar to search hits missing important metadata or parts of the document.

[jimczi]

If that's roughly correct, we agreed that hiding the partial suggestions would be the most consistent thing to do. 👍 Seems very trappy for clients to get back partially-formed results... would be similar to search hits missing important metadata or parts of the document.

+1, thanks @polyfractal

[elasticsearchmachine]

Pinging @elastic/es-search (Team:Search)

[elasticsearchmachine]

Pinging @elastic/es-search-relevance (Team:Search Relevance)