Using _score as a metric in visualizations

[markharwood]

Background to requirement

"Relevance engineer" is an emerging job title and Kibana has the potential to be a useful tool in helping tune relevance. Relevance engineers tweak various query DSL settings and want to review the effects of these changes.
Visualizations like the histogram can be very useful, showing the distribution of result scores produced by a query and the types of things they match :

In this diagram we can see the strict, high-scoring clauses have produced matches that belong to a small cluster of product categories on the right. We can also see that the fuzzier, low-scoring clauses have produced an enormous number of matches from a huge variety of product departments on the left. Seeing this, the relevance engineer might turn-down the fuzzy dial on the query settings. Although the above diagram shows results of a query broken down by `_score` on the x-axis configuring this visualization involved a horrible hack.

What's wrong with analyzing scores in Kibana today?

1. The drop-down for choice of numeric field in visualizations like histogram does not offer _score.
2. Since Kibana 7.x (or thereabouts) it is not possible to configure a scripted-field in the index pattern that returns _score.
3. The histogram field forces you to select a physical field before it enables the "advanced" settings that allow you to enter the JSON to have a script that returns _score BUT - when running the visualization it errors, complaining that an aggregation can't have both the physical field and the scripted data source in the same request. (This happens when the "advanced" JSON is merged with the settings defined in the GUI).

Given all of the above setbacks, the hack that gave me the workaround was:

• define a scripted-field in the index pattern called bogus that returns "1".
• select the bogus field in the histogram
• enter the JSON in the advanced settings to run a script that returns "_score"

It's perhaps a lucky accident that any of this works.

Can we make _score a valid metric for use in visualizations?

[elasticmachine]

Pinging @elastic/kibana-app (Team:KibanaApp)