Return more information in RefreshStats

[romseygeek]

A common cause of performance issues in elasticsearch is long-lasting or too-frequent refreshes (eg indexing multiple individual docs with refresh=true rather than using bulk loads, or enabling eager_global_orginals on high cardinality fields). Refresh stats currently give us the total number of refreshes and the total time they took, but this isn't particularly useful when trying to track down these refresh issues.

One possible enhancement would be to keep track of the number and length of refreshes in the previous 60 mins, and return that; we could also track the tail percentile refresh lengths and warn if a refresh takes longer than a certain amount of time.

[elasticmachine]

Pinging @elastic/es-search (:Search/Search)

[jimczi]

If eager_global_ordinals is the culprit of these slow refreshes this should show up in the warmer stats. For high cardinality field the cost of this option should not change upon refreshes so the load time should be reflected in the stats. Maybe we could add dedicated entries for each warmer or add the loading time in the fielddata stats to ease debugging ?

[elasticsearchmachine]

Pinging @elastic/es-search (Team:Search)

[elasticsearchmachine]

Pinging @elastic/es-distributed (Team:Distributed)

[javanna]

Given the recent discussions around refresh , I tagged the Distributed team to double check what they think about this issue that we opened a while ago but did not follow up on. Is it still something we'd like to address?

[javanna]

We have no plans to work on this issue, closing.