Upgrade t-digest to 3.2 #28305
1 similar comment
Thanks @colings86 ! I'm not a expert on t-digest, please correct me if I'm wrong. I made a test for both t-digest 3.2 and 3.1, I found the big change is introduced in 3.2:
The output is:
For 3.1, the output:
Then I found another implementation https://github.com/CamDavidsonPilon/tdigest and made a test:
The output is:
So I trend to think it's more accurate than before.
@liketic I had a look into the differences between t-digest 3.0 and t-digest 3.2 specifically around the AVLTreeDigest that we use in Elasticsearch. The key difference that is causing the tests here to need changing seems to be a different approach to interpolating the value when the requested quantile is between two centroids.
As an example, if we are looking for the 81.25th percentile and we have 8 values the percentile is located at the 6.5th value. Let say that we have a centroid that contains the 6th value with a centroid value of 4 and a centroid containing the 7th value with centroid value of 9.
In 3.0 the interpolation was linear between the two centroids. So it will return the value that is half way between 5 and 8, since the value we want is half way between the 6th and 7th value. So the returned value is 8.5
In 3.2 the interpolation is a weighted average between the two value which takes into account the count of values in the 6th and 7th centroids. So if the 7th centroid has more values than the 6th centroid, the interpolation is biased more towards the 7th centroid.
I think the weighted interpolation is a better approach so I am happy that the change is a positive one. Would you mind updating your branch with the latest master and then I can set of another CI build to make sure we are still green before I merge this?
Thanks for working on this and also for your patience waiting for my response (sorry for the delay)
@liketic it looks like there are some test failures left in the documentation tests:
I think the expected outputs in the percentiles aggregation page in the docs just need adjusting. Are you able to make those changes?
* master: Add a note to the docs that _cat api `help` option cannot be used if an optional url param is used (elastic#28686) Lift error finding utility to exceptions helpers Change "tweet" type to "_doc" (elastic#28690) [Docs] Add missing word in nested.asciidoc (elastic#28507) Simplify the Translog constructor by always expecting an existing translog (elastic#28676) Upgrade t-digest to 3.2 (elastic#28295) (elastic#28305) Add comment explaining lazy declared versions
…on on Percentiles aggregations (#1531) I've updated the: - docker-compose to a newer version (3.4) - updated to ElasticSearch 6.4.x (available on [HERE](https://www.docker.elastic.co/)) - Updated Percentiles Aggregation Tests as the T-Digest algorithm has been updated and so all the tests failed. Have a look at [the PR on ES](elastic/elasticsearch#28305)