Slightly more accurate terms sorting on sub-aggs
#72684
Assignees
Labels
:Analytics/Aggregations
Aggregations
>enhancement
Team:Analytics
Meta label for analytical engine team (ESQL/Aggs/Geo)
Comments
elasticmachine
added
the
Team:Analytics
|
Pinging @elastic/es-analytics-geo (Team:Analytics) |
|
A few of us go to talking about this and figured out we don't really know how often we get into this state. We document that this sort of thing can be an error. We're not super clear about the side effects here. I think we use kind of dense language. I think we can improve that. Anyway! It'd be useful to collect a counter of how often this happens. I'm not sure it's worth the effort right now. We're kind of swamped and we're not sure that we'd really do with the counter. We don't have a good way to fix this. The errors are still unbounded. Even if we bump the shard_size. It'd help. Some. But we don't know how much and it'd absolutely come with performance changes. And, even better, you can do this yourself. Just set the So! I'm going to convert this into a docs issue and just update those docs and leave the rest as it. |
nik9000
added a commit
to nik9000/elasticsearch
that referenced
this issue
Oct 14, 2021
The `terms` agg picks the top `size` terms in a single scatter/gather pass across all the shards. For the default `order` and if you `order` by `_key` this works quite well. Some errors creep in, but it's fairly easy to point to them and understand them. But ordering by doc count ascending is like inviting the error vampire into your agg. It's super easy to get inaccurate results. This updates the docs to be more stark about it. Closes elastic#72684
elasticsearchmachine
pushed a commit
that referenced
this issue
Nov 1, 2021
The `terms` agg picks the top `size` terms in a single scatter/gather pass across all the shards. For the default `order` and if you `order` by `_key` this works quite well. Some errors creep in, but it's fairly easy to point to them and understand them. But ordering by doc count ascending is like inviting the error vampire into your agg. It's super easy to get inaccurate results. This updates the docs to be more stark about it. Closes #72684
nik9000
added a commit
to nik9000/elasticsearch
that referenced
this issue
Nov 1, 2021
The `terms` agg picks the top `size` terms in a single scatter/gather pass across all the shards. For the default `order` and if you `order` by `_key` this works quite well. Some errors creep in, but it's fairly easy to point to them and understand them. But ordering by doc count ascending is like inviting the error vampire into your agg. It's super easy to get inaccurate results. This updates the docs to be more stark about it. Closes elastic#72684
nik9000
added a commit
to nik9000/elasticsearch
that referenced
this issue
Nov 2, 2021
The `terms` agg picks the top `size` terms in a single scatter/gather pass across all the shards. For the default `order` and if you `order` by `_key` this works quite well. Some errors creep in, but it's fairly easy to point to them and understand them. But ordering by doc count ascending is like inviting the error vampire into your agg. It's super easy to get inaccurate results. This updates the docs to be more stark about it. Closes elastic#72684
nik9000
added a commit
that referenced
this issue
Nov 2, 2021
The `terms` agg picks the top `size` terms in a single scatter/gather pass across all the shards. For the default `order` and if you `order` by `_key` this works quite well. Some errors creep in, but it's fairly easy to point to them and understand them. But ordering by doc count ascending is like inviting the error vampire into your agg. It's super easy to get inaccurate results. This updates the docs to be more stark about it. Closes #72684
nik9000
added a commit
to nik9000/elasticsearch
that referenced
this issue
Nov 2, 2021
…elastic#80223) The `terms` agg picks the top `size` terms in a single scatter/gather pass across all the shards. For the default `order` and if you `order` by `_key` this works quite well. Some errors creep in, but it's fairly easy to point to them and understand them. But ordering by doc count ascending is like inviting the error vampire into your agg. It's super easy to get inaccurate results. This updates the docs to be more stark about it. Closes elastic#72684
nik9000
added a commit
to nik9000/elasticsearch
that referenced
this issue
Nov 2, 2021
…elastic#80223) The `terms` agg picks the top `size` terms in a single scatter/gather pass across all the shards. For the default `order` and if you `order` by `_key` this works quite well. Some errors creep in, but it's fairly easy to point to them and understand them. But ordering by doc count ascending is like inviting the error vampire into your agg. It's super easy to get inaccurate results. This updates the docs to be more stark about it. Closes elastic#72684
nik9000
added a commit
that referenced
this issue
Nov 2, 2021
… (#80227) The `terms` agg picks the top `size` terms in a single scatter/gather pass across all the shards. For the default `order` and if you `order` by `_key` this works quite well. Some errors creep in, but it's fairly easy to point to them and understand them. But ordering by doc count ascending is like inviting the error vampire into your agg. It's super easy to get inaccurate results. This updates the docs to be more stark about it. Closes #72684
nik9000
added a commit
that referenced
this issue
Nov 2, 2021
… (#80226) The `terms` agg picks the top `size` terms in a single scatter/gather pass across all the shards. For the default `order` and if you `order` by `_key` this works quite well. Some errors creep in, but it's fairly easy to point to them and understand them. But ordering by doc count ascending is like inviting the error vampire into your agg. It's super easy to get inaccurate results. This updates the docs to be more stark about it. Closes #72684
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Right now we default the
shard_sizeto1.5*size+10intermsin an effort to keep the doc count errors lower. When you sort the terms by something other than doc_count descending the error is unbounded. But largershard_sizewill lead to more accurate results because we ship more data back to the coordinating node. Maybe we should bump theshard_sizeif you are sorting on anything other than doc_count descending to get you more accurate results by default. Sure, it'll be slower, but it's the price you pay for a little more accuracy. And the performance when sorting by the default won't change.The text was updated successfully, but these errors were encountered: