terms facet gives wrong count with n_shards > 1

[jmchambers]

I'm working with nested documents and have noticed that my faceted search interface is giving the wrong counts when I have more than one shard. To be more specific, I'm working with RDF triples (entity > attribute > value) and I'm nesting the attributes (called predicates in my example):

{
  "_id" : "512a2c022f0b4e3daa341e6c8bcf6c2f",
  "url": "http://dbpedia.org/resource/Alan_Shepard",
  "predicates": [
    {
      "type": "type",
      "string_value": ["thing", "person", "astronaut"]
    }, {
      "type": "label",
      "string_value": ["Alan Shepard"]
    }, {
      "type": "time in space",
      "float_value": [216.950]
    },
    ... lots more
  ]
}

I've created a shell script (https://gist.github.com/1196986) that recreates the problem with a fresh index. The created data set has these totals:

• thing (30)
• creative work (20)
• video game (10)
• tv show (10)
• people (10)

With only one shard the following query gives the correct counts no matter what the size parameter is set to:

{
  "size": 0,
  "query": {
    "match_all": {}
  },
  "facets": {
    "type_counts": {
      "terms": {
        "field": "string_value",
        "size": 5
      },
      "nested": "predicates",
      "facet_filter": {
        "term": {
          "type": "type"
        }
      }
    }
  }
}

However, with more than one shard the size parameter affects the accuracy of the counts. If it is equal to or greater than the number of terms returned by the facet query (5 in this case) then it works fine. However, the terms at the bottom of the list start to display low counts as you reduce the size parameter:

With "size" : 4

• thing (30)
• creative work (20)
• video game (10)
• tv show (9)

With "size" : 3

• thing (30)
• creative work (15)
• video game (9)

With "size" : 2

• thing (30)
• creative work (15)

So it looks like the sub-totals from some of the shards aren't being included for some reason. BTW I'm on ubuntu and the problem seems to affect all versions of ES I've tried (17.0, 17.1 and 17.6). Any ideas...?

P.S. absolutely loving ES - it's made my life a lot easier :)

[losomo]

+1 for this bug. I have reproduced the problem using just documents with one field.
Complete test (as run on version 0.17.8):
as Perl script: https://gist.github.com/1292897
generated shell test: https://gist.github.com/1292912
The first error can be seen on line 951.
Expected:

"terms" : [
 {
 "count" : 10,
 "term" : "user 10"
 }
 ],

Got:

"terms" : [
 {
 "count" : 7,
 "term" : "user 9"
 }
 ],

[losomo]

After more experimenting I'd say that it's caused by naïve top-N facet merging. Something like Phase three here should be added:

http://wiki.apache.org/solr/DistributedSearchDesign#Phase_3:_REFINE_FACETS_.28only_for_faceted_search.29

Or something smarter: http://netcins.ceid.upatras.gr/papers/Klee_VLDB.pdf

[kimchy]

Right, the way top N facets work now is by getting the top N from each shard, and merging the results. This can give inaccurate results. The phase 3 thingy is not really a solution, will read the paper though :)

[losomo]

Right. The "phase 3 thingy" only solves the second problem:

1. The recall is not 100% (some values may be simply missing)
2. The numbers are often lower than they should be.

But while the first problem can easily stay unnoticed, the second one leads to quite a lousy user experience in very common faceting scenarios: The web displays top 10 commenters with number of their comments in parentheses. You click the last one with 30 comments and Voilà, it shows her 48 comments. This is hard not to notice.

Meanwhile a slow and not-really-always-working workaround:
Set size to what you want + 150 or another magical constant when requesting the facets and then filter out the top size facets on the client side.

[karussell]

The phase 3 thingy is not really a solution

@kimchy: why do you think so? isn't it a similar approach as query then fetch? now only for facets and its counts instead of docs and its score? (it even could be implemented in the second query to avoid traffic, no?)

Another workaround would be 'routing': every facet value should go to the same shard. But then the data should have a lot facet values and all of them should have not a too high count to avoid wrong balancing ...

[agnellvj]

Is this on the roadmap? We discovered this same problem with regular fields and large datasets with multiple nodes and shards. This is problematic for us since the faceted counts can fluctuate wildly based on what filters we apply. Is there a workaround?

[piskvorky]

I still see this bug in 19.4, and the wrong counts are also a show-stopper for us.

Adding "150 or another magical constant" only helps for tiny datasets; does there exist a more robust workaround until this is fixed properly?

[jmchambers]

+1 for a better workaround or fix

The only completely robust workaround I know of is to limit yourself to one shard. Obviously that pretty much takes the 'elastic' out of 'elasticsearch', but if accurate counts are critical to your app and your index isn't too big you might get away with it...

[tarunjangra]

Any update on this? I do have an application where counts are critically important. As you are saying to limit to one shard for a particular entity which suppose to undergo such queries.
Does this make sense to route entities by entity type in corresponding shards?

[karmi]

The only completely robust workaround I know of is to limit yourself to one shard. Obviously that pretty much takes the 'elastic' out of 'elasticsearch' (...)

@jmchambers Not really -- you can slice the data into many one-shard indices, with a weekly (daily, hourly, ...) rotation. You can use aliases (or wildcards, ...) to query them in a reasonable way from your application. Would mulitple one-shard indices work around the facet problem?

[piskvorky]

@karmi: only if the indices are large enough. To my knowledge, the scoring stats (IDF etc) are computed per-index (actually, per-shard by default). So comparing relevancy scores across indices is like comparing apples to oranges, even with dfs_query_then_fetch.

But once the indices are large enough, this doesn't matter anymore as the stats converge (assuming identical doc/word distribution in each index, for the stats to actually converge).

[ajhalani]

+1 for a fix. The workaround (one-shard, multiple one-shard index, etc.) don't sound convincing.

[Downchuck]

I'm a bit confused on this issue: I have five shards and the top value occurs on each shard as #1, by a large margin. Still, when I facet, I get a 20% smaller number than when I simply query for the value directly.

[jrydberg]

@karmi

you can slice the data into many one-shard indices, with a weekly (daily, hourly, ...) rotation

Won't you run into the same problem when doing a faceted count over multiple indices?

[giamma]

Any news about this issue?