Skip to content

Troubleshooting

Jump to bottom
Bernadette Varga edited this page Jan 13, 2017 · 6 revisions

Encountered problems

  1. Poor document relevance
  2. Inconsistent results

Poor document relevance

Maintaining only a small amount of data across many shards will tend to result in poor document relevance. Some advices about shard optimisation are documented in this article. The main ideas are:

  • max shard size should be between of 30-32 giga
  • total number of shards has to be between 1.5-3 x nr of nodes
  • in our case we have 5 shards/index for 64 gigas - a bit high, but should do it

For balancing our results, we applied term query boosting - which doubles the relevance of an exact match. Read more about query boosting here.

Inconsistent results

We experienced that executing the same query multiple times resulted in different results (different scoring).

This is an expected behaviour, if the scores, where the ordering differs, are equal. In this case, using a preference parameter helps forcing the requests to be executed on the same resource, so that the results will be consistent for the same user.

It is much more inconvenient, when the scores are different, depending on which of nodes the request ends-up, and whether a replica or a primary shard gets interrogated. In this case, we have a primary-replica shard inconsistency.

Things we can check:

  • number of docs of a shard and its replica
  • number of max_docs of a shard and its replica
  • deleted docs of a shard and its replica

You can check all of these information doing a request to:

curl -i {elasticsearch-domain}/concept/_stats?level=shards

and looking at indices/{index-name}/shards/{nr-of-shard}/../docs/{count/deleted}

In our case, the cluster was green, because the number of documents in both the primary shard and its replica were the same. However max_docs (= docs + deleted docs) differed. Deleted documents were different in primary and its replica. Deleted documents contribute to the score calculation, which lead to different scores.

Solutions:

There isn't an easy way to force a healthy-looks-like replica to resync. What can be done:

  1. Rerouting or reallocation Force reallocation of a shard (in our case a replica). While reallocation is done, a resynch will also be applied, which can fix the corrupted replica. Check how to do this here.

However /reroute is not an allowed operation for amazon elasticsearch service.

  1. Setting replicas to 0 The number of replicas can be modified at any time, even after an index creation. A possible way to fix the corrupt replicas is to set the number of replicas to 0, wait for the changes to apply, then reset the number of replicas to 1 (or the number of replicas you want to have). This can be done using the command:

curl -XPUT {elasticsearch-doman}/{indexName}/_settings -d '{"index":{"number_of_replicas":1}}'

Clone this wiki locally