Remove cluster.routing.allocation.balance.primary

[s1monw]

The cluster.routing.allocation.balance.primary setting has caused
a lot of confusion in the past while it has very little benefit form a
shard allocatioon point of view. Users tend to modify this value to
evently distribute primaries across the nodes which is dangerous since
a prmiary flag on it's own can trigger relocations. The primary flag for a shard
is should not have any impact on cluster performance unless the high level feature
suffereing from primary hotspots is buggy. Yet, this setting was intended to be a
tie-breaker which is not necessary anymore since the algorithm is deterministic.

This commit removes this setting entriely.

[dakrone]

Do we want to bump the shard balance factor back up to 0.5 to keep the default weight sum at 1.0?

[nik9000]

Probably a good call. I try to always keep them totally 1.0 for my sanity.

[dakrone]

LGTM, left one minor comment

[s1monw]

@dakrone pushed a new commit

[dakrone]

LGTM

[gkukal]

How can we avoid all primary shards to not go to same node? In heavy write use cases, all shards on same node can cause issue for that node. Any alternative?

[clintongormley]

@gkukal primaries and replicas do the same amount of work, so it doesn't matter if one node holds more primaries than another. The only exception to this rule is if you make heavy use of the update API. (see #8369)

[chaitd]

@gkukal
Have you resolved the problem! I have the problem too.

[chaitd]

@gkukal primaries and replicas do the same amount of work, so it doesn't matter if one node holds more primaries than another. The only exception to this rule is if you make heavy use of the update API. (see #8369)

@clintongormley
If I set the preference param, primaries and replicas do NOT do the same amount of work, and there is a problem that one node becomes hotspot.
Could u tell me how to solve it？

[chaitd]

@gkukal @clintongormley I get a solution. We can cancel an replica shard, and move the primary shard to the other node. Like this:
{ "commands": [{ "cancel": { "index": "indexName", "shard": 1, "node": "node2" } }, { "move": { "index": "indexName", "shard": 1, "from_node": "node1", "to_node": "node2" } }] }
And I test it, the es responses quickly and the result is successful.
Do I use the proper way?