Consider only forcemerging primary shards

[n0othing]

Describe the feature:

Today, running a _forcemerge?max_num_segments=1 against an index will result in both primaries and replicas asynchronously merging down to a single segment. This ultimately results in all shard copies containing a single segment that look different from one another (e.g the primary might have segment _c, while the replica might have segment _e).

It might be better to only forcemerge the primary shard --> then recover the replicas, resulting in identical segments for all shard copies.

Several benefits come to mind:

• Having "different" looking segments amongst shard copies can result in excessive Snapshot repo disk usage (e.g replica gets promoted to primary and its single segment looks different so it gets backed up in full).
• Should avoid the "bouncing results" problem.
• Less load on nodes hosting replicas (assuming a recovery is cheaper than full blown merging)

[elasticmachine]

Pinging @elastic/es-core-features (:Core/Features/Features)

[elasticmachine]

Pinging @elastic/es-distributed (:Distributed/Distributed)

[dnhatn]

@n0othing Thanks for raising this. I don't think we should implement this. Even without force merge, segment files on the primary and replicas would be very different because of: (1) concurrent indexing, (2) different refresh schedule (depending on search requests and memory pressure), (3) background merges.