[Segment Replication] Bump segment infos counter before commit during replica promotion

[dreamer-89]

Signed-off-by: Suraj Singh surajrider@gmail.com

Description

This change bumps the in memory SegmentInfos's counter in order to decrease the chances of conflict on replica due to same file from different primaries. This solution does not guarantees that conflict will never happen but rather decreases the probability of happening. It does not harm to bump the counter. Increasing counter helps in avoiding this conflict as counter is used to generate new segment file names, thus higher counter create files with different names. The ideal solution is to identify the counter from previous primary.

Credits: @mch2 for suggesting this smart workaround.

Issues Resolved

#4295

Alternatives

1. Add a new variable counter in cluster state, during replica promotion, shard can refer the cluster state to fetch existing primary's counter. This involves more changes to keep cluster state upto date.
2. [Segment Replication] Track last primary term used for copied files to ignore different files from new primary term #4304 Resolve difference on replica nodes. This is slightly complex as it needs re-naming or deleting existing conflicting files from store.
3. Make timed transport request to in-sync replicas to fetch counter value. This is problematic as older primary (having latest SegmentInfos) can be unreachable during that time.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: SUCCESS ✅
• URL: https://build.ci.opensearch.org/job/gradle-check/2395/
• CommitID: 6995db4

[codecov-commenter]

Codecov Report

Merging #4365 (158e753) into main (70d911c) will decrease coverage by 0.32%.
The diff coverage is 100.00%.

@@             Coverage Diff              @@
##               main    #4365      +/-   ##
============================================
- Coverage     71.08%   70.75%   -0.33%     
+ Complexity    57622    57364     -258     
============================================
  Files          4616     4616              
  Lines        275441   275443       +2     
  Branches      40314    40314              
============================================
- Hits         195801   194894     -907     
- Misses        63602    64214     +612     
- Partials      16038    16335     +297     

Impacted Files	Coverage Δ
.../opensearch/index/engine/NRTReplicationEngine.java	77.27% <100.00%> (+0.34%)	⬆️
...java/org/opensearch/client/indices/DataStream.java	0.00% <0.00%> (-76.09%)	⬇️
.../opensearch/client/indices/CloseIndexResponse.java	17.50% <0.00%> (-73.75%)	⬇️
...ava/org/opensearch/action/NoSuchNodeException.java	0.00% <0.00%> (-50.00%)	⬇️
...cluster/coordination/PendingClusterStateStats.java	20.00% <0.00%> (-48.00%)	⬇️
.../admin/cluster/reroute/ClusterRerouteResponse.java	60.00% <0.00%> (-40.00%)	⬇️
...luster/routing/allocation/RoutingExplanations.java	62.06% <0.00%> (-37.94%)	⬇️
...java/org/opensearch/threadpool/ThreadPoolInfo.java	56.25% <0.00%> (-37.50%)	⬇️
...ain/java/org/opensearch/search/sort/MinAndMax.java	63.15% <0.00%> (-36.85%)	⬇️
...cluster/routing/allocation/RerouteExplanation.java	70.00% <0.00%> (-30.00%)	⬇️
... and 464 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

[mch2]

I'm not opposed to this as a "best-effort" approach but it would be great if we could somehow fetch and increment from the previous primary's counter instead of this replica's latest infos. latestSegmentInfos on the replica here could still potentially be more than 10 generations behind and get promoted as primary.

I'm not sure exactly how we'd do that though with any accuracy.

I'm ok with this as best-effort, because we can still fail & recover the replicas that have conflicts as needed and attempt to select the furthest ahead during promotion.

[dreamer-89]

I'm not sure exactly how we'd do that though with any accuracy.

I added few alternatives in the description. But this solution works well and prevents file conflicts on nodes. I ran tests in continuation (with some tweaks on refresh/commits) but did not face any issue or side-effects. If this doesn't hurt let's move forward with this solution. This is also in general helpful not just for #4295 but overall working of segment replication.

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: SUCCESS ✅
• URL: https://build.ci.opensearch.org/job/gradle-check/2555/
• CommitID: 158e753

[opensearch-trigger-bot]

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 1

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add .worktrees/backport-2.x 2.x
# Navigate to the new working tree
cd .worktrees/backport-2.x
# Create a new branch
git switch --create backport/backport-4365-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 c885686b0fb12d9b0397d38c37518c65cbb466c5
# Push it to GitHub
git push --set-upstream origin backport/backport-4365-to-2.x
# Go back to the original working tree
cd ../..
# Delete the working tree
git worktree remove .worktrees/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport/backport-4365-to-2.x.

[hydrogen666]

I think you should take SegmentCommitInfo#delGen, SegmentCommitInfo#fieldInfosGen, SegmentCommitInfo#docValuesGen into consideration too. Especially SegmentCommitInfo#delGen, if we are don't bump this counter, it may cause duplicate *.liv file generation.

[dreamer-89]

I think you should take SegmentCommitInfo#delGen, SegmentCommitInfo#fieldInfosGen, SegmentCommitInfo#docValuesGen into consideration too. Especially SegmentCommitInfo#delGen, if we are don't bump this counter, it may cause duplicate *.liv file generation.

Thank you so much @hydrogen666 for raising this. I was not aware of this. Created #5701 to track this.