Adding back [Time series based workload desc order optimization through reverse segment read (#7244)] with fixes #7967

gashutos · 2023-06-08T15:03:21Z

Description

This PR adds back reverted PR #7244 due to bug introduced in re-index in this issue #7878.
The issue was, re-index operation on timeseries based workload was only able to re-index documents in its last segment and was skipping all other segments, hence we had data loss.
The reason behind this, we do early terminate the sort queries if lucene index is indexsorted or sort field is _doc. As per TopFieldCollector code, in case of IndexSort or _doc field sort, it does earlyTerminate and searches only very first segment. ReIndex plugin queries scroll query on _doc field in asc order. Hence the issue was with Reindex.

In this PR, I am adding check not to enable this optimization in case if sorting is not on @timestamp field as well if it is IndexSorted.

Related Issues

Resolves #7878

Check List

New functionality includes testing.
- All tests pass
New functionality has been documented.
- New functionality has javadoc added
Commits are signed per the DCO using --signoff
Commit changes are listed out in CHANGELOG.md file (See: Changelog)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

…rough reverse segment read (opensearch-project#7244)" (opensearch-project#7892)" This reverts commit bb26536. Signed-off-by: gashutos <gashutos@amazon.com>

@timestamp

…also ASC order reverse should only consider in @timestamp field Signed-off-by: gashutos <gashutos@amazon.com>

gashutos · 2023-06-08T15:07:12Z

Please reivew second commit 59285c02385bb13bf98f710c6e683b67a2af041c only, The first commit in this PR contains just revert of #7244

@reta @andrross @Hailong-am.

many thanks for Hailong-am for catching this and @andrross to help reverting to unblock 2.8 release.

Signed-off-by: gashutos <gashutos@amazon.com>

github-actions · 2023-06-08T15:40:48Z

Gradle Check (Jenkins) Run Completed with:

RESULT: UNSTABLE ❕
TEST FAILURES:

      1 org.opensearch.search.backpressure.SearchBackpressureIT.testSearchShardTaskCancellationWithHighCpu
      1 org.opensearch.remotestore.SegmentReplicationUsingRemoteStoreIT.testDropPrimaryDuringReplication
      1 org.opensearch.remotestore.RemoteStoreRefreshListenerIT.testRemoteRefreshRetryOnFailure
      1 org.opensearch.indices.replication.SegmentReplicationIT.testScrollCreatedOnReplica

URL: https://build.ci.opensearch.org/job/gradle-check/17138/
CommitID: d01c3c1
Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

github-actions · 2023-06-08T15:44:29Z

Gradle Check (Jenkins) Run Completed with:

RESULT: UNSTABLE ❕
TEST FAILURES:

      1 org.opensearch.search.backpressure.SearchBackpressureIT.testSearchShardTaskCancellationWithHighCpu
      1 org.opensearch.remotestore.SegmentReplicationUsingRemoteStoreIT.testDropPrimaryDuringReplication

URL: https://build.ci.opensearch.org/job/gradle-check/17139/
CommitID: 8d8f924
Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

codecov · 2023-06-08T15:47:01Z

Codecov Report

Merging #7967 (52cd6bd) into main (584617e) will increase coverage by 0.53%.
The diff coverage is 35.71%.

@@             Coverage Diff              @@
##               main    #7967      +/-   ##
============================================
+ Coverage     70.87%   71.40%   +0.53%     
- Complexity    56504    56940     +436     
============================================
  Files          4719     4719              
  Lines        267408   267446      +38     
  Branches      39196    39210      +14     
============================================
+ Hits         189521   190970    +1449     
+ Misses        61861    60657    -1204     
+ Partials      16026    15819     -207

Impacted Files	Coverage Δ
.../main/java/org/opensearch/index/IndexSettings.java	`85.93% <0.00%> (-0.67%)`	⬇️
...n/java/org/opensearch/index/MergePolicyConfig.java	`74.66% <ø> (ø)`
...va/org/opensearch/index/engine/InternalEngine.java	`75.35% <0.00%> (+0.58%)`	⬆️
...ensearch/search/internal/ContextIndexSearcher.java	`71.60% <20.00%> (-5.73%)`	⬇️
...va/org/opensearch/cluster/metadata/DataStream.java	`84.15% <22.22%> (-6.06%)`	⬇️
...ava/org/opensearch/index/mapper/MappingLookup.java	`92.79% <50.00%> (-0.79%)`	⬇️
...in/java/org/opensearch/index/shard/IndexShard.java	`70.63% <66.66%> (-0.15%)`	⬇️
...java/org/opensearch/index/engine/EngineConfig.java	`98.50% <100.00%> (+0.04%)`	⬆️
...g/opensearch/index/engine/EngineConfigFactory.java	`92.95% <100.00%> (+0.10%)`	⬆️

... and 456 files with indirect coverage changes

reta · 2023-06-08T18:07:06Z

The issue was, re-index operation on timeseries based workload was only able to re-index documents in its last segment and was skipping all other segments, hence we had data loss.

@gashutos would be great if we could somehow add test case(s) for that, what do you think?

andrross · 2023-06-09T17:29:43Z

would be great if we could somehow add test case(s) for that, what do you think?

Agreed. @gashutos Can you add an integration or unit test to ensure we're not still hitting the early terminate behavior?

…termination Signed-off-by: gashutos <gashutos@amazon.com>

gashutos · 2023-06-12T09:55:15Z

Report

would be great if we could somehow add test case(s) for that, what do you think?

Agreed. @gashutos Can you add an integration or unit test to ensure we're not still hitting the early terminate behavior?

Oops, I missed adding file to commit. Just added now.
Sorry, just got back from short vacation so delay in response. Added integ test to commit now.

github-actions · 2023-06-12T10:24:12Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/17301/
CommitID: 106f43a
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-06-12T10:56:50Z

Gradle Check (Jenkins) Run Completed with:

RESULT: UNSTABLE ❕
TEST FAILURES:

      2 org.opensearch.remotestore.RemoteStoreRefreshListenerIT.testRemoteRefreshRetryOnFailure
      1 org.opensearch.search.backpressure.SearchBackpressureIT.testSearchShardTaskCancellationWithHighCpu
      1 org.opensearch.action.admin.indices.create.SplitIndexIT.testSplitIndexPrimaryTerm

URL: https://build.ci.opensearch.org/job/gradle-check/17304/
CommitID: 52cd6bd
Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

github-actions · 2023-06-12T11:01:21Z

Gradle Check (Jenkins) Run Completed with:

RESULT: UNSTABLE ❕
TEST FAILURES:

      1 org.opensearch.search.backpressure.SearchBackpressureIT.testSearchShardTaskCancellationWithHighCpu
      1 org.opensearch.remotestore.SegmentReplicationUsingRemoteStoreIT.testReplicaHasDiffFilesThanPrimary
      1 org.opensearch.indices.replication.SegmentReplicationIT.testScrollCreatedOnReplica
      1 org.opensearch.indices.replication.SegmentReplicationIT.testReplicationPostDeleteAndForceMerge

URL: https://build.ci.opensearch.org/job/gradle-check/17303/
CommitID: 52cd6bd
Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

gashutos · 2023-06-12T13:22:53Z

would be great if we could somehow add test case(s) for that, what do you think?

Agreed. @gashutos Can you add an integration or unit test to ensure we're not still hitting the early terminate behavior?

Oops, I missed adding file to commit. Just added now.
Sorry, just got back from short vacation so delay in response. Added integ test to commit now.

I was trying a lot about adding unit test to cover codcov happy. I think for this scenario, Integ tests are fine but unit tests doesnt add much value even if we end up covering lines. Let me know if you guys think otherwise.

opensearch-trigger-bot · 2023-06-12T17:00:50Z

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/backport-2.x 2.x
# Navigate to the new working tree
pushd ../.worktrees/backport-2.x
# Create a new branch
git switch --create backport/backport-7967-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 5c3225692dcea1eddbb3e76ae19f47de5ea23a96
# Push it to GitHub
git push --set-upstream origin backport/backport-7967-to-2.x
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport/backport-7967-to-2.x.

reta · 2023-06-12T19:29:11Z

@gashutos mind please to create a manual backport to 2.x, thank you

@timestamp

…gh reverse segment read (opensearch-project#7244)] with fixes (opensearch-project#7967) * Revert "Revert "Time series based workload desc order optimization through reverse segment read (opensearch-project#7244)" (opensearch-project#7892)" This reverts commit bb26536. Signed-off-by: gashutos <gashutos@amazon.com> * Enable time series optimization only if it is not IndexSorted index, also ASC order reverse should only consider in @timestamp field Signed-off-by: gashutos <gashutos@amazon.com> * Modifying CHANGELOG Signed-off-by: gashutos <gashutos@amazon.com> * Adding integ test for scroll API where sort by _doc is getting early termination Signed-off-by: gashutos <gashutos@amazon.com> --------- Signed-off-by: gashutos <gashutos@amazon.com>

…gh reverse segment read (#7244)] with fixes (#7967) (#8037) Signed-off-by: gashutos <gashutos@amazon.com>

…gh reverse segment read (opensearch-project#7244)] with fixes (opensearch-project#7967) (opensearch-project#8037) Signed-off-by: gashutos <gashutos@amazon.com>

@timestamp

…gh reverse segment read (opensearch-project#7244)] with fixes (opensearch-project#7967) * Revert "Revert "Time series based workload desc order optimization through reverse segment read (opensearch-project#7244)" (opensearch-project#7892)" This reverts commit bb26536. Signed-off-by: gashutos <gashutos@amazon.com> * Enable time series optimization only if it is not IndexSorted index, also ASC order reverse should only consider in @timestamp field Signed-off-by: gashutos <gashutos@amazon.com> * Modifying CHANGELOG Signed-off-by: gashutos <gashutos@amazon.com> * Adding integ test for scroll API where sort by _doc is getting early termination Signed-off-by: gashutos <gashutos@amazon.com> --------- Signed-off-by: gashutos <gashutos@amazon.com> Signed-off-by: Rishab Nahata <rnnahata@amazon.com>

@timestamp

…gh reverse segment read (opensearch-project#7244)] with fixes (opensearch-project#7967) * Revert "Revert "Time series based workload desc order optimization through reverse segment read (opensearch-project#7244)" (opensearch-project#7892)" This reverts commit bb26536. Signed-off-by: gashutos <gashutos@amazon.com> * Enable time series optimization only if it is not IndexSorted index, also ASC order reverse should only consider in @timestamp field Signed-off-by: gashutos <gashutos@amazon.com> * Modifying CHANGELOG Signed-off-by: gashutos <gashutos@amazon.com> * Adding integ test for scroll API where sort by _doc is getting early termination Signed-off-by: gashutos <gashutos@amazon.com> --------- Signed-off-by: gashutos <gashutos@amazon.com> Signed-off-by: Shivansh Arora <hishiv@amazon.com>

gashutos added 2 commits June 8, 2023 20:21

Revert "Revert "Time series based workload desc order optimization th…

59285c0

…rough reverse segment read (opensearch-project#7244)" (opensearch-project#7892)" This reverts commit bb26536. Signed-off-by: gashutos <gashutos@amazon.com>

Enable time series optimization only if it is not IndexSorted index, …

d01c3c1

…also ASC order reverse should only consider in @timestamp field Signed-off-by: gashutos <gashutos@amazon.com>

gashutos requested review from reta, anasalkouz, andrross, Bukhtawar, CEHENKLE, dblock, gbbafna, setiah, kartg, kotwanikunal, mch2, nknize, owaiskazi19, Rishikesh1159, ryanbogan, saratvemulapalli, shwetathareja, dreamer-89, tlfeng, VachaShah and dbwiddis as code owners June 8, 2023 15:03

Modifying CHANGELOG

8d8f924

Signed-off-by: gashutos <gashutos@amazon.com>

reta approved these changes Jun 8, 2023

View reviewed changes

Adding integ test for scroll API where sort by _doc is getting early …

106f43a

…termination Signed-off-by: gashutos <gashutos@amazon.com>

Merge branch 'opensearch-project:main' into main

52cd6bd

andrross approved these changes Jun 12, 2023

View reviewed changes

andrross added the backport 2.x Backport to 2.x branch label Jun 12, 2023

andrross merged commit 5c32256 into opensearch-project:main Jun 12, 2023
19 of 20 checks passed

gashutos mentioned this pull request Jun 13, 2023

[Backport 2.x] [#7967] Adding back [Time series based workload desc order optimization throu… #8037

Merged

gbbafna pushed a commit that referenced this pull request Jun 13, 2023

Adding back [Time series based workload desc order optimization throu…

2b83949

…gh reverse segment read (#7244)] with fixes (#7967) (#8037) Signed-off-by: gashutos <gashutos@amazon.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Adding back [Time series based workload desc order optimization through reverse segment read (#7244)] with fixes #7967

Adding back [Time series based workload desc order optimization through reverse segment read (#7244)] with fixes #7967

gashutos commented Jun 8, 2023 •

edited

gashutos commented Jun 8, 2023

github-actions bot commented Jun 8, 2023

github-actions bot commented Jun 8, 2023

codecov bot commented Jun 8, 2023 •

edited

reta commented Jun 8, 2023

andrross commented Jun 9, 2023

gashutos commented Jun 12, 2023

github-actions bot commented Jun 12, 2023

github-actions bot commented Jun 12, 2023

github-actions bot commented Jun 12, 2023

gashutos commented Jun 12, 2023

opensearch-trigger-bot bot commented Jun 12, 2023

reta commented Jun 12, 2023

Adding back [Time series based workload desc order optimization through reverse segment read (#7244)] with fixes #7967

Adding back [Time series based workload desc order optimization through reverse segment read (#7244)] with fixes #7967

Conversation

gashutos commented Jun 8, 2023 • edited

Description

Related Issues

Check List

gashutos commented Jun 8, 2023

github-actions bot commented Jun 8, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Jun 8, 2023

Gradle Check (Jenkins) Run Completed with:

codecov bot commented Jun 8, 2023 • edited

Codecov Report

reta commented Jun 8, 2023

andrross commented Jun 9, 2023

gashutos commented Jun 12, 2023

github-actions bot commented Jun 12, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Jun 12, 2023

Gradle Check (Jenkins) Run Completed with:

github-actions bot commented Jun 12, 2023

Gradle Check (Jenkins) Run Completed with:

gashutos commented Jun 12, 2023

opensearch-trigger-bot bot commented Jun 12, 2023

reta commented Jun 12, 2023

gashutos commented Jun 8, 2023 •

edited

codecov bot commented Jun 8, 2023 •

edited