[Tiered Caching] Moving query recomputation logic outside of write lock #14187

sgup432 · 2024-06-11T16:51:11Z

Description

This is related to performance improvement of Tiered Caching. Earlier during a cache miss, we were recomputing the query under the a single write lock which made the other threads wait and degraded performance. This change moves that logic outside the write lock. More details here #13989. We will follow up with another PR to make TieredCache segmented.

Related Issues

Resolves #[Issue number to be closed when this PR is merged]

Check List

Functionality includes testing.
API changes companion pull request created, if applicable.
~~[] Public documentation issue/PR created, if applicable.~~

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com>

github-actions · 2024-06-11T17:01:25Z

❌ Gradle check result for b0f6550: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com>

github-actions · 2024-06-11T18:27:48Z

❌ Gradle check result for 1dcb7e9: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

modules/cache-common/src/main/java/org/opensearch/cache/common/tier/TieredSpilloverCache.java

Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com>

github-actions · 2024-06-18T07:20:09Z

❌ Gradle check result for 6cf2248: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com>

github-actions · 2024-06-21T22:19:05Z

❌ Gradle check result for 5262362: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com>

Signed-off-by: Sagar <99425694+sgup432@users.noreply.github.com>

github-actions · 2024-06-21T23:18:56Z

❌ Gradle check result for 7fb3893: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2024-06-21T23:25:23Z

✅ Gradle check result for acdc6e7: SUCCESS

...s/cache-common/src/test/java/org/opensearch/cache/common/tier/TieredSpilloverCacheTests.java

Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com>

...s/cache-common/src/test/java/org/opensearch/cache/common/tier/TieredSpilloverCacheTests.java

github-actions · 2024-06-25T00:37:12Z

❌ Gradle check result for c7881df: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

sgup432 · 2024-06-25T00:41:12Z

Flaky test

[org.opensearch.action.admin.indices.create.RemoteCloneIndexIT.testCreateCloneIndexFailure]

#14292

Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com>

github-actions · 2024-06-25T02:24:25Z

❕ Gradle check result for 3686d2c: UNSTABLE

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

sohami · 2024-06-25T19:04:03Z

❕ Gradle check result for 3686d2c: UNSTABLE

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

#11997

opensearch-trigger-bot · 2024-06-25T19:05:03Z

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Navigate to the root of your repository
cd $(git rev-parse --show-toplevel)
# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/OpenSearch/backport-2.x 2.x
# Navigate to the new working tree
pushd ../.worktrees/OpenSearch/backport-2.x
# Create a new branch
git switch --create backport/backport-14187-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 563375de28b16870ab42b9fb4260127598d47d91
# Push it to GitHub
git push --set-upstream origin backport/backport-14187-to-2.x
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/OpenSearch/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport/backport-14187-to-2.x.

…ck (opensearch-project#14187) * Moving query recompute out of write lock Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com> * [Tiered Caching] Moving query recomputation logic outside of write lock Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com> * Adding java doc for the completable map Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com> * Changes to call future handler only once per key Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com> * Fixing spotless check Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com> * Added changelog Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com> * Addressing comments Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com> * Fixing gradle fail Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com> * Addressing comments to refactor unit test Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com> * minor UT refactor Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com> --------- Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com> Signed-off-by: Sagar <99425694+sgup432@users.noreply.github.com> Co-authored-by: Sagar Upadhyaya <upasagar@amazon.com>

* Fix flaky test in range aggregation yaml test (opensearch-project#14486) Signed-off-by: bowenlan-amzn <bowenlan23@gmail.com> * Use CODECOV_TOKEN (opensearch-project#14536) Signed-off-by: Prudhvi Godithi <pgodithi@amazon.com> * [Tiered Caching] Moving query recomputation logic outside of write lock (opensearch-project#14187) * Moving query recompute out of write lock Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com> * [Tiered Caching] Moving query recomputation logic outside of write lock Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com> * Adding java doc for the completable map Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com> * Changes to call future handler only once per key Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com> * Fixing spotless check Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com> * Added changelog Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com> * Addressing comments Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com> * Fixing gradle fail Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com> * Addressing comments to refactor unit test Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com> * minor UT refactor Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com> --------- Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com> Signed-off-by: Sagar <99425694+sgup432@users.noreply.github.com> Co-authored-by: Sagar Upadhyaya <upasagar@amazon.com> * Fix Flaky Test ClusterRerouteIT.testDelayWithALargeAmountOfShards (opensearch-project#14510) Signed-off-by: kkewwei kkewwei@163.com Signed-off-by: kkewwei kkewwei@163.com Signed-off-by: kkewwei <kkewwei@163.com> * Add doc for debugging rest tests (opensearch-project#14491) * add doc for debugging rest tests Signed-off-by: bowenlan-amzn <bowenlan23@gmail.com> * Update TESTING.md Co-authored-by: Marc Handalian <handalm@amazon.com> Signed-off-by: bowenlan-amzn <bowenlan23@gmail.com> * Address comment Signed-off-by: bowenlan-amzn <bowenlan23@gmail.com> --------- Signed-off-by: bowenlan-amzn <bowenlan23@gmail.com> Co-authored-by: Marc Handalian <handalm@amazon.com> * Fix flaky DefaultCacheStatsHolderTests (opensearch-project#14462) Signed-off-by: Peter Alfonsi <petealft@amazon.com> Co-authored-by: Peter Alfonsi <petealft@amazon.com> * Startreemapping (#24) * Star tree mapping changes with feature flag Signed-off-by: Bharathwaj G <bharath78910@gmail.com> * Fixes and tests Signed-off-by: Bharathwaj G <bharath78910@gmail.com> * addressing review comments and adding validations, integ tests Signed-off-by: Bharathwaj G <bharath78910@gmail.com> * addressing review comments Signed-off-by: Bharathwaj G <bharath78910@gmail.com> --------- Signed-off-by: Bharathwaj G <bharath78910@gmail.com> * OnHeap Star Tree Implementation Signed-off-by: Sarthak Aggarwal <sarthagg@amazon.com> * addressed nits Signed-off-by: Sarthak Aggarwal <sarthagg@amazon.com> * addressed major nits Signed-off-by: Sarthak Aggarwal <sarthagg@amazon.com> * includes Count Aggregator Signed-off-by: Sarthak Aggarwal <sarthagg@amazon.com> --------- Signed-off-by: bowenlan-amzn <bowenlan23@gmail.com> Signed-off-by: Prudhvi Godithi <pgodithi@amazon.com> Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com> Signed-off-by: Sagar <99425694+sgup432@users.noreply.github.com> Signed-off-by: kkewwei kkewwei@163.com Signed-off-by: kkewwei <kkewwei@163.com> Signed-off-by: Peter Alfonsi <petealft@amazon.com> Signed-off-by: Bharathwaj G <bharath78910@gmail.com> Signed-off-by: Sarthak Aggarwal <sarthagg@amazon.com> Co-authored-by: bowenlan-amzn <bowenlan23@gmail.com> Co-authored-by: Prudhvi Godithi <pgodithi@amazon.com> Co-authored-by: Sagar <99425694+sgup432@users.noreply.github.com> Co-authored-by: Sagar Upadhyaya <upasagar@amazon.com> Co-authored-by: kkewwei <kkewwei@163.com> Co-authored-by: Marc Handalian <handalm@amazon.com> Co-authored-by: Peter Alfonsi <peter.alfonsi@gmail.com> Co-authored-by: Peter Alfonsi <petealft@amazon.com> Co-authored-by: Sarthak Aggarwal <sarthagg@amazon.com>

* Startreemapping (#24) * Star tree mapping changes with feature flag Signed-off-by: Bharathwaj G <bharath78910@gmail.com> * Fixes and tests Signed-off-by: Bharathwaj G <bharath78910@gmail.com> * addressing review comments and adding validations, integ tests Signed-off-by: Bharathwaj G <bharath78910@gmail.com> * addressing review comments Signed-off-by: Bharathwaj G <bharath78910@gmail.com> --------- Signed-off-by: Bharathwaj G <bharath78910@gmail.com> * Onheap impl startree (#25) * Fix flaky test in range aggregation yaml test (opensearch-project#14486) Signed-off-by: bowenlan-amzn <bowenlan23@gmail.com> * Use CODECOV_TOKEN (opensearch-project#14536) Signed-off-by: Prudhvi Godithi <pgodithi@amazon.com> * [Tiered Caching] Moving query recomputation logic outside of write lock (opensearch-project#14187) * Moving query recompute out of write lock Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com> * [Tiered Caching] Moving query recomputation logic outside of write lock Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com> * Adding java doc for the completable map Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com> * Changes to call future handler only once per key Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com> * Fixing spotless check Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com> * Added changelog Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com> * Addressing comments Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com> * Fixing gradle fail Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com> * Addressing comments to refactor unit test Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com> * minor UT refactor Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com> --------- Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com> Signed-off-by: Sagar <99425694+sgup432@users.noreply.github.com> Co-authored-by: Sagar Upadhyaya <upasagar@amazon.com> * Fix Flaky Test ClusterRerouteIT.testDelayWithALargeAmountOfShards (opensearch-project#14510) Signed-off-by: kkewwei kkewwei@163.com Signed-off-by: kkewwei kkewwei@163.com Signed-off-by: kkewwei <kkewwei@163.com> * Add doc for debugging rest tests (opensearch-project#14491) * add doc for debugging rest tests Signed-off-by: bowenlan-amzn <bowenlan23@gmail.com> * Update TESTING.md Co-authored-by: Marc Handalian <handalm@amazon.com> Signed-off-by: bowenlan-amzn <bowenlan23@gmail.com> * Address comment Signed-off-by: bowenlan-amzn <bowenlan23@gmail.com> --------- Signed-off-by: bowenlan-amzn <bowenlan23@gmail.com> Co-authored-by: Marc Handalian <handalm@amazon.com> * Fix flaky DefaultCacheStatsHolderTests (opensearch-project#14462) Signed-off-by: Peter Alfonsi <petealft@amazon.com> Co-authored-by: Peter Alfonsi <petealft@amazon.com> * Startreemapping (#24) * Star tree mapping changes with feature flag Signed-off-by: Bharathwaj G <bharath78910@gmail.com> * Fixes and tests Signed-off-by: Bharathwaj G <bharath78910@gmail.com> * addressing review comments and adding validations, integ tests Signed-off-by: Bharathwaj G <bharath78910@gmail.com> * addressing review comments Signed-off-by: Bharathwaj G <bharath78910@gmail.com> --------- Signed-off-by: Bharathwaj G <bharath78910@gmail.com> * OnHeap Star Tree Implementation Signed-off-by: Sarthak Aggarwal <sarthagg@amazon.com> * addressed nits Signed-off-by: Sarthak Aggarwal <sarthagg@amazon.com> * addressed major nits Signed-off-by: Sarthak Aggarwal <sarthagg@amazon.com> * includes Count Aggregator Signed-off-by: Sarthak Aggarwal <sarthagg@amazon.com> --------- Signed-off-by: bowenlan-amzn <bowenlan23@gmail.com> Signed-off-by: Prudhvi Godithi <pgodithi@amazon.com> Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com> Signed-off-by: Sagar <99425694+sgup432@users.noreply.github.com> Signed-off-by: kkewwei kkewwei@163.com Signed-off-by: kkewwei <kkewwei@163.com> Signed-off-by: Peter Alfonsi <petealft@amazon.com> Signed-off-by: Bharathwaj G <bharath78910@gmail.com> Signed-off-by: Sarthak Aggarwal <sarthagg@amazon.com> Co-authored-by: bowenlan-amzn <bowenlan23@gmail.com> Co-authored-by: Prudhvi Godithi <pgodithi@amazon.com> Co-authored-by: Sagar <99425694+sgup432@users.noreply.github.com> Co-authored-by: Sagar Upadhyaya <upasagar@amazon.com> Co-authored-by: kkewwei <kkewwei@163.com> Co-authored-by: Marc Handalian <handalm@amazon.com> Co-authored-by: Peter Alfonsi <peter.alfonsi@gmail.com> Co-authored-by: Peter Alfonsi <petealft@amazon.com> Co-authored-by: Sarthak Aggarwal <sarthagg@amazon.com> --------- Signed-off-by: Bharathwaj G <bharath78910@gmail.com> Signed-off-by: bowenlan-amzn <bowenlan23@gmail.com> Signed-off-by: Prudhvi Godithi <pgodithi@amazon.com> Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com> Signed-off-by: Sagar <99425694+sgup432@users.noreply.github.com> Signed-off-by: kkewwei kkewwei@163.com Signed-off-by: kkewwei <kkewwei@163.com> Signed-off-by: Peter Alfonsi <petealft@amazon.com> Signed-off-by: Sarthak Aggarwal <sarthagg@amazon.com> Co-authored-by: bowenlan-amzn <bowenlan23@gmail.com> Co-authored-by: Prudhvi Godithi <pgodithi@amazon.com> Co-authored-by: Sagar <99425694+sgup432@users.noreply.github.com> Co-authored-by: Sagar Upadhyaya <upasagar@amazon.com> Co-authored-by: kkewwei <kkewwei@163.com> Co-authored-by: Marc Handalian <handalm@amazon.com> Co-authored-by: Peter Alfonsi <peter.alfonsi@gmail.com> Co-authored-by: Peter Alfonsi <petealft@amazon.com> Co-authored-by: Sarthak Aggarwal <sarthagg@amazon.com>

…ck (#14187) (#14565) * Moving query recompute out of write lock * [Tiered Caching] Moving query recomputation logic outside of write lock * Adding java doc for the completable map * Changes to call future handler only once per key * Fixing spotless check * Added changelog * Addressing comments * Fixing gradle fail * Addressing comments to refactor unit test * minor UT refactor --------- Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com> Signed-off-by: Sagar <99425694+sgup432@users.noreply.github.com> Co-authored-by: Sagar Upadhyaya <upasagar@amazon.com>

sgup432 added 2 commits June 4, 2024 14:31

Moving query recompute out of write lock

c06cdf2

Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com>

[Tiered Caching] Moving query recomputation logic outside of write lock

b0f6550

Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com>

sgup432 requested review from anasalkouz, andrross, Bukhtawar, CEHENKLE, dblock, dbwiddis, dreamer-89, gbbafna, kotwanikunal, mch2, msfroh, nknize, owaiskazi19, reta, Rishikesh1159, sachinpkale, saratvemulapalli, shwetathareja, sohami, tlfeng and VachaShah as code owners June 11, 2024 16:51

Adding java doc for the completable map

1dcb7e9

Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com>

jainankitk reviewed Jun 17, 2024

View reviewed changes

modules/cache-common/src/main/java/org/opensearch/cache/common/tier/TieredSpilloverCache.java Show resolved Hide resolved

jainankitk reviewed Jun 17, 2024

View reviewed changes

Changes to call future handler only once per key

6cf2248

Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com>

Addressing comments

5262362

Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com>

sgup432 and others added 2 commits June 21, 2024 15:26

Fixing gradle fail

7fb3893

Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com>

Merge branch 'main' into with_query_recompute_moved_out

acdc6e7

Signed-off-by: Sagar <99425694+sgup432@users.noreply.github.com>

prudhvigodithi mentioned this pull request Jun 23, 2024

[AUTOCUT] Gradle Check Flaky Test Report for IndicesRequestCacheIT prudhvigodithi/OpenSearch#27

Open

sohami reviewed Jun 24, 2024

View reviewed changes

Addressing comments to refactor unit test

c7881df

Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com>

sgup432 commented Jun 24, 2024

View reviewed changes

...s/cache-common/src/test/java/org/opensearch/cache/common/tier/TieredSpilloverCacheTests.java Show resolved Hide resolved

minor UT refactor

3686d2c

Signed-off-by: Sagar Upadhyaya <sagar.upadhyaya.121@gmail.com>

sohami approved these changes Jun 25, 2024

View reviewed changes

opensearch-ci-bot mentioned this pull request Jun 25, 2024

[AUTOCUT] Gradle Check Flaky Test Report for RemoteCloneIndexIT #14292

Open

sohami merged commit 563375d into opensearch-project:main Jun 25, 2024
31 checks passed

opensearch-trigger-bot bot added the backport-failed label Jun 25, 2024

sgup432 mentioned this pull request Jun 25, 2024

[BUG] org.opensearch.cache.common.tier.TieredSpilloverCacheTests.testComputeIfAbsentConcurrently is flaky #14544

Closed

sgup432 mentioned this pull request Jun 25, 2024

Fix flaky test TieredSpilloverCacheTests.testComputeIfAbsentConcurrently #14550

Merged

2 tasks

opensearch-ci-bot mentioned this pull request Jun 26, 2024

[AUTOCUT] Gradle Check Flaky Test Report for RemoteSplitIndexIT #14296

Open

sgup432 mentioned this pull request Jun 26, 2024

[Backport 2.x] [Tiered Caching] Moving query recomputation logic outside of write lock #14563

Closed

1 task

sgup432 mentioned this pull request Jun 26, 2024

[Backport 2.x][Tiered Caching] Moving query recomputation logic outside of write lock #14565

Merged

1 task

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Tiered Caching] Moving query recomputation logic outside of write lock #14187

[Tiered Caching] Moving query recomputation logic outside of write lock #14187

sgup432 commented Jun 11, 2024

github-actions bot commented Jun 11, 2024

github-actions bot commented Jun 11, 2024

github-actions bot commented Jun 18, 2024

github-actions bot commented Jun 21, 2024

github-actions bot commented Jun 21, 2024

github-actions bot commented Jun 21, 2024

github-actions bot commented Jun 25, 2024

sgup432 commented Jun 25, 2024

github-actions bot commented Jun 25, 2024

sohami commented Jun 25, 2024

opensearch-trigger-bot bot commented Jun 25, 2024

[Tiered Caching] Moving query recomputation logic outside of write lock #14187

[Tiered Caching] Moving query recomputation logic outside of write lock #14187

Conversation

sgup432 commented Jun 11, 2024

Description

Related Issues

Check List

github-actions bot commented Jun 11, 2024

github-actions bot commented Jun 11, 2024

github-actions bot commented Jun 18, 2024

github-actions bot commented Jun 21, 2024

github-actions bot commented Jun 21, 2024

github-actions bot commented Jun 21, 2024

github-actions bot commented Jun 25, 2024

sgup432 commented Jun 25, 2024

github-actions bot commented Jun 25, 2024

sohami commented Jun 25, 2024

opensearch-trigger-bot bot commented Jun 25, 2024