Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Tiered Caching] Stats rework (1/3): Interfaces and implementations for individual tiers #12531

Merged
merged 82 commits into from
Apr 12, 2024

Conversation

peteralfonsi
Copy link
Contributor

@peteralfonsi peteralfonsi commented Mar 5, 2024

Description

As part of tiered caching stats, changes the common ICache interface to use ICacheKey as its key. This key contains dimensions (for example, shard ID, index name, or tier) that can be used to aggregate stats. Also changes the CacheStats interface to store the necessary cache stats, and to support getting stats either as a total or aggregated by these dimensions.

Integrates these changes with OpenSearchOnHeapCache and EhcacheDiskCache. The stats implementation for the TieredSpilloverCache will be in a followup PR.

Check List

  • New functionality includes testing.
    • All tests pass
  • New functionality has been documented.
    • New functionality has javadoc added
  • Failing checks are inspected and point to the corresponding known issue(s) (See: Troubleshooting Failing Builds)
  • Commits are signed per the DCO using --signoff
  • Commit changes are listed out in CHANGELOG.md file (See: Changelog)
  • [N/A] Public documentation issue/PR created

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Peter Alfonsi added 12 commits March 4, 2024 11:50
Signed-off-by: Peter Alfonsi <petealft@amazon.com>
Signed-off-by: Peter Alfonsi <petealft@amazon.com>
Signed-off-by: Peter Alfonsi <petealft@amazon.com>
Signed-off-by: Peter Alfonsi <petealft@amazon.com>
Signed-off-by: Peter Alfonsi <petealft@amazon.com>
Signed-off-by: Peter Alfonsi <petealft@amazon.com>
Signed-off-by: Peter Alfonsi <petealft@amazon.com>
Signed-off-by: Peter Alfonsi <petealft@amazon.com>
Signed-off-by: Peter Alfonsi <petealft@amazon.com>
Signed-off-by: Peter Alfonsi <petealft@amazon.com>
Signed-off-by: Peter Alfonsi <petealft@amazon.com>
Signed-off-by: Peter Alfonsi <petealft@amazon.com>
Copy link
Contributor

❌ Gradle check result for 84381f0: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Signed-off-by: Peter Alfonsi <petealft@amazon.com>
Copy link
Contributor

❌ Gradle check result for 50da5dc: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Copy link
Contributor

❌ Gradle check result for 2f59ee7: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Copy link
Contributor

❌ Gradle check result for da9e485: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Copy link
Contributor

❌ Gradle check result for f60fb08: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Copy link
Contributor

❌ Gradle check result for 5330225: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Signed-off-by: Peter Alfonsi <petealft@amazon.com>
Copy link
Contributor

❌ Gradle check result for f465a22: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Signed-off-by: Peter Alfonsi <petealft@amazon.com>
Copy link
Contributor

✅ Gradle check result for 5283470: SUCCESS

Copy link

codecov bot commented Apr 12, 2024

Codecov Report

Attention: Patch coverage is 92.70073% with 30 lines in your changes are missing coverage. Please review.

Project coverage is 71.62%. Comparing base (b15cb0c) to head (54e12a3).
Report is 158 commits behind head on main.

Files Patch % Lines
...in/java/org/opensearch/common/cache/ICacheKey.java 75.75% 3 Missing and 5 partials ⚠️
.../org/opensearch/common/cache/stats/CacheStats.java 87.50% 4 Missing and 3 partials ⚠️
...pensearch/common/cache/stats/CacheStatsHolder.java 95.32% 1 Missing and 4 partials ⚠️
.../opensearch/cache/store/disk/EhcacheDiskCache.java 93.75% 2 Missing and 1 partial ⚠️
...h/common/cache/serializer/ICacheKeySerializer.java 91.17% 3 Missing ⚠️
...search/cache/common/tier/TieredSpilloverCache.java 83.33% 1 Missing ⚠️
...search/common/cache/stats/ImmutableCacheStats.java 96.42% 0 Missing and 1 partial ⚠️
.../common/cache/stats/ImmutableCacheStatsHolder.java 96.55% 1 Missing ⚠️
...arch/common/cache/store/OpenSearchOnHeapCache.java 96.96% 0 Missing and 1 partial ⚠️
Additional details and impacted files
@@             Coverage Diff              @@
##               main   #12531      +/-   ##
============================================
+ Coverage     71.42%   71.62%   +0.19%     
- Complexity    59978    60706     +728     
============================================
  Files          4985     5039      +54     
  Lines        282275   285316    +3041     
  Branches      40946    41326     +380     
============================================
+ Hits         201603   204344    +2741     
- Misses        63999    64092      +93     
- Partials      16673    16880     +207     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Signed-off-by: Peter Alfonsi <petealft@amazon.com>
Copy link
Contributor

❕ Gradle check result for 54e12a3: UNSTABLE

  • TEST FAILURES:
      2 org.opensearch.cluster.MinimumClusterManagerNodesIT.testThreeNodesNoClusterManagerBlock
      1 org.opensearch.repositories.azure.AzureBlobStoreRepositoryTests.testSnapshotWithLargeSegmentFiles

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

@msfroh msfroh merged commit cc22310 into opensearch-project:main Apr 12, 2024
31 checks passed
peteralfonsi added a commit to peteralfonsi/OpenSearch that referenced this pull request Apr 16, 2024
…or individual tiers (opensearch-project#12531)

As part of tiered caching stats, changes the common ICache interface to use ICacheKey as its key. This key contains
dimensions (for example, shard ID, index name, or tier) that can be used to aggregate stats. Also changes the
CacheStats interface to store the necessary cache stats, and to support getting stats either as a total or aggregated by
these dimensions.

Integrates these changes with OpenSearchOnHeapCache and EhcacheDiskCache. The stats implementation for the
TieredSpilloverCache will be in a followup PR.

---------

Signed-off-by: Peter Alfonsi <petealft@amazon.com>
Co-authored-by: Peter Alfonsi <petealft@amazon.com>
(cherry picked from commit cc22310)
dblock pushed a commit that referenced this pull request Apr 25, 2024
…plementations for individual tiers (#13247)

* [Tiered Caching] Stats rework (1/3): Interfaces and implementations for individual tiers (#12531)

As part of tiered caching stats, changes the common ICache interface to use ICacheKey as its key. This key contains
dimensions (for example, shard ID, index name, or tier) that can be used to aggregate stats. Also changes the
CacheStats interface to store the necessary cache stats, and to support getting stats either as a total or aggregated by
these dimensions.

Integrates these changes with OpenSearchOnHeapCache and EhcacheDiskCache. The stats implementation for the
TieredSpilloverCache will be in a followup PR.

---------

Signed-off-by: Peter Alfonsi <petealft@amazon.com>
Co-authored-by: Peter Alfonsi <petealft@amazon.com>
(cherry picked from commit cc22310)

* Rerunning gradle check

Signed-off-by: Peter Alfonsi <petealft@amazon.com>

---------

Signed-off-by: Peter Alfonsi <petealft@amazon.com>
Co-authored-by: Peter Alfonsi <petealft@amazon.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants