[Remove] Segment memory estimation and tracking #2029

nknize · 2022-02-01T18:40:13Z

Lucene 9 removed CodecReader#ramBytesUsed and all file formats that no longer
consume large amounts of memory. As a result RAM estimation for segments is no
longer possible and is removed in order to upgrade to lucene 9. backwards compatibility is retained for the transport layer.

opensearch-ci-bot · 2022-02-01T18:42:03Z

Can one of the admins verify this patch?

opensearch-ci-bot · 2022-02-01T18:56:34Z

❌ Gradle Check failure a649a14fa58df6113d02ce12f8a259b3ef625ff9
Log 2151

Reports 2151

opensearch-ci-bot · 2022-02-01T19:54:51Z

❌ Gradle Check failure ee32118c2ddcf37eaeb5bc66feecb8555a6d617a
Log 2155

Reports 2155

opensearch-ci-bot · 2022-02-01T20:41:42Z

❌ Gradle Check failure b3f066040127120242b920df90ed956ef022387d
Log 2158

Reports 2158

Lucene 9 removed CodecReader#ramBytesUsed and all file formats that no longer consume large amounts of memory. As a result RAM estimation for segments is no longer possible and needs to be removed. backwards compatibility is retained for the tranport layer. Signed-off-by: Nicholas Walter Knize <nknize@apache.org>

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>

opensearch-ci-bot · 2022-02-01T22:47:29Z

✅ Gradle Check success 693ed1c
Log 2161

Reports 2161

adnapibar

👍

adnapibar · 2022-02-04T18:02:05Z

server/src/main/java/org/opensearch/rest/action/cat/RestIndicesAction.java

-            table.addCell(totalStats.getSegments() == null ? null : totalStats.getSegments().getMemory());
-            table.addCell(primaryStats.getSegments() == null ? null : primaryStats.getSegments().getMemory());
+            table.addCell(totalStats.getSegments() == null ? null : totalStats.getSegments().getZeroMemory());
+            table.addCell(primaryStats.getSegments() == null ? null : primaryStats.getSegments().getZeroMemory());



why are we not removing these instead - for backward compatibility?

Yes, they're needed for bwc so we just report 0 memory usage.

reta · 2022-02-04T18:30:05Z

server/src/main/java/org/opensearch/index/engine/RamAccountingRefreshListener.java

-
-    @Override
-    public void accept(OpenSearchDirectoryReader reader, OpenSearchDirectoryReader previousReader) {
-        final CircuitBreaker breaker = breakerService.getBreaker(CircuitBreaker.ACCOUNTING);


Since we removed it, I believe CircuitBreaker.ACCOUNTING becomes useless as well, seems like this is the only place where it was managed (please correct me if I am wrong).

Ah, good catch. We can remove the Accountable. I'll take care of that here as well.

On second thought. I'm going to remove that in a follow up PR to keep the changes contained.

`OpenSearch` 2.0 uses newer version of `Lucene` (9.0) which doesn't provide segments stats info. Ref: opensearch-project/OpenSearch#2029 opensearch-project/OpenSearch#1109 See also history for `server/src/main/java/org/opensearch/index/engine/SegmentsStats.java` in `OpenSearch` repo. Signed-off-by: Yury Fridlyand <yuryf@bitquilltech.com>

* Update nuget packages (opensearch-project#88) * Cleanup - removing stale dependencies. * Fix 2 typos in scripts project * Update `scripts` project file to include all relevant objects. * Rename `master` node role to `cluster_manager` as it was done in OpenSearch. Ref: opensearch-project/OpenSearch#2480 * Remove validation for indices segments stats. `OpenSearch` 2.0 uses newer version of `Lucene` (9.0) which doesn't provide segments stats info. Ref: opensearch-project/OpenSearch#2029 opensearch-project/OpenSearch#1109 See also history for `server/src/main/java/org/opensearch/index/engine/SegmentsStats.java` in `OpenSearch` repo. * Remove tests for `_type` validation in mapping APIs as it was removed from `OpenSearch`. Ref: opensearch-project/OpenSearch#2238 opensearch-project/OpenSearch#2480 * Remove usage of deprecated `search.remote` settings. Ref: opensearch-project/OpenSearch#1870 * Update abstractions package - patch to support OpenSearch 2.0. Update integration workflow to run tests on OpenSearch 2.0. * Rename `master_timeout` to `cluster_manager_timeout` in all APIs where it is used. * Enrich comments to already renamed `CatMaster`/`CatClusterManager` API. * Rename in `/_cluster/stats`/`cluster.stats` and `/_cluster/state`/`cluster.state`. * Add deprecation info. * Rename in comments. * Rename in test data. * Renamings in tests including `MasterEligible`, but mark it is obsolete. * Rename branch reference in scripting. * Mark `indices.exists_type`/`TypeExists` APIs as deprecated. * Update compatibility matrix and include it into `sln` file. * Add deprecation notice to all reference of `include_type_name`/`IncludeTypeName`. * Update compatibility matrix. * Remove `OpenDistro` compatibility notice. * Update repo link. * Add small README for each project being released. * Address PR opensearch-project#51 feedback. Signed-off-by: Yury-Fridlyand <yuryf@bitquilltech.com>

* Update nuget packages (#88) * Cleanup - removing stale dependencies. * Fix 2 typos in scripts project * Update `scripts` project file to include all relevant objects. * Rename `master` node role to `cluster_manager` as it was done in OpenSearch. Ref: opensearch-project/OpenSearch#2480 * Remove validation for indices segments stats. `OpenSearch` 2.0 uses newer version of `Lucene` (9.0) which doesn't provide segments stats info. Ref: opensearch-project/OpenSearch#2029 opensearch-project/OpenSearch#1109 See also history for `server/src/main/java/org/opensearch/index/engine/SegmentsStats.java` in `OpenSearch` repo. * Remove tests for `_type` validation in mapping APIs as it was removed from `OpenSearch`. Ref: opensearch-project/OpenSearch#2238 opensearch-project/OpenSearch#2480 * Remove usage of deprecated `search.remote` settings. Ref: opensearch-project/OpenSearch#1870 * Update abstractions package - patch to support OpenSearch 2.0. Update integration workflow to run tests on OpenSearch 2.0. * Rename `master_timeout` to `cluster_manager_timeout` in all APIs where it is used. * Enrich comments to already renamed `CatMaster`/`CatClusterManager` API. * Rename in `/_cluster/stats`/`cluster.stats` and `/_cluster/state`/`cluster.state`. * Add deprecation info. * Rename in comments. * Rename in test data. * Renamings in tests including `MasterEligible`, but mark it is obsolete. * Rename branch reference in scripting. * Mark `indices.exists_type`/`TypeExists` APIs as deprecated. * Update compatibility matrix and include it into `sln` file. * Add deprecation notice to all reference of `include_type_name`/`IncludeTypeName`. * Update compatibility matrix. * Remove `OpenDistro` compatibility notice. * Update repo link. * Add small README for each project being released. * Address PR #51 feedback. Signed-off-by: Yury-Fridlyand <yuryf@bitquilltech.com> Signed-off-by: Yury-Fridlyand <yuryf@bitquilltech.com>

nknize added v2.0.0 Version 2.0.0 deprecate labels Feb 1, 2022

nknize requested a review from a team as a code owner February 1, 2022 18:40

nknize force-pushed the remove/segmentMemoryEstimate branch from a649a14 to ee32118 Compare February 1, 2022 19:30

nknize mentioned this pull request Feb 1, 2022

Remove LegacyVersion.V_6.x Constants and Deprecated Code in OpenSearch 2.0 #1674

Closed

19 tasks

nknize added 4 commits February 1, 2022 16:04

fix test failure

d60171c

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>

remove unnecessary assert from ExternalTestCluster

4bf5a51

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>

fix RestSegmentsAction to pass long not ByteSizeValue

693ed1c

Signed-off-by: Nicholas Walter Knize <nknize@apache.org>

nknize force-pushed the remove/segmentMemoryEstimate branch from b3f0660 to 693ed1c Compare February 1, 2022 22:04

dblock requested a review from andrross February 2, 2022 19:07

adnapibar approved these changes Feb 4, 2022

View reviewed changes

reta reviewed Feb 4, 2022

View reviewed changes

nknize merged commit fc0d3a3 into opensearch-project:main Feb 4, 2022

nknize mentioned this pull request Feb 4, 2022

[Remove] CircuitBreaker Accounting #2056

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Remove] Segment memory estimation and tracking #2029

[Remove] Segment memory estimation and tracking #2029

nknize commented Feb 1, 2022

opensearch-ci-bot commented Feb 1, 2022

opensearch-ci-bot commented Feb 1, 2022

opensearch-ci-bot commented Feb 1, 2022

opensearch-ci-bot commented Feb 1, 2022

opensearch-ci-bot commented Feb 1, 2022

adnapibar left a comment

adnapibar Feb 4, 2022

nknize Feb 4, 2022

reta Feb 4, 2022

nknize Feb 4, 2022

nknize Feb 4, 2022

[Remove] Segment memory estimation and tracking #2029

[Remove] Segment memory estimation and tracking #2029

Conversation

nknize commented Feb 1, 2022

opensearch-ci-bot commented Feb 1, 2022

opensearch-ci-bot commented Feb 1, 2022

opensearch-ci-bot commented Feb 1, 2022

opensearch-ci-bot commented Feb 1, 2022

opensearch-ci-bot commented Feb 1, 2022

adnapibar left a comment

Choose a reason for hiding this comment

adnapibar Feb 4, 2022

Choose a reason for hiding this comment

nknize Feb 4, 2022

Choose a reason for hiding this comment

reta Feb 4, 2022

Choose a reason for hiding this comment

nknize Feb 4, 2022

Choose a reason for hiding this comment

nknize Feb 4, 2022

Choose a reason for hiding this comment