[To dev/1.3] Pipe: Added total floating memory usage for mem-ctrl of linked but deleted tsFile resource and insertNode memory (#15205)#15268
Merged
SteveYurongSu merged 2 commits intoapache:dev/1.3from Apr 3, 2025
Conversation
…leted tsFile resource and insertNode memory (apache#15205)
Contributor
There was a problem hiding this comment.
Pull Request Overview
This PR updates the memory control framework for pipes by introducing a new floating memory proportion configuration and refactoring memory calculations to distinguish between total non-floating and floating memory. The key changes include:
- Adding a new configuration property (pipeTotalFloatingMemoryProportion) and its corresponding getter/setter methods.
- Updating memory calculations in multiple components (e.g., PipeMemoryManager, subscription caches, extractors, and task agents) to use non-floating memory metrics.
- Renaming methods and variables for consistency (e.g., tsfile → tsFile).
Reviewed Changes
Copilot reviewed 13 out of 13 changed files in this pull request and generated no comments.
Show a summary per file
| File | Description |
|---|---|
| iotdb-core/node-commons/src/main/java/org/apache/iotdb/commons/pipe/config/PipeConfig.java | Added getter for pipeTotalFloatingMemoryProportion and logging update |
| iotdb-core/node-commons/src/main/java/org/apache/iotdb/commons/pipe/agent/task/PipeTaskAgent.java | Added aggregation method for floating memory usage |
| iotdb-core/node-commons/src/main/java/org/apache/iotdb/commons/conf/CommonDescriptor.java | Injected new property for floating memory proportion |
| iotdb-core/node-commons/src/main/java/org/apache/iotdb/commons/conf/CommonConfig.java | Added and exposed pipeTotalFloatingMemoryProportion configuration |
| iotdb-core/datanode/src/main/java/org/apache/iotdb/db/subscription/event/cache/SubscriptionPollResponseCache.java | Changed memory calculation to use non-floating memory |
| iotdb-core/datanode/src/main/java/org/apache/iotdb/db/subscription/broker/SubscriptionPrefetchingQueueStates.java | Updated memory check with non-floating memory calculations |
| iotdb-core/datanode/src/main/java/org/apache/iotdb/db/pipe/resource/tsfile/PipeTsFileResourceManager.java | Renamed method for consistency in tsFile naming |
| iotdb-core/datanode/src/main/java/org/apache/iotdb/db/pipe/resource/memory/PipeMemoryManager.java | Refactored memory allocation logic to use non-floating and floating memory sizes |
| iotdb-core/datanode/src/main/java/org/apache/iotdb/db/pipe/metric/overview/PipeResourceMetrics.java | Bound metric to non-floating memory size |
| iotdb-core/datanode/src/main/java/org/apache/iotdb/db/pipe/extractor/dataregion/realtime/PipeRealtimeDataRegionHybridExtractor.java | Adjusted threshold calculations based on total floating memory size |
| iotdb-core/datanode/src/main/java/org/apache/iotdb/db/pipe/event/common/tsfile/container/TsFileInsertionDataContainerProvider.java | Updated threshold check using non-floating memory calculation |
| iotdb-core/datanode/src/main/java/org/apache/iotdb/db/pipe/connector/client/IoTDBDataNodeCacheLeaderClientManager.java | Updated client’s memory calculation to use non-floating memory size |
| iotdb-core/datanode/src/main/java/org/apache/iotdb/db/pipe/agent/task/PipeDataNodeTaskAgent.java | Revised memory checks and logging with updates to floating memory aggregation |
SteveYurongSu
approved these changes
Apr 3, 2025
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
As the title said.
#15205
This PR has:
for an unfamiliar reader.
for code coverage.
Key changed/added classes (or packages if there are too many classes) in this PR