Skip to content

[To dev/1.3] Pipe: Added total floating memory usage for mem-ctrl of linked but deleted tsFile resource and insertNode memory (#15205)#15268

Merged
SteveYurongSu merged 2 commits intoapache:dev/1.3from
Caideyipi:cp-floating-13
Apr 3, 2025

Conversation

@Caideyipi
Copy link
Collaborator

@Caideyipi Caideyipi commented Apr 3, 2025

Description

As the title said.
#15205


This PR has:

  • been self-reviewed.
    • concurrent read
    • concurrent write
    • concurrent read and write
  • added documentation for new or modified features or behaviors.
  • added Javadocs for most classes and all non-trivial methods.
  • added or updated version, license, or notice information
  • added comments explaining the "why" and the intent of the code wherever would not be obvious
    for an unfamiliar reader.
  • added unit tests or modified existing tests to cover new code paths, ensuring the threshold
    for code coverage.
  • added integration tests.
  • been tested in a test IoTDB cluster.

Key changed/added classes (or packages if there are too many classes) in this PR

Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR updates the memory control framework for pipes by introducing a new floating memory proportion configuration and refactoring memory calculations to distinguish between total non-floating and floating memory. The key changes include:

  • Adding a new configuration property (pipeTotalFloatingMemoryProportion) and its corresponding getter/setter methods.
  • Updating memory calculations in multiple components (e.g., PipeMemoryManager, subscription caches, extractors, and task agents) to use non-floating memory metrics.
  • Renaming methods and variables for consistency (e.g., tsfile → tsFile).

Reviewed Changes

Copilot reviewed 13 out of 13 changed files in this pull request and generated no comments.

Show a summary per file
File Description
iotdb-core/node-commons/src/main/java/org/apache/iotdb/commons/pipe/config/PipeConfig.java Added getter for pipeTotalFloatingMemoryProportion and logging update
iotdb-core/node-commons/src/main/java/org/apache/iotdb/commons/pipe/agent/task/PipeTaskAgent.java Added aggregation method for floating memory usage
iotdb-core/node-commons/src/main/java/org/apache/iotdb/commons/conf/CommonDescriptor.java Injected new property for floating memory proportion
iotdb-core/node-commons/src/main/java/org/apache/iotdb/commons/conf/CommonConfig.java Added and exposed pipeTotalFloatingMemoryProportion configuration
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/subscription/event/cache/SubscriptionPollResponseCache.java Changed memory calculation to use non-floating memory
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/subscription/broker/SubscriptionPrefetchingQueueStates.java Updated memory check with non-floating memory calculations
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/pipe/resource/tsfile/PipeTsFileResourceManager.java Renamed method for consistency in tsFile naming
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/pipe/resource/memory/PipeMemoryManager.java Refactored memory allocation logic to use non-floating and floating memory sizes
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/pipe/metric/overview/PipeResourceMetrics.java Bound metric to non-floating memory size
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/pipe/extractor/dataregion/realtime/PipeRealtimeDataRegionHybridExtractor.java Adjusted threshold calculations based on total floating memory size
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/pipe/event/common/tsfile/container/TsFileInsertionDataContainerProvider.java Updated threshold check using non-floating memory calculation
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/pipe/connector/client/IoTDBDataNodeCacheLeaderClientManager.java Updated client’s memory calculation to use non-floating memory size
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/pipe/agent/task/PipeDataNodeTaskAgent.java Revised memory checks and logging with updates to floating memory aggregation

@SteveYurongSu SteveYurongSu changed the title [To dev/1.3] Pipe: Added total floating memory usage for mem-ctrl of linked but deleted tsFile resource and insertNode memory [To dev/1.3] Pipe: Added total floating memory usage for mem-ctrl of linked but deleted tsFile resource and insertNode memory (#15205) Apr 3, 2025
@SteveYurongSu SteveYurongSu self-assigned this Apr 3, 2025
@SteveYurongSu SteveYurongSu merged commit ccfa23e into apache:dev/1.3 Apr 3, 2025
25 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants