[IOTDB-6273] Load: Memory Management Framework#11756
Merged
yschengzi merged 30 commits intoapache:masterfrom Dec 29, 2023
Merged
[IOTDB-6273] Load: Memory Management Framework#11756yschengzi merged 30 commits intoapache:masterfrom
yschengzi merged 30 commits intoapache:masterfrom
Conversation
MarcosZyk
reviewed
Dec 26, 2023
iotdb-core/node-commons/src/main/java/org/apache/iotdb/commons/conf/CommonConfig.java
Outdated
Show resolved
Hide resolved
yschengzi
added a commit
that referenced
this pull request
Dec 29, 2023
This reverts commit afde755.
SzyWilliam
pushed a commit
to SzyWilliam/iotdb
that referenced
this pull request
Nov 26, 2024
* done * fix * refactor tsfile analyze schema * add memory control for data cache * add release policy for data cache memory block * fix iterator * try to load liu zhen dataset * fix send data and improve writing deletions * improve deletion data writing on load tsfile manager * done * fix * refactor tsfile analyze schema * add memory control for data cache * add release policy for data cache memory block * fix iterator * try to load liu zhen dataset * fix send data and improve writing deletions * improve deletion data writing on load tsfile manager * add LoadMemoryMetrics * remove debug log * fix config * queryEngine provides tryAllocate, forceAllocate & release methods for load * fix UsedMemorySize calculation inaccuracy * adjust args --------- Co-authored-by: yschengzi <yscysc99@126.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Memory Manager core class
LoadTsFileMemoryManager, which provides four interfaces:allocatedFromQueryandreleaseToQueryfor query module andforceAllocateandtryAllocatefor load module.Analyzer part of the memory manager class
DeviceToTimeseriesSchemasMap, which encapsulatesMap<String, Set<MeasurementSchema>> currentBatchDevice2TimeseriesSchemas, providing add(), isEmpty(), and isEmpty(). add(), isEmpty(), clear() and other interfaces for TsFileAnalyzer to call, mainly in the add method to add the calculation of the memory process.MemoryBlock is the block of memory that the caller gets after requesting memory, which contains the size of the requested memory and the variables that mark whether the memory has been freed or not, etc.
QueryMemoryBlockis the block of memory that the caller gets after requesting memory from the query module, andLoadMemoryBlockis the block of memory that the caller provides to the load module, and the requested memory will be used to calculate the size of the memory.QueryMemoryBlockis a block of memory requested from the query module andLoadMemoryBlockis a block of memory provided to the load module, and the requested QueryMemoryBlock is stuffed into the corresponding LoadMemoryBlock and returned to load.内存控制核心类 LoadTsFileMemoryManager, 向上(查询模块)提供allocatedFromQuery 和 releaseFromQuery 两个接口,向下(load模块)提供forceAllocate和tryAllocate两个接口
analyzer部分内存控制类 DeviceToTimeseriesSchemasMap, 其中封装了Map<String, Set> currentBatchDevice2TimeseriesSchemas ,向外提供 add(),isEmpty(),clear()等接口供TsFileAnalyzer调用,主要在add方法中添加计算内存流程
memoryBlock 是调用者申请内存后得到的内存块,其中包含申请到的内存大小,以及标记是否已释放的变量等。QueryMemoryBlock是向查询模块申请得到的内存块,LoadMemoryBlock是向 load 模块提供的内存块,申请到的 QueryMemoryBlock 会塞到相应的LoadMemoryBlock返回给load,其目的是当load模块用完内存块需要释放时,查询模块相应的内存也需要被释放。