Skip to content

[IOTDB-6273] Load: Memory Management Framework#11756

Merged
yschengzi merged 30 commits intoapache:masterfrom
MiniSho:load_memory_framework
Dec 29, 2023
Merged

[IOTDB-6273] Load: Memory Management Framework#11756
yschengzi merged 30 commits intoapache:masterfrom
MiniSho:load_memory_framework

Conversation

@MiniSho
Copy link
Copy Markdown
Contributor

@MiniSho MiniSho commented Dec 20, 2023

Description

Memory Manager core class LoadTsFileMemoryManager , which provides four interfaces: allocatedFromQuery and releaseToQuery for query module and forceAllocate and tryAllocate for load module.

Analyzer part of the memory manager class DeviceToTimeseriesSchemasMap, which encapsulates Map<String, Set<MeasurementSchema>> currentBatchDevice2TimeseriesSchemas , providing add(), isEmpty(), and isEmpty(). add(), isEmpty(), clear() and other interfaces for TsFileAnalyzer to call, mainly in the add method to add the calculation of the memory process.

MemoryBlock is the block of memory that the caller gets after requesting memory, which contains the size of the requested memory and the variables that mark whether the memory has been freed or not, etc. QueryMemoryBlock is the block of memory that the caller gets after requesting memory from the query module, and LoadMemoryBlock is the block of memory that the caller provides to the load module, and the requested memory will be used to calculate the size of the memory. QueryMemoryBlock is a block of memory requested from the query module and LoadMemoryBlock is a block of memory provided to the load module, and the requested QueryMemoryBlock is stuffed into the corresponding LoadMemoryBlock and returned to load.


内存控制核心类 LoadTsFileMemoryManager, 向上(查询模块)提供allocatedFromQuery 和 releaseFromQuery 两个接口,向下(load模块)提供forceAllocate和tryAllocate两个接口

analyzer部分内存控制类 DeviceToTimeseriesSchemasMap, 其中封装了Map<String, Set> currentBatchDevice2TimeseriesSchemas ,向外提供 add(),isEmpty(),clear()等接口供TsFileAnalyzer调用,主要在add方法中添加计算内存流程

memoryBlock 是调用者申请内存后得到的内存块,其中包含申请到的内存大小,以及标记是否已释放的变量等。QueryMemoryBlock是向查询模块申请得到的内存块,LoadMemoryBlock是向 load 模块提供的内存块,申请到的 QueryMemoryBlock 会塞到相应的LoadMemoryBlock返回给load,其目的是当load模块用完内存块需要释放时,查询模块相应的内存也需要被释放。

@yschengzi yschengzi merged commit afde755 into apache:master Dec 29, 2023
yschengzi added a commit that referenced this pull request Dec 29, 2023
@MiniSho MiniSho deleted the load_memory_framework branch January 28, 2024 12:16
SzyWilliam pushed a commit to SzyWilliam/iotdb that referenced this pull request Nov 26, 2024
* done

* fix

* refactor tsfile analyze schema

* add memory control for data cache

* add release policy for data cache memory block

* fix iterator

* try to load liu zhen dataset

* fix send data and improve writing deletions

* improve deletion data writing on load tsfile manager

* done

* fix

* refactor tsfile analyze schema

* add memory control for data cache

* add release policy for data cache memory block

* fix iterator

* try to load liu zhen dataset

* fix send data and improve writing deletions

* improve deletion data writing on load tsfile manager

* add LoadMemoryMetrics

* remove debug log

* fix config

* queryEngine provides tryAllocate, forceAllocate & release methods for load

* fix UsedMemorySize calculation inaccuracy

* adjust args

---------

Co-authored-by: yschengzi <yscysc99@126.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants