[Improve] Estimated column reader memory to control segment cache #34526
Conversation
|
Thank you for your contribution to Apache Doris. Since 2024-03-18, the Document has been moved to doris-website. |
|
run buildall |
|
clang-tidy review says "All clean, LGTM! 👍" |
|
change to |
| @@ -408,6 +408,7 @@ Status Segment::_create_column_readers(const SegmentFooterPB& footer) { | |||
| RETURN_IF_ERROR(ColumnReader::create(opts, footer.columns(iter->second), footer.num_rows(), | |||
| _file_reader, &reader)); | |||
| _column_readers.emplace(column.unique_id(), std::move(reader)); | |||
| _meta_mem_usage += config::estimated_mem_per_column_reader; | |||
There was a problem hiding this comment.
in past, _meta_mem_usage only tracking some indexes memory,
now, add column reader memory
will there be other memory in segment?
There was a problem hiding this comment.
It still has some unique_ptr/shared_ptr object and the object SubcolumnColumnReaders.
|
run buildall |
|
clang-tidy review says "All clean, LGTM! 👍" |
|
TeamCity be ut coverage result: |
github-actions
bot
added
the
approved
|
PR approved by at least one committer and no changes requested. |
|
PR approved by anyone and no changes requested. |
| @@ -1053,6 +1053,10 @@ DEFINE_mInt32(schema_cache_sweep_time_sec, "100"); | |||
|
|
|||
| // max number of segment cache, default -1 for backward compatibility fd_number*2/5 | |||
| DEFINE_mInt32(segment_cache_capacity, "-1"); | |||
| DEFINE_mInt32(estimated_num_columns_per_segment, "30"); | |||
| DEFINE_mInt32(estimated_mem_per_column_reader, "1024"); | |||
| // The value is calculate by storage_page_cache_limit * index_page_cache_percentage | |||
Proposed changes
Issue Number: close #xxx
If a segment has many columns, it will use a lot more memory than a segment with few columns, but it is not observable at present. So we estimated column reader memory to to control the memory.
Further comments
If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...