[Enhancement] Early release stale rowsets after compaction. #5056

xiaoyong-z · 2022-04-13T13:01:15Z

After compaction, the stale rowsets will still stay in memory for a long time before gc thread free them. And these stale rowsets take up a lot of memory. In fact, there may be a few old readers still read the stale rowsets, all new readers will read the new rowset generated by the compaction, so we can free the stale rowsets if there are no readers on it. Experiments show that this optimization significantly reduces the memory footprint of rowsets metadata.

Signed-off-by: xyz a997647204@gmail.com

What type of PR is this：

bug
feature
enhancement
others

Which issues of this PR fixes ：

Fixes #4873

Problem Summary(Required) ：

After compaction, the stale rowsets will still stay in memory for a long time before gc thread free them. And these stale rowsets take up a lot of memory. In fact, there may be a few old readers still reading the stale rowsets, all new readers will read the new rowset generated by the compaction, so we can free the stale rowsets if there are no readers on it. Experiments show that this optimization significantly reduces the memory footprint of rowsets metadata.

xiaoyong-z · 2022-04-13T13:03:41Z

Experimental setup

Table contains 10800 columns, and divided into 100 bucket.
Create 100 broker load tasks, each task load 1 file contains 10,000 lines of data(1.2G).

Memory Before optimization

The peak of rss is about 81.7G, the peak of tablet meta memory is 34.6G, the peak of column_reader_index_mem is 17.8G, and the peak of column_reader_meta_mem is 16.8G.

Memory after sharing data field between ColumnReader optimiztion #4875

Rss peaks at about 74.2G, tablet meta memory peaks at 27.3G, column_reader_index_mem peaks at 15.3G, and column_reader_meta_mem peaks at 11.9G, the memory of column_reader_meta_mem significantly lower than before.

Memory after early releasing stale rowsets #5056

Rss peaks at about 40G, tablet meta memory peaks at 4.5G, the memory footprint of the tablet meta data is significantly reduced

be/src/storage/meta_reader.cpp

imay · 2022-04-13T13:17:53Z

be/src/storage/compaction.cpp

@@ -245,6 +245,10 @@ Status Compaction::_merge_rowsets_vertically(size_t segment_iterator_num, Statis
    TRACE_COUNTER_SCOPE_LATENCY_US("merge_rowsets_latency_us");
    auto mask_buffer = std::make_unique<RowSourceMaskBuffer>(_tablet->tablet_id(), _tablet->data_dir()->path());
    auto source_masks = std::make_unique<std::vector<RowSourceMask>>();
+    // ensure all input rowsets are loaded into memory
+    for (auto& rowset : _input_rowsets) {
+        RETURN_IF_ERROR(rowset->load());


Why can this operation assure that rowset won't be released?
I think other thread can release these rowsets after loading()

Only stale rowsets can be released and rowsets can only transfer from not-stale to stale. Now the compaction has not begun, the input rowsets are all not-stale. So i think no other threads can release these rowsets?
If the code is written correctly, rowset should be loaded into memory here. Just in case, I've added three lines safeguards here, because the segments of rowsets will be used directly later.

be/src/storage/rowset/rowset.h

xiaoyong-z · 2022-04-20T07:07:03Z

@Mergifyio rebase

mergify · 2022-04-20T07:07:44Z

rebase

❌ Base branch update has failed

Git reported the following error:

Rebasing (1/6)
Auto-merging be/src/storage/compaction.cpp
Auto-merging be/src/storage/rowset/rowset.h
CONFLICT (content): Merge conflict in be/src/storage/rowset/rowset.h
Auto-merging be/src/storage/tablet.cpp
error: could not apply 88fcfff9... [optimize] Early release stale rowsets after compaction.
hint: Resolve all conflicts manually, mark them as resolved with
hint: "git add/rm <conflicted_files>", then run "git rebase --continue".
hint: You can instead skip this commit: run "git rebase --skip".
hint: To abort and get back to the state before "git rebase", run "git rebase --abort".
Could not apply 88fcfff9... [optimize] Early release stale rowsets after compaction. After compaction, the stale rowsets will still stay in memory for a long time before gc thread free them. And these stale rowsets take up a lot of memory. In fact, there may be a few old readers still read the stale rowsets, all new readers will read the new rowset generated by the compaction, so we can free the stale rowsets if there are no readers on it. Experiments show that this optimization significantly reduces the memory footprint of rowsets meta data.

err-code: 5F68D

After compaction, the stale rowsets will still stay in memory for a long time before gc thread free them. And these stale rowsets take up a lot of memory. In fact, there may be a few old readers still read the stale rowsets, all new readers will read the new rowset generated by the compaction, so we can free the stale rowsets if there are no readers on it. Experiments show that this optimization significantly reduces the memory footprint of rowsets meta data.

xiaoyong-z · 2022-04-20T07:48:13Z

@Mergifyio rebase

mergify · 2022-04-20T07:48:52Z

rebase

❌ Base branch update has failed

Git reported the following error:

Rebasing (1/1)
Auto-merging be/src/storage/tablet_reader.cpp
CONFLICT (content): Merge conflict in be/src/storage/tablet_reader.cpp
error: could not apply 4e3060ce... [optimize] Early release stale rowsets after compaction.
hint: Resolve all conflicts manually, mark them as resolved with
hint: "git add/rm <conflicted_files>", then run "git rebase --continue".
hint: You can instead skip this commit: run "git rebase --skip".
hint: To abort and get back to the state before "git rebase", run "git rebase --abort".
Could not apply 4e3060ce... [optimize] Early release stale rowsets after compaction. After compaction, the stale rowsets will still stay in memory for a long time before gc thread free them. And these stale rowsets take up a lot of memory. In fact, there may be a few old readers still read the stale rowsets, all new readers will read the new rowset generated by the compaction, so we can free the stale rowsets if there are no readers on it. Experiments show that this optimization significantly reduces the memory footprint of rowsets meta data.

err-code: D7F44

be/src/storage/meta_reader.cpp

imay · 2022-04-20T11:18:20Z

be/src/storage/tablet.cpp

@@ -276,6 +276,7 @@ void Tablet::modify_rowsets(const std::vector<RowsetSharedPtr>& to_add, const st

    // add rs_metas_to_delete to tracker
    _timestamped_version_tracker.add_stale_path_version(rs_metas_to_delete);
+    Rowset::close_rowsets(to_delete);


Actually I don't think this is a good place to close_rowsets.
I think caller should do this things.
This function just do meta modification. It don't know if the rowsets will be closed.
I think the compaction manager will decide if these rowsets should be closed not this function.

be/src/storage/tablet_reader.cpp

Signed-off-by: xyz <a997647204@gmail.com>

be/src/storage/tablet_reader.cpp

imay · 2022-04-20T15:21:32Z

be/src/storage/vertical_compaction_task.cpp

@@ -59,6 +59,11 @@ Status VerticalCompactionTask::_vertical_compaction_data(Statistics* statistics)
          "size:$1",
          max_rows_per_segment, column_groups.size());

+    // ensure all input rowsets are loaded into memory
+    for (auto& rowset : _input_rowsets) {


When these _input_rowsets is acquired and when is these rowsets is released?

I move these lines in tablet reader prepare() function. I will first acquire these rowsets and then loads them to ensure there input rowsets are not freed.
Also i move the tablet reader prepare() function before the direct use of rowset's segment() function. Thus these segments are always loaded into the memory.

Signed-off-by: xyz <a997647204@gmail.com>

chaoyli · 2022-04-21T06:55:35Z

run starrocks_clang-format

imay

LGTM

…s#5056) * [optimize] Early release stale rowsets after compaction. After compaction, the stale rowsets will still stay in memory for a long time before gc thread free them. And these stale rowsets take up a lot of memory. In fact, there may be a few old readers still read the stale rowsets, all new readers will read the new rowset generated by the compaction, so we can free the stale rowsets if there are no readers on it. Experiments show that this optimization significantly reduces the memory footprint of rowsets meta data.

* [optimize] Early release stale rowsets after compaction. After compaction, the stale rowsets will still stay in memory for a long time before gc thread free them. And these stale rowsets take up a lot of memory. In fact, there may be a few old readers still read the stale rowsets, all new readers will read the new rowset generated by the compaction, so we can free the stale rowsets if there are no readers on it. Experiments show that this optimization significantly reduces the memory footprint of rowsets meta data.

(cherry picked from commit 1f27dbb) Co-authored-by: evelyn.zhaojie <everlyn.zhaojie@gmail.com>

xiaoyong-z mentioned this pull request Apr 13, 2022

Compaction memory optimization #4873

Closed

3 tasks

imay reviewed Apr 13, 2022

View reviewed changes

chaoyli changed the title ~~[optimize] Early release stale rowsets after compaction.~~ [Enhancement] Early release stale rowsets after compaction. Apr 13, 2022

chaoyli reviewed Apr 13, 2022

View reviewed changes

be/src/storage/rowset/rowset.h Outdated Show resolved Hide resolved

xiaoyong-z changed the title ~~[Enhancement] Early release stale rowsets after compaction.~~ [Enhancement] Early release stale rowsets after compaction. (WIP) Apr 14, 2022

xiaoyong-z changed the title ~~[Enhancement] Early release stale rowsets after compaction. (WIP)~~ [Enhancement] Early release stale rowsets after compaction. Apr 14, 2022

xiaoyong-z force-pushed the Stale-Rowsets-early-release-after-commpaction branch 3 times, most recently from a38f858 to 2ba39e8 Compare April 16, 2022 16:18

chaoyli requested changes Apr 17, 2022

View reviewed changes

be/src/storage/rowset/rowset.h Outdated Show resolved Hide resolved

xiaoyong-z requested review from chaoyli and imay April 18, 2022 08:56

chaoyli previously approved these changes Apr 20, 2022

View reviewed changes

xiaoyong-z dismissed chaoyli’s stale review via 4e3060c April 20, 2022 07:39

xiaoyong-z force-pushed the Stale-Rowsets-early-release-after-commpaction branch from 1e59c3e to 4e3060c Compare April 20, 2022 07:39

Merge branch 'main' into Stale-Rowsets-early-release-after-commpaction

32e8f6f

chaoyli previously approved these changes Apr 20, 2022

View reviewed changes

meegoo previously approved these changes Apr 20, 2022

View reviewed changes

imay reviewed Apr 20, 2022

View reviewed changes

move close_rowsets out of the modify_rowsets

739fc69

Signed-off-by: xyz <a997647204@gmail.com>

xiaoyong-z dismissed stale reviews from meegoo and chaoyli via 739fc69 April 20, 2022 12:27

chaoyli previously approved these changes Apr 20, 2022

View reviewed changes

imay reviewed Apr 20, 2022

View reviewed changes

add rowsets.clear() in tabletreader::close()

4c13be0

Signed-off-by: xyz <a997647204@gmail.com>

xiaoyong-z dismissed chaoyli’s stale review via 4c13be0 April 20, 2022 16:23

move rowset load into tablet reader prepare()

7ffb8c2

Signed-off-by: xyz <a997647204@gmail.com>

chaoyli approved these changes Apr 21, 2022

View reviewed changes

imay approved these changes Apr 21, 2022

View reviewed changes

imay merged commit d3b8930 into StarRocks:main Apr 21, 2022

ZiheLiu mentioned this pull request Nov 22, 2022

[BugFix] Acquire rowsets at querying #13830

Merged

12 tasks

jaogoy pushed a commit to jaogoy/starrocks that referenced this pull request Nov 15, 2023

add unit for flink load para (StarRocks#5053) (StarRocks#5056)

afcf938

(cherry picked from commit 1f27dbb) Co-authored-by: evelyn.zhaojie <everlyn.zhaojie@gmail.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Enhancement] Early release stale rowsets after compaction. #5056

[Enhancement] Early release stale rowsets after compaction. #5056

xiaoyong-z commented Apr 13, 2022 •

edited by chaoyli

xiaoyong-z commented Apr 13, 2022

imay Apr 13, 2022

xiaoyong-z Apr 13, 2022

xiaoyong-z commented Apr 20, 2022

mergify bot commented Apr 20, 2022

xiaoyong-z commented Apr 20, 2022

mergify bot commented Apr 20, 2022

imay Apr 20, 2022

imay Apr 20, 2022

xiaoyong-z Apr 20, 2022 •

edited

chaoyli commented Apr 21, 2022

imay left a comment

[Enhancement] Early release stale rowsets after compaction. #5056

[Enhancement] Early release stale rowsets after compaction. #5056

Conversation

xiaoyong-z commented Apr 13, 2022 • edited by chaoyli

What type of PR is this：

Which issues of this PR fixes ：

Problem Summary(Required) ：

xiaoyong-z commented Apr 13, 2022

Experimental setup

Memory Before optimization

Memory after sharing data field between ColumnReader optimiztion #4875

Memory after early releasing stale rowsets #5056

imay Apr 13, 2022

Choose a reason for hiding this comment

xiaoyong-z Apr 13, 2022

Choose a reason for hiding this comment

xiaoyong-z commented Apr 20, 2022

mergify bot commented Apr 20, 2022

❌ Base branch update has failed

xiaoyong-z commented Apr 20, 2022

mergify bot commented Apr 20, 2022

❌ Base branch update has failed

imay Apr 20, 2022

Choose a reason for hiding this comment

imay Apr 20, 2022

Choose a reason for hiding this comment

xiaoyong-z Apr 20, 2022 • edited

Choose a reason for hiding this comment

chaoyli commented Apr 21, 2022

imay left a comment

Choose a reason for hiding this comment

xiaoyong-z commented Apr 13, 2022 •

edited by chaoyli

xiaoyong-z Apr 20, 2022 •

edited