[fix](inverted index) cloud mode supports time series #33414

zzzxl1993 · 2024-04-09T06:10:36Z

Proposed changes

Issue Number: close #xxx

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

doris-robot · 2024-04-09T06:10:42Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

zzzxl1993 · 2024-04-09T06:10:48Z

run buildall

github-actions

clang-tidy made some suggestions

github-actions · 2024-04-09T06:17:16Z

be/src/cloud/cloud_cumulative_compaction_policy.cpp

@@ -49,7 +49,7 @@ int64_t CloudSizeBasedCumulativeCompactionPolicy::_level_size(const int64_t size
    return (int64_t)1 << (sizeof(size) * 8 - 1 - __builtin_clzl(size));
 }

-int CloudSizeBasedCumulativeCompactionPolicy::pick_input_rowsets(
+int32_t CloudSizeBasedCumulativeCompactionPolicy::pick_input_rowsets(


warning: function 'pick_input_rowsets' exceeds recommended size/complexity thresholds [readability-function-size]

int32_t CloudSizeBasedCumulativeCompactionPolicy::pick_input_rowsets( ^

Additional context

be/src/cloud/cloud_cumulative_compaction_policy.cpp:51: 127 lines including whitespace and comments (threshold 80)

int32_t CloudSizeBasedCumulativeCompactionPolicy::pick_input_rowsets( ^

github-actions · 2024-04-09T06:17:17Z

be/src/cloud/cloud_cumulative_compaction_policy.cpp

@@ -213,4 +213,143 @@
                   : last_cumulative_point;
 }

+int32_t CloudTimeSeriesCumulativeCompactionPolicy::pick_input_rowsets(


warning: method 'pick_input_rowsets' can be made static [readability-convert-member-functions-to-static]

Suggested change

int32_t CloudTimeSeriesCumulativeCompactionPolicy::pick_input_rowsets(

static int32_t CloudTimeSeriesCumulativeCompactionPolicy::pick_input_rowsets(

github-actions · 2024-04-09T06:17:17Z

be/src/cloud/cloud_cumulative_compaction_policy.cpp

@@ -213,4 +213,143 @@
                   : last_cumulative_point;
 }

+int32_t CloudTimeSeriesCumulativeCompactionPolicy::pick_input_rowsets(


warning: function 'pick_input_rowsets' exceeds recommended size/complexity thresholds [readability-function-size]

int32_t CloudTimeSeriesCumulativeCompactionPolicy::pick_input_rowsets( ^

Additional context

be/src/cloud/cloud_cumulative_compaction_policy.cpp:215: 118 lines including whitespace and comments (threshold 80)

int32_t CloudTimeSeriesCumulativeCompactionPolicy::pick_input_rowsets( ^

github-actions · 2024-04-09T06:17:17Z

be/src/cloud/cloud_cumulative_compaction_policy.h

+
+    virtual int32_t pick_input_rowsets(CloudTablet* tablet,
+                                       const std::vector<RowsetSharedPtr>& candidate_rowsets,
+                                       const int64_t max_compaction_score,


warning: parameter 'max_compaction_score' is const-qualified in the function declaration; const-qualification of parameters only has an effect in function definitions [readability-avoid-const-params-in-decls]

Suggested change

const int64_t max_compaction_score,

int64_t max_compaction_score,

github-actions · 2024-04-09T06:17:17Z

be/src/cloud/cloud_cumulative_compaction_policy.h

+    virtual int32_t pick_input_rowsets(CloudTablet* tablet,
+                                       const std::vector<RowsetSharedPtr>& candidate_rowsets,
+                                       const int64_t max_compaction_score,
+                                       const int64_t min_compaction_score,


warning: parameter 'min_compaction_score' is const-qualified in the function declaration; const-qualification of parameters only has an effect in function definitions [readability-avoid-const-params-in-decls]

Suggested change

const int64_t min_compaction_score,

int64_t min_compaction_score,

github-actions · 2024-04-09T06:17:17Z

be/src/cloud/cloud_cumulative_compaction_policy.h

-                           bool allow_delete = false);
+    int32_t pick_input_rowsets(CloudTablet* tablet,
+                               const std::vector<RowsetSharedPtr>& candidate_rowsets,
+                               const int64_t max_compaction_score,


warning: parameter 'max_compaction_score' is const-qualified in the function declaration; const-qualification of parameters only has an effect in function definitions [readability-avoid-const-params-in-decls]

Suggested change

const int64_t max_compaction_score,

int64_t max_compaction_score,

github-actions · 2024-04-09T06:17:17Z

be/src/cloud/cloud_cumulative_compaction_policy.h

+    int32_t pick_input_rowsets(CloudTablet* tablet,
+                               const std::vector<RowsetSharedPtr>& candidate_rowsets,
+                               const int64_t max_compaction_score,
+                               const int64_t min_compaction_score,


warning: parameter 'min_compaction_score' is const-qualified in the function declaration; const-qualification of parameters only has an effect in function definitions [readability-avoid-const-params-in-decls]

Suggested change

const int64_t min_compaction_score,

int64_t min_compaction_score,

github-actions · 2024-04-09T06:17:17Z

be/src/cloud/cloud_cumulative_compaction_policy.h

+
+    int32_t pick_input_rowsets(CloudTablet* tablet,
+                               const std::vector<RowsetSharedPtr>& candidate_rowsets,
+                               const int64_t max_compaction_score,


warning: parameter 'max_compaction_score' is const-qualified in the function declaration; const-qualification of parameters only has an effect in function definitions [readability-avoid-const-params-in-decls]

Suggested change

const int64_t max_compaction_score,

int64_t max_compaction_score,

github-actions · 2024-04-09T06:17:17Z

be/src/cloud/cloud_cumulative_compaction_policy.h

+    int32_t pick_input_rowsets(CloudTablet* tablet,
+                               const std::vector<RowsetSharedPtr>& candidate_rowsets,
+                               const int64_t max_compaction_score,
+                               const int64_t min_compaction_score,


warning: parameter 'min_compaction_score' is const-qualified in the function declaration; const-qualification of parameters only has an effect in function definitions [readability-avoid-const-params-in-decls]

Suggested change

const int64_t min_compaction_score,

int64_t min_compaction_score,

github-actions · 2024-04-09T06:17:17Z

be/src/cloud/cloud_storage_engine.cpp

@@ -766,4 +770,12 @@ Status CloudStorageEngine::get_compaction_status_json(std::string* result) {
    return Status::OK();
 }

+std::shared_ptr<CloudCumulativeCompactionPolicy> CloudStorageEngine::cumu_compaction_policy(


warning: method 'cumu_compaction_policy' can be made static [readability-convert-member-functions-to-static]

be/src/cloud/cloud_storage_engine.h:108:

- std::shared_ptr<CloudCumulativeCompactionPolicy> cumu_compaction_policy( + static std::shared_ptr<CloudCumulativeCompactionPolicy> cumu_compaction_policy(

doris-robot · 2024-04-09T06:34:02Z

TeamCity be ut coverage result:
Function Coverage: 35.57% (8888/24985)
Line Coverage: 27.32% (72984/267156)
Region Coverage: 26.48% (37725/142440)
Branch Coverage: 23.29% (19225/82536)
Coverage Report: http://coverage.selectdb-in.cc/coverage/b16e9947d10d3d4804014efd41ac21b68f3a5fca_b16e9947d10d3d4804014efd41ac21b68f3a5fca/report/index.html

csun5285 · 2024-04-09T09:51:06Z

be/src/cloud/cloud_tablet.cpp

@@ -409,6 +410,10 @@ Result<std::unique_ptr<RowsetWriter>> CloudTablet::create_transient_rowset_write
 }

 int64_t CloudTablet::get_cloud_base_compaction_score() const {
+    if (_tablet_meta->compaction_policy() == CUMULATIVE_TIME_SERIES_POLICY) {


It should be checked whether there is a rowset containing the delete predicate. If so, return the score for base compaction.

csun5285 · 2024-04-09T09:52:36Z

be/src/cloud/cloud_cumulative_compaction_policy.cpp

+    // when single replica compaction is enabled and BE1 fetchs merged rowsets from BE2, and then BE2 goes offline.
+    // BE1 should performs compaction on its own, the time series compaction may re-compact previously fetched rowsets.
+    // time series compaction policy needs to skip over the fetched rowset
+    const auto& first_rowset_iter = std::find_if(


These codes should be deleted. Single compaction will not occur in the cloud engine

csun5285 · 2024-04-09T09:54:25Z

fe/fe-core/src/main/java/org/apache/doris/catalog/OlapTable.java

@@ -2194,7 +2194,7 @@ public Long getTimeSeriesCompactionGoalSizeMbytes() {
        if (tableProperty != null) {
            return tableProperty.timeSeriesCompactionGoalSizeMbytes();
        }
-        return null;
+        return PropertyAnalyzer.TIME_SERIES_COMPACTION_GOAL_SIZE_MBYTES_DEFAULT_VALUE;


getCompactionPolicy（） return PropertyAnalyzer.SIZE_BASED_COMPACTION_POLICY.

zzzxl1993 · 2024-04-09T10:37:00Z

run buildall

github-actions

clang-tidy made some suggestions

github-actions · 2024-04-09T10:42:47Z

be/src/cloud/cloud_cumulative_compaction_policy.cpp

@@ -213,4 +213,135 @@ int64_t CloudSizeBasedCumulativeCompactionPolicy::new_cumulative_point(
                   : last_cumulative_point;
 }

+int32_t CloudTimeSeriesCumulativeCompactionPolicy::pick_input_rowsets(


warning: function 'pick_input_rowsets' exceeds recommended size/complexity thresholds [readability-function-size]

int32_t CloudTimeSeriesCumulativeCompactionPolicy::pick_input_rowsets( ^

Additional context

be/src/cloud/cloud_cumulative_compaction_policy.cpp:215: 110 lines including whitespace and comments (threshold 80)

int32_t CloudTimeSeriesCumulativeCompactionPolicy::pick_input_rowsets( ^

doris-robot · 2024-04-09T11:17:33Z

TeamCity be ut coverage result:
Function Coverage: 35.63% (8906/24993)
Line Coverage: 27.36% (73147/267379)
Region Coverage: 26.52% (37802/142525)
Branch Coverage: 23.32% (19258/82590)
Coverage Report: http://coverage.selectdb-in.cc/coverage/2d649c3091e2afac1c1cd1cc50435a011f41db2b_2d649c3091e2afac1c1cd1cc50435a011f41db2b/report/index.html

zzzxl1993 · 2024-04-09T12:30:31Z

run buildall

doris-robot · 2024-04-09T12:49:09Z

TeamCity be ut coverage result:
Function Coverage: 35.63% (8906/24993)
Line Coverage: 27.35% (73140/267379)
Region Coverage: 26.52% (37798/142525)
Branch Coverage: 23.32% (19258/82590)
Coverage Report: http://coverage.selectdb-in.cc/coverage/c59322002d904f6c2909c51edb79ab5253d77bde_c59322002d904f6c2909c51edb79ab5253d77bde/report/index.html

csun5285

LGTM

github-actions · 2024-04-10T02:11:38Z

PR approved by anyone and no changes requested.

github-actions · 2024-04-10T02:36:21Z

PR approved by at least one committer and no changes requested.

zzzxl1993 · 2024-04-10T02:38:40Z

run p0

zzzxl1993 · 2024-04-10T03:25:11Z

run cloud_p0

zzzxl1993 · 2024-04-10T07:01:14Z

run buildall

zzzxl1993 · 2024-04-10T07:07:08Z

run buildall

github-actions

clang-tidy made some suggestions

github-actions · 2024-04-10T07:07:26Z

be/src/cloud/cloud_cumulative_compaction_policy.cpp

+    return 0;
+}
+
+int64_t CloudTimeSeriesCumulativeCompactionPolicy::new_compaction_level(


warning: method 'new_compaction_level' can be made static [readability-convert-member-functions-to-static]

Suggested change

int64_t CloudTimeSeriesCumulativeCompactionPolicy::new_compaction_level(

static int64_t CloudTimeSeriesCumulativeCompactionPolicy::new_compaction_level(

github-actions · 2024-04-10T07:07:26Z

be/src/cloud/cloud_cumulative_compaction_policy.h

+                                 Version& last_delete_version,
+                                 int64_t last_cumulative_point) override;
+
+    int64_t new_compaction_level(const std::vector<RowsetSharedPtr>& input_rowsets) override {


warning: method 'new_compaction_level' can be made static [readability-convert-member-functions-to-static]

Suggested change

int64_t new_compaction_level(const std::vector<RowsetSharedPtr>& input_rowsets) override {

static int64_t new_compaction_level(const std::vector<RowsetSharedPtr>& input_rowsets) override {

zzzxl1993 · 2024-04-10T07:24:47Z

run buildall

doris-robot · 2024-04-10T08:02:51Z

TeamCity be ut coverage result:
Function Coverage: 35.66% (8910/24986)
Line Coverage: 27.37% (73166/267340)
Region Coverage: 26.53% (37806/142501)
Branch Coverage: 23.33% (19266/82584)
Coverage Report: http://coverage.selectdb-in.cc/coverage/be8f4debb0774480a939d331746ea77eeec81928_be8f4debb0774480a939d331746ea77eeec81928/report/index.html

csun5285

LGTM

github-actions · 2024-04-10T09:56:27Z

PR approved by at least one committer and no changes requested.

github-actions bot added the meta-change label Apr 9, 2024

github-actions bot reviewed Apr 9, 2024

View reviewed changes

csun5285 suggested changes Apr 9, 2024

View reviewed changes

zzzxl1993 force-pushed the time_series_cloud branch from b16e994 to 2d649c3 Compare April 9, 2024 10:36

github-actions bot reviewed Apr 9, 2024

View reviewed changes

zzzxl1993 force-pushed the time_series_cloud branch from 2d649c3 to c593220 Compare April 9, 2024 12:28

csun5285 approved these changes Apr 10, 2024

View reviewed changes

github-actions bot added the reviewed label Apr 10, 2024

qidaye changed the title ~~[fix](inverted index) cloud mod supports time series~~ [fix](inverted index) cloud mode supports time series Apr 10, 2024

qidaye previously approved these changes Apr 10, 2024

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Apr 10, 2024

zzzxl1993 dismissed qidaye’s stale review via e922705 April 10, 2024 07:01

zzzxl1993 force-pushed the time_series_cloud branch from c593220 to e922705 Compare April 10, 2024 07:01

github-actions bot removed the approved Indicates a PR has been approved by one committer. label Apr 10, 2024

zzzxl1993 force-pushed the time_series_cloud branch from e922705 to 600e290 Compare April 10, 2024 07:06

github-actions bot reviewed Apr 10, 2024

View reviewed changes

zzzxl1993 force-pushed the time_series_cloud branch from 600e290 to 956e238 Compare April 10, 2024 07:12

[fix](inverted index) cloud mod supports time series

be8f4de

zzzxl1993 force-pushed the time_series_cloud branch from 956e238 to be8f4de Compare April 10, 2024 07:21

csun5285 approved these changes Apr 10, 2024

View reviewed changes

qidaye approved these changes Apr 10, 2024

View reviewed changes

qidaye merged commit 62a5bc3 into apache:master Apr 10, 2024
23 of 29 checks passed

github-actions bot added the approved Indicates a PR has been approved by one committer. label Apr 10, 2024

dataroaring added the dev/3.0.0-merged label Apr 13, 2024

dataroaring pushed a commit that referenced this pull request Apr 13, 2024

[fix](inverted index) cloud mod supports time series (#33414)

241981e

	int32_t CloudTimeSeriesCumulativeCompactionPolicy::pick_input_rowsets(
	static int32_t CloudTimeSeriesCumulativeCompactionPolicy::pick_input_rowsets(

	const int64_t max_compaction_score,
	int64_t max_compaction_score,

	const int64_t min_compaction_score,
	int64_t min_compaction_score,

	int64_t CloudTimeSeriesCumulativeCompactionPolicy::new_compaction_level(
	static int64_t CloudTimeSeriesCumulativeCompactionPolicy::new_compaction_level(

	int64_t new_compaction_level(const std::vector<RowsetSharedPtr>& input_rowsets) override {
	static int64_t new_compaction_level(const std::vector<RowsetSharedPtr>& input_rowsets) override {

[fix](inverted index) cloud mode supports time series #33414

[fix](inverted index) cloud mode supports time series #33414

Conversation

zzzxl1993 commented Apr 9, 2024

Proposed changes

Further comments

doris-robot commented Apr 9, 2024

zzzxl1993 commented Apr 9, 2024

github-actions bot left a comment

Choose a reason for hiding this comment

github-actions bot Apr 9, 2024

Choose a reason for hiding this comment

github-actions bot Apr 9, 2024

Choose a reason for hiding this comment

github-actions bot Apr 9, 2024

Choose a reason for hiding this comment

github-actions bot Apr 9, 2024

Choose a reason for hiding this comment

github-actions bot Apr 9, 2024

Choose a reason for hiding this comment

github-actions bot Apr 9, 2024

Choose a reason for hiding this comment

github-actions bot Apr 9, 2024

Choose a reason for hiding this comment

github-actions bot Apr 9, 2024

Choose a reason for hiding this comment

github-actions bot Apr 9, 2024

Choose a reason for hiding this comment

github-actions bot Apr 9, 2024

Choose a reason for hiding this comment

doris-robot commented Apr 9, 2024

csun5285 Apr 9, 2024

Choose a reason for hiding this comment

csun5285 Apr 9, 2024

Choose a reason for hiding this comment

csun5285 Apr 9, 2024

Choose a reason for hiding this comment

zzzxl1993 commented Apr 9, 2024

github-actions bot left a comment

Choose a reason for hiding this comment

github-actions bot Apr 9, 2024

Choose a reason for hiding this comment

doris-robot commented Apr 9, 2024

zzzxl1993 commented Apr 9, 2024

doris-robot commented Apr 9, 2024

csun5285 left a comment

Choose a reason for hiding this comment

github-actions bot commented Apr 10, 2024

github-actions bot commented Apr 10, 2024

zzzxl1993 commented Apr 10, 2024

zzzxl1993 commented Apr 10, 2024

zzzxl1993 commented Apr 10, 2024

zzzxl1993 commented Apr 10, 2024

github-actions bot left a comment

Choose a reason for hiding this comment

github-actions bot Apr 10, 2024

Choose a reason for hiding this comment

github-actions bot Apr 10, 2024

Choose a reason for hiding this comment

zzzxl1993 commented Apr 10, 2024

doris-robot commented Apr 10, 2024

csun5285 left a comment

Choose a reason for hiding this comment

github-actions bot commented Apr 10, 2024