[Feature] Adds routine_load_unstable_threshold_second #36222

rickif · 2023-12-01T03:27:21Z

Why I'm doing:
When a routine load job gets sustained failure, the retention of the Kafka partition may expire, which leads to data loss.

What I'm doing:

adding TimeProgress to the result of show routine load.
The TimestampProgress denotes the latest partition timestamp of the consumed message.
adding FE config routine_load_unstable_threshold_second.
When the time latency is more than this config value, the job would be set as UNSTABLE state.
In this state, the routine load job would be scheduled continuously, and the ReasonOfChangeState would be set.

Fixes #35296

What type of PR is this:

Does this PR entail a change in behavior?

Yes, this PR will result in a change in behavior.
No, this PR will not result in a change in behavior.

If yes, please specify the type of change:

Interface/UI changes: syntax, type conversion, expression evaluation, display information
Parameter changes: default values, similar parameters but with different default values
Policy changes: use new policy to replace old one, functionality automatically enabled
Feature removed
Miscellaneous: upgrade & downgrade compatibility, etc.

Checklist:

I have added test cases for my bug fix or my new feature
This pr needs user documentation (for new or modified features or behaviors)
- I have added documentation for my new feature or new function
  [Doc] Add routine_load_unstable_threshold_second #36967

Bugfix cherry-pick branch check:

rickif · 2023-12-01T07:20:45Z

@Mergifyio rebase

mergify · 2023-12-01T07:21:07Z

rebase

✅ Branch has been successfully rebased

Signed-off-by: ricky <rickif@qq.com>

gensrc/thrift/FrontendService.thrift

Signed-off-by: ricky <rickif@qq.com>

fe/fe-core/src/main/java/com/starrocks/load/routineload/RLTaskTxnCommitAttachment.java

…TxnCommitAttachment.java Co-authored-by: wyb <wybb86@gmail.com> Signed-off-by: ricky <rickif@qq.com>

sonarcloud · 2023-12-13T13:15:30Z

Quality Gate passed

The SonarCloud Quality Gate passed, but some issues were introduced.

5 New issues
0 Security Hotspots
0.0% Coverage on New Code
2.5% Duplication on New Code

See analysis details on SonarCloud

github-actions · 2023-12-13T14:43:46Z

[FE Incremental Coverage Report]

✅ pass : 57 / 66 (86.36%)

file detail

	path	covered_line	new_line	coverage	not_covered_line_detail
🔵	com/starrocks/load/routineload/RoutineLoadTaskScheduler.java	0	2	00.00%	[193, 198]
🔵	com/starrocks/load/routineload/PulsarRoutineLoadJob.java	0	3	00.00%	[293, 294, 300]
🔵	com/starrocks/load/routineload/PulsarProgress.java	0	1	00.00%	[129]
🔵	com/starrocks/load/routineload/RLTaskTxnCommitAttachment.java	5	6	83.33%	[94]
🔵	com/starrocks/load/routineload/KafkaRoutineLoadJob.java	19	20	95.00%	[349]
🔵	com/starrocks/load/routineload/RoutineLoadJob.java	26	27	96.30%	[1990]
🔵	com/starrocks/load/routineload/KafkaProgress.java	3	3	100.00%	[]
🔵	com/starrocks/common/Config.java	1	1	100.00%	[]
🔵	com/starrocks/sql/ast/ShowRoutineLoadStmt.java	1	1	100.00%	[]
🔵	com/starrocks/common/InternalErrorCode.java	2	2	100.00%	[]

github-actions · 2023-12-13T14:44:54Z

[BE Incremental Coverage Report]

❌ fail : 0 / 9 (00.00%)

file detail

	path	covered_line	new_line	coverage	not_covered_line_detail
🔵	src/runtime/routine_load/data_consumer_group.cpp	0	8	00.00%	[120, 168, 215, 216, 217, 229, 230, 231]
🔵	src/runtime/stream_load/stream_load_executor.cpp	0	1	00.00%	[463]

…ocks#36222

…ocks#36222 Signed-off-by: ricky <rickif@qq.com>

rickif · 2023-12-14T02:05:15Z

@Mergifyio backport branch-3.2 branch-3.1 branch-3.0 branch-2.5

mergify · 2023-12-14T02:05:24Z

backport branch-3.2 branch-3.1 branch-3.0 branch-2.5

✅ Backports have been created

#36997 [Feature] Adds routine_load_unstable_threshold_second (backport #36222) has been created for branch branch-3.2 but encountered conflicts
#36998 [Feature] Adds routine_load_unstable_threshold_second (backport #36222) has been created for branch branch-3.1 but encountered conflicts
#36999 [Feature] Adds routine_load_unstable_threshold_second (backport #36222) has been created for branch branch-3.0 but encountered conflicts
#37000 [Feature] Adds routine_load_unstable_threshold_second (backport #36222) has been created for branch branch-2.5 but encountered conflicts

Signed-off-by: ricky <rickif@qq.com> Co-authored-by: wyb <wybb86@gmail.com> (cherry picked from commit 4c9756f) # Conflicts: # fe/fe-core/src/main/java/com/starrocks/common/Config.java # fe/fe-core/src/main/java/com/starrocks/load/routineload/RoutineLoadJob.java # fe/fe-core/src/test/java/com/starrocks/analysis/ShowRoutineLoadStmtTest.java # fe/fe-core/src/test/java/com/starrocks/load/routineload/RoutineLoadJobTest.java

Signed-off-by: ricky <rickif@qq.com> Co-authored-by: wyb <wybb86@gmail.com>

… (branch-2.5) (#37010) Signed-off-by: ricky <rickif@qq.com> Co-authored-by: wyb <wybb86@gmail.com>

… (branch-3.0) (#37009) Signed-off-by: ricky <rickif@qq.com> Co-authored-by: wyb <wybb86@gmail.com>

… (branch-3.1) (#37008) Signed-off-by: ricky <rickif@qq.com> Co-authored-by: wyb <wybb86@gmail.com>

… (branch-3.2) (#37007) Signed-off-by: ricky <rickif@qq.com> Co-authored-by: wyb <wybb86@gmail.com>

Signed-off-by: ricky <rickif@qq.com> Co-authored-by: wyb <wybb86@gmail.com> Signed-off-by: 张敢 <zhanggan@deepexi.com>

rickif requested a review from a team as a code owner December 1, 2023 03:27

github-actions bot added behavior_changed META-REVIEW PROTO-REVIEW labels Dec 1, 2023

mergify bot assigned rickif Dec 1, 2023

rickif force-pushed the feat/timestamp-progress branch 2 times, most recently from 1e20ec0 to c87f0b9 Compare December 1, 2023 06:06

rickif force-pushed the feat/timestamp-progress branch from cdc2906 to 4c0bf28 Compare December 1, 2023 07:21

rickif changed the title ~~[Feature] Adds timestamp progress to routine load~~ [Feature] Adds routine_load_unstable_threshold_second Dec 5, 2023

rickif force-pushed the feat/timestamp-progress branch from 46e5ed7 to 2d9372e Compare December 6, 2023 07:37

rickif added 5 commits December 6, 2023 17:13

*: routine load timestamp progress

4d2e734

Signed-off-by: ricky <rickif@qq.com>

test: update RoutineLoadJobTest

fed6e47

Signed-off-by: ricky <rickif@qq.com>

test: update GlobalTransactionMgrTest

16a0ea2

Signed-off-by: ricky <rickif@qq.com>

routine_load: add substate UNSTABLE

716e747

Signed-off-by: ricky <rickif@qq.com>

test: update testGetShowInfo

ccb56d6

Signed-off-by: ricky <rickif@qq.com>

rickif force-pushed the feat/timestamp-progress branch from 2d9372e to ccb56d6 Compare December 6, 2023 09:13

test: update testGetShowInfo

4ee27c4

Signed-off-by: ricky <rickif@qq.com>

meegoo reviewed Dec 12, 2023

View reviewed changes

gensrc/thrift/FrontendService.thrift Outdated Show resolved Hide resolved

thrift: set partitionCmtOffsetTimestamp as optional

a41ce8b

Signed-off-by: ricky <rickif@qq.com>

zaorangyang approved these changes Dec 13, 2023

View reviewed changes

wyb reviewed Dec 13, 2023

View reviewed changes

fe/fe-core/src/main/java/com/starrocks/load/routineload/RLTaskTxnCommitAttachment.java Outdated Show resolved Hide resolved

Update fe/fe-core/src/main/java/com/starrocks/load/routineload/RLTask…

11c82b6

…TxnCommitAttachment.java Co-authored-by: wyb <wybb86@gmail.com> Signed-off-by: ricky <rickif@qq.com>

wyb approved these changes Dec 13, 2023

View reviewed changes

rickif added a commit to rickif/starrocks that referenced this pull request Dec 13, 2023

backport: [Feature] Adds routine_load_unstable_threshold_second StarR…

11e5484

…ocks#36222

rickif mentioned this pull request Dec 13, 2023

[Feature] Adds routine_load_unstable_threshold_second (backport #36222) (branch-2.5) #36990

Closed

rickif added a commit to rickif/starrocks that referenced this pull request Dec 13, 2023

backport: [Feature] Adds routine_load_unstable_threshold_second StarR…

5d0bb98

…ocks#36222 Signed-off-by: ricky <rickif@qq.com>

meegoo enabled auto-merge (squash) December 13, 2023 18:50

meegoo approved these changes Dec 13, 2023

View reviewed changes

HangyuanLiu approved these changes Dec 14, 2023

View reviewed changes

meegoo merged commit 4c9756f into StarRocks:main Dec 14, 2023
40 of 45 checks passed

rickif deleted the feat/timestamp-progress branch December 14, 2023 02:01

This was referenced Dec 14, 2023

[Feature] Adds routine_load_unstable_threshold_second (backport #36222) #36997

Closed

[Feature] Adds routine_load_unstable_threshold_second (backport #36222) #36998

Closed

[Feature] Adds routine_load_unstable_threshold_second (backport #36222) #36999

Closed

mergify bot mentioned this pull request Dec 14, 2023

[Feature] Adds routine_load_unstable_threshold_second (backport #36222) #37000

Closed

rickif added a commit to rickif/starrocks that referenced this pull request Dec 14, 2023

[Feature] Adds routine_load_unstable_threshold_second (StarRocks#36222)

f706f23

Signed-off-by: ricky <rickif@qq.com> Co-authored-by: wyb <wybb86@gmail.com>

rickif mentioned this pull request Dec 14, 2023

[Feature] Adds routine_load_unstable_threshold_second (backpor #36222) (branch-3.2) #37007

Merged

17 tasks

rickif added a commit to rickif/starrocks that referenced this pull request Dec 14, 2023

[Feature] Adds routine_load_unstable_threshold_second (StarRocks#36222)

902dd55

Signed-off-by: ricky <rickif@qq.com> Co-authored-by: wyb <wybb86@gmail.com>

rickif mentioned this pull request Dec 14, 2023

[Feature] Adds routine_load_unstable_threshold_second (backport #36222) (branch-3.1) #37008

Merged

rickif added a commit to rickif/starrocks that referenced this pull request Dec 14, 2023

[Feature] Adds routine_load_unstable_threshold_second (StarRocks#36222)

48e8259

Signed-off-by: ricky <rickif@qq.com> Co-authored-by: wyb <wybb86@gmail.com>

rickif mentioned this pull request Dec 14, 2023

[Feature] Adds routine_load_unstable_threshold_second (backport #36222) (branch-3.0) #37009

Merged

rickif added a commit to rickif/starrocks that referenced this pull request Dec 14, 2023

[Feature] Adds routine_load_unstable_threshold_second (StarRocks#36222)

31a78f0

Signed-off-by: ricky <rickif@qq.com> Co-authored-by: wyb <wybb86@gmail.com>

rickif mentioned this pull request Dec 14, 2023

[Feature] Adds routine_load_unstable_threshold_second (backport #36222) (branch-2.5) #37010

Merged

rickif added a commit to rickif/starrocks that referenced this pull request Dec 14, 2023

[Feature] Adds routine_load_unstable_threshold_second (StarRocks#36222)

5c16ad8

Signed-off-by: ricky <rickif@qq.com> Co-authored-by: wyb <wybb86@gmail.com>

wyb added a commit that referenced this pull request Dec 14, 2023

[Feature] Adds routine_load_unstable_threshold_second (backport #36222)…

48d0dfa

… (branch-2.5) (#37010) Signed-off-by: ricky <rickif@qq.com> Co-authored-by: wyb <wybb86@gmail.com>

wyb added a commit that referenced this pull request Dec 14, 2023

[Feature] Adds routine_load_unstable_threshold_second (backport #36222)…

ebac4b8

… (branch-3.0) (#37009) Signed-off-by: ricky <rickif@qq.com> Co-authored-by: wyb <wybb86@gmail.com>

wanpengfei-git pushed a commit that referenced this pull request Dec 15, 2023

[Feature] Adds routine_load_unstable_threshold_second (backport #36222)…

c3af3db

… (branch-3.1) (#37008) Signed-off-by: ricky <rickif@qq.com> Co-authored-by: wyb <wybb86@gmail.com>

andyziye pushed a commit that referenced this pull request Dec 21, 2023

[Feature] Adds routine_load_unstable_threshold_second (backpor #36222)…

2ddb971

… (branch-3.2) (#37007) Signed-off-by: ricky <rickif@qq.com> Co-authored-by: wyb <wybb86@gmail.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Feature] Adds routine_load_unstable_threshold_second #36222

[Feature] Adds routine_load_unstable_threshold_second #36222

rickif commented Dec 1, 2023 •

edited

Loading

rickif commented Dec 1, 2023

mergify bot commented Dec 1, 2023

sonarcloud bot commented Dec 13, 2023

github-actions bot commented Dec 13, 2023

github-actions bot commented Dec 13, 2023

rickif commented Dec 14, 2023

mergify bot commented Dec 14, 2023 •

edited

Loading

[Feature] Adds routine_load_unstable_threshold_second #36222

[Feature] Adds routine_load_unstable_threshold_second #36222

Conversation

rickif commented Dec 1, 2023 • edited Loading

What type of PR is this:

Checklist:

Bugfix cherry-pick branch check:

rickif commented Dec 1, 2023

mergify bot commented Dec 1, 2023

✅ Branch has been successfully rebased

sonarcloud bot commented Dec 13, 2023

Quality Gate passed

github-actions bot commented Dec 13, 2023

[FE Incremental Coverage Report]

file detail

github-actions bot commented Dec 13, 2023

[BE Incremental Coverage Report]

file detail

rickif commented Dec 14, 2023

mergify bot commented Dec 14, 2023 • edited Loading

✅ Backports have been created

rickif commented Dec 1, 2023 •

edited

Loading

mergify bot commented Dec 14, 2023 •

edited

Loading