Skip to content

[enhance](job) add zero-row hint for Kafka read_committed load#63664

Open
sollhui wants to merge 1 commit into
apache:masterfrom
sollhui:hint_for_rl
Open

[enhance](job) add zero-row hint for Kafka read_committed load#63664
sollhui wants to merge 1 commit into
apache:masterfrom
sollhui:hint_for_rl

Conversation

@sollhui
Copy link
Copy Markdown
Contributor

@sollhui sollhui commented May 26, 2026

What problem does this PR solve?

When Kafka routine load is configured with isolation.level=read_committed, the consumer may consume 0 rows while the task partition lag is still positive. This can happen when upstream producers use Kafka transactions and some records are not committed yet, so they are invisible to the read_committed consumer.

This PR adds an OtherMsg hint for this case:

  • routine load task consumes 0 rows
  • task partition lag is positive
  • Kafka property isolation.level=read_committed is configured

The message helps users distinguish this case from ordinary no-data consumption. A FE debug point and regression test are added to ensure the warning can be reported deterministically.

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@hello-stephen
Copy link
Copy Markdown
Contributor

hello-stephen commented May 26, 2026

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@sollhui
Copy link
Copy Markdown
Contributor Author

sollhui commented May 26, 2026

run buildall

@sollhui
Copy link
Copy Markdown
Contributor Author

sollhui commented May 26, 2026

run buildall

@hello-stephen
Copy link
Copy Markdown
Contributor

hello-stephen commented May 26, 2026

TPC-H: Total hot run time: 31912 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 9ec418d61049def531f33d232cf8fd0f3f1c8c7c, data reload: false

------ Round 1 ----------------------------------
orders	Doris	NULL	NULL	0	0	0	NULL	0	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:42:55	NULL	utf-8	NULL	NULL	
============================================
q1	17703	4224	4076	4076
q2	q3	10763	1411	848	848
q4	4689	486	349	349
q5	7532	2323	2116	2116
q6	242	180	136	136
q7	944	780	639	639
q8	9383	1704	1599	1599
q9	5172	5017	4965	4965
q10	6399	2206	1882	1882
q11	434	286	241	241
q12	622	443	298	298
q13	18189	3366	2821	2821
q14	263	256	246	246
q15	q16	825	768	711	711
q17	1001	1010	983	983
q18	7110	5842	5440	5440
q19	1459	1289	1259	1259
q20	565	461	297	297
q21	6307	2877	2690	2690
q22	459	399	316	316
Total cold run time: 100061 ms
Total hot run time: 31912 ms

----- Round 2, with runtime_filter_mode=off -----
orders	Doris	NULL	NULL	150000000	42	6422171781	NULL	22778155	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:42:55	NULL	utf-8	NULL	NULL	
============================================
q1	4931	4879	4852	4852
q2	q3	4998	5287	4601	4601
q4	2156	2261	1446	1446
q5	5045	4772	4719	4719
q6	232	175	128	128
q7	1899	1832	1638	1638
q8	2485	2209	2264	2209
q9	8206	7603	7468	7468
q10	4793	4716	4257	4257
q11	553	393	362	362
q12	748	757	557	557
q13	3023	3358	2805	2805
q14	279	290	269	269
q15	q16	694	708	632	632
q17	1308	1281	1278	1278
q18	7259	7041	6931	6931
q19	1072	1094	1108	1094
q20	2262	2218	1959	1959
q21	5393	4679	4544	4544
q22	530	457	450	450
Total cold run time: 57866 ms
Total hot run time: 52199 ms

@hello-stephen
Copy link
Copy Markdown
Contributor

hello-stephen commented May 26, 2026

TPC-DS: Total hot run time: 173704 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 9ec418d61049def531f33d232cf8fd0f3f1c8c7c, data reload: false

query5	4322	652	533	533
query6	436	233	215	215
query7	4229	585	326	326
query8	339	245	228	228
query9	8873	4181	4195	4181
query10	448	352	303	303
query11	5822	2675	2289	2289
query12	185	131	135	131
query13	1312	644	426	426
query14	6168	5588	5260	5260
query14_1	4597	4561	4546	4546
query15	220	213	188	188
query16	1030	495	465	465
query17	1171	743	620	620
query18	2705	501	377	377
query19	227	207	175	175
query20	142	141	131	131
query21	221	141	120	120
query22	13733	13569	13462	13462
query23	17499	16583	16370	16370
query23_1	16518	16487	16390	16390
query24	7456	1809	1334	1334
query24_1	1355	1349	1331	1331
query25	570	499	452	452
query26	1306	333	185	185
query27	2656	577	354	354
query28	4444	2055	2027	2027
query29	1003	658	518	518
query30	307	243	204	204
query31	1142	1087	968	968
query32	97	81	76	76
query33	571	369	350	350
query34	1190	1142	679	679
query35	769	797	689	689
query36	1435	1452	1276	1276
query37	161	104	87	87
query38	3226	3222	3074	3074
query39	933	923	890	890
query39_1	874	865	898	865
query40	228	149	126	126
query41	65	64	62	62
query42	111	108	106	106
query43	340	341	300	300
query44	
query45	207	205	201	201
query46	1087	1196	757	757
query47	2360	2421	2301	2301
query48	404	429	298	298
query49	641	496	387	387
query50	1030	343	260	260
query51	4340	4328	4257	4257
query52	111	106	93	93
query53	262	291	208	208
query54	309	278	253	253
query55	95	98	86	86
query56	303	298	311	298
query57	1441	1418	1353	1353
query58	302	278	273	273
query59	1616	1679	1451	1451
query60	326	315	309	309
query61	162	158	157	157
query62	704	660	595	595
query63	252	200	204	200
query64	2404	809	677	677
query65	
query66	1690	502	369	369
query67	29920	29923	29784	29784
query68	
query69	460	345	320	320
query70	1051	1022	1006	1006
query71	305	274	268	268
query72	3092	2776	2434	2434
query73	809	724	427	427
query74	5132	5027	4810	4810
query75	2696	2631	2308	2308
query76	2306	1225	782	782
query77	415	421	328	328
query78	12548	12570	11922	11922
query79	1517	1036	774	774
query80	733	559	462	462
query81	471	282	252	252
query82	1375	155	122	122
query83	329	268	246	246
query84	254	146	108	108
query85	889	544	465	465
query86	404	327	348	327
query87	3446	3387	3266	3266
query88	3647	2796	2756	2756
query89	453	404	344	344
query90	1838	182	183	182
query91	179	169	142	142
query92	82	80	76	76
query93	1598	1553	863	863
query94	594	355	326	326
query95	691	475	343	343
query96	1075	835	353	353
query97	2742	2752	2616	2616
query98	236	231	232	231
query99	1199	1153	1027	1027
Total cold run time: 256023 ms
Total hot run time: 173704 ms

@hello-stephen
Copy link
Copy Markdown
Contributor

hello-stephen commented May 26, 2026

FE Regression Coverage Report

Increment line coverage 3.37% (3/89) 🎉
Increment coverage report
Complete coverage report

@sollhui
Copy link
Copy Markdown
Contributor Author

sollhui commented May 26, 2026

run buildall

@hello-stephen
Copy link
Copy Markdown
Contributor

hello-stephen commented May 26, 2026

TPC-H: Total hot run time: 31818 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 20878c79937957f899283ecb1d07d5fbecffff40, data reload: false

------ Round 1 ----------------------------------
orders	Doris	NULL	NULL	0	0	0	NULL	0	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:42:55	NULL	utf-8	NULL	NULL	
============================================
q1	17677	4216	4126	4126
q2	q3	10803	1390	849	849
q4	4686	476	342	342
q5	7557	2246	2107	2107
q6	240	175	136	136
q7	911	785	664	664
q8	9419	1691	1533	1533
q9	5190	4982	4966	4966
q10	6416	2211	1854	1854
q11	428	278	253	253
q12	632	420	287	287
q13	18226	3429	2776	2776
q14	266	260	240	240
q15	q16	823	766	706	706
q17	1011	975	987	975
q18	7092	5674	5488	5488
q19	1325	1277	1206	1206
q20	574	438	294	294
q21	6229	2929	2708	2708
q22	562	362	308	308
Total cold run time: 100067 ms
Total hot run time: 31818 ms

----- Round 2, with runtime_filter_mode=off -----
orders	Doris	NULL	NULL	150000000	42	6422171781	NULL	22778155	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:42:55	NULL	utf-8	NULL	NULL	
============================================
q1	4926	4894	4903	4894
q2	q3	5054	5318	4638	4638
q4	2140	2209	1443	1443
q5	5019	4672	4725	4672
q6	230	177	128	128
q7	1952	1785	1585	1585
q8	2417	2137	2141	2137
q9	8127	7440	7490	7440
q10	4746	4675	4251	4251
q11	548	389	369	369
q12	749	742	539	539
q13	3026	3341	2774	2774
q14	278	286	257	257
q15	q16	685	704	616	616
q17	1313	1273	1271	1271
q18	7413	6835	6797	6797
q19	1125	1060	1117	1060
q20	2247	2221	1951	1951
q21	5312	4620	4510	4510
q22	509	448	415	415
Total cold run time: 57816 ms
Total hot run time: 51747 ms

@hello-stephen
Copy link
Copy Markdown
Contributor

hello-stephen commented May 26, 2026

TPC-DS: Total hot run time: 172935 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 20878c79937957f899283ecb1d07d5fbecffff40, data reload: false

query5	4336	678	527	527
query6	323	217	207	207
query7	4283	572	314	314
query8	332	231	220	220
query9	8833	4096	4066	4066
query10	469	353	305	305
query11	5796	2601	2246	2246
query12	185	129	130	129
query13	1279	591	432	432
query14	6244	5557	5182	5182
query14_1	4473	4498	4485	4485
query15	210	209	187	187
query16	1008	448	429	429
query17	1126	717	591	591
query18	2718	476	354	354
query19	217	204	162	162
query20	137	130	137	130
query21	217	138	115	115
query22	13685	13628	13499	13499
query23	17265	16599	16298	16298
query23_1	16305	16393	16451	16393
query24	7642	1786	1310	1310
query24_1	1363	1320	1313	1313
query25	588	526	448	448
query26	1307	322	176	176
query27	2690	586	349	349
query28	4415	1984	2040	1984
query29	994	659	529	529
query30	305	245	203	203
query31	1142	1085	957	957
query32	92	80	78	78
query33	555	363	312	312
query34	1183	1139	659	659
query35	780	799	713	713
query36	1446	1437	1257	1257
query37	155	105	95	95
query38	3228	3217	3102	3102
query39	961	915	898	898
query39_1	881	873	886	873
query40	230	156	139	139
query41	72	69	69	69
query42	113	113	109	109
query43	341	363	309	309
query44	
query45	215	208	196	196
query46	1081	1240	745	745
query47	2373	2427	2261	2261
query48	408	448	310	310
query49	687	510	398	398
query50	987	375	289	289
query51	4364	4333	4276	4276
query52	108	107	98	98
query53	260	290	210	210
query54	338	292	280	280
query55	99	99	89	89
query56	316	326	314	314
query57	1433	1433	1314	1314
query58	299	302	281	281
query59	1650	1736	1492	1492
query60	334	331	324	324
query61	184	178	190	178
query62	708	654	597	597
query63	252	204	207	204
query64	2374	817	645	645
query65	
query66	1690	481	358	358
query67	29780	29791	29553	29553
query68	
query69	459	339	306	306
query70	1113	970	1022	970
query71	304	276	267	267
query72	3045	2684	2451	2451
query73	839	812	415	415
query74	5135	4968	4825	4825
query75	2687	2616	2267	2267
query76	2323	1142	789	789
query77	426	427	344	344
query78	12500	12406	11920	11920
query79	1500	1081	721	721
query80	654	548	448	448
query81	455	290	244	244
query82	1404	162	121	121
query83	361	276	252	252
query84	268	145	115	115
query85	887	571	452	452
query86	408	378	316	316
query87	3433	3410	3280	3280
query88	3616	2747	2745	2745
query89	448	391	344	344
query90	1964	182	180	180
query91	181	176	141	141
query92	81	81	74	74
query93	1469	1405	849	849
query94	570	371	321	321
query95	684	466	344	344
query96	1102	774	354	354
query97	2740	2741	2586	2586
query98	237	232	229	229
query99	1198	1149	1027	1027
Total cold run time: 255685 ms
Total hot run time: 172935 ms

@hello-stephen
Copy link
Copy Markdown
Contributor

hello-stephen commented May 26, 2026

FE UT Coverage Report

Increment line coverage 57.14% (8/14) 🎉
Increment coverage report
Complete coverage report

@hello-stephen
Copy link
Copy Markdown
Contributor

hello-stephen commented May 26, 2026

FE Regression Coverage Report

Increment line coverage 21.43% (3/14) 🎉
Increment coverage report
Complete coverage report

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@liaoxin01 liaoxin01 Awaiting requested review from liaoxin01 liaoxin01 is a code owner

@gavinchou gavinchou Awaiting requested review from gavinchou gavinchou is a code owner

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@sollhui @hello-stephen