Skip to content

[feat](dynamic table) support table stream part 3: docking olap table stram with binlog#63850

Open
TsukiokaKogane wants to merge 9 commits into
apache:masterfrom
TsukiokaKogane:feature/dock_table_stream_and_binlog
Open

[feat](dynamic table) support table stream part 3: docking olap table stram with binlog#63850
TsukiokaKogane wants to merge 9 commits into
apache:masterfrom
TsukiokaKogane:feature/dock_table_stream_and_binlog

Conversation

@TsukiokaKogane
Copy link
Copy Markdown
Contributor

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #57921

Problem Summary:

  1. docking olap table stram with binlog
  • tso meta refactor
  • min delta / append-only scan mode support
  1. support changes syntax
  2. table stream manager now supports to clean up stale db/stream/partition info

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@hello-stephen
Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@TsukiokaKogane
Copy link
Copy Markdown
Contributor Author

run buildall

@TsukiokaKogane
Copy link
Copy Markdown
Contributor Author

/review

@TsukiokaKogane
Copy link
Copy Markdown
Contributor Author

run buildall

@hello-stephen
Copy link
Copy Markdown
Contributor

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 78.06% (1854/2375)
Line Coverage 64.50% (33326/51665)
Region Coverage 65.21% (16525/25343)
Branch Coverage 55.74% (8837/15854)

@hello-stephen
Copy link
Copy Markdown
Contributor

FE UT Coverage Report

Increment line coverage 27.08% (231/853) 🎉
Increment coverage report
Complete coverage report

@TsukiokaKogane TsukiokaKogane force-pushed the feature/dock_table_stream_and_binlog branch from 83e734b to 3fe0d33 Compare May 29, 2026 08:46
@TsukiokaKogane
Copy link
Copy Markdown
Contributor Author

run buildall

@TsukiokaKogane
Copy link
Copy Markdown
Contributor Author

/review

@TsukiokaKogane
Copy link
Copy Markdown
Contributor Author

run fe_ut

@TsukiokaKogane
Copy link
Copy Markdown
Contributor Author

run buildall

@hello-stephen
Copy link
Copy Markdown
Contributor

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 78.06% (1854/2375)
Line Coverage 64.51% (33326/51663)
Region Coverage 65.18% (16518/25343)
Branch Coverage 55.71% (8833/15854)

@TsukiokaKogane
Copy link
Copy Markdown
Contributor Author

run buildall

@hello-stephen
Copy link
Copy Markdown
Contributor

FE UT Coverage Report

Increment line coverage 46.15% (396/858) 🎉
Increment coverage report
Complete coverage report

@hello-stephen
Copy link
Copy Markdown
Contributor

TPC-H: Total hot run time: 31461 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 452362c2f8c8ea6cee4e7d66ac3296c6b60af999, data reload: false

------ Round 1 ----------------------------------
orders	Doris	NULL	NULL	0	0	0	NULL	0	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:42:55	NULL	utf-8	NULL	NULL	
============================================
q1	17602	4105	4072	4072
q2	q3	10766	1418	851	851
q4	4688	479	354	354
q5	7711	2279	2176	2176
q6	241	177	141	141
q7	932	789	644	644
q8	9397	1742	1544	1544
q9	6466	4994	5004	4994
q10	6457	2256	1881	1881
q11	433	283	249	249
q12	693	430	307	307
q13	18177	3842	2849	2849
q14	266	261	235	235
q15	q16	842	779	701	701
q17	937	992	996	992
q18	6922	5781	5489	5489
q19	1186	1301	1088	1088
q20	526	400	268	268
q21	5721	2665	2322	2322
q22	436	361	304	304
Total cold run time: 100399 ms
Total hot run time: 31461 ms

----- Round 2, with runtime_filter_mode=off -----
orders	Doris	NULL	NULL	150000000	42	6422171781	NULL	22778155	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:42:55	NULL	utf-8	NULL	NULL	
============================================
q1	4411	4379	4311	4311
q2	q3	4562	4989	4328	4328
q4	2099	2217	1376	1376
q5	4465	4372	4952	4372
q6	275	214	159	159
q7	2075	1880	1642	1642
q8	2577	2267	2267	2267
q9	8306	8054	8258	8054
q10	4868	4874	4354	4354
q11	621	473	421	421
q12	793	793	583	583
q13	3422	3718	3087	3087
q14	330	329	295	295
q15	q16	733	749	686	686
q17	1423	1572	1399	1399
q18	8086	7398	7373	7373
q19	1150	1072	1120	1072
q20	2236	2228	1950	1950
q21	5302	4668	4511	4511
q22	537	468	422	422
Total cold run time: 58271 ms
Total hot run time: 52662 ms

@hello-stephen
Copy link
Copy Markdown
Contributor

TPC-H: Total hot run time: 31897 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 938e4460688bf5c0eec911c75e20766113b8f4a3, data reload: false

------ Round 1 ----------------------------------
orders	Doris	NULL	NULL	0	0	0	NULL	0	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:42:55	NULL	utf-8	NULL	NULL	
============================================
q1	17645	4237	4052	4052
q2	q3	10844	1420	852	852
q4	4757	488	347	347
q5	9948	2335	2123	2123
q6	363	183	138	138
q7	981	784	630	630
q8	9588	1730	1610	1610
q9	7076	5030	5021	5021
q10	6467	2239	1924	1924
q11	456	285	242	242
q12	689	429	296	296
q13	18206	3420	2785	2785
q14	275	256	232	232
q15	q16	817	768	711	711
q17	1021	861	983	861
q18	6975	5758	6389	5758
q19	1229	1324	1087	1087
q20	527	443	272	272
q21	6050	2883	2631	2631
q22	452	370	325	325
Total cold run time: 104366 ms
Total hot run time: 31897 ms

----- Round 2, with runtime_filter_mode=off -----
orders	Doris	NULL	NULL	150000000	42	6422171781	NULL	22778155	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:42:55	NULL	utf-8	NULL	NULL	
============================================
q1	4794	4839	4770	4770
q2	q3	4837	5507	4678	4678
q4	2122	2225	1426	1426
q5	4822	4809	4651	4651
q6	231	178	131	131
q7	1931	1665	1619	1619
q8	2434	2092	1958	1958
q9	7499	7469	7431	7431
q10	4771	4650	4213	4213
q11	537	385	356	356
q12	730	738	518	518
q13	3014	3393	2851	2851
q14	270	280	265	265
q15	q16	679	695	615	615
q17	1305	1299	1300	1299
q18	7326	7130	7009	7009
q19	1145	1101	1110	1101
q20	2247	2223	1940	1940
q21	5270	4636	4503	4503
q22	539	474	411	411
Total cold run time: 56503 ms
Total hot run time: 51745 ms

@hello-stephen
Copy link
Copy Markdown
Contributor

TPC-DS: Total hot run time: 172176 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 452362c2f8c8ea6cee4e7d66ac3296c6b60af999, data reload: false

query5	4360	665	520	520
query6	345	223	193	193
query7	4256	559	324	324
query8	328	234	215	215
query9	8777	4120	4069	4069
query10	466	345	324	324
query11	5799	2394	2214	2214
query12	177	129	129	129
query13	1293	629	443	443
query14	6141	5467	5161	5161
query14_1	4526	4476	4473	4473
query15	211	214	185	185
query16	1008	476	434	434
query17	1125	727	596	596
query18	2527	475	351	351
query19	220	202	160	160
query20	139	142	141	141
query21	221	137	117	117
query22	13814	13639	13440	13440
query23	17594	16401	16125	16125
query23_1	16351	16304	16380	16304
query24	7448	1753	1348	1348
query24_1	1329	1331	1318	1318
query25	558	486	441	441
query26	1328	321	178	178
query27	2683	581	351	351
query28	4439	2041	2018	2018
query29	995	648	534	534
query30	298	242	207	207
query31	1114	1087	968	968
query32	91	79	78	78
query33	551	374	322	322
query34	1204	1129	654	654
query35	805	795	703	703
query36	1398	1418	1235	1235
query37	163	109	95	95
query38	3212	3179	3097	3097
query39	937	919	900	900
query39_1	886	880	889	880
query40	232	154	130	130
query41	73	68	68	68
query42	111	115	114	114
query43	342	340	295	295
query44	
query45	219	206	207	206
query46	1078	1189	745	745
query47	2379	2384	2308	2308
query48	418	416	303	303
query49	665	549	407	407
query50	1018	359	261	261
query51	4396	4267	4316	4267
query52	112	109	97	97
query53	259	290	207	207
query54	333	291	286	286
query55	98	95	89	89
query56	327	332	323	323
query57	1457	1453	1319	1319
query58	318	291	270	270
query59	1615	1687	1453	1453
query60	340	356	333	333
query61	192	188	190	188
query62	698	664	596	596
query63	255	208	205	205
query64	2466	819	629	629
query65	
query66	1677	488	361	361
query67	29923	29648	29578	29578
query68	
query69	463	346	316	316
query70	1040	1025	1032	1025
query71	312	277	273	273
query72	2984	2724	2473	2473
query73	874	851	446	446
query74	5078	4941	4794	4794
query75	2701	2593	2276	2276
query76	2320	1134	807	807
query77	415	409	333	333
query78	12463	12475	11940	11940
query79	1501	1066	746	746
query80	635	543	456	456
query81	460	280	240	240
query82	1329	160	124	124
query83	351	285	249	249
query84	272	145	108	108
query85	882	536	448	448
query86	406	335	334	334
query87	3431	3419	3253	3253
query88	3651	2771	2771	2771
query89	445	392	341	341
query90	2016	194	183	183
query91	180	173	141	141
query92	80	79	78	78
query93	1416	1393	927	927
query94	526	349	321	321
query95	693	481	381	381
query96	1103	797	341	341
query97	2762	2732	2630	2630
query98	237	231	228	228
query99	1184	1160	1033	1033
Total cold run time: 255383 ms
Total hot run time: 172176 ms

@hello-stephen
Copy link
Copy Markdown
Contributor

TPC-DS: Total hot run time: 172656 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 938e4460688bf5c0eec911c75e20766113b8f4a3, data reload: false

query5	4327	678	538	538
query6	341	219	214	214
query7	4311	585	311	311
query8	323	237	250	237
query9	8811	4119	4128	4119
query10	449	348	305	305
query11	5826	2413	2295	2295
query12	191	129	125	125
query13	1291	598	442	442
query14	6180	5501	5202	5202
query14_1	4562	4547	4449	4449
query15	225	206	186	186
query16	1060	478	456	456
query17	1141	721	578	578
query18	2550	481	359	359
query19	211	202	172	172
query20	138	133	128	128
query21	222	135	120	120
query22	13607	13564	13323	13323
query23	17521	16573	16310	16310
query23_1	16375	16394	16309	16309
query24	7506	1778	1334	1334
query24_1	1338	1296	1323	1296
query25	541	476	424	424
query26	1329	337	175	175
query27	2671	581	366	366
query28	4424	2047	2010	2010
query29	989	626	538	538
query30	301	239	194	194
query31	1136	1086	964	964
query32	85	75	77	75
query33	543	359	320	320
query34	1212	1126	660	660
query35	786	805	720	720
query36	1396	1453	1271	1271
query37	156	114	96	96
query38	3261	3168	3126	3126
query39	944	926	885	885
query39_1	879	873	884	873
query40	227	144	128	128
query41	65	63	61	61
query42	112	109	110	109
query43	330	336	304	304
query44	
query45	218	205	196	196
query46	1105	1229	710	710
query47	2396	2395	2253	2253
query48	392	423	297	297
query49	638	513	393	393
query50	970	343	256	256
query51	4408	4247	4270	4247
query52	106	106	98	98
query53	256	277	206	206
query54	318	286	262	262
query55	94	93	97	93
query56	299	299	316	299
query57	1434	1451	1336	1336
query58	298	276	270	270
query59	1582	1656	1403	1403
query60	325	332	314	314
query61	154	157	155	155
query62	707	642	608	608
query63	248	204	212	204
query64	2433	861	661	661
query65	
query66	1699	487	365	365
query67	29809	29626	29584	29584
query68	
query69	480	345	315	315
query70	1016	993	986	986
query71	310	283	272	272
query72	3069	2770	2432	2432
query73	850	802	441	441
query74	5131	5003	4826	4826
query75	2691	2634	2297	2297
query76	2317	1194	783	783
query77	427	446	361	361
query78	12416	12371	11826	11826
query79	1274	1016	794	794
query80	631	579	497	497
query81	459	287	251	251
query82	246	161	134	134
query83	281	290	261	261
query84	272	147	115	115
query85	948	630	552	552
query86	378	337	326	326
query87	3436	3394	3289	3289
query88	3635	2762	2740	2740
query89	443	383	347	347
query90	2196	201	192	192
query91	197	185	159	159
query92	84	85	113	85
query93	1427	1483	825	825
query94	557	372	318	318
query95	684	398	366	366
query96	1040	817	383	383
query97	2706	2741	2682	2682
query98	238	231	225	225
query99	1172	1148	1040	1040
Total cold run time: 253641 ms
Total hot run time: 172656 ms

@hello-stephen
Copy link
Copy Markdown
Contributor

FE Regression Coverage Report

Increment line coverage 17.13% (147/858) 🎉
Increment coverage report
Complete coverage report

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants