[Feature](tvf) Support cdc stream tvf for mysql and pg by JNSimba · Pull Request #60116 · apache/doris

JNSimba · 2026-01-21T09:52:45Z

What problem does this PR solve?

#58898 #59461 These two features support consuming data from MySQL and PostgreSQL.

Based on this, this PR introduces cdc_stream tvf, which allows consuming data from MySQL and PostgreSQL in TVF format.

Example

select * from cdc_stream(
    "type" = "mysql",
    "jdbc_url" = "jdbc:mysql://127.0.0.1:3306",
    "driver_url" = "mysql-connector-j-8.0.31.jar",
    "driver_class" = "com.mysql.cj.jdbc.Driver",
    "user" = "root",
    "password" = "123456",
    "database" = "test_cdc_db",
    "table" = "user_info",
    "offset" = '{"file":"binlog.000003","pos":"496"}'
)

Release note

None

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

hello-stephen · 2026-01-21T09:52:50Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

Copilot

Pull request overview

This PR introduces a cdc_stream table-valued function (TVF) that enables consuming Change Data Capture (CDC) data from MySQL and PostgreSQL databases directly through SQL queries. The feature builds upon existing CDC streaming job capabilities (#58898, #59461) by exposing them through a TVF interface.

Changes:

Added cdc_stream TVF support for MySQL and PostgreSQL CDC streaming
Refactored CDC client to use String-based job IDs instead of Long to support both jobs and TVF queries
Implemented HTTP streaming response support in the backend for efficient CDC data delivery

Reviewed changes

Copilot reviewed 31 out of 31 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
test_cdc_stream_tvf_mysql.groovy	Regression test for MySQL CDC stream TVF functionality
test_cdc_stream_tvf_postgres.groovy	Regression test for PostgreSQL CDC stream TVF functionality
test_cdc_stream_tvf_mysql.out	Expected output for MySQL CDC stream TVF test
ConfigUtil.java	Added utility method for table list extraction and changed serverId parameter type from long to String
PostgresSourceReader.java	Updated to support String job IDs and simplified offset handling for TVF
MySqlSourceReader.java	Refactored to always create new split readers and support String job IDs
SourceReader.java	Changed interface to use String job IDs
JdbcIncrementalSourceReader.java	Simplified split reader logic to always create new readers
DorisBatchStreamLoad.java	Updated to use String job IDs
PipelineCoordinator.java	Added streaming response support and TVF-specific record fetching
StreamException.java	New exception for streaming response errors
CommonException.java	New generic exception for common errors
ClientController.java	Added REST endpoints for streaming responses and task offset retrieval
GlobalExceptionHandler.java	Added handlers for new exception types
Env.java	Updated to use String-based job context keys
CdcStreamTableValuedFunction.java	New TVF implementation for CDC streaming
TableValuedFunctionVisitor.java	Added visitor method for CDC stream TVF
CdcStream.java	Nereids expression class for CDC stream TVF
StreamingJobUtils.java	Made utility methods public for TVF usage
JdbcSourceOffsetProvider.java	Updated to convert job IDs to String for CDC client
StreamingMultiTblTask.java	Updated to pass job ID as String
BuiltinTableValuedFunctions.java	Registered cdc_stream TVF
WriteRecordRequest.java	Removed unused abstract method implementations
JobBaseRecordRequest.java	Removed unused abstract method declarations
JobBaseConfig.java	Changed jobId type from Long to String
FetchTableSplitsRequest.java	Updated to convert Long job ID to String
FetchRecordRequest.java	Simplified by removing unused fields
CompareOffsetRequest.java	Updated to convert Long job ID to String
DataSourceConfigKeys.java	Added TYPE and TABLE configuration keys
http_file_reader.h	Added chunk response support flag
http_file_reader.cpp	Implemented CDC client integration, chunk response support, and HTTP error handling

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

...main/java/org/apache/doris/nereids/trees/expressions/visitor/TableValuedFunctionVisitor.java

...core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/table/CdcStream.java

fs_brokers/cdc_client/src/main/java/org/apache/doris/cdcclient/service/PipelineCoordinator.java

fs_brokers/cdc_client/src/main/java/org/apache/doris/cdcclient/utils/ConfigUtil.java

be/src/io/fs/http_file_reader.cpp

JNSimba · 2026-01-21T10:04:45Z

run buildall

doris-robot · 2026-01-21T11:20:12Z

TPC-H: Total hot run time: 32113 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit d84fcddbe4aa9142e912da69ab6bafe20075ca2e, data reload: false

------ Round 1 ----------------------------------
q1	17610	4269	4073	4073
q2	2069	361	255	255
q3	10100	1284	721	721
q4	10193	773	301	301
q5	7490	2100	1882	1882
q6	189	168	134	134
q7	940	791	644	644
q8	9260	1468	1110	1110
q9	5010	4541	4588	4541
q10	6807	1781	1393	1393
q11	527	299	274	274
q12	760	745	605	605
q13	17803	3861	3109	3109
q14	300	302	269	269
q15	597	522	515	515
q16	731	689	647	647
q17	665	736	598	598
q18	6856	6500	6937	6500
q19	1661	1078	702	702
q20	412	373	301	301
q21	3327	2643	2501	2501
q22	1113	1064	1038	1038
Total cold run time: 104420 ms
Total hot run time: 32113 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4342	4275	4369	4275
q2	333	425	326	326
q3	2253	2871	2521	2521
q4	1426	1851	1408	1408
q5	4578	4358	4279	4279
q6	213	161	127	127
q7	1972	1936	1809	1809
q8	2547	2399	2357	2357
q9	7287	7500	7152	7152
q10	2518	2648	2296	2296
q11	540	480	461	461
q12	683	727	565	565
q13	3381	3845	3113	3113
q14	264	280	265	265
q15	533	486	476	476
q16	627	649	639	639
q17	1087	1255	1306	1255
q18	7352	7303	7056	7056
q19	846	792	833	792
q20	1872	1956	1797	1797
q21	4521	4234	4070	4070
q22	1070	1079	1009	1009
Total cold run time: 50245 ms
Total hot run time: 48048 ms

doris-robot · 2026-01-21T11:30:58Z

TPC-DS: Total hot run time: 174132 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit d84fcddbe4aa9142e912da69ab6bafe20075ca2e, data reload: false

query5	4430	629	510	510
query6	335	228	220	220
query7	4204	471	258	258
query8	347	242	237	237
query9	8640	2905	2870	2870
query10	523	382	353	353
query11	15309	15088	14903	14903
query12	176	119	121	119
query13	1266	471	392	392
query14	6338	3048	2824	2824
query14_1	2704	2657	2697	2657
query15	205	191	170	170
query16	1004	503	455	455
query17	1106	667	581	581
query18	2509	437	337	337
query19	224	225	197	197
query20	130	117	118	117
query21	232	137	119	119
query22	3944	4032	3935	3935
query23	16131	15574	15325	15325
query23_1	15411	15553	15299	15299
query24	7160	1574	1182	1182
query24_1	1181	1197	1227	1197
query25	545	448	411	411
query26	1241	267	155	155
query27	2750	464	288	288
query28	4502	2170	2160	2160
query29	799	549	451	451
query30	313	231	206	206
query31	804	637	577	577
query32	86	77	75	75
query33	538	366	329	329
query34	897	871	562	562
query35	747	759	685	685
query36	869	877	861	861
query37	137	99	89	89
query38	2791	2765	2672	2672
query39	763	737	739	737
query39_1	715	736	710	710
query40	224	140	128	128
query41	79	69	68	68
query42	107	106	106	106
query43	458	450	423	423
query44	1372	751	750	750
query45	188	204	188	188
query46	860	1011	582	582
query47	1458	1444	1388	1388
query48	311	339	241	241
query49	616	436	341	341
query50	625	275	215	215
query51	3786	3758	3711	3711
query52	105	108	98	98
query53	294	325	273	273
query54	295	279	255	255
query55	84	80	79	79
query56	318	329	300	300
query57	1018	1011	950	950
query58	269	263	268	263
query59	2015	2095	1964	1964
query60	346	337	317	317
query61	148	143	141	141
query62	387	362	286	286
query63	304	270	263	263
query64	4923	1256	955	955
query65	3774	3722	3738	3722
query66	1444	421	322	322
query67	15653	15616	15430	15430
query68	2467	1106	754	754
query69	441	372	372	372
query70	988	971	951	951
query71	332	329	299	299
query72	5313	3193	3290	3193
query73	601	740	319	319
query74	8734	8778	8606	8606
query75	2760	2835	2524	2524
query76	2289	1068	672	672
query77	375	392	300	300
query78	9722	9937	9162	9162
query79	1101	934	588	588
query80	1293	583	494	494
query81	551	264	233	233
query82	991	146	110	110
query83	337	265	250	250
query84	258	111	95	95
query85	922	489	425	425
query86	411	285	320	285
query87	2912	2926	2786	2786
query88	3521	2586	2570	2570
query89	405	356	328	328
query90	1960	180	168	168
query91	169	154	133	133
query92	72	71	71	71
query93	1039	940	545	545
query94	650	331	300	300
query95	586	340	320	320
query96	653	520	234	234
query97	2366	2431	2349	2349
query98	211	204	203	203
query99	587	569	471	471
Total cold run time: 247625 ms
Total hot run time: 174132 ms

doris-robot · 2026-01-21T11:35:58Z

ClickBench: Total hot run time: 26.75 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit d84fcddbe4aa9142e912da69ab6bafe20075ca2e, data reload: false

query1	0.05	0.04	0.05
query2	0.09	0.05	0.04
query3	0.26	0.09	0.08
query4	1.61	0.12	0.11
query5	0.28	0.27	0.25
query6	1.14	0.67	0.65
query7	0.02	0.02	0.03
query8	0.05	0.04	0.04
query9	0.57	0.50	0.49
query10	0.55	0.54	0.53
query11	0.14	0.10	0.10
query12	0.14	0.11	0.11
query13	0.60	0.58	0.58
query14	0.95	0.93	0.93
query15	0.78	0.77	0.78
query16	0.38	0.41	0.41
query17	1.08	1.04	1.06
query18	0.22	0.21	0.22
query19	1.95	1.89	1.84
query20	0.02	0.01	0.01
query21	15.46	0.25	0.14
query22	5.32	0.05	0.04
query23	16.11	0.28	0.10
query24	1.62	0.24	0.52
query25	0.09	0.07	0.08
query26	0.14	0.14	0.13
query27	0.06	0.08	0.06
query28	3.74	1.07	0.88
query29	12.53	3.89	3.13
query30	0.28	0.13	0.12
query31	2.84	0.63	0.39
query32	3.23	0.55	0.46
query33	3.00	3.05	3.04
query34	15.85	5.07	4.43
query35	4.47	4.46	4.44
query36	0.66	0.50	0.49
query37	0.10	0.07	0.06
query38	0.07	0.03	0.03
query39	0.04	0.03	0.03
query40	0.17	0.13	0.13
query41	0.09	0.04	0.03
query42	0.05	0.03	0.03
query43	0.05	0.03	0.03
Total cold run time: 96.85 s
Total hot run time: 26.75 s

hello-stephen · 2026-01-21T14:12:43Z

BE UT Coverage Report

Increment line coverage 0.00% (0/79) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	52.41% (19093/36431)
Line Coverage	35.77% (177218/495488)
Region Coverage	32.25% (137009/424848)
Branch Coverage	33.17% (59277/178717)

hello-stephen · 2026-01-21T15:45:41Z

BE Regression && UT Coverage Report

Increment line coverage 65.82% (52/79) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	73.15% (26126/35718)
Line Coverage	56.24% (278395/494978)
Region Coverage	54.02% (231974/429398)
Branch Coverage	55.69% (99963/179498)

hello-stephen · 2026-01-21T15:58:07Z

FE Regression Coverage Report

Increment line coverage 3.08% (2/65) 🎉
Increment coverage report
Complete coverage report

Copilot

Pull request overview

Copilot reviewed 31 out of 31 changed files in this pull request and generated 25 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

fs_brokers/cdc_client/src/main/java/org/apache/doris/cdcclient/controller/ClientController.java

fs_brokers/cdc_client/src/main/java/org/apache/doris/cdcclient/utils/ConfigUtil.java

regression-test/suites/job_p0/streaming_job/cdc/tvf/test_cdc_stream_tvf_postgres.groovy

fe/fe-common/src/main/java/org/apache/doris/job/cdc/request/JobBaseRecordRequest.java

fe/fe-common/src/main/java/org/apache/doris/job/cdc/DataSourceConfigKeys.java

fe/fe-core/src/main/java/org/apache/doris/tablefunction/CdcStreamTableValuedFunction.java

...nt/src/main/java/org/apache/doris/cdcclient/source/reader/postgres/PostgresSourceReader.java

...c_client/src/main/java/org/apache/doris/cdcclient/source/reader/mysql/MySqlSourceReader.java

JNSimba · 2026-01-22T03:20:09Z

run buildall

doris-robot · 2026-01-22T04:39:49Z

TPC-H: Total hot run time: 31405 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit b56d1d11f9b7fee728fc06d90ddd84506f443adf, data reload: false

------ Round 1 ----------------------------------
q1	17687	4722	4601	4601
q2	2044	330	197	197
q3	10188	1326	733	733
q4	10220	908	315	315
q5	7608	2118	1897	1897
q6	215	171	142	142
q7	903	724	599	599
q8	9279	1402	1126	1126
q9	5362	4603	4754	4603
q10	6826	1662	1276	1276
q11	525	281	266	266
q12	372	388	225	225
q13	17787	3834	3108	3108
q14	235	233	213	213
q15	615	518	526	518
q16	671	665	583	583
q17	653	793	503	503
q18	6645	6460	6819	6460
q19	1625	997	671	671
q20	436	377	241	241
q21	3109	2363	2125	2125
q22	1089	1071	1003	1003
Total cold run time: 104094 ms
Total hot run time: 31405 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5195	4900	5000	4900
q2	326	394	346	346
q3	2416	2956	2469	2469
q4	1392	1887	1461	1461
q5	4568	4452	4299	4299
q6	230	176	133	133
q7	2129	1970	1813	1813
q8	2591	2444	2469	2444
q9	7191	7151	7154	7151
q10	2777	2685	2356	2356
q11	568	474	451	451
q12	726	756	566	566
q13	3407	3856	3081	3081
q14	273	287	259	259
q15	547	509	497	497
q16	634	664	612	612
q17	1107	1282	1321	1282
q18	7268	7312	7141	7141
q19	866	807	828	807
q20	1935	1978	1862	1862
q21	4543	4244	4125	4125
q22	1085	1056	1014	1014
Total cold run time: 51774 ms
Total hot run time: 49069 ms

doris-robot · 2026-01-22T04:50:32Z

TPC-DS: Total hot run time: 173013 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit b56d1d11f9b7fee728fc06d90ddd84506f443adf, data reload: false

query5	4379	658	525	525
query6	330	221	226	221
query7	4224	471	259	259
query8	342	260	242	242
query9	8688	2876	2904	2876
query10	464	349	295	295
query11	15207	15167	14874	14874
query12	185	126	117	117
query13	1281	472	383	383
query14	5639	3477	2953	2953
query14_1	2816	2838	2804	2804
query15	200	194	171	171
query16	955	499	394	394
query17	1108	660	564	564
query18	2429	430	331	331
query19	196	186	153	153
query20	120	117	115	115
query21	212	137	116	116
query22	3799	3890	3926	3890
query23	16020	15667	15351	15351
query23_1	15441	15325	15628	15325
query24	7088	1534	1187	1187
query24_1	1177	1188	1176	1176
query25	520	437	400	400
query26	1249	280	147	147
query27	2755	439	275	275
query28	4612	2152	2140	2140
query29	835	529	444	444
query30	318	236	199	199
query31	772	645	561	561
query32	84	74	74	74
query33	512	364	326	326
query34	920	891	553	553
query35	724	754	684	684
query36	831	916	803	803
query37	136	101	85	85
query38	2750	2706	2614	2614
query39	774	765	722	722
query39_1	706	739	749	739
query40	218	142	122	122
query41	68	65	63	63
query42	98	90	95	90
query43	460	434	401	401
query44	1381	742	745	742
query45	188	187	177	177
query46	866	950	586	586
query47	1402	1389	1311	1311
query48	324	327	245	245
query49	611	423	353	353
query50	707	276	224	224
query51	3748	3755	3756	3755
query52	94	95	85	85
query53	215	226	171	171
query54	284	257	244	244
query55	84	80	75	75
query56	306	319	309	309
query57	1022	984	920	920
query58	276	271	269	269
query59	2180	2077	2041	2041
query60	336	337	318	318
query61	146	147	146	146
query62	378	358	312	312
query63	203	171	177	171
query64	5037	1157	850	850
query65	3759	3699	3796	3699
query66	1454	409	323	323
query67	15578	15666	15601	15601
query68	2349	1068	727	727
query69	396	320	282	282
query70	997	875	882	875
query71	302	303	287	287
query72	5307	3240	3358	3240
query73	623	749	321	321
query74	8714	8790	8580	8580
query75	2370	2327	1913	1913
query76	2286	1097	666	666
query77	363	393	316	316
query78	9743	9840	9199	9199
query79	1073	934	596	596
query80	619	577	436	436
query81	474	262	228	228
query82	1372	150	122	122
query83	364	266	243	243
query84	260	124	99	99
query85	824	466	406	406
query86	369	296	321	296
query87	2932	2864	2809	2809
query88	3463	2587	2571	2571
query89	311	267	253	253
query90	1879	190	178	178
query91	168	161	129	129
query92	81	77	75	75
query93	1079	1093	638	638
query94	457	340	292	292
query95	590	348	345	345
query96	652	523	233	233
query97	2391	2383	2282	2282
query98	214	227	213	213
query99	609	619	509	509
Total cold run time: 245476 ms
Total hot run time: 173013 ms

doris-robot · 2026-01-22T04:55:32Z

ClickBench: Total hot run time: 26.8 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit b56d1d11f9b7fee728fc06d90ddd84506f443adf, data reload: false

query1	0.04	0.05	0.05
query2	0.10	0.05	0.05
query3	0.26	0.09	0.08
query4	1.61	0.12	0.11
query5	0.28	0.25	0.25
query6	1.17	0.66	0.66
query7	0.03	0.03	0.03
query8	0.05	0.04	0.04
query9	0.55	0.50	0.49
query10	0.56	0.54	0.55
query11	0.14	0.10	0.10
query12	0.15	0.11	0.12
query13	0.60	0.58	0.58
query14	0.94	0.94	0.95
query15	0.78	0.79	0.79
query16	0.39	0.39	0.40
query17	1.04	1.05	1.04
query18	0.23	0.21	0.21
query19	1.81	1.79	1.85
query20	0.02	0.01	0.01
query21	15.44	0.28	0.14
query22	4.91	0.08	0.05
query23	15.74	0.29	0.11
query24	1.04	0.30	0.54
query25	0.12	0.08	0.05
query26	0.14	0.14	0.13
query27	0.07	0.05	0.08
query28	4.01	1.10	0.89
query29	12.60	3.90	3.11
query30	0.28	0.14	0.12
query31	2.82	0.64	0.40
query32	3.24	0.56	0.45
query33	2.94	3.09	3.05
query34	16.06	5.12	4.40
query35	4.49	4.54	4.45
query36	0.67	0.50	0.49
query37	0.12	0.06	0.07
query38	0.07	0.04	0.03
query39	0.04	0.03	0.03
query40	0.17	0.15	0.14
query41	0.09	0.03	0.03
query42	0.04	0.03	0.03
query43	0.06	0.03	0.03
Total cold run time: 95.91 s
Total hot run time: 26.8 s

doris-robot · 2026-01-22T06:27:30Z

BE UT Coverage Report

Increment line coverage 0.00% (0/79) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	52.46% (19116/36440)
Line Coverage	35.83% (177584/495676)
Region Coverage	32.30% (137314/425099)
Branch Coverage	33.24% (59437/178833)

JNSimba · 2026-01-22T06:36:33Z

run buildall

doris-robot · 2026-01-22T07:14:48Z

TPC-H: Total hot run time: 30904 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit e02811eaefda6a3a1631ee0683594c271015510b, data reload: false

------ Round 1 ----------------------------------
q1	17682	4836	4545	4545
q2	2077	315	199	199
q3	10199	1292	717	717
q4	10211	845	300	300
q5	7515	2086	1816	1816
q6	183	170	139	139
q7	829	710	588	588
q8	9271	1345	1101	1101
q9	4933	4566	4573	4566
q10	6785	1618	1278	1278
q11	491	289	276	276
q12	339	368	219	219
q13	18275	3825	3274	3274
q14	241	249	208	208
q15	594	529	527	527
q16	652	648	581	581
q17	654	783	507	507
q18	6733	6561	6354	6354
q19	1234	966	618	618
q20	388	342	229	229
q21	2636	1982	1916	1916
q22	1045	969	946	946
Total cold run time: 102967 ms
Total hot run time: 30904 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4704	4688	4654	4654
q2	324	379	336	336
q3	2148	2663	2304	2304
q4	1313	1779	1309	1309
q5	4101	3963	4066	3963
q6	204	171	135	135
q7	1904	1807	1759	1759
q8	2739	2487	2464	2464
q9	7392	7104	7184	7104
q10	2487	2830	2308	2308
q11	556	472	458	458
q12	708	741	631	631
q13	3616	4143	3503	3503
q14	320	327	283	283
q15	558	522	513	513
q16	639	660	661	660
q17	1136	1318	1282	1282
q18	7832	7747	7842	7747
q19	860	853	856	853
q20	2041	2228	1857	1857
q21	4812	4420	4114	4114
q22	1094	1015	984	984
Total cold run time: 51488 ms
Total hot run time: 49221 ms

doris-robot · 2026-01-22T07:25:28Z

TPC-DS: Total hot run time: 172737 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit e02811eaefda6a3a1631ee0683594c271015510b, data reload: false

query5	4427	642	490	490
query6	335	214	193	193
query7	4238	452	260	260
query8	337	240	229	229
query9	8697	2851	2844	2844
query10	476	346	298	298
query11	15121	15111	14900	14900
query12	177	121	116	116
query13	1239	459	373	373
query14	6094	3036	2780	2780
query14_1	2693	2662	2660	2660
query15	201	196	177	177
query16	968	478	450	450
query17	1113	676	584	584
query18	2439	436	339	339
query19	210	182	174	174
query20	132	129	115	115
query21	216	138	128	128
query22	3964	3849	3921	3849
query23	16043	15500	15406	15406
query23_1	15574	15489	15536	15489
query24	7152	1527	1171	1171
query24_1	1159	1162	1157	1157
query25	548	457	411	411
query26	1250	271	159	159
query27	2756	448	280	280
query28	4540	2147	2123	2123
query29	789	560	464	464
query30	309	238	205	205
query31	781	643	548	548
query32	83	77	75	75
query33	538	362	325	325
query34	887	880	528	528
query35	727	756	674	674
query36	871	914	865	865
query37	148	117	95	95
query38	2704	2707	2635	2635
query39	777	757	736	736
query39_1	718	712	710	710
query40	221	138	123	123
query41	76	71	68	68
query42	101	97	95	95
query43	443	476	415	415
query44	1328	746	753	746
query45	190	192	179	179
query46	837	939	577	577
query47	1434	1473	1335	1335
query48	311	326	238	238
query49	614	422	344	344
query50	671	264	201	201
query51	3686	3771	3715	3715
query52	94	99	84	84
query53	215	230	158	158
query54	276	264	273	264
query55	83	81	75	75
query56	296	306	297	297
query57	1014	1014	872	872
query58	268	256	268	256
query59	1873	2172	2075	2075
query60	320	325	314	314
query61	163	143	142	142
query62	375	359	319	319
query63	193	161	164	161
query64	4895	1173	853	853
query65	3834	3752	3728	3728
query66	1454	427	307	307
query67	15359	15508	15401	15401
query68	2409	1101	712	712
query69	390	308	271	271
query70	1008	945	929	929
query71	310	307	274	274
query72	5260	3055	3189	3055
query73	597	713	321	321
query74	8726	8798	8543	8543
query75	2342	2327	1885	1885
query76	2259	1043	661	661
query77	349	378	292	292
query78	9667	9843	9125	9125
query79	1082	917	619	619
query80	1298	540	436	436
query81	562	263	236	236
query82	1011	151	121	121
query83	324	274	243	243
query84	255	118	94	94
query85	872	480	408	408
query86	436	301	286	286
query87	2870	2834	2739	2739
query88	3550	2586	2558	2558
query89	300	259	238	238
query90	1961	169	170	169
query91	166	160	142	142
query92	75	70	67	67
query93	1084	1032	652	652
query94	650	333	293	293
query95	591	331	317	317
query96	650	510	230	230
query97	2323	2369	2346	2346
query98	214	199	207	199
query99	605	580	511	511
Total cold run time: 245391 ms
Total hot run time: 172737 ms

doris-robot · 2026-01-22T07:30:31Z

ClickBench: Total hot run time: 26.76 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit e02811eaefda6a3a1631ee0683594c271015510b, data reload: false

query1	0.05	0.05	0.05
query2	0.09	0.04	0.04
query3	0.26	0.09	0.08
query4	1.60	0.11	0.12
query5	0.27	0.26	0.25
query6	1.20	0.65	0.65
query7	0.03	0.03	0.03
query8	0.06	0.04	0.04
query9	0.57	0.49	0.50
query10	0.55	0.54	0.53
query11	0.16	0.10	0.10
query12	0.15	0.11	0.12
query13	0.60	0.59	0.58
query14	0.96	0.96	0.94
query15	0.78	0.78	0.78
query16	0.39	0.39	0.41
query17	1.04	1.04	1.05
query18	0.22	0.21	0.22
query19	1.96	1.89	1.90
query20	0.02	0.01	0.02
query21	15.41	0.26	0.14
query22	5.15	0.06	0.05
query23	16.00	0.29	0.11
query24	2.90	0.35	0.72
query25	0.09	0.09	0.05
query26	0.14	0.13	0.13
query27	0.08	0.05	0.04
query28	4.93	1.06	0.88
query29	12.58	3.85	3.14
query30	0.28	0.14	0.12
query31	2.81	0.63	0.40
query32	3.24	0.56	0.45
query33	2.97	2.95	3.03
query34	15.97	5.05	4.42
query35	4.41	4.39	4.46
query36	0.64	0.50	0.49
query37	0.12	0.07	0.06
query38	0.08	0.05	0.04
query39	0.05	0.03	0.03
query40	0.17	0.14	0.13
query41	0.09	0.03	0.02
query42	0.05	0.03	0.02
query43	0.04	0.03	0.04
Total cold run time: 99.16 s
Total hot run time: 26.76 s

hello-stephen · 2026-01-22T07:56:26Z

FE UT Coverage Report

Increment line coverage 3.08% (2/65) 🎉
Increment coverage report
Complete coverage report

hello-stephen · 2026-01-22T08:02:38Z

BE UT Coverage Report

Increment line coverage 0.00% (0/79) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	52.46% (19116/36440)
Line Coverage	35.83% (177582/495676)
Region Coverage	32.26% (137141/425099)
Branch Coverage	33.22% (59417/178833)

hello-stephen · 2026-01-22T09:11:50Z

BE Regression && UT Coverage Report

Increment line coverage 65.82% (52/79) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	73.12% (26118/35719)
Line Coverage	56.20% (278285/495144)
Region Coverage	54.08% (232333/429612)
Branch Coverage	55.65% (99946/179604)

hello-stephen · 2026-01-22T09:25:52Z

FE Regression Coverage Report

Increment line coverage 3.08% (2/65) 🎉
Increment coverage report
Complete coverage report

liaoxin01 · 2026-01-27T02:47:25Z

Code review

No issues found. Checked for bugs and CLAUDE.md compliance.

🤖 Generated with Claude Code

JNSimba · 2026-03-25T07:16:58Z

run buildall

JNSimba · 2026-03-25T07:28:07Z

run buildall

doris-robot · 2026-03-25T08:02:54Z

TPC-H: Total hot run time: 26636 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 7afa17eeb0f2d0735b05ec72eeaf3bf3cb6a96a4, data reload: false

------ Round 1 ----------------------------------
orders	Doris	NULL	NULL	0	0	0	NULL	0	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:42:55	NULL	utf-8	NULL	NULL	
============================================
q1	17607	4544	4339	4339
q2	q3	10649	825	547	547
q4	4678	347	249	249
q5	7581	1197	1016	1016
q6	179	174	146	146
q7	788	850	670	670
q8	9313	1501	1309	1309
q9	4978	4768	4745	4745
q10	6246	1935	1650	1650
q11	462	250	251	250
q12	700	581	464	464
q13	18038	2730	1948	1948
q14	225	236	207	207
q15	q16	759	734	665	665
q17	722	856	454	454
q18	6143	5501	5185	5185
q19	1230	1006	601	601
q20	545	513	382	382
q21	4754	1855	1519	1519
q22	473	366	290	290
Total cold run time: 96070 ms
Total hot run time: 26636 ms

----- Round 2, with runtime_filter_mode=off -----
orders	Doris	NULL	NULL	150000000	42	6422171781	NULL	22778155	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:42:55	NULL	utf-8	NULL	NULL	
============================================
q1	4743	4641	4595	4595
q2	q3	3920	4443	3808	3808
q4	892	1209	765	765
q5	4084	4426	4344	4344
q6	198	179	140	140
q7	1818	1663	1529	1529
q8	2532	2750	2629	2629
q9	8083	7527	7441	7441
q10	3788	4024	3607	3607
q11	544	432	417	417
q12	488	571	461	461
q13	2502	3233	2076	2076
q14	283	300	283	283
q15	q16	728	775	725	725
q17	1194	1411	1366	1366
q18	7124	6911	6686	6686
q19	927	883	921	883
q20	2067	2167	2093	2093
q21	4028	3526	3380	3380
q22	497	427	388	388
Total cold run time: 50440 ms
Total hot run time: 47616 ms

doris-robot · 2026-03-25T08:13:41Z

TPC-DS: Total hot run time: 169257 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 7afa17eeb0f2d0735b05ec72eeaf3bf3cb6a96a4, data reload: false

query5	4355	630	542	542
query6	343	249	227	227
query7	4224	471	270	270
query8	354	251	231	231
query9	8749	2736	2728	2728
query10	538	382	346	346
query11	6913	5151	4894	4894
query12	196	133	125	125
query13	1278	451	349	349
query14	5773	3757	3523	3523
query14_1	2868	2846	2830	2830
query15	211	196	177	177
query16	985	472	435	435
query17	891	719	624	624
query18	2461	448	355	355
query19	221	218	195	195
query20	141	127	129	127
query21	221	140	109	109
query22	13273	14174	14694	14174
query23	16889	16157	15966	15966
query23_1	16203	15609	15736	15609
query24	7111	1630	1214	1214
query24_1	1231	1251	1213	1213
query25	553	466	404	404
query26	1237	264	148	148
query27	2785	489	292	292
query28	4460	1824	1815	1815
query29	847	565	475	475
query30	303	233	193	193
query31	1015	950	871	871
query32	83	71	73	71
query33	511	337	293	293
query34	881	872	511	511
query35	646	663	596	596
query36	1107	1139	1003	1003
query37	135	97	87	87
query38	2957	2923	2881	2881
query39	873	842	830	830
query39_1	797	807	835	807
query40	234	159	136	136
query41	63	61	61	61
query42	257	254	255	254
query43	244	245	224	224
query44	
query45	202	188	183	183
query46	872	986	611	611
query47	2120	2156	2050	2050
query48	330	310	226	226
query49	641	472	378	378
query50	679	272	209	209
query51	4053	4091	4093	4091
query52	262	272	260	260
query53	299	336	289	289
query54	304	275	264	264
query55	91	87	85	85
query56	308	331	315	315
query57	1917	1684	1686	1684
query58	282	276	272	272
query59	2791	2977	2750	2750
query60	352	340	327	327
query61	164	150	159	150
query62	641	589	537	537
query63	315	278	278	278
query64	5098	1324	999	999
query65	
query66	1466	451	356	356
query67	24375	24305	24271	24271
query68	
query69	414	311	298	298
query70	968	992	961	961
query71	354	302	295	295
query72	2823	2731	2456	2456
query73	544	551	319	319
query74	9662	9605	9433	9433
query75	2885	2788	2483	2483
query76	2289	1019	667	667
query77	364	428	314	314
query78	10956	11113	10462	10462
query79	1109	787	578	578
query80	1342	625	539	539
query81	553	265	234	234
query82	970	157	122	122
query83	337	262	247	247
query84	304	120	102	102
query85	929	494	463	463
query86	442	310	322	310
query87	3163	3181	3095	3095
query88	3523	2648	2624	2624
query89	441	369	349	349
query90	2018	169	178	169
query91	175	180	135	135
query92	76	72	76	72
query93	914	852	494	494
query94	657	328	310	310
query95	592	403	317	317
query96	648	509	227	227
query97	2459	2495	2383	2383
query98	246	231	227	227
query99	973	995	916	916
Total cold run time: 250262 ms
Total hot run time: 169257 ms

JNSimba · 2026-03-25T09:08:54Z

run buildall

doris-robot · 2026-03-25T09:28:12Z

TPC-H: Total hot run time: 26241 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 9ea6fe06a5cddc20a17b3f4565082396852f0a56, data reload: false

------ Round 1 ----------------------------------
orders	Doris	NULL	NULL	0	0	0	NULL	0	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:42:55	NULL	utf-8	NULL	NULL	
============================================
q1	17633	4446	4292	4292
q2	q3	10643	789	515	515
q4	4679	360	246	246
q5	7567	1233	1032	1032
q6	174	176	146	146
q7	765	840	672	672
q8	9302	1480	1304	1304
q9	4833	4746	4669	4669
q10	6241	1900	1642	1642
q11	461	261	232	232
q12	694	585	463	463
q13	18021	2691	1912	1912
q14	222	230	214	214
q15	q16	735	737	669	669
q17	726	846	418	418
q18	6042	5435	5170	5170
q19	1150	993	601	601
q20	540	507	385	385
q21	4469	1844	1419	1419
q22	346	296	240	240
Total cold run time: 95243 ms
Total hot run time: 26241 ms

----- Round 2, with runtime_filter_mode=off -----
orders	Doris	NULL	NULL	150000000	42	6422171781	NULL	22778155	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:42:55	NULL	utf-8	NULL	NULL	
============================================
q1	4990	4707	4609	4609
q2	q3	3828	4320	3830	3830
q4	875	1202	797	797
q5	4076	4351	4328	4328
q6	183	175	143	143
q7	1761	1657	1515	1515
q8	2478	2724	2569	2569
q9	7592	7306	7388	7306
q10	3739	4023	3618	3618
q11	504	425	411	411
q12	497	608	458	458
q13	2422	2833	2018	2018
q14	282	305	299	299
q15	q16	749	795	759	759
q17	1166	1357	1388	1357
q18	7246	6791	6631	6631
q19	895	925	936	925
q20	2054	2209	2056	2056
q21	3928	3743	3452	3452
q22	432	407	381	381
Total cold run time: 49697 ms
Total hot run time: 47462 ms

doris-robot · 2026-03-25T09:38:56Z

TPC-DS: Total hot run time: 167899 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 9ea6fe06a5cddc20a17b3f4565082396852f0a56, data reload: false

query5	4335	648	512	512
query6	346	237	222	222
query7	4210	454	265	265
query8	348	247	233	233
query9	8720	2710	2710	2710
query10	529	398	365	365
query11	6956	5090	4856	4856
query12	181	139	130	130
query13	1302	458	359	359
query14	5769	3661	3456	3456
query14_1	2863	2822	2835	2822
query15	203	191	177	177
query16	974	501	453	453
query17	1130	721	613	613
query18	2439	450	355	355
query19	219	211	191	191
query20	138	127	128	127
query21	216	143	113	113
query22	13357	13221	13086	13086
query23	16236	15737	16314	15737
query23_1	16204	16212	16097	16097
query24	7395	1669	1254	1254
query24_1	1348	1275	1267	1267
query25	616	589	463	463
query26	1235	257	149	149
query27	2794	472	289	289
query28	4538	1830	1813	1813
query29	850	562	476	476
query30	299	225	190	190
query31	1005	956	860	860
query32	81	74	69	69
query33	510	328	283	283
query34	910	859	506	506
query35	660	664	606	606
query36	1074	1097	1004	1004
query37	133	92	83	83
query38	2931	2918	2839	2839
query39	855	838	808	808
query39_1	798	791	796	791
query40	234	152	135	135
query41	62	99	59	59
query42	257	256	259	256
query43	238	241	224	224
query44	
query45	202	189	182	182
query46	875	991	605	605
query47	2087	2103	2068	2068
query48	315	326	229	229
query49	639	458	386	386
query50	683	276	211	211
query51	4053	4030	3938	3938
query52	259	267	260	260
query53	288	334	289	289
query54	309	276	266	266
query55	92	83	84	83
query56	327	319	315	315
query57	1911	1788	1764	1764
query58	298	282	269	269
query59	2783	2954	2718	2718
query60	334	339	320	320
query61	159	159	158	158
query62	620	587	535	535
query63	306	281	279	279
query64	5108	1285	991	991
query65	
query66	1450	454	346	346
query67	24267	24270	24148	24148
query68	
query69	423	327	296	296
query70	868	970	920	920
query71	350	319	300	300
query72	2864	2740	2486	2486
query73	548	542	313	313
query74	9663	9531	9350	9350
query75	2899	2760	2477	2477
query76	2281	1035	679	679
query77	358	374	315	315
query78	10935	11103	10468	10468
query79	1424	760	563	563
query80	1363	607	554	554
query81	546	265	233	233
query82	1003	153	125	125
query83	333	262	241	241
query84	246	120	99	99
query85	936	505	441	441
query86	410	309	324	309
query87	3084	3113	3048	3048
query88	3578	2660	2648	2648
query89	428	371	341	341
query90	2027	173	167	167
query91	166	162	139	139
query92	72	72	73	72
query93	1003	862	504	504
query94	635	323	291	291
query95	578	352	313	313
query96	645	515	228	228
query97	2478	2486	2377	2377
query98	232	245	224	224
query99	1025	996	897	897
Total cold run time: 250279 ms
Total hot run time: 167899 ms

JNSimba · 2026-03-25T09:55:19Z

run beut

JNSimba · 2026-03-25T10:29:30Z

run cloud_p0

JNSimba · 2026-03-25T10:55:13Z

run p0

doris-robot · 2026-03-25T11:06:51Z

BE UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	52.82% (19872/37622)
Line Coverage	36.32% (185652/511210)
Region Coverage	32.55% (143743/441670)
Branch Coverage	33.77% (62962/186442)

JNSimba · 2026-03-25T11:28:57Z

run cloud_p0

JNSimba · 2026-03-25T12:01:25Z

run p0

JNSimba · 2026-03-25T12:37:39Z

run p0

hello-stephen · 2026-03-25T14:37:45Z

BE Regression && UT Coverage Report

Increment line coverage 100% (0/0) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	63.07% (23233/36839)
Line Coverage	46.50% (236962/509647)
Region Coverage	43.56% (194198/445796)
Branch Coverage	44.87% (83913/187006)

dataroaring

LGTM

github-actions · 2026-03-27T09:23:06Z

PR approved by at least one committer and no changes requested.

liaoxin01

LGTM

### What problem does this PR solve? apache#58898 apache#59461 These two features support consuming data from MySQL and PostgreSQL. Based on this, this PR introduces cdc_stream tvf, which allows consuming data from MySQL and PostgreSQL in TVF format. Example ``` select * from cdc_stream( "type" = "mysql", "jdbc_url" = "jdbc:mysql://127.0.0.1:3306", "driver_url" = "mysql-connector-j-8.0.31.jar", "driver_class" = "com.mysql.cj.jdbc.Driver", "user" = "root", "password" = "123456", "database" = "test_cdc_db", "table" = "user_info", "offset" = '{"file":"binlog.000003","pos":"496"}' ) ```

### What problem does this PR solve? #58898 #59461 These two features support consuming data from MySQL and PostgreSQL. Based on this, this PR introduces cdc_stream tvf, which allows consuming data from MySQL and PostgreSQL in TVF format. Example ``` select * from cdc_stream( "type" = "mysql", "jdbc_url" = "jdbc:mysql://127.0.0.1:3306", "driver_url" = "mysql-connector-j-8.0.31.jar", "driver_class" = "com.mysql.cj.jdbc.Driver", "user" = "root", "password" = "123456", "database" = "test_cdc_db", "table" = "user_info", "offset" = '{"file":"binlog.000003","pos":"496"}' ) ```

JNSimba requested review from dataroaring and gavinchou as code owners January 21, 2026 09:52

JNSimba requested a review from Copilot January 21, 2026 09:53

Copilot started reviewing on behalf of JNSimba January 21, 2026 09:54 View session

Copilot AI reviewed Jan 21, 2026

View reviewed changes

JNSimba requested a review from Copilot January 22, 2026 03:00

Copilot started reviewing on behalf of JNSimba January 22, 2026 03:01 View session

Copilot AI reviewed Jan 22, 2026

View reviewed changes

github-actions bot removed the approved Indicates a PR has been approved by one committer. label Mar 25, 2026

fix

7afa17e

fix

9ea6fe0

dataroaring approved these changes Mar 27, 2026

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Mar 27, 2026

liaoxin01 approved these changes Mar 27, 2026

View reviewed changes

JNSimba merged commit c26652a into apache:master Mar 27, 2026
27 of 30 checks passed

github-actions bot added the dev/4.1.x-conflict label Mar 27, 2026

JNSimba added dev/4.1.x and removed dev/4.1.x dev/4.1.x-conflict labels Mar 28, 2026

github-actions bot mentioned this pull request Mar 28, 2026

branch-4.1: [Feature](tvf) Support cdc stream tvf for mysql and pg #60116 #61840

Open

Conversation

JNSimba commented Jan 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

Uh oh!

hello-stephen commented Jan 21, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

JNSimba commented Jan 21, 2026

Uh oh!

doris-robot commented Jan 21, 2026

Uh oh!

doris-robot commented Jan 21, 2026

Uh oh!

doris-robot commented Jan 21, 2026

Uh oh!

hello-stephen commented Jan 21, 2026

BE UT Coverage Report

Uh oh!

hello-stephen commented Jan 21, 2026

BE Regression && UT Coverage Report

Uh oh!

hello-stephen commented Jan 21, 2026

FE Regression Coverage Report

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

JNSimba commented Jan 22, 2026

Uh oh!

doris-robot commented Jan 22, 2026

Uh oh!

doris-robot commented Jan 22, 2026

Uh oh!

doris-robot commented Jan 22, 2026

Uh oh!

doris-robot commented Jan 22, 2026

BE UT Coverage Report

Uh oh!

JNSimba commented Jan 22, 2026

Uh oh!

doris-robot commented Jan 22, 2026

Uh oh!

doris-robot commented Jan 22, 2026

Uh oh!

doris-robot commented Jan 22, 2026

Uh oh!

hello-stephen commented Jan 22, 2026

FE UT Coverage Report

Uh oh!

hello-stephen commented Jan 22, 2026

BE UT Coverage Report

Uh oh!

hello-stephen commented Jan 22, 2026

BE Regression && UT Coverage Report

Uh oh!

hello-stephen commented Jan 22, 2026

JNSimba commented Jan 21, 2026 •

edited

Loading