Skip to content

[be](serde) support more arrow::Type in read_column_from_arrow - #66489

Merged
yiguolei merged 6 commits into
apache:masterfrom
zhangstar333:master_serde
Aug 7, 2026
Merged

[be](serde) support more arrow::Type in read_column_from_arrow#66489
yiguolei merged 6 commits into
apache:masterfrom
zhangstar333:master_serde

Conversation

@zhangstar333

Copy link
Copy Markdown
Contributor

What problem does this PR solve?

Problem Summary:

  • Support Arrow LargeList and FixedSizeList for Doris ARRAY.
  • Support Binary, LargeBinary, and FixedSizeBinary for VARBINARY.
  • Support additional Arrow numeric and temporal types, including Float16, unsigned integers, Date64, TIME, and TIMESTAMPTZ.
  • Improve Arrow offset, range, type, and temporal-unit validation.

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@hello-stephen

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@zhangstar333

Copy link
Copy Markdown
Contributor Author

run buildall

@zhangstar333

Copy link
Copy Markdown
Contributor Author

run buildall

@zhangstar333

Copy link
Copy Markdown
Contributor Author

run buildall

@zhangstar333

Copy link
Copy Markdown
Contributor Author

run buildall

@hello-stephen

Copy link
Copy Markdown
Contributor

BE UT Coverage Report

Increment line coverage 87.29% (316/362) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 59.78% (26082/43629)
Line Coverage 44.14% (264821/599939)
Region Coverage 39.89% (210896/528656)
Branch Coverage 41.30% (96574/233824)

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-H: Total hot run time: 28801 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit da820936d2fefb778dd6442da83cfeb357afcc72, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17772	3960	3940	3940
q2	1978	310	200	200
q3	10321	1413	786	786
q4	4681	468	343	343
q5	7481	888	557	557
q6	178	167	135	135
q7	743	817	603	603
q8	9623	1528	1567	1528
q9	5956	4057	4027	4027
q10	6792	1628	1376	1376
q11	502	362	325	325
q12	741	576	448	448
q13	18122	3281	2741	2741
q14	261	260	246	246
q15	q16	735	732	658	658
q17	971	1001	985	985
q18	6619	5570	5511	5511
q19	1185	1254	1058	1058
q20	820	691	583	583
q21	5778	2826	2443	2443
q22	457	376	308	308
Total cold run time: 101716 ms
Total hot run time: 28801 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	4726	4600	4561	4561
q2	310	346	307	307
q3	4881	5172	4659	4659
q4	2251	2265	1472	1472
q5	4666	4680	4664	4664
q6	240	182	134	134
q7	1837	1770	1578	1578
q8	2404	2015	1910	1910
q9	6892	6791	6815	6791
q10	4231	4213	3808	3808
q11	512	370	334	334
q12	710	715	496	496
q13	3001	3272	2755	2755
q14	279	289	247	247
q15	q16	664	675	631	631
q17	1236	1227	1197	1197
q18	12118	10983	11815	10983
q19	1099	1048	1052	1048
q20	2197	2202	1906	1906
q21	5254	4522	4505	4505
q22	529	455	392	392
Total cold run time: 60037 ms
Total hot run time: 54378 ms

@hello-stephen

Copy link
Copy Markdown
Contributor

BE Regression && UT Coverage Report

Increment line coverage 87.29% (316/362) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 75.91% (32296/42544)
Line Coverage 60.78% (362181/595886)
Region Coverage 57.38% (304336/530393)
Branch Coverage 58.84% (137601/233848)

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-DS: Total hot run time: 165967 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit da820936d2fefb778dd6442da83cfeb357afcc72, data reload: false

query5	4328	604	456	456
query6	445	210	201	201
query7	4868	561	341	341
query8	324	168	149	149
query9	8776	4049	4025	4025
query10	514	361	302	302
query11	5811	2165	2001	2001
query12	159	100	95	95
query13	1251	611	421	421
query14	6063	4261	4018	4018
query14_1	3816	3801	3778	3778
query15	194	187	172	172
query16	987	452	468	452
query17	889	681	542	542
query18	2448	464	326	326
query19	204	191	154	154
query20	99	98	103	98
query21	227	157	135	135
query22	13090	12999	12761	12761
query23	15569	14926	14511	14511
query23_1	14689	14519	14585	14519
query24	7560	1704	1204	1204
query24_1	1234	1195	1235	1195
query25	519	422	353	353
query26	1337	367	208	208
query27	2595	567	384	384
query28	4578	2065	2026	2026
query29	1094	621	492	492
query30	346	274	223	223
query31	1180	1127	1055	1055
query32	109	62	62	62
query33	578	298	234	234
query34	1172	1206	644	644
query35	730	749	623	623
query36	789	761	698	698
query37	154	106	89	89
query38	1823	1789	1684	1684
query39	842	827	798	798
query39_1	787	793	780	780
query40	254	164	141	141
query41	68	62	61	61
query42	97	99	89	89
query43	314	318	273	273
query44	1425	780	763	763
query45	185	187	172	172
query46	1000	1198	723	723
query47	1503	1514	1408	1408
query48	398	387	305	305
query49	591	408	306	306
query50	1038	439	344	344
query51	10428	10612	10221	10221
query52	88	89	81	81
query53	255	282	205	205
query54	272	231	221	221
query55	79	72	67	67
query56	309	299	289	289
query57	994	977	901	901
query58	284	260	296	260
query59	1549	1555	1412	1412
query60	313	271	260	260
query61	145	154	147	147
query62	391	318	266	266
query63	240	190	197	190
query64	2859	1119	956	956
query65	3866	3829	3859	3829
query66	1845	493	372	372
query67	28165	28089	27958	27958
query68	3142	1608	1066	1066
query69	419	308	279	279
query70	887	774	769	769
query71	391	346	314	314
query72	3157	2768	2324	2324
query73	814	826	452	452
query74	4618	4522	4277	4277
query75	2378	2345	1988	1988
query76	2300	1167	777	777
query77	361	362	266	266
query78	11180	11269	10703	10703
query79	1385	1245	786	786
query80	1277	548	454	454
query81	570	337	290	290
query82	656	170	142	142
query83	385	331	300	300
query84	327	157	132	132
query85	1016	607	529	529
query86	407	241	220	220
query87	1975	1971	1836	1836
query88	3776	2826	2818	2818
query89	394	313	282	282
query90	1889	199	196	196
query91	204	197	166	166
query92	64	58	58	58
query93	1788	1567	970	970
query94	700	364	309	309
query95	790	498	475	475
query96	1070	808	351	351
query97	2478	2467	2363	2363
query98	191	187	187	187
query99	740	730	607	607
Total cold run time: 253061 ms
Total hot run time: 165967 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
ClickBench: Total hot run time: 23.66 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit da820936d2fefb778dd6442da83cfeb357afcc72, data reload: false

query1	0.01	0.01	0.01
query2	0.09	0.05	0.05
query3	0.24	0.14	0.13
query4	1.60	0.14	0.14
query5	0.24	0.22	0.23
query6	1.16	0.81	0.81
query7	0.04	0.01	0.01
query8	0.06	0.04	0.04
query9	0.36	0.32	0.32
query10	0.55	0.57	0.54
query11	0.18	0.13	0.14
query12	0.17	0.14	0.14
query13	0.46	0.46	0.46
query14	1.01	0.97	0.97
query15	0.61	0.59	0.58
query16	0.32	0.34	0.31
query17	1.05	1.11	1.07
query18	0.21	0.20	0.19
query19	2.06	1.90	1.93
query20	0.01	0.02	0.01
query21	15.45	0.21	0.13
query22	5.01	0.06	0.05
query23	16.14	0.31	0.12
query24	2.93	0.42	0.31
query25	0.12	0.05	0.04
query26	0.73	0.20	0.14
query27	0.04	0.03	0.04
query28	3.52	0.76	0.35
query29	12.48	3.99	3.18
query30	0.27	0.15	0.15
query31	2.79	0.55	0.32
query32	3.22	0.58	0.49
query33	3.18	3.20	3.12
query34	15.58	3.94	3.24
query35	3.23	3.22	3.20
query36	0.54	0.45	0.42
query37	0.08	0.07	0.06
query38	0.04	0.04	0.03
query39	0.03	0.02	0.02
query40	0.18	0.15	0.14
query41	0.09	0.03	0.04
query42	0.03	0.04	0.03
query43	0.05	0.03	0.03
Total cold run time: 96.16 s
Total hot run time: 23.66 s

@yiguolei
yiguolei merged commit 193c0ca into apache:master Aug 7, 2026
30 of 31 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants