Skip to content

[fix](array/map) Fix semi-structure data column#40363

Closed
Gabriel39 wants to merge 1 commit intoapache:masterfrom
Gabriel39:fix_0904
Closed

[fix](array/map) Fix semi-structure data column#40363
Gabriel39 wants to merge 1 commit intoapache:masterfrom
Gabriel39:fix_0904

Conversation

@Gabriel39
Copy link
Contributor

Proposed changes

Function convert_to_full_column_if_const will return a shared pointer with the origin column and add 1 to use count of this column if it is not a Array/Map column.

For Array/Map columns, convert_to_full_column_if_const will return a new pointer which shares the internal nested columns with the origin columns.

During execution, a column which is exclusive means it is not shared by another pointer so we can use this column as we want otherwise we should copy the data into another exclusive column.

Because convert_to_full_column_if_const will return a new pointer for Array/Map columns, we can not just use count == 1 to decide if the column is exclusive.

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@Gabriel39
Copy link
Contributor Author

run buildall

@amorynan
Copy link
Contributor

amorynan commented Sep 4, 2024

please add cases!

@github-actions
Copy link
Contributor

github-actions bot commented Sep 4, 2024

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TPC-H: Total hot run time: 38215 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 99150632e24ee70221d0bbdf81ec5a7bb51b0b5b, data reload: false

------ Round 1 ----------------------------------
q1	17623	4547	4293	4293
q2	2028	179	178	178
q3	11800	968	1161	968
q4	10509	757	667	667
q5	7762	2833	2820	2820
q6	231	142	139	139
q7	962	618	604	604
q8	9307	2079	2063	2063
q9	7179	6517	6557	6517
q10	6994	2214	2158	2158
q11	444	246	241	241
q12	400	226	221	221
q13	17757	3084	3066	3066
q14	284	232	231	231
q15	531	490	498	490
q16	587	517	504	504
q17	988	793	750	750
q18	7355	6832	6938	6832
q19	1374	1152	1056	1056
q20	684	328	325	325
q21	4301	3076	3196	3076
q22	1148	1018	1016	1016
Total cold run time: 110248 ms
Total hot run time: 38215 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4350	4310	4272	4272
q2	389	279	263	263
q3	2869	2616	2701	2616
q4	1949	1664	1617	1617
q5	5628	5643	5694	5643
q6	236	138	143	138
q7	2245	1812	1808	1808
q8	3317	3431	3441	3431
q9	8828	8827	8750	8750
q10	3575	3393	3356	3356
q11	620	504	505	504
q12	842	666	626	626
q13	14740	3250	3235	3235
q14	317	296	298	296
q15	540	513	501	501
q16	637	588	592	588
q17	1847	1535	1556	1535
q18	8311	7823	8031	7823
q19	1745	1634	1589	1589
q20	2158	1957	1944	1944
q21	5744	5548	5355	5355
q22	1143	1070	1048	1048
Total cold run time: 72030 ms
Total hot run time: 56938 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 192406 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 99150632e24ee70221d0bbdf81ec5a7bb51b0b5b, data reload: false

query1	1278	884	866	866
query2	6368	1948	1951	1948
query3	10600	4128	3946	3946
query4	59293	25832	23136	23136
query5	5286	510	488	488
query6	414	173	174	173
query7	5761	305	297	297
query8	278	212	204	204
query9	8978	2490	2502	2490
query10	470	262	271	262
query11	18065	15016	15182	15016
query12	154	108	99	99
query13	1579	411	401	401
query14	10930	7081	7320	7081
query15	213	184	186	184
query16	7438	468	482	468
query17	1092	553	572	553
query18	2107	302	301	301
query19	271	147	146	146
query20	129	111	107	107
query21	207	109	103	103
query22	4658	4598	4526	4526
query23	34405	33362	33211	33211
query24	6005	2861	2838	2838
query25	536	405	399	399
query26	659	157	154	154
query27	1736	298	282	282
query28	3548	2086	2064	2064
query29	703	426	424	424
query30	235	158	158	158
query31	947	773	736	736
query32	88	55	57	55
query33	422	287	289	287
query34	868	480	481	480
query35	849	704	729	704
query36	1046	930	905	905
query37	150	89	97	89
query38	4152	4010	3908	3908
query39	1488	1381	1386	1381
query40	201	116	118	116
query41	49	49	47	47
query42	119	95	95	95
query43	517	488	467	467
query44	1082	737	751	737
query45	198	166	166	166
query46	1093	742	720	720
query47	1906	1819	1865	1819
query48	369	303	300	300
query49	780	440	455	440
query50	824	417	414	414
query51	7049	6949	6935	6935
query52	101	85	89	85
query53	248	182	178	178
query54	565	457	461	457
query55	82	76	80	76
query56	293	272	269	269
query57	1216	1076	1073	1073
query58	233	295	235	235
query59	3014	2775	2746	2746
query60	300	282	282	282
query61	128	123	123	123
query62	765	667	657	657
query63	212	187	190	187
query64	2498	755	733	733
query65	3199	3150	3134	3134
query66	801	340	346	340
query67	15394	15027	15095	15027
query68	4416	573	560	560
query69	422	288	287	287
query70	1123	1027	1121	1027
query71	373	271	279	271
query72	6615	4022	3950	3950
query73	758	327	364	327
query74	9225	8835	9056	8835
query75	3387	2719	2676	2676
query76	1740	989	991	989
query77	521	303	307	303
query78	9902	9098	9052	9052
query79	1301	523	517	517
query80	938	523	514	514
query81	563	240	238	238
query82	254	146	143	143
query83	172	148	146	146
query84	263	72	75	72
query85	883	330	277	277
query86	396	290	294	290
query87	4500	4370	4396	4370
query88	3285	2335	2352	2335
query89	391	293	280	280
query90	1912	191	188	188
query91	123	102	101	101
query92	65	51	51	51
query93	1825	539	543	539
query94	804	297	299	297
query95	337	253	254	253
query96	600	268	271	268
query97	3193	3069	3125	3069
query98	224	202	208	202
query99	1808	1251	1280	1251
Total cold run time: 308961 ms
Total hot run time: 192406 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 32.3 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 99150632e24ee70221d0bbdf81ec5a7bb51b0b5b, data reload: false

query1	0.05	0.05	0.04
query2	0.09	0.04	0.04
query3	0.23	0.05	0.05
query4	1.68	0.07	0.07
query5	0.52	0.51	0.50
query6	1.13	0.74	0.73
query7	0.01	0.01	0.01
query8	0.05	0.05	0.05
query9	0.55	0.48	0.49
query10	0.55	0.55	0.53
query11	0.16	0.12	0.12
query12	0.15	0.12	0.12
query13	0.60	0.60	0.59
query14	2.04	2.06	2.12
query15	0.90	0.85	0.83
query16	0.39	0.39	0.38
query17	1.07	1.01	1.08
query18	0.21	0.21	0.21
query19	1.97	1.80	1.75
query20	0.01	0.01	0.01
query21	15.38	0.66	0.65
query22	4.05	7.26	2.02
query23	18.31	1.42	1.34
query24	2.15	0.21	0.21
query25	0.16	0.08	0.08
query26	0.27	0.18	0.18
query27	0.08	0.08	0.07
query28	13.22	1.02	1.00
query29	12.62	3.40	3.38
query30	0.24	0.06	0.06
query31	2.88	0.40	0.40
query32	3.24	0.48	0.48
query33	2.99	3.03	3.04
query34	16.96	4.38	4.35
query35	4.45	4.47	4.42
query36	0.66	0.50	0.46
query37	0.18	0.15	0.15
query38	0.16	0.15	0.15
query39	0.05	0.04	0.04
query40	0.15	0.12	0.13
query41	0.09	0.05	0.05
query42	0.05	0.04	0.04
query43	0.04	0.04	0.04
Total cold run time: 110.74 s
Total hot run time: 32.3 s

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants