[fix](group commit) Fix group commit load complex data type in nereids planner #36553

mymeiyi · 2024-06-19T10:03:28Z

Proposed changes

when use group commit in nereids planner to insert struct data type, may get too many filtered rows, because the converted string value is not correct.
Fix case: group commit can be executed in observer fe, the case try to get profile from observer to ensure the group commit is not forwarded, but no profiler can get, so the case fail.

doris-robot · 2024-06-19T10:03:33Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

mymeiyi · 2024-06-19T10:08:00Z

run buildall

doris-robot · 2024-06-19T10:48:28Z

TPC-H: Total hot run time: 39527 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 8ca882bf180355a7b3764bd54ac19d9a2b72f58c, data reload: false

------ Round 1 ----------------------------------
q1	17639	4316	4253	4253
q2	2022	186	195	186
q3	10473	1166	1123	1123
q4	10187	808	738	738
q5	7561	2641	2586	2586
q6	215	140	134	134
q7	943	620	601	601
q8	9222	2051	2073	2051
q9	8827	6511	6439	6439
q10	8967	3733	3767	3733
q11	454	236	240	236
q12	501	233	229	229
q13	17773	2972	2975	2972
q14	270	211	211	211
q15	527	475	476	475
q16	523	380	388	380
q17	954	669	725	669
q18	7975	7396	7284	7284
q19	3746	1598	1527	1527
q20	662	323	316	316
q21	4926	3046	3217	3046
q22	383	338	341	338
Total cold run time: 114750 ms
Total hot run time: 39527 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4403	4206	4235	4206
q2	371	254	267	254
q3	2958	2745	2980	2745
q4	1974	1693	1688	1688
q5	5528	5554	5507	5507
q6	222	130	144	130
q7	2227	1800	1921	1800
q8	3294	3422	3413	3413
q9	8711	8674	8761	8674
q10	4112	3886	3817	3817
q11	582	488	493	488
q12	799	624	648	624
q13	17282	3155	3158	3155
q14	322	264	292	264
q15	531	478	483	478
q16	514	445	437	437
q17	1815	1536	1496	1496
q18	8032	7941	8594	7941
q19	1839	1672	1712	1672
q20	3087	1862	1847	1847
q21	5096	5049	4668	4668
q22	638	546	585	546
Total cold run time: 74337 ms
Total hot run time: 55850 ms

doris-robot · 2024-06-19T10:59:14Z

TPC-DS: Total hot run time: 169964 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 8ca882bf180355a7b3764bd54ac19d9a2b72f58c, data reload: false

query1	937	379	376	376
query2	6450	2393	2351	2351
query3	6632	202	205	202
query4	19083	17323	17297	17297
query5	3600	489	471	471
query6	237	154	155	154
query7	4579	297	284	284
query8	319	292	283	283
query9	8482	2376	2373	2373
query10	588	285	273	273
query11	10538	10008	10118	10008
query12	120	91	81	81
query13	1648	372	363	363
query14	9386	6249	7680	6249
query15	234	190	182	182
query16	7712	271	268	268
query17	1783	542	517	517
query18	1806	274	266	266
query19	199	155	152	152
query20	92	85	85	85
query21	207	138	126	126
query22	4397	3986	3909	3909
query23	33473	33718	33576	33576
query24	11175	2918	2860	2860
query25	612	400	365	365
query26	1077	150	155	150
query27	2340	325	332	325
query28	6442	2104	2105	2104
query29	891	623	637	623
query30	225	159	155	155
query31	985	752	755	752
query32	92	53	55	53
query33	747	290	303	290
query34	1037	486	490	486
query35	745	631	642	631
query36	1137	966	979	966
query37	145	74	71	71
query38	2945	2918	2803	2803
query39	909	856	814	814
query40	221	133	135	133
query41	60	52	61	52
query42	114	118	111	111
query43	566	518	533	518
query44	1236	720	728	720
query45	191	160	164	160
query46	1085	718	734	718
query47	1814	1744	1749	1744
query48	372	294	309	294
query49	849	413	422	413
query50	767	392	396	392
query51	6717	6704	6696	6696
query52	99	97	90	90
query53	366	290	298	290
query54	883	444	454	444
query55	77	76	77	76
query56	292	274	295	274
query57	1130	1050	1077	1050
query58	268	254	252	252
query59	3709	3215	3127	3127
query60	294	266	277	266
query61	90	92	87	87
query62	590	458	432	432
query63	313	286	296	286
query64	8631	2279	1752	1752
query65	3154	3112	3129	3112
query66	750	325	323	323
query67	15572	14995	15012	14995
query68	4656	561	537	537
query69	624	451	385	385
query70	1143	1093	1097	1093
query71	463	266	277	266
query72	7024	2801	2585	2585
query73	740	320	321	320
query74	5843	5488	5449	5449
query75	3531	2697	2701	2697
query76	3186	945	889	889
query77	635	288	304	288
query78	10245	9914	9811	9811
query79	1926	510	508	508
query80	1522	474	454	454
query81	566	214	219	214
query82	731	102	103	102
query83	201	169	164	164
query84	267	86	83	83
query85	1225	283	271	271
query86	403	306	298	298
query87	3256	3132	3088	3088
query88	3652	2353	2327	2327
query89	485	381	377	377
query90	1765	194	189	189
query91	124	102	98	98
query92	57	51	60	51
query93	2688	517	504	504
query94	1002	192	183	183
query95	408	309	315	309
query96	582	265	268	265
query97	3255	3039	3021	3021
query98	207	203	195	195
query99	1295	853	842	842
Total cold run time: 267923 ms
Total hot run time: 169964 ms

doris-robot · 2024-06-19T11:04:44Z

ClickBench: Total hot run time: 30.87 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 8ca882bf180355a7b3764bd54ac19d9a2b72f58c, data reload: false

query1	0.04	0.03	0.03
query2	0.07	0.04	0.04
query3	0.23	0.04	0.05
query4	1.67	0.07	0.07
query5	0.50	0.48	0.52
query6	1.13	0.72	0.72
query7	0.02	0.01	0.01
query8	0.05	0.04	0.04
query9	0.55	0.50	0.50
query10	0.53	0.55	0.54
query11	0.16	0.12	0.11
query12	0.15	0.12	0.13
query13	0.59	0.59	0.61
query14	0.78	0.77	0.78
query15	0.83	0.81	0.82
query16	0.36	0.36	0.36
query17	0.96	1.05	0.96
query18	0.22	0.24	0.24
query19	1.80	1.69	1.84
query20	0.02	0.01	0.01
query21	15.42	0.66	0.65
query22	4.13	6.90	2.16
query23	18.26	1.49	1.36
query24	2.19	0.22	0.21
query25	0.17	0.08	0.08
query26	0.27	0.18	0.17
query27	0.10	0.07	0.08
query28	13.20	1.02	1.00
query29	12.61	3.34	3.29
query30	0.26	0.06	0.06
query31	2.87	0.39	0.39
query32	3.27	0.47	0.47
query33	2.91	2.95	2.89
query34	17.23	4.46	4.44
query35	4.46	4.46	4.58
query36	0.65	0.49	0.45
query37	0.18	0.15	0.16
query38	0.15	0.15	0.14
query39	0.05	0.03	0.04
query40	0.18	0.13	0.14
query41	0.10	0.05	0.05
query42	0.05	0.05	0.05
query43	0.05	0.04	0.04
Total cold run time: 109.42 s
Total hot run time: 30.87 s

dataroaring

LGTM

github-actions · 2024-06-20T12:09:01Z

PR approved by at least one committer and no changes requested.

github-actions · 2024-06-20T12:09:04Z

PR approved by anyone and no changes requested.

fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/insert/InsertUtils.java

mymeiyi · 2024-06-21T04:13:05Z

run buildall

doris-robot · 2024-06-21T04:36:02Z

TPC-H: Total hot run time: 39943 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit b2b9ee9fb06a430d724a1db2191b555d6b7b3346, data reload: false

------ Round 1 ----------------------------------
q1	17679	4487	4285	4285
q2	2027	194	193	193
q3	10445	1207	1103	1103
q4	10191	876	863	863
q5	7576	2724	2668	2668
q6	229	139	140	139
q7	966	623	613	613
q8	9572	2128	2058	2058
q9	8911	6495	6451	6451
q10	8870	3732	3714	3714
q11	443	248	234	234
q12	423	231	227	227
q13	17776	2972	2965	2965
q14	272	219	221	219
q15	528	495	483	483
q16	539	395	376	376
q17	963	767	656	656
q18	8063	7543	7450	7450
q19	3369	1457	1462	1457
q20	642	318	342	318
q21	4858	3138	3933	3138
q22	390	348	333	333
Total cold run time: 114732 ms
Total hot run time: 39943 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4320	4226	4406	4226
q2	387	260	268	260
q3	2980	2757	2731	2731
q4	1863	1639	1576	1576
q5	5251	5285	5265	5265
q6	216	129	129	129
q7	2108	1756	1717	1717
q8	3196	3327	3307	3307
q9	8365	8330	8379	8330
q10	3864	3705	3683	3683
q11	584	478	507	478
q12	764	636	612	612
q13	17593	2970	2972	2970
q14	285	268	262	262
q15	512	476	469	469
q16	470	405	422	405
q17	1768	1478	1452	1452
q18	7695	7634	7390	7390
q19	2063	1698	1546	1546
q20	1998	1788	1772	1772
q21	4874	4799	4738	4738
q22	593	533	520	520
Total cold run time: 71749 ms
Total hot run time: 53838 ms

doris-robot · 2024-06-21T04:47:04Z

TPC-DS: Total hot run time: 173069 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit b2b9ee9fb06a430d724a1db2191b555d6b7b3346, data reload: false

query1	921	391	370	370
query2	6456	2315	2335	2315
query3	6652	210	217	210
query4	18987	17376	17233	17233
query5	4149	496	483	483
query6	256	165	165	165
query7	4593	298	300	298
query8	309	293	291	291
query9	8795	2452	2423	2423
query10	617	313	291	291
query11	10554	9984	10041	9984
query12	145	87	85	85
query13	1638	407	356	356
query14	9945	7688	6681	6681
query15	218	185	188	185
query16	7810	272	258	258
query17	1919	536	541	536
query18	1901	266	275	266
query19	194	154	159	154
query20	91	84	82	82
query21	214	137	129	129
query22	4222	4221	4230	4221
query23	33956	33176	33159	33159
query24	12141	2834	2795	2795
query25	661	358	361	358
query26	1793	150	157	150
query27	2998	323	322	322
query28	7596	2076	2049	2049
query29	1145	622	611	611
query30	284	159	148	148
query31	945	748	745	745
query32	94	55	57	55
query33	793	287	320	287
query34	943	478	473	473
query35	750	622	610	610
query36	1101	941	942	941
query37	305	73	75	73
query38	2891	2807	2728	2728
query39	851	773	779	773
query40	283	123	123	123
query41	55	51	54	51
query42	127	99	104	99
query43	570	525	518	518
query44	1242	739	740	739
query45	189	176	160	160
query46	1085	752	735	735
query47	1824	1792	1803	1792
query48	365	302	287	287
query49	1183	407	411	407
query50	763	387	393	387
query51	7020	6821	6803	6803
query52	110	90	93	90
query53	367	287	287	287
query54	981	475	444	444
query55	74	74	74	74
query56	289	259	269	259
query57	1136	1073	1052	1052
query58	258	261	253	253
query59	3358	3285	2989	2989
query60	298	278	286	278
query61	91	92	104	92
query62	651	447	449	447
query63	324	290	287	287
query64	9926	2266	1739	1739
query65	3207	3157	3093	3093
query66	1381	340	328	328
query67	15454	15076	14979	14979
query68	4582	535	538	535
query69	462	307	308	307
query70	1198	1140	1128	1128
query71	409	279	275	275
query72	7168	5626	5559	5559
query73	744	326	325	325
query74	5986	5466	5474	5466
query75	3464	2657	2652	2652
query76	2887	1023	938	938
query77	473	305	304	304
query78	10384	9862	9831	9831
query79	2331	519	524	519
query80	1093	471	471	471
query81	579	221	222	221
query82	694	108	102	102
query83	236	168	175	168
query84	247	82	91	82
query85	2001	279	278	278
query86	496	319	337	319
query87	3272	3139	3088	3088
query88	4219	2350	2359	2350
query89	475	389	388	388
query90	1859	195	193	193
query91	131	100	101	100
query92	63	50	50	50
query93	2367	518	497	497
query94	1264	187	188	187
query95	413	323	326	323
query96	597	270	270	270
query97	3269	3045	3018	3018
query98	229	204	194	194
query99	1277	846	854	846
Total cold run time: 277313 ms
Total hot run time: 173069 ms

doris-robot · 2024-06-21T04:52:34Z

ClickBench: Total hot run time: 31.13 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit b2b9ee9fb06a430d724a1db2191b555d6b7b3346, data reload: false

query1	0.04	0.03	0.03
query2	0.08	0.04	0.04
query3	0.23	0.06	0.05
query4	1.65	0.10	0.10
query5	0.52	0.49	0.49
query6	1.13	0.73	0.73
query7	0.02	0.02	0.02
query8	0.06	0.04	0.05
query9	0.56	0.48	0.50
query10	0.55	0.53	0.55
query11	0.15	0.11	0.12
query12	0.15	0.12	0.12
query13	0.59	0.58	0.59
query14	0.79	0.78	0.78
query15	0.86	0.82	0.82
query16	0.35	0.37	0.38
query17	1.04	1.03	1.00
query18	0.21	0.25	0.24
query19	1.84	1.68	1.80
query20	0.02	0.01	0.01
query21	15.42	0.65	0.64
query22	3.90	6.40	2.34
query23	18.33	1.35	1.24
query24	2.12	0.24	0.21
query25	0.16	0.09	0.08
query26	0.27	0.18	0.17
query27	0.08	0.08	0.08
query28	13.25	1.02	1.00
query29	12.64	3.36	3.32
query30	0.26	0.07	0.06
query31	2.84	0.39	0.38
query32	3.28	0.47	0.47
query33	2.88	2.94	2.90
query34	17.14	4.47	4.43
query35	4.51	4.48	4.51
query36	0.65	0.49	0.50
query37	0.19	0.16	0.16
query38	0.16	0.15	0.15
query39	0.05	0.04	0.04
query40	0.17	0.14	0.14
query41	0.10	0.05	0.05
query42	0.05	0.04	0.05
query43	0.04	0.05	0.04
Total cold run time: 109.33 s
Total hot run time: 31.13 s

morrySnow · 2024-06-21T06:28:30Z

fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/insert/InsertUtils.java

+                        "\"")) {
+                    row.addColBuilder().setValue(String.format("\"%s\"", stringValue));
+                } else {
+                    row.addColBuilder().setValue(String.format("%s", stringValue));


could process string literal abc"dbf correctly?

mymeiyi changed the title ~~[fix](group commit) Fix group commit load struct type in nereids planner~~ [fix](group commit) Fix group commit load complex data type in nereids planner Jun 19, 2024

dataroaring previously approved these changes Jun 20, 2024

View reviewed changes

github-actions bot added approved Indicates a PR has been approved by one committer. reviewed labels Jun 20, 2024

morrySnow reviewed Jun 20, 2024

View reviewed changes

fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/insert/InsertUtils.java Show resolved Hide resolved

mymeiyi added 3 commits June 21, 2024 12:01

fix data type

877d9e8

fix case

880414d

fix

b2b9ee9

mymeiyi dismissed dataroaring’s stale review via b2b9ee9 June 21, 2024 04:11

mymeiyi force-pushed the gc-nere branch from 8ca882b to b2b9ee9 Compare June 21, 2024 04:11

github-actions bot removed the approved Indicates a PR has been approved by one committer. label Jun 21, 2024

morrySnow reviewed Jun 21, 2024

View reviewed changes

mymeiyi closed this Jun 28, 2024

[fix](group commit) Fix group commit load complex data type in nereids planner #36553

[fix](group commit) Fix group commit load complex data type in nereids planner #36553

Uh oh!

Conversation

mymeiyi commented Jun 19, 2024

Proposed changes

Uh oh!

doris-robot commented Jun 19, 2024

Uh oh!

mymeiyi commented Jun 19, 2024

Uh oh!

doris-robot commented Jun 19, 2024

Uh oh!

doris-robot commented Jun 19, 2024

Uh oh!

doris-robot commented Jun 19, 2024

Uh oh!

dataroaring left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Jun 20, 2024

Uh oh!

github-actions bot commented Jun 20, 2024

Uh oh!

Uh oh!

mymeiyi commented Jun 21, 2024

Uh oh!

doris-robot commented Jun 21, 2024

Uh oh!

doris-robot commented Jun 21, 2024

Uh oh!

doris-robot commented Jun 21, 2024

Uh oh!

morrySnow Jun 21, 2024

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants