Skip to content

[fix](Nereids) remove useless substr of string like type cast#29997

Merged
yiguolei merged 1 commit intoapache:masterfrom
morrySnow:fix_sink_varchar
Jan 16, 2024
Merged

[fix](Nereids) remove useless substr of string like type cast#29997
yiguolei merged 1 commit intoapache:masterfrom
morrySnow:fix_sink_varchar

Conversation

@morrySnow
Copy link
Contributor

@morrySnow morrySnow commented Jan 15, 2024

Proposed changes

Issue Number: close #xxx

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@morrySnow
Copy link
Contributor Author

morrySnow commented Jan 15, 2024

run buildall

@doris-robot
Copy link

doris-robot commented Jan 15, 2024

TPC-H: Total hot run time: 39323 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 4e835b4f653a9dd95b0e87ab7fbe667f55e123f7, data reload: false

------ Round 1 ----------------------------------
q1	18036	6409	5677	5677
q2	2038	154	132	132
q3	10873	1160	1118	1118
q4	10523	772	784	772
q5	7850	3144	3062	3062
q6	204	123	125	123
q7	909	497	487	487
q8	9249	1985	2009	1985
q9	7374	6390	6426	6390
q10	8180	3127	3069	3069
q11	424	207	214	207
q12	363	199	198	198
q13	18119	3406	3445	3406
q14	253	223	215	215
q15	553	513	512	512
q16	447	376	376	376
q17	940	504	487	487
q18	7685	6976	6723	6723
q19	2012	1330	1305	1305
q20	584	296	308	296
q21	2835	2450	2455	2450
q22	383	334	333	333
Total cold run time: 109834 ms
Total hot run time: 39323 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5902	5553	5889	5553
q2	342	217	226	217
q3	3332	3140	3052	3052
q4	2208	2058	2014	2014
q5	6133	5979	5794	5794
q6	202	123	126	123
q7	2426	1983	2000	1983
q8	3640	3751	3776	3751
q9	9836	26848	9487	9487
q10	8944	3796	3799	3796
q11	652	468	446	446
q12	1144	655	642	642
q13	17106	3272	3233	3233
q14	378	259	274	259
q15	576	505	509	505
q16	540	495	494	494
q17	9013	2065	1979	1979
q18	9545	9753	12953	9753
q19	13478	1565	1636	1565
q20	4664	1979	1918	1918
q21	15042	5530	5595	5530
q22	1261	621	589	589
Total cold run time: 116364 ms
Total hot run time: 62683 ms

@doris-robot
Copy link

doris-robot commented Jan 15, 2024

TPC-DS: Total hot run time: 178250 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 4e835b4f653a9dd95b0e87ab7fbe667f55e123f7, data reload: false

query1	942	338	332	332
query2	6715	1859	1751	1751
query3	6739	207	201	201
query4	32605	22335	22100	22100
query5	6933	584	540	540
query6	260	180	178	178
query7	4611	280	277	277
query8	235	192	207	192
query9	9231	2735	2749	2735
query10	564	253	239	239
query11	16699	15296	15263	15263
query12	125	76	67	67
query13	1715	395	379	379
query14	11009	7289	7290	7289
query15	221	181	183	181
query16	4641	250	239	239
query17	1017	495	482	482
query18	1812	278	261	261
query19	181	151	147	147
query20	80	74	76	74
query21	193	147	124	124
query22	5079	4827	4742	4742
query23	31675	31006	30972	30972
query24	7400	2820	2822	2820
query25	410	327	333	327
query26	727	147	148	147
query27	2365	276	277	276
query28	3936	1922	1904	1904
query29	980	671	663	663
query30	207	137	134	134
query31	936	746	767	746
query32	62	62	59	59
query33	433	247	235	235
query34	832	468	461	461
query35	867	826	798	798
query36	1423	1285	1319	1285
query37	102	67	71	67
query38	3412	3315	3262	3262
query39	1317	1271	1271	1271
query40	217	98	91	91
query41	39	35	35	35
query42	111	98	94	94
query43	549	518	494	494
query44	1033	722	722	722
query45	201	189	183	183
query46	1043	639	633	633
query47	1727	1526	1554	1526
query48	412	332	308	308
query49	624	321	315	315
query50	689	307	320	307
query51	5329	5212	5264	5212
query52	105	90	99	90
query53	355	294	276	276
query54	738	459	455	455
query55	91	96	87	87
query56	206	190	185	185
query57	987	925	966	925
query58	205	184	187	184
query59	2890	2573	2630	2573
query60	240	206	221	206
query61	81	81	79	79
query62	488	412	386	386
query63	314	290	299	290
query64	2612	1655	1633	1633
query65	3355	3280	3260	3260
query66	1057	323	343	323
query67	15695	15338	15220	15220
query68	11826	526	490	490
query69	648	337	342	337
query70	1689	1587	1526	1526
query71	518	238	254	238
query72	5489	2840	2852	2840
query73	2561	321	318	318
query74	7037	6413	6487	6413
query75	5091	2366	2293	2293
query76	6185	884	1055	884
query77	707	267	281	267
query78	9144	8615	8581	8581
query79	1028	488	496	488
query80	580	340	339	339
query81	476	204	200	200
query82	205	94	92	92
query83	150	124	122	122
query84	278	81	75	75
query85	1057	342	316	316
query86	427	364	376	364
query87	3632	3385	3352	3352
query88	3013	2223	2248	2223
query89	455	379	370	370
query90	2265	197	208	197
query91	152	133	129	129
query92	61	57	52	52
query93	1174	400	418	400
query94	1499	174	176	174
query95	521	478	471	471
query96	612	315	319	315
query97	4310	4215	4172	4172
query98	212	184	186	184
query99	1070	716	722	716
Total cold run time: 284156 ms
Total hot run time: 178250 ms

@doris-robot
Copy link

doris-robot commented Jan 15, 2024

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 45.54 seconds
stream load tsv: 559 seconds loaded 74807831229 Bytes, about 127 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 21.8 seconds inserted 10000000 Rows, about 458K ops/s
storage size: 17183987863 Bytes

@doris-robot
Copy link

doris-robot commented Jan 15, 2024

ClickBench: Total hot run time: 29.93 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 4e835b4f653a9dd95b0e87ab7fbe667f55e123f7, data reload: false

query1	0.06	0.05	0.05
query2	0.07	0.03	0.02
query3	0.26	0.12	0.11
query4	1.76	0.12	0.11
query5	0.53	0.52	0.53
query6	1.31	0.64	0.64
query7	0.02	0.01	0.01
query8	0.04	0.02	0.02
query9	0.56	0.52	0.50
query10	0.57	0.56	0.54
query11	0.12	0.09	0.09
query12	0.12	0.09	0.09
query13	0.61	0.60	0.61
query14	0.78	0.79	0.80
query15	0.79	0.77	0.77
query16	0.38	0.38	0.39
query17	1.00	1.03	1.01
query18	0.26	0.26	0.24
query19	1.87	1.81	1.81
query20	0.02	0.01	0.01
query21	15.40	0.55	0.54
query22	2.84	2.76	1.30
query23	17.06	0.80	0.82
query24	2.58	0.55	0.46
query25	0.31	0.26	0.05
query26	0.35	0.15	0.14
query27	0.07	0.05	0.06
query28	12.30	0.80	0.78
query29	12.60	3.19	3.19
query30	0.52	0.50	0.47
query31	2.78	0.34	0.36
query32	3.35	0.47	0.49
query33	3.23	3.22	3.21
query34	15.70	4.30	4.26
query35	4.26	4.22	4.23
query36	1.13	1.08	1.08
query37	0.06	0.05	0.05
query38	0.03	0.02	0.02
query39	0.02	0.01	0.02
query40	0.16	0.13	0.13
query41	0.07	0.01	0.02
query42	0.02	0.02	0.02
query43	0.03	0.02	0.02
Total cold run time: 106 s
Total hot run time: 29.93 s

@doris-robot
Copy link

doris-robot commented Jan 15, 2024

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 4e835b4f653a9dd95b0e87ab7fbe667f55e123f7 with default session variables
Stream load json:         20 seconds loaded 2358488459 Bytes, about 112 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      33 seconds loaded 861443392 Bytes, about 24 MB/s
Insert into select:       14.7 seconds inserted 10000000 Rows, about 680K ops/s

@morrySnow
Copy link
Contributor Author

morrySnow commented Jan 16, 2024

run p0

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jan 16, 2024
@github-actions
Copy link
Contributor

github-actions bot commented Jan 16, 2024

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

github-actions bot commented Jan 16, 2024

PR approved by anyone and no changes requested.

@yiguolei yiguolei merged commit c3e4f0c into apache:master Jan 16, 2024
@morrySnow morrySnow deleted the fix_sink_varchar branch January 16, 2024 12:17
yiguolei pushed a commit that referenced this pull request Jan 16, 2024
@morrySnow @Doris-Extras
[fix](Nereids) remove useless substr of string like type cast (#29997)
e693249
hello-stephen pushed a commit to hello-stephen/doris that referenced this pull request Jan 19, 2024
@morrySnow
[fix](Nereids) remove useless substr of string like type cast (apache…
71724c8 
…#29997)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@starocean999 starocean999 starocean999 approved these changes

+1 more reviewer

@keanji-x keanji-x keanji-x approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/3.0.0-merged reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@morrySnow @doris-robot @keanji-x @starocean999 @yiguolei @dataroaring