Skip to content

Conversation

@zddr
Copy link
Contributor

@zddr zddr commented Jan 29, 2024

Proposed changes

LinkedList is not thread safe and there may be issues with concurrent queries and writes. Change it to ConcurrentLinkedQueue

Caused by: java.util.ConcurrentModificationException
	at java.util.LinkedList$ListItr.checkForComodification(LinkedList.java:966) ~[?:1.8.0_131]
	at java.util.LinkedList$ListItr.next(LinkedList.java:888) ~[?:1.8.0_131]
	at org.apache.doris.tablefunction.MetadataGenerator.taskMetadataResult(MetadataGenerator.java:694) ~[doris-fe.jar:1.2-SNAPSHOT]
	at org.apache.doris.tablefunction.MetadataGenerator.getMetadataTable(MetadataGenerator.java:119) ~[doris-fe.jar:1.2-SNAPSHOT]
	at org.apache.doris.service.FrontendServiceImpl.fetchSchemaTableData(FrontendServiceImpl.java:2195) ~[doris-fe.jar:1.2-SNAPSHOT]
	... 13 more

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@zddr
Copy link
Contributor Author

zddr commented Jan 29, 2024

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 37371 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 225063eaecb056ae7978fa0abe5ea1da31df387a, data reload: false

------ Round 1 ----------------------------------
q1	17644	4934	5036	4934
q2	2045	144	134	134
q3	10671	958	908	908
q4	4660	764	696	696
q5	7661	2833	2837	2833
q6	185	119	119	119
q7	1141	733	719	719
q8	9320	1988	2017	1988
q9	7143	6341	6286	6286
q10	8084	2414	2426	2414
q11	412	198	206	198
q12	771	267	290	267
q13	17991	3299	3303	3299
q14	280	241	240	240
q15	544	484	483	483
q16	458	415	395	395
q17	919	543	482	482
q18	6837	6156	5995	5995
q19	4112	1351	1287	1287
q20	582	322	318	318
q21	6611	3138	3084	3084
q22	800	297	292	292
Total cold run time: 108871 ms
Total hot run time: 37371 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5242	4918	4919	4918
q2	348	239	229	229
q3	3049	2798	2608	2608
q4	1765	1559	1608	1559
q5	5219	5274	5203	5203
q6	207	117	116	116
q7	2126	1762	1759	1759
q8	3121	3190	3217	3190
q9	8825	8569	19274	8569
q10	5508	3469	3415	3415
q11	538	471	471	471
q12	748	592	596	592
q13	16959	3089	3074	3074
q14	283	252	257	252
q15	534	483	491	483
q16	496	480	478	478
q17	2089	1610	1706	1610
q18	8758	8375	9559	8375
q19	23794	1583	1519	1519
q20	4618	1915	1882	1882
q21	14578	4768	4825	4768
q22	990	488	457	457
Total cold run time: 109795 ms
Total hot run time: 55527 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 172929 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 225063eaecb056ae7978fa0abe5ea1da31df387a, data reload: false

query1	933	336	326	326
query2	6576	1922	1917	1917
query3	6701	196	195	195
query4	34077	22215	22121	22121
query5	4449	356	408	356
query6	238	160	151	151
query7	4606	274	262	262
query8	242	176	173	173
query9	8570	2267	2253	2253
query10	412	206	199	199
query11	17946	15365	15364	15364
query12	126	66	64	64
query13	1661	381	365	365
query14	9143	6432	6444	6432
query15	215	176	177	176
query16	5691	258	247	247
query17	953	457	468	457
query18	1804	261	251	251
query19	176	128	142	128
query20	67	68	66	66
query21	199	134	134	134
query22	4883	4709	4771	4709
query23	30973	30422	30501	30422
query24	7322	2763	2804	2763
query25	413	307	305	305
query26	811	137	141	137
query27	2816	277	286	277
query28	3911	1839	1831	1831
query29	913	606	621	606
query30	208	135	140	135
query31	899	709	708	708
query32	70	49	49	49
query33	413	215	202	202
query34	805	448	471	448
query35	843	775	749	749
query36	1280	1098	1060	1060
query37	89	57	59	57
query38	3289	3141	3111	3111
query39	1304	1261	1241	1241
query40	193	89	79	79
query41	37	36	34	34
query42	86	83	81	81
query43	517	452	491	452
query44	1052	698	704	698
query45	190	176	171	171
query46	1051	652	658	652
query47	1545	1517	1547	1517
query48	382	298	302	298
query49	620	278	282	278
query50	685	314	306	306
query51	5270	5189	5253	5189
query52	91	78	82	78
query53	341	254	260	254
query54	224	187	192	187
query55	74	74	76	74
query56	180	171	163	163
query57	1001	911	895	895
query58	180	157	148	148
query59	2629	2567	2536	2536
query60	194	172	175	172
query61	84	85	84	84
query62	452	380	354	354
query63	275	252	270	252
query64	2884	1767	1746	1746
query65	3293	3253	3191	3191
query66	1176	316	314	314
query67	14241	13837	14449	13837
query68	3909	499	510	499
query69	446	290	290	290
query70	1506	1552	1503	1503
query71	265	205	229	205
query72	3797	2828	2871	2828
query73	684	315	307	307
query74	6647	6267	6242	6242
query75	2751	2259	2312	2259
query76	2646	1028	1036	1028
query77	377	227	227	227
query78	9012	8903	8542	8542
query79	2255	493	491	491
query80	1376	321	309	309
query81	503	194	203	194
query82	670	84	78	78
query83	240	114	114	114
query84	278	69	76	69
query85	1650	330	338	330
query86	496	390	428	390
query87	3396	3277	3253	3253
query88	3655	2179	2174	2174
query89	412	388	361	361
query90	2015	186	185	185
query91	152	113	116	113
query92	57	43	44	43
query93	2348	461	449	449
query94	1282	155	158	155
query95	496	464	456	456
query96	611	320	316	316
query97	4273	4118	4144	4118
query98	209	194	183	183
query99	1133	752	731	731
Total cold run time: 264521 ms
Total hot run time: 172929 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.31 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 225063eaecb056ae7978fa0abe5ea1da31df387a, data reload: false

query1	0.04	0.03	0.04
query2	0.06	0.02	0.02
query3	0.23	0.06	0.06
query4	1.69	0.09	0.09
query5	0.54	0.51	0.51
query6	1.21	0.64	0.63
query7	0.02	0.01	0.02
query8	0.03	0.02	0.02
query9	0.55	0.52	0.51
query10	0.56	0.55	0.57
query11	0.11	0.09	0.09
query12	0.12	0.09	0.09
query13	0.60	0.60	0.61
query14	0.79	0.79	0.80
query15	0.81	0.77	0.78
query16	0.38	0.38	0.39
query17	1.02	1.04	0.97
query18	0.21	0.27	0.25
query19	1.88	1.79	1.76
query20	0.02	0.01	0.01
query21	15.40	0.62	0.59
query22	2.72	2.20	1.45
query23	17.30	0.79	0.76
query24	2.36	1.65	0.71
query25	0.41	0.18	0.09
query26	0.49	0.15	0.15
query27	0.06	0.05	0.04
query28	11.42	0.86	0.83
query29	12.51	3.21	3.08
query30	0.64	0.56	0.55
query31	2.78	0.35	0.36
query32	3.35	0.48	0.49
query33	3.20	3.20	3.24
query34	15.81	4.31	4.26
query35	4.33	4.29	4.36
query36	1.10	1.07	1.07
query37	0.07	0.05	0.05
query38	0.04	0.02	0.03
query39	0.02	0.02	0.02
query40	0.16	0.13	0.13
query41	0.07	0.02	0.02
query42	0.02	0.02	0.01
query43	0.03	0.02	0.02
Total cold run time: 105.16 s
Total hot run time: 30.31 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 225063eaecb056ae7978fa0abe5ea1da31df387a with default session variables
Stream load json:         18 seconds loaded 2358488459 Bytes, about 124 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select:       14.2 seconds inserted 10000000 Rows, about 704K ops/s

@github-actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added approved Indicates a PR has been approved by one committer. reviewed labels Jan 30, 2024
@github-actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@yiguolei yiguolei merged commit 6d23a11 into apache:master Jan 30, 2024
yiguolei pushed a commit that referenced this pull request Jan 30, 2024
LinkedList is not thread safe and there may be issues with concurrent queries and writes. Change it to ConcurrentLinkedQueue

Caused by: java.util.ConcurrentModificationException
	at java.util.LinkedList$ListItr.checkForComodification(LinkedList.java:966) ~[?:1.8.0_131]
	at java.util.LinkedList$ListItr.next(LinkedList.java:888) ~[?:1.8.0_131]
	at org.apache.doris.tablefunction.MetadataGenerator.taskMetadataResult(MetadataGenerator.java:694) ~[doris-fe.jar:1.2-SNAPSHOT]
	at org.apache.doris.tablefunction.MetadataGenerator.getMetadataTable(MetadataGenerator.java:119) ~[doris-fe.jar:1.2-SNAPSHOT]
	at org.apache.doris.service.FrontendServiceImpl.fetchSchemaTableData(FrontendServiceImpl.java:2195) ~[doris-fe.jar:1.2-SNAPSHOT]
	... 13 more
@zddr zddr deleted the mv_task_current branch March 28, 2024 02:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by one committer. reviewed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants