Skip to content

[Improvement](local shuffle) Reduce locking scope in local exchanger …#46293

Merged
Gabriel39 merged 1 commit intoapache:branch-3.0from
Gabriel39:pick_0102_3_1
Jan 2, 2025
Merged

[Improvement](local shuffle) Reduce locking scope in local exchanger …#46293
Gabriel39 merged 1 commit intoapache:branch-3.0from
Gabriel39:pick_0102_3_1

Conversation

@Gabriel39
Copy link
Contributor

…(#46251)

Reduce lock scope from global level to data queue level.

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

…pache#46251)

Reduce lock scope from global level to data queue level.
@hello-stephen
Copy link
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@Gabriel39
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 40892 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit aa73d2c6928ea04ade9e9ea5e058e935c2db23f4, data reload: false

------ Round 1 ----------------------------------
q1	17580	7523	7303	7303
q2	2070	182	179	179
q3	10590	1067	1132	1067
q4	10568	733	758	733
q5	7753	2859	2819	2819
q6	234	145	147	145
q7	960	621	606	606
q8	9384	1942	2001	1942
q9	6584	6430	6453	6430
q10	7025	2266	2324	2266
q11	469	276	262	262
q12	415	219	207	207
q13	17802	3030	3001	3001
q14	239	203	207	203
q15	560	522	499	499
q16	686	592	621	592
q17	982	548	552	548
q18	7291	6669	6692	6669
q19	1388	1102	1110	1102
q20	462	202	196	196
q21	4061	3268	3125	3125
q22	1126	1022	998	998
Total cold run time: 108229 ms
Total hot run time: 40892 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7332	7355	7315	7315
q2	326	232	232	232
q3	2945	2933	2948	2933
q4	2089	1793	1758	1758
q5	5935	5777	5728	5728
q6	221	145	150	145
q7	2247	1819	1841	1819
q8	3343	3655	3510	3510
q9	8790	8902	8892	8892
q10	3576	3557	3550	3550
q11	598	517	490	490
q12	787	590	599	590
q13	9973	3182	3142	3142
q14	304	275	264	264
q15	569	536	544	536
q16	722	689	672	672
q17	1844	1659	1617	1617
q18	8188	7667	7742	7667
q19	1667	1578	1536	1536
q20	2146	1883	1876	1876
q21	5358	5444	5484	5444
q22	1200	1005	1027	1005
Total cold run time: 70160 ms
Total hot run time: 60721 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 197268 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit aa73d2c6928ea04ade9e9ea5e058e935c2db23f4, data reload: false

query1	1267	961	903	903
query2	6229	2091	2059	2059
query3	10912	4227	4460	4227
query4	66080	28740	23441	23441
query5	4992	476	459	459
query6	413	177	176	176
query7	5528	307	311	307
query8	316	231	222	222
query9	8972	2701	2674	2674
query10	447	270	253	253
query11	17677	15178	16063	15178
query12	155	103	104	103
query13	1479	454	451	451
query14	9840	7597	7302	7302
query15	200	186	180	180
query16	7134	526	514	514
query17	1091	594	596	594
query18	1798	328	324	324
query19	213	160	157	157
query20	123	112	115	112
query21	68	47	53	47
query22	4783	4560	4470	4470
query23	34581	34430	34167	34167
query24	6127	2915	2942	2915
query25	522	440	438	438
query26	651	170	171	170
query27	1750	304	311	304
query28	4233	2504	2459	2459
query29	739	438	424	424
query30	249	171	161	161
query31	1018	806	816	806
query32	70	52	56	52
query33	433	284	296	284
query34	910	531	501	501
query35	872	714	723	714
query36	1085	942	974	942
query37	125	71	74	71
query38	4081	4091	4014	4014
query39	1526	1520	1442	1442
query40	144	84	79	79
query41	49	48	47	47
query42	112	100	101	100
query43	517	493	490	490
query44	1170	836	826	826
query45	187	172	168	168
query46	1126	723	728	723
query47	2032	1939	1909	1909
query48	478	374	392	374
query49	712	396	406	396
query50	811	422	422	422
query51	7338	7323	7203	7203
query52	105	85	89	85
query53	256	180	179	179
query54	554	443	443	443
query55	81	73	75	73
query56	266	244	249	244
query57	1214	1115	1098	1098
query58	214	204	221	204
query59	3132	2879	2960	2879
query60	284	262	251	251
query61	106	111	114	111
query62	778	671	674	671
query63	209	190	185	185
query64	1441	661	657	657
query65	3226	3186	3207	3186
query66	700	303	315	303
query67	15732	15634	15370	15370
query68	4133	577	572	572
query69	424	258	265	258
query70	1181	1109	1147	1109
query71	323	256	245	245
query72	6489	4056	4024	4024
query73	743	341	345	341
query74	10167	9009	9030	9009
query75	3385	2668	2663	2663
query76	1838	1021	1048	1021
query77	512	275	265	265
query78	10558	9677	9613	9613
query79	1832	598	595	595
query80	1129	439	433	433
query81	548	242	244	242
query82	209	124	117	117
query83	166	139	148	139
query84	291	78	79	78
query85	1004	308	283	283
query86	404	293	300	293
query87	4540	4240	4275	4240
query88	4265	2361	2350	2350
query89	406	287	288	287
query90	2037	187	188	187
query91	185	150	149	149
query92	63	53	48	48
query93	2575	536	542	536
query94	935	280	319	280
query95	352	258	251	251
query96	615	275	270	270
query97	3367	3181	3191	3181
query98	211	206	205	205
query99	1621	1291	1296	1291
Total cold run time: 318171 ms
Total hot run time: 197268 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 32.88 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit aa73d2c6928ea04ade9e9ea5e058e935c2db23f4, data reload: false

query1	0.03	0.04	0.04
query2	0.07	0.03	0.02
query3	0.23	0.07	0.07
query4	1.62	0.10	0.10
query5	0.52	0.50	0.51
query6	1.15	0.72	0.73
query7	0.02	0.02	0.02
query8	0.04	0.03	0.04
query9	0.56	0.49	0.51
query10	0.54	0.56	0.56
query11	0.14	0.10	0.10
query12	0.16	0.12	0.12
query13	0.60	0.59	0.59
query14	3.01	2.89	2.91
query15	0.89	0.83	0.82
query16	0.38	0.38	0.39
query17	1.03	1.06	1.06
query18	0.23	0.22	0.22
query19	1.82	1.86	2.05
query20	0.01	0.01	0.02
query21	15.36	0.60	0.59
query22	2.75	4.02	1.47
query23	17.10	0.98	0.77
query24	3.19	1.17	1.12
query25	0.18	0.18	0.24
query26	0.48	0.15	0.13
query27	0.04	0.04	0.05
query28	10.40	1.12	1.07
query29	12.55	3.20	3.19
query30	0.25	0.06	0.06
query31	2.84	0.41	0.39
query32	3.46	0.46	0.46
query33	3.02	3.00	3.06
query34	16.93	4.50	4.51
query35	4.56	4.55	4.53
query36	0.66	0.48	0.48
query37	0.10	0.06	0.06
query38	0.05	0.03	0.04
query39	0.03	0.02	0.02
query40	0.17	0.14	0.12
query41	0.08	0.02	0.02
query42	0.03	0.03	0.02
query43	0.04	0.03	0.03
Total cold run time: 107.32 s
Total hot run time: 32.88 s

@Gabriel39 Gabriel39 merged commit 70da785 into apache:branch-3.0 Jan 2, 2025
11 of 12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants