Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[fix](load) acquire latest token instead of oldest token in TokenManager (#34424) #34535

Merged
merged 1 commit into from
May 8, 2024

Conversation

kaijchen
Copy link
Contributor

@kaijchen kaijchen commented May 8, 2024

Proposed changes

cherry-pick #34424

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

…ger (apache#34424)

* [fix](load) acquire latest token instead of oldest token

* fixup
@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@kaijchen
Copy link
Contributor Author

kaijchen commented May 8, 2024

run buildall

@github-actions github-actions bot added the area/load Issues or PRs related to all kinds of load label May 8, 2024
@doris-robot
Copy link

TPC-H: Total hot run time: 49367 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit cadca047ae6b0412d914b0ff2b5e08043d11de14, data reload: false

------ Round 1 ----------------------------------
q1	17868	4453	4365	4365
q2	2032	152	148	148
q3	10400	1945	1920	1920
q4	10382	1223	1345	1223
q5	8419	3832	3879	3832
q6	229	123	124	123
q7	1995	1602	1619	1602
q8	9330	2689	2726	2689
q9	10464	10291	10226	10226
q10	8674	3489	3496	3489
q11	428	237	243	237
q12	472	293	293	293
q13	18358	3936	4095	3936
q14	345	326	324	324
q15	511	452	454	452
q16	666	565	564	564
q17	1106	955	955	955
q18	7223	6802	6924	6802
q19	1717	1552	1479	1479
q20	520	292	306	292
q21	4416	4077	4024	4024
q22	496	392	400	392
Total cold run time: 116051 ms
Total hot run time: 49367 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4310	4256	4283	4256
q2	320	222	218	218
q3	4151	4203	4138	4138
q4	2750	2737	2714	2714
q5	7158	7082	7051	7051
q6	235	120	117	117
q7	3213	2892	2873	2873
q8	4350	4571	4451	4451
q9	16886	16712	16772	16712
q10	4207	4307	4212	4212
q11	748	673	674	673
q12	1012	846	858	846
q13	6796	3712	3739	3712
q14	451	435	418	418
q15	506	451	459	451
q16	730	706	670	670
q17	3750	3825	3891	3825
q18	8764	8793	8787	8787
q19	1707	1680	1625	1625
q20	2420	2145	2131	2131
q21	8494	8512	8538	8512
q22	1006	969	991	969
Total cold run time: 83964 ms
Total hot run time: 79361 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 204502 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit cadca047ae6b0412d914b0ff2b5e08043d11de14, data reload: false

query1	916	389	377	377
query2	6546	2883	2645	2645
query3	6913	205	204	204
query4	21643	17920	17887	17887
query5	19726	6529	6464	6464
query6	275	212	209	209
query7	4149	292	300	292
query8	268	262	233	233
query9	3113	2679	2598	2598
query10	406	293	317	293
query11	11393	10785	10897	10785
query12	119	73	69	69
query13	5578	707	678	678
query14	18239	13805	13842	13805
query15	385	231	234	231
query16	6482	274	266	266
query17	1757	1427	903	903
query18	2305	421	410	410
query19	212	151	148	148
query20	82	80	72	72
query21	195	92	88	88
query22	5385	5124	5085	5085
query23	33303	32677	32760	32677
query24	7047	6557	6500	6500
query25	526	413	410	410
query26	627	164	156	156
query27	2031	289	290	289
query28	6204	2346	2307	2307
query29	2912	2804	2858	2804
query30	243	164	164	164
query31	916	781	724	724
query32	68	59	58	58
query33	392	257	238	238
query34	835	463	470	463
query35	1131	898	904	898
query36	1235	1279	1312	1279
query37	92	60	58	58
query38	3124	2927	2919	2919
query39	1365	1320	1318	1318
query40	247	94	98	94
query41	43	35	35	35
query42	90	89	81	81
query43	899	728	708	708
query44	1128	714	703	703
query45	242	225	228	225
query46	1231	956	953	953
query47	1777	1809	1727	1727
query48	998	712	697	697
query49	621	357	371	357
query50	867	578	598	578
query51	4837	4644	4681	4644
query52	93	75	85	75
query53	439	325	316	316
query54	2654	2450	2451	2450
query55	89	80	77	77
query56	205	195	215	195
query57	1180	1076	1325	1076
query58	232	196	190	190
query59	3923	4202	3742	3742
query60	216	188	188	188
query61	91	84	84	84
query62	803	461	446	446
query63	463	345	338	338
query64	2667	1646	1450	1450
query65	3629	3534	3566	3534
query66	804	368	380	368
query67	15886	15418	15908	15418
query68	8508	663	649	649
query69	561	346	330	330
query70	1705	1452	1404	1404
query71	410	298	301	298
query72	6477	3421	3468	3421
query73	742	311	332	311
query74	6377	5840	5882	5840
query75	4685	3728	3714	3714
query76	4723	1139	1202	1139
query77	567	247	254	247
query78	12615	11748	14235	11748
query79	8159	636	642	636
query80	1159	400	386	386
query81	496	234	226	226
query82	804	104	101	101
query83	170	137	134	134
query84	259	70	70	70
query85	917	302	296	296
query86	339	291	297	291
query87	3237	2994	3004	2994
query88	4801	2376	2350	2350
query89	350	310	290	290
query90	2080	208	205	205
query91	173	133	148	133
query92	56	56	52	52
query93	2971	523	559	523
query94	765	198	201	198
query95	1110	1041	1049	1041
query96	642	334	329	329
query97	6487	6341	6454	6341
query98	199	168	171	168
query99	2596	913	884	884
Total cold run time: 309632 ms
Total hot run time: 204502 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.94 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit cadca047ae6b0412d914b0ff2b5e08043d11de14, data reload: false

query1	0.03	0.02	0.02
query2	0.07	0.02	0.02
query3	0.25	0.04	0.05
query4	1.80	0.11	0.11
query5	0.53	0.54	0.53
query6	1.33	0.60	0.62
query7	0.01	0.01	0.01
query8	0.03	0.02	0.02
query9	0.51	0.49	0.48
query10	0.54	0.54	0.53
query11	0.11	0.09	0.08
query12	0.11	0.08	0.08
query13	0.62	0.61	0.62
query14	0.77	0.80	0.78
query15	0.78	0.75	0.78
query16	0.40	0.36	0.36
query17	1.02	1.00	1.00
query18	0.23	0.26	0.26
query19	1.93	1.83	1.84
query20	0.01	0.01	0.01
query21	15.47	0.55	0.55
query22	2.12	2.29	1.48
query23	17.35	1.13	1.00
query24	5.81	1.19	1.45
query25	0.33	0.09	0.06
query26	0.75	0.17	0.15
query27	0.05	0.03	0.04
query28	6.99	0.74	0.71
query29	12.60	2.40	2.31
query30	0.65	0.54	0.55
query31	2.81	0.39	0.37
query32	3.40	0.50	0.49
query33	3.08	3.09	3.06
query34	15.26	4.81	4.78
query35	4.85	4.86	4.85
query36	1.04	1.02	1.01
query37	0.06	0.04	0.05
query38	0.04	0.02	0.02
query39	0.02	0.01	0.01
query40	0.16	0.14	0.13
query41	0.07	0.01	0.01
query42	0.02	0.01	0.01
query43	0.02	0.02	0.02
Total cold run time: 104.03 s
Total hot run time: 30.94 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit cadca047ae6b0412d914b0ff2b5e08043d11de14 with default session variables
Stream load json:         20 seconds loaded 2358488459 Bytes, about 112 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select:       21.4 seconds inserted 10000000 Rows, about 467K ops/s

@xiaokang xiaokang merged commit 9676cb0 into apache:branch-2.0 May 8, 2024
24 of 26 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/load Issues or PRs related to all kinds of load
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants