Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[chore](fe) Add some important information in fe log for debug (#34352) #37627

Merged
merged 1 commit into from
Jul 11, 2024
Merged

[chore](fe) Add some important information in fe log for debug (#34352) #37627

merged 1 commit into from
Jul 11, 2024

Conversation

SWJTU-ZhangLei
Copy link
Contributor

@SWJTU-ZhangLei SWJTU-ZhangLei commented Jul 10, 2024 •
edited by dataroaring
Loading

  • Add git commit info in fe.log/fe.out when fe start
  • Add metadata recovery mode log in fe.log/fe.out
  • Add bdb journal info in fe.log when size over 1M

pick #34352

Proposed changes

Issue Number: close #xxx

@SWJTU-ZhangLei
[chore](fe) Add some important information in fe log for debug (apach…
5617833 
…e#34352)

* Add git commit info in fe.log/fe.out when fe start
* Add `metadata recovery mode` log in fe.log/fe.out
* Add bdb journal info in fe.log when size over `1M`
@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@SWJTU-ZhangLei
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 49829 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 56178333ecaf2af43fd0537d607856447290958e, data reload: false

------ Round 1 ----------------------------------
q1	17735	4409	4343	4343
q2	2063	158	147	147
q3	10328	1888	1940	1888
q4	10349	1261	1340	1261
q5	8416	3984	3879	3879
q6	237	125	123	123
q7	2033	1587	1577	1577
q8	9544	2725	2704	2704
q9	13253	10282	10141	10141
q10	8651	3501	3497	3497
q11	416	250	252	250
q12	470	306	310	306
q13	18410	3980	4021	3980
q14	363	323	328	323
q15	514	456	465	456
q16	668	576	580	576
q17	1140	947	973	947
q18	7242	6922	6966	6922
q19	1802	1730	1676	1676
q20	509	318	301	301
q21	4461	4113	4092	4092
q22	512	440	453	440
Total cold run time: 119116 ms
Total hot run time: 49829 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4364	4257	4327	4257
q2	320	229	226	226
q3	4179	4152	4104	4104
q4	2753	2739	2744	2739
q5	7204	7168	7113	7113
q6	240	121	121	121
q7	3223	2834	2811	2811
q8	4389	4504	4533	4504
q9	16959	16836	16837	16836
q10	4258	4274	4235	4235
q11	720	688	683	683
q12	1021	874	878	874
q13	7061	3786	3823	3786
q14	484	463	440	440
q15	509	466	465	465
q16	778	720	738	720
q17	3904	3884	3934	3884
q18	9261	9007	8917	8917
q19	1784	1732	1718	1718
q20	2428	2107	2135	2107
q21	8489	8373	8340	8340
q22	1019	983	973	973
Total cold run time: 85347 ms
Total hot run time: 79853 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 203197 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 56178333ecaf2af43fd0537d607856447290958e, data reload: false

query1	929	419	383	383
query2	6528	2594	2922	2594
query3	6923	204	209	204
query4	20379	18054	17974	17974
query5	19758	6485	6557	6485
query6	348	216	238	216
query7	4769	297	299	297
query8	436	471	450	450
query9	3087	2662	2609	2609
query10	423	313	300	300
query11	11661	11789	10770	10770
query12	127	77	74	74
query13	5599	705	701	701
query14	18945	13409	13197	13197
query15	372	242	252	242
query16	6445	292	272	272
query17	1386	1538	882	882
query18	2259	416	416	416
query19	214	144	151	144
query20	79	83	82	82
query21	187	97	98	97
query22	5187	5052	5158	5052
query23	32299	31955	31741	31741
query24	6781	6461	6519	6461
query25	532	429	405	405
query26	456	166	162	162
query27	1560	299	296	296
query28	6175	2374	2345	2345
query29	3065	2930	2589	2589
query30	255	164	171	164
query31	906	724	742	724
query32	72	54	60	54
query33	373	259	268	259
query34	840	470	469	469
query35	1122	877	903	877
query36	1374	1100	1390	1100
query37	93	60	58	58
query38	3086	2930	2945	2930
query39	1371	1336	1327	1327
query40	212	96	98	96
query41	55	46	45	45
query42	80	86	85	85
query43	761	689	668	668
query44	1108	733	740	733
query45	256	235	241	235
query46	1253	959	994	959
query47	2034	1712	1738	1712
query48	1025	735	703	703
query49	604	364	375	364
query50	865	633	596	596
query51	4784	4647	4652	4647
query52	97	80	86	80
query53	449	327	321	321
query54	2632	2466	2425	2425
query55	92	81	76	76
query56	228	222	215	215
query57	1370	1034	1128	1034
query58	229	202	206	202
query59	4068	4079	4055	4055
query60	212	197	215	197
query61	100	96	98	96
query62	854	455	447	447
query63	492	338	343	338
query64	2233	1524	1501	1501
query65	3652	3514	3548	3514
query66	786	374	382	374
query67	15805	15553	15326	15326
query68	9031	640	662	640
query69	574	357	351	351
query70	1569	1573	1556	1556
query71	423	310	314	310
query72	6508	3482	3501	3482
query73	738	320	319	319
query74	6277	5813	5850	5813
query75	5341	3676	3732	3676
query76	5599	1135	1158	1135
query77	948	258	251	251
query78	12664	11676	11519	11519
query79	8622	654	656	654
query80	1462	404	399	399
query81	497	239	231	231
query82	1691	101	99	99
query83	168	137	131	131
query84	259	72	70	70
query85	864	327	336	327
query86	344	297	313	297
query87	3245	3008	3039	3008
query88	4957	2346	2333	2333
query89	386	283	303	283
query90	1969	207	218	207
query91	191	156	140	140
query92	58	55	53	53
query93	6156	604	598	598
query94	714	198	206	198
query95	1084	1037	1061	1037
query96	637	319	334	319
query97	6511	6377	6297	6297
query98	188	179	177	177
query99	2912	861	841	841
Total cold run time: 315324 ms
Total hot run time: 203197 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.28 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 56178333ecaf2af43fd0537d607856447290958e, data reload: false

query1	0.02	0.02	0.02
query2	0.07	0.03	0.02
query3	0.26	0.04	0.04
query4	1.79	0.07	0.07
query5	0.54	0.53	0.52
query6	1.29	0.62	0.59
query7	0.01	0.01	0.01
query8	0.03	0.02	0.03
query9	0.54	0.50	0.47
query10	0.54	0.54	0.53
query11	0.12	0.09	0.09
query12	0.11	0.09	0.09
query13	0.62	0.62	0.60
query14	0.78	0.78	0.78
query15	0.78	0.76	0.76
query16	0.37	0.39	0.37
query17	0.99	1.02	1.01
query18	0.23	0.26	0.24
query19	1.95	1.80	1.85
query20	0.01	0.01	0.01
query21	15.47	0.56	0.55
query22	2.38	2.37	1.46
query23	17.14	1.08	0.85
query24	6.24	4.33	0.83
query25	0.43	0.09	0.06
query26	0.90	0.17	0.14
query27	0.05	0.04	0.04
query28	3.84	0.71	0.73
query29	12.66	2.32	2.31
query30	0.65	0.50	0.54
query31	2.81	0.38	0.38
query32	3.38	0.50	0.48
query33	3.10	3.06	3.13
query34	15.27	4.79	4.79
query35	4.87	4.85	4.83
query36	1.07	1.01	1.03
query37	0.06	0.05	0.04
query38	0.04	0.02	0.02
query39	0.02	0.01	0.01
query40	0.17	0.14	0.14
query41	0.07	0.01	0.02
query42	0.02	0.02	0.01
query43	0.02	0.01	0.01
Total cold run time: 101.71 s
Total hot run time: 30.28 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 56178333ecaf2af43fd0537d607856447290958e with default session variables
Stream load json:         20 seconds loaded 2358488459 Bytes, about 112 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       21.1 seconds inserted 10000000 Rows, about 473K ops/s

@dataroaring dataroaring merged commit 47a865b into apache:branch-2.0 Jul 11, 2024
24 of 25 checks passed
mongo360 pushed a commit to mongo360/doris that referenced this pull request Aug 16, 2024
@SWJTU-ZhangLei
[chore](fe) Add some important information in fe log for debug (apach…
15f2a04 
…e#34352) (apache#37627)

* Add git commit info in fe.log/fe.out when fe start
* Add `metadata recovery mode` log in fe.log/fe.out
* Add bdb journal info in fe.log when size over `1M`

## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@SWJTU-ZhangLei @doris-robot @dataroaring