Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[chore](fe) Add some important information in fe log for debug #34352

Merged
merged 1 commit into from
May 6, 2024

Conversation

SWJTU-ZhangLei
Copy link
Contributor

  • Add git commit info in fe.log/fe.out when fe start
  • Add metadata recovery mode log in fe.log/fe.out
  • Add bdb journal info in fe.log when size over 1M

Proposed changes

Issue Number: close #xxx

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@SWJTU-ZhangLei
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 41637 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 974a34b0833712e24ff4e3da3c046b4ee20fec57, data reload: false

------ Round 1 ----------------------------------
q1	17601	4322	4197	4197
q2	2016	190	191	190
q3	10460	1208	1190	1190
q4	10201	881	743	743
q5	7528	2730	2696	2696
q6	224	136	136	136
q7	1033	601	601	601
q8	9248	2139	2094	2094
q9	9812	6719	6766	6719
q10	9146	3945	3851	3851
q11	466	244	241	241
q12	490	230	216	216
q13	18121	3191	3136	3136
q14	274	231	243	231
q15	522	474	470	470
q16	519	408	397	397
q17	996	700	690	690
q18	8278	7767	7747	7747
q19	1847	1554	1474	1474
q20	657	311	310	310
q21	5264	4026	4234	4026
q22	348	282	288	282
Total cold run time: 115051 ms
Total hot run time: 41637 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4498	4447	4367	4367
q2	388	264	254	254
q3	3178	2901	2909	2901
q4	1963	1615	1621	1615
q5	5377	5485	5472	5472
q6	210	124	125	124
q7	2333	1959	1964	1959
q8	3277	3380	3417	3380
q9	8652	8766	8726	8726
q10	4032	3705	3823	3705
q11	576	501	503	501
q12	768	640	638	638
q13	17106	3088	3162	3088
q14	306	273	250	250
q15	519	485	495	485
q16	481	432	418	418
q17	1744	1473	1438	1438
q18	7684	7585	7585	7585
q19	3140	1602	1526	1526
q20	1993	1762	1793	1762
q21	10271	4751	4729	4729
q22	574	505	501	501
Total cold run time: 79070 ms
Total hot run time: 55424 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 186541 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 974a34b0833712e24ff4e3da3c046b4ee20fec57, data reload: false

query1	921	367	354	354
query2	6444	2409	2431	2409
query3	6645	215	226	215
query4	22862	21196	21285	21196
query5	4133	422	424	422
query6	257	172	172	172
query7	4575	294	291	291
query8	250	201	205	201
query9	8528	2434	2412	2412
query10	436	243	255	243
query11	14718	14143	14163	14143
query12	138	88	84	84
query13	1651	372	397	372
query14	10008	7663	8377	7663
query15	247	177	175	175
query16	8103	265	271	265
query17	1698	582	562	562
query18	2123	288	277	277
query19	205	153	155	153
query20	91	87	91	87
query21	193	128	126	126
query22	5102	4968	4830	4830
query23	33617	33089	33141	33089
query24	6741	2920	2921	2920
query25	537	369	363	363
query26	689	151	147	147
query27	1946	311	332	311
query28	3783	2069	2071	2069
query29	854	606	607	606
query30	216	155	149	149
query31	994	746	709	709
query32	98	51	52	51
query33	499	246	239	239
query34	885	492	491	491
query35	755	680	657	657
query36	1053	905	896	896
query37	101	68	67	67
query38	3187	3013	3005	3005
query39	1619	1513	1696	1513
query40	197	140	125	125
query41	41	38	39	38
query42	103	96	100	96
query43	572	548	570	548
query44	1079	722	747	722
query45	264	248	255	248
query46	1058	707	728	707
query47	1998	1865	1883	1865
query48	367	297	292	292
query49	770	391	404	391
query50	780	392	392	392
query51	6864	6784	6651	6651
query52	103	88	91	88
query53	342	294	284	284
query54	265	230	238	230
query55	79	76	73	73
query56	233	219	214	214
query57	1238	1144	1183	1144
query58	211	193	193	193
query59	3427	3222	3234	3222
query60	254	226	230	226
query61	89	87	87	87
query62	579	451	450	450
query63	303	273	274	273
query64	8257	7272	7199	7199
query65	3132	3013	3066	3013
query66	796	330	343	330
query67	15497	15181	15028	15028
query68	5323	537	529	529
query69	496	304	306	304
query70	1152	1124	1121	1121
query71	398	275	265	265
query72	7698	2527	2330	2330
query73	715	326	334	326
query74	6519	6104	6074	6074
query75	3338	2644	2724	2644
query76	2842	998	1015	998
query77	482	265	260	260
query78	10958	10340	10131	10131
query79	4763	517	523	517
query80	2027	430	426	426
query81	512	229	226	226
query82	1232	91	101	91
query83	293	169	176	169
query84	263	90	84	84
query85	1732	286	263	263
query86	519	279	304	279
query87	3248	3118	3136	3118
query88	4948	2389	2397	2389
query89	496	383	375	375
query90	2059	177	176	176
query91	124	95	98	95
query92	63	47	46	46
query93	5735	515	500	500
query94	1251	184	182	182
query95	397	305	300	300
query96	601	272	267	267
query97	3150	2939	2954	2939
query98	235	222	215	215
query99	1212	839	864	839
Total cold run time: 280581 ms
Total hot run time: 186541 ms

Copy link
Contributor

PR approved by anyone and no changes requested.

Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

-v does not work any more.

@SWJTU-ZhangLei
Copy link
Contributor Author

-v does not work any more.

 // pid file
 if (!cmdLineOpts.isVersion() && !createAndLockPidFile(pidDir + "/fe.pid")) {
     throw new IOException("pid file is already locked.");
 }

-v option still work

* Add git commit info in fe.log/fe.out when fe start
* Add `metadata recovery mode` log in fe.log/fe.out
* Add bdb journal info in fe.log when size over `1M`
@SWJTU-ZhangLei
Copy link
Contributor Author

run buildall

Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Copy link
Contributor

github-actions bot commented May 6, 2024

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label May 6, 2024
@doris-robot
Copy link

TPC-H: Total hot run time: 39795 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 271f7dfbe18c2661be441d39a1957bce55584e94, data reload: false

------ Round 1 ----------------------------------
q1	17798	4533	4375	4375
q2	2583	193	194	193
q3	11326	1162	1177	1162
q4	10297	846	799	799
q5	7548	2732	2656	2656
q6	222	133	132	132
q7	1019	577	552	552
q8	9218	2037	2019	2019
q9	8842	6547	6524	6524
q10	9024	3715	3733	3715
q11	450	235	239	235
q12	420	217	211	211
q13	17772	2964	2913	2913
q14	258	210	216	210
q15	511	471	470	470
q16	496	390	372	372
q17	949	698	675	675
q18	8084	7363	7306	7306
q19	6917	1545	1500	1500
q20	662	317	316	316
q21	4971	3190	3995	3190
q22	341	270	286	270
Total cold run time: 119708 ms
Total hot run time: 39795 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4381	4173	4220	4173
q2	380	258	260	258
q3	2928	2785	2685	2685
q4	1823	1573	1583	1573
q5	5266	5255	5263	5255
q6	214	122	122	122
q7	2261	1874	1870	1870
q8	3213	3350	3345	3345
q9	8480	8405	8424	8405
q10	3878	3670	3633	3633
q11	575	492	479	479
q12	749	594	576	576
q13	16335	2957	3678	2957
q14	283	276	248	248
q15	515	467	456	456
q16	483	398	426	398
q17	1770	1501	1473	1473
q18	7772	7499	7336	7336
q19	1636	1492	1487	1487
q20	1931	1766	1762	1762
q21	4920	4933	4931	4931
q22	568	482	481	481
Total cold run time: 70361 ms
Total hot run time: 53903 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 185894 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 271f7dfbe18c2661be441d39a1957bce55584e94, data reload: false

query1	913	352	350	350
query2	6444	2331	2312	2312
query3	6702	201	212	201
query4	22901	21203	21191	21191
query5	4178	423	418	418
query6	280	192	181	181
query7	4592	286	296	286
query8	234	190	179	179
query9	8499	2351	2326	2326
query10	439	235	240	235
query11	14783	14165	14101	14101
query12	128	93	95	93
query13	1641	357	384	357
query14	8614	7520	8004	7520
query15	280	175	173	173
query16	8200	245	255	245
query17	1894	551	537	537
query18	2100	269	268	268
query19	219	148	148	148
query20	93	87	85	85
query21	202	129	129	129
query22	5133	4904	4993	4904
query23	33871	33162	33103	33103
query24	10857	2881	2816	2816
query25	595	361	358	358
query26	1136	152	145	145
query27	2488	310	318	310
query28	7224	2039	2024	2024
query29	856	601	598	598
query30	284	150	151	150
query31	967	727	715	715
query32	94	52	52	52
query33	758	238	233	233
query34	1088	473	479	473
query35	790	654	652	652
query36	1091	921	891	891
query37	137	66	73	66
query38	3179	3030	3012	3012
query39	1597	1557	1551	1551
query40	209	133	124	124
query41	42	41	39	39
query42	102	95	95	95
query43	583	544	539	539
query44	1198	715	734	715
query45	247	250	259	250
query46	1071	730	711	711
query47	1957	1878	1885	1878
query48	359	295	292	292
query49	1081	390	395	390
query50	775	382	370	370
query51	6871	6635	6511	6511
query52	108	90	87	87
query53	348	281	276	276
query54	297	232	224	224
query55	78	70	70	70
query56	232	214	217	214
query57	1220	1144	1154	1144
query58	217	195	195	195
query59	3492	3231	3224	3224
query60	246	244	228	228
query61	92	119	85	85
query62	641	441	458	441
query63	305	278	278	278
query64	8659	7172	7136	7136
query65	3069	3082	3054	3054
query66	809	336	331	331
query67	15317	15051	15224	15051
query68	5203	524	524	524
query69	475	304	316	304
query70	1155	1078	1125	1078
query71	387	273	269	269
query72	7801	2647	2455	2455
query73	708	315	319	315
query74	6433	6043	6185	6043
query75	3361	2645	2633	2633
query76	2886	1018	959	959
query77	417	265	267	265
query78	11108	10277	10323	10277
query79	8419	519	512	512
query80	2103	457	462	457
query81	549	225	228	225
query82	1536	97	94	94
query83	275	171	175	171
query84	275	89	97	89
query85	2026	315	313	313
query86	482	300	260	260
query87	3323	3141	3084	3084
query88	5242	2306	2338	2306
query89	497	379	370	370
query90	1994	186	182	182
query91	122	96	96	96
query92	65	51	49	49
query93	6372	514	497	497
query94	1209	191	183	183
query95	392	300	303	300
query96	602	257	257	257
query97	3177	2939	2920	2920
query98	242	220	214	214
query99	1220	856	853	853
Total cold run time: 295162 ms
Total hot run time: 185894 ms

@dataroaring dataroaring merged commit 5c5cf37 into apache:master May 6, 2024
24 of 26 checks passed
ByteYue pushed a commit to ByteYue/doris that referenced this pull request May 15, 2024
…e#34352)

* Add git commit info in fe.log/fe.out when fe start
* Add `metadata recovery mode` log in fe.log/fe.out
* Add bdb journal info in fe.log when size over `1M`
SWJTU-ZhangLei added a commit to SWJTU-ZhangLei/incubator-doris that referenced this pull request Jul 10, 2024
…e#34352)

* Add git commit info in fe.log/fe.out when fe start
* Add `metadata recovery mode` log in fe.log/fe.out
* Add bdb journal info in fe.log when size over `1M`
dataroaring pushed a commit that referenced this pull request Jul 11, 2024
… (#37627)

* Add git commit info in fe.log/fe.out when fe start
* Add `metadata recovery mode` log in fe.log/fe.out
* Add bdb journal info in fe.log when size over `1M`

## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->
mongo360 pushed a commit to mongo360/doris that referenced this pull request Aug 16, 2024
…e#34352) (apache#37627)

* Add git commit info in fe.log/fe.out when fe start
* Add `metadata recovery mode` log in fe.log/fe.out
* Add bdb journal info in fe.log when size over `1M`

## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by one committer. dev/2.0.13-merged dev/3.0.0-merged reviewed
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants