Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Improvement](executor)Add queue columns for active_queries #32259

Merged
merged 1 commit into from Mar 15, 2024
Merged

[Improvement](executor)Add queue columns for active_queries #32259

merged 1 commit into from Mar 15, 2024

Conversation

wangbo
Copy link
Contributor

@wangbo wangbo commented Mar 14, 2024

Add QUEUE_START_TIME/QUEUE_END_TIME/QUERY_STATUS column for active_queries;

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

@wangbo
Copy link
Contributor Author

wangbo commented Mar 14, 2024

run buildall

@wangbo
Copy link
Contributor Author

wangbo commented Mar 14, 2024

run buildall

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

1 similar comment
@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TPC-H: Total hot run time: 38312 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 0ea8ee4d4a2b18bd703d6f6c59a5becbb6ce2311, data reload: false

------ Round 1 ----------------------------------
q1	17705	4368	4129	4129
q2	2029	151	146	146
q3	10685	1120	912	912
q4	7813	785	796	785
q5	7507	2594	2715	2594
q6	185	121	122	121
q7	1223	833	814	814
q8	9380	2073	1983	1983
q9	7104	6451	6457	6451
q10	8429	3463	3621	3463
q11	424	230	225	225
q12	651	301	290	290
q13	17810	2838	2860	2838
q14	269	246	245	245
q15	494	457	456	456
q16	491	396	380	380
q17	949	548	544	544
q18	7385	6493	6481	6481
q19	4542	1406	1482	1406
q20	555	286	278	278
q21	6284	3478	3484	3478
q22	359	293	348	293
Total cold run time: 112273 ms
Total hot run time: 38312 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4171	4122	4051	4051
q2	324	231	228	228
q3	2949	2866	2830	2830
q4	1880	1565	1586	1565
q5	5216	5282	5264	5264
q6	197	123	114	114
q7	2251	1848	1825	1825
q8	3163	3283	3286	3283
q9	8604	8618	8600	8600
q10	3739	3676	3715	3676
q11	541	440	441	440
q12	725	568	518	518
q13	16913	2865	2851	2851
q14	272	244	256	244
q15	484	443	441	441
q16	452	408	400	400
q17	1729	1479	1473	1473
q18	7566	7270	7032	7032
q19	1632	1563	1574	1563
q20	1905	1703	1733	1703
q21	4829	4735	4766	4735
q22	530	473	425	425
Total cold run time: 70072 ms
Total hot run time: 53261 ms

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 34.99% (8573/24499)
Line Coverage: 26.73% (69458/259807)
Region Coverage: 26.01% (36079/138691)
Branch Coverage: 22.97% (18423/80208)
Coverage Report: http://coverage.selectdb-in.cc/coverage/0ea8ee4d4a2b18bd703d6f6c59a5becbb6ce2311_0ea8ee4d4a2b18bd703d6f6c59a5becbb6ce2311/report/index.html

yiguolei
yiguolei reviewed Mar 14, 2024
View reviewed changes
fe/fe-core/src/main/java/org/apache/doris/qe/Coordinator.java Outdated
LOG.error("query (id=" + DebugUtil.printId(queryId) + ") " + queueToken.getOfferResultDetail());
queryQueue.returnToken(queueToken);
throw new UserException(queueToken.getOfferResultDetail());
long queueStartTime = System.currentTimeMillis();
Copy link
Contributor

@yiguolei yiguolei Mar 14, 2024 •
edited

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

这么改不好，容易锁死。
直接把queue start time和queue end time 放到queue token 里，然后token 本身是coordinator的一个属性就ok了。
当我们显示的时候，直接从token 里拿start time和end time 就可以了

@wangbo
Copy link
Contributor Author

wangbo commented Mar 15, 2024

run buildall

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

yiguolei
yiguolei reviewed Mar 15, 2024
View reviewed changes
fe/fe-core/src/main/java/org/apache/doris/qe/QeProcessorImpl.java Outdated
@@ -105,6 +106,26 @@ public void registerQuery(TUniqueId queryId, QueryInfo info) throws UserExceptio
}
}

@Override
public QueryInfo getQueryInfoByQueryId(TUniqueId queryId) throws UserException {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Both query start time and token are in coordinator, we could get these info directly from coordinator.
And QueryInfo contains a coordinator. So that we do not need these API any more.

yiguolei
yiguolei reviewed Mar 15, 2024
View reviewed changes
fe/fe-core/src/main/java/org/apache/doris/tablefunction/MetadataGenerator.java Outdated
@@ -574,6 +537,36 @@ private static TFetchSchemaTableDataResult queriesMetadataResult(TSchemaTableReq
trow.addToColumnValue(new TCell().setStringVal(""));
}
trow.addToColumnValue(new TCell().setStringVal(selfNode));

QueueToken queueToken = queryInfo.getQueueToken();
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do not use token directly here.
Add related method in QueryInfo, for example getQueueStartTime, getQueueEndTime
and then QueryInfo could use coordinator to get these time.

yiguolei
yiguolei reviewed Mar 15, 2024
View reviewed changes
fe/fe-core/src/main/java/org/apache/doris/qe/QeProcessorImpl.java Outdated
@@ -295,6 +318,18 @@ public String getSql() {
public long getStartExecTime() {
return startExecTime;
}

public void setQueueToken(QueueToken queueToken) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

remove setQueueToken and getQueueToken setStartExecTime
Get these info from coordinator directly.

@doris-robot
Copy link

TPC-H: Total hot run time: 38578 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 1d1c8f8996f40604d3b1943c5eea236b7fd24fa0, data reload: false

------ Round 1 ----------------------------------
q1	17642	4254	4159	4159
q2	2021	155	147	147
q3	10592	1098	905	905
q4	5001	729	750	729
q5	7413	2828	2814	2814
q6	200	120	127	120
q7	1167	817	825	817
q8	9342	2016	2018	2016
q9	7045	6496	6442	6442
q10	8553	3514	3655	3514
q11	440	218	217	217
q12	777	307	293	293
q13	17820	2850	2835	2835
q14	276	249	245	245
q15	503	452	444	444
q16	499	389	383	383
q17	957	550	641	550
q18	7328	6555	6404	6404
q19	3923	1425	1391	1391
q20	537	274	273	273
q21	7073	3570	3666	3570
q22	359	310	311	310
Total cold run time: 109468 ms
Total hot run time: 38578 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4143	4066	4081	4066
q2	316	229	220	220
q3	2946	2832	2870	2832
q4	1840	1569	1544	1544
q5	5273	5263	5253	5253
q6	200	117	116	116
q7	2190	1853	1815	1815
q8	3176	3304	3291	3291
q9	8532	8575	8616	8575
q10	3740	3693	3662	3662
q11	550	457	443	443
q12	728	538	549	538
q13	16930	2854	2845	2845
q14	273	243	246	243
q15	485	445	454	445
q16	440	424	402	402
q17	1730	1503	1464	1464
q18	7476	7296	7070	7070
q19	1619	1556	1557	1556
q20	1919	1697	1740	1697
q21	4762	4774	4710	4710
q22	543	455	482	455
Total cold run time: 69811 ms
Total hot run time: 53242 ms

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 34.95% (8574/24534)
Line Coverage: 26.64% (69470/260727)
Region Coverage: 25.94% (36086/139116)
Branch Coverage: 22.89% (18425/80484)
Coverage Report: http://coverage.selectdb-in.cc/coverage/1d1c8f8996f40604d3b1943c5eea236b7fd24fa0_1d1c8f8996f40604d3b1943c5eea236b7fd24fa0/report/index.html

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

2 similar comments
@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@wangbo
Copy link
Contributor Author

wangbo commented Mar 15, 2024

run buildall

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

1 similar comment
@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

yiguolei
yiguolei approved these changes Mar 15, 2024
View reviewed changes
Copy link
Contributor

@yiguolei yiguolei left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added approved Indicates a PR has been approved by one committer. reviewed labels Mar 15, 2024
@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@doris-robot
Copy link

TPC-H: Total hot run time: 37997 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 94af99aa45e4beab758ed3b318f13d439a6041f6, data reload: false

------ Round 1 ----------------------------------
q1	17626	4176	4097	4097
q2	2023	153	142	142
q3	10613	1052	896	896
q4	7086	706	692	692
q5	7448	2611	2679	2611
q6	181	121	120	120
q7	1182	789	794	789
q8	9339	1921	2004	1921
q9	7150	6387	6396	6387
q10	8495	3527	3640	3527
q11	438	225	211	211
q12	769	305	283	283
q13	18139	2810	2821	2810
q14	274	244	252	244
q15	497	462	442	442
q16	509	382	383	382
q17	945	557	512	512
q18	7004	6396	6509	6396
q19	1567	1388	1372	1372
q20	564	285	278	278
q21	6303	3594	3663	3594
q22	357	291	333	291
Total cold run time: 108509 ms
Total hot run time: 37997 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4090	4194	4103	4103
q2	318	226	229	226
q3	2928	2798	2792	2792
q4	1839	1579	1530	1530
q5	5230	5240	5210	5210
q6	190	116	116	116
q7	2210	1874	1844	1844
q8	3163	3287	3280	3280
q9	8536	8486	8504	8486
q10	3652	3707	3662	3662
q11	540	443	458	443
q12	708	553	559	553
q13	17211	2887	2979	2887
q14	289	268	264	264
q15	482	448	449	448
q16	474	445	423	423
q17	1747	1478	1459	1459
q18	7700	7150	7031	7031
q19	1675	1459	1451	1451
q20	1890	1670	1695	1670
q21	4868	4634	4621	4621
q22	539	449	436	436
Total cold run time: 70279 ms
Total hot run time: 52935 ms

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 34.95% (8574/24534)
Line Coverage: 26.65% (69485/260731)
Region Coverage: 25.94% (36087/139107)
Branch Coverage: 22.90% (18427/80478)
Coverage Report: http://coverage.selectdb-in.cc/coverage/94af99aa45e4beab758ed3b318f13d439a6041f6_94af99aa45e4beab758ed3b318f13d439a6041f6/report/index.html

morningman
morningman approved these changes Mar 15, 2024
View reviewed changes
Copy link
Contributor

@morningman morningman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@yiguolei yiguolei merged commit b1d6340 into apache:master Mar 15, 2024
24 of 29 checks passed
yiguolei pushed a commit to yiguolei/incubator-doris that referenced this pull request Mar 16, 2024
@wangbo @Doris-Extras
Add QUEUE_START_TIME/QUEUE_END_TIME/QUERY_STATUS column for active_qu…
83ab61a 
…eries (apache#32259)
@yiguolei yiguolei mentioned this pull request Mar 24, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@morningman morningman morningman approved these changes

@yiguolei yiguolei yiguolei approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. reviewed workload-group
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
4 participants
@wangbo @doris-robot @morningman @yiguolei