New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Introduce query limits at the resource group level #16303
Introduce query limits at the resource group level #16303
Conversation
5e3db7e
to
85f46c1
Compare
...com/facebook/presto/resourceGroups/reloading/ReloadingResourceGroupConfigurationManager.java
Outdated
Show resolved
Hide resolved
presto-main/src/main/java/com/facebook/presto/ExceededCpuLimitException.java
Outdated
Show resolved
Hide resolved
presto-main/src/main/java/com/facebook/presto/dispatcher/FailedDispatchQuery.java
Outdated
Show resolved
Hide resolved
presto-main/src/main/java/com/facebook/presto/execution/QueryTracker.java
Outdated
Show resolved
Hide resolved
...ource-group-managers/src/main/java/com/facebook/presto/resourceGroups/ResourceGroupSpec.java
Outdated
Show resolved
Hide resolved
presto-main/src/main/java/com/facebook/presto/execution/SqlQueryManager.java
Outdated
Show resolved
Hide resolved
...resource-group-managers/src/main/java/com/facebook/presto/resourceGroups/StaticSelector.java
Outdated
Show resolved
Hide resolved
d4b4731
to
84c028e
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
At a high level - 2 main comments here -
- Lets rename the field names to start with
perQuery
and end withLimit
. If the variable type isDataSize
/Duration
- then the name can end withLimit
. However, if the fields have the typelong
- in that case we need to put the unit at the end of the name likesMillis
orBytes
so that it is clear to the reader of the code what is the unit in question for the long value. - Use
requireNonNull
wherever possible in constructors to ensure that we are not passing aroundnull
s
...o-main/src/main/java/com/facebook/presto/execution/resourceGroups/InternalResourceGroup.java
Outdated
Show resolved
Hide resolved
...o-main/src/main/java/com/facebook/presto/execution/resourceGroups/InternalResourceGroup.java
Outdated
Show resolved
Hide resolved
...o-main/src/main/java/com/facebook/presto/execution/resourceGroups/InternalResourceGroup.java
Outdated
Show resolved
Hide resolved
...o-main/src/main/java/com/facebook/presto/execution/resourceGroups/InternalResourceGroup.java
Outdated
Show resolved
Hide resolved
...o-main/src/main/java/com/facebook/presto/execution/resourceGroups/InternalResourceGroup.java
Outdated
Show resolved
Hide resolved
...ource-group-managers/src/main/java/com/facebook/presto/resourceGroups/ResourceGroupSpec.java
Outdated
Show resolved
Hide resolved
...p-managers/src/main/java/com/facebook/presto/resourceGroups/db/ResourceGroupSpecBuilder.java
Outdated
Show resolved
Hide resolved
...ce-group-managers/src/main/java/com/facebook/presto/resourceGroups/db/ResourceGroupsDao.java
Outdated
Show resolved
Hide resolved
...ce-group-managers/src/main/java/com/facebook/presto/resourceGroups/db/ResourceGroupsDao.java
Outdated
Show resolved
Hide resolved
presto-spi/src/main/java/com/facebook/presto/spi/resourceGroups/ResourceGroupQueryLimits.java
Outdated
Show resolved
Hide resolved
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
One thing I realized after reviewing this PR is that the values that we passed to InternalResourceGroup
is not really being used since we are just using the values from ResourceGroupConfigurationManager
- lets see if we need those changes - we can make the changes a lot simpler
presto-main/src/main/java/com/facebook/presto/ExceededCpuLimitException.java
Outdated
Show resolved
Hide resolved
presto-main/src/main/java/com/facebook/presto/ExceededCpuLimitException.java
Outdated
Show resolved
Hide resolved
presto-main/src/main/java/com/facebook/presto/ExceededMemoryLimitException.java
Outdated
Show resolved
Hide resolved
presto-main/src/main/java/com/facebook/presto/dispatcher/FailedDispatchQuery.java
Outdated
Show resolved
Hide resolved
presto-main/src/main/java/com/facebook/presto/execution/QueryTracker.java
Outdated
Show resolved
Hide resolved
presto-main/src/main/java/com/facebook/presto/execution/SqlQueryManager.java
Outdated
Show resolved
Hide resolved
presto-main/src/main/java/com/facebook/presto/execution/SqlQueryManager.java
Outdated
Show resolved
Hide resolved
presto-main/src/main/java/com/facebook/presto/execution/SqlQueryManager.java
Outdated
Show resolved
Hide resolved
presto-main/src/main/java/com/facebook/presto/execution/LimitWithLevelContext.java
Outdated
Show resolved
Hide resolved
presto-main/src/main/java/com/facebook/presto/memory/ClusterMemoryManager.java
Outdated
Show resolved
Hide resolved
haven't reviewed the full PR - will continue the same tomorrow |
6afefbf
to
1ed5b7d
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Review for commit " Add resource limits per query on a resource group "
...o-main/src/main/java/com/facebook/presto/execution/resourceGroups/InternalResourceGroup.java
Outdated
Show resolved
Hide resolved
presto-spi/src/main/java/com/facebook/presto/spi/resourceGroups/ResourceGroup.java
Outdated
Show resolved
Hide resolved
1ed5b7d
to
97d8dfe
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
flushing some more comments
presto-main/src/main/java/com/facebook/presto/ExceededCpuLimitException.java
Outdated
Show resolved
Hide resolved
presto-main/src/main/java/com/facebook/presto/ExceededCpuLimitException.java
Outdated
Show resolved
Hide resolved
presto-main/src/main/java/com/facebook/presto/dispatcher/LocalDispatchQuery.java
Outdated
Show resolved
Hide resolved
presto-main/src/main/java/com/facebook/presto/dispatcher/LocalDispatchQuery.java
Outdated
Show resolved
Hide resolved
presto-main/src/main/java/com/facebook/presto/execution/DataDefinitionExecution.java
Outdated
Show resolved
Hide resolved
presto-main/src/main/java/com/facebook/presto/execution/LimitWithLevelContext.java
Outdated
Show resolved
Hide resolved
presto-main/src/main/java/com/facebook/presto/execution/QueryTracker.java
Outdated
Show resolved
Hide resolved
presto-main/src/main/java/com/facebook/presto/execution/SqlQueryManager.java
Outdated
Show resolved
Hide resolved
presto-main/src/main/java/com/facebook/presto/memory/ClusterMemoryManager.java
Outdated
Show resolved
Hide resolved
...resource-group-managers/src/main/java/com/facebook/presto/resourceGroups/SelectorResult.java
Outdated
Show resolved
Hide resolved
presto-spi/src/main/java/com/facebook/presto/spi/resourceGroups/ResourceGroupQueryLimits.java
Outdated
Show resolved
Hide resolved
presto-spi/src/main/java/com/facebook/presto/spi/resourceGroups/ResourceGroupQueryLimits.java
Outdated
Show resolved
Hide resolved
presto-spi/src/main/java/com/facebook/presto/spi/resourceGroups/ResourceGroupQueryLimits.java
Show resolved
Hide resolved
7969aa4
to
9b45461
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Add resource limits per query on a resource group
For this commit - mostly good - just some naming issues
You should rename the commit title to Support passing per query limits in resource group configuration
The commit description starts with saying Allows for more intelligent thread allocation for queues
- this diff commit doesn't do that. Please ensure that the commit description and title matches with what the commit actually does
...ource-group-managers/src/main/java/com/facebook/presto/resourceGroups/ResourceGroupSpec.java
Outdated
Show resolved
Hide resolved
...ce-group-managers/src/test/java/com/facebook/presto/resourceGroups/TestingResourceGroup.java
Outdated
Show resolved
Hide resolved
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Also - please make the SelectorResult/SelectionContext changes we talked about
presto-main/src/main/java/com/facebook/presto/execution/QueryLimit.java
Outdated
Show resolved
Hide resolved
presto-main/src/main/java/com/facebook/presto/execution/SqlQueryExecution.java
Outdated
Show resolved
Hide resolved
presto-main/src/main/java/com/facebook/presto/execution/SqlQueryManager.java
Outdated
Show resolved
Hide resolved
presto-main/src/test/java/com/facebook/presto/dispatcher/TestLocalDispatchQuery.java
Outdated
Show resolved
Hide resolved
presto-main/src/test/java/com/facebook/presto/dispatcher/TestLocalDispatchQuery.java
Outdated
Show resolved
Hide resolved
.../src/main/java/com/facebook/presto/resourceGroups/FileResourceGroupConfigurationManager.java
Outdated
Show resolved
Hide resolved
9b45461
to
77026a9
Compare
@tdcmeehan - this diff is ready for an initial look. We are still looking at unittests and code style fixes - but otherwise - would love to get an idea on the approach and the PR should be ready for final review by tomorrow. |
0757343
to
9361c39
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Getting there - this should be last major set of review comments
presto-main/src/main/java/com/facebook/presto/execution/DataDefinitionExecution.java
Outdated
Show resolved
Hide resolved
...resource-group-managers/src/main/java/com/facebook/presto/resourceGroups/SelectorResult.java
Outdated
Show resolved
Hide resolved
...s/src/main/java/com/facebook/presto/resourceGroups/AbstractResourceConfigurationManager.java
Outdated
Show resolved
Hide resolved
...s/src/main/java/com/facebook/presto/resourceGroups/AbstractResourceConfigurationManager.java
Outdated
Show resolved
Hide resolved
...s/src/main/java/com/facebook/presto/resourceGroups/AbstractResourceConfigurationManager.java
Outdated
Show resolved
Hide resolved
...managers/src/main/java/com/facebook/presto/resourceGroups/db/DbSourceExactMatchSelector.java
Outdated
Show resolved
Hide resolved
...com/facebook/presto/resourceGroups/reloading/ReloadingResourceGroupConfigurationManager.java
Outdated
Show resolved
Hide resolved
...managers/src/main/java/com/facebook/presto/resourceGroups/db/DbSourceExactMatchSelector.java
Outdated
Show resolved
Hide resolved
.../src/main/java/com/facebook/presto/resourceGroups/FileResourceGroupConfigurationManager.java
Outdated
Show resolved
Hide resolved
...s/src/main/java/com/facebook/presto/resourceGroups/AbstractResourceConfigurationManager.java
Outdated
Show resolved
Hide resolved
9361c39
to
4dd45d8
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Commit 1
...ource-group-managers/src/main/java/com/facebook/presto/resourceGroups/ResourceGroupSpec.java
Outdated
Show resolved
Hide resolved
presto-spi/src/main/java/com/facebook/presto/spi/resourceGroups/ResourceGroupQueryLimits.java
Outdated
Show resolved
Hide resolved
presto-spi/src/main/java/com/facebook/presto/spi/resourceGroups/ResourceGroupQueryLimits.java
Outdated
Show resolved
Hide resolved
409786c
to
ccb9d9a
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
mostly naming changes and things here and there
...o-main/src/main/java/com/facebook/presto/execution/resourceGroups/InternalResourceGroup.java
Outdated
Show resolved
Hide resolved
...ource-group-managers/src/main/java/com/facebook/presto/resourceGroups/ResourceGroupSpec.java
Outdated
Show resolved
Hide resolved
...p-managers/src/main/java/com/facebook/presto/resourceGroups/db/ResourceGroupSpecBuilder.java
Outdated
Show resolved
Hide resolved
presto-spi/src/main/java/com/facebook/presto/spi/resourceGroups/ResourceGroupQueryLimits.java
Outdated
Show resolved
Hide resolved
presto-main/src/main/java/com/facebook/presto/execution/SqlQueryExecution.java
Outdated
Show resolved
Hide resolved
presto-main/src/main/java/com/facebook/presto/execution/SqlQueryExecution.java
Outdated
Show resolved
Hide resolved
...o-main/src/main/java/com/facebook/presto/execution/resourceGroups/InternalResourceGroup.java
Outdated
Show resolved
Hide resolved
...facebook/presto/resourceGroups/reloading/TestReloadingResourceGroupConfigurationManager.java
Outdated
Show resolved
Hide resolved
...facebook/presto/resourceGroups/reloading/TestReloadingResourceGroupConfigurationManager.java
Outdated
Show resolved
Hide resolved
ccb9d9a
to
a20caa3
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM !
...o-main/src/main/java/com/facebook/presto/execution/resourceGroups/InternalResourceGroup.java
Outdated
Show resolved
Hide resolved
presto-tests/src/test/resources/resource_groups_config_simple.json
Outdated
Show resolved
Hide resolved
ebecbcf
to
27fe716
Compare
presto-main/src/main/java/com/facebook/presto/dispatcher/LocalDispatchQuery.java
Outdated
Show resolved
Hide resolved
presto-main/src/main/java/com/facebook/presto/dispatcher/LocalDispatchQuery.java
Outdated
Show resolved
Hide resolved
presto-main/src/main/java/com/facebook/presto/dispatcher/LocalDispatchQuery.java
Outdated
Show resolved
Hide resolved
presto-main/src/main/java/com/facebook/presto/execution/QueryTracker.java
Outdated
Show resolved
Hide resolved
presto-main/src/main/java/com/facebook/presto/execution/QueryTracker.java
Outdated
Show resolved
Hide resolved
presto-main/src/main/java/com/facebook/presto/execution/SqlQueryManager.java
Outdated
Show resolved
Hide resolved
...o-main/src/main/java/com/facebook/presto/execution/resourceGroups/InternalResourceGroup.java
Outdated
Show resolved
Hide resolved
presto-main/src/main/java/com/facebook/presto/memory/ClusterMemoryManager.java
Outdated
Show resolved
Hide resolved
presto-main/src/main/java/com/facebook/presto/execution/QueryLimit.java
Outdated
Show resolved
Hide resolved
4029568
to
5ff2403
Compare
Setting resource group resources on a query for cpu time, execution time, and memory usage, adds greater granularity to resource allocation.
5ff2403
to
a151fbf
Compare
presto-main/src/main/java/com/facebook/presto/execution/DataDefinitionExecution.java
Show resolved
Hide resolved
presto-main/src/main/java/com/facebook/presto/execution/SqlQueryExecution.java
Outdated
Show resolved
Hide resolved
Enables query manager to kill a query which exceeds its resource group allocation.
a151fbf
to
b237980
Compare
testTwoQueriesAtSameTime is failing most of the times. I have several open PRs and i have seen that failing at least 3 times. https://github.com/prestodb/presto/pull/16434/checks?check_run_id=3128611696#step:7:35072 Error: Tests run: 1874, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 1,800.32 s <<< FAILURE! - in TestSuite Often fails. This is the last change that touched this code. Could this be a regression or should the test be just disabled ? |
Introduce the ability to add query limits (cpu time, execution time, and memory limit) at the resource group level. These limits can complement the
SelectorResourceEstimate
passed toSelectorSpec
to ensure that if the query exceeds the indented query limits, we can fail the query and protect the resource group.This is enabled by adding to the
ResourceGroupSpec
the fieldperQueryLimits
, a JSON object of the optional fieldsexecutionTimeLimit
,totalMemoryLimit
,cpuTimeLimit
which is then passed to the coordinator to implement the appropriate limits.Test plan - Unit tests added and build success are sufficient for first commit. Second commit will have e2e by deploying fbpkg to test cluster to ensure resource group limits can be enforced by
SqlQueryManager
. Unit testing also on second commit.Depends on https://github.com/facebookexternal/presto-facebook/pull/1599