-
Notifications
You must be signed in to change notification settings - Fork 13.8k
[FLINK-14504][rest] Add Cluster DataSet REST API #11628
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
|
Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community Automated ChecksLast check on commit 44066c5 (Fri Apr 03 10:45:56 UTC 2020) Warnings:
Mention the bot in a comment to re-run the automated checks. Review Progress
Please see the Pull Request Review Guide for a full explanation of the review process. DetailsThe Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commandsThe @flinkbot bot supports the following commands:
|
...untime/src/main/java/org/apache/flink/runtime/rest/messages/dataset/ClusterDataSetEntry.java
Show resolved
Hide resolved
.../main/java/org/apache/flink/runtime/rest/messages/dataset/ClusterDataSetIdPathParameter.java
Outdated
Show resolved
Hide resolved
.../java/org/apache/flink/runtime/rest/messages/dataset/ClusterDataSetListResponseBodyTest.java
Outdated
Show resolved
Hide resolved
...va/org/apache/flink/runtime/rest/handler/resourcemanager/AbstractResourceManagerHandler.java
Show resolved
Hide resolved
...untime/src/main/java/org/apache/flink/runtime/rest/messages/dataset/ClusterDataSetEntry.java
Show resolved
Hide resolved
...n/java/org/apache/flink/runtime/rest/handler/dataset/ClusterDataSetPartitionListHandler.java
Outdated
Show resolved
Hide resolved
flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/DataSetMetaInfo.java
Show resolved
Hide resolved
.../java/org/apache/flink/runtime/rest/messages/dataset/ClusterDataSetListResponseBodyTest.java
Outdated
Show resolved
Hide resolved
The name was misleading since the handler only interacts with the ResourceManager, and just happened to only be used by TaskExecutor-related handlers.
We don't have a mechanism to ensure that either all partitions of a data set are registered or the remaining partitions are released at some point. In these cases the user has to step in, but without being able to list incomplete partitions there's nothing that can be done. This commit relaxes the filtering, and introduces an additional optional count for the number of currently registered partitions. This count is only determined when the listing is requested for performance and complexity reasons.
Adds the REST API for managing cluster partitions. The collection of all cluster partitions belonging to a single data set is referred to as a "Cluster DataSet".
The API allows the user to list and delete partitions. The handlers are relatively straight-forward and pretty much just forward calls to the RM (which forwards them to the Tracker), at most converting some data-structures.
As a prerequisite we no longer filter incomplete data sets in
PartitionTracker#listDataSets.We don't have a mechanism to ensure that either all partitions of a data set are registered or the remaining partitions are released at some point.
In these cases the user has to step in, but without being able to list incomplete partitions there's nothing that can be done.
To differentiate between complete/incomplete partitions we expose the number of currently registered partitions.
This count is only determined when the listing is requested for performance and complexity reasons.