[FLINK-4871] [mini cluster] Add memory calculation for TaskManagers to MiniCluster #2669
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
asfgit
force-pushed
the
flip-6
branch
2 times, most recently
from
7041f93
to
930334e
Compare
tillrohrmann
force-pushed
the
miniClusterUpdate
branch
from
58e8d6c
to
4918b63
Compare
mxm
reviewed
Oct 21, 2016
| * | ||
| * @return | ||
| */ | ||
| private long calculateManagedMemoryPerTaskManager() { |
There was a problem hiding this comment.
getOrCalculateManagedMemoryPerTaskManager? 😃
There was a problem hiding this comment.
Yes, will rename the method.
| // ------------------------------------------------------------------------ | ||
| // getters | ||
| // ------------------------------------------------------------------------ | ||
|
|
||
| public Configuration getConfiguration() { | ||
| // update the memory in case that we've changed the number of components (TM, RM, JM) | ||
| long memory = calculateManagedMemoryPerTaskManager(); |
There was a problem hiding this comment.
Getters should usually not perform any calculation. How about changing the method name to updateConfiguration()?
There was a problem hiding this comment.
After this method has been called, you can't change the memory configuration anymore because the config value will prevent new calculation in calculateManagedMemoryPerTaskManager. Is that desired?
There was a problem hiding this comment.
True. I guess we should create a new configuration every time this method is called. Will rename the method to then togenerateConfiguration.
|
Thanks for the review @mxm. I will address your comments and if Travis gives green light, I will merge the PR. |
Allowing the RpcEndpoint.start/shutDown to throw exceptions will help to let rpc endpoints to quickly fail without having to use a callback like the FatalErrorHandler. This closes apache#2651.
This PR introduces a FatalErrorHandler and the MetricRegistry to the RM. The FatalErrorHandler is used to handle fatal errors. Additionally, the PR adds the MetricRegistry to the RM which can be used to register metrics. Apart from these changes the PR restructures the code of the RM a little bit and fixes some blocking operations. The PR also moves the TestingFatalErrorHandler into the util package of flink-runtime test. That it is usable across multiple tests. Introduce ResourceManagerRunner to handle errors in the ResourceManager This closes apache#2655.
…anager Introduce the JobLeaderIdService which automatically retrieves the current job leader id. This job leader id is used to validate job manager registartion attempts. Additionally, it is used to disconnect old job leaders from the resource manager. Add comments This closes apache#2657.
…o MiniCluster If the managed memory size for the task manager has not been set in the Configuration, then it is automatically calculated by dividing the available memory by the number of distributed components. Additionally this PR allows to provide a MetricRegistry to the TaskManagerRunner. That way it is possible to use the MiniCluster's MetricRegistry. Add memory calculation for task managers This closes apache#2669.
tillrohrmann
force-pushed
the
miniClusterUpdate
branch
from
4918b63
to
f2cdd2f
Compare
asfgit
pushed a commit
that referenced
this pull request
Oct 23, 2016
…o MiniCluster If the managed memory size for the task manager has not been set in the Configuration, then it is automatically calculated by dividing the available memory by the number of distributed components. Additionally this PR allows to provide a MetricRegistry to the TaskManagerRunner. That way it is possible to use the MiniCluster's MetricRegistry. Add memory calculation for task managers This closes #2669.
|
Merged |
tillrohrmann
added a commit
to tillrohrmann/flink
that referenced
this pull request
Oct 31, 2016
…o MiniCluster If the managed memory size for the task manager has not been set in the Configuration, then it is automatically calculated by dividing the available memory by the number of distributed components. Additionally this PR allows to provide a MetricRegistry to the TaskManagerRunner. That way it is possible to use the MiniCluster's MetricRegistry. Add memory calculation for task managers This closes apache#2669.
tillrohrmann
added a commit
to tillrohrmann/flink
that referenced
this pull request
Oct 31, 2016
…o MiniCluster If the managed memory size for the task manager has not been set in the Configuration, then it is automatically calculated by dividing the available memory by the number of distributed components. Additionally this PR allows to provide a MetricRegistry to the TaskManagerRunner. That way it is possible to use the MiniCluster's MetricRegistry. Add memory calculation for task managers This closes apache#2669.
asfgit
pushed a commit
that referenced
this pull request
Nov 1, 2016
…o MiniCluster If the managed memory size for the task manager has not been set in the Configuration, then it is automatically calculated by dividing the available memory by the number of distributed components. Additionally this PR allows to provide a MetricRegistry to the TaskManagerRunner. That way it is possible to use the MiniCluster's MetricRegistry. Add memory calculation for task managers This closes #2669.
StephanEwen
pushed a commit
to StephanEwen/flink
that referenced
this pull request
Dec 23, 2016
…o MiniCluster If the managed memory size for the task manager has not been set in the Configuration, then it is automatically calculated by dividing the available memory by the number of distributed components. Additionally this PR allows to provide a MetricRegistry to the TaskManagerRunner. That way it is possible to use the MiniCluster's MetricRegistry. Add memory calculation for task managers This closes apache#2669.
StephanEwen
pushed a commit
to StephanEwen/flink
that referenced
this pull request
Dec 23, 2016
…o MiniCluster If the managed memory size for the task manager has not been set in the Configuration, then it is automatically calculated by dividing the available memory by the number of distributed components. Additionally this PR allows to provide a MetricRegistry to the TaskManagerRunner. That way it is possible to use the MiniCluster's MetricRegistry. Add memory calculation for task managers This closes apache#2669.
StephanEwen
pushed a commit
to StephanEwen/flink
that referenced
this pull request
Dec 23, 2016
…o MiniCluster If the managed memory size for the task manager has not been set in the Configuration, then it is automatically calculated by dividing the available memory by the number of distributed components. Additionally this PR allows to provide a MetricRegistry to the TaskManagerRunner. That way it is possible to use the MiniCluster's MetricRegistry. Add memory calculation for task managers This closes apache#2669.
liuyuzhong7
pushed a commit
to liuyuzhong7/flink
that referenced
this pull request
Jan 4, 2017
…o MiniCluster If the managed memory size for the task manager has not been set in the Configuration, then it is automatically calculated by dividing the available memory by the number of distributed components. Additionally this PR allows to provide a MetricRegistry to the TaskManagerRunner. That way it is possible to use the MiniCluster's MetricRegistry. Add memory calculation for task managers This closes apache#2669.
liuyuzhong7
pushed a commit
to liuyuzhong7/flink
that referenced
this pull request
Jan 17, 2017
…o MiniCluster If the managed memory size for the task manager has not been set in the Configuration, then it is automatically calculated by dividing the available memory by the number of distributed components. Additionally this PR allows to provide a MetricRegistry to the TaskManagerRunner. That way it is possible to use the MiniCluster's MetricRegistry. Add memory calculation for task managers This closes apache#2669.
joseprupi
pushed a commit
to joseprupi/flink
that referenced
this pull request
Feb 12, 2017
…o MiniCluster If the managed memory size for the task manager has not been set in the Configuration, then it is automatically calculated by dividing the available memory by the number of distributed components. Additionally this PR allows to provide a MetricRegistry to the TaskManagerRunner. That way it is possible to use the MiniCluster's MetricRegistry. Add memory calculation for task managers This closes apache#2669.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR is based on #2651, #2655 and #2657. Only 4918b63 is relevant.
If the managed memory size for the task manager has not been set in the Configuration, then
it is automatically calculated by dividing the available memory by the number of distributed
components. Additionally this PR allows to provide a MetricRegistry to the TaskManagerRunner.
That way it is possible to use the MiniCluster's MetricRegistry.
Add memory calculation for task managers.