[REEF-1900] Add option for number of cores for the Driver in REEF Java #1388
Conversation
This adds all the plumbing to allow for the configuration of the Driver's CPU cores in the REEF Java implementation. Notes: * Only the YARN and HDInsight runtimes read the new field. JIRA: [REEF-1900](https://issues.apache.org/jira/browse/REEF-1900) Pull Request: This closes #
|
Tests pass locally, but I have not tested this on YARN. |
|
LGTM Sergiy and I will test it on Yarn. I am requesting access to the CISL cluster. I will commit if it passes. |
|
Local tests pass. Waiting for access to CISL cluster for YARN testing. |
| * Number of cores assigned to the Driver Container. | ||
| */ | ||
| @NamedParameter(doc = "Number of CPU Cores allocated to the Driver.", default_value = "1") | ||
| public final class DriverCPUCores implements Name<Integer> { |
There was a problem hiding this comment.
VirtualCores or CPUCores which one is better?
There was a problem hiding this comment.
I went with the generic CPUCores, as this API is for more than YARN.
| @@ -136,12 +139,11 @@ public YarnSubmissionHelper setApplicationName(final String applicationName) { | |||
| } | |||
|
|
|||
| /** | |||
| * Set the amount of memory to be allocated to the Driver. | |||
| * @param megabytes | |||
| * Set the resources for the Driver. | |||
There was a problem hiding this comment.
Set memory and number of cores to be allocated to the driver.
There was a problem hiding this comment.
Thanks, I addressed this.
|
I have done a pass. It looks good. Should have a test to set the core or hello to show example and test the code pass. |
How would we test this? If we run on YARN, we'd have to restrict the tests to clusters with large enough machines, right? |
|
@jwang98052 I addressed your comments, Please have another look. |
|
Can we set it to event 2 in Hello and run it as a test? |
We can. But what do we do if the cluster doesn't have 2 core containers? Also, I don't know how to inquire how many cores we actually got to make sure the test passes. Any idea how to do that? |
|
I will add it into yarn test once .Net change is made. Other than it, LGTM. I will test and merge. |
This adds all the plumbing to allow for the configuration of the Driver's CPU cores in the REEF Java implementation.
Notes:
JIRA:
REEF-1900