[SPARK-10514] [Mesos] waiting for min no of total cores acquired by Spark by implementing the sufficientResourcesRegistered method #8672
Conversation
…park by implementing the sufficientResourcesRegistered method
|
You need to update documentation as well. It's not clear to me how you determine the total number of expected resources. Is that |
|
@dragos This change is for Coarse grained mode hence "spark.cores.max" is used to determine total no of resource. The documentation is already in place. You can see it in the image.
|
|
Sorry, got your point. I will update the Documentation. |
|
Hm, I definitely don't see Mesos in http://spark.apache.org/docs/latest/configuration.html (only Yarn). That's what I meant. Where is the screenshot from? |
…egisteredResourcesRatio parameter
|
@dragos I have updated the Documentation. Feel free to give any feedback. |
|
@tnachen [Friendly Reminder] Can you please take a look to the patch? |
|
Looks good to me |
|
ok to test |
|
OK, LGTM. Although there was some discussion to remove this back then (I can't find the PR now) it was inconclusive and never merged. This patch just makes it aligned with the docs so that's a strict improvement. |
| @@ -1109,10 +1109,11 @@ Apart from these, the following properties are also available, and may be useful | |||
| </tr> | |||
| <tr> | |||
| <td><code>spark.scheduler.minRegisteredResourcesRatio</code></td> | |||
| <td>0.8 for YARN mode; 0.0 otherwise</td> | |||
| <td>0.8 for YARN mode; 0.0 for standalone mode and Mesos Coarse grained mode</td> | |||
There was a problem hiding this comment.
coarse-grained, I'll fix this on merge
|
Test build #42283 has finished for PR 8672 at commit
|
|
Thanks @andrewor14. I will close the issue in JIRA. |
…park by implementing the sufficientResourcesRegistered method spark.scheduler.minRegisteredResourcesRatio configuration parameter works for YARN mode but not for Mesos Coarse grained mode. If the parameter specified default value of 0 will be set for spark.scheduler.minRegisteredResourcesRatio in base class and this method will always return true. There are no existing test for YARN mode too. Hence not added test for the same. Author: Akash Mishra <akash.mishra20@gmail.com> Closes apache#8672 from SleepyThread/master.
spark.scheduler.minRegisteredResourcesRatio configuration parameter works for YARN mode but not for Mesos Coarse grained mode.
If the parameter specified default value of 0 will be set for spark.scheduler.minRegisteredResourcesRatio in base class and this method will always return true.
There are no existing test for YARN mode too. Hence not added test for the same.