fix: spark.executor.cores' default value based on master when counting workers

[ocworld]

This pr is fixing spark.executor.cores' default value based on master when counting workers.

The default value of spark.executor.cores is different as what master is.

It is 1 in YARN mode. In other case, It is all the available cores on the worker in standalone and Mesos coarse-grained modes.
(https://spark.apache.org/docs/latest/configuration.html)

Currently, I've tried to use mmlspark based on spark on k8s. When I did not set spark.executor.cores, error was occured. It is because not matched expected num of mmlspark workers with actual value.

[imatiach-msft]

/azp run

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[imatiach-msft]

will need to test this out a bit since there have been a lot of issues in the past around this specific code section

[codecov]

Codecov Report

Merging #855 into master will increase coverage by 0.66%.
The diff coverage is 77.14%.

@@            Coverage Diff             @@
##           master     #855      +/-   ##
==========================================
+ Coverage   84.56%   85.22%   +0.66%     
==========================================
  Files         189      189              
  Lines        8709     8733      +24     
  Branches      544      543       -1     
==========================================
+ Hits         7365     7443      +78     
+ Misses       1344     1290      -54     

Impacted Files	Coverage Δ
...om/microsoft/ml/spark/core/utils/ClusterUtil.scala	69.44% <77.14%> (+2.77%)	⬆️
...osoft/ml/spark/io/http/PartitionConsolidator.scala	93.33% <0.00%> (-2.23%)	⬇️
...a/com/microsoft/ml/spark/io/image/ImageUtils.scala	88.23% <0.00%> (+9.80%)	⬆️
...rosoft/ml/spark/core/schema/BinaryFileSchema.scala	100.00% <0.00%> (+12.50%)	⬆️
...icrosoft/ml/spark/io/binary/BinaryFileFormat.scala	97.72% <0.00%> (+14.77%)	⬆️
...icrosoft/ml/spark/io/binary/BinaryFileReader.scala	73.91% <0.00%> (+17.39%)	⬆️
...spark/ml/source/image/PatchedImageFileFormat.scala	88.88% <0.00%> (+40.74%)	⬆️
.../microsoft/ml/spark/core/env/StreamUtilities.scala	85.18% <0.00%> (+59.25%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 4ae0fe8...aff36e6. Read the comment docs.

[imatiach-msft]

/azp run

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[imatiach-msft]

/azp run

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[imatiach-msft]

the build was all green but it's not updating in github, weird, looks like some github bug

[ocworld]

@imatiach-msft I wrote new unittest and pushed. However, It is hard to tested mmlspark's unittest in my local environment . Please check it :)

[imatiach-msft]

/azp run

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[ocworld]

@imatiach-msft Can I see why the build failed?

[imatiach-msft]

/azp run

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[imatiach-msft]

/azp run

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[imatiach-msft]

@ocworld the style checker is failing on:

[error] /home/vsts/work/1/s/src/test/scala/com/microsoft/ml/spark/core/utils/VerifyClusterUtil.scala: Header does not match expected text

It looks like you need to add this copyright message at the top of the file VerifyClusterUtil.scala to pass the style checker:

// Copyright (C) Microsoft Corporation. All rights reserved.
// Licensed under the MIT License. See LICENSE in project root for information.

[imatiach-msft]

@ocworld I tried this PR out on a cluster (azure databricks) with Higgs dataset and it didn't change the training time for me (compared to previous runs), so it seems safe to me at least in that scenario. I think they use spark standalone mode for databricks clusters.

[ocworld]

@ocworld the style checker is failing on:

[error] /home/vsts/work/1/s/src/test/scala/com/microsoft/ml/spark/core/utils/VerifyClusterUtil.scala: Header does not match expected text

It looks like you need to add this copyright message at the top of the file VerifyClusterUtil.scala to pass the style checker:

// Copyright (C) Microsoft Corporation. All rights reserved.
// Licensed under the MIT License. See LICENSE in project root for information.

@imatiach-msft Copyright is added in the file

[ocworld]

@ocworld I tried this PR out on a cluster (azure databricks) with Higgs dataset and it didn't change the training time for me (compared to previous runs), so it seems safe to me at least in that scenario. I think they use spark standalone mode for databricks clusters.

@imatiach-msft Thanks for your test

[imatiach-msft]

/azp run

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[imatiach-msft]

it looks like @mhamilton723 is required to review as a code owner (can't seem to merge), can you please review this PR @mhamilton723 ?

[ocworld]

@AhnLab-OSS