Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix barrier execution mode with repartition for spark standalone #651

Merged

Conversation

@imatiach-msft
Copy link
Contributor

commented Aug 19, 2019

No description provided.

@imatiach-msft

This comment has been minimized.

Copy link
Contributor Author

commented Aug 19, 2019

/azp run

@azure-pipelines

This comment has been minimized.

Copy link

commented Aug 19, 2019

Azure Pipelines successfully started running 1 pipeline(s).
@codecov

This comment has been minimized.

Copy link

commented Aug 19, 2019

Codecov Report

Merging #651 into master will decrease coverage by 1.66%.
The diff coverage is 92.3%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master     #651      +/-   ##
==========================================
- Coverage   79.81%   78.14%   -1.67%     
==========================================
  Files         225      225              
  Lines        9025     9034       +9     
  Branches      456      490      +34     
==========================================
- Hits         7203     7060     -143     
- Misses       1822     1974     +152
Impacted Files Coverage Δ
...com/microsoft/ml/spark/lightgbm/LightGBMBase.scala 84.31% <92.3%> (+0.59%) ⬆️
...microsoft/ml/spark/automl/DefaultHyperparams.scala 0% <0%> (-100%) ⬇️
.../com/microsoft/ml/spark/automl/FindBestModel.scala 15.38% <0%> (-70.77%) ⬇️
...icrosoft/ml/spark/automl/TuneHyperparameters.scala 11.42% <0%> (-62.86%) ⬇️
...ala/com/microsoft/ml/spark/automl/ParamSpace.scala 0% <0%> (-60%) ⬇️
...om/microsoft/ml/spark/train/AutoTrainedModel.scala 0% <0%> (-50%) ⬇️
.../org/apache/spark/injections/RegressionUtils.scala 0% <0%> (-50%) ⬇️
...om/microsoft/ml/spark/automl/EvaluationUtils.scala 4% <0%> (-44%) ⬇️
.../microsoft/ml/spark/automl/HyperparamBuilder.scala 4.76% <0%> (-42.86%) ⬇️
...rosoft/ml/spark/train/ComputeModelStatistics.scala 77.56% <0%> (-8.3%) ⬇️
... and 5 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 1e186ad...636a2e2. Read the comment docs.

@imatiach-msft imatiach-msft force-pushed the imatiach-msft:ilmat/fix-rdd-unsupported-ex branch 2 times, most recently from ecaea96 to c7041c2 Aug 22, 2019

@imatiach-msft imatiach-msft changed the title [WIP] remove coalesce on barrier execution fix barrier execution mode with repartition for spark standalone Aug 22, 2019

@imatiach-msft

This comment has been minimized.

Copy link
Contributor Author

commented Aug 22, 2019

/azp run

@azure-pipelines

This comment has been minimized.

Copy link

commented Aug 22, 2019

Azure Pipelines successfully started running 1 pipeline(s).
@drdarshan
Copy link
Contributor

left a comment

The code change itself looks good, just left a minor comment in the Scala style override.

@imatiach-msft

This comment has been minimized.

Copy link
Contributor Author

commented Aug 23, 2019

@drdarshan replied to comment

@imatiach-msft imatiach-msft force-pushed the imatiach-msft:ilmat/fix-rdd-unsupported-ex branch from c7041c2 to 636a2e2 Aug 26, 2019

@imatiach-msft

This comment has been minimized.

Copy link
Contributor Author

commented Aug 26, 2019

/azp run

@azure-pipelines

This comment has been minimized.

Copy link

commented Aug 26, 2019

Azure Pipelines successfully started running 1 pipeline(s).
@imatiach-msft

This comment has been minimized.

Copy link
Contributor Author

commented Aug 26, 2019

/azp run

@azure-pipelines

This comment has been minimized.

Copy link

commented Aug 26, 2019

Azure Pipelines successfully started running 1 pipeline(s).

@mhamilton723 mhamilton723 merged commit 9805996 into Azure:master Aug 26, 2019

26 of 27 checks passed

Semantic Pull Request add a semantic commit or PR title
Details
Azure.mmlspark Build #20190826.3 succeeded
Details
Azure.mmlspark (PublishAndE2E) PublishAndE2E succeeded
Details
Azure.mmlspark (PublishDocker) PublishDocker succeeded
Details
Azure.mmlspark (PythonTests) PythonTests succeeded
Details
Azure.mmlspark (Release) Release succeeded
Details
Azure.mmlspark (Style) Style succeeded
Details
Azure.mmlspark (UnitTests automl) UnitTests automl succeeded
Details
Azure.mmlspark (UnitTests cntk) UnitTests cntk succeeded
Details
Azure.mmlspark (UnitTests cognitive) UnitTests cognitive succeeded
Details
Azure.mmlspark (UnitTests core) UnitTests core succeeded
Details
Azure.mmlspark (UnitTests downloader) UnitTests downloader succeeded
Details
Azure.mmlspark (UnitTests featurize) UnitTests featurize succeeded
Details
Azure.mmlspark (UnitTests flaky) UnitTests flaky succeeded
Details
Azure.mmlspark (UnitTests image) UnitTests image succeeded
Details
Azure.mmlspark (UnitTests io) UnitTests io succeeded
Details
Azure.mmlspark (UnitTests lightgbm1) UnitTests lightgbm1 succeeded
Details
Azure.mmlspark (UnitTests lightgbm2) UnitTests lightgbm2 succeeded
Details
Azure.mmlspark (UnitTests lime) UnitTests lime succeeded
Details
Azure.mmlspark (UnitTests opencv) UnitTests opencv succeeded
Details
Azure.mmlspark (UnitTests recommendation) UnitTests recommendation succeeded
Details
Azure.mmlspark (UnitTests stages) UnitTests stages succeeded
Details
Azure.mmlspark (UnitTests train) UnitTests train succeeded
Details
Azure.mmlspark (UnitTests vw) UnitTests vw succeeded
Details
Codacy/PR Quality Review Up to standards. A positive pull request.
Details
WIP Ready for review
Details
license/cla All CLA requirements met.
Details
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
3 participants
You can’t perform that action at this time.