Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: refactor to have separate dataset utils and partition processor #1089

Merged

Conversation

imatiach-msft
Copy link
Contributor

This is in preparation for PR #1066.
This PR refactors the dataset-related code into the new dataset utils file and the training-related code into the partition processor.

@imatiach-msft
Copy link
Contributor Author

/azp run

@azure-pipelines
Copy link

Azure Pipelines successfully started running 1 pipeline(s).

@codecov
Copy link

codecov bot commented Jun 16, 2021

Codecov Report

Merging #1089 (6ee13f9) into master (e8a97ed) will increase coverage by 0.04%.
The diff coverage is 86.66%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #1089      +/-   ##
==========================================
+ Coverage   84.95%   84.99%   +0.04%     
==========================================
  Files         206      208       +2     
  Lines        9776     9786      +10     
  Branches      566      564       -2     
==========================================
+ Hits         8305     8318      +13     
+ Misses       1471     1468       -3     
Impacted Files Coverage Δ
...crosoft/ml/spark/lightgbm/params/TrainParams.scala 100.00% <ø> (ø)
...a/com/microsoft/ml/spark/lightgbm/TrainUtils.scala 85.35% <75.00%> (-2.19%) ⬇️
...osoft/ml/spark/lightgbm/dataset/DatasetUtils.scala 82.96% <82.96%> (ø)
...com/microsoft/ml/spark/lightgbm/LightGBMBase.scala 92.85% <100.00%> (ø)
...crosoft/ml/spark/lightgbm/PartitionProcessor.scala 100.00% <100.00%> (ø)
.../execution/streaming/continuous/HTTPSourceV2.scala 92.80% <0.00%> (+0.71%) ⬆️
...a/com/microsoft/ml/spark/io/http/HTTPClients.scala 76.66% <0.00%> (+3.33%) ⬆️
...com/microsoft/ml/spark/core/utils/AsyncUtils.scala 70.00% <0.00%> (+5.00%) ⬆️
... and 1 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e8a97ed...6ee13f9. Read the comment docs.

@imatiach-msft imatiach-msft force-pushed the ilmat/ref-part-proc-data-utils branch from 412a17b to 0929de8 Compare June 16, 2021 20:02
@imatiach-msft
Copy link
Contributor Author

/azp run

@azure-pipelines
Copy link

Azure Pipelines successfully started running 1 pipeline(s).

@imatiach-msft imatiach-msft force-pushed the ilmat/ref-part-proc-data-utils branch from 0929de8 to 6ee13f9 Compare June 16, 2021 20:45
@imatiach-msft
Copy link
Contributor Author

/azp run

@azure-pipelines
Copy link

Azure Pipelines successfully started running 1 pipeline(s).

@imatiach-msft
Copy link
Contributor Author

/azp run

@azure-pipelines
Copy link

Azure Pipelines successfully started running 1 pipeline(s).

@imatiach-msft
Copy link
Contributor Author

/azp run

@azure-pipelines
Copy link

Azure Pipelines successfully started running 1 pipeline(s).

@imatiach-msft
Copy link
Contributor Author

/azp run

@azure-pipelines
Copy link

Azure Pipelines successfully started running 1 pipeline(s).

@imatiach-msft imatiach-msft merged commit 21d6c04 into microsoft:master Jun 17, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants