[SPARK-41589][PYTHON][ML] PyTorch Distributor Baseline API Changes#39146
Closed
rithwik-db wants to merge 2 commits intoapache:masterfrom
Closed
[SPARK-41589][PYTHON][ML] PyTorch Distributor Baseline API Changes#39146rithwik-db wants to merge 2 commits intoapache:masterfrom
rithwik-db wants to merge 2 commits intoapache:masterfrom
Conversation
rithwik-db
commented
Dec 21, 2022
rithwik-db
commented
Dec 21, 2022
rithwik-db
commented
Dec 21, 2022
HyukjinKwon
reviewed
Dec 21, 2022
HyukjinKwon
reviewed
Dec 21, 2022
HyukjinKwon
reviewed
Dec 21, 2022
HyukjinKwon
reviewed
Dec 21, 2022
|
Can one of the admins verify this patch? |
rithwik-db
force-pushed
the
baseline-api
branch
from
55750cd to
7c51eff
Compare
HyukjinKwon
reviewed
Dec 23, 2022
HyukjinKwon
reviewed
Dec 23, 2022
HyukjinKwon
reviewed
Dec 23, 2022
HyukjinKwon
reviewed
Dec 23, 2022
HyukjinKwon
reviewed
Dec 23, 2022
HyukjinKwon
reviewed
Dec 23, 2022
Member
|
cc @WeichenXu123 and @mengxr |
rithwik-db
force-pushed
the
baseline-api
branch
2 times, most recently
from
814820c to
43c2213
Compare
rithwik-db
force-pushed
the
baseline-api
branch
from
43c2213 to
0153c6c
Compare
Contributor
|
@rithwik-db could you please fix the python lint? |
HyukjinKwon
reviewed
Jan 6, 2023
HyukjinKwon
reviewed
Jan 6, 2023
HyukjinKwon
reviewed
Jan 6, 2023
HyukjinKwon
reviewed
Jan 6, 2023
HyukjinKwon
reviewed
Jan 6, 2023
HyukjinKwon
reviewed
Jan 6, 2023
rithwik-db
force-pushed
the
baseline-api
branch
2 times, most recently
from
518438f to
3f57b2d
Compare
lu-wang-dl
reviewed
Jan 6, 2023
Contributor
There was a problem hiding this comment.
Why we need to add F403? I saw other test files only include F401.
Contributor
Author
There was a problem hiding this comment.
mypy raises errors otherwise
lu-wang-dl
reviewed
Jan 6, 2023
Contributor
There was a problem hiding this comment.
And also not seeing # type: ignore in other test files.
Contributor
Author
There was a problem hiding this comment.
mypy raises errors otherwise
rithwik-db
force-pushed
the
baseline-api
branch
from
9c28473 to
5c434f9
Compare
WeichenXu123
approved these changes
Jan 9, 2023
rithwik-db
changed the title
rithwik-db
force-pushed
the
baseline-api
branch
from
5c434f9 to
f78f91c
Compare
HyukjinKwon
approved these changes
Jan 10, 2023
Member
|
Test failures are not related to this PR. Merged to master. |
HyukjinKwon
added a commit
that referenced
this pull request
Jan 11, 2023
…etup.py ### What changes were proposed in this pull request? This PR is a followup of #39146 that adds `pyspark.ml.torch` to `setup.py`. ### Why are the changes needed? In order for PyPI users to be able to use `pyspark.ml.torch` package. ### Does this PR introduce _any_ user-facing change? No, the main change has not been released yet. It adds the package into PyPI-packaged PySpark. ### How was this patch tested? CI in pip packaging test should check the change. Closes #39490 from HyukjinKwon/SPARK-41589-followup. Authored-by: Hyukjin Kwon <gurwls223@apache.org> Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>
HyukjinKwon
pushed a commit
that referenced
this pull request
Jan 11, 2023
… GPU ### What changes were proposed in this pull request? This is an addition to #39146 to add support for single node training using PyTorch files. The users would follow the second workflow in the [design document](https://docs.google.com/document/d/1QPO1Ly8WteL6aIPvVcR7Xne9qVtJiB3fdrRn7NwBcpA/edit#heading=h.8yvw9xq428fh) to run training. I added some new utility functions as well as built on top of current functions. ### Why are the changes needed? Look at the [main ticket](https://issues.apache.org/jira/browse/SPARK-41589) for more details. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Some unit tests were added and integration tests will be added in a later PR (https://issues.apache.org/jira/browse/SPARK-41777). Closes #39188 from rithwik-db/pytorch-file-local-training. Authored-by: Rithwik Ediga Lakhamsani <rithwik.ediga@databricks.com> Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
6 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Just creating a small PR to start progress on the Spark-PyTorch Distributor. This is a WIP project and I left questions and comments to discuss how I will be approaching certain aspects of the code.
What changes were proposed in this pull request?
This just proposes the baseline API for how users will interact with the Spark PyTorch distributor (Design Document).
Why are the changes needed?
The design document's background section goes into more detail about the why.
Does this PR introduce any user-facing change?
Yes, this proposes an API for how users will interact with the PyTorch Distributor. The user workflow is also proposed in that design document.
How was this patch tested?
I just added some basic tests. These will need to be improved to correctly match the style that PySpark requires.