[SPARK-41775][PYTHON][ML] Adding support for PyTorch functions #39369
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Can one of the admins verify this patch? |
rithwik-db
force-pushed
the
pytorch-functions
branch
4 times, most recently
from
9becf31
to
0d169aa
Compare
|
overall good. |
rithwik-db
force-pushed
the
pytorch-functions
branch
from
0d169aa
to
c6b96a5
Compare
rithwik-db
changed the title
rithwik-db
force-pushed
the
pytorch-functions
branch
2 times, most recently
from
2d1ad9f
to
ad662be
Compare
WeichenXu123
approved these changes
Jan 20, 2023
srowen
changed the title
rithwik-db
force-pushed
the
pytorch-functions
branch
from
ad662be
to
5f10396
Compare
lu-wang-dl
reviewed
Jan 20, 2023
rithwik-db
force-pushed
the
pytorch-functions
branch
from
5f10396
to
d8715ba
Compare
lu-wang-dl
approved these changes
Jan 20, 2023
rithwik-db
force-pushed
the
pytorch-functions
branch
from
36a9a00
to
177b291
Compare
rithwik-db
force-pushed
the
pytorch-functions
branch
from
37df2c6
to
691fc3b
Compare
rithwik-db
force-pushed
the
pytorch-functions
branch
from
691fc3b
to
71f78f7
Compare
rithwik-db
force-pushed
the
pytorch-functions
branch
from
41aebc2
to
73a62f6
Compare
|
Merged to master. |
| @@ -15,16 +15,21 @@ | |||
| # limitations under the License. | |||
| # | |||
|
|
|||
| import cloudpickle # type: ignore | |||
There was a problem hiding this comment.
This should be our cloudpickle.
There was a problem hiding this comment.
Hi, @rithwik-db , @WeichenXu123 , @HyukjinKwon .
Apache Spark should not require two versions of cloudpickle at the same time.
I made a PR, #39715.
dongjoon-hyun
added a commit
that referenced
this pull request
Jan 24, 2023
…`cloudpickle` ### What changes were proposed in this pull request? This is a follow-up of #39369 which aims to use `pyspark.cloudpickle` instead of outside `cloudpickle` dependency. ### Why are the changes needed? Apache PySpark should not use two versions of `cloudpickle`. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Pass the CIs. Closes #39715 from dongjoon-hyun/SPARK-41775. Authored-by: Dongjoon Hyun <dongjoon@apache.org> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
HyukjinKwon
pushed a commit
that referenced
this pull request
Jan 25, 2023
### What changes were proposed in this pull request? This is a follow-up of #39369 which aims to fix `stderr` rerouting to `stdout` which is required for users seeing any errors that come up due to distributed training, ### Why are the changes needed? Previously, the error would be lost since it's never returned to the user. We are fixing that issue. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? N/A Closes #39724 from rithwik-db/sterr-followup. Authored-by: Rithwik Ediga Lakhamsani <rithwik.ediga@databricks.com> Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
NOTE: If you want to only view the diff from the other WIP PR regarding the baseline changes, look at the LAST COMMIT in this PR's commit history (titled "WIP adding notebook functionality"). Since I am sending out parallel PRs that are related, you should view this commit to see the diff pertaining to this ticket.
What changes were proposed in this pull request?
This is an addition to #39299 to add support for using functions as the input for distributed training. The users would follow the first workflow in the design document to run training.
Why are the changes needed?
We want to make it easier for users to run distributed training from a notebook setting.
Does this PR introduce any user-facing change?
Users can now input a training function into the
PyTorchDistributor().run(...)api. This will require a lot of additional documentation though since since we are internally using cloudpickle to run training so we need to be able to have the user's train() function be picklable.How was this patch tested?
This is a WIP PR so it hasn't been tested yet.