Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Train] Improve Lightning and Train API Documentations #38276

Merged
merged 57 commits into from
Aug 11, 2023
Merged

[Train] Improve Lightning and Train API Documentations #38276

merged 57 commits into from
Aug 11, 2023

Conversation

woshiyyya
Copy link
Member

@woshiyyya woshiyyya commented Aug 9, 2023 •
edited

Why are these changes needed?

Rendered doc: https://anyscale-ray--38276.com.readthedocs.build/en/38276/

From user study, we discovered multiple issues around current documentation. This PR makes the following changes:

  • Update Ray Train API page
  • standardize def train_func(config) in the user guide
  • set accelerate="auto" in lightning user guide
  • Improve lightning migration guides
  • Add framework build-in data utilities section in "Data Loading and Preprocessing" section

Related issue number

Checks

  • I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
  • I've run scripts/format.sh to lint the changes in this PR.
  • I've included any doc changes needed for https://docs.ray.io/en/master/.
    • I've added any new APIs to the API Reference. For example, if I added a
      method in Tune, I've added it in doc/source/tune/api/ under the
      corresponding .rst file.
  • I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
  • Testing Strategy
    • Unit tests
    • Release tests
    • This PR is not tested :(

woshiyyya and others added 30 commits August 1, 2023 13:49
@woshiyyya
init
3a7dc6e 
Signed-off-by: woshiyyya <xiaoyunxuan1998@gmail.com>
@woshiyyya
Merge remote-tracking branch 'upstream/master' into train/unified-api…
c7b91a7 
…/add_lightning_utilities
@woshiyyya
add docstring
4ef88c9 
Signed-off-by: woshiyyya <xiaoyunxuan1998@gmail.com>
@woshiyyya @matthewdeng
Apply suggestions from code review
7b5da30 
Co-authored-by: matthewdeng <matthew.j.deng@gmail.com>
Signed-off-by: Yunxuan Xiao <xiaoyunxuan1998@gmail.com>
@woshiyyya
add deepspeed tests
95e2b9b 
Signed-off-by: woshiyyya <xiaoyunxuan1998@gmail.com>
@woshiyyya
fix import error
4df1ec6 
Signed-off-by: woshiyyya <xiaoyunxuan1998@gmail.com>
@woshiyyya
Merge remote-tracking branch 'upstream/master' into train/unified-api…
3914a21 
…/add_lightning_utilities
@woshiyyya
make report callback public
21a8ce3 
Signed-off-by: woshiyyya <xiaoyunxuan1998@gmail.com>
@woshiyyya
fix lint
abdf842 
Signed-off-by: woshiyyya <xiaoyunxuan1998@gmail.com>
@woshiyyya
Merge remote-tracking branch 'upstream/master' into train/unified-api…
d3b01f0 
…/add_lightning_utilities
@woshiyyya
use new ray.train api
0ecd61a 
Signed-off-by: woshiyyya <xiaoyunxuan1998@gmail.com>
@woshiyyya
switch to new api in LightningTrainer
0de4bc4 
Signed-off-by: woshiyyya <xiaoyunxuan1998@gmail.com>
@woshiyyya
Merge remote-tracking branch 'upstream/master' into train/unified-api…
2b2589e 
…/add_lightning_utilities
@woshiyyya
WIP: add lightning user guides
7c1b1e2 
Signed-off-by: woshiyyya <xiaoyunxuan1998@gmail.com>
@woshiyyya
change default ckpt name
e8285ab 
Signed-off-by: woshiyyya <xiaoyunxuan1998@gmail.com>
@woshiyyya
add migration guides and api
6c326a0 
Signed-off-by: woshiyyya <xiaoyunxuan1998@gmail.com>
@woshiyyya @matthewdeng
Apply suggestions from code review
2f9f5a4 
Co-authored-by: matthewdeng <matthew.j.deng@gmail.com>
Signed-off-by: Yunxuan Xiao <xiaoyunxuan1998@gmail.com>
@woshiyyya @matthewdeng
Update doc/source/train/distributed-pytorch/migration-guides.rst
4f1412f 
Co-authored-by: matthewdeng <matthew.j.deng@gmail.com>
Signed-off-by: Yunxuan Xiao <xiaoyunxuan1998@gmail.com>
@woshiyyya
address comments
c9f3c03 
Signed-off-by: woshiyyya <xiaoyunxuan1998@gmail.com>
@woshiyyya
change the resume training snippet
8264549 
Signed-off-by: woshiyyya <xiaoyunxuan1998@gmail.com>
@woshiyyya
add mnist example
ba81c15 
Signed-off-by: woshiyyya <xiaoyunxuan1998@gmail.com>
@woshiyyya
fix doc lint
89c7388 
Signed-off-by: woshiyyya <xiaoyunxuan1998@gmail.com>
@woshiyyya
Merge branch 'master' into train/unified-api/add_lightning_utilities
0d9bdaf
@woshiyyya
fix
c186f5b 
Signed-off-by: woshiyyya <xiaoyunxuan1998@gmail.com>
@woshiyyya
fixing
6925a05 
Signed-off-by: woshiyyya <xiaoyunxuan1998@gmail.com>
@woshiyyya
Merge remote-tracking branch 'upstream/master' into train/unified-api…
079ddec 
…/add_lightning_utilities
@woshiyyya
fix ut
7c19ffe 
Signed-off-by: woshiyyya <xiaoyunxuan1998@gmail.com>
@woshiyyya
Merge remote-tracking branch 'upstream/master' into train/unified-api…
8a65e36 
…/add_lightning_utilities
@woshiyyya
update semgrep
9d9ed17 
Signed-off-by: woshiyyya <xiaoyunxuan1998@gmail.com>
@woshiyyya @matthewdeng
Apply suggestions from code review
87aa763 
Co-authored-by: matthewdeng <matthew.j.deng@gmail.com>
Signed-off-by: Yunxuan Xiao <xiaoyunxuan1998@gmail.com>
@woshiyyya
fix
02c224d 
Signed-off-by: woshiyyya <xiaoyunxuan1998@gmail.com>
@woshiyyya
add links to new api
d35419a 
Signed-off-by: woshiyyya <xiaoyunxuan1998@gmail.com>
matthewdeng
matthewdeng reviewed Aug 11, 2023
View reviewed changes
Copy link
Contributor

@matthewdeng matthewdeng left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall looks good :)

doc/source/ray-air/api/configs.rst Outdated Show resolved Hide resolved
doc/source/train/api/api.rst Outdated Show resolved Hide resolved
doc/source/train/distributed-pytorch/data-loading-preprocessing.rst Outdated Show resolved Hide resolved
woshiyyya and others added 7 commits August 10, 2023 17:30
@woshiyyya @matthewdeng
Update doc/source/train/distributed-pytorch/data-loading-preprocessin…
2e03a01 
…g.rst

Co-authored-by: matthewdeng <matthew.j.deng@gmail.com>
Signed-off-by: Yunxuan Xiao <xiaoyunxuan1998@gmail.com>
@woshiyyya @matthewdeng
Update doc/source/ray-air/api/configs.rst
c07bab9 
Co-authored-by: matthewdeng <matthew.j.deng@gmail.com>
Signed-off-by: Yunxuan Xiao <xiaoyunxuan1998@gmail.com>
@woshiyyya
remove dup block
996787a 
Signed-off-by: woshiyyya <xiaoyunxuan1998@gmail.com>
@woshiyyya
Merge remote-tracking branch 'upstream/master' into train/improve_lig…
e57acfa 
…htning_doc
@woshiyyya
update getting start page
7ab7aca 
Signed-off-by: woshiyyya <xiaoyunxuan1998@gmail.com>
@woshiyyya
fix wording
002ccf3 
Signed-off-by: woshiyyya <xiaoyunxuan1998@gmail.com>
@woshiyyya
refract
9f9bfcf 
Signed-off-by: woshiyyya <xiaoyunxuan1998@gmail.com>
@woshiyyya
remove old link
c0d3f68 
Signed-off-by: woshiyyya <xiaoyunxuan1998@gmail.com>
@woshiyyya
rm migration guides and converting page
bf6c079 
Signed-off-by: woshiyyya <xiaoyunxuan1998@gmail.com>
@woshiyyya woshiyyya added the tests-ok The tagger certifies test failures are unrelated and assumes personal liability. label Aug 11, 2023
c21
c21 approved these changes Aug 11, 2023
View reviewed changes
Copy link
Contributor

@c21 c21 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Accept from Data side.

@matthewdeng matthewdeng merged commit 5ac2110 into ray-project:master Aug 11, 2023
112 of 117 checks passed
NripeshN pushed a commit to NripeshN/ray that referenced this pull request Aug 15, 2023
@woshiyyya @matthewdeng @NripeshN
[Train] Improve Lightning and Train API Documentations (ray-project#3…
e431136 
…8276)

Signed-off-by: woshiyyya <xiaoyunxuan1998@gmail.com>
Signed-off-by: Yunxuan Xiao <xiaoyunxuan1998@gmail.com>
Signed-off-by: Yunxuan Xiao <yunxuanx@anyscale.com>
Co-authored-by: matthewdeng <matthew.j.deng@gmail.com>
Signed-off-by: NripeshN <nn2012@hw.ac.uk>
harborn pushed a commit to harborn/ray that referenced this pull request Aug 17, 2023
@woshiyyya @matthewdeng @harborn
[Train] Improve Lightning and Train API Documentations (ray-project#3…
d6cad68 
…8276)

Signed-off-by: woshiyyya <xiaoyunxuan1998@gmail.com>
Signed-off-by: Yunxuan Xiao <xiaoyunxuan1998@gmail.com>
Signed-off-by: Yunxuan Xiao <yunxuanx@anyscale.com>
Co-authored-by: matthewdeng <matthew.j.deng@gmail.com>
Signed-off-by: harborn <gangsheng.wu@intel.com>
harborn pushed a commit to harborn/ray that referenced this pull request Aug 17, 2023
@woshiyyya @matthewdeng @harborn
[Train] Improve Lightning and Train API Documentations (ray-project#3…
ec3f878 
…8276)



Signed-off-by: woshiyyya <xiaoyunxuan1998@gmail.com>
Signed-off-by: Yunxuan Xiao <xiaoyunxuan1998@gmail.com>
Signed-off-by: Yunxuan Xiao <yunxuanx@anyscale.com>
Co-authored-by: matthewdeng <matthew.j.deng@gmail.com>
arvind-chandra pushed a commit to lmco/ray that referenced this pull request Aug 31, 2023
@woshiyyya @matthewdeng @arvind-chandra
[Train] Improve Lightning and Train API Documentations (ray-project#3…
3104dc5 
…8276)

Signed-off-by: woshiyyya <xiaoyunxuan1998@gmail.com>
Signed-off-by: Yunxuan Xiao <xiaoyunxuan1998@gmail.com>
Signed-off-by: Yunxuan Xiao <yunxuanx@anyscale.com>
Co-authored-by: matthewdeng <matthew.j.deng@gmail.com>
Signed-off-by: e428265 <arvind.chandramouli@lmco.com>
vymao pushed a commit to vymao/ray that referenced this pull request Oct 11, 2023
@woshiyyya @matthewdeng
[Train] Improve Lightning and Train API Documentations (ray-project#3…
919d225 
…8276)

Signed-off-by: woshiyyya <xiaoyunxuan1998@gmail.com>
Signed-off-by: Yunxuan Xiao <xiaoyunxuan1998@gmail.com>
Signed-off-by: Yunxuan Xiao <yunxuanx@anyscale.com>
Co-authored-by: matthewdeng <matthew.j.deng@gmail.com>
Signed-off-by: Victor <vctr.y.m@example.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@matthewdeng matthewdeng matthewdeng approved these changes

@c21 c21 c21 approved these changes

@richardliaw richardliaw Awaiting requested review from richardliaw

@gjoliver gjoliver Awaiting requested review from gjoliver

@krfricke krfricke Awaiting requested review from krfricke

@xwjiang2010 xwjiang2010 Awaiting requested review from xwjiang2010

@amogkam amogkam Awaiting requested review from amogkam amogkam is a code owner

@Yard1 Yard1 Awaiting requested review from Yard1

@maxpumperla maxpumperla Awaiting requested review from maxpumperla

@ericl ericl Awaiting requested review from ericl ericl is a code owner

@scv119 scv119 Awaiting requested review from scv119 scv119 is a code owner

@scottjlee scottjlee Awaiting requested review from scottjlee scottjlee is a code owner

@bveeramani bveeramani Awaiting requested review from bveeramani bveeramani is a code owner

@raulchen raulchen Awaiting requested review from raulchen raulchen is a code owner

Assignees

@matthewdeng matthewdeng

@xwjiang2010 xwjiang2010

Labels
tests-ok The tagger certifies test failures are unrelated and assumes personal liability.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
4 participants
@woshiyyya @matthewdeng @c21 @xwjiang2010