Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[data][train] Update "Data Loading and Preprocessing" doc #44165

Merged
merged 24 commits into from
Mar 26, 2024
Merged

[data][train] Update "Data Loading and Preprocessing" doc #44165

merged 24 commits into from
Mar 26, 2024

Conversation

raulchen
Copy link
Contributor

Why are these changes needed?

Related issue number

Checks

  • I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
  • I've run scripts/format.sh to lint the changes in this PR.
  • I've included any doc changes needed for https://docs.ray.io/en/master/.
    • I've added any new APIs to the API Reference. For example, if I added a
      method in Tune, I've added it in doc/source/tune/api/ under the
      corresponding .rst file.
  • I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
  • Testing Strategy
    • Unit tests
    • Release tests
    • This PR is not tested :(

@raulchen
refine data-loading-preprocessing doc
53eff33 
Signed-off-by: Hao Chen <chenh1024@gmail.com>
@raulchen
fix
c6604dc 
Signed-off-by: Hao Chen <chenh1024@gmail.com>
@raulchen
fix
9f77b95 
Signed-off-by: Hao Chen <chenh1024@gmail.com>
@raulchen
collate_fn
a70ab08 
Signed-off-by: Hao Chen <chenh1024@gmail.com>
@raulchen
add todo
ff7cca1 
Signed-off-by: Hao Chen <chenh1024@gmail.com>
scottjlee
scottjlee approved these changes Mar 20, 2024
View reviewed changes
Copy link
Contributor

@scottjlee scottjlee left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

pending fixing docs/premerge failures

doc/source/train/user-guides/data-loading-preprocessing.rst Outdated Show resolved Hide resolved
doc/source/train/user-guides/data-loading-preprocessing.rst Outdated Show resolved Hide resolved
doc/source/train/user-guides/data-loading-preprocessing.rst Outdated Show resolved Hide resolved
doc/source/train/user-guides/data-loading-preprocessing.rst Outdated Show resolved Hide resolved
doc/source/train/user-guides/data-loading-preprocessing.rst Outdated Show resolved Hide resolved
doc/source/train/user-guides/data-loading-preprocessing.rst Outdated Show resolved Hide resolved
doc/source/train/user-guides/data-loading-preprocessing.rst Outdated Show resolved Hide resolved
angelinalg
angelinalg approved these changes Mar 20, 2024
View reviewed changes
Copy link
Contributor

@angelinalg angelinalg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please correct any subjects that I added to convert passive voice sentences to active voice.

Also, consider using Vale to catch spelling and style errors. (go/vale)

doc/source/train/user-guides/data-loading-preprocessing.rst Outdated Show resolved Hide resolved
doc/source/train/user-guides/data-loading-preprocessing.rst Outdated Show resolved Hide resolved
doc/source/train/user-guides/data-loading-preprocessing.rst Outdated Show resolved Hide resolved
doc/source/train/user-guides/data-loading-preprocessing.rst
@@ -71,6 +72,9 @@ Data ingestion can be set up with four basic steps:
# Step 4: Access the dataset shard for the training worker via
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The indentation and step order looks wrong. This should be step 3?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

from runtime's perspective, this happens after the step 3 below

doc/source/train/user-guides/data-loading-preprocessing.rst Outdated Show resolved Hide resolved
doc/source/train/user-guides/data-loading-preprocessing.rst Outdated Show resolved Hide resolved
doc/source/train/user-guides/data-loading-preprocessing.rst Outdated Show resolved Hide resolved
doc/source/train/user-guides/data-loading-preprocessing.rst Outdated Show resolved Hide resolved
doc/source/train/user-guides/data-loading-preprocessing.rst Outdated Show resolved Hide resolved
doc/source/train/user-guides/data-loading-preprocessing.rst Outdated Show resolved Hide resolved
raulchen and others added 14 commits March 21, 2024 12:49
@raulchen @scottjlee
Update doc/source/train/user-guides/data-loading-preprocessing.rst
6610d47 
Co-authored-by: Scott Lee <scottjlee@users.noreply.github.com>
Signed-off-by: Hao Chen <chenh1024@gmail.com>
@raulchen @scottjlee
Update doc/source/train/user-guides/data-loading-preprocessing.rst
7c586fc 
Co-authored-by: Scott Lee <scottjlee@users.noreply.github.com>
Signed-off-by: Hao Chen <chenh1024@gmail.com>
@raulchen @scottjlee
Update doc/source/train/user-guides/data-loading-preprocessing.rst
b2e509e 
Co-authored-by: Scott Lee <scottjlee@users.noreply.github.com>
Signed-off-by: Hao Chen <chenh1024@gmail.com>
@raulchen @angelinalg
Update doc/source/train/user-guides/data-loading-preprocessing.rst
605e23f 
Co-authored-by: angelinalg <122562471+angelinalg@users.noreply.github.com>
Signed-off-by: Hao Chen <chenh1024@gmail.com>
@raulchen @angelinalg
Update doc/source/train/user-guides/data-loading-preprocessing.rst
5bc4b60 
Co-authored-by: angelinalg <122562471+angelinalg@users.noreply.github.com>
Signed-off-by: Hao Chen <chenh1024@gmail.com>
@raulchen @angelinalg
Update doc/source/train/user-guides/data-loading-preprocessing.rst
39892f4 
Co-authored-by: angelinalg <122562471+angelinalg@users.noreply.github.com>
Signed-off-by: Hao Chen <chenh1024@gmail.com>
@raulchen @angelinalg
Update doc/source/train/user-guides/data-loading-preprocessing.rst
6c2f715 
Co-authored-by: angelinalg <122562471+angelinalg@users.noreply.github.com>
Signed-off-by: Hao Chen <chenh1024@gmail.com>
@raulchen @angelinalg
Update doc/source/train/user-guides/data-loading-preprocessing.rst
03db162 
Co-authored-by: angelinalg <122562471+angelinalg@users.noreply.github.com>
Signed-off-by: Hao Chen <chenh1024@gmail.com>
@raulchen @angelinalg
Update doc/source/train/user-guides/data-loading-preprocessing.rst
92be6c7 
Co-authored-by: angelinalg <122562471+angelinalg@users.noreply.github.com>
Signed-off-by: Hao Chen <chenh1024@gmail.com>
@raulchen @angelinalg
Update doc/source/train/user-guides/data-loading-preprocessing.rst
0686a61 
Co-authored-by: angelinalg <122562471+angelinalg@users.noreply.github.com>
Signed-off-by: Hao Chen <chenh1024@gmail.com>
@raulchen @angelinalg
Update doc/source/train/user-guides/data-loading-preprocessing.rst
e38c00e 
Co-authored-by: angelinalg <122562471+angelinalg@users.noreply.github.com>
Signed-off-by: Hao Chen <chenh1024@gmail.com>
@raulchen @angelinalg
Update doc/source/train/user-guides/data-loading-preprocessing.rst
a5f86e9 
Co-authored-by: angelinalg <122562471+angelinalg@users.noreply.github.com>
Signed-off-by: Hao Chen <chenh1024@gmail.com>
@raulchen @angelinalg
Update doc/source/train/user-guides/data-loading-preprocessing.rst
513894d 
Co-authored-by: angelinalg <122562471+angelinalg@users.noreply.github.com>
Signed-off-by: Hao Chen <chenh1024@gmail.com>
@raulchen @angelinalg
Update doc/source/train/user-guides/data-loading-preprocessing.rst
c85fc19 
Co-authored-by: angelinalg <122562471+angelinalg@users.noreply.github.com>
Signed-off-by: Hao Chen <chenh1024@gmail.com>
raulchen and others added 4 commits March 21, 2024 14:12
@raulchen @angelinalg
Update doc/source/train/user-guides/data-loading-preprocessing.rst
922e63d 
Co-authored-by: angelinalg <122562471+angelinalg@users.noreply.github.com>
Signed-off-by: Hao Chen <chenh1024@gmail.com>
@raulchen
address comments
3541256 
Signed-off-by: Hao Chen <chenh1024@gmail.com>
@raulchen
Merge branch 'master' into ray-train-data-doc
51540cb
@raulchen
link shuffling
88e8915 
Signed-off-by: Hao Chen <chenh1024@gmail.com>
@raulchen
fix
a8d13cb 
Signed-off-by: Hao Chen <chenh1024@gmail.com>
@raulchen raulchen merged commit 3cb72ac into ray-project:master Mar 26, 2024
5 checks passed
@raulchen raulchen deleted the ray-train-data-doc branch March 26, 2024 18:39
raulchen added a commit to raulchen/ray that referenced this pull request Mar 27, 2024
@raulchen @scottjlee @angelinalg
[data][train] Update "Data Loading and Preprocessing" doc (ray-projec…
e70154b 
…t#44165)

---------

Signed-off-by: Hao Chen <chenh1024@gmail.com>
Co-authored-by: Scott Lee <scottjlee@users.noreply.github.com>
Co-authored-by: angelinalg <122562471+angelinalg@users.noreply.github.com>
Signed-off-by: Hao Chen <chenh1024@gmail.com>
@raulchen raulchen mentioned this pull request Mar 27, 2024
Merged
8 tasks
stephanie-wang pushed a commit to stephanie-wang/ray that referenced this pull request Mar 27, 2024
@raulchen @scottjlee
@angelinalg @stephanie-wang
[data][train] Update "Data Loading and Preprocessing" doc (ray-projec…
6675991 
…t#44165)


---------

Signed-off-by: Hao Chen <chenh1024@gmail.com>
Co-authored-by: Scott Lee <scottjlee@users.noreply.github.com>
Co-authored-by: angelinalg <122562471+angelinalg@users.noreply.github.com>
khluu pushed a commit that referenced this pull request Mar 29, 2024
@raulchen @scottjlee @angelinalg
[data][train] Update "Data Loading and Preprocessing" doc (#44165) (#…
0222cb1 
…44316)

Pick #44165 (doc changes only) to 2.10 branch
---------

Signed-off-by: Hao Chen <chenh1024@gmail.com>
Co-authored-by: Scott Lee <scottjlee@users.noreply.github.com>
Co-authored-by: angelinalg <122562471+angelinalg@users.noreply.github.com>
ryanaoleary pushed a commit to ryanaoleary/ray that referenced this pull request Jun 7, 2024
@raulchen @scottjlee
@angelinalg @ryanaoleary
[data][train] Update "Data Loading and Preprocessing" doc (ray-projec…
78162c0 
…t#44165)


---------

Signed-off-by: Hao Chen <chenh1024@gmail.com>
Co-authored-by: Scott Lee <scottjlee@users.noreply.github.com>
Co-authored-by: angelinalg <122562471+angelinalg@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@scottjlee scottjlee scottjlee approved these changes

@angelinalg angelinalg angelinalg approved these changes

@matthewdeng matthewdeng Awaiting requested review from matthewdeng matthewdeng is a code owner

@justinvyu justinvyu Awaiting requested review from justinvyu justinvyu is a code owner

@woshiyyya woshiyyya Awaiting requested review from woshiyyya woshiyyya is a code owner

@ericl ericl Awaiting requested review from ericl ericl is a code owner

@scv119 scv119 Awaiting requested review from scv119 scv119 is a code owner

@c21 c21 Awaiting requested review from c21 c21 is a code owner

@amogkam amogkam Awaiting requested review from amogkam amogkam is a code owner

@bveeramani bveeramani Awaiting requested review from bveeramani bveeramani is a code owner

@stephanie-wang stephanie-wang Awaiting requested review from stephanie-wang stephanie-wang is a code owner

@omatthew98 omatthew98 Awaiting requested review from omatthew98 omatthew98 is a code owner

Assignees

@justinvyu justinvyu

@matthewdeng matthewdeng

@c21 c21

@scottjlee scottjlee

@woshiyyya woshiyyya

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
7 participants
@raulchen @scottjlee @angelinalg @justinvyu @matthewdeng @c21 @woshiyyya