-
Notifications
You must be signed in to change notification settings - Fork 5.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[docs] Remove trainer references from preprocessors #38348
[docs] Remove trainer references from preprocessors #38348
Conversation
doc/source/data/preprocessors.rst
Outdated
This page covers *preprocessors*, which are a higher level API on top of existing Ray Data operations like `map_batches`, | ||
targeted towards tabular and structured data use cases. | ||
|
||
The recommended way to perform preprocessing is to :ref:`use existing Ray Data operations <transforming_data>` instead |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can we update the wording here? It's a little bit confusing right now if we say the recommended way for preprocessing is existing Ray Data operations, but then say you should consider built-in preprocessors.
Maybe just "While Ray Data supports generic transformations on datasets, for tabular data it also provides built-in preprocessors"
@@ -415,6 +415,56 @@ If your model is sensitive to shuffle quality, call :meth:`Dataset.random_shuffl | |||
|
|||
For more information on how to optimize shuffling, and which approach to choose, see the :ref:`Optimize shuffling guide <optimizing_shuffles>`. | |||
|
|||
Preprocessing Data |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Preprocessing Data | |
Preprocessing Tabular Data |
We should also update the example https://docs.ray.io/en/master/ray-air/examples/gptj_deepspeed_fine_tuning.html to remove the preprocessor stuff and instead use |
…/ray into preprocessor-changes
Signed-off-by: harborn <gangsheng.wu@intel.com>
Signed-off-by: e428265 <arvind.chandramouli@lmco.com>
Signed-off-by: Victor <vctr.y.m@example.com>
Why are these changes needed?
This PR contains the following changes:
map_batches
approachRelated issue number
Checks
git commit -s
) in this PR.scripts/format.sh
to lint the changes in this PR.method in Tune, I've added it in
doc/source/tune/api/
under thecorresponding
.rst
file.