Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[docs] Remove trainer references from preprocessors #38348

Merged
merged 15 commits into from
Aug 16, 2023

Conversation

richardliaw
Copy link
Contributor

@richardliaw richardliaw commented Aug 11, 2023

Why are these changes needed?

This PR contains the following changes:

  1. Removes a lot of the AIR trainer <> preprocessor usage and lifecycle text that is no longer relevant
  2. Adds references and recommendation to the map_batches approach
  3. Also adds hooks from Ray Train docs (especially tabular-oriented trainers like XGBoost).

Related issue number

Checks

  • I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
  • I've run scripts/format.sh to lint the changes in this PR.
  • I've included any doc changes needed for https://docs.ray.io/en/master/.
    • I've added any new APIs to the API Reference. For example, if I added a
      method in Tune, I've added it in doc/source/tune/api/ under the
      corresponding .rst file.
  • I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
  • Testing Strategy
    • Unit tests
    • Release tests
    • This PR is not tested :(

doc/source/data/preprocessors.rst Outdated Show resolved Hide resolved
doc/source/data/preprocessors.rst Outdated Show resolved Hide resolved
doc/source/data/preprocessors.rst Outdated Show resolved Hide resolved
doc/source/data/preprocessors.rst Outdated Show resolved Hide resolved
This page covers *preprocessors*, which are a higher level API on top of existing Ray Data operations like `map_batches`,
targeted towards tabular and structured data use cases.

The recommended way to perform preprocessing is to :ref:`use existing Ray Data operations <transforming_data>` instead
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we update the wording here? It's a little bit confusing right now if we say the recommended way for preprocessing is existing Ray Data operations, but then say you should consider built-in preprocessors.

Maybe just "While Ray Data supports generic transformations on datasets, for tabular data it also provides built-in preprocessors"

doc/source/data/preprocessors.rst Outdated Show resolved Hide resolved
doc/source/train/distributed-xgboost-lightgbm.rst Outdated Show resolved Hide resolved
@@ -415,6 +415,56 @@ If your model is sensitive to shuffle quality, call :meth:`Dataset.random_shuffl

For more information on how to optimize shuffling, and which approach to choose, see the :ref:`Optimize shuffling guide <optimizing_shuffles>`.

Preprocessing Data
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
Preprocessing Data
Preprocessing Tabular Data

@richardliaw richardliaw mentioned this pull request Aug 14, 2023
8 tasks
@pcmoritz
Copy link
Contributor

We should also update the example https://docs.ray.io/en/master/ray-air/examples/gptj_deepspeed_fine_tuning.html to remove the preprocessor stuff and instead use map_batches :)

@richardliaw richardliaw merged commit 604aaa2 into ray-project:master Aug 16, 2023
18 of 21 checks passed
harborn pushed a commit to harborn/ray that referenced this pull request Aug 17, 2023
arvind-chandra pushed a commit to lmco/ray that referenced this pull request Aug 31, 2023
Signed-off-by: e428265 <arvind.chandramouli@lmco.com>
vymao pushed a commit to vymao/ray that referenced this pull request Oct 11, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

6 participants