-
Notifications
You must be signed in to change notification settings - Fork 5.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Data] Remove legacy iteration code path #40013
Conversation
Signed-off-by: Balaji Veeramani <balaji@anyscale.com>
@@ -3975,9 +3973,7 @@ def to_tf( | |||
prefetch_batches: The number of batches to fetch ahead of the current batch | |||
to fetch. If set to greater than 0, a separate threadpool is used | |||
to fetch the objects to the local node, format the batches, and apply | |||
the collate_fn. Defaults to 1. You can revert back to the old | |||
prefetching behavior that uses `prefetch_blocks` by setting | |||
`use_legacy_iter_batches` to True in the :class:`~ray.data.DataContext`. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Also remove the deprecated prefetch_blocks args from all these methods, and the ones in DataIterator
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
DatasetPipeline
has methods that depend on prefetch_blocks
. To minimize code conflicts, I'm planning on sequencing the deprecations like this:
- Remove legacy iteration code path
- Remove
DatasetPipeline
- Remove
prefetch_blocks
We changed our iteration implementation in Ray 2.4. To use the old implementation, we exposed a use_legacy_iter_batches option. Since it's been several releases since the change, this PR removes the legacy code path and corresponding option. Signed-off-by: Balaji Veeramani <balaji@anyscale.com>
We changed our iteration implementation in Ray 2.4. To use the old implementation, we exposed a use_legacy_iter_batches option. Since it's been several releases since the change, this PR removes the legacy code path and corresponding option. Signed-off-by: Balaji Veeramani <balaji@anyscale.com> Signed-off-by: Victor <vctr.y.m@example.com>
Why are these changes needed?
We changed our iteration implementation in Ray 2.4. To use the old implementation, we exposed a
use_legacy_iter_batches
option. Since it's been several releases since the change, this PR removes the legacy code path and corresponding option.Related issue number
Checks
git commit -s
) in this PR.scripts/format.sh
to lint the changes in this PR.method in Tune, I've added it in
doc/source/tune/api/
under thecorresponding
.rst
file.