Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Data] Add concurrency and deprecate parallelism for all read APIs #43113

Merged
merged 2 commits into from
Feb 14, 2024
Merged

[Data] Add concurrency and deprecate parallelism for all read APIs #43113

merged 2 commits into from
Feb 14, 2024

Conversation

c21
Copy link
Contributor

@c21 c21 commented Feb 12, 2024

Why are these changes needed?

This PR is to add concurrency and deprecate parallelism w/ override_num_blocks for all read APIs.

In addition, adding a unit test for read_parquet for concurrency.

Related issue number

Checks

  • I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
  • I've run scripts/format.sh to lint the changes in this PR.
  • I've included any doc changes needed for https://docs.ray.io/en/master/.
    • I've added any new APIs to the API Reference. For example, if I added a
      method in Tune, I've added it in doc/source/tune/api/ under the
      corresponding .rst file.
  • I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
  • Testing Strategy
    • Unit tests
    • Release tests
    • This PR is not tested :(

@c21
Add concurrency and deprecate parallelism for all read APIs
d9baaa8 
Signed-off-by: Cheng Su <scnju13@gmail.com>
bveeramani
bveeramani reviewed Feb 12, 2024
View reviewed changes
python/ray/data/read_api.py Outdated Show resolved Hide resolved
python/ray/data/read_api.py Show resolved Hide resolved
python/ray/data/read_api.py Outdated Show resolved Hide resolved
python/ray/data/read_api.py Outdated Show resolved Hide resolved
python/ray/data/read_api.py Outdated Show resolved Hide resolved
raulchen
raulchen reviewed Feb 12, 2024
View reviewed changes
python/ray/data/read_api.py Show resolved Hide resolved
python/ray/data/read_api.py Outdated Show resolved Hide resolved
python/ray/data/read_api.py Outdated Show resolved Hide resolved
@c21
Address comments
64ced49 
Signed-off-by: Cheng Su <scnju13@gmail.com>
@c21 c21 merged commit e973f2e into ray-project:master Feb 14, 2024
9 checks passed
@c21 c21 deleted the read-other branch February 14, 2024 00:26
@c21 c21 mentioned this pull request Mar 4, 2024
Merged
8 tasks
c21 added a commit that referenced this pull request Mar 5, 2024
@c21
[Data] Remove Ray Data read parallelism from all libraries and tests (#…
9a36755 
…43692)

This PR is to remove Ray Data read `parallelism`, replace with `override_num_blocks` from all libraries and unit tests. The PR change on Ray Data is #43113 .

Signed-off-by: Cheng Su <scnju13@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@bveeramani bveeramani bveeramani approved these changes

@raulchen raulchen raulchen approved these changes

@ericl ericl Awaiting requested review from ericl

@scv119 scv119 Awaiting requested review from scv119

@amogkam amogkam Awaiting requested review from amogkam

@scottjlee scottjlee Awaiting requested review from scottjlee scottjlee is a code owner

@stephanie-wang stephanie-wang Awaiting requested review from stephanie-wang stephanie-wang is a code owner

@omatthew98 omatthew98 Awaiting requested review from omatthew98 omatthew98 is a code owner

Assignees

@raulchen raulchen

@bveeramani bveeramani

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@c21 @raulchen @bveeramani