Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Data] Add FileDatasink subclasses #40693

Merged
merged 89 commits into from
Nov 3, 2023
Merged

[Data] Add FileDatasink subclasses #40693

merged 89 commits into from
Nov 3, 2023

Conversation

bveeramani
Copy link
Member

@bveeramani bveeramani commented Oct 26, 2023 •
edited
Loading

bveeramani/ray@add-file-based-datasink...bveeramani:ray:add-image-datasink

Stacked on

Why are these changes needed?

This PR is part of a larger effort to clean up Datasource interfaces (#40296). #40691 added the new FileDatasink base class, and this PR migrates FileDatasource implementations to the new API.

The primary motivation for these changes is to reduced complexity of our internal code base. For more information, see https://docs.google.com/document/d/1Bqhbzvxv7liwpOhyBzRVy5tOzXdy-NiMSFa-6hupr18/edit#heading=h.rytitv546vx5.

Related issue number

See #40296

Checks

  • I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
  • I've run scripts/format.sh to lint the changes in this PR.
  • I've included any doc changes needed for https://docs.ray.io/en/master/.
    • I've added any new APIs to the API Reference. For example, if I added a
      method in Tune, I've added it in doc/source/tune/api/ under the
      corresponding .rst file.
  • I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
  • Testing Strategy
    • Unit tests
    • Release tests
    • This PR is not tested :(

bveeramani and others added 30 commits October 6, 2023 16:50
@bveeramani
Initial commit
d7fbc8a 
Signed-off-by: Balaji Veeramani <balaji@anyscale.com>
@bveeramani
Update files
d4a1d3a 
Signed-off-by: Balaji Veeramani <balaji@anyscale.com>
@bveeramani
Update input_output.rst
71f8217 
Signed-off-by: Balaji Veeramani <balaji@anyscale.com>
@bveeramani
Add stuff
5d8a7f9 
Signed-off-by: Balaji Veeramani <balaji@anyscale.com>
@bveeramani
Appease lint
9f54f73 
Signed-off-by: Balaji Veeramani <balaji@anyscale.com>
@bveeramani
Initial commit
e0dcb91 
Signed-off-by: Balaji Veeramani <balaji@anyscale.com>
@bveeramani
Add unit test
d40d56b 
Signed-off-by: Balaji Veeramani <balaji@anyscale.com>
@bveeramani
Appease lint
2d4beaf 
Signed-off-by: Balaji Veeramani <balaji@anyscale.com>
@bveeramani
Remove extra file
3c51ebd 
Signed-off-by: Balaji Veeramani <balaji@anyscale.com>
@bveeramani
Add deprecation labels
5d57c07 
Signed-off-by: Balaji Veeramani <balaji@anyscale.com>
@bveeramani
Merge branch 'master' into add-datasink
d64ba38 
Signed-off-by: Balaji Veeramani <balaji@anyscale.com>
@bveeramani @stephanie-wang
Update python/ray/data/datasource/file_based_datasource.py
53fae87 
Co-authored-by: Stephanie Wang <swang@cs.berkeley.edu>
Signed-off-by: Balaji Veeramani <bveeramani@berkeley.edu>
@bveeramani @stephanie-wang
Update python/ray/data/datasource/filename_provider.py
4785b74 
Co-authored-by: Stephanie Wang <swang@cs.berkeley.edu>
Signed-off-by: Balaji Veeramani <bveeramani@berkeley.edu>
@bveeramani @stephanie-wang
Update python/ray/data/datasource/filename_provider.py
068acd5 
Co-authored-by: Stephanie Wang <swang@cs.berkeley.edu>
Signed-off-by: Balaji Veeramani <bveeramani@berkeley.edu>
@bveeramani @stephanie-wang
Update python/ray/data/datasource/filename_provider.py
2651fad 
Co-authored-by: Stephanie Wang <swang@cs.berkeley.edu>
Signed-off-by: Balaji Veeramani <bveeramani@berkeley.edu>
@bveeramani
Address review comments
9cd7f38 
Signed-off-by: Balaji Veeramani <balaji@anyscale.com>
@bveeramani
Address review comments
228f76a 
Signed-off-by: Balaji Veeramani <balaji@anyscale.com>
@bveeramani
Merge branch 'master' of https://github.com/ray-project/ray into file…
fd108fd 
…-path-provider

Signed-off-by: Balaji Veeramani <balaji@anyscale.com>
@bveeramani
Appease lint
8ba3fc1 
Signed-off-by: Balaji Veeramani <balaji@anyscale.com>
@bveeramani
Fix link
99b2859 
Signed-off-by: Balaji Veeramani <balaji@anyscale.com>
@bveeramani
Merge branch 'master' into add-datasink
158e8b1 
Signed-off-by: Balaji Veeramani <balaji@anyscale.com>
@bveeramani
Merge branch 'file-path-provider' into add-file-based-datasink
5a39618 
Signed-off-by: Balaji Veeramani <balaji@anyscale.com>
@bveeramani
Update default implementation
f6086ed 
Signed-off-by: Balaji Veeramani <balaji@anyscale.com>
@bveeramani
Fix bug
9f6f9f6 
Signed-off-by: Balaji Veeramani <balaji@anyscale.com>
@bveeramani
Merge branch 'file-path-provider' into add-file-based-datasink
6d8e6ea 
Signed-off-by: Balaji Veeramani <balaji@anyscale.com>
@bveeramani
Initial commit
9b5917d 
Signed-off-by: Balaji Veeramani <balaji@anyscale.com>
@bveeramani
Appease lint
8645be9 
Signed-off-by: Balaji Veeramani <balaji@anyscale.com>
@bveeramani
Fix bug
8f5ceee 
Signed-off-by: Balaji Veeramani <balaji@anyscale.com>
@bveeramani
Fix bug
1c43cea 
Signed-off-by: Balaji Veeramani <balaji@anyscale.com>
@bveeramani
Merge branch 'add-datasink' into add-file-based-datasink
89f8104 
Signed-off-by: Balaji Veeramani <balaji@anyscale.com>
@bveeramani
Fix typo
68036ac 
Signed-off-by: Balaji Veeramani <balaji@anyscale.com>
@bveeramani
Fix bug
8692966 
Signed-off-by: Balaji Veeramani <balaji@anyscale.com>
@bveeramani
Merge branch 'add-file-based-datasink' into add-image-datasink
80db6af 
Signed-off-by: Balaji Veeramani <balaji@anyscale.com>
@bveeramani
Make classes internal
d063106 
Signed-off-by: Balaji Veeramani <balaji@anyscale.com>
@bveeramani
Fix docs
7d032af 
Signed-off-by: Balaji Veeramani <balaji@anyscale.com>
@bveeramani
Fix bug
4ef6a0a 
Signed-off-by: Balaji Veeramani <balaji@anyscale.com>
@bveeramani
Merge branch 'add-datasink' into add-file-based-datasink
6c60471 
Signed-off-by: Balaji Veeramani <balaji@anyscale.com>
@bveeramani
Rename JSON
622c7a9 
Signed-off-by: Balaji Veeramani <balaji@anyscale.com>
@bveeramani
Merge branch 'add-file-based-datasink' into add-image-datasink
b8fb78f 
Signed-off-by: Balaji Veeramani <balaji@anyscale.com>
@bveeramani
Add deprecation decorator
1498022 
Signed-off-by: Balaji Veeramani <balaji@anyscale.com>
@bveeramani
Merge branch 'add-datasink' into add-file-based-datasink
622c3c2 
Signed-off-by: Balaji Veeramani <balaji@anyscale.com>
@bveeramani
Merge branch 'add-file-based-datasink' into add-image-datasink
4e39948 
Signed-off-by: Balaji Veeramani <balaji@anyscale.com>
@bveeramani
Fix bug and appease lint
b4b6d26 
Signed-off-by: Balaji Veeramani <balaji@anyscale.com>
@bveeramani
Fix bug
35347ae 
Signed-off-by: Balaji Veeramani <balaji@anyscale.com>
@bveeramani
Fix bug
dabcd31 
Signed-off-by: Balaji Veeramani <balaji@anyscale.com>
@bveeramani
Clean up args
0f82485 
Signed-off-by: Balaji Veeramani <balaji@anyscale.com>
@bveeramani
Appease lint
7236d76 
Signed-off-by: Balaji Veeramani <balaji@anyscale.com>
@bveeramani
Appease lint
088d338 
Signed-off-by: Balaji Veeramani <balaji@anyscale.com>
@bveeramani
Fix bug
f440ed2 
Signed-off-by: Balaji Veeramani <balaji@anyscale.com>
@bveeramani
Merge branch 'add-file-based-datasink' into add-image-datasink
b07464c 
Signed-off-by: Balaji Veeramani <balaji@anyscale.com>
@bveeramani
Address review comments
0c792c8 
Signed-off-by: Balaji Veeramani <balaji@anyscale.com>
@bveeramani
Merge branch 'add-datasink' into add-file-based-datasink
624fd50 
Signed-off-by: Balaji Veeramani <balaji@anyscale.com>
@bveeramani
Merge branch 'add-file-based-datasink' into add-image-datasink
d496f71 
Signed-off-by: Balaji Veeramani <balaji@anyscale.com>
@bveeramani bveeramani mentioned this pull request Nov 3, 2023
Merged
10 tasks
bveeramani added a commit that referenced this pull request Nov 3, 2023
@bveeramani @stephanie-wang
[Data] Add FileDatasink (#40691)
af59384 
This PR adds `_FileDatasink`, and it's user-facing subclasses `RowBasedFileDatasink` and `BockBasedFileDatasink`. #40693 migrates `FileDatasource` implementations to the new APIs. These changes are part of a larger effort to clean up `Datasource` interfaces (#40296).  

---------

Signed-off-by: Balaji Veeramani <balaji@anyscale.com>
Signed-off-by: Balaji Veeramani <bveeramani@berkeley.edu>
Co-authored-by: Stephanie Wang <swang@cs.berkeley.edu>
@bveeramani
Merge branch 'master' into add-image-datasink
da72ffc 
Signed-off-by: Balaji Veeramani <balaji@anyscale.com>
@bveeramani bveeramani merged commit 7b8ec8a into ray-project:master Nov 3, 2023
15 of 28 checks passed
@bveeramani bveeramani deleted the add-image-datasink branch November 3, 2023 01:40
ujjawal-khare pushed a commit to ujjawal-khare-27/ray that referenced this pull request Nov 29, 2023
@bveeramani @stephanie-wang @ujjawal-khare
[Data] Add FileDatasink (ray-project#40691)
c9c2386 
This PR adds `_FileDatasink`, and it's user-facing subclasses `RowBasedFileDatasink` and `BockBasedFileDatasink`. ray-project#40693 migrates `FileDatasource` implementations to the new APIs. These changes are part of a larger effort to clean up `Datasource` interfaces (ray-project#40296).  

---------

Signed-off-by: Balaji Veeramani <balaji@anyscale.com>
Signed-off-by: Balaji Veeramani <bveeramani@berkeley.edu>
Co-authored-by: Stephanie Wang <swang@cs.berkeley.edu>
ujjawal-khare pushed a commit to ujjawal-khare-27/ray that referenced this pull request Nov 29, 2023
@bveeramani @stephanie-wang @ujjawal-khare
[Data] Add FileDatasink subclasses (ray-project#40693)
12bed94 
This PR is part of a larger effort to clean up Datasource interfaces (ray-project#40296). This ray-project#40691 added the new FileDatasink base class, and this PR migrates FileDatasource implementations to the new API.

The primary motivation for these changes is to reduced complexity of our internal code base. For more information, see https://docs.google.com/document/d/1Bqhbzvxv7liwpOhyBzRVy5tOzXdy-NiMSFa-6hupr18/edit#heading=h.rytitv546vx5.

---------

Signed-off-by: Balaji Veeramani <balaji@anyscale.com>
Signed-off-by: Balaji Veeramani <bveeramani@berkeley.edu>
Co-authored-by: Stephanie Wang <swang@cs.berkeley.edu>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@stephanie-wang stephanie-wang stephanie-wang approved these changes

@ericl ericl Awaiting requested review from ericl ericl is a code owner

@scv119 scv119 Awaiting requested review from scv119 scv119 is a code owner

@c21 c21 Awaiting requested review from c21 c21 is a code owner

@amogkam amogkam Awaiting requested review from amogkam amogkam is a code owner

@scottjlee scottjlee Awaiting requested review from scottjlee scottjlee is a code owner

@raulchen raulchen Awaiting requested review from raulchen raulchen is a code owner

@Zandew Zandew Awaiting requested review from Zandew

Assignees

@stephanie-wang stephanie-wang

@c21 c21

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@bveeramani @stephanie-wang @c21