Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Data] Add ParquetDatasink #40818

Merged
merged 78 commits into from
Nov 3, 2023
Merged

[Data] Add ParquetDatasink #40818

merged 78 commits into from
Nov 3, 2023

Conversation

bveeramani
Copy link
Member

@bveeramani bveeramani commented Oct 30, 2023 •
edited
Loading

bveeramani/ray@add-file-based-datasink...bveeramani:ray:add-parquet-datasink

Stacked on

Why are these changes needed?

This PR is part of a larger effort to clean up Datasource interfaces (#40296). #40691 added the new FileDatasink base class, and this PR migrates ParquetDatasource to the new API.

The primary motivation for these changes is to reduced complexity of our internal code base. For more information, see https://docs.google.com/document/d/1Bqhbzvxv7liwpOhyBzRVy5tOzXdy-NiMSFa-6hupr18/edit#heading=h.rytitv546vx5.

Related issue number

See #40296

Checks

  • I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
  • I've run scripts/format.sh to lint the changes in this PR.
  • I've included any doc changes needed for https://docs.ray.io/en/master/.
    • I've added any new APIs to the API Reference. For example, if I added a
      method in Tune, I've added it in doc/source/tune/api/ under the
      corresponding .rst file.
  • I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
  • Testing Strategy
    • Unit tests
    • Release tests
    • This PR is not tested :(

bveeramani and others added 30 commits October 6, 2023 16:50
@bveeramani
Initial commit
d7fbc8a 
Signed-off-by: Balaji Veeramani <balaji@anyscale.com>
@bveeramani
Update files
d4a1d3a 
Signed-off-by: Balaji Veeramani <balaji@anyscale.com>
@bveeramani
Update input_output.rst
71f8217 
Signed-off-by: Balaji Veeramani <balaji@anyscale.com>
@bveeramani
Add stuff
5d8a7f9 
Signed-off-by: Balaji Veeramani <balaji@anyscale.com>
@bveeramani
Appease lint
9f54f73 
Signed-off-by: Balaji Veeramani <balaji@anyscale.com>
@bveeramani
Initial commit
e0dcb91 
Signed-off-by: Balaji Veeramani <balaji@anyscale.com>
@bveeramani
Add unit test
d40d56b 
Signed-off-by: Balaji Veeramani <balaji@anyscale.com>
@bveeramani
Appease lint
2d4beaf 
Signed-off-by: Balaji Veeramani <balaji@anyscale.com>
@bveeramani
Remove extra file
3c51ebd 
Signed-off-by: Balaji Veeramani <balaji@anyscale.com>
@bveeramani
Add deprecation labels
5d57c07 
Signed-off-by: Balaji Veeramani <balaji@anyscale.com>
@bveeramani
Merge branch 'master' into add-datasink
d64ba38 
Signed-off-by: Balaji Veeramani <balaji@anyscale.com>
@bveeramani @stephanie-wang
Update python/ray/data/datasource/file_based_datasource.py
53fae87 
Co-authored-by: Stephanie Wang <swang@cs.berkeley.edu>
Signed-off-by: Balaji Veeramani <bveeramani@berkeley.edu>
@bveeramani @stephanie-wang
Update python/ray/data/datasource/filename_provider.py
4785b74 
Co-authored-by: Stephanie Wang <swang@cs.berkeley.edu>
Signed-off-by: Balaji Veeramani <bveeramani@berkeley.edu>
@bveeramani @stephanie-wang
Update python/ray/data/datasource/filename_provider.py
068acd5 
Co-authored-by: Stephanie Wang <swang@cs.berkeley.edu>
Signed-off-by: Balaji Veeramani <bveeramani@berkeley.edu>
@bveeramani @stephanie-wang
Update python/ray/data/datasource/filename_provider.py
2651fad 
Co-authored-by: Stephanie Wang <swang@cs.berkeley.edu>
Signed-off-by: Balaji Veeramani <bveeramani@berkeley.edu>
@bveeramani
Address review comments
9cd7f38 
Signed-off-by: Balaji Veeramani <balaji@anyscale.com>
@bveeramani
Address review comments
228f76a 
Signed-off-by: Balaji Veeramani <balaji@anyscale.com>
@bveeramani
Merge branch 'master' of https://github.com/ray-project/ray into file…
fd108fd 
…-path-provider

Signed-off-by: Balaji Veeramani <balaji@anyscale.com>
@bveeramani
Appease lint
8ba3fc1 
Signed-off-by: Balaji Veeramani <balaji@anyscale.com>
@bveeramani
Fix link
99b2859 
Signed-off-by: Balaji Veeramani <balaji@anyscale.com>
@bveeramani
Merge branch 'master' into add-datasink
158e8b1 
Signed-off-by: Balaji Veeramani <balaji@anyscale.com>
@bveeramani
Merge branch 'file-path-provider' into add-file-based-datasink
5a39618 
Signed-off-by: Balaji Veeramani <balaji@anyscale.com>
@bveeramani
Update default implementation
f6086ed 
Signed-off-by: Balaji Veeramani <balaji@anyscale.com>
@bveeramani
Fix bug
9f6f9f6 
Signed-off-by: Balaji Veeramani <balaji@anyscale.com>
@bveeramani
Merge branch 'file-path-provider' into add-file-based-datasink
6d8e6ea 
Signed-off-by: Balaji Veeramani <balaji@anyscale.com>
@bveeramani
Initial commit
9b5917d 
Signed-off-by: Balaji Veeramani <balaji@anyscale.com>
@bveeramani
Appease lint
8645be9 
Signed-off-by: Balaji Veeramani <balaji@anyscale.com>
@bveeramani
Fix bug
8f5ceee 
Signed-off-by: Balaji Veeramani <balaji@anyscale.com>
@bveeramani
Fix bug
1c43cea 
Signed-off-by: Balaji Veeramani <balaji@anyscale.com>
@bveeramani
Merge branch 'add-datasink' into add-file-based-datasink
89f8104 
Signed-off-by: Balaji Veeramani <balaji@anyscale.com>
@bveeramani
Merge branch 'master' into add-file-based-datasink
50aa82e 
Signed-off-by: Balaji Veeramani <balaji@anyscale.com>
@bveeramani
Merge branch 'master' into add-datasink
d197193 
Signed-off-by: Balaji Veeramani <balaji@anyscale.com>
@bveeramani
Merge branch 'add-datasink' into add-file-based-datasink
3c145ff 
Signed-off-by: Balaji Veeramani <balaji@anyscale.com>
@bveeramani
Fix typo
68036ac 
Signed-off-by: Balaji Veeramani <balaji@anyscale.com>
@bveeramani
Merge branch 'add-file-based-datasink' into add-parquet-datasink
7a8ad02 
Signed-off-by: Balaji Veeramani <balaji@anyscale.com>
@bveeramani
Make class internal
2f4c398 
Signed-off-by: Balaji Veeramani <balaji@anyscale.com>
@bveeramani
Appease lint
295ab5f 
Signed-off-by: Balaji Veeramani <balaji@anyscale.com>
@bveeramani
Fix docs
7d032af 
Signed-off-by: Balaji Veeramani <balaji@anyscale.com>
@bveeramani
Fix bug
4ef6a0a 
Signed-off-by: Balaji Veeramani <balaji@anyscale.com>
@bveeramani
Merge branch 'add-datasink' into add-file-based-datasink
6c60471 
Signed-off-by: Balaji Veeramani <balaji@anyscale.com>
@bveeramani
Merge branch 'add-file-based-datasink' into add-parquet-datasink
f7114fc 
Signed-off-by: Balaji Veeramani <balaji@anyscale.com>
@bveeramani
Add deprecation decorator
1498022 
Signed-off-by: Balaji Veeramani <balaji@anyscale.com>
@bveeramani
Merge branch 'add-datasink' into add-file-based-datasink
622c3c2 
Signed-off-by: Balaji Veeramani <balaji@anyscale.com>
@bveeramani
Merge branch 'add-file-based-datasink' into add-parquet-datasink
117cb1c 
Signed-off-by: Balaji Veeramani <balaji@anyscale.com>
@bveeramani
Fix bug
f0f143e 
Signed-off-by: Balaji Veeramani <balaji@anyscale.com>
@stephanie-wang
Copy link
Contributor

Please update the PR description. Thanks!

@bveeramani
Fix bug
35347ae 
Signed-off-by: Balaji Veeramani <balaji@anyscale.com>
@bveeramani
Merge branch 'add-datasink' into add-parquet-datasink
13ea0c4 
Signed-off-by: Balaji Veeramani <balaji@anyscale.com>
@bveeramani
Fix bug
dabcd31 
Signed-off-by: Balaji Veeramani <balaji@anyscale.com>
@bveeramani
Merge branch 'add-file-based-datasink' into add-parquet-datasink
fead0b2 
Signed-off-by: Balaji Veeramani <balaji@anyscale.com>
@bveeramani
Address review comments
0c792c8 
Signed-off-by: Balaji Veeramani <balaji@anyscale.com>
@bveeramani
Merge branch 'add-datasink' into add-file-based-datasink
624fd50 
Signed-off-by: Balaji Veeramani <balaji@anyscale.com>
@bveeramani
Merge branch 'add-file-based-datasink' into add-parquet-datasink
efce505 
Signed-off-by: Balaji Veeramani <balaji@anyscale.com>
@bveeramani
Merge branch 'master' into add-parquet-datasink
7f28997 
Signed-off-by: Balaji Veeramani <balaji@anyscale.com>
@bveeramani
Remove dead test
970bbd8 
Signed-off-by: Balaji Veeramani <balaji@anyscale.com>
@bveeramani
Merge branch 'master' into add-parquet-datasink
b2e9861 
Signed-off-by: Balaji Veeramani <balaji@anyscale.com>
@bveeramani
Merge branch 'master' into add-parquet-datasink
5a3c5b0 
Signed-off-by: Balaji Veeramani <balaji@anyscale.com>
@bveeramani bveeramani merged commit 852ad9b into ray-project:master Nov 3, 2023
11 of 28 checks passed
ujjawal-khare pushed a commit to ujjawal-khare-27/ray that referenced this pull request Nov 29, 2023
@bveeramani @stephanie-wang @ujjawal-khare
[Data] Add ParquetDatasink (ray-project#40818)
f3612dc 
This PR is part of a larger effort to clean up Datasource interfaces (ray-project#40296). ray-project#40691 added the new FileDatasink base class, and this PR migrates ParquetDatasource to the new API.

The primary motivation for these changes is to reduced complexity of our internal code base. For more information, see https://docs.google.com/document/d/1Bqhbzvxv7liwpOhyBzRVy5tOzXdy-NiMSFa-6hupr18/edit#heading=h.rytitv546vx5.

---------

Signed-off-by: Balaji Veeramani <balaji@anyscale.com>
Signed-off-by: Balaji Veeramani <bveeramani@berkeley.edu>
Co-authored-by: Stephanie Wang <swang@cs.berkeley.edu>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@stephanie-wang stephanie-wang stephanie-wang approved these changes

@ericl ericl Awaiting requested review from ericl ericl is a code owner

@scv119 scv119 Awaiting requested review from scv119 scv119 is a code owner

@c21 c21 Awaiting requested review from c21 c21 is a code owner

@amogkam amogkam Awaiting requested review from amogkam amogkam is a code owner

@scottjlee scottjlee Awaiting requested review from scottjlee scottjlee is a code owner

@raulchen raulchen Awaiting requested review from raulchen raulchen is a code owner

@Zandew Zandew Awaiting requested review from Zandew

Assignees

@stephanie-wang stephanie-wang

@c21 c21

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
3 participants
@bveeramani @stephanie-wang @c21