Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Data] Add read_lance API to read Lance Dataset #45106

Merged
merged 25 commits into from
May 2, 2024
Merged

[Data] Add read_lance API to read Lance Dataset #45106

merged 25 commits into from
May 2, 2024

Conversation

c21
Copy link
Contributor

@c21 c21 commented May 2, 2024 •
edited

Why are these changes needed?

This PR is an update on top of #44853 .

Related issue number

Checks

  • I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
  • I've run scripts/format.sh to lint the changes in this PR.
  • I've included any doc changes needed for https://docs.ray.io/en/master/.
    • I've added any new APIs to the API Reference. For example, if I added a
      method in Tune, I've added it in doc/source/tune/api/ under the
      corresponding .rst file.
  • I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
  • Testing Strategy
    • Unit tests
    • Release tests
    • This PR is not tested :(

brent-anyscale and others added 22 commits May 2, 2024 02:22
@brent-anyscale @c21
add: New LanceDB datasource for Ray Data
aa157ed 
This PR adds a new datasource for Ray Data that reads from LanceDB.
This datasource is a thin wrapper around the LanceDB Python client that allows users to read data from LanceDB into Ray Data.

On branch anyscalebrent/lancedb_datasource
Changes to be committed:
	modified:   python/ray/data/__init__.py
	modified:   python/ray/data/datasource/__init__.py
	new file:   python/ray/data/datasource/lancedb_datasource.py
	modified:   python/ray/data/read_api.py
@brent-anyscale @c21
upd: datasource __init__
478ee95 
Signed-off-by: Brent Bain <brent@anyscale.com>

Changes to be committed:
	modified:   python/ray/data/datasource/__init__.py
	modified:   python/ray/data/datasource/lancedb_datasource.py
@brent-anyscale @c21
upd: read_api.py - fix linting errors with line length
614368f 
Signed-off-by: Brent Bain <brent@anyscale.com>

Changes to be committed:
	modified:   python/ray/data/read_api.py
@brent-anyscale @c21
upd: rename lancedb resources to lance resources
999732b 
Signed-off-by: Brent Bain <brent@anyscale.com>

Changes to be committed:
	modified:   python/ray/data/__init__.py
	modified:   python/ray/data/datasource/__init__.py
	renamed:    python/ray/data/datasource/lancedb_datasource.py -> python/ray/data/datasource/lance_datasource.py
	modified:   python/ray/data/read_api.py
@brent-anyscale @c21
upd: Additional updates to remove DB from Lance resources
051962a 
Signed-off-by: Brent Bain <brent@anyscale.com>

Changes to be committed:
	modified:   python/ray/data/datasource/__init__.py
	modified:   python/ray/data/datasource/lance_datasource.py
@brent-anyscale @c21
upd: Additional updates to remove DB from lance name
c876592 
Signed-off-by: Brent Bain <brent@anyscale.com>

Changes to be committed:
	modified:   python/ray/data/__init__.py
	modified:   python/ray/data/datasource/__init__.py
@brent-anyscale @c21
upd: Lance ReadAPI comment for AZ support
34ce942 
Signed-off-by: Brent Bain <brent@anyscale.com>

Changes to be committed:
	modified:   python/ray/data/read_api.py

Signed-off-by: Brent Bain <brent@anyscale.com>
@brent-anyscale @c21
upd: Include limk to LanceDB docs in read_api.py
bad7e38 
Signed-off-by: Brent Bain <brent@anyscale.com>

Changes to be committed:
	modified:   python/ray/data/read_api.py
@brent-anyscale @c21
upd: lance_datasource - remove header comment
7c18a26 
Signed-off-by: Brent Bain <brent@anyscale.com>

Changes to be committed:
	modified:   python/ray/data/datasource/lance_datasource.py
@brent-anyscale @c21
upd: Change init params to Optional instead of Unions
dc0a0a4 
Signed-off-by: Brent Bain <brent@anyscale.com>

The __init__ method of the LanceDatasource class now uses Optional instead of Union for the parameters.

Changes to be committed:
	modified:   python/ray/data/datasource/lance_datasource.py
@brent-anyscale @c21
upd: lance_datasource - change to use to_batches
a5c507e 
This change updates lance_datasource to a simpler implementation
of to_batches.

Signed-off-by: Brent Bain <brent@anyscale.com>

Changes to be committed:
	modified:   python/ray/data/datasource/lance_datasource.py
@brent-anyscale @c21
upd: lance_datasource - set parallelism based on number of fragments
1c2196e 
Signed-off-by: Brent Bain <brent@anyscale.com>

Changes to be committed:
	modified:   python/ray/data/datasource/lance_datasource.py
@brent-anyscale @c21
upd: lance_datasource - change from yield to return
fb3b285 
Yield isn't working as expected. Changing back to return.

Signed-off-by: Brent Bain <brent@anyscale.com>

Changes to be committed:
	modified:   python/ray/data/datasource/lance_datasource.py
@brent-anyscale @c21
upd: lance_dataset comment - changing for consistent naming
cc42356 
Signed-off-by: Brent Bain <brent@anyscale.com>

Changes to be committed:
	modified:   python/ray/data/datasource/lance_datasource.py
@brent-anyscale @c21
upd: lance_datasource - changed how fragment reading is performed
caba01e 
Signed-off-by: Brent Bain <brent@anyscale.com>

Changes to be committed:
	modified:   python/ray/data/datasource/lance_datasource.py
@brent-anyscale @c21
upd: lance datasource - comments updated
c6e931a 
Signed-off-by: Brent Bain <brent@anyscale.com>

Changes to be committed:
	modified:   python/ray/data/datasource/lance_datasource.py
@brent-anyscale @c21
upd: lance_datasource Add storage options to pass to Lance
276f1fe 
Signed-off-by: Brent Bain <brent@anyscale.com>

Changes to be committed:
	modified:   python/ray/data/datasource/lance_datasource.py
@brent-anyscale @c21
upd: lance datasource
9d88cdb 
Changes to lance_datasource parallelism handling.
Added initial test for lance_datasource.
Signed-off-by: Brent Bain <brent@anyscale.com>

Changes to be committed:
	modified:   python/ray/data/datasource/lance_datasource.py
	modified:   python/ray/data/read_api.py
	new file:   python/ray/data/tests/test_lance.py
@brent-anyscale @c21
upd: lance tests linting updates
3a4ffd4 
Signed-off-by: Brent Bain <brent@anyscale.com>

Changes to be committed:
	modified:   python/ray/data/datasource/lance_datasource.py
	modified:   python/ray/data/tests/test_lance.py
@brent-anyscale @c21
upd: data-test-requirements - add lancedb
d8d4714 
Signed-off-by: Brent Bain <brent@anyscale.com>

Changes to be committed:
	modified:   python/requirements/ml/data-test-requirements.txt
@brent-anyscale @c21
upd: data BUILD - and Lance test
24cd74b 
Signed-off-by: Brent Bain <brent@anyscale.com>

Changes to be committed:
	modified:   python/ray/data/BUILD
@c21
Address all comments
4e3d120
@c21
Fix format
9bb0356 
Signed-off-by: Cheng Su <scnju13@gmail.com>
@c21 c21 force-pushed the anyscalebrent/lancedb_datasource branch from 871ee78 to 9bb0356 Compare May 2, 2024 09:22
@c21 c21 changed the title [Data] Add API to read Lance Dataset [Data] Add read_lance API to read Lance Dataset May 2, 2024
@c21
Skip unit test if PyArrow is older than 12
d44df90 
Signed-off-by: Cheng Su <scnju13@gmail.com>
@zhe-thoughts zhe-thoughts assigned c21 and unassigned c21 May 2, 2024
@c21
Fix import
3fb9355 
Signed-off-by: Cheng Su <scnju13@gmail.com>
@brent-anyscale brent-anyscale mentioned this pull request May 2, 2024
Closed
8 tasks
@c21 c21 merged commit 24bf8a1 into ray-project:master May 2, 2024
5 checks passed
@c21 c21 deleted the anyscalebrent/lancedb_datasource branch May 2, 2024 21:04
harborn pushed a commit to harborn/ray that referenced this pull request May 8, 2024
@c21 @brent-anyscale @harborn
[Data] Add read_lance API to read Lance Dataset (ray-project#45106)
c796b68 
This PR adds the capability to load a LanceDB dataset into a Ray Dataset.

Signed-off-by: Brent Bain <brent@anyscale.com>
Signed-off-by: Cheng Su <scnju13@gmail.com>
Co-authored-by: Brent Bain <brent@anyscale.com>
peytondmurray pushed a commit to peytondmurray/ray that referenced this pull request May 13, 2024
@c21 @brent-anyscale @peytondmurray
[Data] Add read_lance API to read Lance Dataset (ray-project#45106)
b1d0dee 
This PR adds the capability to load a LanceDB dataset into a Ray Dataset.

Signed-off-by: Brent Bain <brent@anyscale.com>
Signed-off-by: Cheng Su <scnju13@gmail.com>
Co-authored-by: Brent Bain <brent@anyscale.com>
ryanaoleary pushed a commit to ryanaoleary/ray that referenced this pull request Jun 6, 2024
@c21 @brent-anyscale @ryanaoleary
[Data] Add read_lance API to read Lance Dataset (ray-project#45106)
c96d2b9 
This PR adds the capability to load a LanceDB dataset into a Ray Dataset.

Signed-off-by: Brent Bain <brent@anyscale.com>
Signed-off-by: Cheng Su <scnju13@gmail.com>
Co-authored-by: Brent Bain <brent@anyscale.com>
Signed-off-by: Ryan O'Leary <ryanaoleary@google.com>
ryanaoleary pushed a commit to ryanaoleary/ray that referenced this pull request Jun 7, 2024
@c21 @brent-anyscale @ryanaoleary
[Data] Add read_lance API to read Lance Dataset (ray-project#45106)
9ccef1f 
This PR adds the capability to load a LanceDB dataset into a Ray Dataset.

Signed-off-by: Brent Bain <brent@anyscale.com>
Signed-off-by: Cheng Su <scnju13@gmail.com>
Co-authored-by: Brent Bain <brent@anyscale.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@scottjlee scottjlee scottjlee approved these changes

@ericl ericl Awaiting requested review from ericl ericl is a code owner

@scv119 scv119 Awaiting requested review from scv119 scv119 is a code owner

@amogkam amogkam Awaiting requested review from amogkam amogkam is a code owner

@bveeramani bveeramani Awaiting requested review from bveeramani bveeramani is a code owner

@raulchen raulchen Awaiting requested review from raulchen raulchen is a code owner

@stephanie-wang stephanie-wang Awaiting requested review from stephanie-wang stephanie-wang is a code owner

@omatthew98 omatthew98 Awaiting requested review from omatthew98 omatthew98 is a code owner

Assignees

@scottjlee scottjlee

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
3 participants
@c21 @scottjlee @brent-anyscale