Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Datasets] Do not eagerly execute first block for read_xxx API #31558

Merged
merged 2 commits into from Jan 11, 2023

Conversation

c21
Copy link
Contributor

@c21 c21 commented Jan 10, 2023

Signed-off-by: Cheng Su scnju13@gmail.com

Why are these changes needed?

This PR is the followup of #31286 (review). The change includes:

  • read_api.py:read_datasource(): Remove the logic to eagerly execute first block for read.
  • Dataset.schema(): Change default value of fetch_if_missing from False to True. So always trigger execution if schame is missing.
  • ExecutionPlan.schema(): if plan is having lazy block list as output, execute the first block only to get schema, instead of executing all blocks.
  • Other files: unit test change to work with new logic.

Related issue number

Checks

  • I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
  • I've run scripts/format.sh to lint the changes in this PR.
  • I've included any doc changes needed for https://docs.ray.io/en/master/.
  • I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
  • Testing Strategy
    • Unit tests
    • Release tests
    • This PR is not tested :(

Signed-off-by: Cheng Su <scnju13@gmail.com>
Signed-off-by: Cheng Su <scnju13@gmail.com>
@c21
Copy link
Contributor Author

c21 commented Jan 10, 2023

The failed book-documentation looks unrelated to this PR. Seeing it also has same failure on other PR.

Copy link
Contributor

@clarkzinzow clarkzinzow left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

python/ray/data/_internal/plan.py Show resolved Hide resolved
@clarkzinzow clarkzinzow merged commit 3fabd7f into ray-project:master Jan 11, 2023
@c21 c21 deleted the schema branch January 12, 2023 01:08
AmeerHajAli pushed a commit that referenced this pull request Jan 12, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants