Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[CHORE] Convert GlobScanOperator to perform streaming into result and take a list of glob paths #1577

Merged
merged 4 commits into from
Nov 8, 2023

Conversation

jaychia
Copy link
Contributor

@jaychia jaychia commented Nov 7, 2023

  1. Converts GlobScanOperator to utilize Iterators and tokio channels to provide end-to-end streaming of glob results
  2. Allows GlobScanOperator to be created from a list of glob files, which will chain the results into a single return stream

@github-actions github-actions bot added the chore label Nov 7, 2023
@jaychia jaychia changed the title [CHORE] Convert GlobScanOperator to perform streaming into result [CHORE] Convert GlobScanOperator to perform streaming into result and take a list of glob paths Nov 7, 2023
Base automatically changed from clark/scan-operator-integration to main November 7, 2023 03:28
src/daft-scan/src/glob.rs Outdated Show resolved Hide resolved
src/daft-io/src/google_cloud.rs Show resolved Hide resolved
src/daft-scan/src/glob.rs Outdated Show resolved Hide resolved
Copy link

codecov bot commented Nov 7, 2023

Codecov Report

Merging #1577 (8a3238d) into main (f80dc85) will increase coverage by 0.04%.
Report is 1 commits behind head on main.
The diff coverage is 0.00%.

❗ Current head 8a3238d differs from pull request most recent head 4ab23ae. Consider uploading reports for the commit 4ab23ae to get more accurate results

Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##             main    #1577      +/-   ##
==========================================
+ Coverage   84.85%   84.90%   +0.04%     
==========================================
  Files          54       54              
  Lines        5165     5162       -3     
==========================================
  Hits         4383     4383              
+ Misses        782      779       -3     
Files Coverage Δ
daft/io/common.py 70.27% <0.00%> (+5.27%) ⬆️

@jaychia jaychia enabled auto-merge (squash) November 8, 2023 02:07
@jaychia jaychia merged commit 96b73d6 into main Nov 8, 2023
35 checks passed
@jaychia jaychia deleted the jay/streaming-globs branch November 8, 2023 17:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants