Skip to content

fix(#667): bound Parquet storage streaming - #671

Draft
cofin wants to merge 2 commits into
mainfrom
fix/storage-parquet-streaming
Draft

fix(#667): bound Parquet storage streaming#671
cofin wants to merge 2 commits into
mainfrom
fix/storage-parquet-streaming

Conversation

@cofin

@cofin cofin commented Aug 1, 2026

Copy link
Copy Markdown
Member

Summary

  • make Arrow storage streaming explicitly Parquet-only and row-group-bounded
  • stream local, fsspec, and obstore objects without draining obstore into BytesIO
  • add deterministic reader cleanup, input validation, and consistent sync/async behavior

Details

All streaming contracts now expose keyword-only file_format="parquet" and batch_size=65_536. Validation occurs before storage access. Each backend iterates one Parquet row group at a time, and obstore uses open_reader() through a file-compatible proxy. The change also fixes base-path duplication.

Closes #667.

@codecov-commenter

codecov-commenter commented Aug 1, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 83.83838% with 16 lines in your changes missing coverage. Please review.
✅ Project coverage is 77.26%. Comparing base (a3a3012) to head (e129289).

Files with missing lines Patch % Lines
sqlspec/storage/backends/obstore.py 81.25% 7 Missing and 2 partials ⚠️
sqlspec/storage/backends/base.py 58.82% 4 Missing and 3 partials ⚠️
Additional details and impacted files
@@           Coverage Diff           @@
##             main     #671   +/-   ##
=======================================
  Coverage   77.26%   77.26%           
=======================================
  Files         475      476    +1     
  Lines       67901    67972   +71     
  Branches     9335     9343    +8     
=======================================
+ Hits        52462    52517   +55     
- Misses      12035    12046   +11     
- Partials     3404     3409    +5     
Flag Coverage Δ
integration 60.78% <40.40%> (-0.03%) ⬇️
py3.10 75.57% <82.82%> (-0.01%) ⬇️
py3.11 75.58% <82.82%> (+<0.01%) ⬆️
py3.12 75.58% <82.82%> (+<0.01%) ⬆️
py3.13 75.59% <82.82%> (+1.36%) ⬆️
py3.14 76.46% <83.83%> (+<0.01%) ⬆️
unit 64.79% <83.83%> (+0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
sqlspec/protocols.py 100.00% <100.00%> (ø)
sqlspec/storage/_arrow_stream.py 100.00% <100.00%> (ø)
sqlspec/storage/backends/fsspec.py 90.64% <100.00%> (+0.09%) ⬆️
sqlspec/storage/backends/local.py 81.50% <100.00%> (+0.18%) ⬆️
sqlspec/storage/backends/base.py 88.65% <58.82%> (-6.47%) ⬇️
sqlspec/storage/backends/obstore.py 83.12% <81.25%> (-0.89%) ⬇️
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@cofin
cofin force-pushed the fix/storage-parquet-streaming branch from 369bcfe to e129289 Compare August 1, 2026 22:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Make Arrow batch streaming row-group-bounded and explicitly Parquet-only

2 participants