Problem
Arrow batch streaming currently exposes broader storage-format behavior than it can safely provide and can eagerly buffer an entire object, especially through the obstore backend. PyArrow Parquet pre-buffering can also fetch substantially more data than a record batch before the first yield.
Proposed behavior
- Make every sync and async
stream_arrow contract explicitly Parquet-only with keyword-only file_format="parquet" and batch_size=65_536 arguments.
- Reject unsupported formats, recognized non-Parquet suffixes, and nonpositive batch sizes before storage access.
- Iterate one Parquet row group at a time across local, fsspec, and obstore backends.
- Use
obstore.open_reader() through a file-compatible proxy with deterministic sync and async cleanup.
- Remove the full
BytesIO drain and correct obstore base-path handling.
Acceptance criteria
- Multi-file and multi-row-group ordering is stable across all backends.
- Sync and async behavior is equivalent.
- Early generator close and reader exceptions close underlying resources.
- A tracked seekable reader proves that yielding the first batch does not fetch later row-group data beyond the mandatory Parquet footer probe.
- Documentation states the one-row-group I/O bound.
Problem
Arrow batch streaming currently exposes broader storage-format behavior than it can safely provide and can eagerly buffer an entire object, especially through the obstore backend. PyArrow Parquet pre-buffering can also fetch substantially more data than a record batch before the first yield.
Proposed behavior
stream_arrowcontract explicitly Parquet-only with keyword-onlyfile_format="parquet"andbatch_size=65_536arguments.obstore.open_reader()through a file-compatible proxy with deterministic sync and async cleanup.BytesIOdrain and correct obstore base-path handling.Acceptance criteria