Skip to content

Make Arrow batch streaming row-group-bounded and explicitly Parquet-only #667

Description

@cofin

Problem

Arrow batch streaming currently exposes broader storage-format behavior than it can safely provide and can eagerly buffer an entire object, especially through the obstore backend. PyArrow Parquet pre-buffering can also fetch substantially more data than a record batch before the first yield.

Proposed behavior

  • Make every sync and async stream_arrow contract explicitly Parquet-only with keyword-only file_format="parquet" and batch_size=65_536 arguments.
  • Reject unsupported formats, recognized non-Parquet suffixes, and nonpositive batch sizes before storage access.
  • Iterate one Parquet row group at a time across local, fsspec, and obstore backends.
  • Use obstore.open_reader() through a file-compatible proxy with deterministic sync and async cleanup.
  • Remove the full BytesIO drain and correct obstore base-path handling.

Acceptance criteria

  • Multi-file and multi-row-group ordering is stable across all backends.
  • Sync and async behavior is equivalent.
  • Early generator close and reader exceptions close underlying resources.
  • A tracked seekable reader proves that yielding the first batch does not fetch later row-group data beyond the mandatory Parquet footer probe.
  • Documentation states the one-row-group I/O bound.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions