Skip to content

feat: streaming FromRow mapping — fetch_rows_as / stream_as for constant-memory struct-mapped queries #91

@StefanSteiner

Description

@StefanSteiner

Summary

fetch_all_as::<T>() is the only public API that wires FromRow into typed struct mapping, but it calls fetch_all first — collecting all rows into a Vec<Row> before any mapping occurs. Memory usage is O(total rows), which makes it unusable for large or unbounded result sets.

The streaming path (execute_query + next_chunk / rows()) gives back untyped Row values only. RowAccessor::new and RowAccessor::build_indices are both pub(crate), so callers cannot wire a #[derive(FromRow)] struct into the chunk loop themselves.

Proposed API

Add a streaming typed iterator to Connection (and the async equivalent on AsyncConnection):

/// Streams rows mapped to `T` via `FromRow`, one at a time.
/// Builds the column-name → index map once from the first chunk's schema;
/// memory usage is O(chunk_size), not O(total_rows).
fn stream_as<T: FromRow>(&self, query: &str) -> impl Iterator<Item = Result<T>> + '_;

The implementation would:

  1. Call execute_query to get a streaming Rowset
  2. Build RowAccessor::build_indices once from the first chunk's schema
  3. Yield T::from_row(RowAccessor::new(&row, &indices)) for each row across all chunks

Workaround today

For streaming millions of rows with a named struct, callers must use rows() + row.get_by_name("col") (linear scan per field per row) or positional row.get(N) (fragile column ordering). Neither composes with #[derive(FromRow)].

Acceptance criteria

  • Connection::stream_as::<T>(query) returns a lazy iterator of Result<T>
  • Index map is built once per query, not once per row or once per chunk
  • Memory stays O(chunk_size) regardless of total result set size
  • AsyncConnection gets an equivalent stream_as returning impl Stream<Item = Result<T>>
  • Documented in docs/ROW_MAPPING.md as Form 5

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels
    No fields configured for Feature.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions