Skip to content

fpstreams 2.3.0

Latest

Choose a tag to compare

@steventimes steventimes released this 15 Oct 06:24

fpstreams 2.3.0 adds explicit batch mapping and lets parallel maps reuse an executor supplied by the caller. It also fixes async callback type inference and reduces setup work in supported Python record pipelines.

Batch mapping

Flow.map_batches() and AsyncFlow.map_batches() pass each batch to a callback as a tuple and emit one result per batch. The final batch can be shorter, and empty input never calls the callback.

from fpstreams import flow

totals = flow(range(7)).map_batches(sum, size=3).to_list()
assert totals == [3, 12, 6]

Sync batches inherit parallel() settings. Async batches support bounded concurrency, ordering, buffering, and cooperative timeouts. Returned iterables stay intact; use flat_map(iter) to expand them.

Reusable executors

Pass executor=pool to map_parallel() or parallel() to reuse a thread or process pool across queries. Each query cancels its queued work and waits for its own running callbacks during cleanup. The pool stays open, and unrelated tasks are left alone. The caller closes the pool.

Existing calls without an executor keep their worker and backend defaults, including spawn for library-created process pools.

Fixes and internal changes

  • Async map(), map_async(), and map_batches() infer the resolved return type for async callbacks and callbacks that mix immediate and awaitable results.
  • Supported single-stage Python select() and with_columns() queries reuse a loop template while preserving selector behavior and row lifetimes.
  • Single-key sorted grouping avoids a temporary selector generator and wrapper tuple per row. Key identity, validation, lookahead, and callback errors retain their existing behavior.
  • New analytics examples cover generator inputs, weighted aggregation, parameterized SQLite queries, and atomic batch writes.
  • CI engine benchmarks collect 21 samples per scenario. Reports retain every wall-time sample and add process CPU time for diagnosis; regression limits are unchanged.

Python 3.11 or newer is required. See the performance guide for batch and executor examples, and the changelog for the release details.