v0.28.0

Release 0.28.0

This release adds streaming reads, so you can bulk-load large datasets on memory-constrained hosts without buffering the whole file. Peak read memory becomes O(batch) instead of O(file), which lets flows that used to require an over-provisioned box run on a small Lambda or Fargate container.

What's New

• Streaming Reads with FlowSource<T>: You can now derive a streaming view of any collection item with .AsStream(), turning an IItem<IEnumerable<T>> into a lazy FlowSource<T> that pulls one batch at a time rather than materializing the entire dataset. This is the answer to reading a multi-row-group Parquet file that doesn't fit in memory: the peak is bounded by the batch size, not the file size.
  • AddBulkLoad for source-to-sink loads: For the common "stream a source straight into a database" case, flow.AddBulkLoad(orders.AsStream(), sink) wires the load as a normal on-DAG step, so it participates in scheduling, caching, and pre-flight like any other step. You write intent, not Compile/Into mechanics.
  • Lazy .Map / .Where combinators: Stateless transforms compose directly on the stream: raw.Map(Normalize).Where(e => e.IsValid). No row is pulled until a sink drains it, so the transform stays O(batch). For "keep the good rows, quarantine the bad," Attempt, SkipErrors, and Rethrow move per-row failures in-band so a single corrupt row no longer aborts the whole load.
  • EF Core streaming bulk sink: Flowthru.Extensions.EFCore.Bulk now ships BulkSink.Insert<T, TContext>(...), a streaming sink that bulk-inserts one batch at a time inside a single transaction. Peak memory stays O(batch), and a mid-stream failure rolls the whole write back, so you never end up with a half-loaded table.
  • StreamingBulkLoad example: A complete, runnable advanced example that streams a Parquet dataset into SQLite and measures the memory win against the eager path over the same data. It's the teachable form of the constrained-host case, and it proves its own thesis by reporting on itself. See StreamingBulkLoad.
  • FUnit streaming test support: FUnit gains streaming affordances so you can test streaming steps without leaving the framework: lift in-memory samples into a FlowSource<T>, drive a stream through your transform, and assert on both the success and error paths.
• FT1201 Streaming-Honesty Analyzer: A new diagnostic catches a format that declares it can stream but doesn't actually stream — either by claiming the capability without implementing the streaming read path, or by materializing the whole input inside a read that advertises itself as streaming. This moves a class of "streaming that silently isn't" from a runtime memory surprise to a design-time compiler error, keeping the streaming promise honest before a flow ever runs.

Bug Fixes

• Constrained-Host Memory Blowups: A flow reading a large file-backed dataset previously buffered the entire file into memory on load, so a wide layer of such reads could exhaust a small host and crash. Reading through .AsStream() now bounds peak memory to a single batch, so those loads survive on memory-limited containers.

❤️ Thank You

• Spencer Elkington