ToObjectStream

[Grauenwolf]

When working with really large result sets, it may be preferable to stream the data instead of materializing a list.

Both NPoco and Dapper already support this:

• https://github.com/StackExchange/dapper-dot-net
• https://github.com/schotime/NPoco/wiki/Query-List

To do better we need to marry it with something that still manages lifecycles. For example, the exposed enumerator needs to close the connection.

Maybe include Reactive Extensions support with this.

[Grauenwolf]

See also #281

[Grauenwolf]

class AsyncObectStream : IAsyncEnumerator<T>, IAsyncDisposable

async using var stream = dataSource.From<T>().ToObjectStream().ExecuteAsync();
async foreach (var item in stream )
{
    //do something
}

[Grauenwolf]

This won't implement ILink<T>. You can't perform a transformation or caching operation on it.

[Grauenwolf]

This is going to need new Data Source methods. Namely ExecuteStream and ExecuteStreamAsync.

These methods will create a connection, but never close it. The result object will have to handle closing itself via Dispose/DisposeAsync.

[Grauenwolf]

For a open or transactional data source, the Dispose call will be a no-op.

[Grauenwolf]

Before we implement this, we should document the full pipeline for a normal Execute call.

[Grauenwolf]

We're going to need the pre-existing StreamingObjectConstructor class for this.

[Grauenwolf]

Previously, Chain would fully manage database connections by default. Specifically, it would open and close connections automatically unless a transaction was involved. In that case, the developer only needed to manage the transactional data source itself.

However, there are times when a result set is too large to handle at one time. In this case the developer will want an IEnumerable or IAsyncEnumerable instead of a collection. To support this, the ToObjectStream materializer was created.

When used in place of ToCollection, the caller gets a ObjectStream object. This object implements IEnumerable<TObject>, IDisposable, IAsyncDisposable, abd IAsyncEnumerable<TObject>. (That latter two are only available in .NET 6 or later.)

This object stream may be used directly, as shown below, or attached to an RX Observable or TPL Dataflow just like any other enumerable data structure.

//sync pattern

using var objectStream = dataSource.From<Employee>(new { Title = uniqueKey }).ToObjectStream<Employee>().Execute();
foreach (var item in objectStream)
{
	Assert.AreEqual(uniqueKey, item.Title);
}

//async pattern

await using var objectStream = await dataSource.From<Employee>(new { Title = uniqueKey }).ToObjectStream<Employee>().ExecuteAsync();
await foreach (var item in objectStream)
{
	Assert.AreEqual(uniqueKey, item.Title);
}

It is vital that the object stream is disposed after use. If that doesn't occur, the database can suffer from thread exhaustion or deadlocks.