Skip to content

fix: true streaming reads in KurrentDB store and paged read-to-end API #568

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
alexeyzimarev wants to merge 1 commit into
base: dev
Choose a base branch
from
Open

fix: true streaming reads in KurrentDB store and paged read-to-end API #568

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions src/Core/src/Eventuous.Persistence/EventStore/IEventReader.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,10 @@ public interface IEventReader {
/// <summary>
/// Read a fixed number of events from an existing stream as an async enumerable.
/// Throws <see cref="StreamNotFound"/> if the stream does not exist.
/// Implementations either stream events as they arrive from the store, or buffer up to <paramref name="count"/>
/// events before yielding, so memory usage can grow with <paramref name="count"/>. To read a whole stream,
/// use <see cref="StoreFunctions.ReadStreamToEnd"/>, which reads in pages, instead of passing
/// <see cref="int.MaxValue"/> as the count.
/// </summary>
/// <param name="stream">Stream name</param>
/// <param name="start">Where to start reading events</param>
Expand All @@ -18,6 +22,8 @@ public interface IEventReader {
/// <summary>
/// Read a number of events from a given stream, backwards (from the stream end).
/// Throws <see cref="StreamNotFound"/> if the stream does not exist.
/// Implementations either stream events as they arrive from the store, or buffer up to <paramref name="count"/>
/// events before yielding, so memory usage can grow with <paramref name="count"/>.
/// </summary>
/// <param name="stream">Stream name</param>
/// <param name="start">Where to start reading events</param>
Expand Down
72 changes: 57 additions & 15 deletions src/Core/src/Eventuous.Persistence/EventStore/StoreFunctions.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,8 @@
// Copyright (C) Eventuous HQ OÜ. All rights reserved
// Licensed under the Apache License, Version 2.0.

using System.Runtime.CompilerServices;

namespace Eventuous;

public static class StoreFunctions {
Expand Down Expand Up @@ -148,6 +150,59 @@ CancellationToken cancellationToken
}
}

/// <summary>
/// Reads a stream from the given position to the end, as an async enumerable.
/// Events are read in pages of <paramref name="pageSize"/> and yielded as they arrive, so the whole stream
/// is never buffered in memory. Use this instead of calling <see cref="IEventReader.ReadEvents"/>
/// with <see cref="int.MaxValue"/> as the count.
/// </summary>
/// <param name="streamName">Name of the stream to read from</param>
/// <param name="start">Stream position to start reading from</param>
/// <param name="pageSize">Number of events to read per page. It caps the amount of events a buffering
/// implementation of <see cref="IEventReader"/> holds in memory at a time.</param>
/// <param name="failIfNotFound">Set to false to complete without yielding anything when the stream isn't found,
/// instead of throwing <see cref="StreamNotFound"/>. Default is true.</param>
/// <param name="cancellationToken">Cancellation token</param>
/// <returns>An async enumerable of events retrieved from the stream</returns>
public async IAsyncEnumerable<StreamEvent> ReadStreamToEnd(
StreamName streamName,
StreamReadPosition start,
int pageSize = 500,

@chatgpt-codex-connector chatgpt-codex-connector Bot Aug 12, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Reject non-positive page sizes

When pageSize is zero, readers such as SqlEventStoreBase return an empty page, but yielded < pageSize is false, so the outer loop repeats indefinitely and continuously queries the store until cancellation. Negative values can behave similarly for readers that treat non-positive counts as empty. Validate that this public argument is greater than zero before entering the paging loop.

Useful? React with 👍 / 👎.

bool failIfNotFound = true,
[EnumeratorCancellation] CancellationToken cancellationToken = default
) {
var position = start;

while (true) {
var yielded = 0;
long lastRevision = 0;

await using var enumerator = eventReader.ReadEvents(streamName, position, pageSize, cancellationToken).GetAsyncEnumerator(cancellationToken);

while (true) {
bool moved;

try {
moved = await enumerator.MoveNextAsync().NoContext();
} catch (StreamNotFound) when (!failIfNotFound) {
yield break;
}

if (!moved) break;

var evt = enumerator.Current;
yielded++;
lastRevision = evt.Revision;

yield return evt;
}

if (yielded < pageSize) yield break;

@chatgpt-codex-connector chatgpt-codex-connector Bot Aug 12, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Track source-page exhaustion instead of yielded events

When a KurrentDB page contains an event that EnumerateStream suppresses, such as an unresolved $> link event whose deserialization fails, the enumerable yields fewer than pageSize items even though the underlying raw page was full and later pages exist. Treating the number of yielded user events as proof that the source reached its end therefore makes ReadStreamToEnd silently omit the remaining events; page exhaustion must be tracked independently of filtered events.

Useful? React with 👍 / 👎.


position = new(lastRevision + 1);
}
Comment on lines +200 to +203

@qodo-free-for-open-source-projects qodo-free-for-open-source-projects Bot Aug 12, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Action required

1. Infinite loop on pagesize 🐞 Bug ☼ Reliability

StoreFunctions.ReadStreamToEnd can loop forever when pageSize <= 0 because it only exits when
yielded < pageSize, which is false for yielded == 0 and pageSize <= 0, so it keeps advancing
position and issuing empty reads.
Agent Prompt 
### Issue description
`ReadStreamToEnd` does not validate `pageSize`. When `pageSize <= 0`, the paging loop can become non-terminating (especially for implementations that yield no events for `count <= 0`), repeatedly performing empty reads.

### Issue Context
This method is a new public read-to-end API intended for safe, bounded-memory paging. Invalid `pageSize` values should fail fast (or be normalized) to prevent hangs.

### Fix Focus Areas
- src/Core/src/Eventuous.Persistence/EventStore/StoreFunctions.cs[167-204]

### Suggested fix
- Add an argument guard at the start of `ReadStreamToEnd`, e.g.:
  - `if (pageSize <= 0) throw new ArgumentOutOfRangeException(nameof(pageSize), "pageSize must be > 0");`
- (Optional) Add a regression test ensuring `pageSize: 0` (and negative) throws `ArgumentOutOfRangeException`.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

}

/// <summary>
/// Reads a stream from the event store to a collection of <seealso cref="StreamEvent"/>
/// </summary>
Expand All @@ -163,23 +218,10 @@ public async Task<StreamEvent[]> ReadStream(
bool failIfNotFound = true,
CancellationToken cancellationToken = default
) {
const int pageSize = 500;

var streamEvents = new List<StreamEvent>();

var position = start;

try {
while (true) {
var events = await eventReader.ReadEvents(streamName, position, pageSize, failIfNotFound, cancellationToken).NoContext();
streamEvents.AddRange(events);

if (events.Length < pageSize) break;

position = new(position.Value + events.Length);
}
} catch (StreamNotFound) when (!failIfNotFound) {
return [];
await foreach (var evt in eventReader.ReadStreamToEnd(streamName, start, failIfNotFound: failIfNotFound, cancellationToken: cancellationToken).NoContext(cancellationToken)) {
streamEvents.Add(evt);
}

return [.. streamEvents];
Expand Down
96 changes: 96 additions & 0 deletions src/Core/test/Eventuous.Tests.Persistence.Base/Store/Read.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -146,6 +146,102 @@ public async Task ShouldReturnWhenReadingBackwards(CancellationToken cancellatio
await Assert.That(result.Length).IsEqualTo(5);
}

[Test]
[Category("Store")]
public async Task ShouldThrowWhenReadingMissingStream(CancellationToken cancellationToken) {
var streamName = Helpers.GetStreamName();

await Assert.ThrowsAsync<StreamNotFound>(() => _fixture.EventStore.ReadEvents(streamName, StreamReadPosition.Start, 10, true, cancellationToken));
}

[Test]
[Category("Store")]
public async Task ShouldThrowWhenReadingMissingStreamBackwards(CancellationToken cancellationToken) {
var streamName = Helpers.GetStreamName();

await Assert.ThrowsAsync<StreamNotFound>(() => _fixture.EventStore.ReadEventsBackwards(streamName, StreamReadPosition.End, 10, true, cancellationToken));
}

[Test]
[Category("Store")]
public async Task ShouldReadStreamToEnd(CancellationToken cancellationToken) {
object[] events = [.. _fixture.CreateEvents(25)];
var streamName = Helpers.GetStreamName();
await _fixture.AppendEvents(streamName, events, ExpectedStreamVersion.NoStream);

var result = new List<StreamEvent>();

await foreach (var evt in _fixture.EventStore.ReadStreamToEnd(streamName, StreamReadPosition.Start, pageSize: 10, cancellationToken: cancellationToken)) {
result.Add(evt);
}

IEnumerable<object> actual = result.Select(x => x.Payload)!;
await Assert.That(actual).IsEquivalentTo(events);
}

[Test]
[Category("Store")]
public async Task ShouldReadStreamToEndWithExactPageMultiple(CancellationToken cancellationToken) {
object[] events = [.. _fixture.CreateEvents(20)];
var streamName = Helpers.GetStreamName();
await _fixture.AppendEvents(streamName, events, ExpectedStreamVersion.NoStream);

var result = new List<StreamEvent>();

await foreach (var evt in _fixture.EventStore.ReadStreamToEnd(streamName, StreamReadPosition.Start, pageSize: 10, cancellationToken: cancellationToken)) {
result.Add(evt);
}

IEnumerable<object> actual = result.Select(x => x.Payload)!;
await Assert.That(actual).IsEquivalentTo(events);
}

[Test]
[Category("Store")]
public async Task ShouldReadStreamToEndFromPosition(CancellationToken cancellationToken) {
object[] events = [.. _fixture.CreateEvents(25)];
var streamName = Helpers.GetStreamName();
await _fixture.AppendEvents(streamName, events, ExpectedStreamVersion.NoStream);

var result = new List<StreamEvent>();

await foreach (var evt in _fixture.EventStore.ReadStreamToEnd(streamName, new(10), pageSize: 10, cancellationToken: cancellationToken)) {
result.Add(evt);
}

var expected = events.Skip(10);
var actual = result.Select(x => x.Payload!);
await Assert.That(actual).IsEquivalentTo(expected);
}

[Test]
[Category("Store")]
public async Task ShouldThrowWhenReadingMissingStreamToEnd(CancellationToken cancellationToken) {
var streamName = Helpers.GetStreamName();

await Assert.ThrowsAsync<StreamNotFound>(ReadFunc);

return;

async Task ReadFunc() {
await foreach (var _ in _fixture.EventStore.ReadStreamToEnd(streamName, StreamReadPosition.Start, cancellationToken: cancellationToken)) { }
}
}

[Test]
[Category("Store")]
public async Task ShouldReturnNothingWhenReadingMissingStreamToEnd(CancellationToken cancellationToken) {
var streamName = Helpers.GetStreamName();

var result = new List<StreamEvent>();

await foreach (var evt in _fixture.EventStore.ReadStreamToEnd(streamName, StreamReadPosition.Start, failIfNotFound: false, cancellationToken: cancellationToken)) {
result.Add(evt);
}

await Assert.That(result).IsEmpty();
}

[Test]
[Category("Store")]
public async Task ShouldThrowWhenReadingBackwardsFromNegativePosition(CancellationToken cancellationToken) {
Expand Down
87 changes: 47 additions & 40 deletions src/KurrentDB/src/Eventuous.KurrentDB/KurrentDBEventStore.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -216,48 +216,63 @@ EventData ToEventData(NewStreamEvent streamEvent) {
}

/// <inheritdoc/>
public async IAsyncEnumerable<StreamEvent> ReadEvents(StreamName stream, StreamReadPosition start, int count, [EnumeratorCancellation] CancellationToken cancellationToken = default) {
var read = _client.ReadStreamAsync(Direction.Forwards, stream, start.AsStreamPosition(), count, cancellationToken: cancellationToken);

var events = await TryExecute(
async () => {
var resolvedEvents = await read.ToArrayAsync(cancellationToken).NoContext();

return ToStreamEvents(resolvedEvents);
},
public IAsyncEnumerable<StreamEvent> ReadEvents(StreamName stream, StreamReadPosition start, int count, CancellationToken cancellationToken = default)
=> EnumerateStream(
() => _client.ReadStreamAsync(Direction.Forwards, stream, start.AsStreamPosition(), count, cancellationToken: cancellationToken),
stream,
true,
() => new("Unable to read {Count} starting at {Start} events from {Stream}", count, start, stream),
(s, ex) => new ReadFromStreamException(s, ex)
cancellationToken
);

foreach (var evt in events) yield return evt;
}

/// <inheritdoc/>
public async IAsyncEnumerable<StreamEvent> ReadEventsBackwards(StreamName stream, StreamReadPosition start, int count, [EnumeratorCancellation] CancellationToken cancellationToken = default) {
var read = _client.ReadStreamAsync(
Direction.Backwards,
public IAsyncEnumerable<StreamEvent> ReadEventsBackwards(StreamName stream, StreamReadPosition start, int count, CancellationToken cancellationToken = default)
=> EnumerateStream(
() => _client.ReadStreamAsync(Direction.Backwards, stream, start.AsStreamPosition(), count, resolveLinkTos: true, cancellationToken: cancellationToken),
stream,
start.AsStreamPosition(),
count,
resolveLinkTos: true,
cancellationToken: cancellationToken
() => new("Unable to read {Count} events backwards from {Stream}", count, stream),
cancellationToken
);

var events = await TryExecute(
async () => {
var resolvedEvents = await read.ToArrayAsync(cancellationToken).NoContext();
// Events are yielded as they arrive from the server, so a read holds at most one
// deserialized event at a time, regardless of the requested count.
// The exception mapping wraps each advance of the source enumerator instead of the whole
// loop because iterators can't yield from inside a try block with a catch clause.
async IAsyncEnumerable<StreamEvent> EnumerateStream(
Func<IAsyncEnumerable<ResolvedEvent>> read,
string stream,
Func<ErrorInfo> getError,
[EnumeratorCancellation] CancellationToken cancellationToken
) {
await using var enumerator = read().GetAsyncEnumerator(cancellationToken);

return ToStreamEvents(resolvedEvents);
},
stream,
true,
() => new("Unable to read {Count} events backwards from {Stream}", count, stream),
(s, ex) => new ReadFromStreamException(s, ex)
);
while (true) {
var moved = false;
StreamEvent? streamEvent = null;

try {
moved = await enumerator.MoveNextAsync().NoContext();

foreach (var evt in events) yield return evt;
if (moved) streamEvent = ToStreamEvent(enumerator.Current);
} catch (StreamNotFoundException) {
LogStreamStreamNotFound(stream);

throw new StreamNotFound(stream);
} catch (OperationCanceledException) {
throw;
} catch (Exception ex) {
var (message, args) = getError();
// ReSharper disable once TemplateIsNotCompileTimeConstantProblem
#pragma warning disable CA2254
_logger.LogWarning(ex, message, args);
#pragma warning restore CA2254

throw new ReadFromStreamException(stream, ex);
}
Comment on lines +262 to +270

if (!moved) yield break;

if (streamEvent != null) yield return streamEvent.Value;
}
}

/// <inheritdoc/>
Expand Down Expand Up @@ -362,14 +377,6 @@ StreamEvent AsStreamEvent(object payload)
);
}

StreamEvent[] ToStreamEvents(ResolvedEvent[] resolvedEvents)
=> [
.. resolvedEvents
.Select(ToStreamEvent)
.Where(x => x != null)
.Select(x => x!.Value)
];

record ErrorInfo(string Message, params object[] Args);

[LoggerMessage(LogLevel.Warning, "Stream {stream} not found")]
Expand Down
Loading
Loading