-
-
Notifications
You must be signed in to change notification settings - Fork 445
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add testing helpers for async projections #2624
Comments
What you're asking for here is syntactic sugar on top of the existing functionality we already use today in testing the async daemon. See this: https://github.com/JasperFx/marten/blob/master/src/Marten/Events/Daemon/ShardStateTracker.cs#L92 It's using IObservables, and the daemon already publishes events for its progress. We could easily just expose that on IProjectionDaemon and add your suggested API's if you want. |
I found this tracking helper but I didn't have much luck with it. https://discord.com/channels/1074998995086225460/1074999076896112661/1123886607695618079 |
I think that it’d be good to expose such methods to make it more accessible plus document them. |
This is a little bigger than it might have sounded because of the potential for multi-tenancy. Will have to be done database by database. |
I'm pushing this back since there are usable workarounds right now |
hi guys, found this issue while dealing with integration tests as well 😄 inspired by your suggestions, this gist helped me to await all projections: // when you start with a cold DB (like a clean db in docker), AllProjectionProgress will return 0 projections
// so we need to await that as well
var asyncProjectionsCount =
_store.Options.Events.Projections().Count(x => x.Lifecycle == ProjectionLifecycle.Async)
+ 1 /* one additional for HighWaterMark projection */;
var awaitTask = Task.Run(async () =>
{
do
{
await Task.Delay(TimeSpan.FromSeconds(1), ct);
var statistics = await _store.Storage.Database.FetchEventStoreStatistics(ct);
var projections = await _store.Storage.Database.AllProjectionProgress(ct);
if (projections.Count == asyncProjectionsCount && projections.All(x => x.Sequence >= statistics.EventSequenceNumber)) return;
} while (true);
}, ct);
await Task.WhenAny(awaitTask, Task.Delay(TimeSpan.FromSeconds(15), ct)); i placed it into a test endpoint and i'm calling this endpoint from tests before other code that requires projections |
I'm going to take this today, and pretty well do a polling approach like @a-shtifanov-laya was doing so there's no issue about what or where the daemon is already running. |
@jeremydmiller It seems that This is what we do right now in our WebApplicationFactory subclass to support our tests which relies on async projections (heavily based on the comment of @a-shtifanov-laya above). We do not use database per tenant yet, so that would complicate this a bit. public class MyWebApplicationFactory : WebApplicationFactory<Program>, IAsyncLifetime, ICollectionFixture<MyWebApplicationFactory>
{
// ....
public async Task WaitForAsyncProjections()
{
CancellationTokenSource cts = new(10.Seconds());
// Since WaitForNonStaleProjectionDataAsync does not wait until the projections exists in the tables,
// we need to wait for the async daemon to prepare first. We do that by waiting for the projection count
// is the same in both configuration and actual database data.
IDocumentStore store = Services.GetRequiredService<IDocumentStore>();
IReadOnlyProjectionData[] asyncProjections = store.Options.Events.Projections()
.Where(x => x.Lifecycle == ProjectionLifecycle.Async)
.ToArray();
while (!cts.IsCancellationRequested)
{
IReadOnlyList<ShardState> projectionProgress = await store.Storage.Database.AllProjectionProgress(cts.Token);
if (projectionProgress.Count == asyncProjections.Length + 1) // +1 to include HighWaterMark
break;
await Task.Delay(100.Milliseconds(), cts.Token);
}
await store.WaitForNonStaleProjectionDataAsync(10.Seconds());
}
} |
And now that I got a bit further, it seems marten/src/Marten/Events/AsyncProjectionTestingExtensions.cs Lines 67 to 72 in 61de7fc
AllProjectionProgress returns an IReadOnlyList<ShardState> .
|
Some of my tests require async projections to be up-to-date in order to be successful.
Let's say I build a test targeting code that queries models created by an async projection:
The second step today is a bit cumbersome.
I tried a suggestion by @oskardudycz using an
IDocumentSessionListener
that writes projection update information to aSystem.Threading.Channels.Channel
. Later I could read from the channel, but every event can only be read once. So I needed to cache all information a second time in aConcurrentBag
. When asserting whether a (>=1) projection was updated I first inspected the bag and if nothing was found attempted to read from the channel. It kind of worked, but also was unreliable (flaky tests, even with an additionalThread.Sleep
for good measure) and tedious to define all necessary projections for every integration scenario.The solution I currently use disables the Async Daemon for testing scenarios and explicitly runs the daemon to update all projections w.r.t. the current event count. It's reliable and works so far.
Since I think that I am not the only person to work with async projections I believe this "wait for everything" to be up-to-date should be generalized and easy to access. Especially it should not require disabling the async daemon for testing scenarios, but rather asking it to be up to date within some timeout. I tried that, but I could not find a way to access the running agent from the
IDocumentStore
.The APIs that would be helpful:
store.WaitForAllAsyncProjections(timeout)
store.WaitForAsyncProjection<TModel>(id, timeout)
store.WaitForAsyncProjection<TModel>(predicate, timeout)
The text was updated successfully, but these errors were encountered: