Avoidable Concurrency Exception?

[ylbillyli]

I run into situation where two event subscribers firing commands against the same aggregate in parallel (because eventflow's event dispatcher dispatch events to subscriber in parallel using "WhenAll()"), which was causing a concurrency exception. Although they are handled fine by the retry, these commands involve calling other integrated systems which cannot be roll-back/redo ending up duplicated operations.

This might be best solved by designing the integration strategy better to external systems, but it is too late for our project and I have solved the problem by using the "Keyed" AsyncLocker in http://stackoverflow.com/questions/31138179/asynchronous-locking-based-on-a-key and modified the command bus class adding a lock around the Loading of Aggregate until it is committed using the AggregateId as the locking key, such that multiple commands firing against the same aggregate will be synchronized.

My main argument here is, when multiple commands firing against the same aggregate at the same time which would guarantee a race condition, could acutally be avoided by thread safe locking. It make sense to me to prevent multiple threads modifying the same aggregate at the same time.

[rasmus]

If I understand you problem correctly, the problem is that multiple instances of the same aggregate are invoked simultaneously due to commands fired from e.g. subscribers.

Concurrency in EventFlow is maintained through the event store, which ensures that only one set of events are committed when multiple attempts to change the aggregate are done. You can see this in the event store integration test TestSuiteForEventStore.OptimisticConcurrency.

If I understand your problem correct, you solution will only hold if the application and the execution of commands are limited to a single server. In our setup, we have multiple web servers running and each are accessing the same aggregate through web requests. In this case, another server might load the aggregate, perform an update, and then commit it before any of the subscribers are finished processing.

1. Server 1 executes command C1, which results in event E1
2. Server 1 starts processing subscribers for E1
3. Server 2 executes command C2, which results in event E2
4. Server 1 subscribers have already started processing based on E1 but commands fail due to an optimistic concurrency exception as E2 have been committed to the event store

EventFlow only ensures that events are correctly ordered within a single aggregate as providing any other synchronization would be hard when having an application spanning multiple servers.

I hope I understand problem correctly and please correct me if I'm wrong.

P.S.: You could use the decorator pattern and decorating the ICommandBusinstead of modifying the EventFlow code.

[rasmus]

Just a clarification, synchronization across several servers is hard to generalize and if your application has that requirement, you should implement it using through e.g. the decorator pattern described in the customize documentation for EventFlow.

If your application has these synchronization issues, then it might (and I stress the might again), be due to an inaccurate model of your domain as your subscribers are required to finish in a specific order.

Again, I would like to know if my comments are completely off target.

[rasmus]

@ylbillyli Would using the AsyncLocker create a deadlock if you have a chain of subscribers executing commands.

• Command C1 results in event E1 on aggregate A1
• Subscriber to E1 fires Command C2
• Command C2 results in event E2 on aggregate A2
• Subscriber to E2 fires Command C3
• Command C3 is deadlocked as it cannot access A1

[ylbillyli]

@rasmus Thanks a lot for the reply and details. You were completely right about my problem.

I agree with you that the concurrency exception is still required for multi server as they are hard to sync, but how about on the same server? Should/Could we prevent race condition on the same server to reduce the chance of this exception? (My project only uses single server anyway)

In regards to the domain modelling, in my context, the two subscribers have no dependency in each other and could be done in any order. In fact, the second subscriber is added later to provide an additional feature required by the business, after which we encountered this problem. With my limited experience in DDD world, I am not confident to tell whether our modelling is valid. Do you think in real world it is valid to have two independent subscribers firing commands simultaneously against the same aggregate? Couldn't think of a better example but let say OrderReceived fires ShipProduct and GenerateInvoice on the FactoryAggregate at the same time?

For deadlock, this is what I worry too, but couldn't come up with an example yet. I might understand it wrong, but I don't think your example will encounter deadlock. If you look at my screenshot above, I am not locking on the whole PublishAsync(), but only locking within the _transientFaultHandler.TryAsync body, with a using block starting from _eventStore.LoadAggregateAsync until return await aggregate.CommitAsync. Since events are dispatched to subscribers only after the events are committed, the lock on A1 is released before E1 is dispatched, which means way before C3 is fired.

This is also one of the reason why I couldn't use the decorator pattern which I actually love quite much on the framework. I only want to lock preciously in the middle of the PublishAsync instead of the whole thing, decorator won't work unless this PublishAsync is split into smaller pieces. On the other hand, I did decorate the IOptimisticConcurrencyRetryStrategy to retry only for OptimisticConcurrencyException instead of all Exception.

Any further comment would be greatly appreciated. The main point I would like to discuss further is, do you agree that access to the same aggregate on the same server should be sychned by locking to avoid concurrency ex?

[ylbillyli]

In addition to the deadlock problem mentioned above, I think it will deadlock only when CommandHandler firing another command which is incorrect I think.

[rasmus]

@ylbillyli In our system we have several commands that fire on the same aggregate at the same time, but these almost never represent a domain error.

Typically the flow would be something similar to this.

• Command C1 starts to execute and loads aggregate with version 1
• Command C2 starts to execute and loads aggregate with version 1
• Command C1 finishes and event E1 is committed to the event store and aggregate is now version 2, event E1 is published to subscribers
• Command C2 finishes and resulted in E2 but an optimistic concurrency exception occurs when trying to commit to the event store as the aggregate has been updated
• Command C2 is now retried on aggregate version 2, the command is still valid and event E3 is committed to the event store and published to subscribers

As long as the operations are valid in your domain, then the order doesn't matter. As an example, changing the title first and then the description on an item is basically the same as first changing the description and then the title.

The important thing to remember is to ensure that the methods of the aggregate doesn't have any side effects, that way it doesn't matter if they are retired any number of times. Retrying commands due to optimistic concurrency exceptions is key for EventFlow, as its to be expected that many requests and/or threads try to manipulate the same aggregates. In a traditional DB world you would have a pessimistic lock in your application that would block other operations, but when working with event sourcing you try to avoid that.

We do have situations where commands are fire, retired and then fail with a domain error as the operation is no longer valid, but that's to be expected in any application and its important how you handle these situations. In our case its usually simple though, as we can merely present an error message to the user. These situations are typical when a user has more than one browser tab open and tries to manipulate items in more than one.

If you have any other comments, please let me know. Especially if there's a problem with EventFlow.

[ylbillyli]

@rasmus thanks for the comment. This make it a lot clearer for me. I still have a question. If C1 and C2 are fired on the same server, do you still prefer relying on concurrency exception auto retry more than completely avoiding it with synchronized locking?

I have been using EventFlow for nearly half year now and found that it is really great and easy to use. The only thing that I am still not quite familiar with is how to use EventFlow to talk to external system that are not event sourced. In many situation including our project, we need to integrate with multiple external systems that only expose there interface via a dll business layer or REST api. They are not eventflow aggregate, however, correctly me if I am wrong, eventflow commands must be fired against eventflow aggregate and eventflow events can only be emited by an eventflow aggregate. The way we get around this is to wrap their API in an eventflow aggregate such that they can receive commands and emit events. However, as they are not event sourced, any re-try attempts will end up duplicated operations. Do you think I may be using Eventflow wrong or are there a better way to achieve this?

[JC008]

@rasmus, @ylbillyli The bridging of external systems to an EventFlow implementation is why I asked for DistinctCommand to be added to identify duplicate commands and handle them in the ICommandHandler for an ICommand.

[rasmus]

@ylbillyli Its great to get some feedback on EventFlow, thanks.

I would prefer the optimistic concurrency exception to the pessimistic locking as in most systems it happens rarely. I know its basically using exceptions for control flow but its modeled around how e.g. the MSSQL event store detects the collision.

For systems that issue requests to EventFlow aggregates, yes, that would have to go through a command, e.g. publish from a REST endpoint.

For when EventFlow has to issue requests to other systems it gets a bit hard and it basically comes down to how much you of it you would like to model the interaction with the external system. Here's are two examples on how it could be modeled.

• Keep it totally out of your aggregates and model it using an application service as e.g. the BookingApplicationService is modeled i shipping example found in the EventFlow code base.
• If the external system interaction is important for the domain, then you could model it using a series of events, e.g. ExternalResourceRequestedEvent -> ExternalResourceAcceptedEvent/ExternalResourceDeniedEvent and call the external system from a subscriber.

We have used the last model for e.g. requesting payments. We emit a PaymentRequestedEvent which a subscriber listens to, tries to actually perform the payment and then issues either a payment succeeded or failed command depending on the outcome. In this case we have the entire flow represented in our domain model.

Again, it depends if the interaction is important enough to be represented in your domain.

[ylbillyli]

Thanks for the comments. They are very helpful.

[rasmus]

@ylbillyli Can I close the issue? Did you get all the answers you need? If not, then let me know.

[ylbillyli]

@rasmus sure. All good. Thanks again.