Guaranteed eventual consistency

[Timovzl]

What are recommended ways to achieve guaranteed eventual consistency in Orleans?

I am talking about the following scenario:

• We are mutating a grain and saving its state.
• As a part of that occurrence, one or more additional things need to eventually happen. Other grains need to make followup changes in response. (Example: A monetary transaction succeeds. At some point, someone's balance needs to be incremented as a result.)
• The followup changes are guaranteed to be possible. (Example: We can always increase someone's balance. Counterexample: Decreasing someone's balance requires it to be positive at that point, and thus is not a relevant use case for this discussion.)
• The time constraints are loose. Think hours.
• The followup work is idempotent.
• If the grain state is saved successfully, then the follow changes must happen at some point. If transient failures prevent them from happening, they must be retried. If permanent failures prevent them from happening (or we run out of retries), an alerting mechanism must kick in.
• If the grain state is not saved (perhaps due to a failure, anywhere), then the followup changes must not happen.
• For emphasis: Node failures, or any other failures, must not cause the constraints to be violated.

For the discussion, I don't think it matters much whether our mental model is that of an event with handlers or simply one where the originating grain needs to make some followup requests.

To further illustrate the scenario, here is one way in which we might traditionally solve the problem:

• Start a database transaction.
• In the database, save the mutation.
• In the database, save the fact that we still need to do followup work, i.e. save some deferred operation.
• Commit. The OLTP part is done.
• Out-of-band, periodically pick up deferred operations. Process them idempotently, and remove them on success.
• Either retry indefinitely and alert if a pending operation has become older than expected, or limit the retry count and alert if we run out.

Such a system allows us guarantee that we can keep distributed systems in-sync eventually, or in the very worst case, get an alert when we fail to.

Can you recommend a way to achieve the described constraints with Orleans?

[JorgeCandeias]

TLDR; This is what sagas are for, but there are simpler ways to go about it, such as plain queues.

Note that any approach we use requires some "thing" to act as a coordinator for eventual consistency. The responsibility to ensure everything eventually happens, rain or shine, needs to lie somewhere. In a transactional case, for example, it lies with whatever is acting as a transaction coordinator, if it's capable of self-retrying. With a job retry scheme, the responsibility lies with whatever code is retrying in the first place and keeping tabs on completed steps. In a queue case, it's the queue itself, by making the item available again if the consumer did not confirm receipt within a timeout. Something to keep in mind when looking at options.

And here are a couple:

• Just use a set of queues on some external provider that supports delivery confirmation by the consumer. This is the vanilla approach to these problems and therefore the benchmark to compare other options with. Stuff doesn't have to go through Orleans if it doesn't make sense to do so.

(if you still want to use Orleans):

• Use a grain to manage the job and keep tabs on individual steps and retry failed ones. Make the grain persistent. Implement a reminder to make the grain immortal until the job is complete. The coordinator here is the managing grain itself.
• Use a persistent streaming provider that supports at-least-once delivery. Make subscribing grains idempotent in their stream message handling. Implicit subscriptions help here. Each grain representing the first step can take a message from Stream A, do its work and then push to the result to Stream B. The second grain can take from Stream B, do its work and push to Stream C. And so on. These grains can notify some shared grain that keeps tabs on the overall state of the job. The coordinator here is the streaming provider, assuming it is capable of re-delivering on consumer failure, e.g. by handling the underlying queue delivery confirmation call.
• Implement the above but using grains as persistent work queues with delivery confirmation. This is less sophisticated, and more code, but it works well in some scenarios, as queue grains can also actively push items, instead of just letting grains ask for them. Coordination is shared by all the queue grains you end up having.
• Use a Saga pattern. Grain A makes a list of steps and targets and gives it to Grain B. Grain B does it work, marks the step as done on the list and passes it to the next grain declared on the list, e.g. Grain C. This repeats until the list is complete. The list structure itself can change within the saga, so this allows for dynamic workflows, where you don't know the full structure upfront, and implementation can stay highly decoupled. This pattern can be implemented in-band (all the calls are chained in one go), or out-of-band (Grain B replies to Grain A after confirming persisting the list but before starting to work on it in the background, e.g. on a timer). All grains implement reminders to keep them awake until their step completes. The responsibility here is shared across all the grains that form part of the saga. The first agent in the chain (client or grain) has the responsibility to retry the call, to make sure the entire saga executes if in-band, or at least the first step kicks off, if out-of-band.

These are just from the top of my head.

[Timovzl]

Thanks a lot for the detailed reply, @JorgeCandeias. Great summary and suggestions.

One option that appeals to me for its simplicity is including the "I still need to do something" as part of the grain's state, and using a reminder to perform it eventually. I note that both the reminder and the grain state need to be saved. What would you recommend with regards to that?

1. Is it possible to save the state and the new reminder as a single transaction?
2. We could have the reminder always running, but that would be overly wasteful if there are many grains.
3. We could save the reminder first and the state after. If a failure occurs in between, the grain's custom ReceiveReminder() implementation should be resilient enough to recognize that no intention was stored and thus do nothing. For clarification, is the reminder persisted on-the-spot by RegisterOrUpdateReminder()?

[JorgeCandeias]

TLDR; Option 3.

1. Is it possible to save the state and the new reminder as a single transaction?

It's possible, if both persistence and reminders providers support distributed transactions using a context (e.g. with TransactionScope), but implementing this breaks the abstraction that Orleans provides, as you're relying on them supporting this. Performance will also take a hit even on the happy path, if this hack works at all, so I don't recommend this.

Orleans does support managed transactions, but this doesn't apply to reminders.

2. We could have the reminder always running, but that would be overly wasteful if there are many grains.

When a reminder is no longer needed, we can, and should, clear it. Same goes for grain state and the grain activation itself. So, if well managed, grains will self-regulate over time and work fine.

3. We could save the reminder first and the state after.

Of the three options, I suggest this one first.

For clarification, is the reminder persisted on-the-spot by RegisterOrUpdateReminder()?

Yes, make sure to await it.

In general I suggest making grain methods internally idempotent, so the caller can safely retry as many times as needed, until the call fully succeeds. Making them idempotent, by virtue of ordering internal steps correctly, lets us avoid the performance drop of transactions, and still handle the minimal cases where calls fail.

So in your case, I suggest option 3, plus adding some simple retry logic on the caller - we can do this in a few lines of Polly code anyway. And if the caller itself fails to retry until success, and leaves the grain with a reminder and no state, then the reminder can handle that, and remove itself when it ticks.

[Timovzl]

Perfect. Agreed.

And the retry logic in the caller depends on its scenario, of course. If it itself is handling some form of eventual consistency, it will usually want to retry. If it originates from a user request, it will usually just want to forward the failure. The very origin of the call chain is usually a great place to determine how to deal with failures, IMHO.

Finally, just to be sure, reminders are invoked on their own turn, correct? In other words, can we be certain that the current operation is completed before the reminder is invoked? That would allows us to set the "due time" extremely short without worry of the corresponding state being saved too late.

[JorgeCandeias]

Finally, just to be sure, reminders are invoked on their own turn, correct?

Reminder callbacks are scheduled as external calls, so they respect whatever reentrancy rules the grain is configured for. So they won't interleave by default, but they can if we want them to.