Handling failures of WriteStateAsync

[juho-hanhimaki]

public async Task SetNameAsync(string name)
{
    _profile.State.Name = name;
    await _profile.WriteStateAsync();
}

Let's say the persistence layer (Azure Storage) is down. My understanding is that the in-memory state of the grain (the Name) will be left modified, even though an exception will be returned to the caller. I have hard time understanding how this could be desirable behaviour as now the persisted state and in-memory state are inconsistent. None of the samples I have seen address this issue in any manner.

So how should one resolve the issue? What comes to mind is making the state immutable and reverting back to original state object if WriteStateAsync fails. C# record types are immutable by default so using them is a natural fit.

    [Immutable]
    [Serializable]
    public record ProfileState(string Name);

    public async Task SetNameAsync(string name)
    {
        var originalState = _profile.State;
        _profile.State = originalState with { Name = name };
        try
        {
            await _profile.WriteStateAsync();
        }
        catch (Exception)
        {
            _profile.State = originalState;
            throw;
        }
    }

Does this make sense? I find it extremely weird that none of the documentation/samples I have seen address this issue.

[ReubenBond]

What you're suggesting makes sense. One option is to copy the state before modification using DeepCopier/DeepCopier<T> (which you can inject) so that you can restore it after a failed update attempt. A potentially cheaper alternative is to set a dirty flag and to ensure that state is reloaded from storage if the flag is set when necessary. You can implement that using IIncomingGrainCallFilter or by wrapping the state type in your own type which manages the flag.

This strange event sourcing toy uses the approach of deactivating the grain if it has a storage error: https://github.com/ReubenBond/OrleansEventJournal/blob/8fc000c469bb1aba58b9c9497877362093911e6c/EventJournal/JournaledActor.cs#L347

[juho-hanhimaki]

Thanks, wasn't aware of the dirty flag approach yet! It would be nice alternative.

Having some documented guidance around this matter could be very useful for newcomers as it is probably something most developers face as soon as they try build something actually useful.

[juho-hanhimaki]

On a deeper thought marking the state dirty would be really bad for availability of read operations when the storage is down as the next read would require re-reading the state from storage. So, it is not as nice of an option as it initially sounded.

[ReubenBond]

The copying approach is rather similar. It's a trade-off between performance on the happy path and availability in the event of a storage outage. If storage is down, though, you're going to have a bad time regardless, so I'm not sure it's worth it (but you know your domain)

[juho-hanhimaki]

Came up with potential solution using the IIncomingGrainCallFilter in combination with immutable state.

    public async Task Invoke(IIncomingGrainCallContext context)
    {
        var originalEtag = _state.Etag;
        var originalState = _state.State;
        try
        {
            await context.Invoke();
        }
        catch (Exception)
        {
            if (_state.Etag == originalEtag)
            {
                // As the Etag hasn't changed, there is a potential that WriteStateAsync did fail, make sure the original state is preserved.
                _state.State = originalState;
            }
            throw;
        }
    }

I guess there is a small chance that Etag collides, so this might need some refinement. I think it would be best to catch storage provider specific exceptions directly so there is absolute certainty that exception originated from persistence.

If storage is down, though, you're going to have a bad time regardless, so I'm not sure it's worth it (but you know your domain)

There is definitely some value in having the lucky in-memory grains available as read-only on storage outage scenario.

[juho-hanhimaki]

Ended up rolling out custom IGrainStorage implementation that throws all other exceptions except InconsistentStateException as custom PersistenceWriteException type. That way IIncomingGrainCallFilter can determine with certainty that WriteStateAsync or ClearStateAsync is the source of the exception and state should be rolled back.

However for some reason the IIncomingGrainCallFilter implementation sees the exception wrapped in OrleansException. I did except an PersistenceWriteException in the filter. Not really a problem but I am curious what is behind this behaviour?

    public async Task Invoke(IIncomingGrainCallContext context)
    {
        var originalState = _state.State;
        try
        {
            await context.Invoke();
        }
        catch (OrleansException ex) when (ex.InnerException is PersistenceWriteException)
        {
            // IGrainStorage WriteStateAsync or ClearStateAsync failed, restore the original state.
            _state.State = originalState;
            throw;
        }
    }