Implementing ToIO on "custom" Transformer Stacks

[OlliMartin]

Hi, I've been messing around (dipping my head into) with Language.Ext for a while and my next project is retries (or unlocking the potential of the IO stuff).

Originally I had the following setup (see discussion):
record Flow<A>(StateT<FlowState, EitherT<Error, IO>, A> executeFlow) : K<FlowEither, A>

However, I read in Discussion 1269 that the StateT transformer cannot work around the interface restriction of MonadIO in a senseful ("magicless") way, so for now I exchanged StateT with ReaderT (as it would be sufficient for my main usecase).

-> record Flow<A>(ReaderT<FlowState, EitherT<FlowError, IO>, A> executeFlow)

Still I'm getting the error that ToIO is not implemented, presumably because it's also not there on EitherT.

Following Paul's example in the discussion, which can also be found here, I tried implementing it on Flow myself:

    ///<inheritdoc />
    static K<Flow, IO<A1>> MonadIO<Flow>.ToIO<A1>(K<Flow, A1> ma)
    {
        return new Flow<IO<A1>>(new ReaderT<FlowState, EitherT<FlowError, IO>, IO<A1>>(state =>
        {
            IO<Either<FlowError, A1>> interRes = ma.As().executeFlow.runReader(state).Run().As();
            
            EitherT<FlowError, IO, IO<A1>> result = interRes switch
            {
                Either.Right<FlowError, A1>(var a1) => Either<FlowError, IO<A1>>.Right(IO.pure(a1)).ToIO(),

                Either.Left<FlowError, A1>(var e) => Left(e),

                _ => Left(new FlowError(new BottomError())) /* How did that happen? */
            };

            return result;
        }));
    }

It compiles! But it always returns a wrapped bottom error, as the intermediate result is wrapped in Pure and the pattern matching does not seem to work.

While I continue banging my head at this; Is this going into the right (hehe) direction?

Apologies if there is crucial code missing, I think all of the referenced functions are coming from the core library.

Cheers
Olli

[aloslider]

Doesn't it always match to default, because you are switching on IO and using Either's Left and Right as possible cases?

[OlliMartin]

That's correct. I always end up in the default (_ => Left(..Bottom)) case because the intermediate result (interRes) is wrapped in a Pure.
My problem is that it is wrapped as Pure<Either<FlowError, A1>>, so I cannot build a senseful (?) pattern match based on left or right.

I'm guessing I'm doing something wrong here, but can't pinpoint what exactly (presumably the object that is pattern matched should not be wrapped in Pure?)

Is the point you are making about unwrapping the IO?

[aloslider]

Sorry, didn't have access to PC until now. What I mean is: you are trying to match IO to Eithers cases, but instead you should get the IO's inner monad (which is Either you want) by calling additional .Run() on it at the end:

var interRes = ma.As().executeFlow.runReader(state).Run().As().Run();

Be aware that I'm still learning as well, so I may be wrong😄

[OlliMartin]

I could execute RunAsync(), technically (see below in the other answer) - however I'm not sure if it is the right thing to do.
It doesn't fit together in my head - why should I run IO in a function called ToIO.
I will give it a try tomorrow though (with GetAwaiter().GetResult()), currently I'm working on something completely different :)

Needless to say I appreciate any bit of help I can get :)

[aloslider]

It doesn't fit together in my head - why should I run IO in a function called ToIO.

You are right, I didn't pay attention to your example enough. I guess, if you are making just a simple wrapper, you better delegate all this IO stuff handling to the underlying transformer (ReaderT in your case):

class Flow : Monad<Flow>, MonadIO<Flow>
{
    static K<Flow, IO<A>> MonadIO<Flow>.ToIO<A>(K<Flow, A> ma) =>
        new Flow<IO<A>>(ma.As().executeFlow.ToIO().As());
        ...
}

Edit: oh nvm, didn't notice louthy has already answered with a full example 😅

[louthy]

I see that you have a generic parameter of A1 on the method, which implies there's a generic parameter of A on the enclosing-type. Which in turn implies that you're implementing this for Flow<A> rather than for the trait-type of Flow.

So, make sure you've overrided the correct method in the trait-implementation of Flow, not Flow<A>.

[OlliMartin]

I cant honestly tell why those are named A1 - I have meanwhile migrated my actual project to use the mentioned transformer stack (without retries) and just pulled the code back in to see if I messed up.
Pretty sure I did it on the Flow, not Flow<A>. (I am pretty certain I had it on the correct class because it was being invoked)

Looking into the functions available to Pure I can't find anything that rings a bell to get the actual result (other than calling RunAsync, which feels a bit odd? Especially because I'm not in an async context).

But then again, probably something else is terribly wrong.

Note: The TFlowConfiguration is the environment of the respective reader.
Note2: For completeness the code in editable form. Notice that the inheritdoc maps to the MonadIO function.

  /// <inheritdoc />
  static K<Flow<TFlowConfiguration>, IO<A>> MonadIO<Flow<TFlowConfiguration>>.ToIO<A>(
    K<Flow<TFlowConfiguration>, A> ma
  )
  {
    return new FlowT<TFlowConfiguration, IO<A>>(
      new ReaderT<TFlowConfiguration, EitherT<FlowError, IO>, IO<A>>(
        state =>
        {
          Either<FlowError, A> interRes = ma.As().ExecuteFlow.runReader(state).Run().As();

          EitherT<FlowError, IO, IO<A>> result = interRes switch
          {
            Either.Right<FlowError, A>(var a1) => Either<FlowError, IO<A>>.Right(IO.pure(a1)).ToIO(),

            Either.Left<FlowError, A>(var e) => Left(e),

            var _ => Left(new FlowError(new BottomError())), /* How did that happen? */
          };

          return result;
        }
      )
    );
  }

[louthy]

So, part of the issue here is that EitherT doesn't produce a value of A. It's FlowError | A. This is a similar issue to why you can't use StateT. Getting an IO<A> isn't strictly possible when the result is FlowError | A. Just as it's not possible to use (S, A) without throwing away the state.

However, IO does have support for Error. So, if you want to use your own bespoke error-type (FlowError) in a monad-transformer stack and have it support ToIO, then you need to find a way of converting between Error and FlowError.

So, to make this work, first implement these methods for FlowError:

public record FlowError
{
    public static FlowError FromError(Error err) => throw new System.NotImplementedException();
    public Error ToError() => throw new System.NotImplementedException();
}

You would need these anyway, because you really want to capture exceptions from the inner IO computations and convert them to FlowError types so that you only have one error handling flow.

Then you can do the Flow type, extensions, and trait implementations:

public record Flow<A>(ReaderT<FlowState, EitherT<FlowError, IO>, A> executeFlow) 
    : K<Flow, A>;

public static class FlowExtensions
{
    public static Flow<A> As<A>(this K<Flow, A> ma) =>
        (Flow<A>)ma;

    public static IO<Either<FlowError, A>> Run<A>(this K<Flow, A> ma, FlowState state) =>
        ma.As()
          .executeFlow
          .Run(state)
          .Run()
          .Catch(_ => true, e => Either<FlowError, A>.Left(FlowError.FromError(e)))
          .As();
}

public class Flow :
    Monad<Flow>,
    Readable<Flow, FlowState>,
    Fallible<FlowError, Flow>
{
    static K<Flow, B> Monad<Flow>.Bind<A, B>(K<Flow, A> ma, Func<A, K<Flow, B>> f) => 
        new Flow<B>(ma.As().executeFlow.Bind(x => f(x).As().executeFlow));

    static K<Flow, B> Functor<Flow>.Map<A, B>(Func<A, B> f, K<Flow, A> ma) => 
        new Flow<B>(ma.As().executeFlow.Map(f));

    static K<Flow, A> Applicative<Flow>.Pure<A>(A value) => 
        new Flow<A>(ReaderT.pure<FlowState, EitherT<FlowError, IO>, A>(value));

    static K<Flow, B> Applicative<Flow>.Apply<A, B>(K<Flow, Func<A, B>> mf, K<Flow, A> ma) =>
        new Flow<B>(mf.As().executeFlow.Apply(ma.As().executeFlow).As());

    static K<Flow, A> Readable<Flow, FlowState>.Asks<A>(Func<FlowState, A> f) => 
        new Flow<A>(ReaderT.asks<EitherT<FlowError, IO>, A, FlowState>(f));

    static K<Flow, A> Readable<Flow, FlowState>.Local<A>(Func<FlowState, FlowState> f, K<Flow, A> ma) => 
        new Flow<A>(ReaderT.local(f, ma.As().executeFlow));

    static K<Flow, A> Fallible<FlowError, Flow>.Fail<A>(FlowError error) => 
        new Flow<A>(ReaderT.lift<FlowState, EitherT<FlowError, IO>, A>(EitherT<FlowError, IO, A>.Left(error)));

    static K<Flow, A> Fallible<FlowError, Flow>.Catch<A>(
        K<Flow, A> fa,
        Func<FlowError, bool> Predicate,
        Func<FlowError, K<Flow, A>> Fail) =>
        new Flow<A>(
            ReaderT<FlowState, EitherT<FlowError, IO>, A>.AsksM(
                state =>
                {
                    // Catch and process IO errors
                    var io = fa.As()
                               .executeFlow
                               .Run(state)
                               .Run()
                               .Catch<IO, Either<FlowError, A>>(
                                    Predicate: (Error e) => Predicate(FlowError.FromError(e)),
                                    Fail: (Error e) => Fail(FlowError.FromError(e)).As().Run(state));

                    // Catch and process FlowErrors
                    io = io.Bind(ea => ea switch
                                       {
                                           Either.Left<FlowError, A> (var l)  => IO.pure(Either<FlowError, A>.Left(l)),
                                           Either.Right<FlowError, A> (var r) => IO.pure(Either<FlowError, A>.Right(r))
                                       });

                    // Return
                    return EitherT<FlowError, IO, A>.Lift(io);
                }));

    static K<Flow, A> MonadIO<Flow>.LiftIO<A>(IO<A> ma) =>
        new Flow<A>(
            ReaderT.lift<FlowState, EitherT<FlowError, IO>, A>(
                new EitherT<FlowError, IO, A>(
                    IO.lift(envIO =>
                            {
                                try
                                {
                                    var result = ma.Run(envIO);
                                    return Either<FlowError, A>.Right(result);
                                }
                                catch (Exception e)
                                {
                                    // When lifting random IO computations into Flow, always true to convert
                                    // the exceptions into FlowError
                                    return Either<FlowError, A>.Left(FlowError.FromError(e));
                                }
                            }))));

    static K<Flow, IO<A>> MonadIO<Flow>.ToIO<A>(K<Flow, A> ma) =>
        new Flow<IO<A>>(
            ReaderT.asksM<EitherT<FlowError, IO>, FlowState, IO<A>>(
                state =>
                    EitherT<FlowError, IO, IO<A>>.Right(
                        ma.As()
                          .executeFlow
                          .Run(state)
                          .As()
                          .Run()
                          .Map(ea => ea switch
                                     {                                      // We must throw here so we can catch later
                                         Either.Left<FlowError, A> (var l)  => l.ToError().Throw<A>(),
                                         Either.Right<FlowError, A> (var r) => r
                                     })
                          .As())));
}

Notice:

• The catching of errors in FlowExtensions.Run - this forces all IO errors to be converted to a FlowError
• The implementation of Catch. It catches both IO exceptions and FlowError (in the EitherT) so that you can handle them (with @catch, .Catch, etc.)
• LiftIO is where some random IO computation is lifted into Flow, so this has a try/catch to convert to a FlowError as soon as possible.
• ToIO must yield an IO<A> and not a IO<Either<FlowError, A>>, so if the result is Left then it converts the FlowError back to an Error and throws it. We rely on all the other error handling code (detailed above) to catch those exceptions and convert back to a FlowError. This is a bit hacky, but it's the only way of achieving what you want.

I haven't run any of the code above, so I don't know if it definitely works, but it should do.

By the way, most of these issues go away if you just use the Error types (Exceptional and Expected) as the base-types to FlowError and use the built-in Error support in IO instead of layering EitherT.

This is with EitherT removed from the stack:

public static class FlowExtensions
{
    public static Flow<A> As<A>(this K<Flow, A> ma) =>
        (Flow<A>)ma;

    public static IO<A> Run<A>(this K<Flow, A> ma, FlowState state) =>
        ma.As()
          .executeFlow
          .Run(state)
          .As();
}

public class Flow :
    Monad<Flow>,
    Readable<Flow, FlowState>,
    Fallible<Flow>
{
    static K<Flow, B> Monad<Flow>.Bind<A, B>(K<Flow, A> ma, Func<A, K<Flow, B>> f) => 
        new Flow<B>(ma.As().executeFlow.Bind(x => f(x).As().executeFlow));

    static K<Flow, B> Functor<Flow>.Map<A, B>(Func<A, B> f, K<Flow, A> ma) => 
        new Flow<B>(ma.As().executeFlow.Map(f));

    static K<Flow, A> Applicative<Flow>.Pure<A>(A value) => 
        new Flow<A>(ReaderT.pure<FlowState, IO, A>(value));

    static K<Flow, B> Applicative<Flow>.Apply<A, B>(K<Flow, Func<A, B>> mf, K<Flow, A> ma) =>
        new Flow<B>(mf.As().executeFlow.Apply(ma.As().executeFlow).As());

    static K<Flow, A> Readable<Flow, FlowState>.Asks<A>(Func<FlowState, A> f) => 
        new Flow<A>(ReaderT.asks<IO, A, FlowState>(f));

    static K<Flow, A> Readable<Flow, FlowState>.Local<A>(Func<FlowState, FlowState> f, K<Flow, A> ma) => 
        new Flow<A>(ReaderT.local(f, ma.As().executeFlow));

    static K<Flow, A> Fallible<Error, Flow>.Fail<A>(Error error) =>
        new Flow<A>(ReaderT.lift<FlowState, IO, A>(IO.fail<A>(error)));

    static K<Flow, A> Fallible<Error, Flow>.Catch<A>(
        K<Flow, A> fa,
        Func<Error, bool> Predicate,
        Func<Error, K<Flow, A>> Fail) =>
        from state  in Readable.ask<Flow, FlowState>()
        from result in fa.MapIO(io => io.Catch(Predicate, e => Fail(e).Run(state)))
        select result;

    static K<Flow, A> MonadIO<Flow>.LiftIO<A>(IO<A> ma) =>
        new Flow<A>(ReaderT.liftIO<FlowState, IO, A>(ma));

    static K<Flow, IO<A>> MonadIO<Flow>.ToIO<A>(K<Flow, A> ma) =>
        new Flow<IO<A>>(ma.As().executeFlow.ToIO().As());
}

Notice how much simpler ToIO, LiftIO, Catch, and Run is.

[OlliMartin]

Thanks again for your very detailed answer!
Apologies for having misunderstood the linked discussion above, I thought by "simply" providing a specialized ToIO in my monad I could make it work and that the problem was rather that you cannot provide a general ToIO in the EitherT (and other types) because of the response type violation.

It felt like the right thing to do to include the EitherT in the stack, but I did not think about that there is very likely already something built in for the same..

In the past I had quite some headaches around having Errors bubbling through my application, sometimes nested (ManyErrors) and processing them in a senseful way.
That is, for example, have a flow that returns a result or one (or many) errors (basically the same as Either) and then being required to do the right decision if an API should return 400 (Bad Request) or 500 (Internal Server Error).
Hence in this "project", I wanted to include my base class from the beginning to be able to inject additional metainformation if required.
(FlowError extends Error already)

By the way, most of these issues go away if you just use the Error types (Exceptional and Expected) as the base-types to FlowError and use the built-in Error support in IO instead of layering EitherT.

This statement confuses me a little. In your referenced (simplified and way sexier) code, FlowError is not used - is it still possible to make sure that all errors that are created follow my contract of being FlowError (this was my original requirement - have one error base class that can be injected with additional metadata like the correlationId, as an example)

Or are you hinting at an approach to override the Fallible<Error, Flow>.Fail<A>(Error error) to return a FlowError?

I will have yet another go at this in some time! Again, thank you so much for your help.

Btw, is there an example for error handling (using Language.Ext Error/Exceptional/Expected)? When dealing with these kind of errors I always have the fear that without type restrictions (Error < FlowError) writing error responses could violate API contracts.
Of course this isn't a problem in a one-man project, but if there is 50+ developers working on the same codebase, having everyone remember how to structure an error is not as easy as it sounds.
How would you go about this? Have a specialized "serializer" for Error and implement all the different possibilities for the Inner error?

[louthy]

I could make it work and that the problem was rather that you cannot provide a general ToIO in the EitherT (and other types) because of the response type violation.
It felt like the right thing to do to include the EitherT in the stack, but I did not think about that there is very likely already something built in for the same..

The thing to remember is that the hyper-general solutions (like EitherT) do not know anything about M (the monad they lift), which is why they can't provide a ToIO implementation. When you build your own monad-transformer stacks where you know the type of every monad in the stack, often it is possible to work around those limitations. But, you also need to respect the laws of the original types. An EitherT is there to carry an alternative value. So, as long as you're able to make those rules hold for the constructed-type, you're good.

If weird edge-cases occur under certain situations, then the composition of the types has failed. This would happen if you tried to implement ToIO for StateT. You would have to drop the state, which may have changes that therefore don't propagate. This is where you need to be careful.

This statement confuses me a little. In your referenced (simplified and way sexier) code, FlowError is not used

The implication is that you derive FlowError from Expected (which derives from Error). So, everything can be written in terms of Error, which simplifies everything.

If you want to catch and expose the FlowError as before, you can update the Run:

public static IO<Either<FlowError, A>> Run<A>(this K<Flow, A> ma, FlowState state) =>
    ma.As()
      .executeFlow
      .Run(state)
      .Map(Either<FlowError, A>.Right) 
      .Catch(_ => true, e => IO.pure<Either<FlowError, A>>(FromError(e)))
      .As();

public static FlowError FromError(Error err) =>
    err as FlowError ?? err.Throw<FlowError>();

You just need to decide whether you want non-FlowError errors to Throw as it does above, or if you want to wrap up the Error in a special 'uncaught' FlowError case. You need to have a policy on this regardless of whether you use EitherT or not. Because many of the IO methods test for things like the cancellation-token being triggered, or timeouts, or whatever, and will throw Error values from the built-in Errors. Treating these as exceptional is fine, but you may well want to turn these into well-defined FlowError values.

In the past I had quite some headaches around having Errors bubbling through my application, sometimes nested (ManyErrors) and processing them in a senseful way.

There are a number of different strategies. My preferred are:

• Let genuine exceptions fail the system - failing and restarting asap is the best strategy
• Trap non-exceptional exceptions and turn them into well-defined Error types as soon as possible
  • Use @catch and .Catch to pattern-match the errors you want to catch (for Fallible types)
• Use @catch and .Catch higher-up the stack to give more context to errors that didn't have enough context at the point of failure.
• For simple error propagation use the Choice trait and the | operator
• For any other Error values that you have 'in-hand', outside of the standard catching process, use:
  • error.Is(Error match) - This will test if match exists in error. Even in a ManyErrors collection.
  • error.IsType<E>() - This will test if error is of type E. If it's a ManyErrors type, then IsType will test if any of the errors in the ManyErrors collection are of type E
  • error.HasException<E>() - Like IsType, but tests for the existence of an Exceptional error that holds an Exception of type E.
  • For additional processing of errors, use Filter, ForAllM, and FoldM.
  • Really, it's better to deal with errors using @catch and .Catch as they do all of these checks automatically, but you may well have an Error outside of a monad-bind computation, and so the above tests can be done in if or switch statements to get at the errors you want.
• Create domain specific errors-codes: enum FlowErrorCode or similar. Error-codes are critical to effective pattern-matching of more complex error types. Using error-codes speeds up the error matching process and makes it less likely to fail due a difference in message.

Btw, is there an example for error handling (using Language.Ext Error/Exceptional/Expected)? When dealing with these kind of errors I always have the fear that without type restrictions (Error < FlowError) writing error responses could violate API contracts.

Remember, nothing happens without Run, so it's the place where you can put additional enforcement. As I showed above with the new implementation of Run, you can put in bespoke functionality for dealing with non-FlowError errors. You could wrap them up into a specific FlowError state, or just throw them as exceptional. You could also attach logging, or other processes to that to trap poorly behaved computations.

This is weaker than baking FlowError into the type, obviously, but it's a more pragmatic approach. The knowing that there's a single entry point (Run), makes it much easier to reason about the entire usage of Flow.

How would you go about this? Have a specialized "serializer" for Error and implement all the different possibilities for the Inner error?

Very much the reason for the Exceptional and Expected types was to make it easier to expose error-responses in a more controlled way. Exceptional errors should always turn into a "there was a problem" error in a public API response. Whereas Expected errors should be able to show their codes and responses safely.

There is a DataContract in-place for Error. It will depend on the serialisation library you use to whether it is respected or not. But, the idea is that Code and Message are allowed in the contract, but Inner is not.

Obviously, it is up to you to make sure that your errors serialise correctly, but the tools are there to have a more controlled error environment:

• Exceptional / Expected
• Single entry point to all processing making it easy to control the errors that are exposed
• Robust introspection and filtering of errors (Is, IsType, HasException, IsExceptional, IsExpected, Filter, ForAllM, FoldM, ...)

I don't think there are any specific samples relating to error handling and the full range of capabilities. But EffectsExamples has ErrorAndGuardExample:

public static class ErrorAndGuardExample<RT>
    where RT : 
        Has<Eff<RT>, ConsoleIO>
{
    static readonly Error UserExited = Error.New(100, "user exited");
    static readonly Error SafeError = Error.New(200, "there was a problem");

    public static Eff<RT, Unit> main =>
        from _1 in askUser
                 | @catch(ex => ex.HasException<SystemException>(), Console<RT>.writeLine("system error"))
                 | SafeError
        from _2 in Console<RT>.writeLine("goodbye")
        select unit;

    static Eff<RT, Unit> askUser =>
        repeat(Schedule.spaced(1 * second) | Schedule.recurs(3),
               from ln in Console<RT>.readLine
               from _1 in guard(notEmpty(ln), UserExited)
               from _2 in guard(ln != "sys", () => throw new SystemException())
               from _3 in guard(ln != "err", () => throw new Exception())
               from _4 in Console<RT>.writeLine(ln)
               select unit).As()
      | @catch(UserExited, pure<Eff<RT>, Unit>(unit));
}

It shows:

• Catching of specific Exception types (SystemException in this case)
• A catch-all SafeError that will transform all uncaught errors that get that far into a "there was a problem" response
• Use of errors to exit a repeat - a UserExited error is raised when there is no input. It is later caught and turned into a 'success' unit response to safely exit the program.

Not exactly a demo of the complete suite of error handling behaviours, but gives a flavour.