Interop with MTL classes

[isovector]

@adamConnerSax on #34 wrote:

A question, slightly off-topic maybe.
I feel like there is something similar about this hoist function and the MonadRandom thing we have been talking about over in polysemy-zoo.
All of them are ways of trying to accommodate a different version of an effect that polysemy supports.
So, two questions:

• Maybe these should be in a separate library, polysemy-mtl or polysemy-interop? That would get rid of the dependencies for users who don't care about mtl.
• What is the ideal form for such a thing? I can imagine three versions, though maybe not all would work equally well for all effects.

1. Something like hoist: hoistX:: Sem (X ': r) a -> XT (Sem r) a
2. Some kind of orphan instance: instance (Member X r) => MonadX (Sem r) where ...
3. A function--like liftIO---which discharges the constraint within Sem : liftX :: (MonadX m, Member X r) => m a -> Sem r a.
   I'm not quite sure how that third version would work, but it seems like it keeps most of the convenience of instances without the issue of actually having them.

I'm going to think about how to do that for MonadRandom in RandomFu.

[isovector]

Some points:

• mtl is a tiny dependency, that people are guaranteed already going to pull in transitively from somewhere. I'm not concerned about this.

1. Furthermore, State is a pretty fundamental concept, and I'm not sure that having special cases for it should imply we must also have special cases for other effects. I can imagine maybe also providing hoistErrorIntoExceptT :: Sem (Error e ': r) a -> ExceptT e (Sem r) a, but would be hard-pressed to go further than that.
2. Orphan instances really really really shouldn't exist in library code. They're maybe OK in application code, but in library code it's just asking for your program to explode in mysterious ways in a few years' time. You can get around this by packaging the orphan instance in the same place as you define the effect, but it still strikes me as a bad idea.
3. That function like liftIO is just sendM :)

Point 3 seems like the best to me, especially because you can implement it separately from the effect, and it has no relation on the effect's algebra. For example, you could write:

liftRandomIO 
    :: (Member RandomFu r, Member (Lift IO) r) 
    => (forall m. MonadRandom m => m a) 
    -> Sem r a

by just instantiating your forall'd MonadRandom'd m as IO and then sendMing that.

---

However, working backwards like this I think is the wrong strategy. If you have in mind "I want to use RandomFu" from the get go, you are going to just use RandomFu, and a huge chunk of the abstraction capabilities of polysemy will immediately be lost to you.

We learned this lesson the hard way at Takt, where we wanted a Redis effect. And so we wrote a Redis effect, but then it turns out, you can never mock that thing short of /just writing a working Redis implementation/ which is not really a mock at all! Instead what we found was to delegate Redis to be an interpretation layer only, meaning it wasn't allowed to appear in application code. Instead you'd write against higher abstractions, like having a KVStore k v you could read and write from, and then interpreting that thing (and others) into Redis.

MonadRandom is clearly an "abstraction" for writing application code. If you simply only have access to the effect algebras that you need rather than having access to all that are possible, you're suddenly afforded a great deal of liberty.

[adamConnerSax]

TL;DR: yes.

• mtl dependency is no big deal. I agree. Others on these threads and you had expressed varying levels of grumpiness with mtl dependency/style and so I was thinking of that. But your response to 3 clarifies and make sense of that. Enough said.

1. That makes sense.
2. That also makes sense! So we'll drop the instance. I don't think I can get around it as you say because it's an instance on Sem not RandomFu but maybe I'm missing something?
3. Is that function sendM? Doesn't sendM require a Member (Lift m) r constraint? I wasn't imagining writing this function as a call to something else but instead by "routing" it through the RandomFu effect, which is then interpreted however. Which, I think, is your point.

I agree about building up effects in this way, one which defers and makes flexible all the decisions about implementation by limiting your power to just what you need. And that's sort of the idea of the mtl constraints, right? MonadRandom m just says that I can get pseudo numbers via some function calls but it doesn't require a specific implementation.

So we agree, I think. We don't want to encourage using specific implementations.

[Sidebar 1: That talk is cool! Though I'll need to watch the adjunctions part again. And, maybe I missed it, but when talking about Applicative vs. Monad, I wished he mentioned parallelism. To me, that's one of the coolest examples and clearest examples of why using more general things gets you more power. I mean, also lenses, but ouch.]

And thus the question/idea: Can we write the function I proposed in #3, but without sendM? Can we discharge the constraint by, at the most extreme, creating a dictionary on the fly which routes the calls through the Random or RandomFu effect? Then I think we're accomplishing both of our goals: we encourage and support writing against higher abstractions (either Member Random r or MonadRandom m), we allow the full variety of interpretations, and we make it easy to deal with already written code with an mtl-style constraint.

Is what you meant by "packaging it with the effect" something like

newtype RandomSem r a = RandomSem { liftMonadRandom :: P.Sem r a } deriving (Functor, Applicative, Monad)

$(R.monadRandom [d|
        instance P.Member Random r => R.MonadRandom (RandomSem r) where
            getRandomPrim :: R.Prim t -> RandomSem r t
            getRandomPrim = RandomSem . getRandomPrim
    |])

?
It's not an orphan anymore since you have to use liftMonadRandom to access the instance. That sort of works as I would like but then in my test case it creates an inference issue. But that's likely because I'm using it in a nightmare scenario where the inference breaks if I look at it funny. So when I get a chance, I'll see if that works under simpler circumstances.

[Sidebar 2: Having this conversation around the Random effect(s) is confusing it some, because, as you've already pointed out, there shouldn't be two of them! Or, more generally, not all effects will have one-interface-to-rule-them-all and that presents a puzzle at the level of the abstraction rather than the interpretation.]

[isovector]

Thanks for writing that all out --- I think I understand your point now!

The point is: if people have already done their due diligence and written their apps in an MTL-esque tagless final style using MonadX typeclasses, then we should do them the favor of acknowledging that work? Yes, I agree!

At the wildest, we could use reflection to spin up that MonadRandom dictionary on the fly. I have some other, weirder experiments in dealing with orphan instances too.

But maybe you're right. Maybe the solution is to just embrace orphans, but possibly with the caveat that they must be in their own package and "buyer beware?" It's not a solution for classes with fundeps, but fundamentally I think there's only so much we can do.

[isovector]

This is fixed by polysemy-research/polysemy-zoo#8, when it lands