Correct the Applicative instances

[nikita-volkov]

In my application I used the following stack:

EitherT MyError Concurrently a

where Concurrently is a type from the "async" package, whose instances for Applicative and Alternative utilise concurrency. To my surprise I discovered that this stack behaved sequentially when used with Applicative operators. Then I found out that the cause was in the EitherT instances, which were implemented sequentially in terms of Monad.

This pull request fixes the instances and lightens the constraints.

[glguy]

When a type has a Monad instance its Applicative instance should be identical to using ap and return. What you actually need is an EitherT-like type that only supports Applicative. For example: http://hackage.haskell.org/package/validation

[DanBurton]

Nonetheless, this patch still makes sense, especially in a post-AMP world. It relaxes the constraints necessary for the Functor, Applicative, and other instances.

[ygale]

I agree with @DanBurton that the patch is good.

But @glguy is correct - the bug here is actually in the async package. Concurrently should not have a Monad instance at all. The documentation specifies using the Applicative instance to get concurrency, and there is no way to write a Monad instance which is consistent with that. That broken instance should be removed.

[jwiegley]

@ygale Can you clarify why, if Applicative offers concurrency, there is no Monad instance consistent with it?

[ygale]

@jwiegley because to implement bind, you need to complete the action on the LHS before you can apply the function on the RHS to the result. There is no way to run the actions concurrently.

Hmm, perhaps there is a way to exploit laziness and allow the calculation to begin immediately and only block if it needs to force the value from the LHS. But still, would ap then be provably equivalent to (<*>)?

[DanBurton]

The monad interface is inherently sequential. When m a represents a
computation that must be run in order to extract the a, look at the type
of bind:

(>>=) :: m a -> (a -> m b) -> m b

You can't run the a -> m b computation until you have completed the m a
computation and extracted the a. Therefore ap is sequential, because it
is defined in terms of (>>=).

ap mf mx =
mf >>= \f ->
mx >>= \x ->
return (f x)

I'm surprised that Concurrently has a Monad instance. It is indeed
inconsistent with its Applicative instance, in terms of running
sequentially vs concurrently. But in terms of the actions run and values
produced, regardless of when they are run/produced, they're equivalent (as
long as concurrent actions don't interfere with each other).

-- Dan Burton

On Thu, Apr 23, 2015 at 10:00 AM, John Wiegley notifications@github.com
wrote:

@ygale https://github.com/ygale Can you clarify why, if Applicative
offers concurrency, there is no Monad instance consistent with it?

—
Reply to this email directly or view it on GitHub
#38 (comment).

[evincarofautumn]

Hmm, perhaps there is a way to exploit laziness and allow the calculation to begin immediately and only block if it needs to force the value from the LHS.

That would be unsafeInterleaveIO.

[ygale]

@evincarofautumn Yes, or we could use mdo, which uses unsafeInterleaveIO under the hood.

[ygale]

Could we please continue this discussion in simonmar/async#26? It is veering off-topic here.

[nikita-volkov]

When a type has a Monad instance its Applicative instance should be identical to using ap and return.

I think we've started off on the wrong foot and the above statement is the root of all misunderstandings here. Being implemented identically and producing the same results are two different things. The rule of ap and <*> having to produce the same result I understand, but the requirement for them to be implemented identically - where does that come from?! What's the point in having Applicative if we can't utilise its capabilities?!

I want to stress that nothing in this pull request violates <*> producing the same result as ap. It just uses more efficient mechanisms if the underlying type provides them - that's all. And I absolutely do not get what the problem is in that.

[ekmett]

This is sadly, not possible.

ap in this case doesn't execute the second action at all if the first one fails. (<*>) using the underlying Applicative necessarily does execute both actions. It has no way not to!

This means that the Applicative implemented as a composition of Applicatives has different semantics than the one that can exploit the Monad structure.

This has been repeatedly raised against transformers in the past on MaybeT and ErrorT as an issue and repeatedly shot down for this reason.

[nikita-volkov]

(<*>) using the underlying Applicative necessarily does execute both actions. It has no way not to!

The way I see it, it simply relays this decision on the Applicative instance of the base type (the m). And that makes another point: it does not mess with the semantics of the type it transforms, but reproduces them instead.

[ekmett]

foo = EitherT $ return $ Left "wrong launch codes"
bar = launchMissiles

foo <*> bar

Under the existing semantics this doesn't launch missiles.
Under yours it does.

You need the use of the Monad on m to stop the launch.

[LeifW]

Maybe something less general than IO is needed to indicate this. Effects which don't mutate RealWorld are free to have their execution order commuted (or skipped entirely if their value isn't needed)?

[ygale]

@ekmett I understand that there is a problem with changing the semantics of (<*>). But this PR has much more than that. Could you please look it over once more?

[nikita-volkov]

foo = EitherT $ return $ Left "wrong launch codes"
bar = launchMissiles

foo <*> bar

Under the existing semantics this doesn't launch missiles.
Under yours it does.

You need the use of the Monad on m to stop the launch.

You're right. However, launching missiles in such a scenario is what I would expect from a parallel (i.e., non-sequential) composition of two things, which, AIU, the power of Applicative is all about.

As another point, what's the good of having Applicative completely reproduce what would otherwise be still achievable with ap, if the user wanted? I'm especially frustrated with this in the light of having to sacrifice important features altogether for this. I've provided a vivid example of how it can limit user's abilities in practice.

[ekmett]

LeifW: You don't need IO, but it provides a very visceral example.

foo =  throwError "stack is empty" *> modify tail.

exhibits the same kind of problem using EitherT e (State [s]).

[ekmett]

Nikita, you can build such a type with your chosen instance it just isn't a "monad transformer" and it fails to satisfy the laws you need to be able to refactor code.

As (>>) = (*>), you're break do sugar as well.

It relies on invariants about m -- invariants which are basically only satisfied by things isomorphic to the reader monad. Everything else fails. EitherT can't have the semantics you want and be a monad transformer.

[ekmett]

Yes, it is what the Applicative is all about: Context freedom.

In which case, if that is what you want, then Compose m (Either e) is precisely the Applicative you are looking for. It doesn't have a Monad instance though, because there is no monad that is compatible with that Applicative. Just as ZipList has no Monad despite being a valuable Applicative.

[ekmett]

@ygale: I have no problem with weakening the constraint on Functor, etc.

[ekmett]

There is a similar issue with Either e itself. We can make an Either variant (often called Validation) where we take all the lefts and smash them together with a Semigroup. This is Applicative, but it doesn't extend to a Monad either.

[hadronized]

I guess the part of the patch that uses less restrictive typeclass constraints should be merged. As said above, it makes sense after AMP.

[ekmett]

@phaazon: We'll want to make a small tweak to that part to avoid worsening the constraints on users < 7.10, but otherwise, sure.

[nikita-volkov]

Thank you, Edward, for thorough answers.

Compose m (Either e) is precisely the Applicative you are looking for

I've never considered that. It's a good option. However I just hate to lose the Monad instance as well.

EitherT can't have the semantics you want and be a monad transformer.

That's the part I don't understand. I understand from our conversation, it's not just about monad transformers, but monads in general. I can't expect the following behaviour from a monad without breaking the laws - is that correct?

-- Executes sequentially:
links <- scrapeLinks
-- Executes concurrently:
fmap concat $ traverse scrapeResults links

But, Gods, do I love how neatly it abstracts from the problem of execution order with the most general interfaces. And BTW it is exactly the way that Concurrently happens to behave, and, AIU, GenHaxl as well. Does this mean they're both broken too?

So basically I just don't understand the practical implications of the laws, which prohibit that behaviour. What do we get exactly which is able to compensate the price of losing such features?

it fails to satisfy the laws you need to be able to refactor code

Can you elaborate?

As (>>) = (*>), you're break do sugar as well

Did this happen during AMP?

Why does changing the semantics of the "do" sugar neccessarily imply breaking it? True, it will behave differently in certain cases, but again it's the behaviour I for one would rather expect in such circumstances.

[ekmett]

Nothing prevents you from writing two combinators

foo :: Compose m (Either e) a -> EitherT e m a
bar :: EitherT e m a -> Compose m (Either e) a

or a lens-style Iso, to transition back and forth.

This is more or less how folks work with Validation, and it was the original motivation for the Concurrently newtype, to get just such an Applicative.

Along the way someone bolted on an illegal extension of it to a Monad, but that doesn't make it correct!

You can make more limited concurrency monads where the Applicative is legally doing parallel computation, but it requires removing certain many effects that you are willing to allow people to do.

e.g. haxl quotients out the number of passes to the server in its definition of how parallel an operation is, and under that quotient the notion of parallelism it uses is okay -- if you ignore the ability to lift IO operations in.

Concurrently is not such a principled beast, however.

You can have (<*>) compute a result vastly (potentially even infinitely!) more efficiently than ap, but having it compute a totally different answer is wrong.

[nikita-volkov]

Okay. Thanks.

[obadz]

@ekmett, could you clarify what you mean by:

haxl quotients out the number of passes to the server in its definition of how parallel an operation is, and under that quotient the notion of parallelism it uses is okay

[ekmett]

i mean that haxl doesn't try to duplicate the exact number of passes in terms of number of round trips made to the various data sources just the final result.

So the fact that (<*>) may let both calls to be made in parallel while ap will force them to be executed in sequence, the answer you get will be the same in either case.

The goal of haxl is to maximize the number of queries that can be executed in parallel and performance may be better in the Applicative setting, but the assumptions of the haxl monad are that answers won't change between passes and that the queries are just that, queries, and don't change the data downstream so if it can yield an answer without making a query that doesn't affect its semantics.

For haxl to get its parallel Applicative instance it needs both of those assumptions.

By 'quotienting' out I'm referring to the act of considering equal up to some condition. Here they are equal up to the number of round trips (which can be lower in the applicative setting) and the possible early avoidance of some queries (which can happen in the monadic setting), but both bits of code yield the same answer.