New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Correct the Applicative instances #38

Closed
wants to merge 6 commits into
base: master
from

Conversation

Projects
None yet
10 participants
@nikita-volkov
Copy link

nikita-volkov commented Apr 23, 2015

In my application I used the following stack:

EitherT MyError Concurrently a

where Concurrently is a type from the "async" package, whose instances for Applicative and Alternative utilise concurrency. To my surprise I discovered that this stack behaved sequentially when used with Applicative operators. Then I found out that the cause was in the EitherT instances, which were implemented sequentially in terms of Monad.

This pull request fixes the instances and lightens the constraints.

@glguy

This comment has been minimized.

Copy link
Contributor

glguy commented Apr 23, 2015

When a type has a Monad instance its Applicative instance should be identical to using ap and return. What you actually need is an EitherT-like type that only supports Applicative. For example: http://hackage.haskell.org/package/validation

@DanBurton

This comment has been minimized.

Copy link

DanBurton commented Apr 23, 2015

Nonetheless, this patch still makes sense, especially in a post-AMP world. It relaxes the constraints necessary for the Functor, Applicative, and other instances.

@ygale

This comment has been minimized.

Copy link

ygale commented Apr 23, 2015

I agree with @DanBurton that the patch is good.

But @glguy is correct - the bug here is actually in the async package. Concurrently should not have a Monad instance at all. The documentation specifies using the Applicative instance to get concurrency, and there is no way to write a Monad instance which is consistent with that. That broken instance should be removed.

@jwiegley

This comment has been minimized.

Copy link
Contributor

jwiegley commented Apr 23, 2015

@ygale Can you clarify why, if Applicative offers concurrency, there is no Monad instance consistent with it?

@ygale

This comment has been minimized.

Copy link

ygale commented Apr 23, 2015

@jwiegley because to implement bind, you need to complete the action on the LHS before you can apply the function on the RHS to the result. There is no way to run the actions concurrently.

Hmm, perhaps there is a way to exploit laziness and allow the calculation to begin immediately and only block if it needs to force the value from the LHS. But still, would ap then be provably equivalent to (<*>)?

@DanBurton

This comment has been minimized.

Copy link

DanBurton commented Apr 23, 2015

The monad interface is inherently sequential. When m a represents a
computation that must be run in order to extract the a, look at the type
of bind:

(>>=) :: m a -> (a -> m b) -> m b

You can't run the a -> m b computation until you have completed the m a
computation and extracted the a. Therefore ap is sequential, because it
is defined in terms of (>>=).

ap mf mx =
mf >>= \f ->
mx >>= \x ->
return (f x)

I'm surprised that Concurrently has a Monad instance. It is indeed
inconsistent with its Applicative instance, in terms of running
sequentially vs concurrently. But in terms of the actions run and values
produced, regardless of when they are run/produced, they're equivalent (as
long as concurrent actions don't interfere with each other).

-- Dan Burton

On Thu, Apr 23, 2015 at 10:00 AM, John Wiegley notifications@github.com
wrote:

@ygale https://github.com/ygale Can you clarify why, if Applicative
offers concurrency, there is no Monad instance consistent with it?


Reply to this email directly or view it on GitHub
#38 (comment).

@evincarofautumn

This comment has been minimized.

Copy link

evincarofautumn commented Apr 23, 2015

Hmm, perhaps there is a way to exploit laziness and allow the calculation to begin immediately and only block if it needs to force the value from the LHS.

That would be unsafeInterleaveIO.

@ygale

This comment has been minimized.

Copy link

ygale commented Apr 23, 2015

@evincarofautumn Yes, or we could use mdo, which uses unsafeInterleaveIO under the hood.

@ygale

This comment has been minimized.

Copy link

ygale commented Apr 23, 2015

Could we please continue this discussion in simonmar/async#26? It is veering off-topic here.

@nikita-volkov

This comment has been minimized.

Copy link

nikita-volkov commented Apr 23, 2015

When a type has a Monad instance its Applicative instance should be identical to using ap and return.

I think we've started off on the wrong foot and the above statement is the root of all misunderstandings here. Being implemented identically and producing the same results are two different things. The rule of ap and <*> having to produce the same result I understand, but the requirement for them to be implemented identically - where does that come from?! What's the point in having Applicative if we can't utilise its capabilities?!

I want to stress that nothing in this pull request violates <*> producing the same result as ap. It just uses more efficient mechanisms if the underlying type provides them - that's all. And I absolutely do not get what the problem is in that.

@ekmett

This comment has been minimized.

Copy link
Owner

ekmett commented Apr 23, 2015

This is sadly, not possible.

ap in this case doesn't execute the second action at all if the first one fails. (<*>) using the underlying Applicative necessarily does execute both actions. It has no way not to!

This means that the Applicative implemented as a composition of Applicatives has different semantics than the one that can exploit the Monad structure.

This has been repeatedly raised against transformers in the past on MaybeT and ErrorT as an issue and repeatedly shot down for this reason.

@ekmett ekmett closed this Apr 23, 2015

@nikita-volkov

This comment has been minimized.

Copy link

nikita-volkov commented Apr 23, 2015

(<*>) using the underlying Applicative necessarily does execute both actions. It has no way not to!

The way I see it, it simply relays this decision on the Applicative instance of the base type (the m). And that makes another point: it does not mess with the semantics of the type it transforms, but reproduces them instead.

@ekmett

This comment has been minimized.

Copy link
Owner

ekmett commented Apr 23, 2015

foo = EitherT $ return $ Left "wrong launch codes"
bar = launchMissiles

foo <*> bar

Under the existing semantics this doesn't launch missiles.
Under yours it does.

You need the use of the Monad on m to stop the launch.

@LeifW

This comment has been minimized.

Copy link

LeifW commented Apr 23, 2015

Maybe something less general than IO is needed to indicate this. Effects which don't mutate RealWorld are free to have their execution order commuted (or skipped entirely if their value isn't needed)?

@ygale

This comment has been minimized.

Copy link

ygale commented Apr 23, 2015

@ekmett I understand that there is a problem with changing the semantics of (<*>). But this PR has much more than that. Could you please look it over once more?

@nikita-volkov

This comment has been minimized.

Copy link

nikita-volkov commented Apr 23, 2015

foo = EitherT $ return $ Left "wrong launch codes"
bar = launchMissiles

foo <*> bar

Under the existing semantics this doesn't launch missiles.
Under yours it does.

You need the use of the Monad on m to stop the launch.

You're right. However, launching missiles in such a scenario is what I would expect from a parallel (i.e., non-sequential) composition of two things, which, AIU, the power of Applicative is all about.

As another point, what's the good of having Applicative completely reproduce what would otherwise be still achievable with ap, if the user wanted? I'm especially frustrated with this in the light of having to sacrifice important features altogether for this. I've provided a vivid example of how it can limit user's abilities in practice.

@ekmett

This comment has been minimized.

Copy link
Owner

ekmett commented Apr 23, 2015

LeifW: You don't need IO, but it provides a very visceral example.

foo =  throwError "stack is empty" *> modify tail.

exhibits the same kind of problem using EitherT e (State [s]).

@ekmett

This comment has been minimized.

Copy link
Owner

ekmett commented Apr 23, 2015

Nikita, you can build such a type with your chosen instance it just isn't a "monad transformer" and it fails to satisfy the laws you need to be able to refactor code.

As (>>) = (*>), you're break do sugar as well.

It relies on invariants about m -- invariants which are basically only satisfied by things isomorphic to the reader monad. Everything else fails. EitherT can't have the semantics you want and be a monad transformer.

@ekmett

This comment has been minimized.

Copy link
Owner

ekmett commented Apr 23, 2015

Yes, it is what the Applicative is all about: Context freedom.

In which case, if that is what you want, then Compose m (Either e) is precisely the Applicative you are looking for. It doesn't have a Monad instance though, because there is no monad that is compatible with that Applicative. Just as ZipList has no Monad despite being a valuable Applicative.

@ekmett

This comment has been minimized.

Copy link
Owner

ekmett commented Apr 23, 2015

@ygale: I have no problem with weakening the constraint on Functor, etc.

@ekmett

This comment has been minimized.

Copy link
Owner

ekmett commented Apr 23, 2015

There is a similar issue with Either e itself. We can make an Either variant (often called Validation) where we take all the lefts and smash them together with a Semigroup. This is Applicative, but it doesn't extend to a Monad either.

@phaazon

This comment has been minimized.

Copy link
Contributor

phaazon commented Apr 23, 2015

I guess the part of the patch that uses less restrictive typeclass constraints should be merged. As said above, it makes sense after AMP.

@ekmett

This comment has been minimized.

Copy link
Owner

ekmett commented Apr 23, 2015

@phaazon: We'll want to make a small tweak to that part to avoid worsening the constraints on users < 7.10, but otherwise, sure.

@nikita-volkov

This comment has been minimized.

Copy link

nikita-volkov commented Apr 24, 2015

Thank you, Edward, for thorough answers.

Compose m (Either e) is precisely the Applicative you are looking for

I've never considered that. It's a good option. However I just hate to lose the Monad instance as well.

EitherT can't have the semantics you want and be a monad transformer.

That's the part I don't understand. I understand from our conversation, it's not just about monad transformers, but monads in general. I can't expect the following behaviour from a monad without breaking the laws - is that correct?

-- Executes sequentially:
links <- scrapeLinks
-- Executes concurrently:
fmap concat $ traverse scrapeResults links

But, Gods, do I love how neatly it abstracts from the problem of execution order with the most general interfaces. And BTW it is exactly the way that Concurrently happens to behave, and, AIU, GenHaxl as well. Does this mean they're both broken too?

So basically I just don't understand the practical implications of the laws, which prohibit that behaviour. What do we get exactly which is able to compensate the price of losing such features?

it fails to satisfy the laws you need to be able to refactor code

Can you elaborate?

As (>>) = (*>), you're break do sugar as well

Did this happen during AMP?

Why does changing the semantics of the "do" sugar neccessarily imply breaking it? True, it will behave differently in certain cases, but again it's the behaviour I for one would rather expect in such circumstances.

@ekmett

This comment has been minimized.

Copy link
Owner

ekmett commented Apr 24, 2015

Nothing prevents you from writing two combinators

foo :: Compose m (Either e) a -> EitherT e m a
bar :: EitherT e m a -> Compose m (Either e) a

or a lens-style Iso, to transition back and forth.

This is more or less how folks work with Validation, and it was the original motivation for the Concurrently newtype, to get just such an Applicative.

Along the way someone bolted on an illegal extension of it to a Monad, but that doesn't make it correct!

You can make more limited concurrency monads where the Applicative is legally doing parallel computation, but it requires removing certain many effects that you are willing to allow people to do.

e.g. haxl quotients out the number of passes to the server in its definition of how parallel an operation is, and under that quotient the notion of parallelism it uses is okay -- if you ignore the ability to lift IO operations in.

Concurrently is not such a principled beast, however.

You can have (<*>) compute a result vastly (potentially even infinitely!) more efficiently than ap, but having it compute a totally different answer is wrong.

@nikita-volkov

This comment has been minimized.

Copy link

nikita-volkov commented Apr 24, 2015

Okay. Thanks.

@obadz

This comment has been minimized.

Copy link

obadz commented Jan 15, 2016

@ekmett, could you clarify what you mean by:

haxl quotients out the number of passes to the server in its definition of how parallel an operation is, and under that quotient the notion of parallelism it uses is okay

@ekmett

This comment has been minimized.

Copy link
Owner

ekmett commented Jan 15, 2016

i mean that haxl doesn't try to duplicate the exact number of passes in terms of number of round trips made to the various data sources just the final result.

So the fact that (<*>) may let both calls to be made in parallel while ap will force them to be executed in sequence, the answer you get will be the same in either case.

The goal of haxl is to maximize the number of queries that can be executed in parallel and performance may be better in the Applicative setting, but the assumptions of the haxl monad are that answers won't change between passes and that the queries are just that, queries, and don't change the data downstream so if it can yield an answer without making a query that doesn't affect its semantics.

For haxl to get its parallel Applicative instance it needs both of those assumptions.

By 'quotienting' out I'm referring to the act of considering equal up to some condition. Here they are equal up to the number of round trips (which can be lower in the applicative setting) and the possible early avoidance of some queries (which can happen in the monadic setting), but both bits of code yield the same answer.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment