New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

New Project: Typelevel Schrodinger #66

Closed
alexandru opened this Issue Mar 29, 2017 · 62 comments

Comments

@alexandru
Member

alexandru commented Mar 29, 2017

I have a proposal for a new project, called Typelevel Schrodinger.

What this project aims to do is to provide a common interface between various Task and IO like data-types and would basically be what Reactive Streams is for streaming, the purpose being to allow interoperability between various libraries.

I started an initial draft here, but would be cool if we moved this to the Typelevel organization and collaborate on it:

https://github.com/alexandru/schrodinger

Summary / Current Proposal (Updated: Apr 7, 2017 23:01)

  1. we start project schrodinger-core which:
  • provides Evaluable, Deferrable, Eventual (or Effect) and Async type-classes
  • is meant as middleware, for interoperability purposes (e.g. conversions)
  • this is meant for library authors, not users
  • it should be as stable and as light as possible (i.e. no dependencies) and once we release 1.0.0 I'd like it to remain set in stone as to not create problems for the downstream
  1. we start a cats-effect project which:
  • provides a reference cats.effect.IO implementation, that depends on cats-core and which also implements instances for schrodinger-core
  • this IO implementation is in fact a Task, capable of async evaluation - naming it like this would be nice because Cats already has Eval and an async IO would be a more capable equivalent of Haskell's IO and it would also leave the Task name to be used for other implementations

Open questions:

  1. Is it OK to have an async cats.effect.IO? (as name, instead of Task)
  2. Can project Schrodinger be moved to the Typelevel GitHub organization?
  3. ... (not sure if I missed anything)

A graph of the dependencies, as I see it:

image

/cc @tpolecat @rossabaker @mpilquist @edmundnoble @djspiewak @non @adelbertc

(notifying here the folks that showed interest, sorry for the spam, not sure if I missed anybody)

@cquiroz

This comment has been minimized.

cquiroz commented Mar 29, 2017

Do you think this would be compatible with scala.js?

@alexandru

This comment has been minimized.

Member

alexandru commented Mar 29, 2017

Do you think this would be compatible with scala.js?

@cquiroz yes, it definitely is. I haven't configured that project yet for Scala.js, but it will be.

@djspiewak

This comment has been minimized.

djspiewak commented Mar 29, 2017

Compatibility with Scala.js definitely seems like a 100% critical requirement.

I would actually depend on Cats. We're going to need to figure out fs2 things at some later date. Part of the value of something like this, as opposed to just depending on the soon-to-come fs2-task submodule, is the fact that it would be highly integrated into the Cats ecosystem.

I would rather bring in something approximating the fs2 typeclass hierarchy (Capture, Catchable, Async, etc) rather than adding something as ad hoc as MonadDefer. This also isolates the effectful typeclasses in the effects project, which seems like logically where they belong. You already basically have this, so I'm ok with that.

I'm definitely not a fan of the ExecutionContext constraint, but I acknowledge that there's no way to apply these typeclasses to monix.Task (and similar constructs) without it. It over-constrains implementations like fs2.Task and also results in types that imply the wrong semantics (since fs2 will literally ignore the EC), but I don't see a better way.

Public Domain and/or CC0 licensing are inappropriate for software distributed in the US. Copyright law is different here in some pretty fundamental ways, notably relating to patent and copyright assignment. These are differences that "real" licenses (notably GPL and ASL) handle very carefully, but licenses like Public Domain, CC0, MIT and such essentially just ignore.

Oh, I'm also strongly averse to the Callback approach (as a separate type). I understand that it avoids the extra allocation, but it also means that it's impossible to handle error cases without extra side-effects. Additionally, the use-site syntax becomes a lot worse, since you need a full anonymous inner class. It also represents an extra type which has to be digested, resulting in higher cognitive load for users trying to learn the API.

On another random note, it's worth pointing out that the API as designed is incompatible with approaches like purescript's Aff, which is similar in spirit to what @puffnfresh and I have been experimenting with here.

@mpilquist

This comment has been minimized.

Member

mpilquist commented Mar 29, 2017

Can we make the ExecutionContext parameter path dependent and let the instance decide what type to pass there?

trait Effect[F[_]] {
  type Context
  def unsafeExecuteAsyncIO[A](fa: F[A], cb: Callback[A])(implicit ctx: Context): Unit
}
@adelbertc

This comment has been minimized.

Member

adelbertc commented Mar 29, 2017

@alexandru

This comment has been minimized.

Member

alexandru commented Mar 29, 2017

@mpilquist

Can we make the ExecutionContext parameter path dependent and let the instance decide what type to pass there?

Unfortunately that doesn't help with abstracting over these types at the call site, where this abstraction is needed, because at the call site you have to know what that Context is, so as an abstraction Effect is no longer useful.

@mpilquist

This comment has been minimized.

Member

mpilquist commented Mar 29, 2017

The call site would need an implicit f.Context, though perhaps that's too complex/unwieldy?

I really, really don't like the ExecutonContext parameter.

@djspiewak

This comment has been minimized.

djspiewak commented Mar 29, 2017

@alexandru

def foo[F[_], A, C](fa: F[A])(implicit F: Async.Aux[F, C], C: C) = ???
@alexandru

This comment has been minimized.

Member

alexandru commented Mar 29, 2017

I would love for that to work, unfortunately I don't see how. If the call site needs an f.Context, that needs to come from somewhere.

If we describe a generic foo that takes that F[_] : Async as parameter, you simply move the problem around. At some point, somebody needs to specify an actual f.Context instance. If we don't specify it in that foo function that @djspiewak just described, then we need to specify it at foo's call-site or further down the line.

And if the goal is to generically call unsafePerformIO, then that EC will have to be a type we can agree on. FS2 already has Strategy, which is basically the same thing (minus reportError which IMO is also useful).

I'm definitely not a fan of the ExecutionContext constraint, but I acknowledge that there's no way to apply these typeclasses to monix.Task (and similar constructs) without it. It over-constrains implementations like fs2.Task and also results in types that imply the wrong semantics (since fs2 will literally ignore the EC), but I don't see a better way.

In Monix we'll have the same problem with Task.async, for which we do not require any EC currently. So our implemented Async.create will also ignore that ExecutionContext.

That is fine though, because the Monix Task will still have the same interface as it does today, these APIs being meant for interoperability purposes only. Just to give an example, working directly with the Reactive Streams API is actually very error prone, because those aren't meant for users, but for library authors.

Also Monix is in the same boat. Our runAsync takes a Scheduler and in this case if the given ExecutionContext is not already a Scheduler then we'll have to convert, but that is fine IMO, because an ExecutionContext-like thing is all that's needed.

And I just implemented an Async[Future] instance which does use it, which is a sign that this solution is not that specific to our Tasks.

The way I see it, as soon as you're dealing with (A => Unit) => Unit, you need some sort of EC injected from somewhere, either in create or in run - and to cover both we simply require it for both 😄 and ignore it if possible.

@mpilquist

This comment has been minimized.

Member

mpilquist commented Mar 29, 2017

Note that fs2.Strategy is only used by fs2.Task -- not any of the type classes. All of the FS2 combinators are implemented parametrically in F with some type class constraint. Strategy appears nowhere in these combinators.

Usage wise, who is the audience of this interop library? Do you envision libraries like http4s and doobie working parametrically with these type classes? If so, wouldn't the ExecutionContext constraint propagate all throughout the end-user APIs of these libraries?

@alexandru

This comment has been minimized.

Member

alexandru commented Mar 29, 2017

I would actually depend on Cats. We're going to need to figure out fs2 things at some later date. Part of the value of something like this, as opposed to just depending on the soon-to-come fs2-task submodule, is the fact that it would be highly integrated into the Cats ecosystem.

If everybody agrees, we can initiate a cats-effects project and then we can make use of the cats.Monad.

Public Domain and/or CC0 licensing are inappropriate for software distributed in the US.

I'm not an expert. But we can do Cats's license, that's fine.

Oh, I'm also strongly averse to the Callback approach (as a separate type). I understand that it avoids the extra allocation, but it also means that it's impossible to handle error cases without extra side-effects.

We can get rid of it, had a feeling it wouldn't fly, but I really hate that extra allocation 😄

@alexandru

This comment has been minimized.

Member

alexandru commented Mar 29, 2017

Note that fs2.Strategy is only used by fs2.Task -- not any of the type classes. All of the FS2 combinators are implemented parametrically in F with some type class constraint. Strategy appears nowhere in these combinators.

I know, but that's also because you don't interact with an "edge of the program". A library like Http4s must probably trigger that execution by itself.

Usage wise, who is the audience of this interop library? Do you envision libraries like http4s and doobie working parametrically with these type classes? If so, wouldn't the ExecutionContext constraint propagate all throughout the end-user APIs of these libraries?

Yes, unfortunately that might be a problem 😒

@djspiewak

This comment has been minimized.

djspiewak commented Mar 29, 2017

@alexandru What if we remove the ExecutionContext from the Async functions and simply expect that implementations which require it (e.g. Monix) capture that ExecutionContext when the typeclass instance is created? Basically the same thing that scalaz.Monad[scala.concurrent.Future] does. This is actually what fs2 does when constructing its instances, to ensure that the forked async constructor works properly.

Usage wise, who is the audience of this interop library? Do you envision libraries like http4s and doobie working parametrically with these type classes? If so, wouldn't the ExecutionContext constraint propagate all throughout the end-user APIs of these libraries?

Are there strong objections to just creating a simple implementation? It wouldn't need to be particularly elaborate, and we can punt on questions like cancelability by just shooting for either absolutely minimal or absolutely composable. For example, define type IO[A] = Free[λ[a => () => Either[Throwable, a]], A], implement sealed trait ContT[F[_], R, A] and then define type Task[A] = ContT[IO, Unit, A]. It won't be anywhere near as fast as Monix's or fs2's Task, but it's perfectly serviceable (e.g. I'm pretty sure it's faster than Scalaz's Task) and it would be able to implement Async and friends.

Alternatively, we can just copy over fs2's Task. It's not perfect for everyone's use-cases, but it's just about the simplest thing that could work (other than ContT[IO, Unit, A]). The effect typeclasses ensure that Monix (and more) can still have their own effect type, and it will still work as a first-class citizen in any environment which isn't practically forced to use a concrete effect (e.g. http4s), and all other use-cases can be handled by ensuring that we keep in mind seamless conversions.

@notxcain

This comment has been minimized.

notxcain commented Mar 30, 2017

I've found it very useful to have a CaptureFuture type class, for interaction with Future base API:

trait CaptureFuture[F[_]] {
  def captureF[A](future: => Future[A]): F[A]
}

Instances should defer Future[A] creation until actual execution. So there is no instance for Future itself.

@alexandru

This comment has been minimized.

Member

alexandru commented Mar 30, 2017

@notxcain that's the same thing as the described Async.create, because you can describe your captureF in terms of it.

Async[F].create { cb =>
  future.onComplete {
    case Success(v) => cb.onSuccess(v)
    case Failure(ex) => cb.onFailure(ex)
  }
}

Note that for wrapping Future APIs, because of Monix's design, we can do one better by getting rid of the needed ExecutionContext which happens when creating any Future, since it can get injected by Task itself:

Task.deferFutureAction { implicit ec =>
  yourFuture()
}

Unfortunately we can't abstract over this.

@notxcain

This comment has been minimized.

notxcain commented Mar 30, 2017

Also I think there should be no instance of Effect for Future, because unsafeExecuteAsyncIO is meaningless, as it is already being executed. However, there could be an instance of Effect[Kleisli[Future, Unit, ?]] with unsafePerformIO = kleisli.run(()).

@alexandru

This comment has been minimized.

Member

alexandru commented Mar 30, 2017

Also I think there should be no instance of Effect for Future, because unsafeExecuteAsyncIO is meaningless, as it is already being executed.

It wouldn't be meaningless, because Effect is about getting that value out of the F[_] context, and definitely not about suspension of effects.

@notxcain

This comment has been minimized.

notxcain commented Mar 30, 2017

@alexandru then name of unsafeExecuteAsyncIO is a bit misleading.

@alexandru

This comment has been minimized.

Member

alexandru commented Mar 30, 2017

@djspiewak

What if we remove the ExecutionContext from the Async functions and simply expect that implementations which require it (e.g. Monix) capture that ExecutionContext when the typeclass instance is created? Basically the same thing that scalaz.Monad[scala.concurrent.Future] does. This is actually what fs2 does when constructing its instances, to ensure that the forked async constructor works properly.

I can see three problems with that:

  1. Async wouldn't be a type-class anymore, because instantiation now depends on a second parameter, besides the F[_] type and my concern here is inefficiency, because if you have to create a new instance every time an Async[F] is needed, this is no longer a zero cost abstraction
  2. We can no longer express the UnsafeIO OOP interface, which would be useful for hiding F[_] for certain use-cases ... this was @rossabaker's main complaint and even if he'll want to use any of this or not for Http4s (I'm pretty sure he won't), hiding F[_] is useful and OOP subtyping is the best mechanism we have for hiding it (e.g. Liskov Substitution Principle and all that)
  3. If Effect.unsafeExecuteAsyncIO no longer takes an ExecutionContext, it's only logical to do the same for Async.create as well - and FS2 needs that in Async.create, so will FS2 also take an implicit Strategy for creating an Async[Task] instance, or will it use a stack-unsafe version?

I know how higher-kinded polymorphism works, I've had my own share. Basically most of the time you're deferring to the user the responsibility of executing the thing (unless you're dealing with a Comonad). This Async / Effect / UnsafeIO abstraction would be for those rare use-cases where the library needs to trigger that execution.

For example we can have routines provided by both Monix and FS2 that convert back and forth between these task types, without Monix depending on FS2 or vice-versa. Think of an operation like:

fs2.Task("sample").to[monix.eval.Task]

Conversion would also work between Task and Future and this is important when a generic F[_] is involved. For example if you have a stream-like type, you can then describe a foreach that works as Scala users expect it to:

def foreach[F[_], A](fa: F[A])(cb: A => Unit)
  (implicit A: Async[F], M: Monad[F]): Future[Unit] = ???

You cannot express such conversions without having a way to extract that value from F[_] and even though this to function would need an implicit ExecutionContext in my proposal, the alternative is to not have this function at all.

But indeed, there are differences in these APIs, an ExecutionContext works like an interpreter for async evaluation and putting it in that API makes sense for implementations that use some sort of ExecutionContext in their implementation. But implementations have different designs, priorities and it's precisely these differences in evaluation that make them special.

So at this point I don't have an opinion on how to proceed. If you think an EC-less Async class would still be useful, then that's fine, but I'm less enthusiastic about it.

@alexandru

This comment has been minimized.

Member

alexandru commented Mar 30, 2017

So I am updating the code, taking out the EC, lets see where that takes us.

@alexandru

This comment has been minimized.

Member

alexandru commented Mar 30, 2017

OK, updated the code in https://github.com/alexandru/effects/tree/master/shared/src/main/scala/org/typelevel/effects with:

  1. removed ExecutionContext from the API
  2. replaced Callback with Either[Throwable, A] => Unit
  3. renamed functions on Effect to unsafeExtractAsync and unsafeExtractTrySync, mirroring extract from Comonad (PS: not using unsafePerformIO, run, runAsync or other variations of those is on purpose)

Let me know what you think.

@djspiewak

This comment has been minimized.

djspiewak commented Mar 30, 2017

Async wouldn't be a type-class anymore, because instantiation now depends on a second parameter, besides the F[_] type and my concern here is inefficiency, because if you have to create a new instance every time an Async[F] is needed, this is no longer a zero cost abstraction

I believe it can be encoded as an implicit value class, if allocations are your primary concern. Escape analysis gets 99% of this stuff in practice though, so most of the penalty is felt during JVM warmup. Finally, calls such as unsafeExtractAsync (I like the name, btw) are not generally something you're doing in the hot path, meaning that performance is far from a primary concern.

We can no longer express the UnsafeIO OOP interface, which would be useful for hiding F[] for certain use-cases ... this was @rossabaker's main complaint and even if he'll want to use any of this or not for Http4s (I'm pretty sure he won't), hiding F[] is useful and OOP subtyping is the best mechanism we have for hiding it (e.g. Liskov Substitution Principle and all that)

I haven't looked at the code in the last few hours, so maybe I missed that one. I really don't think @rossabaker using something like this is in the cards. Ross is primarily interested in efforts like this in so far as they can achieve a standard, concrete Task that he can just use. In lieu of a standard concrete Task, he's just going to default to fs2.Task. Abstracting over the effect is a non-starter for him both because of of the F[_] and because of the proliferation of implicits.

If Effect.unsafeExecuteAsyncIO no longer takes an ExecutionContext, it's only logical to do the same for Async.create as well - and FS2 needs that in Async.create, so will FS2 also take an implicit Strategy for creating an Async[Task] instance, or will it use a stack-unsafe version?

It would close over the EC. Note here and here, neither of which capture an EC. This all works because of this, which is doing precisely what I suggest: capturing the Strategy.

I know how higher-kinded polymorphism works, I've had my own share. Basically most of the time you're deferring to the user the responsibility of executing the thing (unless you're dealing with a Comonad). This Async / Effect / UnsafeIO abstraction would be for those rare use-cases where the library needs to trigger that execution.

I'm… not sure what you mean by this? It's clear that the abstractions are intended to provide the ability to both introduce and eliminate effect capture, and I think that is an excellent goal and worth abstracting over. This was never really in dispute.

For example we can have routines provided by both Monix and FS2 that convert back and forth between these task types, without Monix depending on FS2 or vice-versa.

Right, and functions like this (your to example) are precisely why this sort of thing is really great in principle.

Let's loop back to the concrete implementation suggestion though… Is there a reason not to provide a concrete "reference" implementation along with these abstractions?

@alexandru

This comment has been minimized.

Member

alexandru commented Mar 30, 2017

Finally, calls such as unsafeExtractAsync (I like the name, btw) are not generally something you're doing in the hot path, meaning that performance is far from a primary concern.

OK, you're maybe right.

Let's loop back to the concrete implementation suggestion though… Is there a reason not to provide a concrete "reference" implementation along with these abstractions?

I don't know what to think of that one.

My first instinct is to be selfish though. If there is a reference implementation, people will just use that, just like people are using Scala's Future and Scala's Either, because they are good enough.

This is an entirely subjective, superficial, wishy-washy feeling, but personally I never got into Scalaz because of its big, occupy-Scala nature, which might be totally unjustified, but in Scala's ecosystem I get the feeling we tend to prefer monoliths. But then my opinion is biased, of course, since naturally I want people to use Monix's Task.

@djspiewak

This comment has been minimized.

djspiewak commented Mar 30, 2017

My first instinct is to be selfish though. If there is a reference implementation, people will just use that, just like people are using Scala's Future and Scala's Either, because they are good enough.

This is an entirely subjective, superficial, wishy-washy feeling, but personally I never got into Scalaz because of its big, occupy-Scala nature, which might be totally unjustified, but in Scala's ecosystem I get the feeling we tend to prefer monoliths. But then my opinion is biased, of course, since naturally I want people to use Monix's Task.

I 100% agree this is a concern. This is part of why I was thinking that we bias the reference implementation for composability/formal-coolness. We could make it clear in the documentation that "this works, and it's convenient and cool, but if you're putting code in production you should use Monix's or fs2's Task instead". The problem it solves is really providing an "out of the box" IO type for Cats, which is something that I find myself needing far more often than you would think, as well as a type which is sufficient for http4s to assume.

Then again, http4s is already written against fs2 and unlikely to change on that point, so maybe http4s doesn't really need a "standard Task" since it already has a very direct and natural concrete type to select. So perhaps the big win here is really the ability to represent converters like your to function, rather than any reference, "batteries included" goal.

@rossabaker

This comment has been minimized.

Member

rossabaker commented Mar 30, 2017

I've had a hard time selling Cats to experienced functional programmers because the IO story is so murky. I spoke to people at the conference last week who see fs2.Task and monix.eval.Task and see two nice tasks and are afraid to adopt either because "what about the other?" Library developers want people to use their library, but app developers want libraries that click together. It's so simple in Scalaz: scalaz.concurrent.Task is flawed, but it's right there, it's integrated with the core library, and it has great network effects. This choice is missing from, and holding back, Cats.

A production-grade task in a library like this might get adopted in blaze, where dependencies are kept minimal. Then http4s could use that and lose the overhead of Future conversions in its blaze binding. So I'd say http4s would be interested in a production quality implementation here, especially if it could subsume some other extant tasks.

@alexandru

This comment has been minimized.

Member

alexandru commented Mar 31, 2017

@djspiewak

I 100% agree this is a concern. This is part of why I was thinking that we bias the reference implementation for composability/formal-coolness. We could make it clear in the documentation that "this works, and it's convenient and cool, but if you're putting code in production you should use Monix's or fs2's Task instead". The problem it solves is really providing an "out of the box" IO type for Cats, which is something that I find myself needing far more often than you would think, as well as a type which is sufficient for http4s to assume.

So lets see such an implementation, might be a good idea, along with these type-classes. Should we starts a cats-effects project and collaborate on it?

@alexandru

This comment has been minimized.

Member

alexandru commented Apr 3, 2017

Hi folks, so any further thoughts on this?

Coming up with a solution is a big challenge, but maybe we should solve this piecemeal in order to have progress.

At this point I think the Async and Effect type-classes are very useful at least for enabling seamless conversions between types, without those types knowing about each other.

How can we proceed?

I initiated this a different project for proof of concept, but I can make a PR in Cats if you think a sub-project of cats would be in order. I'd very much like to move this forward.

@alexandru

This comment has been minimized.

Member

alexandru commented Apr 3, 2017

Note: I would also like FS2 to depend on it, to have these conversions baked in by default, which is why I think a separate project makes a lot of sense, to avoid the dependency on cats-core.

@edmundnoble

This comment has been minimized.

edmundnoble commented Apr 3, 2017

I'd just like to re-iterate in the strongest possible terms that Future is not a tool for functional programming, and because it is not referentially transparent it already has no business at all even having a Monad instance. Adding an Async instance can only make this worse.

Perhaps I am assuming my point is more obvious than it is: Async.create cannot "sometimes" perform a side-effect and "sometimes" capture that side-effect to be executed later depending on the underlying instance. That is not a useful abstraction, it's an abstraction which will cause a crapload of bugs because Future is not a Task, and you cannot meaningfully abstract over Task and Future. All code that depends on Async will behave entirely differently depending on whether that Async is Future or Task, potentially executing way more effects (or even way less, seeing as Tasks are reusable) than the author intended, using old results silently in some areas and performing extra work in others.

@alexandru

This comment has been minimized.

Member

alexandru commented Apr 7, 2017

@kailuowang I already included Deferrable in this project. It's no longer a Monad, but that was a concern, due to MTL issues. The laws I described require a Monad instance though.

@alexandru alexandru changed the title from New Project: Typelevel Effects to New Project: Typelevel Schrodinger Apr 7, 2017

@mpilquist

This comment has been minimized.

Member

mpilquist commented Apr 7, 2017

Some miscellaneous thoughts:

  • This library is focused on middleware interop, not end user usage. To that end, I think we should have a good interop story for at least 2 libraries. If only Monix supports these type classes, I'm afraid we'll have proliferated another standard with no measurable improvements. https://xkcd.com/927/
  • I don't know if FS2 will directly depend on this or not. As of right now, FS2 has no dependencies. We've been debating lifting that restriction to improve the interop story but we haven't reached a conclusion yet. If FS2 doesn't depend directly on this, we could certainly create an interop bridge like we currently have for Scalaz and Cats.
  • I don't think cats-core should depend on this library. We need a rock solid, stable cats-core 1.0. I don't think effect capture is that stable yet.

Some implementation comments:

  • I don't think UnsafeIO should be in the library if I understand it correctly. It appears to simply add OO-style syntax for an Effect. If that's all it does, I think we should remove it. Alternatively, we could use Simulacrum to add OO syntax to all of the type classes. I think I prefer the former though.
  • The Async type class conflicts (name-wise) with fs2.util.Async - the latter is a much more specialized type class though, requiring an implementation of Async#ref.
@alexandru

This comment has been minimized.

Member

alexandru commented Apr 7, 2017

@mpilquist

Indeed, my hope is to convince you to depend on it and provide implementations, otherwise it's useless.

I understand the need for no dependencies, but the purpose of this library is to be stable. Once we release 1.0.0, there should be no reason to release another version of the "core", unless we did something stupid that needs to be fixed. Bridges or laws can change, but the core should be set in stone once ready.

This is why I haven't included Simulacrum in the project, because I would prefer for this project to be more stable than Simulacrum's encoding is (as that one can change depending on Scala's evolution).

If you want Async renamed or the removal of UnsafeIO, that's totally fine.
I would be thrilled if you accepted to be one of the project's maintainers ❤️

@milessabin

This comment has been minimized.

Member

milessabin commented Apr 7, 2017

On the licensing question, I'm be strongly in favour of Apache 2.0 (as I am for all Typelevel projects which aren't GPL).

@djspiewak

This comment has been minimized.

djspiewak commented Apr 7, 2017

Just wanted to drop a line saying that this hasn't completely fallent off my radar, and I haven't forgotten about my "let's implement a Task!" volunteering. Just been a bit busy this week. Hopefully I'll get back to it this weekend, but generally I think things are moving in a good direction.

Something to spark a little debate so we know where we're going though… This reference Task: how "production ready" do we actually want this to be? Initially I thought "do something formally minimal, composable and cool", but people are going to use it. Like it or not, whatever we do will become the cats Task/IO. The more I think about that, the more I think it's a good thing so long as we are able to get seamless (and by that I don't mean Task { ioa.unsafePerformIO() }) interop with Monix/fs2. fs2 is already parametric in its effect, so basically its own Task is already just a reference implementation. Monix isn't parametric (right?), but I think any Task-like thing is going to be seamlessly convertable. Which is basically a major point of this project. Nobody wants to obsolete anyone's Task, and I think as long as we do our jobs right, we won't.

So basically, are we going to push for production-ish? Or are we going to do something stupid-simple like Cont[IO[Unit], A]? (for reference, I'm almost positive Cont[IO[Unit], A] would be faster than scalaz's Task).

@mpilquist

This comment has been minimized.

Member

mpilquist commented Apr 7, 2017

@djspiewak You are correct that if we provide a reference implementation, folks will use it and it will become the defacto standard. Hence, I'd shoot for production quality or not provide one at all.

@djspiewak

This comment has been minimized.

djspiewak commented Apr 7, 2017

@mpilquist A related question… Do we want IO and Task? Or just Task? I tend to think just the latter, but as djspiewak/cont-exp shows, there is still some value (albeit experimental) in having a non-async IO.

@alexandru

This comment has been minimized.

Member

alexandru commented Apr 7, 2017

@djspiewak @mpilquist I think at this point a good question would be: what does production ready mean?

From my point of view, Monix's Task is production ready because it gets down and dirty in its implementation, using concurrency primitives defined in the monix-execution sub-project, that would have never happened as part of Cats, because it's basically a complement to scala.concurrent.

For example it provides cache-padded atomic references, which are made to use platform intrinsics, so they work on Java 7, 8 and they'll be optimal on Java 9 as well. These are there to help with shared JVM/JS code, for reusable cancelables, queues and locks, schedulers, etc.
I might take this even further for non-blocking queues, as for example monix-reactive already depends on JCTools.org, a bunch of awesome non-blocking queues implementations making use of dark magic basically.

A reference implementation cannot do that, without introducing ugly code in Cats, duplicating the work needlessly, or ending up with a sub-optimal implementation.

And this matters, because performance is a competitive advantage, if you introduce a Task in Cats, it will have to keep up - I was informed for example that the implementation in Scalaz 8 might be inspired by Monix's. And also, based on Task you can move on to other things, other concurrency primitives, like for example MVar, circuit breaker, semaphore and many others are possible.

And in my experience users are glad for these when they have it - not necessarily knowing they are possible. So where do you draw the line if a Task implementation would be in Cats? Because that Task won't be pure and nice at all, or a good community player.

I think we should do this piecemeal though, I really wish for these interfaces to go through.

@djspiewak

This comment has been minimized.

djspiewak commented Apr 7, 2017

From my point of view, Monix's Task is production ready because it gets down and dirty in its implementation, using concurrency primitives defined in the monix-execution sub-project, that would have never happened as part of Cats, because it's basically a complement to scala.concurrent.

This is essentially where we have a very different perspective on what Task "is". Task, to me, is just an IO that supports asynchronous actions. That's it. It shouldn't have any concurrency stuff, or resource management, or anything really other than that bare minimum. So long as the implementation is clean and transparent, concurrency stuff can be built on top of that.

So in other words, I don't see any need to build something which is a complement to scala.concurrent. fs2 and Monix already do that better than we would want to in this project.

And with respect to performance, it's not that hard to get within rounding error of Monix's Task for most straight-line stuff, especially when concurrency is off the table. Monix's Task is faster than fs2's, but only barely for simple things, and only because of some inlined specialized implementations. I don't have a problem replicating that trick, or not replicating it and just leaving things slightly cleaner. Either way it makes relatively little difference. Monix's real performance wins are elsewhere, and undisputed, and there's no real need for a reference implementation in this project to try to "keep up" with that. Especially if we're not implementing concurrency things.

And also, based on Task you can move on to other things, other concurrency primitives, like for example MVar, circuit breaker, semaphore and many others are possible.

If concurrency stuff (even apply!) is out of scope for cats.effect.Task, then clearly all of this stuff is as well. In my view, if the things you listed (and more) are not possible in a third-party library, built on top of cats.effect.Task, then we did something wrong.

I think we should do this piecemeal though, I really wish for these interfaces to go through.

I agree that we need to nail down the typeclasses. But a reference implementation here is really really important. People care about interop, yes, and pruning down on the constant reinventing of these concepts. But ultimately, I hear a lot more from people about the desperate need for "a cats Task".

@alexandru

This comment has been minimized.

Member

alexandru commented Apr 7, 2017

OK then, if you folks are on board with that.

A random thought: Cats already has Eval for IO and it just implemented MonadError as well. And so I would name this reference Task as IO, because on top of Haskell you end up working with IO for async stuff as well, see for example Async#wait. I think Scala's IO should be async, because we don't have the luxury of this wait.

And we don't have to keep familiarity with Scalaz users, the familiarity we should keep is with Haskell and for that purpose an async IO is fine imo.

@alexandru

This comment has been minimized.

Member

alexandru commented Apr 7, 2017

And yes, this naming proposal would have the benefit of allowing different implementations trying to innovate on that concept to have a different name 😄

@djspiewak

This comment has been minimized.

djspiewak commented Apr 7, 2017

@alexandru I'm entirely ok with calling it IO and giving it async structure. I'm a little squeamish about the fact that Eval has a MonadError instance, but that's a freak-out for another day. :-P

Everybody else cool with the "cats Task" being called IO?

@alexandru

This comment has been minimized.

Member

alexandru commented Apr 7, 2017

@alexandru I'm entirely ok with calling it IO and giving it async structure. I'm a little squeamish about the fact that Eval has a MonadError instance, but that's a freak-out for another day. :-P

I swear, it wasn't my doing 😝

@djspiewak

This comment has been minimized.

djspiewak commented Apr 7, 2017

@alexandru THINK OF THE CHILDREN 😇

@fommil

This comment has been minimized.

fommil commented Apr 7, 2017

a common effects library is a good idea. But, perhaps we're taking the naming convention a little too far... I'd be all in favour of pulling a lot of these things back into cats: alleycats, kittens, dogs, schrodinger. It's all fun and all, but maybe it's getting a bit much?

@djspiewak

This comment has been minimized.

djspiewak commented Apr 7, 2017

@fommil The more things in one library (and repo), the more moving parts there are, thus the more compatibility surface area you have to worry about between versions. Think: scalaz 7.1 to 7.2. I'm very strongly in favor of keeping repos as small and as modular as possible until pragmatically forced to squish them together.

@fommil

This comment has been minimized.

fommil commented Apr 7, 2017

@djspiewak I never used scalaz. But I do know that all these things are so tightly coupled already that a release to cats means everything else needs to be updated, and there is no way the infrastructure is in place to ensure proper binary compatibility because that has the cost of something like a community build. As somebody who's trying to learn all this stuff, it's very confusing.

@djspiewak

This comment has been minimized.

djspiewak commented Apr 7, 2017

@fommil A better question is if a binary-incompatible release of dogs requires a binary-incompatible release of cats. The answer to that is "no", but it would flip to "yes" if dogs and cats were squished together. This is precisely why keeping them separated produces a superior dependency story.

@kailuowang

This comment has been minimized.

Member

kailuowang commented Apr 7, 2017

@djspiewak if we decide to do it, is this cats.effect.IO required for cats 1.0.0-RC1? i.e. do you see it being required by the integration between doobie, http4s and cats?

@djspiewak

This comment has been minimized.

djspiewak commented Apr 7, 2017

@kailuowang I don't see it as required for cats 1.0. That's not to diminish the importance of cats-effect, just noting the fact that it's a separate repo and a separate dependency and doesn't need to couple or hinder the upstream effort in any way. Cats 1.0 is more important than getting a standard IO out the door in any case.

@tpolecat

This comment has been minimized.

Member

tpolecat commented Apr 7, 2017

@kailuowang Hi, sorry, I'm behind on this thread and need to read it top to bottom. There have been a number of proposals and I'm not sure where this ended up. But off the top of my head I can't see this being a blocker for doobie. I'll follow up later today.

@alexandru

This comment has been minimized.

Member

alexandru commented Apr 7, 2017

To help people understand this thread /cc @tpolecat @djspiewak

See updated issue description.

Cheers,

@mpilquist

This comment has been minimized.

Member

mpilquist commented Apr 7, 2017

👍 for an async 'IO' reference implementation.

@djspiewak

This comment has been minimized.

djspiewak commented Apr 8, 2017

@alexandru Ok, so if conversion is literally the only goal here, then I think we should pare this down to the following:

trait Catchable[F[_]] {
  def attempt[A](fa: F[A]): F[Either[Throwable, A]]
  def fail[A](t: Throwable): F[A]
}

trait Suspendable[F[_]] extends Catchable[F] {
  def suspend[A](thunk: => Either[Throwable, A]): F[A]
}

trait Async[F[_]] extends Suspendable[F] {
  def async[A](k: (Either[Throwable, A] => Unit) => Unit): F[A]
}

And that's it. Names can be whatever; doesn't make that much of a difference. There's no need for extractors if all you want is conversion. For example, here's a to function that could be copy/pasted into fs2.Task:

def to[F[_]: Async]: F[A] = Async[F].async(cb => attempt.unsafeRunAsync(cb))

In other words, you only need to know how to put effects in to constructors characterized by these typeclasses; you don't need to know how to take them out. This does mean that the typeclasses in question are lawless (it's sort of like defining Pointed), but it avoids muddling the story with different extractors, and it keeps the footprint to the bare minimum.

As for the reference IO, that seems out of scope for what @alexandru wants to do with this project, so I retract the suggestion. If there is interest and consensus, I will start a "cats-effect" project proposal, which would implement IO, probably define implementations of the above typeclasses, and directly extend cats-core. Thus, what I'm soliciting interest in is a standard cats.effect.IO type with support for async effects. But as I said, it really seems like that effort is out of scope for the project here.

@kailuowang

This comment has been minimized.

Member

kailuowang commented Apr 8, 2017

@djspiewak I am interested in a standard cats.effect.IO. I think it is going to be very helpful.

@tpolecat

This comment has been minimized.

Member

tpolecat commented Apr 8, 2017

So, after all this discussion (here and elsewhere) I'm now pretty convinced that abstracting over these data types isn't solvable in a useful way. The runtime considerations of these implementations are just too different, and abstracting over them will end up placing a burden on end users. You can't sweep this under the middleware rug.

The current reality for me is that I really only have Monix Task and fs2 Task to worry about. scalaz is EOL software as far as I'm concerned so I don't care about those data types. I don't know of any project that's attempting to abstract over streaming libraries (and thus associated Task impls) so the whole question may not be important anymore.

At least for now my approach for doobie will be to interpret into fs2.Task, and if someone wants to write an interpreter for another IO type it's straightforward to do. I may be able to push streaming out of my core and add support for Monix as well, but for now my goal is to get to dry land.

The question of what to do if you need an IO type for Cats and you're not using a streaming library has no obvious answer, especially if fs2's Task gets pulled out into a separate library like Monix's, as has been discussed. The ideal solution for everyone would be for fs2 and Monix to agree on a common one true and holy Task and call it good, but I don't get the sense that this is going to happen anytime soon. Providing a third "reference" implementation would be bad. Don't do that.

So I think I'm 👎 on this line of development. At least for me, in the Cats 1.0 timeframe, the best path forward that I can see is the status quo. Things may become more clear as Cats becomes the standard FP library … perhaps a natural winner will emerge. Dunno.

@alexandru

This comment has been minimized.

Member

alexandru commented Apr 8, 2017

I don't know of any project that's attempting to abstract over streaming libraries (and thus associated Task impls) so the whole question may not be important anymore.

There is in fact one such project that I mentioned in the description: https://github.com/reactive-streams/reactive-streams-jvm/blob/v1.0.0/README.md#specification

What it does is to provide the needed interfaces to transport data over library boundaries. This is how Monix Observables, RxJava, Akka Streams, Project Reactor and others are able to communicate ... well, except for FS2, at least until now.

I have been involved in the discussions for defining this "Reactive Streams" protocol, compromises have been made, but the result is a public and simple protocol that you can depend on for interoperability.

What I'm trying with this project is to define an equivalent interface for Task, IO and Future-like types.

When you provide the means to seamlessly convert back and forth, you solve the primary problem that users have.

Unfortunately I'm seeing this conversation sidetracked into issues that can be treated separately and that aren't necessarily related. And people that open this thread are going to be lost in noise.

Therefore I now have the following plan:

  1. This project I'm keeping on the @monix organization - if there's consensus, I'll gladly move it to @typelevel and I'll gladly take contributions too
  2. I'll seek projects that want to participate in depending on those interfaces, I do hope that I'll convince FS2 folks to do it, although it's going to be much harder if this is non-Typelevel

Will try and polish this to make it more desirable, have pushed this idea a little half-baked due to seeking some collaboration. But unfortunately half-baked ideas are also confusing.

So I'm retracting this for now. I invite anyone that has feedback on the project's issue tracker: https://github.com/monix/schrodinger

After the idea is a little more polished I might raise the issue again.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment