Let's discuss the cancelation model

[alexandru]

I may revive an old topic that was already discussed. Sorry about that, better late than never 🙂

Take this snippet of code, that continuously writes lines of text into a file (a resource), and we attempt killing it via a timeout ...

import cats.effect._
import java.io._
import scala.concurrent.duration._

object Hello extends IOApp {

  override def run(args: List[String]): IO[ExitCode] = {
    val task = IO.bracketFull { _ => 
      IO(
        new BufferedWriter(
          new OutputStreamWriter(
            new FileOutputStream(new File("/tmp/hello")),
            "utf-8"
          )))
    } { reader => 
      IO {
        var line = 0
        while (true) {
          println(s"Writing line $line ...")
          reader.write("Hello, world!\n")
          reader.flush()
          line += 1
          Thread.sleep(1000)
        }
        ExitCode.Success
      }
    } { (reader, outcome) =>
      IO {
        println(s"Ending with: $outcome")
        reader.close()
        ()
      }
    }
    
    // Race condition here...
    task.timeout(10.seconds)
  }
}

In Cats Effect 2.x this program behaves as expected (to me anyway) — it triggers a TimeoutException after 10 seconds and exits. It may also log an IOException via the thread's default error handler (because that Writer ends up forcefully closed).

In Cats Effect 3.x this program is stuck forever. It can't be killed by SIGHUP / SIGTERM either, so IOApp too is problematic — it definitely defies common sense, being very unlike normal app behavior (because you can interrupt a Main with a while (true), by default, in most programming environments I've seen).

And I fear this is by design, a design (possibly imported from ZIO) with which I deeply disagree with. If this behavior is by design, it completely misses the main purpose of cancellation.

Cancellation is NOT user input. We have other ways for modeling user input. If we want to detect the client's intent to exit, we could share a Defer (aka fancy Future), or some sort of stream. Cancellation is forceful shutdown in case of a race condition. Cancellation is not a SHOULD, but a MUST. It doesn't matter if we now have an IO.blocking that's well-behaved. Being able to do a forceful shutdown of resources should not rely on good client behavior.

In the old Cats Effect 2.x the problem was that the resource could have been closed while in use, concurrently, leading to an IOException and possibly corrupted data. But that was better than this — one does expect corrupted data in case of cancellation, being normal. Whereas this behavior here can create a zombie process that can't be killed without a SIGKILL, which is not normal, defeating common sense.

---

In terms of network protocols, getting confirmation from the client that a "connection" is no longer used or closed is a pretty bad idea, because the client might misbehave. In case of a race condition (such as a timeout) a cancel() is not an operation that you want to back-pressure, before or after invoking it.

The perennial example is the TCP protocol. Google is filled with articles on detecting "closed TCP connections" or "broken TCP connections". Detecting liveness is its own can of worms, but even closing a connection is pretty challenging, as it goes via a complex handshake, like this:

Note how the protocol has states like FIN_WAIT_1, TIMED_WAIT, FIN_WAIT_2, CLOSE_WAIT. These states are reflected at the kernel level. They are timed-out by the kernel at some point, because either the client or the server can completely stop responding.

Even with that timeout, because it's a generous one, with huge traffic you can still end up with so many opened file handles that you can exhaust the available file handles, with the kernel refusing to allocate more, temporarily, until the timeouts are triggered. You can then, of course, modify the OS's settings, e.g you can increase the file handles available (since they are cheap), you can decrease the timeouts, etc.

The question is ... if a client is misbehaving, should the app freeze because of it? If it's a long-running server we're talking about, the answer is almost always no — the right answer is for the client's connection to get killed. And it is true that sometimes we can't stop a client easily. For example inside a JVM process this can't be easily killed:

IO { while (true) {} }

And that's fine. IO.race and IO.timeout should still work on it though. Yes, this too creates a leak, which may end up crashing the process, but that's a better outcome than a frozen process that can't be restarted by a supervisor 😉

---

Also, IOApp should forcefully kill the app when repeatedly getting SIGHUP / SIGTERM signals. That you can't specify such a behavior via normal mechanisms (other than giving up and manually triggering a System.exit at some point) indicates we have a problem.

---

Speaking of, it always seemed wrong that bracket's release triggers an IO[Unit], which can be asynchronous in case of cancellation. A release in case of cancellation should always be synchronous. If you have to block on anything to do a release in case of cancellation, that's just wrong.

I always disliked bracket, feels like an improvement over the status quo, but it invalidates the reasoning we have with Monad and I'm getting the feeling that it is a bad gift that keeps on giving 🤷‍♂️

---

Sorry for my strong opinions, I just had an emotional reaction 😅

[vasilmkd]

Just to offer the "idiomatic" CE3 solution to this problem:

import cats.effect._
import java.io._
import scala.concurrent.duration._

object Hello extends IOApp {

  override def run(args: List[String]): IO[ExitCode] = {
    val task = IO.bracketFull { _ => 
      IO(
        new BufferedWriter(
          new OutputStreamWriter(
            new FileOutputStream(new File("/tmp/hello")),
            "utf-8"
          )))
    } { reader => 
      IO.interruptible(false) {
        var line = 0
        while (true) {
          println(s"Writing line $line ...")
          reader.write("Hello, world!\n")
          reader.flush()
          line += 1
          Thread.sleep(1000)
        }
        ExitCode.Success
      }
    } { (reader, outcome) =>
      IO {
        println(s"Ending with: $outcome")
        reader.close()
        ()
      }
    }
    
    // Race condition here...
    task.timeout(10.seconds)
  }
}

[alexandru]

I mentioned on Discord that ...

It doesn't matter that you can fix it with special code. The current timeout implementation defeats the purpose of what timing out operations is about.

To add extra food for thought ... what is timeout all about? Why do we need timeouts at all?

Well, we're executing asynchronous operations that could be running on another thread, or another node on the network, etc. Asynchronous programs can fail outside the current thread, silently. A network node can disappear, a thread can get killed, a piece of code can die with an exception that never gets logged, etc.

The only way to reliably notice malfunctions, the only way to notice non-termination actually, is by measuring time and triggering a concurrent operation that makes us understand that an error happened, as we may want to do something about it, like (forcefully) closing the underlying connection, to free it up for other clients.

Asynchronous tasks involve client-server or producer-consumer communications. The client can notice that the server is down via timeouts. The server can notice that the client is stuck via timeouts. Etc.

And here's the problem ... given a malfunctioning actor, if you have to wait on its confirmation, that entirely defeats the purpose of the timeout operation. Context doesn't matter, it's basically baked in the semantics of timeout operations that it gets triggered at the specified time and not later.

If you actually need back-pressuring on those connections being returned to the pool (or something), that can always be done with a Semaphore. And you don't add that cost, and risk, everywhere.

[SystemFw]

. given a malfunctioning actor, if you have to wait on its confirmation, that entirely defeats the purpose of the timeout operation. Context doesn't matter

Well, context does matter in that you assume the only reason to timeout is a malfunctioning actor (that's the context in which you are right, which isn't always the case). I disagree that we cannot give any "communication" meaning to these operations, or at least allowing that possibility, especially if you generalise from timeout to generic interruption (which you might not be proposing, apologies if that's the case).

If you actually need back-pressuring on those connections being returned to the pool (or something), that can always be done with a Semaphore. And you don't add that cost, and risk, everywhere.

Backpressuring a "leaky" interruption model is nigh-impossible, as we've seen in years of using CE2, and at the very least involves changing the implementation of the code you want to wait upon, whereas not waiting on a back pressured model is pretty easy : timeout(2.seconds).start.flatMap(_.joinWithNever)

[alexandru]

Wait, what is timeout(2.seconds).start.flatMap(_.join) supposed to do?

Backpressuring a "leaky" interruption model is nigh-impossible

Maybe it's because back-pressuring interruption might actually be wrong 🤷‍♂️

Note that Cats-Effect (and ZIO) are pretty much alone in doing this, the definition of being opinionated. If you need to back-pressure on something, that's not interruption.

[SystemFw]

what is timeout(2.seconds).start.flatMap(_.join) supposed to do?

Actually scratch that, I over simplified it, but it is possible to write an operation with the semantics you want given this cancellation model, using racePair and cancel.start (which ignores the waiting), whereas the opposite isn't true, if the cancelation model is leaky, and you need to wait, good luck (at the very minimum you need to change the task itself).

Maybe it's because back-pressuring interruption might actually be wrong

yeah but remember that we didn't start with the idea of wanting to backpressure interruption, we started with the model you are describing as good, and ended with a lot of issues related to resource safety (e.g. not respecting reverse order of acquisition for finalisation). The change to a back pressured model happened in CE2 because of these issues (and you added it lol), the CE3 changes only made it more consistent (in CE2 there were some implementation details in IO where back pressure would sometimes happen and sometimes not).

If you need to back-pressure on something, that's not interruption.

That's also very opinionated, and note that we do have precedent here: Haskell's base model is "cancellation does not back pressure" , and as it turns out it works poorly, and people hate its unsafety, hence why the advice is to use the Async library as a higher level wrapper instead of the base api, which does indeed back pressure on cancellation. The unsafety is so high that many people go as far as saying that haskell shouldn't have async exceptions at all.

So on balance, the fact that you can write .leakyTimeout just fine in the current model does address your concerns, imho.

[alexandru]

(2) here's a related problem ... there's no way to "observe" cancelation from the use clause of a bracket ...

IO.bracket {
  IO(new AtomicBoolean(true))
} { isActive =>
  IO {
    while (isActive.get) {
       println("Still active")
       Thread.sleep(1000)
    }
    println("OK, OK, stopping...")
  }
} { isActive =>
  IO(isActive.set(false))
}

This makes bracket a lot less useful, and I might add, confusing, given the concurrent nature of "cancelation".

[oleg-py]

Note that bracket is a lot less necessary in CE3. uncancelable is the most flexible primitive out there, which bracket is built out of, and Resource handles the pains of nesting brackets as it became very pervasive.

I do feel that this "observing cancellation" idea is similar to cooperative vs. preemptive scheduling, except it's cancellation. CE API describes a preemptive cancellation model, and the idea of cooperation simply doesn't fit there. You can only use cooperative bits in Async.async when wrapping a low-level API that supports such model - which, arguably, is what your code should be using instead. Or use IO.blocking(Thread.sleep(1000)).foreverM[Unit] instead of a while loop which would have automatic cancel boundaries.

[SystemFw]

So, I have to say, I don't find bracket confusing at all: it's literally "bracketing" operations: if isActive.set(false) could happen whilst use is running, it would no longer be "bracketing" (like, visually, even).

I understand the semantics you expect in that snippet, but I don't think you're describing bracket , you're describing the old Monix-style cancelable instead.

Those semantics are mostly useful when you need to implement cancellation manually and cooperatively, but the thing is that in ce you rarely need to!

1. Tasks composed of IOs will be canceled
2. With interruptible, a good amount of blocking tasks will be canceled too,
3. For the few things that don't fit either model, well, bracket is not a primitive and it should be possible to implement the semantics you envision (monixCancelable, really) as a separate combinator.

[oleg-py]

Did we just write same ideas within same minute?

[SystemFw]

This issue can be summarised once again as the perennial and unsolvable dilemma between two opposing notions of safety: deadlock safety and resource safety.

Your argument is: if I'm cancelling something, and it's faulty/not cooperating, I will deadlock.
The opposite argument is: if I'm cancelling something with a finaliser, I need to wait on that finaliser, or I will break reverse order of acquisition, and violate resource safety.

The problem is that without back pressure, all nested brackets are broken, and fixing them requires changing the definition of the bracketed tasks (which you might not own) and add waiting code that is often beyond the reach of even maintainers, whereas with this model most code is deadlock safe (because it can be canceled with IO, or even with interruptible), and worst case scenario you can do .cancel.start and not wait.

[djspiewak]

First off, thanks for bringing this up, because it's a fascinating topic that deserves more attention. There's a lot to unpack here!

What you're getting at is exactly what @SystemFw noted: backpressure, and more generally resource leaks. First off, it's important to underscore the fact that cancelation cannot be a must signal, it can only ever be a hint. This isn't just a limitation of the JVM, it's a limitation of resource-safe code in general. The only way to ensure that resources are deterministically acquired and released is to create critical sections in which cancelation is deferred. Now, it is best practice to ensure that such critical sections utilize Poll to guard any asynchronous blocking, entirely avoiding the worst consequences of the issue you're raising, but there is no way to enforce this.

delay blocks in particular are a very special case, since we literally have no idea what they're doing, and we have no way of signaling into them that it's time to stop doing it. This is why interruptible exists (though note that interruptible imposes overhead and also may not even be safe in certain contexts, which is why it is not the default. Since we can't stop the delay from running, our only option is to just let it complete and then respect the cancelation.

Pretending to respect the cancelation immediately would really just be a lie. We haven't canceled anything; the delay block would still be executing! If the user does this sort of thing in a loop, they can leak resources and crash their process. That's not a hypothetical either, this has literally happened to me several times unwittingly on CE2 because I tend to forget that CE2's IO does not implement this semantic. And this is assuming that something even worse doesn't happen, such as a long-running delay which is using a resource which we immediately free when we cancel without waiting for it to complete, resulting in an invalid state and unpredictable results (not always exceptions!). I've seen this happen as well on several occasions, and it's always a very unpleasant and very nondeterministic problem to track down.

The final nail in the coffin, from my perspective, is the fact that users can always emulate this "fire and forget" model of cancelation by using start on their cancel call. For example, it would be easy to define a timeoutAndForget combinator which has exactly the semantics you want here. However, going the other direction (emulating CE3 cancelation semantics on top of CE2) would be almost impossible in general.

Also, IOApp should forcefully kill the app when repeatedly getting SIGHUP / SIGTERM signals. That you can't specify such a behavior via normal mechanisms (other than giving up and manually triggering a System.exit at some point) indicates we have a problem.

IMO this sounds like a good enhancement for IOApp. If someone sends SIGTERM repeatedly in a short time window, I'm cool with just murdering the runtime rather than waiting for the finalizers. It is worth underscoring though that kill -9 still works on the process even when it is "zombied" due to waiting for finalizers. The semantic IOApp is implementing is basically identical to the semantic of application finalizers in any other runtime.

[alexandru]

@djspiewak @SystemFw @oleg-py going to write a general answer, to not repeat the same answer 🙂

Regarding that timeout, the most important bit is that the implementation doesn't do what people expect it to do, and the implementation picked the wrong default.

Resource safety is absolutely meaningless if you don't see that there's misbehavior. And you can't see that there's misbehavior without that TimeoutException being triggered at the correct time, exactly after 10 seconds, no later than that. That app that I just described cannot be killed without a SIGKILL (kill -9) and there's absolutely no indication that there's a problem with it. It won't be noticed by any supervisor that there's a zombie, because by definition of zombie processes, this isn't one, as it has no detectable deadlocks or live locks, and the app responds to input just fine, it just chooses to completely ignore it.

In real life that's awful — and we might be able to fix IOApp somehow, at least to give some sort of warning, however IOApp is really not the problem here, but the cancelation model which is missunderstood to be more than it's supposed to be, manifested in this default of waiting on tasks to finish. I should make clear here that waiting on tasks to finish is never a requirement for actual "race conditions" — race and timeout are not doing what people expect, and this statement is not opinionated at all.

Cats-Effect 2's .timeout isn't leaky BTW, the above example does not leak in Cats Effect 2 at all. This CE3 implementation leaks an entire process, as it can't be shut down.

we started with the model you are describing as good, and ended with a lot of issues related to resource safety (e.g. not respecting reverse order of acquisition for finalisation).

That's also probably because on cancelation we should probably not respect the reverse order of acquisition. On cancelation the important thing is for resources to be closed as fast as possible, to prevent leaks. People ended up expecting that due to issues with the API of bracket, which did not make it clear. In my opinion a more correct and sane API for bracket would be this one:

def bracket[R, A](acquire: IO[R])(use: R => IO[A])(
  complete: R => IO[Unit],
  cancel: R => SyncIO[Unit]
)

cancel should be treated as just a signal. The producer is responsible for correctly closing that resource. But it's not the responsibility of the producer to fix broken clients (such as use of IO.apply instead of IO.interruptibly), and cancelation happens with broken clients.

If you have to do "one last thing" on cancelation, then that's not cancelation. That's just normal user input that can be treated differently. SIGHUP in IOApp should probably not be treated as cancelation, initially (until there's a timeout or the user insists). The difficulty here is that we have a serious mismatch in semantics.

That Haskell's base model is "cancelation does not back pressure" is IMO yet another confirmation that our model in CE3 might be wrong. Popularity isn't proof enough, but it's the strongest signal we have. Reactive Streams, RxJava, Project Reactor and others don't back-pressure cancelation. And I don't think Haskell's unsafety comes from that IMO, but from its concept of async exceptions and from believing that bracket has responsibilities that it doesn't really have.

---

Your argument is that we can implement a leakyBracket and a leakyTimeout anyway. Which is true, sort of, except where it matters.

Here's why the order of acquisition shouldn't matter much on cancelation, this being another instance of a leak:

// Faulty release ;-)
val never = IO.interruptibly { 
  while (!Thread.currentThread.interrupted()) {} 
}

val res = for {
  r1 <- Resource(IO { (???, IO.unit) })
  r2 <- Resource(IO { (???, never) })
} yield (r1, r2)

res.use { case (r1, r2) => IO.never }
  .timeout(10.seconds)

Note how:

• the result is stuck, just like that IOApp above
• both resources are stuck instead of a single one
• you cannot override the cancelation model to not behave like this, since Resource's implementation is set in stone
• note that Java's interruption model (in case of usage with try/catch/finally) can be triggered multiple times to "unstuck" such cases, whereas Cats-Effect has no such protocol; in other words, in this regard, Java is safer!
• that timeout is useless as it doesn't do what it's supposed to do — which is to signal that there is a problem

And I don't think it's feasible to have people implement their own LeakyResource too.

In other words, this isn't really a choice. Something that bothered me ever since we modified Cats Effect's protocol to back-pressured cancellation (although CE 2 was IMO far better behaved in this regard, even if it had some confusing corner cases, because at least some operations like timeout worked properly).

[alexandru]

I should also add that this is a major change in behavior that will not be reflected in the type signature and might end up silently breaking people's applications when it matters (when problems happen such that cancelation is actually needed).

The migration guide contains no such thing. That's fine when you consider this to be normal behavior, tough luck on those implementations that do otherwise, but my argument is that this isn't normal behavior at all and will catch people by surprise.

I'm speaking of race, timeout, bracket and Resource here.

https://typelevel.org/cats-effect/docs/migration-guide

[alexandru]

One last note — it's probably fine to back-pressure cancelation in some instances.

But race and timeout should definitely not do that.

And Resource should probably have a use overload that specifies what to do in such situations. Ideally it should be able to time-out the chained finalizers individually too, via some sort of configuration.

[djspiewak]

Cats-Effect 2's .timeout isn't leaky BTW, the above example does not leak in Cats Effect 2 at all. This CE3 implementation leaks an entire process, as it can't be shut down.

This really isn't true though. It can be shut down, you just can't have completely uninterruptible things.

Also Cats Effect 2's timeout does in fact leak. A simple example:

IO(while(true) { doSomethingExpensive() }).timeout(1.second).attempt.foreverM

In Cats Effect 2, this program will infinitely spawn expensive things in the background, never giving you any indication that there is a resource leak. In Cats Effect 3, it will backpressure and wait for the first expensive thing to finish (which it won't) before moving on.

Ultimately, it's on the users to not create blocking uncancelable regions, and delay is exactly this. What this is getting at is the mental model for cancelation itself, and transitively, timeout. Note that race itself trivially derives from start, cancel, and Deferred and has no special semantics in and of itself, and of course timeout is derived from race. But coming back to the mental model…

I'm not sure what you mean by "user input" in this context, but cancelation is most definitely not a system-level signal. We can't interrupt the process, and pretending that we can is just going to get users in trouble. Also this isn't a hypothetical. I've had multiple production bugs directly caused by Cats Effect 2's semantics in this area. We can't interrupt something that is uninterruptible, and attempting to magically give users "fire and forget" semantics while pretending that we can is just not the right default. It's better to be explicit.

While I agree that the special case of IOApp deserves special treatment in and of itself, I don't agree that the system default should be to pretend that cancelation is always achievable. It's a user-land signal by definition, and needs to have semantics consistent with such.

The migration guide contains no such thing. That's fine when you consider this to be normal behavior, tough luck on those implementations that do otherwise, but my argument is that this isn't normal behavior at all and will catch people by surprise.

I do agree that the migration guide should probably mention this, as it is a subtle departure from CE2's semantics. With that said, people have repeatedly been bitten by CE2's semantics, particularly around resource release order, so I suspect that in most cases it will be a welcome course correction.

That's also probably because on cancelation we should probably not respect the reverse order of acquisition. On cancelation the important thing is for resources to be closed as fast as possible, to prevent leaks.

That's unsafe though. For example:

bracket(acquireRsrc1)(r1 => bracket(acquireRsrc2(r1))(r2 => ...)(r2 => r2.release))(r1 => r1.release)

Note that r2 is using r1. Presumably, its release could use r1 as well, but if we just go ahead and release r1 without waiting for r2 to be finalized, then we lose all safety guarantees and create a very subtle nondeterministic bug that users have to track down. This also breaks the try/finally intuition.

In my opinion a more correct and sane API for bracket would be this one:

Actually bracket is a derived combinator in CE3. The true primitives are uncancelable, onError, and onCancel. So you can already define a bracket with that shape if you would like.

you cannot override the cancelation model to not behave like this, since Resource's implementation is set in stone

That's not strictly true. You can wrap every Resource finalizer you define with .start, which will basically get the semantic you're asking for.

can be triggered multiple times to "unstuck" such cases, whereas Cats-Effect has no such protocol

Actually interruptible implements a many = true semantic which achieves exactly this. And #1992 has been filed to track the implementation of the repeated interruption specifically in IOApp. I'm not at all convinced that repeated cancel is a reasonable semantic to implement entirely within code space. Idempotency of cancel is something that I've seen quite a bit of code in the wild just assume and I wouldn't want to break that.

in other words, in this regard, Java is safer!

Java is less likely to deadlock in the primary execution flow. I'm not sure I would call it "safer" since the tradeoff is leaky and unreliable resource handling.

that timeout is useless as it doesn't do what it's supposed to do — which is to signal that there is a problem

I mean, it really depends on what you believe timeout is "supposed to do". Signaling that there's a problem is an important semantic, yes, but so is ensuring that resources have been properly cleaned up. If timeout is unable to do that then it needs to backpressure until such time as those resources have been released. Resources are not infinite! This too is specifically a major production bug that I saw during an outage (as in, literally timeout's semantics here). It's really not hypothetical.

But if you really want the semantics you're asking for, they're quite easy to get:

def leakyTimeout[F[_]: Temporal, A](fa: F[A], duration: FiniteDuration): F[A] =
  MonadCancel[F] uncancelable { _ =>
    Spawn[F].racePair(fa, Temporal[F].sleep(duration)) flatMap {
      case Left((oc, canceler)) => 
        canceler.cancel *> oc.embedNever

      case Right((f, _)) => 
        f.cancel.start *> MonadThrow[F].raiseError(new TimeoutException)
    }
  }

You can't go in the other direction though: it's impossible to take leakyTimeout (or leakyRace for that matter) and derive timeout or race.

[alexandru]

OK, I have thoughts — since yesterday. My proposal is basically this ...

1. timeout and race should revert to previous behavior, but I wouldn't mind a configuration parameter for what to do with those finalizers
2. bracket and Resource are fine with the current default, but we need to be able to override that behavior (in case of faulty finalizers)

So how about a configuration like this:

sealed trait OnCancelBehavior

object OnCancelBehavior {

  case class Backpressure(timeout: Option[FiniteDuration]) 
    extends OnCancelBehavior

  case object FireAndForget 
    extends OnCancelBehavior

  val default = BackPressure(None)
}

We can then pass this configuration where needed, like in overloads for timeout, race, bracket, Resource#use, etc.

def timeout(d: FiniteDuration, c: OnCancelBehavior): IO[A]

// ...
abstract class Resource[F[_], R] {
    def use[B](c: OnCancelBehavior)(f: R => F[B]): F[B] = ???
}

Something like this anyway. At least let's make it configurable.

[alexandru]

Note this doesn't fix bracket's and Resource#use's change — cancel is no longer concurrent with use in case of IO.apply or other misbehaving implementations, this being IMO an API leak.

It may be a leak that we can live with, because indeed, in CE 2 you always have to think of concurrent access to shared resources, whereas this API is saner if you structure the use task as a chaining of IO tasks.

But it's only a good choice if timeout() does what it's supposed to do, which is to unstuck the process and indicate problems.

[djspiewak]

But it's only a good choice if timeout() does what it's supposed to do, which is to unstuck the process and indicate problems.

This is my whole point though: is that what it's supposed to do? Because, again, I very much have the expectation (as a user) that timeout also ensures that things are cleaned up after it kills them, and you're suggesting that it shouldn't do that. As I've said, that semantic in and of itself has been the source of major bugs dropping servers in production, so it's gonna be pretty hard to sell me on the idea that it's the expected behavior.

A lot of your examples here are boiling down to IO.delay or IO.blocking with long-running side-effect snippets that just hog the thread forever. We simply can't do anything about those scenarios, particularly if the user hasn't opted-in to interruptibility. How do you expect to get the thread back? Do we just leak it? What should happen to the underlying pool? What about the underlying CPU? How do we discover those problems?

Pervasive backpressure is really the best default. It gives users the most subtle and most complex semantic for free out of the box without them having to do anything. Adjusting things to optimize certain areas for throughput or lossyness is an easy thing to do explicitly given the existence of Queue and Fiber itself. Opt-in backpressure is much more dangerous since it asks users to proactively think about a set of very subtle and unintuitive cases, where the penalty for getting it wrong is losing a server under load.

[alexandru]

A lot of your examples here are boiling down to IO.delay or IO.blocking with long-running side-effect snippets that just hog the thread forever. We simply can't do anything about those scenarios, particularly if the user hasn't opted-in to interruptibility. How do you expect to get the thread back? Do we just leak it? What should happen to the underlying pool? What about the underlying CPU? How do we discover those problems?

Replying late 🙂 as I wanted to digest this more, maybe I'm wrong. But this is basically the ideological divide between static FP folks and the "let it crash" philosophy of dynamic FP, such as Erlang.

If the process leaks resource, I'd argue that it's irrelevant with a supervisor that can detect issues and restart that process when it crashes. A process that gets stuck on back-pressure on the other hand is unable to function, going into a zombie state in which resources are not being used (yay resource safety) and the process is losing money.

To make matters worse, we are relying on developer discipline to make race / timeout work, we have to trust that the task will do the right thing™️ on cancelation, but lack of trust in dealing with async processes (and their non-terminating nature) is why race / timeout exist in the first place.

Imagine working on a payments processor, at a bank. Imagine telling my managers that our process was unable to process transactions, and thus we lost revenue, because some tasks we described were not cancelable, so back-pressure kicked in. Imagine us having to explain that the Dev-Ops were unable to see the problem in time because the process seemed as if it was working as intended, no crashes, no out of place memory usage. You can't even log that a task is faulty without race disregarding any result of the task that lost the race.

Our project right now is on Cats-Effect 2 and I'm afraid of updating it to Cats-Effect 3, because we rely on those timeouts working as intended in every other library handling async cancelable effects in Java's or .NET's or JavaScript's ecosystems, so basically every other library I worked with. To make that transition, at the very least I would have to re-implement race/timeout and hope that's enough, but I have a feeling that it isn't.

You're saying that opt-in back-pressure is the wrong default, with which I agree, sort of, at least for streaming. But I say that Cats-Effect's back-pressure is at this point unable to opt-out of back-pressure and that's a problem. Back-pressure should mean ability to opt-out, and that's definitely not a feature of Cats-Effect.

[djspiewak]

If the process leaks resource, I'd argue that it's irrelevant with a supervisor that can detect issues and restart that process when it crashes. A process that gets stuck on back-pressure on the other hand is unable to function, going into a zombie state in which resources are not being used (yay resource safety) and the process is losing money.

You need a supervisor either way. The problem is that when you're leaking resources, the supervisor itself can degrade when the resources involve kernel-level elements which cannot be isolated. File handles are the classic example of this, but threads also apply to a lesser extent. I have seen, in production, failure scenarios in which a process leaks resources because of lack of back pressure, and it was improperly detected and acted upon by the supervisor because the resources being leaked caused degradation not only of the process but of the virtual and physical host!

Conversely, if your process back pressures, then the supervisor should detect the degradation of metrics and interpret this as a failure to keep up with ongoing load, thus provisioning new tasks. The main difference being that there will always be sufficient resources to make that determination and correctly act upon the information. This "room to act" is a critical property which must always be preserved. The semantics you are advocating for relax this guarantee and lead to situations where degradation cannot be recovered, even with external supervision.

To make matters worse, we are relying on developer discipline to make race / timeout work, we have to trust that the task will do the right thing™️ on cancelation, but lack of trust in dealing with async processes (and their non-terminating nature) is why race / timeout exist in the first place.

I think there's a misunderstanding here. async is cancelable by default (unless you wrap it in uncancelable), which in turn means that you can absolutely rely on timeout behaving the way you expect. It is synchronous, and particularly blocking, effects which are not cancelable by default, because there's no way to do so safely. I would strongly argue that any intuition around timeouts functioning over synchronous effects is simply wrong.

Like, do you really want your server to crash by default in this kind of situation?

val whoops = IO blocking { while (true); }
whoops.timeout(1.millisecond).foreverM

Or would you rather have more aggressive and immediate feedback that something is terribly wrong?

You're saying that opt-in back-pressure is the wrong default, with which I agree, sort of, at least for streaming. But I say that Cats-Effect's back-pressure is at this point unable to opt-out of back-pressure and that's a problem. Back-pressure should mean ability to opt-out, and that's definitely not a feature of Cats-Effect.

Except streaming back pressure is built on top of exactly this back pressure. Without back pressure on cancel, you can't have streaming back pressure in all cases, and that's exactly the situation that CE2 is in.

I would also argue that you're mischaracterizing the present situation. Right now, you can opt-out of back pressure in Cats Effect simply by calling .start on your .cancel. For example, see #2246. In Cats Effect 2, it was effectively impossible to opt-in. Back pressure simply didn't exist in many scenarios and there was no way to really recover that semantic. The defaults today are strictly more expressive, since it's possible to have either semantic (whereas previously it was not possible in all cases), and the default is biased towards safety.

[arosien]

An anecdote from a confused user, summarizing from gitter:

• IO.race(IO.blocking(Thread.sleep(500)), IO.sleep(100) unexpectedly--to the user and, frankly, to me--takes 500+ millis. From reading the above discussion, that's because race and the like wait for the blocking effect to complete because it is uncancelable. Or something like that; I understand the resource argument from the above conversation, but none of this knowledge lies on anything like the main docs or scaladoc at this time.
• AFAICT, here is no documentation about IO.blocking effects being uncancelable.
• It is a bit confusing, naming-wise, that one might naturally use IO.blocking to define an effect that blocks a thread (like the above Thread.sleep), but gets an effect that can't be canceled, but indeed could have been canceled via thread interruption. So users need to be taught to use IO.interruptible in that situation. My concern here is user education; it's not clear what factory method folks should use for various situations, and if I mentally build up a flowchart, it's pretty confusing.

[alexandru]

Extra documentation doesn't help.

The issue is that operations that deal with cancellation (such as race) working as intended relies on developer discipline, which is always a recipe for confusion and failure.

You shouldn't need documentation to have a correct intuition for what IO.race(IO.blocking(Thread.sleep(500)), IO.sleep(100)) does.

[djspiewak]

The issue is that operations that deal with cancellation (such as race) working as intended relies on developer discipline, which is always a recipe for confusion and failure.

I mean, you can make this argument in both directions. I would argue that very few people have a solid and accurate intuition for how cancelation works, and much of why it works that way is simply an essential property of implementation.

Altogether, this really is the classic tradeoff between two different forms of safety: safety from deadlock, and safety from resource leaks. Cats Effect 3 makes very strong guarantees about resource leaks, much stronger than Cats Effect 2, which means that sometimes its guarantees are less strong around deadlock avoidance. I would argue that this is the correct way to bias things though. In Cats Effect 2, we made partial guarantees around resource safety, but those guarantees failed in scenarios which were hard to reason about and even harder to detect. In Cats Effect 3, those guarantees are very strong, and when the implications of those guarantees get you into trouble (such as the above example), it's easier to build tooling to detect the consequences (e.g. fiber traces and dumps).

[alexandru]

fiber traces and dumps

This is an interesting idea. Would help but I'd like to point out that in practice thread-dumps are terrible, looking at thread-dumps often requires human interpretation and skill that few have, and dead-locks or live-locks often can't be detected automatically, even if profilers try hard.

I get your point about safety from deadlock vs safety from resource leaks. It's a good one, and indeed CE2 had issues, that's not under debate. Back-pressuring by default isn't under debate either, that train is gone.

My point is that deadlocks and resource leaks are many times intertwined. Where with a plain resource leak you might leak a file handle, with a deadlock you can leak an entire process. Deadlocks are resource leaks too, pretty bad ones actually.

And back-pressuring needs the ability to opt-out, and currently that ability isn't there. timeoutAndAbandon is a good start. I think more is needed.

[viktorklang]

The rationale for cancel in Reactive Streams not being backpressured is because the signal to cancel typically has urgency—why otherwise send the signal in the first place—you might as well just stop listening for the result. Essentially, cancel is a signal that demand is dropped to 0 forever. Therefor the producer can safely clean up resources.

[alexandru]

Thanks for your input. You are right that at a high level "cancel" in Reactive Streams is about signaling the intent to stop. You may misunderstand the reasoning for back-pressuring. Cancel is also about signaling to the producer that resources should be cleared.

There is no greater example than Rx's switchMap.

stream.switchMap(event => 
  sleep(1.second).flatMapConcat(_ => query(event)))

Imagine that the "query" does database queries, or whatever, and the user doesn't want multiple connections at the same time. This is an actual real-world use-case BTW. If "cancel" is back-pressured, with switchMap actually waiting on the socket to be closed, then the user would get what it expects without anything extra.

Of course, Reactive Streams doesn't do this. Neither does Rx. Which is actually fine, because the use-case can be fixed with a Semaphore / asynchronous lock — I.e. you don't allow access to a new connection until previous connection is released. This is actuallythe classic way to fix it, should the need arise.

The problem is old and has manifested in different ways. Erik Meijer has long argued for example that back-pressuring Rx Observables is a mistake, for example, with similar reasoning. Except that in the case of streams you can always setup buffers that drop events on the floor.

[viktorklang]

Cancel is also about signaling to the producer that resources should be cleared.

That's what I meant by: "Essentially, cancel is a signal that demand is dropped to 0 forever. Therefor the producer can safely clean up resources."

[djspiewak]

I think it's fair to say that I disagree with Erik Meijer on quite a few things, including array covariance. :-)

The problem here stems from the fact that cancelation, in idiomatic Cats Effect applications, is something that happens very frequently, under entirely normal circumstances, and then is often followed by a continuation which performs further operations. If cancelation were quite rare, this wouldn't really matter all that much since there would be limited (or no) possibility of runaway resource leak. That is certainly the case in the reactive streams ecosystem, but not here.

Even outside of resource leaks, it isn't at all difficult to describe common scenarios where backpressuring cancelation produces the desired semantic, and non-backpressured cancelation results in unintuitive or even nondeterministic results. For example, imagine a ResourceApp in which the finalizer flushes buffered log messages to disk. When the application is terminated (e.g. imagine by a container scheduler), the logs should be flushed before the process ends. This should be relatively fast, but if the process ends before the flushing completes, some log messages will be lost nondeterministically.

All of this circles back to a very fundamental idea in the Cats Effect and Fs2 ecosystems: backpressure should always be the default. The reasoning here is relatively simple: resources are finite, therefore we should not design frameworks and applications which assume that resources are infinite. This is equivalent to saying that queues should always be bounded by default, and unbounded queueing should be opt-in rather than opt-out. Every unbounded queue is either an assertion that some external limiter is in place (a detail which most people overlook!) or that your memory is infinitely large (which seems unlikely).

Even aside from the first principles argument, there are several practical scenarios which make default-backpressure an inescapably correct semantic. Load shedding comes to mind as a major one. When everything in an ecosystem preserves backpressure, then graceful load shedding on overloaded services happens effectively for free, without having to take any explicit action. This is an absolutely critical property, since the definition of resilience in any high-scale system is not the ability to successfully drink from a firehose of unbounded size, but rather to automatically self-heal regardless of the flow magnitude. Without load shedding, services will inevitably respond to unexpected traffic volumes by severely degrading, often permanently, and often taking down other systems at the same time.

This brings us around to the Achilles heel of asynchronous applications: backpressure "gaps". The general accepted wisdom around asynchronous programming is that, when it works, it works extremely well, but when it breaks the results are proportionally much more catastrophic. The mere fact that a single process can quite easily accept enormous connection volumes at a time hints at just how catastrophic things can get if you don't balance things correctly. When running in a marathon, tripping in front of the full group will cause a massive chain reaction of problems if those behind you don't react immediately and correctly. The asynchronous analog of this is when some component of an asynchronous system starts slowing down. If you don't have backpressure at every link along the chain back to the producer, the resulting flow will build up at the first point at which unbounded resources were assumed, much like water bursting out of a broken pipe under high pressure.

Thus, removing backpressure from a link in the chain should be a choice that developers make very intentionally, with great care, and with full understanding of the implications. Since cancelation is expected as part of the normal and frequent application operations in any application, it too must exert backpressure as its default mode of operation, otherwise all of the guarantees of the system fall apart.

Anyone who prefers non-backpressuring cancelation can simply append .start, as the timeoutAndAbandon PR does.

[djspiewak]

You covered a lot of conceptual ground. Let's see if I can respond to each in turn…

The ability to do whatever you want with the finalizers is an illusion, due to stuff like bracket being baked in everywhere, like in the evaluation of Resource, which chains finalizers via flatMap.

Finalizers are flatMap (and onError, and onCancel). You've mentioned bracket a few times, but remember that there's nothing magical or illusory about bracket:

  def bracketFull[A, B](acquire: Poll[F] => F[A])(use: A => F[B])(
      release: (A, Outcome[F, E, B]) => F[Unit]): F[B] =
    uncancelable { poll =>
      acquire(poll).flatMap { a =>
        // we need to lazily evaluate `use` so that uncaught exceptions are caught within the effect
        // runtime, otherwise we'll throw here and the error handler will never be registered
        guaranteeCase(poll(unit >> use(a)))(release(a, _))
      }
    }

Technically, Resource doesn't flatMap brackets together anymore. Instead, it uses uncancelable and the various other primitives to achieve a similar effect. I think you're saying that onCancel is the problem and should not be permitted to be asynchronous.

One use-case would be an application's shutdown. Using timeoutAndAbandon in Main is the least we could do.

This would result in resource leaks though. Consider the (very common) case of running an application within some supervised container infrastructure, like Kubernetes. In such environments, when a task is to be shut down, it is sent a SIGTERM followed by a configurable timeout in the supervisor, after which it will send SIGKILL and discard the instance. The SIGTERM is meant to clean up as many resources as possible within that timeout window; there is no need to add an extra timeout which leaves unresolved resources! I actually think that IOApp is the worst example of where timeoutAndAbandon would be useful, since there is always some external supervisor which can and will send KILL.

The ideal would be for the logic in Main to timeout, then for the first finalizer to be attempted then timeout, then for the second finalizer to be attempted and then timeout, and so on and forth. Because each of those layers might attempt doing one last important thing before quitting, and it's unfair to timeout the entire Main instead of just the finalizer that's misbehaving.

Timing out doesn't cancel if you're saying timeout should be timeoutAndAbandon. The finalizer would still be running, and the application would now be in a potentially invalid state since a more-outer finalizer would start running before a more-inner finalizer had completed, potentially meaning that a resource would be closed by the outer finalizer which was needed for the inner one (which is still running!).

My view is that back-pressuring these finalizers have a different meaning than streaming in Fs2, with different effects. And even when describing streaming, you still want the ability to timeout either the producer, or the consumer, depending on who can misbehave.

Except that Fs2 is built on top of Cats Effect. If CE's finalizers don't backpressure, then neither do Fs2's. Everything basically flows upward from the root, and if CE is unable to provide guarantees around backpressure and ordering, then those guarantees cannot be somehow recovered at a higher layer.

TLDR — the problem with asynchronous communication is that the actor you're communicating to can never be trusted.

I think this is where we fundamentally disagree. There's nothing special about asynchronous effects. Really, most synchronous effects are actually asynchronous in some lower-level scheduler (e.g. Thread.sleep interacts with the kernel scheduler to move the current thread off of the processor until the requisite time has elapsed; it is asynchronous from the perspective of the kernel, and synchronous from the perspective of the user space). Extending this intuition upward, all asynchronous effects in user thread space are synchronous effects in user fiber space. There is no particular reason why one effect is more or less trustworthy than another one. And in fact, in Cats Effect's model, they are meant to be treated identically.

If finalizers are not considered critical regions, then all guarantees around resource safety and even finalizer ordering are lost, which can give rise to nondeterministically invalid program states. That seems… quite bad. What's more, it's not a property which can be easily recovered once lost. See also: the difficulties people have on CE2 with exactly these sorts of problems. Opt-in backpressure on finalizers is very surprising and very difficult to work with, whereas CE3 offers opt-out backpressure universally (just call .start), which at least gives the user some amount of control in most circumstances (you still can't control upstream finalizers, but the upstream finalizers can control themselves).

I mean, we like back-pressure because we want to protect slow consumers, and this is like that, except in the other direction — we want to protect against slow producers, as producers can't be trusted either 🙂

I'm not sure I understand this point. Backpressure flows in both directions. And it's worth reiterating here that cancelation still exists in Cats Effect 3, we just have much tighter guarantees around sequentialization of cancelation and finalization. The model does move more in a cooperative direction, but in doing so it gains much higher degrees of safety without sacrificing expressiveness.

[alexandru]

@djspiewak I got your reply, I disagree with a lot of it 🙂 however I also think that I'm doing a poor job communicating why. Plus there's always the possibility that I'm wrong and you're right.

I'm on vacation now, and when I'm back, if you'll indulge me, I think we should go over the points of disagreement, but individually, and maybe reach a consensus point by point, as indeed we covered a lot of conceptual ground, and really good arguments may get lost.

So I'll digest this and then come back 🙂

[SystemFw]

and the application would now be in a potentially invalid state since a more-outer finalizer would start running before a more-inner finalizer had completed, potentially meaning that a resource would be closed by the outer finalizer which was needed for the inner one (which is still running!).

You already know my general opinion from my previous replies , but I just want to point out that the reason we introduced backpressure on finalisers in CE2 was due to exactly this problem/bug in IOApp (I seem to remember it was @rossabaker who initially stumbled on it). The difference is that in CE2 the model accrued ad hoc incremental changes, whereas in CE3 - we baked backpressure guarantees from the ground up, given that you can opt out of them via start as previously said

[alexandru]

On this point, one thing I disagree with is the opt-out nature of finalizers. I don't agree that you can opt-out. To make it clear, this is mostly true...

task.guarantee(f).guarantee(g) <-> task.guarantee(f *> g)

The call-site cannot transform that behavior into this after the fact:

task.guarantee(f.start *> g.start)

The opportunity is gone, that sequencing is now hard-coded at the call-site. This is what I meant by finalizers being flatMapped.

In terms of UX this seems fine until you realize that you have to trust the implementation of an asynchronous task that may suffer from various issues that can lead to non-termination.

Also, I wouldn't say that forcefully closing a resource that another task depends on is a bug. In some situations it is inevitable.

task
  .guarantee {
    IO.raiseError(???)
  }
  .guarantee {
    IO(file.close())
  }

How should that second guarantee behave?

1. If it's not executed, then this is an obvious resource leak caused by back-pressuring;
2. If it is executed, well, tasks yielding exceptions are non-terminating, so then what makes this different from other non-termination cases?

What I mean by (2) is that the thrown exception can mean anything, including that the background task is still running, therefore closing that file may be incorrect according to this philosophy.

[djspiewak]

The opportunity is gone, that sequencing is now hard-coded at the call-site. This is what I meant by finalizers being flatMapped.

This is a fair point, but any other semantic would mean that guarantees about resource safety are at the whim of downstream callers. Laws would have no meaning, and guarantee would not in any way be "guaranteed".

In terms of UX this seems fine until you realize that you have to trust the implementation of an asynchronous task that may suffer from various issues that can lead to non-termination.

Synchronous tasks have the same problems though. This isn't an issue which is unique to asynchrony, it's just that when the task is asynchronous it 1) won't starve the thread pool, and 2) won't show up in a thread dump (but will show up in a fiber dump).

How should that second guarantee behave?

Try it and see. :-) This is actually defined by law.

In particular, guarantee means what it says on the tin: it is guaranteed unless the upstream does not terminate. And by "does not terminate" I mean never or delay { while (true); }. raiseError is not non-termination, rather it is terminating with an error. Similarly, cancelation is not non-termination.

If the calculus of Applicative is that there is a value of type A inside the F, then the calculus of ApplicativeError implies a value of type Either[E, A] and the calculus of MonadCancel implies a value of type Option[Either[E, A]]. All of these are termination cases, just with very different semantics, and none of them are the same as something like never or an infinite loop.

All of which is to say that the file in your example will be closed.