StandardErrors are swallowed

[kwerle]

I'm adding async and I love it so far - with one exception: exceptions. Well, those that are StandardErrors, anyway. Async swallows them.

task.rb:314:

rescue StandardError => error
  failed!(error, false)

That false says to not re-raise it. So the task exits and says there "might have been" an exception and gives the backtrace - all of which is helpful - but not enough. Ideally I would like it to treat StandardErrors the same way as Exceptions. At the least I want some way to get at the swallowed exceptions so I can re-raise them myself.

I'm happy to code a solution - including passing a parameter to Async do -- Async(error_handling: :reraise) do, or something like that.

[trevorturk]

This is definitely easy to get tripped up on, and it's one of the more confusing aspects of Async, IMO, but I think it all makes sense eventually!

There's a bit of documentation around exceptions worth reading, you can check this out to get started: https://socketry.github.io/async/guides/asynchronous-tasks/index.html#exception-handling

Let me know what you think, and if you have more questions I can try to help, or I'm sure someone else can chime in as well!

[kwerle]

Super helpful - thanks. I was doing something wrong, and I'm still not super happy with what I have to do.

My mistake:

Async do
  Fiber.schedule do
    ...
  end
  Fiber.schedule do
    ...
  end
end

Because I was using Fiber.schedule directly I had no access to the exceptions. That's my mistake and it happened because I was migrating to async and failed to convert the schedule calls.

What I have to do:

tasks = []
Async do
  tasks << Async do
    ...
  end
  tasks << Async do
    ...
  end
end
tasks.each(&:wait)

This is sufficient. But I would still prefer something more streamlined:

Async(raise_errors: true) do
  Async do
    ...
  end
  Async do
    ...
  end
end

Or perhaps (if the outer can keep track of the internal fibers)

Async do
  Async do
    ...
  end
  Async do
    ...
  end
end.reraise_errors

Having to gather and call wait to re-raise isn't at all intuitive. If I were coming to the code as a new user, I would probably assume the wait was needed to get the fibers to execute - not that it was being used to handle errors!

[trevorturk]

I use a barrier for this, since it collects the tasks in a "bucket" I can call wait on, but, yeah, dealing with exceptions is a bit more verbose. I suppose I don't think about it much anymore, once I got accustomed to it, tbh.

[ioquatix]

A task may outlive it's parent so the parent can't naturally be a scope for exception handling.

I think ideally, tasks don't raise exceptions as a matter of normal behaviour ("exceptional" cases).

Either you are waiting on a task to complete (fan out) or you are "fire and forget". You need to think about what situation you are in and what is the appropriate error handling for your situation.

As @trevorturk suggested, using an Async::Barrier is a good option for fan-out. https://socketry.github.io/async/source/Async/Barrier/index.html

[kwerle]

It is such an obvious use case. I should think that passing a function to deal with exceptions would be trivial.

Async(exception_handler: -> { |exception, fiber| whatever I want}) do

[ioquatix]

If you want something like this, it's fairly trivial to achieve it without any library changes:

def AsyncWithExceptionHandler(exception_handler)
  Async do |task|
    yield task
  rescue => exception
    handler.call(task, exception)
  end
end

[emiltin]

A task may outlive it's parent so the parent can't naturally be a scope for exception handling.

The question is who should handle the exception?

Elixir/Erlang has the concept of 'let it fail'; Instead of trying to handle all errors in a task, you're encouraged to just let if fail. Tasks can be organised in supervisor trees. If a task fails, a supervisor will restart that task (and others if needed) from a known good state.

Requiring each task to handle all StandardErrors means you have to add similar code to every task. If you want tasks higher up in the hierarchy help out when a task fails, it does not seem straightforward to me how to implement that well.

[emiltin]

Generally not, although the Erlang philosophy seems a bit different, in that a lot of errors are not handled explicitly. Rather the task is simply restarted if something is wrong. But in any case, unexpected errors can always occur.

[ioquatix]

How does Erlang handle restarting tasks that are not idempotent? It seems like you'd have to be pretty careful about that.

[emiltin]

I think the initialization of a process must be idempotent. See e.g.
https://rocket-science.ru/hacking/2018/12/29/idempotent-supervision-tree

[ioquatix]

We have support for this, but it's a little more flat - in async-container. It will keep your containers alive and provides mechanisms to do updates blue/green deploys/etc. The container code itself can be whatever but itself must be defined, i.e. "the initialization of a process must be idempotent" is also true in a sense for async-container.

[emiltin]

Is async-containersuitable for supervisor-tree style task (fiber) management, or is is more for multi-threading?

[emiltin]

tasks.each(&:wait)

This is essentially how Barrier is implemented. But note that errors raised in a task will not be re-raised until all previous tasks complete. E.g. if the first task never completes, errors in other tasks will never be re-raised. See #272.

[emiltin]

Related to the swallowing of errors, is the fact that unhandled errors are logged (printed) only if nobody is waiting for the task. But the task starts running as soon as you create it, and you cannot wait for it until you've created it. So whether the warning is printed can seem arbitrary, because it depends on whether the task yields so that the parent task.wait is reached before the task raises an error.

In this example, a warning is printed, even though we try to wait for the task:

Async do
  task = Async do
    puts 42**43 % 44
    raise RuntimeError.new('ups')
  end
  task.wait
rescue StandardError
  # handle task error
end

If the task does anyting that causes it to yield, no warning is printed:

Async do
  task = Async do
    sleep 0.0000001
    raise RuntimeError.new('ups')
  end
  task.wait
rescue StandardError
  # handle task error
end