Threadpool and panics

[PvdBerg1998]

Currently, if a task panics, the threadpool thread dies and is never respawned. After enough panics this means the entire runtime is dead.

This raises the question: should panics get caught? Or should the runtime at least respawn the dead threads?

[Nemo157]

I remember looking into how tokio handles this at some point. If I recall correctly it will also just let the panic kill the thread, but because it dynamically switches threads in and out of a backup pool I think it would eventually bring in another thread to replace it.

[Nemo157]

Looking into this a little more it's not possible to respawn the dead threads with the current guarantee that after_start will be dropped after the thread pool is created.

I've thrown together a patch to catch panics and pass them to a configured hook, but I'm not certain that's the best solution.

Another alternative is that if any future panics that should take down the whole thread pool. If it had a join method then that would be a good place to route the panic to.

[Nemo157]

Another way to get the same result as my above linked patch is to add a generic combinator on Spawn that will catch panics from any spawned futures. That plus the change I mention where any panic that reaches one of the thread pool threads kills the whole thread pool seems like a decent solution to me.

[vavrusa]

I've thrown together a patch to catch panics and pass them to a configured hook, but I'm not certain that's the best solution.

I just came across this. I would support the approach here as it's much more graceful degradation to unblock the thread pool than have a running unresponsive service.

[cramertj]

I think the correct behavior here is to push the panic back up to the caller running the top-level future.

[xermicus]

FYI I came across this issue too and made a tiny crate that works for me (disclaimer: I have no idea whether this is a solid approach to the problem though).

[Nemo157]

I remember looking into how tokio handles this at some point. If I recall correctly it will also just let the panic kill the thread, but because it dynamically switches threads in and out of a backup pool I think it would eventually bring in another thread to replace it.

[Nemo157]

Looking into this a little more it's not possible to respawn the dead threads with the current guarantee that after_start will be dropped after the thread pool is created.

I've thrown together a patch to catch panics and pass them to a configured hook, but I'm not certain that's the best solution.

Another alternative is that if any future panics that should take down the whole thread pool. If it had a join method then that would be a good place to route the panic to.

[Nemo157]

Another way to get the same result as my above linked patch is to add a generic combinator on Spawn that will catch panics from any spawned futures. That plus the change I mention where any panic that reaches one of the thread pool threads kills the whole thread pool seems like a decent solution to me.

[vavrusa]

I've thrown together a patch to catch panics and pass them to a configured hook, but I'm not certain that's the best solution.

I just came across this. I would support the approach here as it's much more graceful degradation to unblock the thread pool than have a running unresponsive service.

[cramertj]

I think the correct behavior here is to push the panic back up to the caller running the top-level future.

[xermicus]

FYI I came across this issue too and made a tiny crate that works for me (disclaimer: I have no idea whether this is a solid approach to the problem though).