How to handle exceptions when using SPARK mode which run tensorflow in backgroud process

[spencerlisp]

some exceptions in tensorflow function will make backgroud process exit, but the spark thread does not know the situation and will be blocked in the queue operations.

[leewyang]

Unfortunately, that's a known issue with the SPARK mode, which we've highlighted in our talks. I've been looking for a solution, but I don't have a good answer yet (and it may require significant changes). Generally, once your TF code has been stabilized, this should be a rare occurrence... but definitely while debugging, it just looks like a hang.

If you really need to handle exceptions from the TF side, you may want to consider the TENSORFLOW mode, which will surface those exceptions as Spark task failures.

[leewyang]

This should be fixed in #246