You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Describe the bug:
When trying to run MNIST example, I get the error : "Exception in TF background thread" on TFCluster run.
Logs:
from notebook :
2021-06-14 16:44:45,996 INFO (MainThread-22437) Reserving TFSparkNodes w/ TensorBoard
2021-06-14 16:44:45,998 INFO (MainThread-22437) cluster_template: {'ps': range(0, 1), 'worker': range(1, 10)}
2021-06-14 16:44:46,002 INFO (MainThread-22437) listening for reservations at ('XXX.XXX.XXX.XXX', XXXXX)
2021-06-14 16:44:46,003 INFO (MainThread-22437) Starting TensorFlow on executors
2021-06-14 16:44:46,180 INFO (MainThread-22437) Waiting for TFSparkNodes to start
2021-06-14 16:44:46,190 INFO (MainThread-22437) waiting for 10 reservations
2021-06-14 16:44:47,192 INFO (MainThread-22437) waiting for 10 reservations
2021-06-14 16:44:47,290 ERROR (Thread-6-22437) Exception in TF background thread
2021-06-14 16:44:48,195 INFO (MainThread-22437) waiting for 10 reservations
from yarn log :
[...]
21/06/14 16:26:50 INFO YarnAllocator: Received 10 containers from YARN, launching executors on 0 of them.
Did I miss something in Spark/Hadoop/Tensorflow versions ?
I can't get more information of this error in the logs, does it mean that TensorflowOnSpark can't "connect" to Spark ?
The text was updated successfully, but these errors were encountered:
Are you able to run standard Spark jobs on this cluster with those same settings? The Driver requested a total number of 0 executors error seems to be occurring at the Spark level and not the TFoS level.
Please take a look at the Spark executor logs for any exceptions, since you're only showing the driver logs, which aren't showing the root cause of the exception.
Finally, you are using a very old version of tensorflow (1.9.0) and you should be using at least tensorflowonspark==1.4.4 (the last version of TFoS intended for TF 1.x branch). That said, If you are starting from scratch, I'd recommend using TF2.x along with the latest tensorflowonspark.
Environment:
Describe the bug:
When trying to run MNIST example, I get the error : "Exception in TF background thread" on TFCluster run.
Logs:
from notebook :
2021-06-14 16:44:45,996 INFO (MainThread-22437) Reserving TFSparkNodes w/ TensorBoard
2021-06-14 16:44:45,998 INFO (MainThread-22437) cluster_template: {'ps': range(0, 1), 'worker': range(1, 10)}
2021-06-14 16:44:46,002 INFO (MainThread-22437) listening for reservations at ('XXX.XXX.XXX.XXX', XXXXX)
2021-06-14 16:44:46,003 INFO (MainThread-22437) Starting TensorFlow on executors
2021-06-14 16:44:46,180 INFO (MainThread-22437) Waiting for TFSparkNodes to start
2021-06-14 16:44:46,190 INFO (MainThread-22437) waiting for 10 reservations
2021-06-14 16:44:47,192 INFO (MainThread-22437) waiting for 10 reservations
2021-06-14 16:44:47,290 ERROR (Thread-6-22437) Exception in TF background thread
2021-06-14 16:44:48,195 INFO (MainThread-22437) waiting for 10 reservations
from yarn log :
[...]
21/06/14 16:26:50 INFO YarnAllocator: Received 10 containers from YARN, launching executors on 0 of them.
Then when I run :
I get :
21/06/14 16:27:24 INFO YarnAllocator: Driver requested a total number of 0 executor(s).
Spark Submit Command Line:
Ran from notebook with such configuration :
Did I miss something in Spark/Hadoop/Tensorflow versions ?
I can't get more information of this error in the logs, does it mean that TensorflowOnSpark can't "connect" to Spark ?
The text was updated successfully, but these errors were encountered: