How to let the driver stop when there are not enough GPUs?

[zhenyisx]

Hi Team,

I understand that in TFNode.py, start_cluster_server() is responsible to set up GPU devices. If no free GPU is available, it will keep trying for ever until it get free GPU.

Is there a way to let the spark driver know there is no free GPU available on one node and stop the whole job?

Thanks a lot.

Best,
Yi

[leewyang]

@zhenyisx yeah, it retries forever on that node due to this loop:
https://github.com/yahoo/TensorFlowOnSpark/blob/master/tensorflowonspark/TFNode.py#L52-L78

Note that we are somewhat following the notion of YARN resource allocation (i.e. wait until the resources become available)... and also that we are restricted to this machine because of the TF cluster_spec reservation scheme.

Theoretically, you can remove this loop along with the try/except, s.t. the exception is surfaced to Spark. This, in turn, would signal a task failure, which would eventually end up with a job failure.

Lee

[zhenyisx]

Thanks a lot for quick reply, @leewyang .

Actually I have tried you suggestions. It seems that the function start_cluster_server() (https://github.com/yahoo/TensorFlowOnSpark/blob/master/tensorflowonspark/TFNode.py#L38-L90) will run within a background process (https://github.com/yahoo/TensorFlowOnSpark/blob/master/tensorflowonspark/TFSparkNode.py#L403-L405) and won't lead to a job failure. The driver seems don't know what happened and the whole job will keep running with some executors get stuck.

Would you have any comments or suggestions? Thank you very much.

Best,
Yi

[leewyang]

The only other thing I can think of is to just exit that background process, s.t. the data feeding tasks end up failing (which would eventually fail the job).

[zhenyisx]

Can you please advise how to do it with more details? My understanding is that we should not wait for the background process so that we can move on to feed the data. If we could not wait, how can we exit that background process?

Thank you very much.

[leewyang]

I was just imagining replacing this code with a sys.exit, but I'm not absolutely sure if that's enough to raise errors in spark during data feeding.

[zhenyisx]

Hi @leewyang, I tried your approach but it's not enough to raise errors in spark.

It seems to me that sparkcontext has statustracker for jobs and stages, but I am not sure that could be be used to capture the exceptions of background processes here.

[leewyang]

@Eord added a timeout to the driver side reservation loop in #246 which should fail the job if the nodes fail to acquire GPUs and/or fail to respond after a certain amount of time.