Allow running PS nodes on spark driver

[winston-zillow]

My understanding is that the PS nodes are started on one of the spark executors. This could potentially waste the GPU occupied by the PS node executor. It would be nice if one can start the PS node on the driver (in client mode.)

I have submitted a PR #183

[leewyang]

Sorry, I've been in a conference this past week...

For Hadoop/YARN installations, client mode means that the driver is running on the Hadoop gateways, which are shared across all grid users. For this reason, we generally don't recommend running heavy processes in client mode within our clusters, since it only takes a few "abusers" to effectively deny service to others. Also, in cluster mode, it's fairly simple to request more than one PS node (albeit on GPU boxes at the moment), whereas this seems to just run multiple PS nodes as threads. And, finally, have you seen any issues with running the GPU version of TF (required by executors) on a client/gateway w/o CUDA drivers?

[winston-zillow]

It could be abused but one with access to the hadoop master can write any codes to abuse it. That's why this feature should be controlled by an option and is off by default. I think the PS nodes are actually not heavy weight process? (though I couldn't find much details about exactly what it will do and its performance characteristics.)

In environment such as AWS EMR, I can configure the master as a regular non-GPU instance and the slave nodes are GPU instances. I tried to configure more executors than the number of slaves but they will then waiting for GPU forever.

(Not sure what you mean by "have you seen any issues with running the GPU version of TF (required by executors) on a client/gateway w/o CUDA drivers?" I think TF would fall back to CPU automatically? But in my above config, I would configure the master as a non-GPU host.)

[leewyang]

I guess the difference is that in cluster mode, if you abuse the master/driver, only your container is affected, whereas in client mode, if you abuse the driver/gateway, then other hadoop users can be affected.

As for the last bit, for TF, you generally need to install either the CPU version or the GPU version (compiled against CUDA) for your env/hardware. So, for instance, if you run the GPU version of TF on a non-GPU machine, you usually get errors like libcuda.so or libcudnn.so is missing. So, I'm curious if this is an issue for the PS node in this setup. (It's entirely possible that it's not an issue, since I haven't tried this out yet).

[leewyang]

Moving this discussion to the actual PR