yarntf
simplifies the distributed TensorFlow programming model, for running
machine learning applications on Hadoop YARN clusters.
In general it is as simple as follows.
- In your code: replace
tf.train.ClusterSpec()
andtf.train.Server()
withyarntf.createClusterServer()
- On your cluster: submit the application with hops-tensorflow
Your ClusterSpec is generated automaticaly and the parameter servers stopped when all workers are completed. Specify the number of worker, ps and resources on submit.
For more details see the examples.
Development is still in an early stage. Contributions are very welcome!
yarntf
and hops-tensorflow
is released under an Apache 2.0 license.