Skip to content
master
Switch branches/tags
Code

Latest commit

 

Git stats

Files

Permalink
Failed to load latest commit information.
Type
Name
Latest commit message
Commit time
 
 
 
 
 
 

Distributed TensorFlow Example (O'Reilly Blog Post)

There are three pieces of example code here:

  • a Spark program to launch TensorFlow functions inside a Spark 'mapper' function. This is useful for hyperparameter optimization (parameter sweeps)

  • a TensorFlowOnSpark program that can be used with Hops Hadoop

  • a Horovod program that can be used with Hops Hadoop

Hyperparameter Optimization using Spark/TensorFlow

This has been tested on a Ubuntu 16.04 machine, but should work on other platforms. We recommend that you do the following inside a new conda environment, but you may also run it without a virtual environment.

# Download and install standalone Spark
wget http://apache.mirrors.spacedump.net/spark/spark-2.2.1/spark-2.2.1-bin-hadoop2.7.tgz
tar zxf spark-2.2.1-bin-hadoop2.7.tgz
export SPARK_HOME=$(pwd)/spark-2.2.1-bin-hadoop2.7
export PATH=$SPARK_HOME/bin:$PATH
pip install pyspark
pip install tensorflow==1.4.0
pip install jupyter
git clone https://github.com/hopshadoop/distributed-tf.git
cd distributed-tf

export PYSPARK_DRIVER_PYTHON=jupyter
export PYSPARK_DRIVER_PYTHON_OPTS='notebook'
pyspark
#The 'pyspark' command should start a Jupyter Notebook in your web browser. 
#You can then create a new 'pyspark' notebook and test it using this code

TensorFlowOnSpark

You can get more details on TensorFlowOnSpark at its github repo.

Horovod

More details on Horovod at its github repo.

Distributed TensorFlow on Hops

For details on how to install and setup Hops, so that you can run all three programs in a Tour, go to hops.io.

About

Distributed TensorFlow Examples for O'Reilly

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published