Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
38 commits
Select commit Hold shift + click to select a range
740a50e
initial support for tf2
Jul 26, 2019
4f02303
add missing line
Aug 2, 2019
1e04b59
add checkpointing to mnist_keras_2.0.py
Aug 2, 2019
539b0c3
add checkpoints and saved_models
Aug 5, 2019
981e426
remove outdated examples; update mnist_data_setup.py and mnist_infere…
Aug 7, 2019
fec8c07
remove low-level API examples
Aug 7, 2019
c8b47fa
add InputMode.SPARK version of keras
Aug 8, 2019
383bc65
update mnist keras readme
Aug 8, 2019
9ff6a8d
add support for InputMode.SPARK; add TFNodeContext.num_workers
Aug 14, 2019
d15fc4c
remove more low-level API examples
Aug 14, 2019
3d1d3e4
remove low-level API notebooks
Aug 14, 2019
123be55
add parallel inferencing example for estimator
Aug 14, 2019
3b02cbe
mnist streaming w/ estimator; add examples/utils
Aug 15, 2019
497feb2
use examples/utils in keras
Aug 15, 2019
d57c3c0
move model_export.py into examples/util
Aug 15, 2019
e3caa24
remove InputMode.TENSORFLOW and checkpoint inferencing from ML pipeli…
Aug 16, 2019
b3c9b90
remove input_mode param
Aug 19, 2019
9c096be
remove references to InputMode param
Aug 19, 2019
9049b12
add image segmentation example
Aug 26, 2019
1f86976
remove cifar10 and imagenet examples
Aug 27, 2019
b6d454e
initial code from tensorflow/models/official
Aug 27, 2019
036a5c5
add resnet/cifar example
Aug 29, 2019
dcb30c0
remove more TF1.x examples
Aug 30, 2019
3a3d1e2
update versions; remove more outdated examples
Sep 6, 2019
3a32a3e
remove outdated examples/mnist/README.md
Sep 6, 2019
d04f6cb
spark 2.4.4; use sphinx_rtd_theme
Sep 6, 2019
688a237
fix h5py version; use TF2.0.0rc0
Sep 6, 2019
3fd3676
go back to 2.0.0b1
Sep 6, 2019
fea0d58
fix compatibility issues
Sep 16, 2019
14d4aa9
minor edits
Sep 16, 2019
5e6a69b
disable Java build; update README.md
Sep 30, 2019
1239669
Merge branch 'master' into experimental
leewyang Sep 30, 2019
2941b02
try xenial env
Oct 1, 2019
2dbc0be
update pip version
Oct 1, 2019
36b6015
adjust requirements.txt dependencies
Oct 1, 2019
7a9053e
move travis spark setup to script
Oct 1, 2019
6968b3c
more travis setup
Oct 1, 2019
cb0a169
fix test
Oct 1, 2019
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
33 changes: 4 additions & 29 deletions .travis.yml
Original file line number Diff line number Diff line change
Expand Up @@ -2,42 +2,25 @@ matrix:
include:
- language: python
python: 2.7
dist: trusty
dist: xenial
before_install:
- curl -LO http://www-us.apache.org/dist/spark/spark-2.4.4/spark-2.4.4-bin-hadoop2.7.tgz
- export SPARK_HOME=./spark
- mkdir $SPARK_HOME
- tar -xf spark-2.4.4-bin-hadoop2.7.tgz -C $SPARK_HOME --strip-components=1
- export PATH=$SPARK_HOME/bin:$PATH
- export SPARK_LOCAL_IP=127.0.0.1
- export SPARK_CLASSPATH=./lib/tensorflow-hadoop-1.0-SNAPSHOT.jar
- export PYTHONPATH=$(pwd)
- source scripts/travis_before_install.sh
install:
- pip install -r requirements.txt
- pip install mock
script:
- test/run_tests.sh
- language: python
python: 3.6
dist: trusty
dist: xenial
before_install:
- curl -LO http://www-us.apache.org/dist/spark/spark-2.4.4/spark-2.4.4-bin-hadoop2.7.tgz
- export SPARK_HOME=./spark
- mkdir $SPARK_HOME
- tar -xf spark-2.4.4-bin-hadoop2.7.tgz -C $SPARK_HOME --strip-components=1
- export PATH=$SPARK_HOME/bin:$PATH
- export SPARK_LOCAL_IP=127.0.0.1
- export SPARK_CLASSPATH=./lib/tensorflow-hadoop-1.0-SNAPSHOT.jar
- export PYTHONPATH=$(pwd)
- source scripts/travis_before_install.sh
install:
- pip install -r requirements.txt
- pip list
script:
- sphinx-build -b html docs/source docs/build/html
- test/run_tests.sh
- language: java
dist: trusty
jdk: oraclejdk8
notifications:
email: false
deploy:
Expand All @@ -60,11 +43,3 @@ deploy:
python: 3.6
tags: true
condition: "$TRAVIS_TAG =~ ^v.*$"
- provider: script
script: mvn deploy -DskipTests --settings .travis.settings.xml
skip_cleanup: true
on:
branch: master
jdk: oraclejdk8
tags: true
condition: "$TRAVIS_TAG =~ ^scala_.*$"
19 changes: 12 additions & 7 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
<!--
Copyright 2017 Yahoo Inc.
Copyright 2019 Yahoo Inc.
Licensed under the terms of the Apache 2.0 license.
Please see LICENSE file in the project root for terms.
-->
Expand Down Expand Up @@ -41,18 +41,22 @@ deep learning on our Hadoop clusters in Yahoo's private cloud.
TensorFlowOnSpark provides some important benefits (see [our
blog](http://yahoohadoop.tumblr.com/post/157196317141/open-sourcing-tensorflowonspark-distributed-deep))
over alternative deep learning solutions.
* Easily migrate existing TensorFlow programs with <10 lines of code change
* Support all TensorFlow functionalities: synchronous/asynchronous training, model/data parallelism, inferencing and TensorBoard
* Server-to-server direct communication achieves faster learning when available
* Allow datasets on HDFS and other sources pushed by Spark or pulled by TensorFlow
* Easily integrate with your existing Spark data processing pipelines
* Easily migrate existing TensorFlow programs with <10 lines of code change.
* Support all TensorFlow functionalities: synchronous/asynchronous training, model/data parallelism, inferencing and TensorBoard.
* Server-to-server direct communication achieves faster learning when available.
* Allow datasets on HDFS and other sources pushed by Spark or pulled by TensorFlow.
* Easily integrate with your existing Spark data processing pipelines.
* Easily deployed on cloud or on-premise and on CPUs or GPUs.

## Install

TensorFlowOnSpark is provided as a pip package, which can be installed on single machines via:
```
# for tensorflow>=2.0.0
pip install tensorflowonspark

# for tensorflow<2.0.0
pip install tensorflowonspark==1.4.4
```

For distributed clusters, please see our [wiki site](../../wiki) for detailed documentation for specific environments, such as our getting started guides for [single-node Spark Standalone](https://github.com/yahoo/TensorFlowOnSpark/wiki/GetStarted_Standalone), [YARN clusters](../../wiki/GetStarted_YARN) and [AWS EC2](../../wiki/GetStarted_EC2). Note: the Windows operating system is not currently supported due to [this issue](https://github.com/yahoo/TensorFlowOnSpark/issues/36).
Expand All @@ -61,6 +65,8 @@ For distributed clusters, please see our [wiki site](../../wiki) for detailed do

To use TensorFlowOnSpark with an existing TensorFlow application, you can follow our [Conversion Guide](../../wiki/Conversion-Guide) to describe the required changes. Additionally, our [wiki site](../../wiki) has pointers to some presentations which provide an overview of the platform.

**Note: since TensorFlow 2.x breaks API compatibility with TensorFlow 1.x, the examples have been updated accordingly. If you are using TensorFlow 1.x, you will need to checkout the `v1.4.4` tag for compatible examples and instructions.**

## API

[API Documentation](https://yahoo.github.io/TensorFlowOnSpark/) is automatically generated from the code.
Expand All @@ -71,7 +77,6 @@ Please join the [TensorFlowOnSpark user group](https://groups.google.com/forum/#

Contributions are always welcome. For more information, please see our [guide for getting involved](Contributing.md).


## License

The use and distribution terms for this software are covered by the Apache 2.0 license.
Expand Down
4 changes: 2 additions & 2 deletions docs/source/conf.py
Original file line number Diff line number Diff line change
Expand Up @@ -28,9 +28,9 @@
author = 'Yahoo Inc'

# The short X.Y version
version = '1.4.4'
version = '2.0.0'
# The full version, including alpha/beta/rc tags
release = '1.4.4'
release = '2.0.0'


# -- General configuration ---------------------------------------------------
Expand Down
87 changes: 0 additions & 87 deletions examples/cifar10/BUILD

This file was deleted.

69 changes: 0 additions & 69 deletions examples/cifar10/README.md

This file was deleted.

10 changes: 0 additions & 10 deletions examples/cifar10/README_orig.md

This file was deleted.

22 changes: 0 additions & 22 deletions examples/cifar10/__init__.py

This file was deleted.

Loading