Skip to content
Permalink
Browse files

more exercise content, minor script changes (#23)

* added .gitignore.
* more exercise content; add eval metrics to the xfer learning code
* mnist_cnn readme mods
  • Loading branch information
amygdala committed Oct 30, 2016
1 parent f1827de commit e2aeb64e4823640cfc0d17d80f8c2e031e79d359
@@ -0,0 +1,12 @@
# VIM tmp files
*.swp

*.pyc

.tox

*.key.json

.ipynb*
*.bak
*~
@@ -1,24 +1,64 @@

# Creating a Custom MNIST estimator

The [`mnist_cnn.py`](mnist_cnn.py) script follows the [Deep MNIST for Experts](https://www.tensorflow.org/versions/r0.11/tutorials/mnist/pros/index.html#deep-mnist-for-experts) tutorial on the tensorflow.org site.
(Its hidden layers use *convolutions*.)
In a previous lab, we used `tf.contrib.tflearn.DNNClassifier` to easily build our MNIST model.
However, it turns out that a model that uses *convolutions* performs better on this task. There is currently no "pre-baked" class that we can use (instead of `DNNClassifier`) for the CNN model.

The [`mnist_cnn_estimator.py`](mnist_cnn_estimator.py) script shows how you can create a custom [Estimator](https://www.tensorflow.org/versions/r0.11/api_docs/python/contrib.learn.html#Estimator) based on that same model graph, using TensorFlow's high-level `contrib.tflearn` API.
So instead, we can build a *custom* [Estimator](https://www.tensorflow.org/versions/r0.11/api_docs/python/contrib.learn.html#Estimator). We'll still need to define the model specifics, but we can leverage the Estimator's support for running training loops, model evaluations, and model predictions; as well as model checkpointing during training and generating info for TensorBoard.

This lab has two stages. In the first stage, we'll see how we can take a pre-existing model graph definition and essentially plop it into an Estimator, making our new version much simpler.

In the second stage of the lab, we'll look at how we can use [layers](https://www.tensorflow.org/versions/r0.11/api_docs/python/contrib.layers.html#layers-contrib) to make our model specification more concise and easier to build and understand.

### A CNN version of MNIST using TensorFlow's "low-level" API

The [`mnist_cnn.py`](mnist_cnn.py) script follows the [Deep MNIST for Experts](https://www.tensorflow.org/versions/r0.11/tutorials/mnist/pros/index.html#deep-mnist-for-experts) tutorial on the [tensorflow.org](http://tensorflow.org) site.

We can use this code, and its model, as a starting point for the Estimator(s) that we're going to build.

### MNIST-CNN using a Custom Estimator

The [`mnist_cnn_estimator.py`](mnist_cnn_estimator.py) script shows how you can create a custom [Estimator](https://www.tensorflow.org/versions/r0.11/api_docs/python/contrib.learn.html#Estimator) based on that same model graph, using TensorFlow's high-level `contrib.tflearn` API.

As with the "canned" Estimators, like the [DNNClassifier](https://www.tensorflow.org/versions/r0.11/api_docs/python/contrib.learn.html#DNNClassifier) used in one of the [other MNIST labs](../02_README_mnist_tflearn.md), the `Estimator`provides support for the `fit()`, `evaluate()`, and `predict()` functions so that you don't have to write a training loop, manage model checkpointing, etc., yourself.

We'll first walk through this version-- which essentially drops the original model graph into an `Estimator`-- then as an exercise, make it even simpler by using *layers*.
Run it like this:

```sh
$ python mnist_cnn_estimator.py
```

## Exercise
You can take a look at its output in TensorBoard like this, replacing `<your-model-dir>` with the timestamped directory generated by the script.

```sh
$ tensorboard --logdir=/tmp/tfmodels/mnist_estimator/<your-model-dir>
```

For fun, take a look at the model graph in TensorBoard.

<a href="https://storage.googleapis.com/oscon-tf-workshop-materials/images/mnist_cnn_estim_graph.png" target="_blank"><img src="https://storage.googleapis.com/oscon-tf-workshop-materials/images/mnist_cnn_estim_graph.png" width="500"/></a>
We'll first walk through this version of the code— which essentially drops the original model graph into an `Estimator`— then as an exercise, make the code even simpler by using *layers*.
## Exercise: convert the "low-level API" Estimator model to one that uses *layers*
Start with [`mnist_cnn_estimator.py`](mnist_cnn_estimator.py), which shows how you can wrap a model
graph in an `Estimator`.
Modify this script to replace the model definition code-- the code in the `model_fn` function-- with simpler code that defines the model graph using the [tf.contrib.layers](https://www.tensorflow.org/versions/r0.11/api_docs/python/contrib.layers.html#layers-contrib) library.
When you're done, try comparing the output of both training runs-- that of `mnist_cnn_estimator.py` and your new layers version-- in Tensorboard.
If you get stuck, you can take a look at `mnist_cnn_estim_layers.py`, but try not to look too soon.
#### Compare training runs
When you're done, try comparing the output of both training runs— that of `mnist_cnn_estimator.py` and your new layers version— in TensorBoard. You can do this by pointing TensorBoard to the parent directory:
```sh
$ tensorboard --logdir=/tmp/tfmodels/mnist_estimator
```
Take a look again at the TensorBoard info. (You'll notice that the graph nodes are being packaged up a bit more hierarchically this time, due to the use of the 'layers' methods.)
<a href="https://storage.googleapis.com/oscon-tf-workshop-materials/images/mnist_cnn_layers_graph.png" target="_blank"><img src="https://storage.googleapis.com/oscon-tf-workshop-materials/images/mnist_cnn_layers_graph.png" width="500"/></a>
Note: The [transfer learning](../../transfer_learning/README.md) and [word2vec](../../word2vec/README.md) labs show additional examples of building and using custom Estimators.
@@ -14,6 +14,10 @@
# limitations under the License.
# ==============================================================================

"""Trains MNIST using the model described here:
https://www.tensorflow.org/versions/r0.11/tutorials/mnist/pros/index.html#deep-mnist-for-experts
"""

import os
import time

@@ -14,7 +14,9 @@
# limitations under the License.
# ==============================================================================

"""Trains MNIST using tf.contrib.learn.
"""Trains MNIST using a custom estimator, with the model based on the one here:
https://www.tensorflow.org/versions/r0.11/tutorials/mnist/pros/index.html#deep-mnist-for-experts ,
and using tf.contrib.layers to build the model.
"""

from __future__ import absolute_import
@@ -14,7 +14,8 @@
# limitations under the License.
# ==============================================================================

"""Trains MNIST using tf.contrib.learn.
"""Trains MNIST using a custom estimator, with the model based on the one here:
https://www.tensorflow.org/versions/r0.11/tutorials/mnist/pros/index.html#deep-mnist-for-experts
"""

from __future__ import absolute_import
@@ -111,8 +112,9 @@ def model_fn(x, target, mode, params):
# optimizer=tf.train.AdamOptimizer
optimizer="Adam")

predictions = tf.argmax(y_conv, 1)
return predictions, cross_entropy, train_op
prediction = tf.argmax(y_conv, 1)
# you can alternately construct and return a prediction dict
return prediction, cross_entropy, train_op


def run_cnn_classifier():
@@ -13,7 +13,7 @@
- [3. A Custom Esimator for the transfer learning model](#3-a-custom-esimator-for-the-transfer-learning-model)
- [Named Scopes and TensorBoard Summary information](#named-scopes-and-tensorboard-summary-information)
- [4. Exercise: Building the Custom Estimator's model graph](#4-exercise-building-the-custom-estimators-model-graph)
- [If you get stuck...](#if-you-get-stuck)


## Introduction

@@ -134,21 +134,26 @@ However, here, as we're wrapping things in an Estimator, we don't need to an an
## 4. Exercise: Building the Custom Estimator's model graph
Start with [`transfer_learning_skeleton.py`](transfer_learning.py), and complete the `_make_model` function definition. This function builds the model graph for the custom estimator.
As noted above, the Inception model graph is doing the heavy lifting here. We will just train a new top layer to identify our new classes: that is, we will just add a new softmax and fully-connected layer. The input to this layer is the generated "bottleneck" values.
The `add_final_training_ops` function defines this layer, then defines the loss function and the training op.
Start with [`transfer_learning_skeleton.py`](transfer_learning.py), and complete the `_make_model`
function definition. This function builds the model graph for the custom estimator.
Then, the `add_evaluation_step` function adds an op to evaluate the accuracy of the results.
As noted above, the Inception model graph is doing the heavy lifting here. We will just train a new
top layer to identify our new classes: that is, we will just add a new softmax and fully-connected
layer. The input to this layer is the generated "bottleneck" values. The `add_final_training_ops`
function defines this layer, then defines the loss function and the training op.
Put these pieces together to build the bulk of the model graph.
Then, the `add_evaluation_step` function adds an op to evaluate the accuracy of the results. Add
'loss' and 'accuracy' metrics to the prediction_dict, as per the `METRICS` dict below
`make_model_fn` in the code, which we will then pass to the Estimator's `evaluate()` method.
Then add support for generating prediction value(s).
See if you can figure out how to derive the index of the highest-value the entry in the result vector, and store that value at the `"index"` key in the `prediction_dict`.
As a hint, take a look at the ops used in `add_evaluation_step()`.
Then, add support for generating prediction value(s).
See if you can figure out how to derive the index of the highest-value the entry in the result
vector, and store that value at the `"index"` key in the `prediction_dict`. As a hint, take a look
at the ops used in `add_evaluation_step()`.
As shown in the skeleton function, be sure to return the prediction dict, the loss, and the training op. This info sets up the Estimator to handle calls to its `fit()`, `evaluate()`, and `predict()` methods.
As shown in the skeleton of `_make_model`, be sure to return the prediction dict, the loss, and the
training op. This info sets up the Estimator to handle calls to its `fit()`, `evaluate()`, and
`predict()` methods.
### If you get stuck...
If you get stuck, you can take a peek at `transfer_learning.py`, but try not to do that too soon.
@@ -80,6 +80,8 @@

from tensorflow.python.platform import gfile
from tensorflow.python.util import compat
from tensorflow.contrib.learn.python.learn import metric_spec
from tensorflow.contrib.metrics.python.ops import metric_ops

FLAGS = tf.app.flags.FLAGS
ARGFLAGS = None
@@ -587,18 +589,32 @@ def _make_model(bottleneck_input, ground_truth_input, mode, params):
class_count, mode, final_tensor_name,
bottleneck_input, ground_truth_input)

if mode in [ModeKeys.EVAL, ModeKeys.TRAIN]:
if mode == ModeKeys.EVAL:
prediction_dict['loss'] = cross_entropy
# Create the operations we need to evaluate accuracy
add_evaluation_step(final_tensor, ground_truth_input)
acc = add_evaluation_step(final_tensor, ground_truth_input)
prediction_dict['accuracy'] = acc

if mode == ModeKeys.INFER:
predclass = tf.argmax(final_tensor, 1)
prediction_dict = {"class_vector": final_tensor, "index": predclass}
prediction_dict["class_vector"] = final_tensor
prediction_dict["index"] = predclass

return prediction_dict, cross_entropy, train_step

return _make_model

METRICS = {
'loss': metric_spec.MetricSpec(
metric_fn=metric_ops.streaming_mean,
prediction_key='loss'
),
'accuracy': metric_spec.MetricSpec(
metric_fn=metric_ops.streaming_mean,
prediction_key='accuracy'
)
}


def make_image_predictions(
classifier, jpeg_data_tensor, bottleneck_tensor, path_list, labels_list):
@@ -720,7 +736,7 @@ def main(_):
y=train_ground_truth, batch_size=50,
max_steps=ARGFLAGS.num_steps)

# We've completed all our training, so run a final test evaluation on
# We've completed our training, so run a test evaluation on
# some new images we haven't used before.
test_bottlenecks, test_ground_truth = get_all_cached_bottlenecks(
sess, image_lists, 'testing',
@@ -730,7 +746,7 @@ def main(_):
test_ground_truth = np.array(test_ground_truth)
print("evaluating....")
print(classifier.evaluate(
test_bottlenecks.astype(np.float32), test_ground_truth))
test_bottlenecks.astype(np.float32), test_ground_truth, metrics=METRICS))

# write the output labels file if it doesn't already exist
output_labels_file = os.path.join(ARGFLAGS.model_dir, LABELS_FILENAME)
@@ -586,9 +586,13 @@ def _make_model(bottleneck_input, ground_truth_input, mode, params):
# of the results.
# Use the functions 'add_final_training_ops()' and 'add_evaluation_step()'.

# Then, add support for generating prediction information.
# The evaluation part of the graph only needs to be added if the 'mode' is
# ModeKeys.EVAL or ModeKeys.TRAIN.
# ModeKeys.EVAL.
# Add 'loss' and 'accuracy' metrics to the prediction_dict, as per the
# METRICS dict below, which we will pass to the Estimator's evaluate()
# method.

# Then, add support for generating prediction information.
# The prediction part of the graph only needs to be added if the 'mode' is
# ModeKeys.INFER.

@@ -605,6 +609,17 @@ def _make_model(bottleneck_input, ground_truth_input, mode, params):

return _make_model

METRICS = {
'loss': metric_spec.MetricSpec(
metric_fn=metric_ops.streaming_mean,
prediction_key='loss'
),
'accuracy': metric_spec.MetricSpec(
metric_fn=metric_ops.streaming_mean,
prediction_key='accuracy'
)
}


def make_image_predictions(
classifier, jpeg_data_tensor, bottleneck_tensor, path_list, labels_list):
@@ -736,7 +751,7 @@ def main(_):
test_ground_truth = np.array(test_ground_truth)
print("evaluating....")
print(classifier.evaluate(
test_bottlenecks.astype(np.float32), test_ground_truth))
test_bottlenecks.astype(np.float32), test_ground_truth, metrics=METRICS))

# write the output labels file if it doesn't already exist
output_labels_file = os.path.join(ARGFLAGS.model_dir, LABELS_FILENAME)

0 comments on commit e2aeb64

Please sign in to comment.
You can’t perform that action at this time.