New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

No real code example for using the tensorboard embedding tab #6322

Closed
danielgordon10 opened this Issue Dec 14, 2016 · 19 comments

Comments

@danielgordon10
Contributor

danielgordon10 commented Dec 14, 2016

I spent a long time yesterday trying to understand how to get something to show up in the embedding tab on Tensorboard. It is not very well documented, and there are no code examples except for the tiny, incomplete snippets in the tutorial. I was finally able to get something working, but I think it would be helpful for future users to have a real code tutorial. I could pretty easily write one for the MNIST example since I already have my own working. Is that something that people will want?

@danielgordon10

This comment has been minimized.

Show comment
Hide comment
@danielgordon10

danielgordon10 Dec 14, 2016

Contributor

It also seems like there must be some code already working on MNIST to be able to generate the GIFs for the tutorial, so making those available would be nice, and maybe even less work.

Contributor

danielgordon10 commented Dec 14, 2016

It also seems like there must be some code already working on MNIST to be able to generate the GIFs for the tutorial, so making those available would be nice, and maybe even less work.

@smhoang

This comment has been minimized.

Show comment
Hide comment
@smhoang

smhoang Dec 15, 2016

It would be great if you can provide your working version of TensorBoard embedding visualization for MNIST example. I am looking forward to playing around with it.

smhoang commented Dec 15, 2016

It would be great if you can provide your working version of TensorBoard embedding visualization for MNIST example. I am looking forward to playing around with it.

@danielgordon10

This comment has been minimized.

Show comment
Hide comment
@danielgordon10

danielgordon10 Dec 16, 2016

Contributor

Sure. I started with the fully_connected_feed.py file and added in the necessary stuff. It seems to work for me. I tried to follow the Google style as best as I could. I even used comments! It requires Numpy and Scipy, but that isn't too much to ask.

# Copyright 2015 The TensorFlow Authors. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
# @author: Daniel Gordon <xkcd@cs.washington.edu>
#
# ==============================================================================

"""Trains and Evaluates the MNIST network using a feed dictionary."""
from __future__ import absolute_import
from __future__ import division
from __future__ import print_function

# pylint: disable=missing-docstring
import argparse
import os.path
import sys
import time

from six.moves import xrange  # pylint: disable=redefined-builtin
import tensorflow as tf

from tensorflow.contrib.tensorboard.plugins import projector
from tensorflow.examples.tutorials.mnist import input_data
from tensorflow.examples.tutorials.mnist import mnist

import numpy as np
import scipy.misc


# Basic model parameters as external flags.
FLAGS = None


def placeholder_inputs(batch_size):
  """Generate placeholder variables to represent the input tensors.

  These placeholders are used as inputs by the rest of the model building
  code and will be fed from the downloaded data in the .run() loop, below.

  Args:
    batch_size: The batch size will be baked into both placeholders.

  Returns:
    images_placeholder: Images placeholder.
    labels_placeholder: Labels placeholder.
  """
  # Note that the shapes of the placeholders match the shapes of the full
  # image and label tensors, except the first dimension is now batch_size
  # rather than the full size of the train or test data sets.
  images_placeholder = tf.placeholder(tf.float32, shape=(batch_size,
                                                         mnist.IMAGE_PIXELS))
  labels_placeholder = tf.placeholder(tf.int32, shape=(batch_size))
  return images_placeholder, labels_placeholder


def fill_feed_dict(data_set, images_pl, labels_pl):
  """Fills the feed_dict for training the given step.

  A feed_dict takes the form of:
  feed_dict = {
      <placeholder>: <tensor of values to be passed for placeholder>,
      ....
  }

  Args:
    data_set: The set of images and labels, from input_data.read_data_sets()
    images_pl: The images placeholder, from placeholder_inputs().
    labels_pl: The labels placeholder, from placeholder_inputs().

  Returns:
    feed_dict: The feed dictionary mapping from placeholders to values.
  """
  # Create the feed_dict for the placeholders filled with the next
  # `batch size` examples.
  images_feed, labels_feed = data_set.next_batch(FLAGS.batch_size,
                                                 FLAGS.fake_data)
  feed_dict = {
      images_pl: images_feed,
      labels_pl: labels_feed,
  }
  return feed_dict


def do_eval(sess,
            eval_correct,
            images_placeholder,
            labels_placeholder,
            data_set,
            return_results=False):
  """Runs one evaluation against the full epoch of data.

  Args:
    sess: The session in which the model has been trained.
    eval_correct: The Tensor that returns the number of correct predictions.
    images_placeholder: The images placeholder.
    labels_placeholder: The labels placeholder.
    data_set: The set of images and labels to evaluate, from
      input_data.read_data_sets().
    return_results: True if the results should be returned for the embedding.

  Returns:
    all_images: A list of batches of images.
    all_labels: A list of batches of labels.
    all_hidden1_outputs: A list of batches of embeddings from the first hidden
      layer.
    all_hidden2_outputs: A list of batches of embeddings from the second hidden
      layer.
  """
  # And run one epoch of eval.
  true_count = 0  # Counts the number of correct predictions.
  steps_per_epoch = data_set.num_examples // FLAGS.batch_size
  num_examples = steps_per_epoch * FLAGS.batch_size
  if return_results:
    all_images = []
    all_labels = []
    all_hidden1_outputs = []
    all_hidden2_outputs = []
    # Get the outputs before the ReLU.
    hidden1_outputs = tf.get_default_graph().get_tensor_by_name('hidden1/add:0')
    hidden2_outputs = tf.get_default_graph().get_tensor_by_name('hidden2/add:0')
  for step in xrange(steps_per_epoch):
    feed_dict = fill_feed_dict(data_set,
                               images_placeholder,
                               labels_placeholder)
    if return_results:
      all_images.append(feed_dict[images_placeholder])
      all_labels.append(feed_dict[labels_placeholder])
      curr_count, hidden1_output, hidden2_output = sess.run(
              [eval_correct, hidden1_outputs, hidden2_outputs],
              feed_dict=feed_dict)
      true_count += curr_count
      all_hidden1_outputs.append(hidden1_output)
      all_hidden2_outputs.append(hidden2_output)
    else:
      true_count += sess.run(eval_correct, feed_dict=feed_dict)
  precision = float(true_count) / num_examples
  print('  Num examples: %d  Num correct: %d  Precision @ 1: %0.04f' %
        (num_examples, true_count, precision))
  if return_results:
    return (all_images, all_labels, all_hidden1_outputs, all_hidden2_outputs)


def images_to_sprite(data):
    """Creates the sprite image along with any necessary padding

    Args:
      data: NxHxW[x3] tensor containing the images.

    Returns:
      data: Properly shaped HxWx3 image with any necessary padding.
    """
    if len(data.shape) == 3:
        data = np.tile(data[...,np.newaxis], (1,1,1,3))
    data = data.astype(np.float32)
    min = np.min(data.reshape((data.shape[0], -1)), axis=1)
    data = (data.transpose(1,2,3,0) - min).transpose(3,0,1,2)
    max = np.max(data.reshape((data.shape[0], -1)), axis=1)
    data = (data.transpose(1,2,3,0) / max).transpose(3,0,1,2)
    # Inverting the colors seems to look better for MNIST
    data = 1 - data

    n = int(np.ceil(np.sqrt(data.shape[0])))
    padding = ((0, n ** 2 - data.shape[0]), (0, 0),
            (0, 0)) + ((0, 0),) * (data.ndim - 3)
    data = np.pad(data, padding, mode='constant',
            constant_values=0)
    # Tile the individual thumbnails into an image.
    data = data.reshape((n, n) + data.shape[1:]).transpose((0, 2, 1, 3)
            + tuple(range(4, data.ndim + 1)))
    data = data.reshape((n * data.shape[1], n * data.shape[3]) + data.shape[4:])
    data = (data * 255).astype(np.uint8)
    return data

def run_training():
  """Train MNIST for a number of steps."""
  # Get the sets of images and labels for training, validation, and
  # test on MNIST.
  data_sets = input_data.read_data_sets(FLAGS.input_data_dir, FLAGS.fake_data)

  # Tell TensorFlow that the model will be built into the default Graph.
  with tf.Graph().as_default():
    # Generate placeholders for the images and labels.
    images_placeholder, labels_placeholder = placeholder_inputs(
        FLAGS.batch_size)

    # Build a Graph that computes predictions from the inference model.
    logits = mnist.inference(images_placeholder,
                             FLAGS.hidden1,
                             FLAGS.hidden2)

    # Add to the Graph the Ops for loss calculation.
    loss = mnist.loss(logits, labels_placeholder)

    # Add to the Graph the Ops that calculate and apply gradients.
    train_op = mnist.training(loss, FLAGS.learning_rate)

    # Add the Op to compare the logits to the labels during evaluation.
    eval_correct = mnist.evaluation(logits, labels_placeholder)

    # Build the summary Tensor based on the TF collection of Summaries.
    summary = tf.summary.merge_all()

    # Add the variable initializer Op.
    init = tf.global_variables_initializer()

    # Create a saver for writing training checkpoints.
    saver = tf.train.Saver()

    # Create a session for running Ops on the Graph.
    sess = tf.Session()

    # Instantiate a SummaryWriter to output summaries and the Graph.
    summary_writer = tf.summary.FileWriter(FLAGS.log_dir, sess.graph)

    # And then after everything is built:

    # Run the Op to initialize the variables.
    sess.run(init)

    # Start the training loop.
    for step in xrange(FLAGS.max_steps):
      start_time = time.time()

      # Fill a feed dictionary with the actual set of images and labels
      # for this particular training step.
      feed_dict = fill_feed_dict(data_sets.train,
                                 images_placeholder,
                                 labels_placeholder)

      # Run one step of the model.  The return values are the activations
      # from the `train_op` (which is discarded) and the `loss` Op.  To
      # inspect the values of your Ops or variables, you may include them
      # in the list passed to sess.run() and the value tensors will be
      # returned in the tuple from the call.
      _, loss_value = sess.run([train_op, loss],
                               feed_dict=feed_dict)

      duration = time.time() - start_time

      # Write the summaries and print an overview fairly often.
      if step % 100 == 0:
        # Print status to stdout.
        print('Step %d: loss = %.2f (%.3f sec)' % (step, loss_value, duration))
        # Update the events file.
        summary_str = sess.run(summary, feed_dict=feed_dict)
        summary_writer.add_summary(summary_str, step)
        summary_writer.flush()

      # Save a checkpoint and evaluate the model periodically.
      if (step + 1) % 1000 == 0 or (step + 1) == FLAGS.max_steps:
        checkpoint_file = os.path.join(FLAGS.log_dir, 'model.ckpt')
        saver.save(sess, checkpoint_file, global_step=step)
        # Evaluate against the training set.
        print('Training Data Eval:')
        do_eval(sess,
                eval_correct,
                images_placeholder,
                labels_placeholder,
                data_sets.train)
        # Evaluate against the validation set.
        print('Validation Data Eval:')
        do_eval(sess,
                eval_correct,
                images_placeholder,
                labels_placeholder,
                data_sets.validation)
        # Evaluate against the test set.
        print('Test Data Eval:')
        do_eval(sess,
                eval_correct,
                images_placeholder,
                labels_placeholder,
                data_sets.test)

    # Compute embeddings and save them.
    thumbnail_size = int(np.sqrt(mnist.IMAGE_PIXELS))
    for data_set, name in [
            (data_sets.train, 'train'),
            (data_sets.validation, 'validation'),
            (data_sets.test, 'test')]:
      output_path = os.path.join(FLAGS.log_dir, 'embed', name)
      print('Computing %s Embedding' % name)
      (all_images, all_labels, hidden1_vectors, hidden2_vectors) = do_eval(
              sess,
              eval_correct,
              images_placeholder,
              labels_placeholder,
              data_set,
              True)
      embed_tensors = []
      summary_writer = tf.summary.FileWriter(output_path, sess.graph)
      config = projector.ProjectorConfig()
      for layer, embed_vectors in enumerate([hidden1_vectors, hidden2_vectors]):
        embed_tensor = tf.Variable(
                np.array(embed_vectors).reshape(
                    len(embed_vectors) * embed_vectors[0].shape[0], -1),
                name=('%s_layer_%s' % (name, layer)))
        embed_tensors.append(embed_tensor)
        sess.run(embed_tensor.initializer)
        embedding = config.embeddings.add()
        embedding.tensor_name = embed_tensor.name
        embedding.metadata_path = os.path.join(output_path, 'labels.tsv')
        embedding.sprite.image_path = os.path.join(output_path, 'sprite.png')
        embedding.sprite.single_image_dim.extend(
                [thumbnail_size, thumbnail_size])
        projector.visualize_embeddings(summary_writer, config)
      result = sess.run(embed_tensors)
      saver = tf.train.Saver(embed_tensors)
      saver.save(sess, os.path.join(output_path, 'model.ckpt'), layer)

      # Make sprite and labels.
      images = np.array(all_images).reshape(
              -1, thumbnail_size, thumbnail_size).astype(np.float32)
      sprite = images_to_sprite(images)
      scipy.misc.imsave(os.path.join(output_path, 'sprite.png'), sprite)
      all_labels = np.array(all_labels).flatten()
      metadata_file = open(os.path.join(output_path, 'labels.tsv'), 'w')
      metadata_file.write('Name\tClass\n')
      for ll in xrange(len(all_labels)):
        metadata_file.write('%06d\t%d\n' % (ll, all_labels[ll]))
      metadata_file.close()

def main(_):
  if tf.gfile.Exists(FLAGS.log_dir):
    tf.gfile.DeleteRecursively(FLAGS.log_dir)
  tf.gfile.MakeDirs(FLAGS.log_dir)
  run_training()


if __name__ == '__main__':
  parser = argparse.ArgumentParser()
  parser.add_argument(
      '--learning_rate',
      type=float,
      default=0.01,
      help='Initial learning rate.'
  )
  parser.add_argument(
      '--max_steps',
      type=int,
      default=2000,
      help='Number of steps to run trainer.'
  )
  parser.add_argument(
      '--hidden1',
      type=int,
      default=128,
      help='Number of units in hidden layer 1.'
  )
  parser.add_argument(
      '--hidden2',
      type=int,
      default=32,
      help='Number of units in hidden layer 2.'
  )
  parser.add_argument(
      '--batch_size',
      type=int,
      default=100,
      help='Batch size.  Must divide evenly into the dataset sizes.'
  )
  parser.add_argument(
      '--input_data_dir',
      type=str,
      default='/tmp/tensorflow/mnist/input_data',
      help='Directory to put the input data.'
  )
  parser.add_argument(
      '--log_dir',
      type=str,
      default='/tmp/tensorflow/mnist/logs/fully_connected_feed',
      help='Directory to put the log data.'
  )
  parser.add_argument(
      '--fake_data',
      default=False,
      help='If true, uses fake data for unit testing.',
      action='store_true'
  )

  FLAGS, unparsed = parser.parse_known_args()
  tf.app.run(main=main, argv=[sys.argv[0]] + unparsed)
Contributor

danielgordon10 commented Dec 16, 2016

Sure. I started with the fully_connected_feed.py file and added in the necessary stuff. It seems to work for me. I tried to follow the Google style as best as I could. I even used comments! It requires Numpy and Scipy, but that isn't too much to ask.

# Copyright 2015 The TensorFlow Authors. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
# @author: Daniel Gordon <xkcd@cs.washington.edu>
#
# ==============================================================================

"""Trains and Evaluates the MNIST network using a feed dictionary."""
from __future__ import absolute_import
from __future__ import division
from __future__ import print_function

# pylint: disable=missing-docstring
import argparse
import os.path
import sys
import time

from six.moves import xrange  # pylint: disable=redefined-builtin
import tensorflow as tf

from tensorflow.contrib.tensorboard.plugins import projector
from tensorflow.examples.tutorials.mnist import input_data
from tensorflow.examples.tutorials.mnist import mnist

import numpy as np
import scipy.misc


# Basic model parameters as external flags.
FLAGS = None


def placeholder_inputs(batch_size):
  """Generate placeholder variables to represent the input tensors.

  These placeholders are used as inputs by the rest of the model building
  code and will be fed from the downloaded data in the .run() loop, below.

  Args:
    batch_size: The batch size will be baked into both placeholders.

  Returns:
    images_placeholder: Images placeholder.
    labels_placeholder: Labels placeholder.
  """
  # Note that the shapes of the placeholders match the shapes of the full
  # image and label tensors, except the first dimension is now batch_size
  # rather than the full size of the train or test data sets.
  images_placeholder = tf.placeholder(tf.float32, shape=(batch_size,
                                                         mnist.IMAGE_PIXELS))
  labels_placeholder = tf.placeholder(tf.int32, shape=(batch_size))
  return images_placeholder, labels_placeholder


def fill_feed_dict(data_set, images_pl, labels_pl):
  """Fills the feed_dict for training the given step.

  A feed_dict takes the form of:
  feed_dict = {
      <placeholder>: <tensor of values to be passed for placeholder>,
      ....
  }

  Args:
    data_set: The set of images and labels, from input_data.read_data_sets()
    images_pl: The images placeholder, from placeholder_inputs().
    labels_pl: The labels placeholder, from placeholder_inputs().

  Returns:
    feed_dict: The feed dictionary mapping from placeholders to values.
  """
  # Create the feed_dict for the placeholders filled with the next
  # `batch size` examples.
  images_feed, labels_feed = data_set.next_batch(FLAGS.batch_size,
                                                 FLAGS.fake_data)
  feed_dict = {
      images_pl: images_feed,
      labels_pl: labels_feed,
  }
  return feed_dict


def do_eval(sess,
            eval_correct,
            images_placeholder,
            labels_placeholder,
            data_set,
            return_results=False):
  """Runs one evaluation against the full epoch of data.

  Args:
    sess: The session in which the model has been trained.
    eval_correct: The Tensor that returns the number of correct predictions.
    images_placeholder: The images placeholder.
    labels_placeholder: The labels placeholder.
    data_set: The set of images and labels to evaluate, from
      input_data.read_data_sets().
    return_results: True if the results should be returned for the embedding.

  Returns:
    all_images: A list of batches of images.
    all_labels: A list of batches of labels.
    all_hidden1_outputs: A list of batches of embeddings from the first hidden
      layer.
    all_hidden2_outputs: A list of batches of embeddings from the second hidden
      layer.
  """
  # And run one epoch of eval.
  true_count = 0  # Counts the number of correct predictions.
  steps_per_epoch = data_set.num_examples // FLAGS.batch_size
  num_examples = steps_per_epoch * FLAGS.batch_size
  if return_results:
    all_images = []
    all_labels = []
    all_hidden1_outputs = []
    all_hidden2_outputs = []
    # Get the outputs before the ReLU.
    hidden1_outputs = tf.get_default_graph().get_tensor_by_name('hidden1/add:0')
    hidden2_outputs = tf.get_default_graph().get_tensor_by_name('hidden2/add:0')
  for step in xrange(steps_per_epoch):
    feed_dict = fill_feed_dict(data_set,
                               images_placeholder,
                               labels_placeholder)
    if return_results:
      all_images.append(feed_dict[images_placeholder])
      all_labels.append(feed_dict[labels_placeholder])
      curr_count, hidden1_output, hidden2_output = sess.run(
              [eval_correct, hidden1_outputs, hidden2_outputs],
              feed_dict=feed_dict)
      true_count += curr_count
      all_hidden1_outputs.append(hidden1_output)
      all_hidden2_outputs.append(hidden2_output)
    else:
      true_count += sess.run(eval_correct, feed_dict=feed_dict)
  precision = float(true_count) / num_examples
  print('  Num examples: %d  Num correct: %d  Precision @ 1: %0.04f' %
        (num_examples, true_count, precision))
  if return_results:
    return (all_images, all_labels, all_hidden1_outputs, all_hidden2_outputs)


def images_to_sprite(data):
    """Creates the sprite image along with any necessary padding

    Args:
      data: NxHxW[x3] tensor containing the images.

    Returns:
      data: Properly shaped HxWx3 image with any necessary padding.
    """
    if len(data.shape) == 3:
        data = np.tile(data[...,np.newaxis], (1,1,1,3))
    data = data.astype(np.float32)
    min = np.min(data.reshape((data.shape[0], -1)), axis=1)
    data = (data.transpose(1,2,3,0) - min).transpose(3,0,1,2)
    max = np.max(data.reshape((data.shape[0], -1)), axis=1)
    data = (data.transpose(1,2,3,0) / max).transpose(3,0,1,2)
    # Inverting the colors seems to look better for MNIST
    data = 1 - data

    n = int(np.ceil(np.sqrt(data.shape[0])))
    padding = ((0, n ** 2 - data.shape[0]), (0, 0),
            (0, 0)) + ((0, 0),) * (data.ndim - 3)
    data = np.pad(data, padding, mode='constant',
            constant_values=0)
    # Tile the individual thumbnails into an image.
    data = data.reshape((n, n) + data.shape[1:]).transpose((0, 2, 1, 3)
            + tuple(range(4, data.ndim + 1)))
    data = data.reshape((n * data.shape[1], n * data.shape[3]) + data.shape[4:])
    data = (data * 255).astype(np.uint8)
    return data

def run_training():
  """Train MNIST for a number of steps."""
  # Get the sets of images and labels for training, validation, and
  # test on MNIST.
  data_sets = input_data.read_data_sets(FLAGS.input_data_dir, FLAGS.fake_data)

  # Tell TensorFlow that the model will be built into the default Graph.
  with tf.Graph().as_default():
    # Generate placeholders for the images and labels.
    images_placeholder, labels_placeholder = placeholder_inputs(
        FLAGS.batch_size)

    # Build a Graph that computes predictions from the inference model.
    logits = mnist.inference(images_placeholder,
                             FLAGS.hidden1,
                             FLAGS.hidden2)

    # Add to the Graph the Ops for loss calculation.
    loss = mnist.loss(logits, labels_placeholder)

    # Add to the Graph the Ops that calculate and apply gradients.
    train_op = mnist.training(loss, FLAGS.learning_rate)

    # Add the Op to compare the logits to the labels during evaluation.
    eval_correct = mnist.evaluation(logits, labels_placeholder)

    # Build the summary Tensor based on the TF collection of Summaries.
    summary = tf.summary.merge_all()

    # Add the variable initializer Op.
    init = tf.global_variables_initializer()

    # Create a saver for writing training checkpoints.
    saver = tf.train.Saver()

    # Create a session for running Ops on the Graph.
    sess = tf.Session()

    # Instantiate a SummaryWriter to output summaries and the Graph.
    summary_writer = tf.summary.FileWriter(FLAGS.log_dir, sess.graph)

    # And then after everything is built:

    # Run the Op to initialize the variables.
    sess.run(init)

    # Start the training loop.
    for step in xrange(FLAGS.max_steps):
      start_time = time.time()

      # Fill a feed dictionary with the actual set of images and labels
      # for this particular training step.
      feed_dict = fill_feed_dict(data_sets.train,
                                 images_placeholder,
                                 labels_placeholder)

      # Run one step of the model.  The return values are the activations
      # from the `train_op` (which is discarded) and the `loss` Op.  To
      # inspect the values of your Ops or variables, you may include them
      # in the list passed to sess.run() and the value tensors will be
      # returned in the tuple from the call.
      _, loss_value = sess.run([train_op, loss],
                               feed_dict=feed_dict)

      duration = time.time() - start_time

      # Write the summaries and print an overview fairly often.
      if step % 100 == 0:
        # Print status to stdout.
        print('Step %d: loss = %.2f (%.3f sec)' % (step, loss_value, duration))
        # Update the events file.
        summary_str = sess.run(summary, feed_dict=feed_dict)
        summary_writer.add_summary(summary_str, step)
        summary_writer.flush()

      # Save a checkpoint and evaluate the model periodically.
      if (step + 1) % 1000 == 0 or (step + 1) == FLAGS.max_steps:
        checkpoint_file = os.path.join(FLAGS.log_dir, 'model.ckpt')
        saver.save(sess, checkpoint_file, global_step=step)
        # Evaluate against the training set.
        print('Training Data Eval:')
        do_eval(sess,
                eval_correct,
                images_placeholder,
                labels_placeholder,
                data_sets.train)
        # Evaluate against the validation set.
        print('Validation Data Eval:')
        do_eval(sess,
                eval_correct,
                images_placeholder,
                labels_placeholder,
                data_sets.validation)
        # Evaluate against the test set.
        print('Test Data Eval:')
        do_eval(sess,
                eval_correct,
                images_placeholder,
                labels_placeholder,
                data_sets.test)

    # Compute embeddings and save them.
    thumbnail_size = int(np.sqrt(mnist.IMAGE_PIXELS))
    for data_set, name in [
            (data_sets.train, 'train'),
            (data_sets.validation, 'validation'),
            (data_sets.test, 'test')]:
      output_path = os.path.join(FLAGS.log_dir, 'embed', name)
      print('Computing %s Embedding' % name)
      (all_images, all_labels, hidden1_vectors, hidden2_vectors) = do_eval(
              sess,
              eval_correct,
              images_placeholder,
              labels_placeholder,
              data_set,
              True)
      embed_tensors = []
      summary_writer = tf.summary.FileWriter(output_path, sess.graph)
      config = projector.ProjectorConfig()
      for layer, embed_vectors in enumerate([hidden1_vectors, hidden2_vectors]):
        embed_tensor = tf.Variable(
                np.array(embed_vectors).reshape(
                    len(embed_vectors) * embed_vectors[0].shape[0], -1),
                name=('%s_layer_%s' % (name, layer)))
        embed_tensors.append(embed_tensor)
        sess.run(embed_tensor.initializer)
        embedding = config.embeddings.add()
        embedding.tensor_name = embed_tensor.name
        embedding.metadata_path = os.path.join(output_path, 'labels.tsv')
        embedding.sprite.image_path = os.path.join(output_path, 'sprite.png')
        embedding.sprite.single_image_dim.extend(
                [thumbnail_size, thumbnail_size])
        projector.visualize_embeddings(summary_writer, config)
      result = sess.run(embed_tensors)
      saver = tf.train.Saver(embed_tensors)
      saver.save(sess, os.path.join(output_path, 'model.ckpt'), layer)

      # Make sprite and labels.
      images = np.array(all_images).reshape(
              -1, thumbnail_size, thumbnail_size).astype(np.float32)
      sprite = images_to_sprite(images)
      scipy.misc.imsave(os.path.join(output_path, 'sprite.png'), sprite)
      all_labels = np.array(all_labels).flatten()
      metadata_file = open(os.path.join(output_path, 'labels.tsv'), 'w')
      metadata_file.write('Name\tClass\n')
      for ll in xrange(len(all_labels)):
        metadata_file.write('%06d\t%d\n' % (ll, all_labels[ll]))
      metadata_file.close()

def main(_):
  if tf.gfile.Exists(FLAGS.log_dir):
    tf.gfile.DeleteRecursively(FLAGS.log_dir)
  tf.gfile.MakeDirs(FLAGS.log_dir)
  run_training()


if __name__ == '__main__':
  parser = argparse.ArgumentParser()
  parser.add_argument(
      '--learning_rate',
      type=float,
      default=0.01,
      help='Initial learning rate.'
  )
  parser.add_argument(
      '--max_steps',
      type=int,
      default=2000,
      help='Number of steps to run trainer.'
  )
  parser.add_argument(
      '--hidden1',
      type=int,
      default=128,
      help='Number of units in hidden layer 1.'
  )
  parser.add_argument(
      '--hidden2',
      type=int,
      default=32,
      help='Number of units in hidden layer 2.'
  )
  parser.add_argument(
      '--batch_size',
      type=int,
      default=100,
      help='Batch size.  Must divide evenly into the dataset sizes.'
  )
  parser.add_argument(
      '--input_data_dir',
      type=str,
      default='/tmp/tensorflow/mnist/input_data',
      help='Directory to put the input data.'
  )
  parser.add_argument(
      '--log_dir',
      type=str,
      default='/tmp/tensorflow/mnist/logs/fully_connected_feed',
      help='Directory to put the log data.'
  )
  parser.add_argument(
      '--fake_data',
      default=False,
      help='If true, uses fake data for unit testing.',
      action='store_true'
  )

  FLAGS, unparsed = parser.parse_known_args()
  tf.app.run(main=main, argv=[sys.argv[0]] + unparsed)
@oduerr

This comment has been minimized.

Show comment
Hide comment
@oduerr

oduerr Dec 16, 2016

I also was confused like hell by the description. I created a notebook, maybe it helps someone. It's not about MNIST but with CIFAR

https://github.com/oduerr/dl_tutorial/blob/master/tensorflow/debugging/embedding.ipynb

oduerr commented Dec 16, 2016

I also was confused like hell by the description. I created a notebook, maybe it helps someone. It's not about MNIST but with CIFAR

https://github.com/oduerr/dl_tutorial/blob/master/tensorflow/debugging/embedding.ipynb

@ahirner

This comment has been minimized.

Show comment
Hide comment
@ahirner

ahirner Dec 20, 2016

Thanks for ending the confusion @danielgordon10 @oduerr! I first thought there should be a direct way of writing a checkpoint from an operation's output. However, it seem you have to create a temporary variable first, initialize it with embeddings from a previous run and then save it.

ahirner commented Dec 20, 2016

Thanks for ending the confusion @danielgordon10 @oduerr! I first thought there should be a direct way of writing a checkpoint from an operation's output. However, it seem you have to create a temporary variable first, initialize it with embeddings from a previous run and then save it.

@pannous

This comment has been minimized.

Show comment
Hide comment
@pannous

pannous Dec 21, 2016

@danielgordon10 @oduerr most wonderful, thanks!
This should definitely be part of the official docu/samples!

pannous commented Dec 21, 2016

@danielgordon10 @oduerr most wonderful, thanks!
This should definitely be part of the official docu/samples!

@danielgordon10

This comment has been minimized.

Show comment
Hide comment
@danielgordon10

danielgordon10 Dec 27, 2016

Contributor

Should I make a PR or something? Would this go in Contrib somewhere or with the rest of the MNIST examples?

Contributor

danielgordon10 commented Dec 27, 2016

Should I make a PR or something? Would this go in Contrib somewhere or with the rest of the MNIST examples?

@decentralion

This comment has been minimized.

Show comment
Hide comment
@decentralion

decentralion Jan 12, 2017

Contributor

Sorry for the slow response, this was assigned to me while I was on vacation.

We would be grateful to receive pull requests that improve the documentation. @dsmilkov would be the best person to review

Contributor

decentralion commented Jan 12, 2017

Sorry for the slow response, this was assigned to me while I was on vacation.

We would be grateful to receive pull requests that improve the documentation. @dsmilkov would be the best person to review

@danielgordon10

This comment has been minimized.

Show comment
Hide comment
@danielgordon10

danielgordon10 Jan 22, 2017

Contributor

@dsmilkov where should this code live? I can't really do a pull request if I don't know where to put the code.

Contributor

danielgordon10 commented Jan 22, 2017

@dsmilkov where should this code live? I can't really do a pull request if I don't know where to put the code.

@aselle aselle added type:feature and removed enhancement labels Feb 9, 2017

@Franck-Dernoncourt

This comment has been minimized.

Show comment
Hide comment
@Franck-Dernoncourt

Franck-Dernoncourt Mar 8, 2017

Contributor

Corresponding Stack Exchange question: TensorBoard Embedding Example?

Contributor

Franck-Dernoncourt commented Mar 8, 2017

Corresponding Stack Exchange question: TensorBoard Embedding Example?

@dartdog

This comment has been minimized.

Show comment
Hide comment
@dartdog

dartdog Mar 13, 2017

@danielgordon10 Thank you for the sample code, so far it is the only example I have gotten to work at all for visualizing the embedding.. (using TF1.0 and P36 fwiw)However while it produces a pretty picture I cannot make sense of the labels.. they are three digit numbers when hovering over the dots in the 3d graph.. Although when I select the softmax layer I get 1-two digits when hovering.. Some further description of the utility of the projection and the meaning of the various components of the model (Component 1-3 in the selector boxes for instance) would be super helpful!

dartdog commented Mar 13, 2017

@danielgordon10 Thank you for the sample code, so far it is the only example I have gotten to work at all for visualizing the embedding.. (using TF1.0 and P36 fwiw)However while it produces a pretty picture I cannot make sense of the labels.. they are three digit numbers when hovering over the dots in the 3d graph.. Although when I select the softmax layer I get 1-two digits when hovering.. Some further description of the utility of the projection and the meaning of the various components of the model (Component 1-3 in the selector boxes for instance) would be super helpful!

@Franck-Dernoncourt

This comment has been minimized.

Show comment
Hide comment
@Franck-Dernoncourt

Franck-Dernoncourt Mar 14, 2017

Contributor

If interested, some code that uses TensorBoard embeddings to display character and word embeddings:
https://github.com/Franck-Dernoncourt/NeuroNER

Example:

image

image

FYI: How can I select which checkpoint to view in TensorBoard's embeddings tab?

Contributor

Franck-Dernoncourt commented Mar 14, 2017

If interested, some code that uses TensorBoard embeddings to display character and word embeddings:
https://github.com/Franck-Dernoncourt/NeuroNER

Example:

image

image

FYI: How can I select which checkpoint to view in TensorBoard's embeddings tab?

@EddieOne

This comment has been minimized.

Show comment
Hide comment
@EddieOne

EddieOne Apr 30, 2017

For anyone struggling to get tensorboard embeddings working, I would suggest the standalone embeddings. It has example input files which were a massive help for me. For learning, it is much easier to create embedding data using the final testing code rather than the training code as suggested above. So the question is no longer about embedding examples but how to prepare embedding data.

Here's a beautiful example of what you can do http://onefoldmedia.com/sites/default/super_t-sne/

EddieOne commented Apr 30, 2017

For anyone struggling to get tensorboard embeddings working, I would suggest the standalone embeddings. It has example input files which were a massive help for me. For learning, it is much easier to create embedding data using the final testing code rather than the training code as suggested above. So the question is no longer about embedding examples but how to prepare embedding data.

Here's a beautiful example of what you can do http://onefoldmedia.com/sites/default/super_t-sne/

@evaliyev

This comment has been minimized.

Show comment
Hide comment
@evaliyev

evaliyev commented May 21, 2017

Simple example of embedding visualization
https://github.com/ElchinValiyev/tf-embedding-visualization-demo

@decentralion

This comment has been minimized.

Show comment
Hide comment
@decentralion

decentralion Jun 16, 2017

Contributor

You can also check out my TF Dev summit talk and associated code.

Anyway, we should add a canonical demo for the projector. I'm going to migrate this issue to tensorflow/tensorboard.

Contributor

decentralion commented Jun 16, 2017

You can also check out my TF Dev summit talk and associated code.

Anyway, we should add a canonical demo for the projector. I'm going to migrate this issue to tensorflow/tensorboard.

@xzy256

This comment has been minimized.

Show comment
Hide comment
@xzy256

xzy256 Jul 4, 2017

@danielgordon10 I run the code successfully. But when i open localhost:6006/#embeddings, tensorboard fetching sprite image... parsing metadata takes forever. There are #graphand #scalars . I reduce the bach_size, step_max,embedding data size, but embedding doesn't work. How to do?

xzy256 commented Jul 4, 2017

@danielgordon10 I run the code successfully. But when i open localhost:6006/#embeddings, tensorboard fetching sprite image... parsing metadata takes forever. There are #graphand #scalars . I reduce the bach_size, step_max,embedding data size, but embedding doesn't work. How to do?

@guillaume-chevalier

This comment has been minimized.

Show comment
Hide comment
@guillaume-chevalier

guillaume-chevalier Jul 25, 2017

In case anyone is interested, I have built a code example on how to use embeddings in TensorBoard.

Basically, it is based off @normanheckscher 's code for TensorBoard but with @pkmital 's autoencoder. Therefore the autoencoder is used for its embeddings (a.k.a. "code", or "compressed representation") to be visualized in TensorBoard. I managed to merge and document those two pieces of code in the simplest possible way just as to make a quick working demo.

guillaume-chevalier commented Jul 25, 2017

In case anyone is interested, I have built a code example on how to use embeddings in TensorBoard.

Basically, it is based off @normanheckscher 's code for TensorBoard but with @pkmital 's autoencoder. Therefore the autoencoder is used for its embeddings (a.k.a. "code", or "compressed representation") to be visualized in TensorBoard. I managed to merge and document those two pieces of code in the simplest possible way just as to make a quick working demo.

@harveyslash

This comment has been minimized.

Show comment
Hide comment
@harveyslash

harveyslash Oct 12, 2017

@EddieOne , i like your solution the best, but I am not sure what the tensor.bytes are.
I am planning to use keras. What should the format of the .bytes file be ?

harveyslash commented Oct 12, 2017

@EddieOne , i like your solution the best, but I am not sure what the tensor.bytes are.
I am planning to use keras. What should the format of the .bytes file be ?

@altermarkive

This comment has been minimized.

Show comment
Hide comment
@altermarkive

altermarkive Nov 14, 2017

While by now there are many MNIST-based examples, for me they still relied too much on pre-trained data and pre-generated sprite & metadata files. To fill this gap I created such an example myself and decided to share it here for anyone interested - the code is on GitHub and the video run-through is on YouTube. I hope it helps anyone who stumbles upon the same challenge.

altermarkive commented Nov 14, 2017

While by now there are many MNIST-based examples, for me they still relied too much on pre-trained data and pre-generated sprite & metadata files. To fill this gap I created such an example myself and decided to share it here for anyone interested - the code is on GitHub and the video run-through is on YouTube. I hope it helps anyone who stumbles upon the same challenge.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment