contrib.data.Dataset - doc issue with Dataset.map / tf.py_func in 1.3.0rc0

[cheind]

System information

• Have I written custom code (as opposed to using a stock example script provided in TensorFlow): no
• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Windows 10
• TensorFlow installed from (source or binary): 1.3.0rc0
• Python version: 3.6
• CUDA/cuDNN version: 8/6
• GPU model and memory: GTX 1080
• Exact command to reproduce:

The following sample is taken from here and works in TF 1.2.1

import tensorflow as tf
import numpy as np

def _read_py_function(filename, label):
  return np.zeros((100,100,1)), label

def _resize_function(image_decoded, label):
  image_decoded.set_shape([None, None, None])
  image_resized = tf.image.resize_images(image_decoded, [28, 28])
  return image_resized, label

filenames = np.array(["/var/data/image1.jpg", "/var/data/image2.jpg"])
labels = np.array([0, 37])

dataset = tf.contrib.data.Dataset.from_tensor_slices((filenames, labels))
dataset = dataset.map(
    lambda filename, label: tf.py_func(
        _read_py_function, [filename, label], [tf.uint8, label.dtype]))
dataset = dataset.map(_resize_function)

In 1.3.0rc0 the following error is produced

Cannot convert a list containing a tensor of dtype <dtype: 'int32'> to <dtype: 'uint8'> (Tensor is: <tf.Tensor 'PyFunc:1' shape=<unknown> dtype=int32>)

This is due to the breaking change mentioned in release notes. To fix, one now has to introduce an explicit tuple() like so

dataset = dataset.map(
    lambda filename, label: tuple(tf.py_func(
        _read_py_function, [filename, label], [tf.uint8, label.dtype])))

This should at least be mentioned in the API docs / programmer guide.

[aselle]

@mrry, could you update the documentation to reflect this change?

[aselle]

Thanks @cheind for reporting this.

[cheind]

@mrry You're welcome. While this is certainly a doc issue at this point, I want to raise the concern that assigning tuple and list totally different semantics in Python is very uncommon (even in Tensorflow) and could lead to many suprising moments on user side.

[GPhilo]

I have a similar error message, but I'm not sure if I got the proposed solution correctly.
The code raising the error:

dataset = tf.contrib.data.TFRecordDataset(shard_files)
dataset = dataset.map(partial(decoder.decode, items=['label', 'image']))

where decoder is a tf.contrib.slim.tfexample_decoder.TFExampleDecoder()

So, if I got the issue right, since decoder.decode() returns a list [label, decoded_image_data] this is implicitly casted to a tensor (and thus the cast fails because label and image_data have different types).
However, if I write a lambda that wraps tuple() around the result of the call to decoder.decode(), this should fix the problem:

dataset = dataset.map( lambda s : tuple(decoder.decode(s, items=['label', 'image'])))

I still get the error, however:

TypeError: Cannot convert a list containing a tensor of dtype <dtype: 'float32'> to <dtype: 'int64'> (Tensor is: <tf.Tensor 'distort_image/Mul:0' shape=(224, 224, 3) dtype=float32>)

Am I getting the solution wrong? Is this really the intended way to use the API? Even when many other TF APIs return lists instead of tuples?

EDIT: I had map() in several places and, of course, the new error was coming from the line below the one I fixed.

[mrry]

@GPhilo We've fixed this issue in the internal branch, and it should appear at HEAD soon. (Follow #12396 to see when the commit lands.) After the fix, the behavior of Dataset.map() will be the same if a function returns a tuple or a list containing the same elements, and the tuple(...) workaround will no longer be necessary.

[cheind]

@mrry sounds great!

[tamizharasank]

TypeError: Cannot convert a list containing a tensor of dtype <dtype: 'int32'> to <dtype: 'float32'> (Tensor is: <tf.Tensor 'Preprocessor/stack_1:0' shape=(1, 3) dtype=int32>)

[mrry]

@tamizharasank This issue has been closed. If you think you've found a bug, please open a new issue with enough details about your program to reproduce the problem. Otherwise, Stack Overflow may be a more useful venue for your question.