Im2Txt not working for other languages like hindi

[shahparth123]

---

System information

• What is the top-level directory of the model you are using models/Im2Txt:
• OS Platform and Distribution : Linux CentOS 7:
• TensorFlow installed from Binary:
• TensorFlow version:('v1.0.0-rc1-102-g1536a84-dirty', '1.0.0-rc2'):

I am trying to use im2txt model with MSCOCO dataset modified to use hindi caption. When I tried to pre-process dataset using bazel-bin/im2txt/download_and_preprocess_mscoco "${MSCOCO_DIR}" command, it will work until generating word_count file. In step of generating TFrecord it throws following error:

Exception in thread Thread-14:
Traceback (most recent call last):
File "/usr/lib64/python2.7/threading.py", line 811, in __bootstrap_inner
self.run()
File "/usr/lib64/python2.7/threading.py", line 764, in run
self.__target(*self.__args, **self.__kwargs)
File "/root/.cache/bazel/_bazel_root/5d96493270bd616258f6b3292edd375b/execroot/im2txt/bazel-out/local-fastbuild/bin/im2txt/build_mscoco_data.runfiles/im2txt/im2txt/data/build_mscoco_data.py", line 281, in _process_image_files
sequence_example = _to_sequence_example(image, decoder, vocab)
File "/root/.cache/bazel/_bazel_root/5d96493270bd616258f6b3292edd375b/execroot/im2txt/bazel-out/local-fastbuild/bin/im2txt/build_mscoco_data.runfiles/im2txt/im2txt/data/build_mscoco_data.py", line 234, in _to_sequence_example
"image/caption": _bytes_feature_list(caption),
File "/root/.cache/bazel/_bazel_root/5d96493270bd616258f6b3292edd375b/execroot/im2txt/bazel-out/local-fastbuild/bin/im2txt/build_mscoco_data.runfiles/im2txt/im2txt/data/build_mscoco_data.py", line 202, in _bytes_feature_list
return tf.train.FeatureList(feature=[_bytes_feature(v) for v in values])
File "/root/.cache/bazel/_bazel_root/5d96493270bd616258f6b3292edd375b/execroot/im2txt/bazel-out/local-fastbuild/bin/im2txt/build_mscoco_data.runfiles/im2txt/im2txt/data/build_mscoco_data.py", line 192, in _bytes_feature
return tf.train.Feature(bytes_list=tf.train.BytesList(value=str(value)))
UnicodeEncodeError: 'ascii' codec can't encode characters in position 0-2: ordinal not in range(128)

It seems error in encoding data to tfrecord. I have also tried to modify str(value) using unicode encode function.
Sample caption file is available at: https://drive.google.com/file/d/0B8Rng3ofk0uAbDBJekpPR3BXVzA/view?usp=sharing

[aselle]

@cshallue, could you please take a look? Looks like it is not unicode compatible.

[cshallue]

Does the following work?

return tf.train.Feature(bytes_list=tf.train.BytesList(value=value.encode("utf-8")))

[shahparth123]

After modifying to use encode function it gives following UnicodeDecode error:

Exception in thread Thread-20:
Traceback (most recent call last):
File "/usr/lib64/python2.7/threading.py", line 811, in __bootstrap_inner
self.run()
File "/usr/lib64/python2.7/threading.py", line 764, in run
self.__target(*self.__args, **self.__kwargs)
File "/root/.cache/bazel/_bazel_root/5d96493270bd616258f6b3292edd375b/execroot/im2txt/bazel-out/local-fastbuild/bin/im2txt/build_mscoco_data.runfiles/im2txt/im2txt/data/build_mscoco_data.py", line 282, in _process_image_files
sequence_example = _to_sequence_example(image, decoder, vocab)
File "/root/.cache/bazel/_bazel_root/5d96493270bd616258f6b3292edd375b/execroot/im2txt/bazel-out/local-fastbuild/bin/im2txt/build_mscoco_data.runfiles/im2txt/im2txt/data/build_mscoco_data.py", line 228, in _to_sequence_example
"image/data": _bytes_feature(encoded_image),
File "/root/.cache/bazel/_bazel_root/5d96493270bd616258f6b3292edd375b/execroot/im2txt/bazel-out/local-fastbuild/bin/im2txt/build_mscoco_data.runfiles/im2txt/im2txt/data/build_mscoco_data.py", line 191, in _bytes_feature
return tf.train.Feature(bytes_list=tf.train.BytesList(value=value.encode("utf-8")))
UnicodeDecodeError: 'ascii' codec can't decode byte 0xff in position 0: ordinal not in range(128)

[taowenleon]

Hello, friends: I want to use Chinese caption and occured the same problem.
Have you solved this problem?
Thanks a lot.

[cshallue]

Do any of the following work?

return tf.train.Feature(bytes_list=tf.train.BytesList(value=value))
return tf.train.Feature(bytes_list=tf.train.BytesList(value=value.decode("utf-8")))
return tf.train.Feature(bytes_list=tf.train.BytesList(value=value.decode("utf-8").encode("utf-8")))

[shahparth123]

Tried all three options, Error log of each case is as follows:

return tf.train.Feature(bytes_list=tf.train.BytesList(value=value))
Exception in thread Thread-20:
Traceback (most recent call last):
File "/usr/lib64/python2.7/threading.py", line 811, in __bootstrap_inner
self.run()
File "/usr/lib64/python2.7/threading.py", line 764, in run
self.__target(*self.__args, **self.__kwargs)
File "/root/.cache/bazel/_bazel_root/5d96493270bd616258f6b3292edd375b/execroot/im2txt/bazel-out/local-fastbuild/bin/im2txt/build_mscoco_data.runfiles/im2txt/im2txt/data/build_mscoco_data.py", line 283, in _process_image_files
sequence_example = _to_sequence_example(image, decoder, vocab)
File "/root/.cache/bazel/_bazel_root/5d96493270bd616258f6b3292edd375b/execroot/im2txt/bazel-out/local-fastbuild/bin/im2txt/build_mscoco_data.runfiles/im2txt/im2txt/data/build_mscoco_data.py", line 236, in _to_sequence_example
"image/caption": _bytes_feature_list(caption),
File "/root/.cache/bazel/_bazel_root/5d96493270bd616258f6b3292edd375b/execroot/im2txt/bazel-out/local-fastbuild/bin/im2txt/build_mscoco_data.runfiles/im2txt/im2txt/data/build_mscoco_data.py", line 204, in _bytes_feature_list
return tf.train.FeatureList(feature=[_bytes_feature(v) for v in values])
File "/root/.cache/bazel/_bazel_root/5d96493270bd616258f6b3292edd375b/execroot/im2txt/bazel-out/local-fastbuild/bin/im2txt/build_mscoco_data.runfiles/im2txt/im2txt/data/build_mscoco_data.py", line 191, in _bytes_feature
return tf.train.Feature(bytes_list=tf.train.BytesList(value=value))
TypeError: u'\u090f' has type unicode, but expected one of: bytes

return tf.train.Feature(bytes_list=tf.train.BytesList(value=value.decode("utf-8")))
Exception in thread Thread-20:
Traceback (most recent call last):
File "/usr/lib64/python2.7/threading.py", line 811, in __bootstrap_inner
self.run()
File "/usr/lib64/python2.7/threading.py", line 764, in run
self.__target(*self.__args, **self.__kwargs)
File "/root/.cache/bazel/_bazel_root/5d96493270bd616258f6b3292edd375b/execroot/im2txt/bazel-out/local-fastbuild/bin/im2txt/build_mscoco_data.runfiles/im2txt/im2txt/data/build_mscoco_data.py", line 283, in _process_image_files
sequence_example = _to_sequence_example(image, decoder, vocab)
File "/root/.cache/bazel/_bazel_root/5d96493270bd616258f6b3292edd375b/execroot/im2txt/bazel-out/local-fastbuild/bin/im2txt/build_mscoco_data.runfiles/im2txt/im2txt/data/build_mscoco_data.py", line 229, in _to_sequence_example
"image/data": _bytes_feature(encoded_image),
File "/root/.cache/bazel/_bazel_root/5d96493270bd616258f6b3292edd375b/execroot/im2txt/bazel-out/local-fastbuild/bin/im2txt/build_mscoco_data.runfiles/im2txt/im2txt/data/build_mscoco_data.py", line 191, in _bytes_feature
return tf.train.Feature(bytes_list=tf.train.BytesList(value=value.decode("utf-8")))
File "/usr/lib64/python2.7/encodings/utf_8.py", line 16, in decode
return codecs.utf_8_decode(input, errors, True)
UnicodeDecodeError: 'utf8' codec can't decode byte 0xff in position 0: invalid start byte

return tf.train.Feature(bytes_list=tf.train.BytesList(value=value.decode("utf-8").encode("utf-8")))

Exception in thread Thread-20:
Traceback (most recent call last):
File "/usr/lib64/python2.7/threading.py", line 811, in __bootstrap_inner
self.run()
File "/usr/lib64/python2.7/threading.py", line 764, in run
self.__target(*self.__args, **self.__kwargs)
File "/root/.cache/bazel/_bazel_root/5d96493270bd616258f6b3292edd375b/execroot/im2txt/bazel-out/local-fastbuild/bin/im2txt/build_mscoco_data.runfiles/im2txt/im2txt/data/build_mscoco_data.py", line 283, in _process_image_files
sequence_example = _to_sequence_example(image, decoder, vocab)
File "/root/.cache/bazel/_bazel_root/5d96493270bd616258f6b3292edd375b/execroot/im2txt/bazel-out/local-fastbuild/bin/im2txt/build_mscoco_data.runfiles/im2txt/im2txt/data/build_mscoco_data.py", line 229, in _to_sequence_example
"image/data": _bytes_feature(encoded_image),
File "/root/.cache/bazel/_bazel_root/5d96493270bd616258f6b3292edd375b/execroot/im2txt/bazel-out/local-fastbuild/bin/im2txt/build_mscoco_data.runfiles/im2txt/im2txt/data/build_mscoco_data.py", line 191, in _bytes_feature
return tf.train.Feature(bytes_list=tf.train.BytesList(value=value.decode("utf-8").encode("utf-8")))
File "/usr/lib64/python2.7/encodings/utf_8.py", line 16, in decode
return codecs.utf_8_decode(input, errors, True)
UnicodeDecodeError: 'utf8' codec can't decode byte 0xff in position 0: invalid start byte

[taowenleon]

Are your images jpg format? The function bytes_feature(value): converts a jpg image to string. If your images contain png, which may be utf-16, it will throw errors.
UnicodeDecodeError: 'utf8' codec can't decode byte 0xff in position 0: invalid start byte
The 0xff is out of range utf-8.

[sboschang]

try

def _bytes_feature_list(values):
  """Wrapper for inserting a bytes FeatureList into a SequenceExample proto."""
  return tf.train.FeatureList(feature=[_bytes_feature(v.encode('utf-8')) for v in values])

[shahparth123]

@DaveyTao Images are same as MSCOCO dataset, I have changed only caption language. For English it is working fine.