Issue while training WordClassifier-CTC using IAM Dataset

[yasersakkaf]

---

InvalidArgumentError Traceback (most recent call last)
/data/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py in _do_call(self, fn, *args)
1360 try:
-> 1361 return fn(*args)
1362 except errors.OpError as e:

/data/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py in _run_fn(session, feed_dict, fetch_list, target_list, options, run_metadata)
1339 return tf_session.TF_Run(session, options, feed_dict, fetch_list,
-> 1340 target_list, status, run_metadata)
1341

/data/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/errors_impl.py in exit(self, type_arg, value_arg, traceback_arg)
515 compat.as_text(c_api.TF_Message(self.status.status)),
--> 516 c_api.TF_GetCode(self.status.status))
517 # Delete the underlying status object from memory otherwise it stays alive

InvalidArgumentError: Cannot parse tensor from proto: dtype: DT_UINT8
tensor_shape {
dim {
size: 1
}
}
tensor_content: "\000\000\000\000"

 [[Node: DMT/_239 = Const[dtype=DT_UINT8, value=<Invalid TensorProto: dtype: DT_UINT8 tensor_shape { dim { size: 1 } } tensor_content: "\000\000\000\000">, _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]

During handling of the above exception, another exception occurred:

InvalidArgumentError Traceback (most recent call last)
in ()
8 for i_batch in range(TRAIN_STEPS):
9 fd = train_iterator.next_feed(BATCH_SIZE)
---> 10 train_step.run(fd)
11
12 if i_batch % LOSS_ITER == 0:

/data/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/ops.py in run(self, feed_dict, session)
2283 none, the default session will be used.
2284 """
-> 2285 _run_using_default_session(self, feed_dict, self.graph, session)
2286
2287 _gradient_registry = registry.Registry("gradient")

/data/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/ops.py in _run_using_default_session(operation, feed_dict, graph, session)
4934 "the operation's graph is different from the session's "
4935 "graph.")
-> 4936 session.run(operation, feed_dict)
4937
4938

/data/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py in run(self, fetches, feed_dict, options, run_metadata)
903 try:
904 result = self._run(None, fetches, feed_dict, options_ptr,
--> 905 run_metadata_ptr)
906 if run_metadata:
907 proto_data = tf_session.TF_GetBuffer(run_metadata_ptr)

/data/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py in _run(self, handle, fetches, feed_dict, options, run_metadata)
1135 if final_fetches or final_targets or (handle and feed_dict_tensor):
1136 results = self._do_run(handle, final_targets, final_fetches,
-> 1137 feed_dict_tensor, options, run_metadata)
1138 else:
1139 results = []

/data/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py in _do_run(self, handle, target_list, fetch_list, feed_dict, options, run_metadata)
1353 if handle is None:
1354 return self._do_call(_run_fn, self._session, feeds, fetches, targets,
-> 1355 options, run_metadata)
1356 else:
1357 return self._do_call(_prun_fn, self._session, handle, feeds, fetches)

/data/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py in _do_call(self, fn, *args)
1372 except KeyError:
1373 pass
-> 1374 raise type(e)(node_def, op, message)
1375
1376 def _extend_graph(self):

InvalidArgumentError: Cannot parse tensor from proto: dtype: DT_UINT8
tensor_shape {
dim {
size: 1
}
}
tensor_content: "\000\000\000\000"

 [[Node: DMT/_239 = Const[dtype=DT_UINT8, value=<Invalid TensorProto: dtype: DT_UINT8 tensor_shape { dim { size: 1 } } tensor_content: "\000\000\000\000">, _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]

I don't quite understand why I am getting this error.

[Breta01]

I not sure what cause this error either. Can you test if this happens for all words or for only some of them?

Also make sure you change char_size variable to have size corresponding to length of IAM characters array.

[yasersakkaf]

I will test it for sure.
I didn't change the char_size variable. That maybe the issue.
Thanks for responding

[yasersakkaf]

I changed the char_size, but no progress.
The error seems to appear for all the images

[yasersakkaf]

I think it is related to tensorflow version.
In my local machine I have v1.4.0, it is working fine there.
In the GPU that I am using the version is 1.6.0. and I get the error.

[Breta01]

That's true, I also get error on TensorFlow 1.6.0+ (also on non-GPU version).
I don't know what is the breaking change. It would be great if you could investigate this further. I'm busy right now and I won't have a chance to get to it in next two weeks.