Lack of documentation lstm_object_detection

[Freephi]

System information

• What is the top-level directory of the model you are using: lstm_object_detection
• Have I written custom code (as opposed to using a stock example script provided in TensorFlow): no
• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Linux Ubuntu 18.04
• TensorFlow installed from (source or binary): source
• TensorFlow version (use command below): 1.12.0
• Bazel version (if compiling from source):
• CUDA/cuDNN version: 9.0
• GPU model and memory:
• Exact command to reproduce:

Describe the problem

This is NOT a bug issue, but a request for more documentation/information about the training process of the lstm_object_detection. I tired to find help on Stackoverflow, but now I hope for a response of the authors. It would help a lot if there would be a documentation on how to train or in particular on how to prepare the training data. A sample create_record.py file like the ones in object_detection would be great.

[bongmo]

Would you please tell me where can I get pretrained models of lstm object detection?

[ashkanee]

I am new to TensorFlow and I am trying to convert the ImageNet VID 2015 to TensorFlow record (tfrecord) in order to train the video object detection model here. The missing part is the way on how to convert the ImageNet VID 2015 into TensorFlow record (tfrecord).

The data set are extracted images, which each image represents a video frame. The data set can be found here under "Object detection from video (VID)". The "VID initial release snippets" includes the videos and "VID initial release" includes the extracted frames as JPEG images.

There is a code here to convert Pascal VOC data set to tfrecord, which has a similar structure to ImageNet VID 2015 data set. However, the ImageNet VID 2015 data set has the training and validation data and annotation in separate sub-folders and the labels in separate text files.

Probably one way could be adapting the tfrecord converter of Pascal VOC to ImagNet VID so it handles to scattered data in sub-folders and text files.

I would appreciate any help. Thank you!

Best regards,
Ashkan

[wrkgm]

Did anyone end up getting this to train at all?

[zhifanzhu]

A side question: does anyone have problem with equation (4) in the original paper?
I posted a question on stackoverflow as well.

[Aaronreb]

While running train.py from lstm_object_detection, I'm getting following errors:-

TypeError: Expected binary or unicode string, got <object_detection.core.matcher.Match object at 0x7f5b089379b0>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "lstm_object_detection/train.py", line 185, in
tf.app.run()
File "/home/kt-ml1/.local/lib/python3.6/site-packages/tensorflow/python/platform/app.py", line 125, in run
_sys.exit(main(argv))
File "lstm_object_detection/train.py", line 181, in main
worker_job_name, is_chief, FLAGS.train_dir)
File "/home/kt-ml1/models-master/models-master/research/lstm_object_detection/trainer.py", line 293, in train
clones = model_deploy.create_clones(deploy_config, model_fn, [input_queue])
File "/home/kt-ml1/models-master/models-master/research/slim/deployment/model_deploy.py", line 193, in create_clones
outputs = model_fn(*args, **kwargs)
File "/home/kt-ml1/models-master/models-master/research/lstm_object_detection/trainer.py", line 174, in _create_losses
losses_dict = detection_model.loss(prediction_dict, true_image_shapes)
File "/home/kt-ml1/models-master/models-master/research/lstm_object_detection/meta_architectures/lstm_ssd_meta_arch.py", line 165, in loss
match_list = [matcher.Match(match) for match in tf.unstack(batch_match)]
File "/home/kt-ml1/.local/lib/python3.6/site-packages/tensorflow/python/ops/array_ops.py", line 1149, in unstack
value = ops.convert_to_tensor(value)
File "/home/kt-ml1/.local/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1039, in convert_to_tensor
return convert_to_tensor_v2(value, dtype, preferred_dtype, name)
File "/home/kt-ml1/.local/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1097, in convert_to_tensor_v2
as_ref=False)
File "/home/kt-ml1/.local/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1175, in internal_convert_to_tensor
ret = conversion_func(value, dtype=dtype, name=name, as_ref=as_ref)
File "/home/kt-ml1/.local/lib/python3.6/site-packages/tensorflow/python/framework/constant_op.py", line 304, in _constant_tensor_conversion_function
return constant(v, dtype=dtype, name=name)
File "/home/kt-ml1/.local/lib/python3.6/site-packages/tensorflow/python/framework/constant_op.py", line 245, in constant
allow_broadcast=True)
File "/home/kt-ml1/.local/lib/python3.6/site-packages/tensorflow/python/framework/constant_op.py", line 283, in _constant_impl
allow_broadcast=allow_broadcast))
File "/home/kt-ml1/.local/lib/python3.6/site-packages/tensorflow/python/framework/tensor_util.py", line 562, in make_tensor_proto
"supported type." % (type(values), values))
TypeError: Failed to convert object of type <class 'list'> to Tensor. Contents: [<object_detection.core.matcher.Match object at 0x7f5b089379b0>, <object_detection.core.matcher.Match object at 0x7f5b087b75c0>, <object_detection.core.matcher.Match object at 0x7f5b08626fd0>, <object_detection.core.matcher.Match object at 0x7f5b084a9b00>, <object_detection.core.matcher.Match object at 0x7f5b083302b0>, <object_detection.core.matcher.Match object at 0x7f5b0819a6a0>, <object_detection.core.matcher.Match object at 0x7f5b03fe2710>, <object_detection.core.matcher.Match object at 0x7f5b03e672b0>, <object_detection.core.matcher.Match object at 0x7f5b03cd36a0>, <object_detection.core.matcher.Match object at 0x7f5b03b53710>, <object_detection.core.matcher.Match object at 0x7f5b039d92b0>, <object_detection.core.matcher.Match object at 0x7f5b038456a0>, <object_detection.core.matcher.Match object at 0x7f5b036ca710>, <object_detection.core.matcher.Match object at 0x7f5b0354e2b0>, <object_detection.core.matcher.Match object at 0x7f5b034396a0>, <object_detection.core.matcher.Match object at 0x7f5b032bb710>, <object_detection.core.matcher.Match object at 0x7f5b0313f2b0>, <object_detection.core.matcher.Match object at 0x7f5b02faf6a0>, <object_detection.core.matcher.Match object at 0x7f5b02e315f8>, <object_detection.core.matcher.Match object at 0x7f5b02cb32b0>, <object_detection.core.matcher.Match object at 0x7f5b02b226a0>, <object_detection.core.matcher.Match object at 0x7f5b029a5710>, <object_detection.core.matcher.Match object at 0x7f5b028282b0>, <object_detection.core.matcher.Match object at 0x7f5b026936a0>, <object_detection.core.matcher.Match object at 0x7f5b02518710>, <object_detection.core.matcher.Match object at 0x7f5b0239c2b0>, <object_detection.core.matcher.Match object at 0x7f5b022096a0>, <object_detection.core.matcher.Match object at 0x7f5b0208f710>, <object_detection.core.matcher.Match object at 0x7f5b01f0f2b0>, <object_detection.core.matcher.Match object at 0x7f5b01dfc6a0>, <object_detection.core.matcher.Match object at 0x7f5b01c03710>, <object_detection.core.matcher.Match object at 0x7f5b01a842b0>]. Consider casting elements to a supported type.

PS- I have commented data augmentation option in the config file because it was giving me groundtruth_error
the problem seems to be coming from tensors in the file seq_dataset_builder.py, as the tensors are empty.

Any help would be appreciated

[SonaHarutyunyan]

Comment out the line 165 from lstm_object_detection/meta_architectures/lstm_ssd_meta_arch.py
#match_list = [matcher.Match(match) for match in tf.unstack(batch_match)]
Instead use the following code-
match_list=batch_match

I found the solution here:
#6777

[SonaHarutyunyan]

About groundtruth_weights error:

You can comment only lines 3212-3214 in object_detection/core/preprocessor.py and not the whole data augmentation option.
They are the following lines:

if include_label_weights:
groundtruth_label_weights = (
fields.InputDataFields.groundtruth_weights)