models/research/object_detection requires Tensorflow 1.5

[noahstier]

System information

• What is the top-level directory of the model you are using:
  models/research/object_detection
• Have I written custom code (as opposed to using a stock example script provided in TensorFlow):
  no
• OS Platform and Distribution (e.g., Linux Ubuntu 16.04):
  Linux Ubuntu 16.04
• TensorFlow installed from (source or binary):
  binary: "conda install tensorflow-gpu=1.4"
• TensorFlow version (use command below):
  1.4
• CUDA/cuDNN version:
  cuda 8.0-3, cudnn 7.05
• GPU model and memory:
  Nvidia GTX 1080
• Exact command to reproduce:
  from the example in the object_detection readme:
  python object_detection/train.py
  --logtostderr
  --pipeline_config_path=${PATH_TO_YOUR_PIPELINE_CONFIG}
  --train_dir=${PATH_TO_TRAIN_DIR}

Describe the problem

A recent commit to models/research/object_detection introduced a dependency on the Tensorflow 1.5 release. However, the models/research README claims "Currently, the models are compatible with TensorFlow 1.0 or later". The most recent tensorflow release available through Anaconda is 1.4, so this change forces me to compile from source, use a different package manager, or find a conda package outside the official repo, which are not horrible options but still inconvenient.

Here is a link to the commit diff. It calls tf.contrib.data.parallel_interleave which is not present in Tensorflow 1.4.
8f93258#diff-9e49d577701a72acf6f96e468190b191R129

If the 1.5 dependency is deemed acceptable, then I think it should be indicated either in the models/research readme or the models/research/object_detection readme.

[noahstier]

This issue also references a dependency on the 1.5 release:
#3421

Correction--that issue references a dependency on pre-1.5 releases

[asimshankar]

@jch1 @tombstone @derekjchow @jesu9 @dreamdragon : Perhaps the documentation needs to be updated?

[getaleks]

Is there an update on this issue? Or has anybody found a work around to enable training?
Unfortunately, I am unable to update to Tensorflow 1.5 at the moment.

Is there an earlier commit that we can revert to before this issue arose?

[noahstier]

This is the last commit before those changes were merged, I haven't tested it at that point though: d3143cb

[getaleks]

I tested d3143cb and unfortunately got the BackupHandler error:

in __init__ label_handler = slim_example_decoder.BackupHandler( AttributeError: 'module' object has no attribute 'BackupHandler'

This error has already been described here: #2653.

[TyrionChou]

In nvidia-docker，tensorflow1.4，cuda8，how to solve this problem？update tensorflow and cuda？

[noahstier]

Updating tensorflow to 1.5 or 1.6 should solve it without requiring you to update cuda.

[getaleks]

Yes, but doesn’t Tensorflow 1.5 require CUDA 9 with cuDNN 7? So you can’t install Tensorflow 1.5 without CUDA 9?

[noahstier]

Tensorflow 1.7 requires cuda 9. I've built tf 1.5 with cuda 8 and I believe you can do the same with tf 1.6

[getaleks]

Oh ok, that sounds great if that worked! Is the Object Detection API working for you with tf1.5? Can you train a SSD model?

[davidblumntcgeo]

I'm running CUDA Toolkit 8.0, cudnn 6.0. If I run tensorflow-gpu 1.4, I get: tf.contrib.data.parallel_interleave(
AttributeError: 'module' object has no attribute 'parallel_interleave'
as described by @noahstier .

If I instead run tensorflow-gpu 1.5, I get
ImportError: libcublas.so.9.0: cannot open shared object file: No such file or directory
(which is Tensorflow Issue #15604 and #17629)

Many of the commenters in those two issues are under the impression that TF-gpu 1.5 requires CUDA Toolkit 9.0. @noahstier , did you do anything special to make TF 1.5 work with CUDA 8, which I believe is required for the Object Detector API?

Any plans to make restore OD API compatibility with TF-gpu 1.4?

[opalby]

Settings of my computer are the same as you,and I have the same problem with you.Do you have a solution? @davidblumntcgeo

[davidblumntcgeo]

I don’t, sorry. However, Siddarth Dash just posted a solution to tensorflow issue #17629, which I haven’t had time yet to try. If it works, that should address at least my current problem keeping me from running TF-gpu 1.5

[lwohlhart]

Hi everyone,
I finally got it to run on tensorflow-gpu 1.4.1

I replaced the parallel_interleave object_detection/builders/dataset_builder.py line 75

records_dataset = filename_dataset.interleave(file_read_func,
                                              cycle_length=config.num_readers,
                                              block_length=config.read_block_length)

as suggested in a comment by levinwil

and in object_detection/inputs.py line 413

iterator = dataset.make_initializable_iterator()
tf.add_to_collection(tf.GraphKeys.TABLE_INITIALIZERS, iterator.initializer)
return iterator.get_next()

to make it backward compatible

[roitmaster]

In line 413 there is a return dataset
do you mean I need to delete it? and add your code or just to paste it before or after

[tensorflowbutler]

Hi There,
We are checking to see if you still need help on this, as this seems to be considerably old issue. Please update this issue with the latest information, code snippet to reproduce your issue and error you are seeing.
If we don't hear from you in the next 7 days, this issue will be closed automatically. If you don't need help on this issue any more, please consider closing this.