Getting RuntimeError: main thread is not in main loop when running model_main.py

[szm-R]

System information

• What is the top-level directory of the model you are using:
  Well, I'm trying to fine-tune ssd_mobilenet_v1_fpn_shared_box_predictor_640x640_coco14_sync model on my own custom dataset.
• Have I written custom code (as opposed to using a stock example script provided in TensorFlow):
  No
• OS Platform and Distribution (e.g., Linux Ubuntu 16.04):
  Ubuntu 18.04
• TensorFlow installed from (source or binary):
  From source
• TensorFlow version (use command below):
  1.9.0
• Bazel version (if compiling from source):
  0.15.0
• CUDA/cuDNN version:
  9.2/7.1
• GPU model and memory:
  Geforce GTX 850M 4GB
• Exact command to reproduce:
  python3 object_detection/model_main.py --pipeline_config_path=/home/szm/Work/TensorFlow/Models/ObjectDetection/ssd_mobilenet_v1_fpn_DETRAC_516x292_4C_RA1/ssd_mobilenet_v1_fpn.config --model_dir=/home/szm/Work/TensorFlow/Models/ObjectDetection/ssd_mobilenet_v1_fpn_DETRAC_516x292_4C_RA1 --alsologtostderr

Describe the problem

OK! I've been using an older release of Object Detection API (from the README it seems to be February 9, 2018 release) and have trained multiple models with it. Recently, I decided to use the July 13, 2018 release so as to be able to train newly added models like ssd_mobilenet_v1_fpn.

First, I finished the steps mentioned in the installation with model_builder_test.py resulting in:

Ran 18 tests in 0.095s
OK

Then I tried to train my model using the command I mentioned above, which is what I found in Running locally. I'm using the provided config file ssd_mobilenet_v1_fpn_shared_box_predictor_640x640_coco14_sync.config with minor changes in the following parts:

num_classes (changed number)
fixed_shape_resizer (changed number)
matched_threshold (changed number)
unmatched_threshold (changed number)
iou_threshold (changed number)
batch_size (changed number)
fine_tune_checkpoint_type: 'detection' (added)
num_steps (commented)
learning_rate_base (changed number)
warmup_learning_rate (changed number)

The first problem is that the codes (or at least some of them) seem to have problems running with python3 (which is the python I'm using, It's actually version 3.6.5). Here are the errors I got from running model_main.py and their solutions:

File "/.../models-master/research/object_detection/utils/object_detection_evaluation.py", line 842
    print 'Scores and tpfp per class label: {}'.format(class_index)
    solution: print() instead of print 

File "/.../models-master/research/object_detection/models/feature_map_generators.py", line 225, in fpn_top_down_feature_maps
    reversed(zip(output_feature_map_keys, output_feature_maps_list)))
TypeError: 'zip' object is not reversible
	solution: reversed(list(zip(output_feature_map_keys, output_feature_maps_list))))

File "/.../models-master/research/object_detection/model_lib.py", line 282, in model_fn
    losses = [loss_tensor for loss_tensor in losses_dict.itervalues()]
AttributeError: 'dict' object has no attribute 'itervalues'

	solution: using values() instead of itervalues()

I really don't understand why should such errors exist? Is the API meant to be used with python 2 or it's that I'm missing something?!

Anyway, the actual problem is the one mentioned in the title. That's the error I get after fixing the above problems (I'm not exactly sure about the fixes though!) and I don't know how to deal with it. It's worth to note that I also repeated the above steps on an Ubuntu 16.04 with CUDA 9.0,tensorflow 1.7.0 and Geforce GTX 1070 8GB and got the same errors.

Source code / logs

Here's the full Traceback:

2018-07-16 12:37:39.541264: W tensorflow/core/framework/op_kernel.cc:1263] Unknown: RuntimeError: main thread is not in main loop
Traceback (most recent call last):

  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/script_ops.py", line 206, in __call__
    ret = func(*args)

  File "/home/szm/Work/TensorFlow/Codes/models-master/research/object_detection/utils/visualization_utils.py", line 693, in cdf_plot
    fig = plt.figure(frameon=False)

  File "/home/szm/.local/lib/python3.6/site-packages/matplotlib/pyplot.py", line 548, in figure
    **kwargs)

  File "/home/szm/.local/lib/python3.6/site-packages/matplotlib/backend_bases.py", line 161, in new_figure_manager
    return cls.new_figure_manager_given_figure(num, fig)

  File "/home/szm/.local/lib/python3.6/site-packages/matplotlib/backends/_backend_tk.py", line 1053, in new_figure_manager_given_figure
    icon_img = Tk.PhotoImage(file=icon_fname)

  File "/usr/lib/python3.6/tkinter/__init__.py", line 3542, in __init__
    Image.__init__(self, 'photo', name, cnf, master, **kw)

  File "/usr/lib/python3.6/tkinter/__init__.py", line 3498, in __init__
    self.tk.call(('image', 'create', imgtype, name,) + options)

RuntimeError: main thread is not in main loop


Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py", line 1278, in _do_call
    return fn(*args)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py", line 1263, in _run_fn
    options, feed_dict, fetch_list, target_list, run_metadata)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py", line 1350, in _call_tf_sessionrun
    run_metadata)
tensorflow.python.framework.errors_impl.UnknownError: RuntimeError: main thread is not in main loop
Traceback (most recent call last):

  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/script_ops.py", line 206, in __call__
    ret = func(*args)

  File "/home/szm/Work/TensorFlow/Codes/models-master/research/object_detection/utils/visualization_utils.py", line 693, in cdf_plot
    fig = plt.figure(frameon=False)

  File "/home/szm/.local/lib/python3.6/site-packages/matplotlib/pyplot.py", line 548, in figure
    **kwargs)

  File "/home/szm/.local/lib/python3.6/site-packages/matplotlib/backend_bases.py", line 161, in new_figure_manager
    return cls.new_figure_manager_given_figure(num, fig)

  File "/home/szm/.local/lib/python3.6/site-packages/matplotlib/backends/_backend_tk.py", line 1053, in new_figure_manager_given_figure
    icon_img = Tk.PhotoImage(file=icon_fname)

  File "/usr/lib/python3.6/tkinter/__init__.py", line 3542, in __init__
    Image.__init__(self, 'photo', name, cnf, master, **kw)

  File "/usr/lib/python3.6/tkinter/__init__.py", line 3498, in __init__
    self.tk.call(('image', 'create', imgtype, name,) + options)

RuntimeError: main thread is not in main loop


	 [[Node: Loss/PyFunc_1 = PyFunc[Tin=[DT_FLOAT], Tout=[DT_UINT8], token="pyfunc_1", _device="/job:localhost/replica:0/task:0/device:CPU:0"](Loss/Squeeze_1/_3561)]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "object_detection/model_main.py", line 101, in <module>
    tf.app.run()
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/platform/app.py", line 125, in run
    _sys.exit(main(argv))
  File "object_detection/model_main.py", line 97, in main
    tf.estimator.train_and_evaluate(estimator, train_spec, eval_specs[0])
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/estimator/training.py", line 451, in train_and_evaluate
    return executor.run()
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/estimator/training.py", line 590, in run
    return self.run_local()
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/estimator/training.py", line 691, in run_local
    saving_listeners=saving_listeners)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/estimator/estimator.py", line 376, in train
    loss = self._train_model(input_fn, hooks, saving_listeners)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/estimator/estimator.py", line 1143, in _train_model
    return self._train_model_default(input_fn, hooks, saving_listeners)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/estimator/estimator.py", line 1171, in _train_model_default
    saving_listeners)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/estimator/estimator.py", line 1449, in _train_with_estimator_spec
    _, loss = mon_sess.run([estimator_spec.train_op, estimator_spec.loss])
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/training/monitored_session.py", line 583, in run
    run_metadata=run_metadata)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/training/monitored_session.py", line 1059, in run
    run_metadata=run_metadata)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/training/monitored_session.py", line 1150, in run
    raise six.reraise(*original_exc_info)
  File "/usr/lib/python3/dist-packages/six.py", line 693, in reraise
    raise value
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/training/monitored_session.py", line 1135, in run
    return self._sess.run(*args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/training/monitored_session.py", line 1207, in run
    run_metadata=run_metadata)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/training/monitored_session.py", line 987, in run
    return self._sess.run(*args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py", line 877, in run
    run_metadata_ptr)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py", line 1100, in _run
    feed_dict_tensor, options, run_metadata)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py", line 1272, in _do_run
    run_metadata)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py", line 1291, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.UnknownError: RuntimeError: main thread is not in main loop
Traceback (most recent call last):

  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/script_ops.py", line 206, in __call__
    ret = func(*args)

  File "/home/szm/Work/TensorFlow/Codes/models-master/research/object_detection/utils/visualization_utils.py", line 693, in cdf_plot
    fig = plt.figure(frameon=False)

  File "/home/szm/.local/lib/python3.6/site-packages/matplotlib/pyplot.py", line 548, in figure
    **kwargs)

  File "/home/szm/.local/lib/python3.6/site-packages/matplotlib/backend_bases.py", line 161, in new_figure_manager
    return cls.new_figure_manager_given_figure(num, fig)

  File "/home/szm/.local/lib/python3.6/site-packages/matplotlib/backends/_backend_tk.py", line 1053, in new_figure_manager_given_figure
    icon_img = Tk.PhotoImage(file=icon_fname)

  File "/usr/lib/python3.6/tkinter/__init__.py", line 3542, in __init__
    Image.__init__(self, 'photo', name, cnf, master, **kw)

  File "/usr/lib/python3.6/tkinter/__init__.py", line 3498, in __init__
    self.tk.call(('image', 'create', imgtype, name,) + options)

RuntimeError: main thread is not in main loop


	 [[Node: Loss/PyFunc_1 = PyFunc[Tin=[DT_FLOAT], Tout=[DT_UINT8], token="pyfunc_1", _device="/job:localhost/replica:0/task:0/device:CPU:0"](Loss/Squeeze_1/_3561)]]

Caused by op 'Loss/PyFunc_1', defined at:
  File "object_detection/model_main.py", line 101, in <module>
    tf.app.run()
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/platform/app.py", line 125, in run
    _sys.exit(main(argv))
  File "object_detection/model_main.py", line 97, in main
    tf.estimator.train_and_evaluate(estimator, train_spec, eval_specs[0])
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/estimator/training.py", line 451, in train_and_evaluate
    return executor.run()
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/estimator/training.py", line 590, in run
    return self.run_local()
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/estimator/training.py", line 691, in run_local
    saving_listeners=saving_listeners)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/estimator/estimator.py", line 376, in train
    loss = self._train_model(input_fn, hooks, saving_listeners)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/estimator/estimator.py", line 1143, in _train_model
    return self._train_model_default(input_fn, hooks, saving_listeners)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/estimator/estimator.py", line 1168, in _train_model_default
    features, labels, model_fn_lib.ModeKeys.TRAIN, self.config)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/estimator/estimator.py", line 1131, in _call_model_fn
    model_fn_results = self._model_fn(features=features, **kwargs)
  File "/home/szm/Work/TensorFlow/Codes/models-master/research/object_detection/model_lib.py", line 281, in model_fn
    prediction_dict, features[fields.InputDataFields.true_image_shape])
  File "/home/szm/Work/TensorFlow/Codes/models-master/research/object_detection/meta_architectures/ssd_meta_arch.py", line 597, in loss
    flattened_class_ids, flattened_classification_losses)
  File "/home/szm/Work/TensorFlow/Codes/models-master/research/object_detection/meta_architectures/ssd_meta_arch.py", line 660, in _summarize_anchor_classification_loss
    'NegativeAnchorLossCDF')
  File "/home/szm/Work/TensorFlow/Codes/models-master/research/object_detection/utils/visualization_utils.py", line 703, in add_cdf_image_summary
    cdf_plot = tf.py_func(cdf_plot, [values], tf.uint8)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/script_ops.py", line 456, in py_func
    func=func, inp=inp, Tout=Tout, stateful=stateful, eager=False, name=name)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/script_ops.py", line 281, in _internal_py_func
    input=inp, token=token, Tout=Tout, name=name)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/gen_script_ops.py", line 128, in py_func
    "PyFunc", input=input, token=token, Tout=Tout, name=name)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper
    op_def=op_def)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/util/deprecation.py", line 432, in new_func
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/ops.py", line 3212, in create_op
    op_def=op_def)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/ops.py", line 1702, in __init__
    self._traceback = self._graph._extract_stack()  # pylint: disable=protected-access

UnknownError (see above for traceback): RuntimeError: main thread is not in main loop
Traceback (most recent call last):

  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/script_ops.py", line 206, in __call__
    ret = func(*args)

  File "/home/szm/Work/TensorFlow/Codes/models-master/research/object_detection/utils/visualization_utils.py", line 693, in cdf_plot
    fig = plt.figure(frameon=False)

  File "/home/szm/.local/lib/python3.6/site-packages/matplotlib/pyplot.py", line 548, in figure
    **kwargs)

  File "/home/szm/.local/lib/python3.6/site-packages/matplotlib/backend_bases.py", line 161, in new_figure_manager
    return cls.new_figure_manager_given_figure(num, fig)

  File "/home/szm/.local/lib/python3.6/site-packages/matplotlib/backends/_backend_tk.py", line 1053, in new_figure_manager_given_figure
    icon_img = Tk.PhotoImage(file=icon_fname)

  File "/usr/lib/python3.6/tkinter/__init__.py", line 3542, in __init__
    Image.__init__(self, 'photo', name, cnf, master, **kw)

  File "/usr/lib/python3.6/tkinter/__init__.py", line 3498, in __init__
    self.tk.call(('image', 'create', imgtype, name,) + options)

RuntimeError: main thread is not in main loop


	 [[Node: Loss/PyFunc_1 = PyFunc[Tin=[DT_FLOAT], Tout=[DT_UINT8], token="pyfunc_1", _device="/job:localhost/replica:0/task:0/device:CPU:0"](Loss/Squeeze_1/_3561)]]

Error in atexit._run_exitfuncs:
Traceback (most recent call last):
  File "/home/szm/.local/lib/python3.6/site-packages/matplotlib/_pylab_helpers.py", line 78, in destroy_all
    manager.destroy()
  File "/home/szm/.local/lib/python3.6/site-packages/matplotlib/backends/_backend_tk.py", line 558, in destroy
    self.window.destroy()
  File "/usr/lib/python3.6/tkinter/__init__.py", line 2058, in destroy
    for c in list(self.children.values()): c.destroy()
  File "/usr/lib/python3.6/tkinter/__init__.py", line 2302, in destroy
    self.tk.call('destroy', self._w)
RuntimeError: main thread is not in main loop

[derekjchow]

Thank you for the report.

The Tensorflow Object Detection API should work with Python 2 and 3. We'll have a patch out to fix these issues soon.

[szm-R]

Thank you, you mean the runtime error is also related to incompatibility with python3? How can I know that the fix has been added? Should I check the release section often?

[marcbelmont]

The following change did the trick for me:

modified   object_detection/eval_util.py
@@ -22,4 +22,5 @@ import numpy as np
 import tensorflow as tf
 
+from object_detection.utils import visualization_utils as vis_utils
 from object_detection.core import box_list
 from object_detection.core import box_list_ops
@@ -29,5 +30,4 @@ from object_detection.metrics import coco_evaluation
 from object_detection.utils import label_map_util
 from object_detection.utils import ops
-from object_detection.utils import visualization_utils as vis_utils

[szm-R]

Sorry, I don't get it! What do you mean by @@ and + - marks?

[marcbelmont]

It's a diff. It means remove the import and then move it to the top.

[szm-R]

OK, so what about the @@ -22,4 +22,5 @@ and @@ -29,5 +30,4 @@?

[gar1t]

@derekjchow I see pylint directives in the object_detection code - I presume you're linting on Python 2. Would it be possible to lint on Python 3?

Also - and my apologies if these are readily available (I couldn't find them) - are there docs that explain Google's release process wrt object_detection? It seems that you maintain a private repo and collect a large number of changes, which are then applied to the public repo in discrete sync operations. This makes it tough for the community to make even simple contributions (like fixing Python 3 breakage) as there's little visibility into what Google will be merging into the public repo.

Waiting for Google to patch these issues with little-to-no visibility into that process or timeline I think makes this library challenging to work with. I personally would be delighted to fix these directly (and I certainly must do that in my own fork) but knowing that they're already fixed, but in an inaccessible repo, forces me to either suffer the breakage, revert to some earlier commit (which itself may have issues) or to work with my own fork knowing I'll have merge conflicts down the road.

I understand there's often a need to maintain private repos and routinely sync, but I'm wondering if this is outlined somewhere as a policy? Is there any discussion of making the public repo the sole repo for object_detection?

Again, my apologies if this is covering already-documented topics!

[MichaelX99]

One line 703 of utils/visualization_utils.py change cdf_plot=tf.py_func(cdf_plot, [values], tf.uint8) to cdf_plot=tf.py_func(cdf_plot, [values], tf.float32)

this will eliminate the errors, or at least it did for me

[szm-R]

Hello everyone,
@MichaelX99 I tried your solution but got this error:

File "/home/szm/Work/TensorFlow/Codes/models-master/research/object_detection/utils/visualization_utils.py", line 704, in add_cdf_image_summary
    cdf_plot=tf.py_func(cdf_plot, [values], tf.float32)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/script_ops.py", line 456, in py_func
    func=func, inp=inp, Tout=Tout, stateful=stateful, eager=False, name=name)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/script_ops.py", line 281, in _internal_py_func
    input=inp, token=token, Tout=Tout, name=name)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/gen_script_ops.py", line 128, in py_func
    "PyFunc", input=input, token=token, Tout=Tout, name=name)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper
    op_def=op_def)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/util/deprecation.py", line 432, in new_func
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/ops.py", line 3212, in create_op
    op_def=op_def)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/ops.py", line 1702, in __init__
    self._traceback = self._graph._extract_stack()  # pylint: disable=protected-access

Then I turned the line back to what it was and tried @marcbelmont 's solution (still don't have any idea what the @ means here I just moved from object_detection.utils import visualization_utils as vis_utils to the top). After getting an error about dict object and fixing it (AttributeError: 'dict' object has no attribute 'iteritems' in file model_lib.py line 391), I get this:

File "/home/szm/Paper/TensorFlow/Codes/models-master/research/object_detection/metrics/coco_tools.py", line 118, in LoadAnnotations
    results.dataset['categories'] = copy.deepcopy(self.dataset['categories'])

  File "/usr/lib/python3.5/copy.py", line 174, in deepcopy
    rv = reductor(4)

TypeError: can't pickle dict_values objects


	 [[Node: PyFunc_1 = PyFunc[Tin=[], Tout=[DT_FLOAT], token="pyfunc_7", _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]
	 [[Node: map/while/TensorArrayWrite/TensorArrayWriteV3/_2493 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device_incarnation=1, tensor_name="edge_2215_map/while/TensorArrayWrite/TensorArrayWriteV3", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:0"](^_cloopmap/while/NextIteration_1/_2278)]]

The full stack trace is huge, so I don't send all of it (tell me if I should). Now the point is I just want to train the model I don't want TF to perform any evaluation for me (I have a separate code and tool kit for that). So, is there any way to disable evaluation totally? Because it seems to be the source of the problem, unless of course, it's just because it's being done before training and the training process would give the same error as well!

Another thing is, it took ages for the command to get to the error I mentioned! I'm running the code on a Geforce GTX 1070 and usually it's fast enough. So, what's the problem?! Is it perhaps trying to run evaluation on my whole validation set from the beginning or it's somthing else?!

[szm-R]

Hey guys! I'm still hoping someone would tell me how to fix this problem! Any ideas?!

[HongweiXie]

Got the same error!!!

[marcbelmont]

Have you tried with the latest commits?

[szm-R]

OK, today I downloaded the whole models folder and tried to run the same code as the one I mentioned in the actual question, meaning:
python3 object_detection/model_main.py --pipeline_config_path=/home/szm/Work/TensorFlow/Models/ObjectDetection/ssd_mobilenet_v1_fpn_DETRAC_516x292_4C_RA1/ssd_mobilenet_v1_fpn.config --model_dir=/home/szm/Work/TensorFlow/Models/ObjectDetection/ssd_mobilenet_v1_fpn_DETRAC_516x292_4C_RA1 --alsologtostderr

After fixing an error in multiscale_grid_anchor_generator.py file (NameError: name 'xrange' is not defined, line 61), I got the same run time error again:

Error in atexit._run_exitfuncs:

Traceback (most recent call last):
  File "/home/szm/.local/lib/python3.6/site-packages/matplotlib/_pylab_helpers.py", line 78, in destroy_all
    manager.destroy()
  File "/home/szm/.local/lib/python3.6/site-packages/matplotlib/backends/_backend_tk.py", line 558, in destroy
    self.window.destroy()
  File "/usr/lib/python3.6/tkinter/__init__.py", line 2058, in destroy
    for c in list(self.children.values()): c.destroy()
  File "/usr/lib/python3.6/tkinter/__init__.py", line 2302, in destroy
    self.tk.call('destroy', self._w)
RuntimeError: main thread is not in main loop

Again the full Traceback is huge, I don't know if I should send it or not.

[BorisDeletic]

I have the same issue, any fix would be appreciated

[MichaelX99]

@szm2015, I believe your error about the pickle dictionary object may be due to a python version problem? I was having dictionary problems when I was running python3 and not python2 but I am not sure if that will be your specific problem.

Also for the RuntimeError: main thread thing, I believe that this is really just an error saying that an error happened somewhere in the code. I think the only way to solve the problem is to go through the full traceback and comment out whatever line it is that is causing you pain... My fix for py_func was for a plot in tensorboard so not really necessary for training even though it was causing the program to crash

[xtianhb]

It is an python2 vs python3 error. I fixed it by changing file:
\models\research\object_detection\anchor_generators\multiscale_grid_anchor_generator.py

-- for scale in xrange(scales_per_octave)]
++ for scale in range(scales_per_octave)]

[szm-R]

@MichaelX99 and @xtianhb I think we already established the fact that it is a python2 vs. python3 problem and @xtianhb I also did the fix you suggest but that only fixes the initial error and not the pain-in-the-neck runtime one!!! :) The point is why should such errors exist in the first place?! Shouldn't the object detection API be compatible with python3?!
@MichaelX99 going over the full stack trace is probably a working solution (albeit painful!!! as it's so long!). The thing is, I want to disable the evaluation part entirely, as I mentioned before. Is there a way to do that?

[szm-R]

Hello again. I did as @MichaelX99 suggested and went over the full Traceback. There I found the source of the problem, line 703 and 704 of file visualization_utils.py:

cdf_plot = tf.py_func(cdf_plot, [values], tf.uint8)
tf.summary.image(name, cdf_plot)

Which I commented out and so the runtime problem was resolved. The code went on (of course after an error TypeError: can't pickle dict_values objects which was resolved by adding list() to line 385 of model_lib.py as suggested in #4780, again python3 incompatibility). I feel like an idiot now as the fix was so simple!!! :|

However, I still don't want the code to perform any evaluation whatsoever. Now it seems to do nothing but evaluation! any suggestions?! I feel like this is something very simple and I'm missing some very obvious points!

This is the output for running the code for a couple of minutes on a small dataset:

import matplotlib; matplotlib.use('Agg')  # pylint: disable=multiple-statements
WARNING:tensorflow:Estimator's model_fn (<function create_model_fn.<locals>.model_fn at 0x7faa957ea1e0>) includes params argument, but params are not passed to Estimator.
WARNING:tensorflow:num_readers has been reduced to 1 to match input file shards.
WARNING:tensorflow:From /home/szm/Work/TensorFlow/Codes/models-master/research/object_detection/core/preprocessor.py:1205: calling squeeze (from tensorflow.python.ops.array_ops) with squeeze_dims is deprecated and will be removed in a future version.
Instructions for updating:
Use the `axis` argument instead
WARNING:tensorflow:From /home/szm/Work/TensorFlow/Codes/models-master/research/object_detection/builders/dataset_builder.py:146: batch_and_drop_remainder (from tensorflow.contrib.data.python.ops.batching) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.data.Dataset.batch(..., drop_remainder=True)`.
2018-08-11 18:18:22.295782: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:897] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2018-08-11 18:18:22.296720: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1404] Found device 0 with properties: 
name: GeForce GTX 850M major: 5 minor: 0 memoryClockRate(GHz): 0.9015
pciBusID: 0000:0a:00.0
totalMemory: 3.95GiB freeMemory: 3.53GiB
2018-08-11 18:18:22.296740: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1483] Adding visible gpu devices: 0
2018-08-11 18:18:22.661449: I tensorflow/core/common_runtime/gpu/gpu_device.cc:964] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-08-11 18:18:22.661496: I tensorflow/core/common_runtime/gpu/gpu_device.cc:970]      0 
2018-08-11 18:18:22.661505: I tensorflow/core/common_runtime/gpu/gpu_device.cc:983] 0:   N 
2018-08-11 18:18:22.661696: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1096] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 3255 MB memory) -> physical GPU (device: 0, name: GeForce GTX 850M, pci bus id: 0000:0a:00.0, compute capability: 5.0)
2018-08-11 18:18:22.661888: E tensorflow/core/common_runtime/gpu/gpu_device.cc:228] Illegal GPUOptions.experimental.num_dev_to_dev_copy_streams=0 set to 1 instead.
2018-08-11 18:28:37.835099: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1483] Adding visible gpu devices: 0
2018-08-11 18:28:37.835150: I tensorflow/core/common_runtime/gpu/gpu_device.cc:964] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-08-11 18:28:37.835160: I tensorflow/core/common_runtime/gpu/gpu_device.cc:970]      0 
2018-08-11 18:28:37.835169: I tensorflow/core/common_runtime/gpu/gpu_device.cc:983] 0:   N 
2018-08-11 18:28:37.835294: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1096] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 3255 MB memory) -> physical GPU (device: 0, name: GeForce GTX 850M, pci bus id: 0000:0a:00.0, compute capability: 5.0)
creating index...
index created!
creating index...
index created!
Running per image evaluation...
Evaluate annotation type *bbox*
DONE (t=2.25s).
Accumulating evaluation results...
DONE (t=0.59s).
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.440
 Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.813
 Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = 0.422
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = -1.000
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.486
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.166
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  1 ] = 0.428
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 10 ] = 0.603
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.632
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = -1.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.636
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.605
2018-08-11 18:38:38.630020: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1483] Adding visible gpu devices: 0
2018-08-11 18:38:38.630090: I tensorflow/core/common_runtime/gpu/gpu_device.cc:964] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-08-11 18:38:38.630106: I tensorflow/core/common_runtime/gpu/gpu_device.cc:970]      0 
2018-08-11 18:38:38.630118: I tensorflow/core/common_runtime/gpu/gpu_device.cc:983] 0:   N 
2018-08-11 18:38:38.630289: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1096] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 3255 MB memory) -> physical GPU (device: 0, name: GeForce GTX 850M, pci bus id: 0000:0a:00.0, compute capability: 5.0)
creating index...
index created!
creating index...
index created!
Running per image evaluation...
Evaluate annotation type *bbox*
DONE (t=3.22s).
Accumulating evaluation results...
DONE (t=0.45s).
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.028
 Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.069
 Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = 0.019
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = -1.000
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.192
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.001
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  1 ] = 0.005
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 10 ] = 0.245
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.442
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = -1.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.446
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.418
2018-08-11 18:48:38.443520: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1483] Adding visible gpu devices: 0
2018-08-11 18:48:38.443576: I tensorflow/core/common_runtime/gpu/gpu_device.cc:964] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-08-11 18:48:38.443585: I tensorflow/core/common_runtime/gpu/gpu_device.cc:970]      0 
2018-08-11 18:48:38.443591: I tensorflow/core/common_runtime/gpu/gpu_device.cc:983] 0:   N 
2018-08-11 18:48:38.443728: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1096] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 3255 MB memory) -> physical GPU (device: 0, name: GeForce GTX 850M, pci bus id: 0000:0a:00.0, compute capability: 5.0)
creating index...
index created!
creating index...
index created!
Running per image evaluation...
Evaluate annotation type *bbox*
DONE (t=2.74s).
Accumulating evaluation results...
DONE (t=0.35s).
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.373
 Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.604
 Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = 0.441
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = -1.000
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.451
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.260
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  1 ] = 0.356
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 10 ] = 0.576
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.641
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = -1.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.644
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.620

[MichaelX99]

@szm2015, I am glad you are able to run the script without errors now! The system is massive and can error in very annoying places... I no longer can help since I run on the legacy code before estimators and am not knowledgeable enough about their latest code to help any more, good luck!

[tensorflowbutler]

Hi There,
We are checking to see if you still need help on this, as this seems to be considerably old issue. Please update this issue with the latest information, code snippet to reproduce your issue and error you are seeing.
If we don't hear from you in the next 7 days, this issue will be closed automatically. If you don't need help on this issue any more, please consider closing this.

[aligokkaya]

heey tensorflowbutler help please

(tens-test) C:\tensorflow1\models\research\object_detection>python model_main.py --logtostderr --model_dir=training/ --pipeline_config_path=training/faster_rcnn_inception_v2_pets.config
WARNING:tensorflow:
The TensorFlow contrib module will not be included in TensorFlow 2.0.
For more information, please see:

• https://github.com/tensorflow/community/blob/master/rfcs/20180907-contrib-sunset.md
• https://github.com/tensorflow/addons
• https://github.com/tensorflow/io (for I/O related ops)
  If you depend on functionality not listed there, please file an issue.

WARNING:tensorflow:From model_main.py:109: The name tf.app.run is deprecated. Please use tf.compat.v1.app.run instead.

WARNING:tensorflow:From C:\tensorflow1\models\research\object_detection\utils\config_util.py:102: The name tf.gfile.GFile is deprecated. Please use tf.io.gfile.GFile instead.

W0215 03:49:36.531214 11232 module_wrapper.py:139] From C:\tensorflow1\models\research\object_detection\utils\config_util.py:102: The name tf.gfile.GFile is deprecated. Please use tf.io.gfile.GFile instead.

WARNING:tensorflow:From C:\tensorflow1\models\research\object_detection\model_lib.py:628: The name tf.logging.warning is deprecated. Please use tf.compat.v1.logging.warning instead.

W0215 03:49:36.539234 11232 module_wrapper.py:139] From C:\tensorflow1\models\research\object_detection\model_lib.py:628: The name tf.logging.warning is deprecated. Please use tf.compat.v1.logging.warning instead.

WARNING:tensorflow:Forced number of epochs for all eval validations to be 1.
W0215 03:49:36.543213 11232 model_lib.py:629] Forced number of epochs for all eval validations to be 1.
WARNING:tensorflow:From C:\tensorflow1\models\research\object_detection\utils\config_util.py:488: The name tf.logging.info is deprecated. Please use tf.compat.v1.logging.info instead.

W0215 03:49:36.545213 11232 module_wrapper.py:139] From C:\tensorflow1\models\research\object_detection\utils\config_util.py:488: The name tf.logging.info is deprecated. Please use tf.compat.v1.logging.info instead.

INFO:tensorflow:Maybe overwriting train_steps: None
I0215 03:49:36.552228 11232 config_util.py:488] Maybe overwriting train_steps: None
INFO:tensorflow:Maybe overwriting use_bfloat16: False
I0215 03:49:36.554213 11232 config_util.py:488] Maybe overwriting use_bfloat16: False
INFO:tensorflow:Maybe overwriting sample_1_of_n_eval_examples: 1
I0215 03:49:36.555222 11232 config_util.py:488] Maybe overwriting sample_1_of_n_eval_examples: 1
INFO:tensorflow:Maybe overwriting eval_num_epochs: 1
I0215 03:49:36.558212 11232 config_util.py:488] Maybe overwriting eval_num_epochs: 1
INFO:tensorflow:Maybe overwriting load_pretrained: True
I0215 03:49:36.564219 11232 config_util.py:488] Maybe overwriting load_pretrained: True
INFO:tensorflow:Ignoring config override key: load_pretrained
I0215 03:49:36.565212 11232 config_util.py:498] Ignoring config override key: load_pretrained
WARNING:tensorflow:Expected number of evaluation epochs is 1, but instead encountered eval_on_train_input_config.num_epochs = 0. Overwriting num_epochs to 1.
W0215 03:49:36.568212 11232 model_lib.py:645] Expected number of evaluation epochs is 1, but instead encountered eval_on_train_input_config.num_epochs = 0. Overwriting num_epochs to 1.
INFO:tensorflow:create_estimator_and_inputs: use_tpu False, export_to_tpu False
I0215 03:49:36.570220 11232 model_lib.py:680] create_estimator_and_inputs: use_tpu False, export_to_tpu False
INFO:tensorflow:Using config: {'_model_dir': 'training/', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': None, '_save_checkpoints_secs': 600, '_session_config': allow_soft_placement: true
graph_options {
rewrite_options {
meta_optimizer_iterations: ONE
}
}
, '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_service': None, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x00000225771A4860>, '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1}
I0215 03:49:36.572225 11232 estimator.py:212] Using config: {'_model_dir': 'training/', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': None, '_save_checkpoints_secs': 600, '_session_config': allow_soft_placement: true
graph_options {
rewrite_options {
meta_optimizer_iterations: ONE
}
}
, '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_session_creation_timeout_secs': 7200, '_service': None, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x00000225771A4860>, '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1}
WARNING:tensorflow:Estimator's model_fn (<function create_model_fn..model_fn at 0x00000225771A5AE8>) includes params argument, but params are not passed to Estimator.
W0215 03:49:36.580208 11232 model_fn.py:630] Estimator's model_fn (<function create_model_fn..model_fn at 0x00000225771A5AE8>) includes params argument, but params are not passed to Estimator.
INFO:tensorflow:Not using Distribute Coordinator.
I0215 03:49:36.583206 11232 estimator_training.py:186] Not using Distribute Coordinator.
INFO:tensorflow:Running training and evaluation locally (non-distributed).
I0215 03:49:36.585206 11232 training.py:612] Running training and evaluation locally (non-distributed).
INFO:tensorflow:Start train and evaluate loop. The evaluate will happen after every checkpoint. Checkpoint frequency is determined based on RunConfig arguments: save_checkpoints_steps None or save_checkpoints_secs 600.
I0215 03:49:36.587209 11232 training.py:700] Start train and evaluate loop. The evaluate will happen after every checkpoint. Checkpoint frequency is determined based on RunConfig arguments: save_checkpoints_steps None or save_checkpoints_secs 600.
WARNING:tensorflow:From C:\Users\aligk\Anaconda3\envs\tens-test\lib\site-packages\tensorflow_core\python\training\training_util.py:236: Variable.initialized_value (from tensorflow.python.ops.variables) is deprecated and will be removed in a future version.
Instructions for updating:
Use Variable.read_value. Variables in 2.X are initialized automatically both in eager and graph (inside tf.defun) contexts.
W0215 03:49:36.608065 11232 deprecation.py:323] From C:\Users\aligk\Anaconda3\envs\tens-test\lib\site-packages\tensorflow_core\python\training\training_util.py:236: Variable.initialized_value (from tensorflow.python.ops.variables) is deprecated and will be removed in a future version.
Instructions for updating:
Use Variable.read_value. Variables in 2.X are initialized automatically both in eager and graph (inside tf.defun) contexts.
WARNING:tensorflow:From C:\tensorflow1\models\research\object_detection\data_decoders\tf_example_decoder.py:182: The name tf.FixedLenFeature is deprecated. Please use tf.io.FixedLenFeature instead.

W0215 03:49:36.632063 11232 module_wrapper.py:139] From C:\tensorflow1\models\research\object_detection\data_decoders\tf_example_decoder.py:182: The name tf.FixedLenFeature is deprecated. Please use tf.io.FixedLenFeature instead.

WARNING:tensorflow:From C:\tensorflow1\models\research\object_detection\data_decoders\tf_example_decoder.py:197: The name tf.VarLenFeature is deprecated. Please use tf.io.VarLenFeature instead.

W0215 03:49:36.637063 11232 module_wrapper.py:139] From C:\tensorflow1\models\research\object_detection\data_decoders\tf_example_decoder.py:197: The name tf.VarLenFeature is deprecated. Please use tf.io.VarLenFeature instead.

WARNING:tensorflow:From C:\tensorflow1\models\research\object_detection\builders\dataset_builder.py:64: The name tf.gfile.Glob is deprecated. Please use tf.io.gfile.glob instead.

W0215 03:49:36.670076 11232 module_wrapper.py:139] From C:\tensorflow1\models\research\object_detection\builders\dataset_builder.py:64: The name tf.gfile.Glob is deprecated. Please use tf.io.gfile.glob instead.

WARNING:tensorflow:num_readers has been reduced to 1 to match input file shards.
W0215 03:49:36.683053 11232 dataset_builder.py:72] num_readers has been reduced to 1 to match input file shards.
WARNING:tensorflow:From C:\tensorflow1\models\research\object_detection\builders\dataset_builder.py:86: parallel_interleave (from tensorflow.contrib.data.python.ops.interleave_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.data.experimental.parallel_interleave(...).
W0215 03:49:36.698752 11232 deprecation.py:323] From C:\tensorflow1\models\research\object_detection\builders\dataset_builder.py:86: parallel_interleave (from tensorflow.contrib.data.python.ops.interleave_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.data.experimental.parallel_interleave(...).
WARNING:tensorflow:From C:\Users\aligk\Anaconda3\envs\tens-test\lib\site-packages\tensorflow_core\contrib\data\python\ops\interleave_ops.py:77: parallel_interleave (from tensorflow.python.data.experimental.ops.interleave_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.data.Dataset.interleave(map_func, cycle_length, block_length, num_parallel_calls=tf.data.experimental.AUTOTUNE) instead. If sloppy execution is desired, use tf.data.Options.experimental_determinstic.
W0215 03:49:36.703757 11232 deprecation.py:323] From C:\Users\aligk\Anaconda3\envs\tens-test\lib\site-packages\tensorflow_core\contrib\data\python\ops\interleave_ops.py:77: parallel_interleave (from tensorflow.python.data.experimental.ops.interleave_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.data.Dataset.interleave(map_func, cycle_length, block_length, num_parallel_calls=tf.data.experimental.AUTOTUNE) instead. If sloppy execution is desired, use tf.data.Options.experimental_determinstic.
WARNING:tensorflow:From C:\tensorflow1\models\research\object_detection\builders\dataset_builder.py:155: DatasetV1.map_with_legacy_function (from tensorflow.python.data.ops.dataset_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.data.Dataset.map() W0215 03:49:36.758179 11232 deprecation.py:323] From C:\tensorflow1\models\research\object_detection\builders\dataset_builder.py:155: DatasetV1.map_with_legacy_function (from tensorflow.python.data.ops.dataset_ops) is deprecated and will be removed in a future version. Instructions for updating: Use tf.data.Dataset.map()
WARNING:tensorflow:From C:\Users\aligk\Anaconda3\envs\tens-test\lib\site-packages\tensorflow_core\python\autograph\converters\directives.py:119: The name tf.logging.warn is deprecated. Please use tf.compat.v1.logging.warn instead.

W0215 03:49:39.115937 11232 module_wrapper.py:139] From C:\Users\aligk\Anaconda3\envs\tens-test\lib\site-packages\tensorflow_core\python\autograph\converters\directives.py:119: The name tf.logging.warn is deprecated. Please use tf.compat.v1.logging.warn instead.

WARNING:tensorflow:From C:\Users\aligk\Anaconda3\envs\tens-test\lib\site-packages\tensorflow_core\python\autograph\converters\directives.py:119: The name tf.is_nan is deprecated. Please use tf.math.is_nan instead.

W0215 03:49:50.746980 11232 module_wrapper.py:139] From C:\Users\aligk\Anaconda3\envs\tens-test\lib\site-packages\tensorflow_core\python\autograph\converters\directives.py:119: The name tf.is_nan is deprecated. Please use tf.math.is_nan instead.

WARNING:tensorflow:From C:\tensorflow1\models\research\object_detection\utils\ops.py:493: where (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.where in 2.0, which has the same broadcast rule as np.where
W0215 03:49:50.888271 11232 deprecation.py:323] From C:\tensorflow1\models\research\object_detection\utils\ops.py:493: where (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.where in 2.0, which has the same broadcast rule as np.where
WARNING:tensorflow:From C:\Users\aligk\Anaconda3\envs\tens-test\lib\site-packages\tensorflow_core\python\autograph\converters\directives.py:119: The name tf.random_uniform is deprecated. Please use tf.random.uniform instead.

W0215 03:49:54.265079 11232 module_wrapper.py:139] From C:\Users\aligk\Anaconda3\envs\tens-test\lib\site-packages\tensorflow_core\python\autograph\converters\directives.py:119: The name tf.random_uniform is deprecated. Please use tf.random.uniform instead.

WARNING:tensorflow:From C:\Users\aligk\Anaconda3\envs\tens-test\lib\site-packages\tensorflow_core\python\autograph\converters\directives.py:119: The name tf.image.resize_images is deprecated. Please use tf.image.resize instead.

W0215 03:49:57.583898 11232 module_wrapper.py:139] From C:\Users\aligk\Anaconda3\envs\tens-test\lib\site-packages\tensorflow_core\python\autograph\converters\directives.py:119: The name tf.image.resize_images is deprecated. Please use tf.image.resize instead.

WARNING:tensorflow:From C:\tensorflow1\models\research\object_detection\inputs.py:166: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.cast instead.
W0215 03:49:58.692425 11232 deprecation.py:323] From C:\tensorflow1\models\research\object_detection\inputs.py:166: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.cast instead.
WARNING:tensorflow:From C:\Users\aligk\Anaconda3\envs\tens-test\lib\site-packages\tensorflow_core\python\autograph\converters\directives.py:119: The name tf.string_to_hash_bucket_fast is deprecated. Please use tf.strings.to_hash_bucket_fast instead.

W0215 03:50:02.155535 11232 module_wrapper.py:139] From C:\Users\aligk\Anaconda3\envs\tens-test\lib\site-packages\tensorflow_core\python\autograph\converters\directives.py:119: The name tf.string_to_hash_bucket_fast is deprecated. Please use tf.strings.to_hash_bucket_fast instead.

WARNING:tensorflow:From C:\tensorflow1\models\research\object_detection\builders\dataset_builder.py:158: batch_and_drop_remainder (from tensorflow.contrib.data.python.ops.batching) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.data.Dataset.batch(..., drop_remainder=True).
W0215 03:50:03.060607 11232 deprecation.py:323] From C:\tensorflow1\models\research\object_detection\builders\dataset_builder.py:158: batch_and_drop_remainder (from tensorflow.contrib.data.python.ops.batching) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.data.Dataset.batch(..., drop_remainder=True).
INFO:tensorflow:Calling model_fn.
I0215 03:50:03.122578 11232 estimator.py:1148] Calling model_fn.
WARNING:tensorflow:From C:\tensorflow1\models\research\object_detection\meta_architectures\faster_rcnn_meta_arch.py:168: The name tf.variable_scope is deprecated. Please use tf.compat.v1.variable_scope instead.

W0215 03:50:03.181793 11232 module_wrapper.py:139] From C:\tensorflow1\models\research\object_detection\meta_architectures\faster_rcnn_meta_arch.py:168: The name tf.variable_scope is deprecated. Please use tf.compat.v1.variable_scope instead.

WARNING:tensorflow:From C:\Users\aligk\Anaconda3\envs\tens-test\lib\site-packages\tensorflow_core\contrib\layers\python\layers\layers.py:2784: Layer.apply (from tensorflow.python.keras.engine.base_layer) is deprecated and will be removed in a future version.
Instructions for updating:
Please use layer.__call__ method instead.
W0215 03:50:03.209163 11232 deprecation.py:323] From C:\Users\aligk\Anaconda3\envs\tens-test\lib\site-packages\tensorflow_core\contrib\layers\python\layers\layers.py:2784: Layer.apply (from tensorflow.python.keras.engine.base_layer) is deprecated and will be removed in a future version.
Instructions for updating:
Please use layer.__call__ method instead.
WARNING:tensorflow:From C:\tensorflow1\models\research\object_detection\core\anchor_generator.py:171: The name tf.assert_equal is deprecated. Please use tf.compat.v1.assert_equal instead.

W0215 03:50:06.293033 11232 module_wrapper.py:139] From C:\tensorflow1\models\research\object_detection\core\anchor_generator.py:171: The name tf.assert_equal is deprecated. Please use tf.compat.v1.assert_equal instead.

INFO:tensorflow:Scale of 0 disables regularizer.
I0215 03:50:06.315390 11232 regularizers.py:98] Scale of 0 disables regularizer.
WARNING:tensorflow:From C:\tensorflow1\models\research\object_detection\meta_architectures\faster_rcnn_meta_arch.py:558: The name tf.get_variable_scope is deprecated. Please use tf.compat.v1.get_variable_scope instead.

W0215 03:50:06.318396 11232 module_wrapper.py:139] From C:\tensorflow1\models\research\object_detection\meta_architectures\faster_rcnn_meta_arch.py:558: The name tf.get_variable_scope is deprecated. Please use tf.compat.v1.get_variable_scope instead.

INFO:tensorflow:Scale of 0 disables regularizer.
I0215 03:50:06.353806 11232 regularizers.py:98] Scale of 0 disables regularizer.
INFO:tensorflow:depth of additional conv before box predictor: 0
I0215 03:50:06.355805 11232 convolutional_box_predictor.py:151] depth of additional conv before box predictor: 0
WARNING:tensorflow:From C:\tensorflow1\models\research\object_detection\box_coders\faster_rcnn_box_coder.py:82: The name tf.log is deprecated. Please use tf.math.log instead.

W0215 03:50:07.548388 11232 module_wrapper.py:139] From C:\tensorflow1\models\research\object_detection\box_coders\faster_rcnn_box_coder.py:82: The name tf.log is deprecated. Please use tf.math.log instead.

WARNING:tensorflow:From C:\tensorflow1\models\research\object_detection\core\minibatch_sampler.py:85: The name tf.random_shuffle is deprecated. Please use tf.random.shuffle instead.

W0215 03:50:07.637370 11232 module_wrapper.py:139] From C:\tensorflow1\models\research\object_detection\core\minibatch_sampler.py:85: The name tf.random_shuffle is deprecated. Please use tf.random.shuffle instead.

WARNING:tensorflow:From C:\tensorflow1\models\research\object_detection\utils\spatial_transform_ops.py:419: calling crop_and_resize_v1 (from tensorflow.python.ops.image_ops_impl) with box_ind is deprecated and will be removed in a future version.
Instructions for updating:
box_ind is deprecated, use box_indices instead
W0215 03:50:08.094356 11232 deprecation.py:506] From C:\tensorflow1\models\research\object_detection\utils\spatial_transform_ops.py:419: calling crop_and_resize_v1 (from tensorflow.python.ops.image_ops_impl) with box_ind is deprecated and will be removed in a future version.
Instructions for updating:
box_ind is deprecated, use box_indices instead
WARNING:tensorflow:From C:\tensorflow1\models\research\object_detection\meta_architectures\faster_rcnn_meta_arch.py:191: The name tf.AUTO_REUSE is deprecated. Please use tf.compat.v1.AUTO_REUSE instead.

W0215 03:50:08.126353 11232 module_wrapper.py:139] From C:\tensorflow1\models\research\object_detection\meta_architectures\faster_rcnn_meta_arch.py:191: The name tf.AUTO_REUSE is deprecated. Please use tf.compat.v1.AUTO_REUSE instead.

WARNING:tensorflow:From C:\Users\aligk\Anaconda3\envs\tens-test\lib\site-packages\tensorflow_core\contrib\layers\python\layers\layers.py:1634: flatten (from tensorflow.python.layers.core) is deprecated and will be removed in a future version.
Instructions for updating:
Use keras.layers.flatten instead.
W0215 03:50:09.165442 11232 deprecation.py:323] From C:\Users\aligk\Anaconda3\envs\tens-test\lib\site-packages\tensorflow_core\contrib\layers\python\layers\layers.py:1634: flatten (from tensorflow.python.layers.core) is deprecated and will be removed in a future version.
Instructions for updating:
Use keras.layers.flatten instead.
INFO:tensorflow:Scale of 0 disables regularizer.
I0215 03:50:09.174441 11232 regularizers.py:98] Scale of 0 disables regularizer.
INFO:tensorflow:Scale of 0 disables regularizer.
I0215 03:50:09.212453 11232 regularizers.py:98] Scale of 0 disables regularizer.
WARNING:tensorflow:From C:\tensorflow1\models\research\object_detection\utils\variables_helper.py:179: The name tf.global_variables is deprecated. Please use tf.compat.v1.global_variables instead.

W0215 03:50:09.268049 11232 module_wrapper.py:139] From C:\tensorflow1\models\research\object_detection\utils\variables_helper.py:179: The name tf.global_variables is deprecated. Please use tf.compat.v1.global_variables instead.

WARNING:tensorflow:From C:\tensorflow1\models\research\object_detection\meta_architectures\faster_rcnn_meta_arch.py:2768: get_or_create_global_step (from tensorflow.contrib.framework.python.ops.variables) is deprecated and will be removed in a future version.
Instructions for updating:
Please switch to tf.train.get_or_create_global_step
W0215 03:50:09.273027 11232 deprecation.py:323] From C:\tensorflow1\models\research\object_detection\meta_architectures\faster_rcnn_meta_arch.py:2768: get_or_create_global_step (from tensorflow.contrib.framework.python.ops.variables) is deprecated and will be removed in a future version.
Instructions for updating:
Please switch to tf.train.get_or_create_global_step
WARNING:tensorflow:From C:\tensorflow1\models\research\object_detection\utils\variables_helper.py:139: The name tf.train.NewCheckpointReader is deprecated. Please use tf.compat.v1.train.NewCheckpointReader instead.

W0215 03:50:09.282026 11232 module_wrapper.py:139] From C:\tensorflow1\models\research\object_detection\utils\variables_helper.py:139: The name tf.train.NewCheckpointReader is deprecated. Please use tf.compat.v1.train.NewCheckpointReader instead.

Windows fatal exception: access violation

Current thread 0x00002be0 (most recent call first):
File "C:\Users\aligk\Anaconda3\envs\tens-test\lib\site-packages\tensorflow_core\python\pywrap_tensorflow_internal.py", line 885 in init
File "C:\Users\aligk\Anaconda3\envs\tens-test\lib\site-packages\tensorflow_core\python\pywrap_tensorflow_internal.py", line 873 in NewCheckpointReader
File "C:\tensorflow1\models\research\object_detection\utils\variables_helper.py", line 139 in get_variables_available_in_checkpoint
File "C:\tensorflow1\models\research\object_detection\model_lib.py", line 343 in model_fn
File "C:\Users\aligk\Anaconda3\envs\tens-test\lib\site-packages\tensorflow_estimator\python\estimator\estimator.py", line 1149 in _call_model_fn
File "C:\Users\aligk\Anaconda3\envs\tens-test\lib\site-packages\tensorflow_estimator\python\estimator\estimator.py", line 1191 in _train_model_default
File "C:\Users\aligk\Anaconda3\envs\tens-test\lib\site-packages\tensorflow_estimator\python\estimator\estimator.py", line 1161 in _train_model
File "C:\Users\aligk\Anaconda3\envs\tens-test\lib\site-packages\tensorflow_estimator\python\estimator\estimator.py", line 370 in train
File "C:\Users\aligk\Anaconda3\envs\tens-test\lib\site-packages\tensorflow_estimator\python\estimator\training.py", line 714 in run_local
File "C:\Users\aligk\Anaconda3\envs\tens-test\lib\site-packages\tensorflow_estimator\python\estimator\training.py", line 613 in run
File "C:\Users\aligk\Anaconda3\envs\tens-test\lib\site-packages\tensorflow_estimator\python\estimator\training.py", line 473 in train_and_evaluate
File "model_main.py", line 105 in main
File "C:\Users\aligk\Anaconda3\envs\tens-test\lib\site-packages\absl\app.py", line 250 in _run_main
File "C:\Users\aligk\Anaconda3\envs\tens-test\lib\site-packages\absl\app.py", line 299 in run
File "C:\Users\aligk\Anaconda3\envs\tens-test\lib\site-packages\tensorflow_core\python\platform\app.py", line 40 in run
File "model_main.py", line 109 in