eval_image_classifier.py can't predict top 5 accuracy #694

civilman628 · 2016-11-28T18:21:22Z

I get the following error when run eval_image_classifier.py to test inception v2 .ckpt file: I am using the latest code and run a data set which has 269 class labels.

INFO:tensorflow:Error reported to Coordinator: <class 'tensorflow.python.framework.errors.InvalidArgumentError'>, targets[86] is out of range
[[Node: InTopK = InTopK[T=DT_INT64, k=5, _device="/job:localhost/replica:0/task:0/cpu:0"](InceptionV2/Logits/SpatialSqueeze/_1161, Squeeze/_1163)]]
[[Node: InTopK/_1165 = _Recvclient_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_322_InTopK", tensor_type=DT_BOOL, _device="/job:localhost/replica:0/task:0/gpu:0"]]

Caused by op u'InTopK', defined at:
File "eval_image_classifier.py", line 191, in
tf.app.run()
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/platform/app.py", line 30, in run
sys.exit(main(sys.argv[:1] + flags_passthrough))
File "eval_image_classifier.py", line 157, in main
logits, labels, 5),
File "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/framework/python/framework/deprecation.py", line 218, in new_func
return func(*args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/metrics/python/ops/metric_ops.py", line 1277, in streaming_recall_at_k
in_top_k = math_ops.to_float(nn.in_top_k(predictions, labels, k))
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/gen_nn_ops.py", line 1290, in in_top_k
targets=targets, k=k, name=name)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/op_def_library.py", line 749, in apply_op
op_def=op_def)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 2380, in create_op
original_op=self._default_original_op, op_def=op_def)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 1298, in init
self._traceback = _extract_stack()

InvalidArgumentError (see above for traceback): targets[86] is out of range
[[Node: InTopK = InTopK[T=DT_INT64, k=5, _device="/job:localhost/replica:0/task:0/cpu:0"](InceptionV2/Logits/SpatialSqueeze/_1161, Squeeze/_1163)]]
[[Node: InTopK/_1165 = _Recvclient_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_322_InTopK", tensor_type=DT_BOOL, _device="/job:localhost/replica:0/task:0/gpu:0"]]

Traceback (most recent call last):
File "eval_image_classifier.py", line 191, in
tf.app.run()
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/platform/app.py", line 30, in run
sys.exit(main(sys.argv[:1] + flags_passthrough))
File "eval_image_classifier.py", line 187, in main
variables_to_restore=variables_to_restore)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/slim/python/slim/evaluation.py", line 359, in evaluate_once
global_step=global_step)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/slim/python/slim/evaluation.py", line 260, in evaluation
sess.run(eval_op, eval_op_feed_dict)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 717, in run
run_metadata_ptr)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 915, in _run
feed_dict_string, options, run_metadata)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 965, in _do_run
target_list, options, run_metadata)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 985, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors.InvalidArgumentError: targets[86] is out of range
[[Node: InTopK = InTopK[T=DT_INT64, k=5, _device="/job:localhost/replica:0/task:0/cpu:0"](InceptionV2/Logits/SpatialSqueeze/_1161, Squeeze/_1163)]]
[[Node: InTopK/_1165 = _Recvclient_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_322_InTopK", tensor_type=DT_BOOL, _device="/job:localhost/replica:0/task:0/gpu:0"]]

Caused by op u'InTopK', defined at:
File "eval_image_classifier.py", line 191, in
tf.app.run()
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/platform/app.py", line 30, in run
sys.exit(main(sys.argv[:1] + flags_passthrough))
File "eval_image_classifier.py", line 157, in main
logits, labels, 5),
File "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/framework/python/framework/deprecation.py", line 218, in new_func
return func(*args, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/metrics/python/ops/metric_ops.py", line 1277, in streaming_recall_at_k
in_top_k = math_ops.to_float(nn.in_top_k(predictions, labels, k))
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/gen_nn_ops.py", line 1290, in in_top_k
targets=targets, k=k, name=name)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/op_def_library.py", line 749, in apply_op
op_def=op_def)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 2380, in create_op
original_op=self._default_original_op, op_def=op_def)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 1298, in init
self._traceback = _extract_stack()

InvalidArgumentError (see above for traceback): targets[86] is out of range
[[Node: InTopK = InTopK[T=DT_INT64, k=5, _device="/job:localhost/replica:0/task:0/cpu:0"](InceptionV2/Logits/SpatialSqueeze/_1161, Squeeze/_1163)]]
[[Node: InTopK/_1165 = _Recvclient_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_322_InTopK", tensor_type=DT_BOOL, _device="/job:localhost/replica:0/task:0/gpu:0"]]

civilman628 · 2016-11-28T18:22:40Z

if remove out the line: 'Recall@5': slim.metrics.streaming_recall_at_k(
logits, labels, 5),

I can still get top 1 accuracy without any error.

osama · 2016-11-29T13:37:24Z

if remove out the line: 'Recall@5': slim.metrics.streaming_recall_at_k(logits, labels, 5),

Thanks for that! I was seeing a similar error after fine tuning inception_resnet_v2 but removing this line at least allows me to see the top 1 accuracy.

If it helps, my error is as follows (on TensorFlow r0.11 Linux/CPU):

INFO:tensorflow:Scale of 0 disables regularizer.
INFO:tensorflow:Scale of 0 disables regularizer.
INFO:tensorflow:Evaluating /home/osama/checkpoint/training-resnet/model.ckpt-1033
INFO:tensorflow:Starting evaluation at 2016-11-29-13:13:17
INFO:tensorflow:Executing eval ops
INFO:tensorflow:Executing eval_op 1/4
W tensorflow/core/framework/op_kernel.cc:968] Invalid argument: targets[20] is out of range
	 [[Node: InTopK = InTopK[T=DT_INT64, k=5, _device="/job:localhost/replica:0/task:0/cpu:0"](InceptionResnetV2/Logits/Logits/BiasAdd, Squeeze)]]
INFO:tensorflow:Error reported to Coordinator: <class 'tensorflow.python.framework.errors.InvalidArgumentError'>, targets[20] is out of range
	 [[Node: InTopK = InTopK[T=DT_INT64, k=5, _device="/job:localhost/replica:0/task:0/cpu:0"](InceptionResnetV2/Logits/Logits/BiasAdd, Squeeze)]]

Caused by op u'InTopK', defined at:
  File "eval_image_classifier.py", line 191, in <module>
    tf.app.run()
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/platform/app.py", line 30, in run
    sys.exit(main(sys.argv[:1] + flags_passthrough))
  File "eval_image_classifier.py", line 157, in main
    logits, labels, 5),
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/framework/python/framework/deprecation.py", line 218, in new_func
    return func(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/metrics/python/ops/metric_ops.py", line 1277, in streaming_recall_at_k
    in_top_k = math_ops.to_float(nn.in_top_k(predictions, labels, k))
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/gen_nn_ops.py", line 1290, in in_top_k
    targets=targets, k=k, name=name)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/op_def_library.py", line 749, in apply_op
    op_def=op_def)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 2380, in create_op
    original_op=self._default_original_op, op_def=op_def)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 1298, in __init__
    self._traceback = _extract_stack()

InvalidArgumentError (see above for traceback): targets[20] is out of range
	 [[Node: InTopK = InTopK[T=DT_INT64, k=5, _device="/job:localhost/replica:0/task:0/cpu:0"](InceptionResnetV2/Logits/Logits/BiasAdd, Squeeze)]]

Traceback (most recent call last):
  File "eval_image_classifier.py", line 191, in <module>
    tf.app.run()
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/platform/app.py", line 30, in run
    sys.exit(main(sys.argv[:1] + flags_passthrough))
  File "eval_image_classifier.py", line 187, in main
    variables_to_restore=variables_to_restore)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/slim/python/slim/evaluation.py", line 359, in evaluate_once
    global_step=global_step)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/slim/python/slim/evaluation.py", line 260, in evaluation
    sess.run(eval_op, eval_op_feed_dict)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 717, in run
    run_metadata_ptr)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 915, in _run
    feed_dict_string, options, run_metadata)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 965, in _do_run
    target_list, options, run_metadata)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 985, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors.InvalidArgumentError: targets[20] is out of range
	 [[Node: InTopK = InTopK[T=DT_INT64, k=5, _device="/job:localhost/replica:0/task:0/cpu:0"](InceptionResnetV2/Logits/Logits/BiasAdd, Squeeze)]]

Caused by op u'InTopK', defined at:
  File "eval_image_classifier.py", line 191, in <module>
    tf.app.run()
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/platform/app.py", line 30, in run
    sys.exit(main(sys.argv[:1] + flags_passthrough))
  File "eval_image_classifier.py", line 157, in main
    logits, labels, 5),
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/framework/python/framework/deprecation.py", line 218, in new_func
    return func(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/metrics/python/ops/metric_ops.py", line 1277, in streaming_recall_at_k
    in_top_k = math_ops.to_float(nn.in_top_k(predictions, labels, k))
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/gen_nn_ops.py", line 1290, in in_top_k
    targets=targets, k=k, name=name)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/op_def_library.py", line 749, in apply_op
    op_def=op_def)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 2380, in create_op
    original_op=self._default_original_op, op_def=op_def)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 1298, in __init__
    self._traceback = _extract_stack()

InvalidArgumentError (see above for traceback): targets[20] is out of range
	 [[Node: InTopK = InTopK[T=DT_INT64, k=5, _device="/job:localhost/replica:0/task:0/cpu:0"](InceptionResnetV2/Logits/Logits/BiasAdd, Squeeze)]]

shahparth123 · 2016-12-20T05:44:37Z

Same issue with me also. I am using flower dataset for fine tuning. It gives me same error except targets[3] out of range in error. using tensorflow rc11, python 2.7 and CentOS 7 without GPU

salilkapur · 2017-06-17T23:33:44Z

Change
slim.metrics.streaming_recall_at_k(logits, labels, 5)
to
slim.metrics.streaming_sparse_recall_at_k(logits, labels, 5)

aselle · 2017-06-20T17:16:39Z

Does @salikapur's suggestion resolve your issue?

rudifus · 2017-07-07T09:49:47Z

seems not, for my 2 categories finetune train, afer changing to
slim.metrics.streaming_sparse_recall_at_k(logits, labels, 5)

ValueError: input must have last dimension >= k = 5 but is 2 for 'recall_at_5/TopKV2' (op: 'TopKV2') with input shapes: [32,2], [] and with computed input tensors: input[1] = <5>.

salilkapur · 2017-07-07T17:46:22Z

With just 2 labels I dont think you can calculate top 5 error. I think just top 1 error should suffice for your case.

offbye · 2017-10-19T11:51:40Z

Change
slim.metrics.streaming_recall_at_k(logits, labels, 5)
to
slim.metrics.streaming_sparse_recall_at_k(logits, labels, 5)

It works！！

… only - ref: tensorflow#694 - ref: https://stackoverflow.com/questions/43803168/running-tf-slim-evaluation-loop-on-cpu

itsmeolivia · 2018-02-08T00:19:18Z

Automatically closing due to lack of recent activity. Please update the issue when new information becomes available, and we will reopen the issue. Thanks!

kycocotree · 2019-07-22T07:51:16Z

predictions = tf.argmax(logits, 1)
labels = tf.squeeze(labels) -> this code change to #labels = tf.squeeze(labels) or delete

# Define the metrics:
names_to_values, names_to_updates = slim.metrics.aggregate_metric_map({
    'Accuracy': slim.metrics.streaming_accuracy(predictions, labels),
    'Recall_5': slim.metrics.streaming_recall_at_k(
    logits, labels, 5),
})

poxvoculi assigned nathansilberman Nov 28, 2016

salilkapur unassigned nathansilberman Jun 17, 2017

aselle added the stat:awaiting response Waiting on input from the contributor label Jun 20, 2017

kertansul added a commit to kertansul/models that referenced this issue Nov 9, 2017

fix eval recall bug; add FLAGS to adjust GPU MEM alloc or eval on CPU…

e6a99bd

… only - ref: tensorflow#694 - ref: https://stackoverflow.com/questions/43803168/running-tf-slim-evaluation-loop-on-cpu

itsmeolivia closed this as completed Feb 8, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

eval_image_classifier.py can't predict top 5 accuracy #694

eval_image_classifier.py can't predict top 5 accuracy #694

civilman628 commented Nov 28, 2016 •

edited

civilman628 commented Nov 28, 2016

osama commented Nov 29, 2016

shahparth123 commented Dec 20, 2016

salilkapur commented Jun 17, 2017

aselle commented Jun 20, 2017

rudifus commented Jul 7, 2017

salilkapur commented Jul 7, 2017

offbye commented Oct 19, 2017

itsmeolivia commented Feb 8, 2018

kycocotree commented Jul 22, 2019

eval_image_classifier.py can't predict top 5 accuracy #694

eval_image_classifier.py can't predict top 5 accuracy #694

Comments

civilman628 commented Nov 28, 2016 • edited

civilman628 commented Nov 28, 2016

osama commented Nov 29, 2016

shahparth123 commented Dec 20, 2016

salilkapur commented Jun 17, 2017

aselle commented Jun 20, 2017

rudifus commented Jul 7, 2017

salilkapur commented Jul 7, 2017

offbye commented Oct 19, 2017

itsmeolivia commented Feb 8, 2018

kycocotree commented Jul 22, 2019

civilman628 commented Nov 28, 2016 •

edited