Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Value Error: First Step Cannot Be Zero #3794

Closed
ashiqks opened this issue Mar 28, 2018 · 28 comments
Closed

Value Error: First Step Cannot Be Zero #3794

ashiqks opened this issue Mar 28, 2018 · 28 comments
Assignees

Comments

@ashiqks
Copy link

ashiqks commented Mar 28, 2018

  • What is the top-level directory of the model you are using: Object_detection
  • Have I written custom code (as opposed to using a stock example script provided in TensorFlow): No
  • OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Windows 7
  • TensorFlow installed from (source or binary): Binary
  • TensorFlow version (use command below): 1.6.0
  • Bazel version (if compiling from source):
  • CUDA/cuDNN version:
  • GPU model and memory:
  • Exact command to reproduce: python train.py --logtostderr --train_dir=training/ --pipeline_config_path=training/pipeline.config

When running the faster_rcnn_resnet101_kitti_2018_01_28 model it is throwing the error:

(base) C:\Users\MAT\models-master\research\object_detection>python train.py --lo
gtostderr --train_dir=training/ --pipeline_config_path=training/pipeline.config
C:\Users\MAT\Anaconda3\lib\site-packages\h5py\__init__.py:36: FutureWarning: Con
version of the second argument of issubdtype from `float` to `np.floating` is de
precated. In future, it will be treated as `np.float64 == np.dtype(float).type`.

from ._conv import register_converters as _register_converters
INFO:tensorflow:Scale of 0 disables regularizer.
INFO:tensorflow:Scale of 0 disables regularizer.
WARNING:tensorflow:From C:\Users\MAT\models-master\research\object_detection\tra
iner.py:228: create_global_step (from tensorflow.contrib.framework.python.ops.va
riables) is deprecated and will be removed in a future version.
Instructions for updating:
Please switch to tf.train.create_global_step
INFO:tensorflow:Scale of 0 disables regularizer.
INFO:tensorflow:Scale of 0 disables regularizer.
INFO:tensorflow:Scale of 0 disables regularizer.
INFO:tensorflow:depth of additional conv before box predictor: 0
INFO:tensorflow:Scale of 0 disables regularizer.
WARNING:tensorflow:From C:\Users\MAT\models-master\research\object_detection\cor
e\box_predictor.py:396: calling reduce_mean (from tensorflow.python.ops.math_ops
) with keep_dims is deprecated and will be removed in a future version.
Instructions for updating:
keep_dims is deprecated, use keepdims instead
WARNING:tensorflow:From C:\Users\MAT\models-master\research\object_detection\cor
e\losses.py:316: softmax_cross_entropy_with_logits (from tensorflow.python.ops.n
n_ops) is deprecated and will be removed in a future version.
Instructions for updating:

Future major versions of TensorFlow will allow gradients to flow
into the labels input on backprop by default.

See tf.nn.softmax_cross_entropy_with_logits_v2.

Traceback (most recent call last):
File "train.py", line 167, in <module>
tf.app.run()
File "C:\Users\MAT\Anaconda3\lib\site-packages\tensorflow\python\platform\app.
py", line 126, in run
 _sys.exit(main(argv))
File "train.py", line 163, in main
worker_job_name, is_chief, FLAGS.train_dir)
File "C:\Users\MAT\models-master\research\object_detection\trainer.py", line 2
55, in train
train_config.optimizer)
File "C:\Users\MAT\models-master\research\object_detection\builders\optimizer_
builder.py", line 50, in build
learning_rate = _create_learning_rate(config.learning_rate)
File "C:\Users\MAT\models-master\research\object_detection\builders\optimizer_
builder.py", line 109, in _create_learning_rate
learning_rate_sequence, config.warmup)
File "C:\Users\MAT\models-master\research\object_detection\utils\learning_sche
dules.py", line 156, in manual_stepping
raise ValueError('First step cannot be zero.')
@pkulzc
Copy link
Contributor

pkulzc commented Mar 28, 2018

We recently updated the code so this block

          schedule {
            step: 0
            learning_rate: .0001
          }

is not recognized anymore.

The model configs in model zoo still have this block so it crashed the program. You can remove this block or use the latest config instead of the downloaded one in tar.gz file.

@pkulzc pkulzc self-assigned this Mar 28, 2018
@vishalgolcha
Copy link

@pkulzc removing the block is no good , it still throws no variables to save error .
"ValueError: No variables to save"

@pkulzc
Copy link
Contributor

pkulzc commented Mar 29, 2018

@vishalgolcha could you share more details?

@vishalgolcha
Copy link

vishalgolcha commented Mar 29, 2018

What is the top-level directory of the model you are using: Object_detection
Have I written custom code (as opposed to using a stock example script provided in TensorFlow): No
OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Ubuntu 17.04
TensorFlow installed from (source or binary): Binary
TensorFlow version (use command below): 1.6.0
**CUDA/cuDNN version **: cuda 9.0
GPU model and memory: 1080 8GB

i am trying to finetune a faster rcnn inception model for a custom tfrecord file ,
below is my config file
model { faster_rcnn { num_classes: 8 image_resizer { keep_aspect_ratio_resizer { min_dimension: 600 max_dimension: 1024 } } feature_extractor { type: "faster_rcnn_inception_v2" first_stage_features_stride: 16 } first_stage_anchor_generator { grid_anchor_generator { height_stride: 16 width_stride: 16 scales: 0.25 scales: 0.5 scales: 1.0 scales: 2.0 aspect_ratios: 0.5 aspect_ratios: 1.0 aspect_ratios: 2.0 } } first_stage_box_predictor_conv_hyperparams { op: CONV regularizer { l2_regularizer { weight: 0.0 } } initializer { truncated_normal_initializer { stddev: 0.00999999977648 } } } first_stage_nms_score_threshold: 0.0 first_stage_nms_iou_threshold: 0.699999988079 first_stage_max_proposals: 300 first_stage_localization_loss_weight: 2.0 first_stage_objectness_loss_weight: 1.0 initial_crop_size: 14 maxpool_kernel_size: 2 maxpool_stride: 2 second_stage_box_predictor { mask_rcnn_box_predictor { fc_hyperparams { op: FC regularizer { l2_regularizer { weight: 0.0 } } initializer { variance_scaling_initializer { factor: 1.0 uniform: true mode: FAN_AVG } } } use_dropout: false dropout_keep_probability: 1.0 } } second_stage_post_processing { batch_non_max_suppression { score_threshold: 0.0 iou_threshold: 0.600000023842 max_detections_per_class: 100 max_total_detections: 300 } score_converter: SOFTMAX } second_stage_localization_loss_weight: 2.0 second_stage_classification_loss_weight: 1.0 } } train_config { batch_size: 1 data_augmentation_options { random_horizontal_flip { } } optimizer { momentum_optimizer { learning_rate { manual_step_learning_rate { initial_learning_rate: 0.000199999994948 schedule { step: 900000 learning_rate: 1.99999994948e-05 } schedule { step: 1200000 learning_rate: 1.99999999495e-06 } } } momentum_optimizer_value: 0.899999976158 } use_moving_average: false } gradient_clipping_by_norm: 10.0 fine_tune_checkpoint: "/home/lashiv/obj-detect/sop/models/model.ckpt.index" from_detection_checkpoint: true num_steps: 200000 } train_input_reader { label_map_path: "/home/lashiv/obj-detect/sop/data/StringIntLabelMap.pbtxt" tf_record_input_reader { input_path: "/home/lashiv/obj-detect/sop/data/train.record" } } eval_config { num_examples: 8000 max_evals: 10 use_moving_averages: false } eval_input_reader { label_map_path: "/home/lashiv/obj-detect/sop/data/StringIntLabelMap.pbtxt" shuffle: false num_readers: 1 tf_record_input_reader { input_path: "/home/lashiv/obj-detect/sop/data/train.record" } }

the traceback it shows is
Traceback (most recent call last):
File "object_detection/train.py", line 167, in
tf.app.run()
File "/home/lashiv/anaconda3/envs/detect/lib/python3.6/site-packages/tensorflow/python/platform/app.py", line 126, in run
_sys.exit(main(argv))
File "object_detection/train.py", line 163, in main
worker_job_name, is_chief, FLAGS.train_dir)
File "/home/lashiv/obj-detect/models/research/object_detection/trainer.py", line 285, in train
init_saver = tf.train.Saver(available_var_map)
File "/home/lashiv/anaconda3/envs/detect/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 1293, in init
self.build()
File "/home/lashiv/anaconda3/envs/detect/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 1302, in build
self._build(self._filename, build_save=True, build_restore=True)
File "/home/lashiv/anaconda3/envs/detect/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 1327, in _build
raise ValueError("No variables to save")
ValueError: No variables to save

@pkulzc
Copy link
Contributor

pkulzc commented Mar 29, 2018

You're using the wrong file for checkpoint. Use this:
fine_tune_checkpoint: "/home/lashiv/obj-detect/sop/models/model.ckpt"
instead.
Since this is a different problem, please open another thread for this if you have further questions.

@chenyuyi94
Copy link

      schedule {
        step: 0
        learning_rate: .0001
      }

i have the same error . and i can find where is the block.could you please tell me ?

@pkulzc
Copy link
Contributor

pkulzc commented Apr 5, 2018

It's in the config file, train_config/optimizer/learning_rate.

@chenyuyi94
Copy link

chenyuyi94 commented Apr 6, 2018 via email

@krassynesh
Copy link

Mr. pkulzc pls be more specific :

  1. The name of file that must be modifyed
  2. The location of this file
  3. if any link avalible for correct model zoo
    Best Regards

@pkulzc
Copy link
Contributor

pkulzc commented May 10, 2018

An simpler way is to always use the config here instead of the ones in downloaded file.

For example, If you want to train a faster_rcnn_resnet101_kitti model, use models/research/object_detection/samples/configs/faster_rcnn_resnet101_kitti.config instead of the one in the downloaded compressed file. The configs in git repo are guaranteed to be latest.

@Shiv1799
Copy link

Thank you the issue resolved but coming up with new error

(tensorflow1) C:\tensorflow1\models\research\object_detection>python train.py --logtostderr --train_dir=training/ --pipeline_config_path=training/faster_rcnn_inception_v2_pets.config
Traceback (most recent call last):
File "train.py", line 184, in
tf.app.run()
File "C:\Users\Ankit Sharma\Anaconda2\envs\tensorflow1\lib\site-packages\tensorflow\python\platform\app.py", line 126, in run
_sys.exit(main(argv))
File "train.py", line 93, in main
FLAGS.pipeline_config_path)
File "C:\tensorflow1\models\research\object_detection\utils\config_util.py", line 94, in get_configs_from_pipeline_file
text_format.Merge(proto_str, pipeline_config)
File "C:\Users\Ankit Sharma\Anaconda2\envs\tensorflow1\lib\site-packages\google\protobuf\text_format.py", line 533, in Merge
descriptor_pool=descriptor_pool)
File "C:\Users\Ankit Sharma\Anaconda2\envs\tensorflow1\lib\site-packages\google\protobuf\text_format.py", line 587, in MergeLines
return parser.MergeLines(lines, message)
File "C:\Users\Ankit Sharma\Anaconda2\envs\tensorflow1\lib\site-packages\google\protobuf\text_format.py", line 620, in MergeLines
self._ParseOrMerge(lines, message)
File "C:\Users\Ankit Sharma\Anaconda2\envs\tensorflow1\lib\site-packages\google\protobuf\text_format.py", line 635, in _ParseOrMerge
self._MergeField(tokenizer, message)
File "C:\Users\Ankit Sharma\Anaconda2\envs\tensorflow1\lib\site-packages\google\protobuf\text_format.py", line 735, in _MergeField
merger(tokenizer, message, field)
File "C:\Users\Ankit Sharma\Anaconda2\envs\tensorflow1\lib\site-packages\google\protobuf\text_format.py", line 822, in _MergeMessageField
raise tokenizer.ParseErrorPreviousToken('Expected "%s".' % (end_token,))
google.protobuf.text_format.ParseError: 140:16 : Expected "}"

@pkulzc
Copy link
Contributor

pkulzc commented May 29, 2018

@Shiv1799 apparently the config file is missing a "}" , you will want to check it.

@antoine29
Copy link

hi @pkulzc
could you help me with this?
i have a pretrained model (RCNN with Inception v2 model trained on the COCO dataset) with these files :
checkpoint model.ckpt.data-00000-of-00001
example.config model.ckpt.index
frozen_inference_graph.pb model.ckpt.meta
labels.txt saved_model/

what file should i set to the "fine_tune_checkpoint:" value in my file example.config ?

currently my example.config looks like this :

fine_tune_checkpoint: "/home/antoine/Documents/ia/HereIsWally/trained_model/checkpoint"
it is right ?
I have some mistakes in the training of the model and I suspect that it could be by this checkpoint

@pkulzc
Copy link
Contributor

pkulzc commented Jun 12, 2018

@antoine29 you should use "...trained_model/model.ckpt".
BTW, This is a stackoverflow question since it's neither a bug or a feature request. Please ask there if you have future questions like this.

I'm closing this bug since it's straightforward and has been answered.

@pkulzc pkulzc closed this as completed Jun 12, 2018
@osama191
Copy link

(tensorflow1) C:\tensorflow1\models\research\object_detection>python train.py --
logtostderr --train_dir=training/ --pipeline_config_path=training/faster_rcnn_in
ception_v2_pets.config
WARNING:tensorflow:From C:\tensorflow1\models\research\object_detection\trainer.
py:260: create_global_step (from tensorflow.contrib.framework.python.ops.variabl
es) is deprecated and will be removed in a future version.
Instructions for updating:
Please switch to tf.train.create_global_step
WARNING:tensorflow:num_readers has been reduced to 0 to match input file shards.

Traceback (most recent call last):
File "train.py", line 184, in
tf.app.run()
File "C:\Users\Osama\Anaconda3\envs\tensorflow1\lib\site-packages\tensorflow\p
ython\platform\app.py", line 126, in run
_sys.exit(main(argv))
File "train.py", line 180, in main
graph_hook_fn=graph_rewriter_fn)
File "C:\tensorflow1\models\research\object_detection\trainer.py", line 274, i
n train
train_config.prefetch_queue_capacity, data_augmentation_options)
File "C:\tensorflow1\models\research\object_detection\trainer.py", line 59, in
create_input_queue
tensor_dict = create_tensor_dict_fn()
File "train.py", line 121, in get_next
dataset_builder.build(config)).get_next()
File "C:\tensorflow1\models\research\object_detection\builders\dataset_builder
.py", line 186, in build
process_fn, config.input_path[:], input_reader_config)
File "C:\tensorflow1\models\research\object_detection\utils\dataset_util.py",
line 141, in read_dataset
sloppy=config.shuffle))
File "C:\Users\Osama\Anaconda3\envs\tensorflow1\lib\site-packages\tensorflow\p
ython\data\ops\dataset_ops.py", line 973, in apply
dataset = transformation_func(self)
File "C:\Users\Osama\Anaconda3\envs\tensorflow1\lib\site-packages\tensorflow\c
ontrib\data\python\ops\interleave_ops.py", line 76, in _apply_fn
buffer_output_elements, prefetch_input_elements)
File "C:\Users\Osama\Anaconda3\envs\tensorflow1\lib\site-packages\tensorflow\p
ython\data\ops\readers.py", line 130, in init
cycle_length, block_length)
File "C:\Users\Osama\Anaconda3\envs\tensorflow1\lib\site-packages\tensorflow\p
ython\data\ops\dataset_ops.py", line 1959, in init
super(InterleaveDataset, self).init(input_dataset, map_func)
File "C:\Users\Osama\Anaconda3\envs\tensorflow1\lib\site-packages\tensorflow\p
ython\data\ops\dataset_ops.py", line 1928, in init
self._map_func.add_to_graph(ops.get_default_graph())
File "C:\Users\Osama\Anaconda3\envs\tensorflow1\lib\site-packages\tensorflow\p
ython\framework\function.py", line 484, in add_to_graph
self._create_definition_if_needed()
File "C:\Users\Osama\Anaconda3\envs\tensorflow1\lib\site-packages\tensorflow\p
ython\framework\function.py", line 319, in _create_definition_if_needed
self._create_definition_if_needed_impl()
File "C:\Users\Osama\Anaconda3\envs\tensorflow1\lib\site-packages\tensorflow\p
ython\framework\function.py", line 336, in _create_definition_if_needed_impl
outputs = self._func(*inputs)
File "C:\Users\Osama\Anaconda3\envs\tensorflow1\lib\site-packages\tensorflow\p
ython\data\ops\dataset_ops.py", line 1916, in tf_map_func
dataset = map_func(nested_args)
File "C:\Users\Osama\Anaconda3\envs\tensorflow1\lib\site-packages\tensorflow\p
ython\data\ops\readers.py", line 196, in init
filenames = ops.convert_to_tensor(filenames, dtype=dtypes.string)
File "C:\Users\Osama\Anaconda3\envs\tensorflow1\lib\site-packages\tensorflow\p
ython\framework\ops.py", line 1014, in convert_to_tensor
as_ref=False)
File "C:\Users\Osama\Anaconda3\envs\tensorflow1\lib\site-packages\tensorflow\p
ython\framework\ops.py", line 1104, in internal_convert_to_tensor
ret = conversion_func(value, dtype=dtype, name=name, as_ref=as_ref)
File "C:\Users\Osama\Anaconda3\envs\tensorflow1\lib\site-packages\tensorflow\p
ython\framework\ops.py", line 947, in _TensorTensorConversionFunction
(dtype.name, t.dtype.name, str(t)))
ValueError: Tensor conversion requested dtype string for Tensor with dtype float
32: 'Tensor("arg0:0", shape=(), dtype=float32)'

Getting this error anyone who can help with this??

@RolandOne
Copy link

RolandOne commented Jun 23, 2018

After resolving the first step cannot be zero, I encountered this problem below.

(tensorflow) C:\Users\User\Documents\GitHub\models-master\research\object_detection>python train.py --logtostderr --train_dir=training/ --pipeline_config_path=training/faster_rcnn_inception_v2_pets.config
C:\Users\User\AppData\Local\conda\conda\envs\tensorflow\lib\site-packages\h5py_init_.py:36: FutureWarning: Conversion of the second argument of issubdtype from float to np.floating is deprecated. In future, it will be treated as np.float64 == np.dtype(float).type.
from ._conv import register_converters as _register_converters
WARNING:tensorflow:From C:\Users\User\Documents\GitHub\models-master\research\object_detection\trainer.py:260: create_global_step (from tensorflow.contrib.framework.python.ops.variables) is deprecated and will be removed in a future version.
Instructions for updating:
Please switch to tf.train.create_global_step
WARNING:tensorflow:num_readers has been reduced to 1 to match input file shards.
INFO:tensorflow:Scale of 0 disables regularizer.
INFO:tensorflow:Scale of 0 disables regularizer.
INFO:tensorflow:depth of additional conv before box predictor: 0
WARNING:tensorflow:From C:\Users\User\Documents\GitHub\models-master\research\object_detection\core\box_predictor.py:407: calling reduce_mean (from tensorflow.python.ops.math_ops) with keep_dims is deprecated and will be removed in a future version.
Instructions for updating:
keep_dims is deprecated, use keepdims instead
INFO:tensorflow:Scale of 0 disables regularizer.
WARNING:tensorflow:From C:\Users\User\Documents\GitHub\models-master\research\object_detection\core\losses.py:317: softmax_cross_entropy_with_logits (from tensorflow.python.ops.nn_ops) is deprecated and will be removed in a future version.
Instructions for updating:

Future major versions of TensorFlow will allow gradients to flow
into the labels input on backprop by default.

See @{tf.nn.softmax_cross_entropy_with_logits_v2}.

Traceback (most recent call last):
File "C:\Users\User\AppData\Local\conda\conda\envs\tensorflow\lib\site-packages\tensorflow\python\framework\op_def_library.py", line 510, in _apply_op_helper
preferred_dtype=default_dtype)
File "C:\Users\User\AppData\Local\conda\conda\envs\tensorflow\lib\site-packages\tensorflow\python\framework\ops.py", line 1104, in internal_convert_to_tensor
ret = conversion_func(value, dtype=dtype, name=name, as_ref=as_ref)
File "C:\Users\User\AppData\Local\conda\conda\envs\tensorflow\lib\site-packages\tensorflow\python\framework\constant_op.py", line 235, in _constant_tensor_conversion_function
return constant(v, dtype=dtype, name=name)
File "C:\Users\User\AppData\Local\conda\conda\envs\tensorflow\lib\site-packages\tensorflow\python\framework\constant_op.py", line 214, in constant
value, dtype=dtype, shape=shape, verify_shape=verify_shape))
File "C:\Users\User\AppData\Local\conda\conda\envs\tensorflow\lib\site-packages\tensorflow\python\framework\tensor_util.py", line 441, in make_tensor_proto
_GetDenseDimensions(values)))
ValueError: Argument must be a dense tensor: range(0, 3) - got shape [3], but wanted [].

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "C:\Users\User\AppData\Local\conda\conda\envs\tensorflow\lib\site-packages\tensorflow\python\framework\op_def_library.py", line 524, in _apply_op_helper
values, as_ref=input_arg.is_ref).dtype.name
File "C:\Users\User\AppData\Local\conda\conda\envs\tensorflow\lib\site-packages\tensorflow\python\framework\ops.py", line 1104, in internal_convert_to_tensor
ret = conversion_func(value, dtype=dtype, name=name, as_ref=as_ref)
File "C:\Users\User\AppData\Local\conda\conda\envs\tensorflow\lib\site-packages\tensorflow\python\framework\constant_op.py", line 235, in _constant_tensor_conversion_function
return constant(v, dtype=dtype, name=name)
File "C:\Users\User\AppData\Local\conda\conda\envs\tensorflow\lib\site-packages\tensorflow\python\framework\constant_op.py", line 214, in constant
value, dtype=dtype, shape=shape, verify_shape=verify_shape))
File "C:\Users\User\AppData\Local\conda\conda\envs\tensorflow\lib\site-packages\tensorflow\python\framework\tensor_util.py", line 441, in make_tensor_proto
_GetDenseDimensions(values)))
ValueError: Argument must be a dense tensor: range(0, 3) - got shape [3], but wanted [].

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "train.py", line 184, in
tf.app.run()
File "C:\Users\User\AppData\Local\conda\conda\envs\tensorflow\lib\site-packages\tensorflow\python\platform\app.py", line 126, in run
_sys.exit(main(argv))
File "train.py", line 180, in main
graph_hook_fn=graph_rewriter_fn)
File "C:\Users\User\Documents\GitHub\models-master\research\object_detection\trainer.py", line 298, in train
train_config.optimizer)
File "C:\Users\User\Documents\GitHub\models-master\research\object_detection\builders\optimizer_builder.py", line 50, in build
learning_rate = _create_learning_rate(config.learning_rate)
File "C:\Users\User\Documents\GitHub\models-master\research\object_detection\builders\optimizer_builder.py", line 109, in _create_learning_rate
learning_rate_sequence, config.warmup)
File "C:\Users\User\Documents\GitHub\models-master\research\object_detection\utils\learning_schedules.py", line 169, in manual_stepping
[0] * num_boundaries))
File "C:\Users\User\AppData\Local\conda\conda\envs\tensorflow\lib\site-packages\tensorflow\python\ops\array_ops.py", line 2681, in where
return gen_math_ops.select(condition=condition, x=x, y=y, name=name)
File "C:\Users\User\AppData\Local\conda\conda\envs\tensorflow\lib\site-packages\tensorflow\python\ops\gen_math_ops.py", line 7189, in select
"Select", condition=condition, t=x, e=y, name=name)
File "C:\Users\User\AppData\Local\conda\conda\envs\tensorflow\lib\site-packages\tensorflow\python\framework\op_def_library.py", line 528, in _apply_op_helper
(input_name, err))
ValueError: Tried to convert 't' to a tensor and failed. Error: Argument must be a dense tensor: range(0, 3) - got shape [3], but wanted [].

Any one familiar with this?

@lgribeiro
Copy link

@osama191 i have the same problem. Do you fix it ?

Help me please

WARNING:tensorflow:From /home/luiz/model_ssd/research/object_detection/trainer.py:262: create_global_step (from tensorflow.contrib.framework.python.ops.variables) is deprecated and will be removed in a future version.
Instructions for updating:
Please switch to tf.train.create_global_step
WARNING:tensorflow:num_readers has been reduced to 0 to match input file shards.
Traceback (most recent call last):
File "train.py", line 188, in
tf.app.run()
File "/home/luiz/.virtualenvs/virtual-py2/local/lib/python2.7/site-packages/tensorflow/python/platform/app.py", line 126, in run
_sys.exit(main(argv))
File "train.py", line 184, in main
graph_hook_fn=graph_rewriter_fn)
File "/home/luiz/model_ssd/research/object_detection/trainer.py", line 276, in train
train_config.prefetch_queue_capacity, data_augmentation_options)
File "/home/luiz/model_ssd/research/object_detection/trainer.py", line 59, in create_input_queue
tensor_dict = create_tensor_dict_fn()
File "train.py", line 125, in get_next
dataset_builder.build(config)).get_next()
File "/home/luiz/model_ssd/research/object_detection/builders/dataset_builder.py", line 134, in build
config.input_path[:], input_reader_config)
File "/home/luiz/model_ssd/research/object_detection/builders/dataset_builder.py", line 80, in read_dataset
sloppy=config.shuffle))
File "/home/luiz/.virtualenvs/virtual-py2/local/lib/python2.7/site-packages/tensorflow/python/data/ops/dataset_ops.py", line 973, in apply
dataset = transformation_func(self)
File "/home/luiz/.virtualenvs/virtual-py2/local/lib/python2.7/site-packages/tensorflow/contrib/data/python/ops/interleave_ops.py", line 76, in _apply_fn
buffer_output_elements, prefetch_input_elements)
File "/home/luiz/.virtualenvs/virtual-py2/local/lib/python2.7/site-packages/tensorflow/python/data/ops/readers.py", line 130, in init
cycle_length, block_length)
File "/home/luiz/.virtualenvs/virtual-py2/local/lib/python2.7/site-packages/tensorflow/python/data/ops/dataset_ops.py", line 1959, in init
super(InterleaveDataset, self).init(input_dataset, map_func)
File "/home/luiz/.virtualenvs/virtual-py2/local/lib/python2.7/site-packages/tensorflow/python/data/ops/dataset_ops.py", line 1928, in init
self._map_func.add_to_graph(ops.get_default_graph())
File "/home/luiz/.virtualenvs/virtual-py2/local/lib/python2.7/site-packages/tensorflow/python/framework/function.py", line 484, in add_to_graph
self._create_definition_if_needed()
File "/home/luiz/.virtualenvs/virtual-py2/local/lib/python2.7/site-packages/tensorflow/python/framework/function.py", line 319, in _create_definition_if_needed
self._create_definition_if_needed_impl()
File "/home/luiz/.virtualenvs/virtual-py2/local/lib/python2.7/site-packages/tensorflow/python/framework/function.py", line 336, in _create_definition_if_needed_impl
outputs = self._func(*inputs)
File "/home/luiz/.virtualenvs/virtual-py2/local/lib/python2.7/site-packages/tensorflow/python/data/ops/dataset_ops.py", line 1916, in tf_map_func
dataset = map_func(nested_args)
File "/home/luiz/.virtualenvs/virtual-py2/local/lib/python2.7/site-packages/tensorflow/python/data/ops/readers.py", line 196, in init
filenames = ops.convert_to_tensor(filenames, dtype=dtypes.string)
File "/home/luiz/.virtualenvs/virtual-py2/local/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 1014, in convert_to_tensor
as_ref=False)
File "/home/luiz/.virtualenvs/virtual-py2/local/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 1104, in internal_convert_to_tensor
ret = conversion_func(value, dtype=dtype, name=name, as_ref=as_ref)
File "/home/luiz/.virtualenvs/virtual-py2/local/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 947, in _TensorTensorConversionFunction
(dtype.name, t.dtype.name, str(t)))
ValueError: Tensor conversion requested dtype string for Tensor with dtype float32: 'Tensor("arg0:0", shape=(), dtype=float32)'

I did put the "prints" in the function above
def _TensorTensorConversionFunction(t, dtype=None, name=None, as_ref=False):
_ = name, as_ref
print('t',t,'dt', dtype ,'name', name,'ref', as_ref)
print('t.dtype',t.dtype)
print('comp', dtype.is_compatible_with(t.dtype))
if dtype and not dtype.is_compatible_with(t.dtype):
print('entry21',t, dtype , name, as_ref)
raise ValueError(
"Tensor conversion requested dtype %s for Tensor with dtype %s: %r" %
(dtype.name, t.dtype.name, str(t)))
return t

Tensor("global_step:0", shape=(), dtype=int64_ref, device=/device:CPU:0) dt None name ref ref True
t.dtype <dtype: 'int64_ref'>

print('comap', dtype.is_compatible_with(t.dtype))
AttributeError: 'NoneType' object has no attribute 'is_compatible_with'

@dshahrokhian
Copy link

A complete answer: The issue is that step: 0 in the following block is not recognized by the newer releases:

schedule {
            step: 0
            learning_rate: .0001
          }

A fix is to delete this block and changing initial_learning_rate to the learning_rate described in the block. The file that you have to modify is the one that you pass as option --pipeline_config_path to the train.py script.

@dscha09
Copy link

dscha09 commented Aug 17, 2018

@pkulzc there are like 3 blocks for schedule, which one?

@moinkhan3012
Copy link

I am getting this error

(tensorflow1) C:\tensorflow1\models\research\object_detection>python train.py --
logtostderr --train_dir=training/ --pipeline_config_path=training/faster_rcnn_in
ception_v2_pets.config
WARNING:tensorflow:From C:\tensorflow1\models\research\object_detection\trainer.
py:260: create_global_step (from tensorflow.contrib.framework.python.ops.variabl
es) is deprecated and will be removed in a future version.
Instructions for updating:
Please switch to tf.train.create_global_step
WARNING:tensorflow:num_readers has been reduced to 0 to match input file shards.

Traceback (most recent call last):
File "train.py", line 184, in
tf.app.run()
File "C:\Users\Osama\Anaconda3\envs\tensorflow1\lib\site-packages\tensorflow\p
ython\platform\app.py", line 126, in run
_sys.exit(main(argv))
File "train.py", line 180, in main
graph_hook_fn=graph_rewriter_fn)
File "C:\tensorflow1\models\research\object_detection\trainer.py", line 274, i
n train
train_config.prefetch_queue_capacity, data_augmentation_options)
File "C:\tensorflow1\models\research\object_detection\trainer.py", line 59, in
create_input_queue
tensor_dict = create_tensor_dict_fn()
File "train.py", line 121, in get_next
dataset_builder.build(config)).get_next()
File "C:\tensorflow1\models\research\object_detection\builders\dataset_builder
.py", line 186, in build
process_fn, config.input_path[:], input_reader_config)
File "C:\tensorflow1\models\research\object_detection\utils\dataset_util.py",
line 141, in read_dataset
sloppy=config.shuffle))
File "C:\Users\Osama\Anaconda3\envs\tensorflow1\lib\site-packages\tensorflow\p
ython\data\ops\dataset_ops.py", line 973, in apply
dataset = transformation_func(self)
File "C:\Users\Osama\Anaconda3\envs\tensorflow1\lib\site-packages\tensorflow\c
ontrib\data\python\ops\interleave_ops.py", line 76, in _apply_fn
buffer_output_elements, prefetch_input_elements)
File "C:\Users\Osama\Anaconda3\envs\tensorflow1\lib\site-packages\tensorflow\p
ython\data\ops\readers.py", line 130, in init
cycle_length, block_length)
File "C:\Users\Osama\Anaconda3\envs\tensorflow1\lib\site-packages\tensorflow\p
ython\data\ops\dataset_ops.py", line 1959, in init
super(InterleaveDataset, self).init(input_dataset, map_func)
File "C:\Users\Osama\Anaconda3\envs\tensorflow1\lib\site-packages\tensorflow\p
ython\data\ops\dataset_ops.py", line 1928, in init
self._map_func.add_to_graph(ops.get_default_graph())
File "C:\Users\Osama\Anaconda3\envs\tensorflow1\lib\site-packages\tensorflow\p
ython\framework\function.py", line 484, in add_to_graph
self._create_definition_if_needed()
File "C:\Users\Osama\Anaconda3\envs\tensorflow1\lib\site-packages\tensorflow\p
ython\framework\function.py", line 319, in _create_definition_if_needed
self._create_definition_if_needed_impl()
File "C:\Users\Osama\Anaconda3\envs\tensorflow1\lib\site-packages\tensorflow\p
ython\framework\function.py", line 336, in _create_definition_if_needed_impl
outputs = self._func(*inputs)
File "C:\Users\Osama\Anaconda3\envs\tensorflow1\lib\site-packages\tensorflow\p
ython\data\ops\dataset_ops.py", line 1916, in tf_map_func
dataset = map_func(nested_args)
File "C:\Users\Osama\Anaconda3\envs\tensorflow1\lib\site-packages\tensorflow\p
ython\data\ops\readers.py", line 196, in init
filenames = ops.convert_to_tensor(filenames, dtype=dtypes.string)
File "C:\Users\Osama\Anaconda3\envs\tensorflow1\lib\site-packages\tensorflow\p
ython\framework\ops.py", line 1014, in convert_to_tensor
as_ref=False)
File "C:\Users\Osama\Anaconda3\envs\tensorflow1\lib\site-packages\tensorflow\p
ython\framework\ops.py", line 1104, in internal_convert_to_tensor
ret = conversion_func(value, dtype=dtype, name=name, as_ref=as_ref)
File "C:\Users\Osama\Anaconda3\envs\tensorflow1\lib\site-packages\tensorflow\p
ython\framework\ops.py", line 947, in _TensorTensorConversionFunction
(dtype.name, t.dtype.name, str(t)))
ValueError: Tensor conversion requested dtype string for Tensor with dtype float
32: 'Tensor("arg0:0", shape=(), dtype=float32)'

Can anyone explain why I am getting this error?

1 similar comment
@moinkhan3012
Copy link

I am getting this error

(tensorflow1) C:\tensorflow1\models\research\object_detection>python train.py --
logtostderr --train_dir=training/ --pipeline_config_path=training/faster_rcnn_in
ception_v2_pets.config
WARNING:tensorflow:From C:\tensorflow1\models\research\object_detection\trainer.
py:260: create_global_step (from tensorflow.contrib.framework.python.ops.variabl
es) is deprecated and will be removed in a future version.
Instructions for updating:
Please switch to tf.train.create_global_step
WARNING:tensorflow:num_readers has been reduced to 0 to match input file shards.

Traceback (most recent call last):
File "train.py", line 184, in
tf.app.run()
File "C:\Users\Osama\Anaconda3\envs\tensorflow1\lib\site-packages\tensorflow\p
ython\platform\app.py", line 126, in run
_sys.exit(main(argv))
File "train.py", line 180, in main
graph_hook_fn=graph_rewriter_fn)
File "C:\tensorflow1\models\research\object_detection\trainer.py", line 274, i
n train
train_config.prefetch_queue_capacity, data_augmentation_options)
File "C:\tensorflow1\models\research\object_detection\trainer.py", line 59, in
create_input_queue
tensor_dict = create_tensor_dict_fn()
File "train.py", line 121, in get_next
dataset_builder.build(config)).get_next()
File "C:\tensorflow1\models\research\object_detection\builders\dataset_builder
.py", line 186, in build
process_fn, config.input_path[:], input_reader_config)
File "C:\tensorflow1\models\research\object_detection\utils\dataset_util.py",
line 141, in read_dataset
sloppy=config.shuffle))
File "C:\Users\Osama\Anaconda3\envs\tensorflow1\lib\site-packages\tensorflow\p
ython\data\ops\dataset_ops.py", line 973, in apply
dataset = transformation_func(self)
File "C:\Users\Osama\Anaconda3\envs\tensorflow1\lib\site-packages\tensorflow\c
ontrib\data\python\ops\interleave_ops.py", line 76, in _apply_fn
buffer_output_elements, prefetch_input_elements)
File "C:\Users\Osama\Anaconda3\envs\tensorflow1\lib\site-packages\tensorflow\p
ython\data\ops\readers.py", line 130, in init
cycle_length, block_length)
File "C:\Users\Osama\Anaconda3\envs\tensorflow1\lib\site-packages\tensorflow\p
ython\data\ops\dataset_ops.py", line 1959, in init
super(InterleaveDataset, self).init(input_dataset, map_func)
File "C:\Users\Osama\Anaconda3\envs\tensorflow1\lib\site-packages\tensorflow\p
ython\data\ops\dataset_ops.py", line 1928, in init
self._map_func.add_to_graph(ops.get_default_graph())
File "C:\Users\Osama\Anaconda3\envs\tensorflow1\lib\site-packages\tensorflow\p
ython\framework\function.py", line 484, in add_to_graph
self._create_definition_if_needed()
File "C:\Users\Osama\Anaconda3\envs\tensorflow1\lib\site-packages\tensorflow\p
ython\framework\function.py", line 319, in _create_definition_if_needed
self._create_definition_if_needed_impl()
File "C:\Users\Osama\Anaconda3\envs\tensorflow1\lib\site-packages\tensorflow\p
ython\framework\function.py", line 336, in _create_definition_if_needed_impl
outputs = self._func(*inputs)
File "C:\Users\Osama\Anaconda3\envs\tensorflow1\lib\site-packages\tensorflow\p
ython\data\ops\dataset_ops.py", line 1916, in tf_map_func
dataset = map_func(nested_args)
File "C:\Users\Osama\Anaconda3\envs\tensorflow1\lib\site-packages\tensorflow\p
ython\data\ops\readers.py", line 196, in init
filenames = ops.convert_to_tensor(filenames, dtype=dtypes.string)
File "C:\Users\Osama\Anaconda3\envs\tensorflow1\lib\site-packages\tensorflow\p
ython\framework\ops.py", line 1014, in convert_to_tensor
as_ref=False)
File "C:\Users\Osama\Anaconda3\envs\tensorflow1\lib\site-packages\tensorflow\p
ython\framework\ops.py", line 1104, in internal_convert_to_tensor
ret = conversion_func(value, dtype=dtype, name=name, as_ref=as_ref)
File "C:\Users\Osama\Anaconda3\envs\tensorflow1\lib\site-packages\tensorflow\p
ython\framework\ops.py", line 947, in _TensorTensorConversionFunction
(dtype.name, t.dtype.name, str(t)))
ValueError: Tensor conversion requested dtype string for Tensor with dtype float
32: 'Tensor("arg0:0", shape=(), dtype=float32)'

Can anyone explain why I am getting this error?

@GitWR
Copy link

GitWR commented Sep 30, 2018

When I according to the suggestion of @pkulzc to remove that block, the problem can be solved, however, a new problem is followed, which is:

File "D:\Anaconda2\envs\tensorflow1\lib\site-packages\tensorflow\python\ops\script_ops.py", line 158, in call
ret = func(*args)

File "D:\tensorflow1\models\research\object_detection\metrics\coco_evaluation.py", line 346, in first_value_func
self._metrics = self.evaluate()

File "D:\tensorflow1\models\research\object_detection\metrics\coco_evaluation.py", line 212, in evaluate
all_metrics_per_category=self._all_metrics_per_category)

File "D:\tensorflow1\models\research\object_detection\metrics\coco_tools.py", line 236, in ComputeMetrics
self.evaluate()

File "D:\tensorflow1\models\research\pycocotools\cocoeval.py", line 156, in evaluate
for catId in catIds

File "D:\tensorflow1\models\research\pycocotools\cocoeval.py", line 158, in
for imgId in p.imgIds

File "D:\tensorflow1\models\research\pycocotools\cocoeval.py", line 264, in evaluateImg
ious = self.ious[imgId, catId][:, gtind] if len(self.ious[imgId, catId]) > 0 else self.ious[imgId, catId]

TypeError: object of type 'NoneType' has no len()

I hope you can give me some ways to solve it, thanks very much!

@Vinzia
Copy link

Vinzia commented Mar 10, 2019

image

Hi, can someone help me with these errors?

@monadhif
Copy link

monadhif commented Jun 6, 2019

I deleted the block :
schedule {
step: 0
learning_rate: .0002
}
but i found an other error "Attempting to fetch value instead of handling error Internal: no supported devices found for platform CUDA
Abandon (core dumped)"


WARNING: The TensorFlow contrib module will not be included in TensorFlow 2.0.
For more information, please see:

WARNING:tensorflow:From /home/user/anaconda3/lib/python3.7/site-packages/tensorflow/python/platform/app.py:125: main (from main) is deprecated and will be removed in a future version.
Instructions for updating:
Use object_detection/model_main.py.
WARNING:tensorflow:From /home/user/great/model/research/object_detection/legacy/trainer.py:266: create_global_step (from tensorflow.contrib.framework.python.ops.variables) is deprecated and will be removed in a future version.
Instructions for updating:
Please switch to tf.train.create_global_step
WARNING:tensorflow:From /home/user/anaconda3/lib/python3.7/site-packages/tensorflow/python/framework/op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.
Instructions for updating:
Colocations handled automatically by placer.
WARNING:tensorflow:num_readers has been reduced to 1 to match input file shards.
WARNING:tensorflow:From /home/user/great/model/research/object_detection/builders/dataset_builder.py:80: parallel_interleave (from tensorflow.contrib.data.python.ops.interleave_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.data.experimental.parallel_interleave(...).
WARNING:tensorflow:From /home/user/great/model/research/object_detection/legacy/trainer.py:65: to_float (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.cast instead.
WARNING:tensorflow:From /home/user/great/model/research/object_detection/core/batcher.py:96: batch (from tensorflow.python.training.input) is deprecated and will be removed in a future version.
Instructions for updating:
Queue-based input pipelines have been replaced by tf.data. Use tf.data.Dataset.batch(batch_size) (or padded_batch(...) if dynamic_pad=True).
WARNING:tensorflow:From /home/user/anaconda3/lib/python3.7/site-packages/tensorflow/python/training/input.py:753: QueueRunner.init (from tensorflow.python.training.queue_runner_impl) is deprecated and will be removed in a future version.
Instructions for updating:
To construct input pipelines, use the tf.data module.
WARNING:tensorflow:From /home/user/anaconda3/lib/python3.7/site-packages/tensorflow/python/training/input.py:753: add_queue_runner (from tensorflow.python.training.queue_runner_impl) is deprecated and will be removed in a future version.
Instructions for updating:
To construct input pipelines, use the tf.data module.
WARNING:tensorflow:From /home/user/anaconda3/lib/python3.7/site-packages/tensorflow/python/ops/image_ops_impl.py:1055: to_int32 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.cast instead.
INFO:tensorflow:Scale of 0 disables regularizer.
INFO:tensorflow:Scale of 0 disables regularizer.
INFO:tensorflow:depth of additional conv before box predictor: 0
WARNING:tensorflow:From /home/user/anaconda3/lib/python3.7/site-packages/tensorflow/contrib/layers/python/layers/layers.py:1624: flatten (from tensorflow.python.layers.core) is deprecated and will be removed in a future version.
Instructions for updating:
Use keras.layers.flatten instead.
INFO:tensorflow:Scale of 0 disables regularizer.
INFO:tensorflow:Scale of 0 disables regularizer.
WARNING:tensorflow:From /home/user/great/model/research/object_detection/core/losses.py:345: softmax_cross_entropy_with_logits (from tensorflow.python.ops.nn_ops) is deprecated and will be removed in a future version.
Instructions for updating:

Future major versions of TensorFlow will allow gradients to flow
into the labels input on backprop by default.

See tf.nn.softmax_cross_entropy_with_logits_v2.

/home/user/anaconda3/lib/python3.7/site-packages/tensorflow/python/ops/gradients_impl.py:110: UserWarning: Converting sparse IndexedSlices to a dense Tensor of unknown shape. This may consume a large amount of memory.
"Converting sparse IndexedSlices to a dense Tensor of unknown shape. "
WARNING:tensorflow:From /home/user/great/model/research/object_detection/meta_architectures/faster_rcnn_meta_arch.py:2298: get_or_create_global_step (from tensorflow.contrib.framework.python.ops.variables) is deprecated and will be removed in a future version.
Instructions for updating:
Please switch to tf.train.get_or_create_global_step
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Conv2d_1a_7x7/BatchNorm/beta/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Conv2d_1a_7x7/BatchNorm/gamma/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Conv2d_1a_7x7/depthwise_weights/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Conv2d_1a_7x7/pointwise_weights/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Conv2d_2b_1x1/BatchNorm/beta/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Conv2d_2b_1x1/BatchNorm/gamma/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Conv2d_2b_1x1/weights/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Conv2d_2c_3x3/BatchNorm/beta/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Conv2d_2c_3x3/BatchNorm/gamma/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Conv2d_2c_3x3/weights/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_3b/Branch_0/Conv2d_0a_1x1/BatchNorm/beta/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_3b/Branch_0/Conv2d_0a_1x1/BatchNorm/gamma/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_3b/Branch_0/Conv2d_0a_1x1/weights/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_3b/Branch_1/Conv2d_0a_1x1/BatchNorm/beta/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_3b/Branch_1/Conv2d_0a_1x1/BatchNorm/gamma/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_3b/Branch_1/Conv2d_0a_1x1/weights/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_3b/Branch_1/Conv2d_0b_3x3/BatchNorm/beta/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_3b/Branch_1/Conv2d_0b_3x3/BatchNorm/gamma/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_3b/Branch_1/Conv2d_0b_3x3/weights/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_3b/Branch_2/Conv2d_0a_1x1/BatchNorm/beta/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_3b/Branch_2/Conv2d_0a_1x1/BatchNorm/gamma/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_3b/Branch_2/Conv2d_0a_1x1/weights/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_3b/Branch_2/Conv2d_0b_3x3/BatchNorm/beta/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_3b/Branch_2/Conv2d_0b_3x3/BatchNorm/gamma/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_3b/Branch_2/Conv2d_0b_3x3/weights/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_3b/Branch_2/Conv2d_0c_3x3/BatchNorm/beta/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_3b/Branch_2/Conv2d_0c_3x3/BatchNorm/gamma/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_3b/Branch_2/Conv2d_0c_3x3/weights/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_3b/Branch_3/Conv2d_0b_1x1/BatchNorm/beta/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_3b/Branch_3/Conv2d_0b_1x1/BatchNorm/gamma/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_3b/Branch_3/Conv2d_0b_1x1/weights/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_3c/Branch_0/Conv2d_0a_1x1/BatchNorm/beta/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_3c/Branch_0/Conv2d_0a_1x1/BatchNorm/gamma/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_3c/Branch_0/Conv2d_0a_1x1/weights/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_3c/Branch_1/Conv2d_0a_1x1/BatchNorm/beta/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_3c/Branch_1/Conv2d_0a_1x1/BatchNorm/gamma/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_3c/Branch_1/Conv2d_0a_1x1/weights/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_3c/Branch_1/Conv2d_0b_3x3/BatchNorm/beta/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_3c/Branch_1/Conv2d_0b_3x3/BatchNorm/gamma/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_3c/Branch_1/Conv2d_0b_3x3/weights/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_3c/Branch_2/Conv2d_0a_1x1/BatchNorm/beta/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_3c/Branch_2/Conv2d_0a_1x1/BatchNorm/gamma/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_3c/Branch_2/Conv2d_0a_1x1/weights/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_3c/Branch_2/Conv2d_0b_3x3/BatchNorm/beta/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_3c/Branch_2/Conv2d_0b_3x3/BatchNorm/gamma/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_3c/Branch_2/Conv2d_0b_3x3/weights/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_3c/Branch_2/Conv2d_0c_3x3/BatchNorm/beta/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_3c/Branch_2/Conv2d_0c_3x3/BatchNorm/gamma/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_3c/Branch_2/Conv2d_0c_3x3/weights/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_3c/Branch_3/Conv2d_0b_1x1/BatchNorm/beta/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_3c/Branch_3/Conv2d_0b_1x1/BatchNorm/gamma/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_3c/Branch_3/Conv2d_0b_1x1/weights/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4a/Branch_0/Conv2d_0a_1x1/BatchNorm/beta/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4a/Branch_0/Conv2d_0a_1x1/BatchNorm/gamma/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4a/Branch_0/Conv2d_0a_1x1/weights/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4a/Branch_0/Conv2d_1a_3x3/BatchNorm/beta/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4a/Branch_0/Conv2d_1a_3x3/BatchNorm/gamma/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4a/Branch_0/Conv2d_1a_3x3/weights/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4a/Branch_1/Conv2d_0a_1x1/BatchNorm/beta/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4a/Branch_1/Conv2d_0a_1x1/BatchNorm/gamma/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4a/Branch_1/Conv2d_0a_1x1/weights/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4a/Branch_1/Conv2d_0b_3x3/BatchNorm/beta/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4a/Branch_1/Conv2d_0b_3x3/BatchNorm/gamma/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4a/Branch_1/Conv2d_0b_3x3/weights/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4a/Branch_1/Conv2d_1a_3x3/BatchNorm/beta/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4a/Branch_1/Conv2d_1a_3x3/BatchNorm/gamma/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4a/Branch_1/Conv2d_1a_3x3/weights/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4b/Branch_0/Conv2d_0a_1x1/BatchNorm/beta/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4b/Branch_0/Conv2d_0a_1x1/BatchNorm/gamma/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4b/Branch_0/Conv2d_0a_1x1/weights/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4b/Branch_1/Conv2d_0a_1x1/BatchNorm/beta/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4b/Branch_1/Conv2d_0a_1x1/BatchNorm/gamma/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4b/Branch_1/Conv2d_0a_1x1/weights/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4b/Branch_1/Conv2d_0b_3x3/BatchNorm/beta/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4b/Branch_1/Conv2d_0b_3x3/BatchNorm/gamma/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4b/Branch_1/Conv2d_0b_3x3/weights/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4b/Branch_2/Conv2d_0a_1x1/BatchNorm/beta/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4b/Branch_2/Conv2d_0a_1x1/BatchNorm/gamma/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4b/Branch_2/Conv2d_0a_1x1/weights/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4b/Branch_2/Conv2d_0b_3x3/BatchNorm/beta/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4b/Branch_2/Conv2d_0b_3x3/BatchNorm/gamma/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4b/Branch_2/Conv2d_0b_3x3/weights/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4b/Branch_2/Conv2d_0c_3x3/BatchNorm/beta/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4b/Branch_2/Conv2d_0c_3x3/BatchNorm/gamma/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4b/Branch_2/Conv2d_0c_3x3/weights/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4b/Branch_3/Conv2d_0b_1x1/BatchNorm/beta/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4b/Branch_3/Conv2d_0b_1x1/BatchNorm/gamma/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4b/Branch_3/Conv2d_0b_1x1/weights/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4c/Branch_0/Conv2d_0a_1x1/BatchNorm/beta/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4c/Branch_0/Conv2d_0a_1x1/BatchNorm/gamma/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4c/Branch_0/Conv2d_0a_1x1/weights/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4c/Branch_1/Conv2d_0a_1x1/BatchNorm/beta/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4c/Branch_1/Conv2d_0a_1x1/BatchNorm/gamma/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4c/Branch_1/Conv2d_0a_1x1/weights/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4c/Branch_1/Conv2d_0b_3x3/BatchNorm/beta/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4c/Branch_1/Conv2d_0b_3x3/BatchNorm/gamma/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4c/Branch_1/Conv2d_0b_3x3/weights/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4c/Branch_2/Conv2d_0a_1x1/BatchNorm/beta/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4c/Branch_2/Conv2d_0a_1x1/BatchNorm/gamma/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4c/Branch_2/Conv2d_0a_1x1/weights/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4c/Branch_2/Conv2d_0b_3x3/BatchNorm/beta/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4c/Branch_2/Conv2d_0b_3x3/BatchNorm/gamma/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4c/Branch_2/Conv2d_0b_3x3/weights/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4c/Branch_2/Conv2d_0c_3x3/BatchNorm/beta/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4c/Branch_2/Conv2d_0c_3x3/BatchNorm/gamma/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4c/Branch_2/Conv2d_0c_3x3/weights/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4c/Branch_3/Conv2d_0b_1x1/BatchNorm/beta/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4c/Branch_3/Conv2d_0b_1x1/BatchNorm/gamma/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4c/Branch_3/Conv2d_0b_1x1/weights/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4d/Branch_0/Conv2d_0a_1x1/BatchNorm/beta/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4d/Branch_0/Conv2d_0a_1x1/BatchNorm/gamma/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4d/Branch_0/Conv2d_0a_1x1/weights/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4d/Branch_1/Conv2d_0a_1x1/BatchNorm/beta/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4d/Branch_1/Conv2d_0a_1x1/BatchNorm/gamma/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4d/Branch_1/Conv2d_0a_1x1/weights/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4d/Branch_1/Conv2d_0b_3x3/BatchNorm/beta/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4d/Branch_1/Conv2d_0b_3x3/BatchNorm/gamma/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4d/Branch_1/Conv2d_0b_3x3/weights/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4d/Branch_2/Conv2d_0a_1x1/BatchNorm/beta/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4d/Branch_2/Conv2d_0a_1x1/BatchNorm/gamma/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4d/Branch_2/Conv2d_0a_1x1/weights/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4d/Branch_2/Conv2d_0b_3x3/BatchNorm/beta/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4d/Branch_2/Conv2d_0b_3x3/BatchNorm/gamma/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4d/Branch_2/Conv2d_0b_3x3/weights/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4d/Branch_2/Conv2d_0c_3x3/BatchNorm/beta/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4d/Branch_2/Conv2d_0c_3x3/BatchNorm/gamma/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4d/Branch_2/Conv2d_0c_3x3/weights/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4d/Branch_3/Conv2d_0b_1x1/BatchNorm/beta/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4d/Branch_3/Conv2d_0b_1x1/BatchNorm/gamma/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4d/Branch_3/Conv2d_0b_1x1/weights/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4e/Branch_0/Conv2d_0a_1x1/BatchNorm/beta/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4e/Branch_0/Conv2d_0a_1x1/BatchNorm/gamma/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4e/Branch_0/Conv2d_0a_1x1/weights/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4e/Branch_1/Conv2d_0a_1x1/BatchNorm/beta/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4e/Branch_1/Conv2d_0a_1x1/BatchNorm/gamma/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4e/Branch_1/Conv2d_0a_1x1/weights/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4e/Branch_1/Conv2d_0b_3x3/BatchNorm/beta/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4e/Branch_1/Conv2d_0b_3x3/BatchNorm/gamma/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4e/Branch_1/Conv2d_0b_3x3/weights/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4e/Branch_2/Conv2d_0a_1x1/BatchNorm/beta/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4e/Branch_2/Conv2d_0a_1x1/BatchNorm/gamma/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4e/Branch_2/Conv2d_0a_1x1/weights/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4e/Branch_2/Conv2d_0b_3x3/BatchNorm/beta/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4e/Branch_2/Conv2d_0b_3x3/BatchNorm/gamma/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4e/Branch_2/Conv2d_0b_3x3/weights/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4e/Branch_2/Conv2d_0c_3x3/BatchNorm/beta/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4e/Branch_2/Conv2d_0c_3x3/BatchNorm/gamma/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4e/Branch_2/Conv2d_0c_3x3/weights/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4e/Branch_3/Conv2d_0b_1x1/BatchNorm/beta/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4e/Branch_3/Conv2d_0b_1x1/BatchNorm/gamma/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_4e/Branch_3/Conv2d_0b_1x1/weights/Momentum] is not available in checkpoint
WARNING:root:Variable [SecondStageFeatureExtractor/InceptionV2/Mixed_5a/Branch_0/Conv2d_0a_1x1/BatchNorm/beta/Momentum] is not available in checkpoint
WARNING:root:Variable [SecondStageFeatureExtractor/InceptionV2/Mixed_5a/Branch_0/Conv2d_0a_1x1/BatchNorm/gamma/Momentum] is not available in checkpoint
WARNING:root:Variable [SecondStageFeatureExtractor/InceptionV2/Mixed_5a/Branch_0/Conv2d_0a_1x1/weights/Momentum] is not available in checkpoint
WARNING:root:Variable [SecondStageFeatureExtractor/InceptionV2/Mixed_5a/Branch_0/Conv2d_1a_3x3/BatchNorm/beta/Momentum] is not available in checkpoint
WARNING:root:Variable [SecondStageFeatureExtractor/InceptionV2/Mixed_5a/Branch_0/Conv2d_1a_3x3/BatchNorm/gamma/Momentum] is not available in checkpoint
WARNING:root:Variable [SecondStageFeatureExtractor/InceptionV2/Mixed_5a/Branch_0/Conv2d_1a_3x3/weights/Momentum] is not available in checkpoint
WARNING:root:Variable [SecondStageFeatureExtractor/InceptionV2/Mixed_5a/Branch_1/Conv2d_0a_1x1/BatchNorm/beta/Momentum] is not available in checkpoint
WARNING:root:Variable [SecondStageFeatureExtractor/InceptionV2/Mixed_5a/Branch_1/Conv2d_0a_1x1/BatchNorm/gamma/Momentum] is not available in checkpoint
WARNING:root:Variable [SecondStageFeatureExtractor/InceptionV2/Mixed_5a/Branch_1/Conv2d_0a_1x1/weights/Momentum] is not available in checkpoint
WARNING:root:Variable [SecondStageFeatureExtractor/InceptionV2/Mixed_5a/Branch_1/Conv2d_0b_3x3/BatchNorm/beta/Momentum] is not available in checkpoint
WARNING:root:Variable [SecondStageFeatureExtractor/InceptionV2/Mixed_5a/Branch_1/Conv2d_0b_3x3/BatchNorm/gamma/Momentum] is not available in checkpoint
WARNING:root:Variable [SecondStageFeatureExtractor/InceptionV2/Mixed_5a/Branch_1/Conv2d_0b_3x3/weights/Momentum] is not available in checkpoint
WARNING:root:Variable [SecondStageFeatureExtractor/InceptionV2/Mixed_5a/Branch_1/Conv2d_1a_3x3/BatchNorm/beta/Momentum] is not available in checkpoint
WARNING:root:Variable [SecondStageFeatureExtractor/InceptionV2/Mixed_5a/Branch_1/Conv2d_1a_3x3/BatchNorm/gamma/Momentum] is not available in checkpoint
WARNING:root:Variable [SecondStageFeatureExtractor/InceptionV2/Mixed_5a/Branch_1/Conv2d_1a_3x3/weights/Momentum] is not available in checkpoint
WARNING:root:Variable [SecondStageFeatureExtractor/InceptionV2/Mixed_5b/Branch_0/Conv2d_0a_1x1/BatchNorm/beta/Momentum] is not available in checkpoint
WARNING:root:Variable [SecondStageFeatureExtractor/InceptionV2/Mixed_5b/Branch_0/Conv2d_0a_1x1/BatchNorm/gamma/Momentum] is not available in checkpoint
WARNING:root:Variable [SecondStageFeatureExtractor/InceptionV2/Mixed_5b/Branch_0/Conv2d_0a_1x1/weights/Momentum] is not available in checkpoint
WARNING:root:Variable [SecondStageFeatureExtractor/InceptionV2/Mixed_5b/Branch_1/Conv2d_0a_1x1/BatchNorm/beta/Momentum] is not available in checkpoint
WARNING:root:Variable [SecondStageFeatureExtractor/InceptionV2/Mixed_5b/Branch_1/Conv2d_0a_1x1/BatchNorm/gamma/Momentum] is not available in checkpoint
WARNING:root:Variable [SecondStageFeatureExtractor/InceptionV2/Mixed_5b/Branch_1/Conv2d_0a_1x1/weights/Momentum] is not available in checkpoint
WARNING:root:Variable [SecondStageFeatureExtractor/InceptionV2/Mixed_5b/Branch_1/Conv2d_0b_3x3/BatchNorm/beta/Momentum] is not available in checkpoint
WARNING:root:Variable [SecondStageFeatureExtractor/InceptionV2/Mixed_5b/Branch_1/Conv2d_0b_3x3/BatchNorm/gamma/Momentum] is not available in checkpoint
WARNING:root:Variable [SecondStageFeatureExtractor/InceptionV2/Mixed_5b/Branch_1/Conv2d_0b_3x3/weights/Momentum] is not available in checkpoint
WARNING:root:Variable [SecondStageFeatureExtractor/InceptionV2/Mixed_5b/Branch_2/Conv2d_0a_1x1/BatchNorm/beta/Momentum] is not available in checkpoint
WARNING:root:Variable [SecondStageFeatureExtractor/InceptionV2/Mixed_5b/Branch_2/Conv2d_0a_1x1/BatchNorm/gamma/Momentum] is not available in checkpoint
WARNING:root:Variable [SecondStageFeatureExtractor/InceptionV2/Mixed_5b/Branch_2/Conv2d_0a_1x1/weights/Momentum] is not available in checkpoint
WARNING:root:Variable [SecondStageFeatureExtractor/InceptionV2/Mixed_5b/Branch_2/Conv2d_0b_3x3/BatchNorm/beta/Momentum] is not available in checkpoint
WARNING:root:Variable [SecondStageFeatureExtractor/InceptionV2/Mixed_5b/Branch_2/Conv2d_0b_3x3/BatchNorm/gamma/Momentum] is not available in checkpoint
WARNING:root:Variable [SecondStageFeatureExtractor/InceptionV2/Mixed_5b/Branch_2/Conv2d_0b_3x3/weights/Momentum] is not available in checkpoint
WARNING:root:Variable [SecondStageFeatureExtractor/InceptionV2/Mixed_5b/Branch_2/Conv2d_0c_3x3/BatchNorm/beta/Momentum] is not available in checkpoint
WARNING:root:Variable [SecondStageFeatureExtractor/InceptionV2/Mixed_5b/Branch_2/Conv2d_0c_3x3/BatchNorm/gamma/Momentum] is not available in checkpoint
WARNING:root:Variable [SecondStageFeatureExtractor/InceptionV2/Mixed_5b/Branch_2/Conv2d_0c_3x3/weights/Momentum] is not available in checkpoint
WARNING:root:Variable [SecondStageFeatureExtractor/InceptionV2/Mixed_5b/Branch_3/Conv2d_0b_1x1/BatchNorm/beta/Momentum] is not available in checkpoint
WARNING:root:Variable [SecondStageFeatureExtractor/InceptionV2/Mixed_5b/Branch_3/Conv2d_0b_1x1/BatchNorm/gamma/Momentum] is not available in checkpoint
WARNING:root:Variable [SecondStageFeatureExtractor/InceptionV2/Mixed_5b/Branch_3/Conv2d_0b_1x1/weights/Momentum] is not available in checkpoint
WARNING:root:Variable [SecondStageFeatureExtractor/InceptionV2/Mixed_5c/Branch_0/Conv2d_0a_1x1/BatchNorm/beta/Momentum] is not available in checkpoint
WARNING:root:Variable [SecondStageFeatureExtractor/InceptionV2/Mixed_5c/Branch_0/Conv2d_0a_1x1/BatchNorm/gamma/Momentum] is not available in checkpoint
WARNING:root:Variable [SecondStageFeatureExtractor/InceptionV2/Mixed_5c/Branch_0/Conv2d_0a_1x1/weights/Momentum] is not available in checkpoint
WARNING:root:Variable [SecondStageFeatureExtractor/InceptionV2/Mixed_5c/Branch_1/Conv2d_0a_1x1/BatchNorm/beta/Momentum] is not available in checkpoint
WARNING:root:Variable [SecondStageFeatureExtractor/InceptionV2/Mixed_5c/Branch_1/Conv2d_0a_1x1/BatchNorm/gamma/Momentum] is not available in checkpoint
WARNING:root:Variable [SecondStageFeatureExtractor/InceptionV2/Mixed_5c/Branch_1/Conv2d_0a_1x1/weights/Momentum] is not available in checkpoint
WARNING:root:Variable [SecondStageFeatureExtractor/InceptionV2/Mixed_5c/Branch_1/Conv2d_0b_3x3/BatchNorm/beta/Momentum] is not available in checkpoint
WARNING:root:Variable [SecondStageFeatureExtractor/InceptionV2/Mixed_5c/Branch_1/Conv2d_0b_3x3/BatchNorm/gamma/Momentum] is not available in checkpoint
WARNING:root:Variable [SecondStageFeatureExtractor/InceptionV2/Mixed_5c/Branch_1/Conv2d_0b_3x3/weights/Momentum] is not available in checkpoint
WARNING:root:Variable [SecondStageFeatureExtractor/InceptionV2/Mixed_5c/Branch_2/Conv2d_0a_1x1/BatchNorm/beta/Momentum] is not available in checkpoint
WARNING:root:Variable [SecondStageFeatureExtractor/InceptionV2/Mixed_5c/Branch_2/Conv2d_0a_1x1/BatchNorm/gamma/Momentum] is not available in checkpoint
WARNING:root:Variable [SecondStageFeatureExtractor/InceptionV2/Mixed_5c/Branch_2/Conv2d_0a_1x1/weights/Momentum] is not available in checkpoint
WARNING:root:Variable [SecondStageFeatureExtractor/InceptionV2/Mixed_5c/Branch_2/Conv2d_0b_3x3/BatchNorm/beta/Momentum] is not available in checkpoint
WARNING:root:Variable [SecondStageFeatureExtractor/InceptionV2/Mixed_5c/Branch_2/Conv2d_0b_3x3/BatchNorm/gamma/Momentum] is not available in checkpoint
WARNING:root:Variable [SecondStageFeatureExtractor/InceptionV2/Mixed_5c/Branch_2/Conv2d_0b_3x3/weights/Momentum] is not available in checkpoint
WARNING:root:Variable [SecondStageFeatureExtractor/InceptionV2/Mixed_5c/Branch_2/Conv2d_0c_3x3/BatchNorm/beta/Momentum] is not available in checkpoint
WARNING:root:Variable [SecondStageFeatureExtractor/InceptionV2/Mixed_5c/Branch_2/Conv2d_0c_3x3/BatchNorm/gamma/Momentum] is not available in checkpoint
WARNING:root:Variable [SecondStageFeatureExtractor/InceptionV2/Mixed_5c/Branch_2/Conv2d_0c_3x3/weights/Momentum] is not available in checkpoint
WARNING:root:Variable [SecondStageFeatureExtractor/InceptionV2/Mixed_5c/Branch_3/Conv2d_0b_1x1/BatchNorm/beta/Momentum] is not available in checkpoint
WARNING:root:Variable [SecondStageFeatureExtractor/InceptionV2/Mixed_5c/Branch_3/Conv2d_0b_1x1/BatchNorm/gamma/Momentum] is not available in checkpoint
WARNING:root:Variable [SecondStageFeatureExtractor/InceptionV2/Mixed_5c/Branch_3/Conv2d_0b_1x1/weights/Momentum] is not available in checkpoint
WARNING:tensorflow:From /home/user/anaconda3/lib/python3.7/site-packages/tensorflow/contrib/slim/python/slim/learning.py:737: Supervisor.init (from tensorflow.python.training.supervisor) is deprecated and will be removed in a future version.
Instructions for updating:
Please switch to tf.train.MonitoredTrainingSession
WARNING:tensorflow:From /home/user/anaconda3/lib/python3.7/site-packages/tensorflow/contrib/slim/python/slim/learning.py:737: Supervisor.init (from tensorflow.python.training.supervisor) is deprecated and will be removed in a future version.
Instructions for updating:
Please switch to tf.train.MonitoredTrainingSession
2019-06-06 15:50:48.647811: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2494300000 Hz
2019-06-06 15:50:48.648157: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x559ee428ed20 executing computations on platform Host. Devices:
2019-06-06 15:50:48.648256: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (0): ,
2019-06-06 15:50:48.868474: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:998] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-06-06 15:50:48.869202: I tensorflow/compiler/xla/service/platform_util.cc:194] StreamExecutor cuda device (0) is of insufficient compute capability: 3.5 required, device is 2.1
2019-06-06 15:50:48.869415: F tensorflow/stream_executor/lib/statusor.cc:34] Attempting to fetch value instead of handling error Internal: no supported devices found for platform CUDA
Abandon (core dumped)

@bharath5673
Copy link

#tessor flow custom training

ERROR:raise ValueError('First step cannot be zero.') ValueError: First step cannot be zero.

SOLUTION: object_detection\training\ .config

train_config: {
batch_size: 1
optimizer {
momentum_optimizer: {
learning_rate: {
manual_step_learning_rate {
initial_learning_rate: 0.0002
schedule {
step: 900000
learning_rate: .00002
}
schedule {
step: 1200000
learning_rate: .000002
}
}
}
momentum_optimizer_value: 0.9
}
use_moving_average: false
}

asetkn added a commit to asetkn/TensorFlow-Object-Detection-API-Tutorial-Train-Multiple-Objects-Windows-10 that referenced this issue Jul 26, 2019
Updated training/faster_rcnn_inception_v2_pets.config due to updates at Tensorflow Object Detection API. 
Thank you for the tutorial with great details!
Tried following it and training crashed with the following error:
raise ValueError('First step cannot be zero.')
Turns out they updated the code-  tensorflow/models#3794
@irfankilic
Copy link

Hello
In config file change "step: 0" to "step: 1 or non-zero value"

optimizer {
    momentum_optimizer {
      learning_rate {
        manual_step_learning_rate {
          initial_learning_rate: 0.000300000014249
          schedule `{`
            step: 1
            learning_rate: 0.000300000014249
          }

@ISS88
Copy link

ISS88 commented Jul 10, 2020

hello so here I try to train a model of mask r cnn on my own give with the command
python D:\Projet\CV-CommonTools\model_main.py /
--model_dir training /
--pipeline_config_path training\mask_rcnn_inception_v2.config /
--num_train_steps 60000 /
--sample_1_of_n_eval_examples=0 /
--alsologtostderr

but i get the following error

Traceback (most recent call last):
  File "D:\Projet\model_main.py", line 110, in <module>
    tf.app.run()
  File "C:\Users\Ibrahim\AppData\Local\Programs\Python\Python36\lib\site-packages\tensorflow\python\platform\app.py", line 40, in run
    _run(main=main, argv=argv, flags_parser=_parse_flags_tolerate_undef)
  File "C:\Users\Ibrahim\AppData\Local\Programs\Python\Python36\lib\site-packages\absl\app.py", line 299, in run
    _run_main(main, args)
  File "C:\Users\Ibrahim\AppData\Local\Programs\Python\Python36\lib\site-packages\absl\app.py", line 250, in _run_main
    sys.exit(main(argv))
  File "D:\Projet\model_main.py", line 71, in main
    FLAGS.sample_1_of_n_eval_on_train_examples))
TypeError: create_estimator_and_inputs() missing 1 required positional argument: 'hparams'

could someone guided me please

@PelinSuK
Copy link

Helloo i have errors while running model_main_tf2.py.
" python model_main_tf2.py --alsologtostderr --model_dir=training/ --pipeline_config_path=training/faster_rcnn_inception_v2_pets.config "
I dont have any custom steps i did everything from here
But my tensorflow version is 2.5 and i did all setups according to 2.5
So the errors are =
File "C:\tensorflow1\models\research\object_detection\model_main_tf2.py", line 116, in
tf.compat.v1.app.run()
File "C:\Users\pelin\anaconda3\envs\tensorflow1\lib\site-packages\tensorflow\python\platform\app.py", line 40, in run
_run(main=main, argv=argv, flags_parser=_parse_flags_tolerate_undef)
File "C:\Users\pelin\anaconda3\envs\tensorflow1\lib\site-packages\absl\app.py", line 303, in run
_run_main(main, args)
File "C:\Users\pelin\anaconda3\envs\tensorflow1\lib\site-packages\absl\app.py", line 251, in _run_main
sys.exit(main(argv))
File "C:\tensorflow1\models\research\object_detection\model_main_tf2.py", line 106, in main
model_lib_v2.train_loop(
File "C:\tensorflow1\models\research\object_detection\model_lib_v2.py", line 524, in train_loop
raise ValueError('train_pb2.load_all_detection_checkpoint_vars '
ValueError: train_pb2.load_all_detection_checkpoint_vars unsupported in TF2

I searched for 6 days to find a solution. Im so new to this topic and couldnt solve the problems. I would be so appreciate if you help me.

these are my config and other files that i use here

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests