Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Conv2d_1a_7x7/BatchNorm/beta/Momentum] is not available in checkpoint #4862

Closed
navinkeshava opened this issue Jul 22, 2018 · 18 comments
Closed

WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Conv2d_1a_7x7/BatchNorm/beta/Momentum] is not available in checkpoint #4862

navinkeshava opened this issue Jul 22, 2018 · 18 comments
Assignees
@bitfort
Labels
models:research models that come under research directory stale stat:awaiting response Waiting on input from the contributor type:support

Comments

@navinkeshava
Copy link

Please go to Stack Overflow for help and support:

http://stackoverflow.com/questions/tagged/tensorflow

System information

  • What is the top-level directory of the model you are using:
  • Have I written custom code (as opposed to using a stock example script provided in TensorFlow):
  • **OS Platform WINDOWS 10
  • **TensorFlow
  • **TensorFlow version 1.5
  • **CUDA 9.0 cuDNN 7.0 version

python train.py --logtostderr --train_dir=training/ --pipeline_config_path=training/faster_rcnn_inception_v2_pets.config

when i am training the model it is showing these types of warning what is the cause of this what is to do correct it

WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Conv2d_1a_7x7/BatchNorm/gamma/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Conv2d_1a_7x7/depthwise_weights/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Conv2d_1a_7x7/pointwise_weights/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Conv2d_2b_1x1/BatchNorm/beta/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Conv2d_2b_1x1/BatchNorm/gamma/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Conv2d_2b_1x1/weights/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Conv2d_2c_3x3/BatchNorm/beta/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Conv2d_2c_3x3/BatchNorm/gamma/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Conv2d_2c_3x3/weights/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_3b/Branch_0/Conv2d_0a_1x1/BatchNorm/beta/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_3b/Branch_0/Conv2d_0a_1x1/BatchNorm/gamma/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_3b/Branch_0/Conv2d_0a_1x1/weights/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_3b/Branch_1/Conv2d_0a_1x1/BatchNorm/beta/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_3b/Branch_1/Conv2d_0a_1x1/BatchNorm/gamma/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_3b/Branch_1/Conv2d_0a_1x1/weights/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_3b/Branch_1/Conv2d_0b_3x3/BatchNorm/beta/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_3b/Branch_1/Conv2d_0b_3x3/BatchNorm/gamma/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_3b/Branch_1/Conv2d_0b_3x3/weights/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_3b/Branch_2/Conv2d_0a_1x1/BatchNorm/beta/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_3b/Branch_2/Conv2d_0a_1x1/BatchNorm/gamma/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_3b/Branch_2/Conv2d_0a_1x1/weights/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_3b/Branch_2/Conv2d_0b_3x3/BatchNorm/beta/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_3b/Branch_2/Conv2d_0b_3x3/BatchNorm/gamma/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_3b/Branch_2/Conv2d_0b_3x3/weights/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_3b/Branch_2/Conv2d_0c_3x3/BatchNorm/beta/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_3b/Branch_2/Conv2d_0c_3x3/BatchNorm/gamma/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_3b/Branch_2/Conv2d_0c_3x3/weights/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_3b/Branch_3/Conv2d_0b_1x1/BatchNorm/beta/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_3b/Branch_3/Conv2d_0b_1x1/BatchNorm/gamma/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_3b/Branch_3/Conv2d_0b_1x1/weights/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_3c/Branch_0/Conv2d_0a_1x1/BatchNorm/beta/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_3c/Branch_0/Conv2d_0a_1x1/BatchNorm/gamma/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_3c/Branch_0/Conv2d_0a_1x1/weights/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_3c/Branch_1/Conv2d_0a_1x1/BatchNorm/beta/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_3c/Branch_1/Conv2d_0a_1x1/BatchNorm/gamma/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_3c/Branch_1/Conv2d_0a_1x1/weights/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_3c/Branch_1/Conv2d_0b_3x3/BatchNorm/beta/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_3c/Branch_1/Conv2d_0b_3x3/BatchNorm/gamma/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_3c/Branch_1/Conv2d_0b_3x3/weights/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_3c/Branch_2/Conv2d_0a_1x1/BatchNorm/beta/Momentum] is not available in checkpoint
WARNING:root:Variable [FirstStageFeatureExtractor/InceptionV2/Mixed_3c/Branch_2/Conv2d_0a_1x1/BatchNorm/gamma/Momentum] is not available in checkpoint
@tensorflowbutler tensorflowbutler added the stat:awaiting response Waiting on input from the contributor label Jul 24, 2018
@tensorflowbutler
Copy link
Member

Thank you for your post. We noticed you have not filled out the following field in the issue template. Could you update them if they are relevant in your case, or leave them as N/A? Thanks.
OS Platform and Distribution
TensorFlow installed from
Bazel version
CUDA/cuDNN version
GPU model and memory
Exact command to reproduce

@jason-zhu
Copy link

I have the same problem...

@navinkeshava
Copy link
Author

OS Platform and Distribution : windows 10
Tensorflow : 1.5 gpu version
CUDA 9.0 and CuDNN 7.0
GPU N/A
python train.py --logtostderr --train_dir=training/ --pipeline_config_path=training/faster_rcnn_inception_v2_pets.config

@navinkeshava
Copy link
Author

navinkeshava commented Jul 25, 2018 via email

@redzhepdx
Copy link

redzhepdx commented Jul 25, 2018 •
edited

I have the same problem as well, I just changed batch_size parameter. But It don't start to train model , there isn't any training visualization and just repeating model loading process again and again.

OS Platform and Distribution : Ubuntu 16.04
Tensorflow : 1.8 gpu version
CUDA 9.0 and cuDNN 7.0
GPU 1080Ti

python model_main.py --alsologtostderr --train_dir=ssd_fpn_mobilenet_training --pipeline_config_path=ssd_fpn_mobilenet_training/pipeline.config --num_train_steps=10000 --num_eval_steps=100

@tensorflowbutler tensorflowbutler removed the stat:awaiting response Waiting on input from the contributor label Jul 25, 2018
@asxasxdscsd
Copy link

for the issue like this, @jch1 answered >>>> it looks like you are initializing from an Imagenet pretrained classification checkpoint?

If so, don't worry about this; those layers are additional layers added to the Mobilenet backbone by SSD and don't exist in the classification checkpoint.

@YNVoon
Copy link

YNVoon commented Aug 1, 2018

I have the same problem as well. Any update?

@tencyrush
Copy link

i have the same problem

@CreateChance
Copy link

I have the same problem as well. Any one could help?

@Uture
Copy link

Uture commented Oct 24, 2018

Also same problem, does anyone here care? How many people need to address the problem to get any kind of helpful answer?

@yuezhilanyi
Copy link

add
from_detection_checkpoint: true
in your_pipeline.config may help.

see https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/configuring_jobs.md#configuring-the-object-detection-training-pipeline for detail, which is hard to find.

@vishal2612200
Copy link

which is hard to find

This is also not working in my computer

@TomasTrnkaPLC
Copy link

I have a some problem with Windows 10. Python 3.6. Some solution?
from_detection_checkpoint: true is without sense

@shABanty
Copy link

shABanty commented Feb 6, 2020 •
edited

### Some examples of warning
WARNING:root:Variable [FeatureExtractor/InceptionV2/Mixed_5c_2_Conv2d_5_3x3_s2_128/BatchNorm/gamma/RMSProp] is not available in checkpoint
WARNING:root:Variable [FeatureExtractor/InceptionV2/Mixed_5c_2_Conv2d_5_3x3_s2_128/BatchNorm/gamma/RMSProp_1] is not available in checkpoint
WARNING:root:Variable [FeatureExtractor/InceptionV2/Mixed_5c_2_Conv2d_5_3x3_s2_128/weights/ExponentialMovingAverage] is not available in checkpoint
WARNING:root:Variable [FeatureExtractor/InceptionV2/Mixed_5c_2_Conv2d_5_3x3_s2_128/weights/RMSProp] is not available in checkpoint
WARNING:root:Variable [FeatureExtractor/InceptionV2/Mixed_5c_2_Conv2d_5_3x3_s2_128/weights/RMSProp_1] is not available in checkpoint
**Any solutions? I'm using ssd_inception_v2_coco.

pipeline.config looks like that**

odel {
ssd {
num_classes: 10
box_coder {
faster_rcnn_box_coder {
y_scale: 10.0
x_scale: 10.0
height_scale: 5.0
width_scale: 5.0
}
}
matcher {
argmax_matcher {
matched_threshold: 0.5
unmatched_threshold: 0.5
ignore_thresholds: false
negatives_lower_than_unmatched: true
force_match_for_each_row: true
}
}
similarity_calculator {
iou_similarity {
}
}
anchor_generator {
ssd_anchor_generator {
num_layers: 6
min_scale: 0.2
max_scale: 0.95
aspect_ratios: 1.0
aspect_ratios: 2.0
aspect_ratios: 0.5
aspect_ratios: 3.0
aspect_ratios: 0.3333
reduce_boxes_in_lowest_layer: true
}
}
image_resizer {
fixed_shape_resizer {
height: 300
width: 300
}
}
box_predictor {
convolutional_box_predictor {
min_depth: 0
max_depth: 0
num_layers_before_predictor: 0
use_dropout: false
dropout_keep_probability: 0.8
kernel_size: 3
box_code_size: 4
apply_sigmoid_to_scores: false
conv_hyperparams {
activation: RELU_6,
regularizer {
l2_regularizer {
weight: 0.00004
}
}
initializer {
truncated_normal_initializer {
stddev: 0.03
mean: 0.0
}
}
}
}
}
feature_extractor {
type: 'ssd_inception_v2'
min_depth: 16
depth_multiplier: 1.0
conv_hyperparams {
activation: RELU_6,
regularizer {
l2_regularizer {
weight: 0.00004
}
}
initializer {
truncated_normal_initializer {
stddev: 0.03
mean: 0.0
}
}
batch_norm {
train: true,
scale: true,
center: true,
decay: 0.9997,
epsilon: 0.001,
}
}
override_base_feature_extractor_hyperparams: true
}
loss {
classification_loss {
weighted_sigmoid {
}
}
localization_loss {
weighted_smooth_l1 {
}
}
hard_example_miner {
num_hard_examples: 3000
iou_threshold: 0.99
loss_type: CLASSIFICATION
max_negatives_per_positive: 3
min_negatives_per_image: 0
}
classification_weight: 1.0
localization_weight: 1.0
}
normalize_loss_by_num_matches: true
post_processing {
batch_non_max_suppression {
score_threshold: 1e-8
iou_threshold: 0.6
max_detections_per_class: 100
max_total_detections: 100
}
score_converter: SIGMOID
}
}
}
train_config: {
batch_size: 10
optimizer {
rms_prop_optimizer: {
learning_rate: {
exponential_decay_learning_rate {
initial_learning_rate: 0.004
decay_steps: 800720
decay_factor: 0.95
}
}
momentum_optimizer_value: 0.9
decay: 0.9
epsilon: 1.0
}
}
fine_tune_checkpoint: "C:\Users\AB\Tensorflow\workplace\training_demo\pre-trained-model\model.ckpt"
### from_detection_checkpoint: true
Note: The below line limits the training process to 200K steps, which we
empirically found to be sufficient enough to train the pets dataset. This
effectively bypasses the learning rate schedule (the learning rate will
never decay). Remove the below line to train indefinitely.
num_steps: 200000
fine_tune_checkpoint:"C:\Users\AB\Tensorflow\workplace\training_demo\pre-trained-model\model.ckpt"
fine_tune_checkpoint_type: "detection"
data_augmentation_options {
random_horizontal_flip {
}
}
data_augmentation_options {
ssd_random_crop {
}
}
}
train_input_reader: {
tf_record_input_reader {
input_path: "C:\Users\AB\Tensorflow\workplace\training_demo\annotations\train.record"
}
label_map_path: "C:\Users\AB\Tensorflow\workplace\training_demo\annotations\label_map.pbtxt"
}
eval_config: {
num_examples: 8000
Note: The below line limits the evaluation process to 10 evaluations.
Remove the below line to evaluate indefinitely.
max_evals: 10
}
eval_input_reader: {
tf_record_input_reader {
input_path: "C:\Users\AB\Tensorflow\workplace\training_demo\annotations\test.record"
}
label_map_path: "C:\Users\AB\Tensorflow\workplace\training_demo\annotations\label_map.pbtxt"
shuffle: false
num_readers: 1
}

@ravikyram ravikyram added models:research models that come under research directory type:support labels Jul 15, 2020
@kumariko
Copy link

kumariko commented Jan 5, 2022

@navinkeshava We are checking to see if you still need help on this issue? Could you please have a look on the link and let us know if it helps? Thanks!

@kumariko kumariko added the stat:awaiting response Waiting on input from the contributor label Jan 5, 2022
@google-ml-butler
Copy link

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you.

@google-ml-butler
Copy link

Closing as stale. Please reopen if you'd like to work on this further.

@google-ml-butler
Copy link

Are you satisfied with the resolution of your issue?
Yes
No

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@bitfort bitfort

Labels
models:research models that come under research directory stale stat:awaiting response Waiting on input from the contributor type:support
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests