Bias not found in checkpoint #7

abhisuri97 · 2017-06-20T03:30:41Z

I’m having some problems running the Pretrained-Show-and-Tell-model
I am using tensorflow v1.2 and python 2.7 (in addition to checking out the latest versions of your repo and the im2txt model repo).
I am getting the following error:

NotFoundError (see above for traceback): Key lstm/basic_lstm_cell/bias not found in checkpoint
	 [[Node: save/RestoreV2_380 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_380/tensor_names, save/RestoreV2_380/shape_and_slices)]]

My Full Trace is Below:

(venv)  abhi@Abhinavs-MacBook-Pro  ~/Desktop/Various_dev_projects/testing/models/im2txt   master  bazel-bin/im2txt/run_inference \
  --checkpoint_path=${CHECKPOINT_PATH} \
  --vocab_file=${VOCAB_FILE} \
  --input_files=${IMAGE_FILE}


INFO:tensorflow:Building model.
INFO:tensorflow:Initializing vocabulary from file: ptrain/word_counts.txt
INFO:tensorflow:Created vocabulary with 11520 words
INFO:tensorflow:Running caption generation on 1 files matching images/1.jpg
2017-06-19 22:24:00.000062: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.2 instructions, but these are available on your machine and could speed up CPU computations.
2017-06-19 22:24:00.000085: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX instructions, but these are available on your machine and could speed up CPU computations.
2017-06-19 22:24:00.000090: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX2 instructions, but these are available on your machine and could speed up CPU computations.
2017-06-19 22:24:00.000094: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use FMA instructions, but these are available on your machine and could speed up CPU computations.
INFO:tensorflow:Loading model from checkpoint: ptrain/model.ckpt-2000000
INFO:tensorflow:Restoring parameters from ptrain/model.ckpt-2000000
2017-06-19 22:24:02.932954: W tensorflow/core/framework/op_kernel.cc:1158] Not found: Key lstm/basic_lstm_cell/bias not found in checkpoint
2017-06-19 22:24:02.933737: W tensorflow/core/framework/op_kernel.cc:1158] Not found: Key lstm/basic_lstm_cell/kernel not found in checkpoint
Traceback (most recent call last):
  File "/Users/abhi/Desktop/Various_dev_projects/testing/models/im2txt/bazel-bin/im2txt/run_inference.runfiles/im2txt/im2txt/run_inference.py", line 85, in <module>
    tf.app.run()
  File "/Users/abhi/Desktop/Various_dev_projects/testing/models/im2txt/venv/lib/python2.7/site-packages/tensorflow/python/platform/app.py", line 48, in run
    _sys.exit(main(_sys.argv[:1] + flags_passthrough))
  File "/Users/abhi/Desktop/Various_dev_projects/testing/models/im2txt/bazel-bin/im2txt/run_inference.runfiles/im2txt/im2txt/run_inference.py", line 65, in main
    restore_fn(sess)
  File "/Users/abhi/Desktop/Various_dev_projects/testing/models/im2txt/bazel-bin/im2txt/run_inference.runfiles/im2txt/im2txt/inference_utils/inference_wrapper_base.py", line 96, in _restore_fn
    saver.restore(sess, checkpoint_path)
  File "/Users/abhi/Desktop/Various_dev_projects/testing/models/im2txt/venv/lib/python2.7/site-packages/tensorflow/python/training/saver.py", line 1548, in restore
    {self.saver_def.filename_tensor_name: save_path})
  File "/Users/abhi/Desktop/Various_dev_projects/testing/models/im2txt/venv/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 789, in run
    run_metadata_ptr)
  File "/Users/abhi/Desktop/Various_dev_projects/testing/models/im2txt/venv/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 997, in _run
    feed_dict_string, options, run_metadata)
  File "/Users/abhi/Desktop/Various_dev_projects/testing/models/im2txt/venv/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1132, in _do_run
    target_list, options, run_metadata)
  File "/Users/abhi/Desktop/Various_dev_projects/testing/models/im2txt/venv/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1152, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.NotFoundError: Key lstm/basic_lstm_cell/bias not found in checkpoint
	 [[Node: save/RestoreV2_380 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_380/tensor_names, save/RestoreV2_380/shape_and_slices)]]

Caused by op u'save/RestoreV2_380', defined at:
  File "/Users/abhi/Desktop/Various_dev_projects/testing/models/im2txt/bazel-bin/im2txt/run_inference.runfiles/im2txt/im2txt/run_inference.py", line 85, in <module>
    tf.app.run()
  File "/Users/abhi/Desktop/Various_dev_projects/testing/models/im2txt/venv/lib/python2.7/site-packages/tensorflow/python/platform/app.py", line 48, in run
    _sys.exit(main(_sys.argv[:1] + flags_passthrough))
  File "/Users/abhi/Desktop/Various_dev_projects/testing/models/im2txt/bazel-bin/im2txt/run_inference.runfiles/im2txt/im2txt/run_inference.py", line 51, in main
    FLAGS.checkpoint_path)
  File "/Users/abhi/Desktop/Various_dev_projects/testing/models/im2txt/bazel-bin/im2txt/run_inference.runfiles/im2txt/im2txt/inference_utils/inference_wrapper_base.py", line 116, in build_graph_from_config
    saver = tf.train.Saver()
  File "/Users/abhi/Desktop/Various_dev_projects/testing/models/im2txt/venv/lib/python2.7/site-packages/tensorflow/python/training/saver.py", line 1139, in __init__
    self.build()
  File "/Users/abhi/Desktop/Various_dev_projects/testing/models/im2txt/venv/lib/python2.7/site-packages/tensorflow/python/training/saver.py", line 1170, in build
    restore_sequentially=self._restore_sequentially)
  File "/Users/abhi/Desktop/Various_dev_projects/testing/models/im2txt/venv/lib/python2.7/site-packages/tensorflow/python/training/saver.py", line 691, in build
    restore_sequentially, reshape)
  File "/Users/abhi/Desktop/Various_dev_projects/testing/models/im2txt/venv/lib/python2.7/site-packages/tensorflow/python/training/saver.py", line 407, in _AddRestoreOps
    tensors = self.restore_op(filename_tensor, saveable, preferred_shard)
  File "/Users/abhi/Desktop/Various_dev_projects/testing/models/im2txt/venv/lib/python2.7/site-packages/tensorflow/python/training/saver.py", line 247, in restore_op
    [spec.tensor.dtype])[0])
  File "/Users/abhi/Desktop/Various_dev_projects/testing/models/im2txt/venv/lib/python2.7/site-packages/tensorflow/python/ops/gen_io_ops.py", line 640, in restore_v2
    dtypes=dtypes, name=name)
  File "/Users/abhi/Desktop/Various_dev_projects/testing/models/im2txt/venv/lib/python2.7/site-packages/tensorflow/python/framework/op_def_library.py", line 767, in apply_op
    op_def=op_def)
  File "/Users/abhi/Desktop/Various_dev_projects/testing/models/im2txt/venv/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 2506, in create_op
    original_op=self._default_original_op, op_def=op_def)
  File "/Users/abhi/Desktop/Various_dev_projects/testing/models/im2txt/venv/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 1269, in __init__
    self._traceback = _extract_stack()

The text was updated successfully, but these errors were encountered:

KranthiGV · 2017-06-20T05:54:39Z

@abhisuri97
It's very unusual.
Are you sure you put the checkpoint files?
Run ls -l --block-size=K in the ptrain folder and post it here.

abhisuri97 · 2017-06-20T13:28:21Z

Here is the output I get on that command:

ls: illegal option -- -
usage: ls [-ABCFGHLOPRSTUWabcdefghiklmnopqrstuwx1] [file ...]

The regular ls -l command outputs the following

-rw-r--r--@ 1 abhi  staff  149002244 Jun 19 22:21 model.ckpt-2000000.data-00000-of-00001
-rw-r--r--  1 abhi  staff      16876 Jun 19 22:21 model.ckpt-2000000.index
-rw-r--r--  1 abhi  staff     121881 Jun 19 22:21 word_counts.txt

Am I missing a file or something here?

DominikFilipiak · 2017-06-20T19:28:18Z

@abhisuri97, I had tried to run this model on the latest version of TF (1.2.0), but encountered the same error:
Key lstm/basic_lstm_cell/kernel not found in checkpoint
I've found @cshallue answer. The original version didn't work out for me. After some digging, I've changed values in vars_to_rename - did the trick.

OLD_CHECKPOINT_FILE = "model.ckpt-1000000"
NEW_CHECKPOINT_FILE = "model2.ckpt-1000000"

import tensorflow as tf
vars_to_rename = {
    "lstm/basic_lstm_cell/weights": "lstm/basic_lstm_cell/kernel",
    "lstm/basic_lstm_cell/biases": "lstm/basic_lstm_cell/bias",
}
new_checkpoint_vars = {}
reader = tf.train.NewCheckpointReader(OLD_CHECKPOINT_FILE)
for old_name in reader.get_variable_to_shape_map():
  if old_name in vars_to_rename:
    new_name = vars_to_rename[old_name]
  else:
    new_name = old_name
  new_checkpoint_vars[new_name] = tf.Variable(reader.get_tensor(old_name))

init = tf.global_variables_initializer()
saver = tf.train.Saver(new_checkpoint_vars)

with tf.Session() as sess:
  sess.run(init)
  saver.save(sess, NEW_CHECKPOINT_FILE)

abhisuri97 · 2017-06-20T19:37:11Z

@mimol that worked. thank you so much!!!!

KranthiGV · 2017-06-21T05:08:57Z

@mimol
Thank you so much!
Will update the readme with this fix! :)

PapaMadeleine2022 · 2017-10-30T12:37:05Z

@mimol your code finally solve my problem.thank you very much~~

eyaler · 2018-04-07T16:18:36Z

thanks. please add to readme...

monajalal · 2018-04-19T06:39:36Z

@0xDFDFDF What if we can't go back to Tensorflow1.1? Or how can I go to that version having the below error:
[jalal@goku CharLSTM]$ conda install tensorflow-gpu==1.1.0
Solving environment: failed

UnsatisfiableError: The following specifications were found to be in conflict:

tensorflow-gpu-base -> cudatoolkit=8.0
tensorflow-gpu-base -> cudnn=7
tensorflow-gpu==1.1.0 -> cudnn==5.1
Use "conda info " to see the dependencies for each package.

I can't downgrade my cudnn and cudatoolkit due to systemwide settings.

monajalal · 2018-04-19T06:40:59Z

@0xDFDFDF where did you apply these changes? Can you please give more starter points? I am using CharLSTM and got this error charlesashby/CharLSTM#9

danlou · 2018-08-22T10:47:28Z

To anyone wondering, I've just executed @0xDFDFDF 's script (#7 (comment)) using the tensorflow v1.10.0 and it worked fine.

ndwuhuangwei · 2019-10-16T14:29:06Z

@abhisuri97 could you tell me what @mimol said? I just can't see any answer from him

DominikFilipiak · 2019-10-16T14:32:23Z

@ndwuhuangwei that was my old login, so you are perhaps looking for this answer: #7 (comment)

maoao686868 · 2020-11-13T00:54:26Z

So finally how did you solve your problems? I am toubled with this problem either very much. I need your help and I can't see what mimol said.@abhisuri97

abhisuri97 · 2020-11-13T01:02:35Z

@maoao686868 As @DominikFilipiak mentioned earlier, the code is here: #7 (comment) Of course, I haven't looked at this codebase for a while so I am not sure if that is still a valid solution or not.

maoao686868 · 2020-11-13T01:29:38Z

@abhisuri97 ，thanks for your reply,but my question is "biases not found in checkpoint" not "kernel not found in checkpoint". I am expecting your answer

abhisuri97 · 2020-11-13T01:34:31Z

@maoao686868 It seems like this line "lstm/basic_lstm_cell/biases": "lstm/basic_lstm_cell/bias", in the answer I linked should take care of that; however, I don't recall coming across a "biases not found in checkpoint" error.

BassantTolba1234 · 2020-12-15T09:45:23Z

Please @DominikFilipiak , I'm using a pretrained model to my new model , but the same error( about bias does not exist in checkpoints) appears to me.
I tried to use your above solution #7 (comment) but it says that there is no file called model.ckpt-100000
please can you kindly help me ??
here is the figure of my three pretrained saved model..
I'm waiting for your reply
Thanks in advance

BassantTolba1234 · 2020-12-15T09:46:42Z

Please all, I'm using a pretrained model to my new model , but the same error( about bias does not exist in checkpoints) appears to me.
I tried to use your above solution #7 (comment) but it says that there is no file called model.ckpt-100000
please can you kindly help me ??
here is the figure of my three pretrained saved model..
I'm waiting for your reply
Thanks in advance

abhisuri97 closed this as completed Jun 20, 2017

eragonruan mentioned this issue Sep 25, 2017

Key lstm_o/rnn/basic_lstm_cell/kernel not found in checkpoint eragonruan/text-detection-ctpn#1

Closed

hsddlz mentioned this issue Oct 27, 2017

kernel not found in checkpoint eragonruan/text-detection-ctpn#11

Closed

adeshpande3 mentioned this issue May 5, 2018

Running in batch doesn't work? adeshpande3/LSTM-Sentiment-Analysis#35

Closed

mgrenander mentioned this issue May 14, 2018

Error when loading model zwc12/Summarization#1

Closed

xthan mentioned this issue Sep 21, 2019

TensorFlow version xthan/polyvore#2

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Bias not found in checkpoint #7

Bias not found in checkpoint #7

abhisuri97 commented Jun 20, 2017

KranthiGV commented Jun 20, 2017

abhisuri97 commented Jun 20, 2017

DominikFilipiak commented Jun 20, 2017 •

edited

Loading

abhisuri97 commented Jun 20, 2017

KranthiGV commented Jun 21, 2017

PapaMadeleine2022 commented Oct 30, 2017

eyaler commented Apr 7, 2018

monajalal commented Apr 19, 2018 •

edited

Loading

monajalal commented Apr 19, 2018 •

edited

Loading

danlou commented Aug 22, 2018

ndwuhuangwei commented Oct 16, 2019

DominikFilipiak commented Oct 16, 2019

maoao686868 commented Nov 13, 2020

abhisuri97 commented Nov 13, 2020

maoao686868 commented Nov 13, 2020

abhisuri97 commented Nov 13, 2020 •

edited

Loading

BassantTolba1234 commented Dec 15, 2020

BassantTolba1234 commented Dec 15, 2020

Bias not found in checkpoint #7

Bias not found in checkpoint #7

Comments

abhisuri97 commented Jun 20, 2017

KranthiGV commented Jun 20, 2017

abhisuri97 commented Jun 20, 2017

DominikFilipiak commented Jun 20, 2017 • edited Loading

abhisuri97 commented Jun 20, 2017

KranthiGV commented Jun 21, 2017

PapaMadeleine2022 commented Oct 30, 2017

eyaler commented Apr 7, 2018

monajalal commented Apr 19, 2018 • edited Loading

monajalal commented Apr 19, 2018 • edited Loading

danlou commented Aug 22, 2018

ndwuhuangwei commented Oct 16, 2019

DominikFilipiak commented Oct 16, 2019

maoao686868 commented Nov 13, 2020

abhisuri97 commented Nov 13, 2020

maoao686868 commented Nov 13, 2020

abhisuri97 commented Nov 13, 2020 • edited Loading

BassantTolba1234 commented Dec 15, 2020

BassantTolba1234 commented Dec 15, 2020

DominikFilipiak commented Jun 20, 2017 •

edited

Loading

monajalal commented Apr 19, 2018 •

edited

Loading

monajalal commented Apr 19, 2018 •

edited

Loading

abhisuri97 commented Nov 13, 2020 •

edited

Loading