Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Unable to load the saved_model #21

Open
kunal017 opened this issue Jan 28, 2019 · 4 comments
Open

Unable to load the saved_model #21

kunal017 opened this issue Jan 28, 2019 · 4 comments

Comments

@kunal017
Copy link

kunal017 commented Jan 28, 2019 •
edited

I am running the model for countries_s1 model but getting the following error:

Executing python code/model/trainer.py --base_output_dir output/countries_s1/ --path_length 2 --hidden_size 25 --embedding_size 25     --batch_size 256 --beta 0.05 --Lambda 0.05 --use_entity_embeddings 1     --train_entity_embeddings 1 --train_relation_embeddings 1     --data_input_dir datasets/data_preprocessed/countries_s1/ --vocab_dir datasets/data_preprocessed/countries_s1/vocab --model_load_dir saved_models/countries_s1/model.ckpt --load_model 1 --total_iterations 1000 --nell_evaluation 0
HEY
Arguments:
                         LSTM_layers : 1
                              Lambda : 0.05
                     base_output_dir : output/countries_s1/
                          batch_size : 256
                                beta : 0.05
                        create_vocab : 0
                      data_input_dir : datasets/data_preprocessed/countries_s1/
                      embedding_size : 25
                          eval_every : 100
                               gamma : 1
                      grad_clip_norm : 5
                         hidden_size : 25
                          input_file : train.txt
                         input_files : ['datasets/data_preprocessed/countries_s1//train.txt']
                        l2_reg_const : 0.01
                       learning_rate : 0.001
                          load_model : True
                             log_dir : ./logs/
                       log_file_name : output/countries_s1//3f71_2_0.05_100_0.05/log.txt
                     max_num_actions : 200
                           model_dir : output/countries_s1//3f71_2_0.05_100_0.05/model/
                      model_load_dir : saved_models/countries_s1/model.ckpt
                     negative_reward : 0
                     nell_evaluation : 0
                        num_rollouts : 20
                          output_dir : output/countries_s1//3f71_2_0.05_100_0.05
                         output_file :
                         path_length : 2
                    path_logger_file : output/countries_s1//3f71_2_0.05_100_0.05
                                pool : max
                     positive_reward : 1.0
        pretrained_embeddings_action :
        pretrained_embeddings_entity :
                       test_rollouts : 100
                    total_iterations : 1000
             train_entity_embeddings : True
           train_relation_embeddings : True
               use_entity_embeddings : True
                           vocab_dir : datasets/data_preprocessed/countries_s1/vocab
INFO:root:reading vocab files...
01/28/2019 01:56:20 AM: [ reading vocab files... ]
INFO:root:Reading mid to name map
01/28/2019 01:56:20 AM: [ Reading mid to name map ]
INFO:root:Done..
01/28/2019 01:56:20 AM: [ Done.. ]
INFO:root:Total number of entities 273
01/28/2019 01:56:20 AM: [ Total number of entities 273 ]
INFO:root:Total number of relations 6
01/28/2019 01:56:20 AM: [ Total number of relations 6 ]
INFO:root:Skipping training
01/28/2019 01:56:20 AM: [ Skipping training ]
INFO:root:Loading model from saved_models/countries_s1/model.ckpt
01/28/2019 01:56:20 AM: [ Loading model from saved_models/countries_s1/model.ckpt ]
Reading vocab...
batcher loaded
KG constructed
Reading vocab...
Contains full graph
batcher loaded
KG constructed
Reading vocab...
Contains full graph
batcher loaded
KG constructed
2019-01-28 01:56:21.112771: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.1 instructions, but these are available on your machine and could speed up CPU computations.
2019-01-28 01:56:21.112794: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.2 instructions, but these are available on your machine and could speed up CPU computations.
2019-01-28 01:56:21.112801: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX instructions, but these are available on your machine and could speed up CPU computations.
2019-01-28 01:56:21.112806: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX2 instructions, but these are available on your machine and could speed up CPU computations.
2019-01-28 01:56:21.112812: W tensorflow/core/platform/cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use FMA instructions, but these are available on your machine and could speed up CPU computations.
2019-01-28 01:56:21.988648: I tensorflow/core/common_runtime/gpu/gpu_device.cc:955] Found device 0 with properties:
name: GeForce GTX 1080 Ti
major: 6 minor: 1 memoryClockRate (GHz) 1.582
pciBusID 0000:06:00.0
Total memory: 10.91GiB
Free memory: 10.75GiB
2019-01-28 01:56:21.988740: I tensorflow/core/common_runtime/gpu/gpu_device.cc:976] DMA: 0
2019-01-28 01:56:21.988757: I tensorflow/core/common_runtime/gpu/gpu_device.cc:986] 0:   Y
2019-01-28 01:56:21.988799: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1045] Creating TensorFlow device (/gpu:0) -> (device: 0, name: GeForce GTX 1080 Ti, pci bus id: 0000:06:00.0)
INFO:root:Creating TF graph...
01/28/2019 01:56:22 AM: [ Creating TF graph... ]
INFO:root:TF Graph creation done..
01/28/2019 01:56:23 AM: [ TF Graph creation done.. ]
INFO:tensorflow:Restoring parameters from saved_models/countries_s1/model.ckpt
INFO:tensorflow:Restoring parameters from saved_models/countries_s1/model.ckpt
01/28/2019 01:56:23 AM: [ Restoring parameters from saved_models/countries_s1/model.ckpt ]
2019-01-28 01:56:23.830867: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key entity_lookup_table/entity_lookup_table/Adam_1 not found in checkpoint
2019-01-28 01:56:23.846931: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key entity_lookup_table/entity_lookup_table/Adam_1 not found in checkpoint
         [[Node: save/RestoreV2_9 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_9/tensor_names, save/RestoreV2_9/shape_and_slices)]]
2019-01-28 01:56:23.847248: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key entity_lookup_table/entity_lookup_table/Adam_1 not found in checkpoint
         [[Node: save/RestoreV2_9 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_9/tensor_names, save/RestoreV2_9/shape_and_slices)]]
2019-01-28 01:56:23.847249: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key entity_lookup_table/entity_lookup_table/Adam_1 not found in checkpoint
         [[Node: save/RestoreV2_9 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_9/tensor_names, save/RestoreV2_9/shape_and_slices)]]
2019-01-28 01:56:23.847266: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key entity_lookup_table/entity_lookup_table/Adam_1 not found in checkpoint
         [[Node: save/RestoreV2_9 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_9/tensor_names, save/RestoreV2_9/shape_and_slices)]]
2019-01-28 01:56:23.847290: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key entity_lookup_table/entity_lookup_table/Adam_1 not found in checkpoint
         [[Node: save/RestoreV2_9 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_9/tensor_names, save/RestoreV2_9/shape_and_slices)]]
2019-01-28 01:56:23.847248: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key entity_lookup_table/entity_lookup_table/Adam_1 not found in checkpoint
         [[Node: save/RestoreV2_9 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_9/tensor_names, save/RestoreV2_9/shape_and_slices)]]
2019-01-28 01:56:23.847342: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key entity_lookup_table/entity_lookup_table/Adam_1 not found in checkpoint
         [[Node: save/RestoreV2_9 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_9/tensor_names, save/RestoreV2_9/shape_and_slices)]]
2019-01-28 01:56:23.847358: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key entity_lookup_table/entity_lookup_table/Adam_1 not found in checkpoint
         [[Node: save/RestoreV2_9 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_9/tensor_names, save/RestoreV2_9/shape_and_slices)]]
2019-01-28 01:56:23.847437: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key entity_lookup_table/entity_lookup_table/Adam_1 not found in checkpoint
         [[Node: save/RestoreV2_9 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_9/tensor_names, save/RestoreV2_9/shape_and_slices)]]
2019-01-28 01:56:23.847529: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key entity_lookup_table/entity_lookup_table/Adam_1 not found in checkpoint
         [[Node: save/RestoreV2_9 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_9/tensor_names, save/RestoreV2_9/shape_and_slices)]]
2019-01-28 01:56:23.847785: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key entity_lookup_table/entity_lookup_table/Adam_1 not found in checkpoint
         [[Node: save/RestoreV2_9 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_9/tensor_names, save/RestoreV2_9/shape_and_slices)]]
2019-01-28 01:56:23.847753: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key entity_lookup_table/entity_lookup_table/Adam_1 not found in checkpoint
         [[Node: save/RestoreV2_9 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_9/tensor_names, save/RestoreV2_9/shape_and_slices)]]
2019-01-28 01:56:23.847603: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key entity_lookup_table/entity_lookup_table/Adam_1 not found in checkpoint
         [[Node: save/RestoreV2_9 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_9/tensor_names, save/RestoreV2_9/shape_and_slices)]]
2019-01-28 01:56:23.847794: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key entity_lookup_table/entity_lookup_table/Adam_1 not found in checkpoint
         [[Node: save/RestoreV2_9 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_9/tensor_names, save/RestoreV2_9/shape_and_slices)]]
2019-01-28 01:56:23.847881: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key entity_lookup_table/entity_lookup_table/Adam_1 not found in checkpoint
         [[Node: save/RestoreV2_9 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_9/tensor_names, save/RestoreV2_9/shape_and_slices)]]
2019-01-28 01:56:23.847816: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key entity_lookup_table/entity_lookup_table/Adam_1 not found in checkpoint
         [[Node: save/RestoreV2_9 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_9/tensor_names, save/RestoreV2_9/shape_and_slices)]]
2019-01-28 01:56:23.848095: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key entity_lookup_table/entity_lookup_table/Adam_1 not found in checkpoint
         [[Node: save/RestoreV2_9 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_9/tensor_names, save/RestoreV2_9/shape_and_slices)]]
2019-01-28 01:56:23.848317: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key entity_lookup_table/entity_lookup_table/Adam_1 not found in checkpoint
         [[Node: save/RestoreV2_9 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_9/tensor_names, save/RestoreV2_9/shape_and_slices)]]
2019-01-28 01:56:23.848375: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key entity_lookup_table/entity_lookup_table/Adam_1 not found in checkpoint
         [[Node: save/RestoreV2_9 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_9/tensor_names, save/RestoreV2_9/shape_and_slices)]]
2019-01-28 01:56:23.854893: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key entity_lookup_table/entity_lookup_table/Adam_1 not found in checkpoint
         [[Node: save/RestoreV2_9 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_9/tensor_names, save/RestoreV2_9/shape_and_slices)]]
2019-01-28 01:56:23.854890: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key entity_lookup_table/entity_lookup_table/Adam_1 not found in checkpoint
         [[Node: save/RestoreV2_9 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_9/tensor_names, save/RestoreV2_9/shape_and_slices)]]
2019-01-28 01:56:23.854966: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key entity_lookup_table/entity_lookup_table/Adam_1 not found in checkpoint
         [[Node: save/RestoreV2_9 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_9/tensor_names, save/RestoreV2_9/shape_and_slices)]]
2019-01-28 01:56:23.854933: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key entity_lookup_table/entity_lookup_table/Adam_1 not found in checkpoint
         [[Node: save/RestoreV2_9 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_9/tensor_names, save/RestoreV2_9/shape_and_slices)]]
2019-01-28 01:56:23.855353: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key entity_lookup_table/entity_lookup_table/Adam_1 not found in checkpoint
         [[Node: save/RestoreV2_9 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_9/tensor_names, save/RestoreV2_9/shape_and_slices)]]
2019-01-28 01:56:23.855395: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key entity_lookup_table/entity_lookup_table/Adam not found in checkpoint
2019-01-28 01:56:23.855675: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key entity_lookup_table/entity_lookup_table/Adam_1 not found in checkpoint
         [[Node: save/RestoreV2_9 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_9/tensor_names, save/RestoreV2_9/shape_and_slices)]]
2019-01-28 01:56:23.855750: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key entity_lookup_table/entity_lookup_table/Adam_1 not found in checkpoint
         [[Node: save/RestoreV2_9 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_9/tensor_names, save/RestoreV2_9/shape_and_slices)]]
2019-01-28 01:56:23.855791: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key entity_lookup_table/entity_lookup_table/Adam_1 not found in checkpoint
         [[Node: save/RestoreV2_9 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_9/tensor_names, save/RestoreV2_9/shape_and_slices)]]
2019-01-28 01:56:23.855820: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key entity_lookup_table/entity_lookup_table/Adam_1 not found in checkpoint
         [[Node: save/RestoreV2_9 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_9/tensor_names, save/RestoreV2_9/shape_and_slices)]]
2019-01-28 01:56:23.855785: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key entity_lookup_table/entity_lookup_table/Adam_1 not found in checkpoint
         [[Node: save/RestoreV2_9 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_9/tensor_names, save/RestoreV2_9/shape_and_slices)]]
2019-01-28 01:56:23.856273: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key entity_lookup_table/entity_lookup_table/Adam_1 not found in checkpoint
         [[Node: save/RestoreV2_9 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_9/tensor_names, save/RestoreV2_9/shape_and_slices)]]
2019-01-28 01:56:23.856468: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key entity_lookup_table/entity_lookup_table/Adam_1 not found in checkpoint
         [[Node: save/RestoreV2_9 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_9/tensor_names, save/RestoreV2_9/shape_and_slices)]]
2019-01-28 01:56:23.856486: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key entity_lookup_table/entity_lookup_table/Adam_1 not found in checkpoint
         [[Node: save/RestoreV2_9 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_9/tensor_names, save/RestoreV2_9/shape_and_slices)]]
2019-01-28 01:56:23.856592: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key entity_lookup_table/entity_lookup_table/Adam_1 not found in checkpoint
         [[Node: save/RestoreV2_9 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_9/tensor_names, save/RestoreV2_9/shape_and_slices)]]
2019-01-28 01:56:23.856623: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key entity_lookup_table/entity_lookup_table/Adam_1 not found in checkpoint
         [[Node: save/RestoreV2_9 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_9/tensor_names, save/RestoreV2_9/shape_and_slices)]]
2019-01-28 01:56:23.856677: W tensorflow/core/framework/op_kernel.cc:1192] Not found: Key entity_lookup_table/entity_lookup_table/Adam_1 not found in checkpoint
         [[Node: save/RestoreV2_9 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_9/tensor_names, save/RestoreV2_9/shape_and_slices)]]
Traceback (most recent call last):
  File "code/model/trainer.py", line 573, in <module>
    trainer.initialize(restore=save_path, sess=sess)
  File "code/model/trainer.py", line 143, in initialize
    return  self.model_saver.restore(sess, restore)
  File "/home/ext_user2/.local/lib/python2.7/site-packages/tensorflow/python/training/saver.py", line 1560, in restore
    {self.saver_def.filename_tensor_name: save_path})
  File "/home/ext_user2/.local/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 895, in run
    run_metadata_ptr)
  File "/home/ext_user2/.local/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1124, in _run
    feed_dict_tensor, options, run_metadata)
  File "/home/ext_user2/.local/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1321, in _do_run
    options, run_metadata)
  File "/home/ext_user2/.local/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1340, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.NotFoundError: Key entity_lookup_table/entity_lookup_table/Adam_1 not found in checkpoint
         [[Node: save/RestoreV2_9 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_9/tensor_names, save/RestoreV2_9/shape_and_slices)]]
         [[Node: save/RestoreV2_17/_3 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_80_save/RestoreV2_17", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/gpu:0"]()]]

Caused by op u'save/RestoreV2_9', defined at:
  File "code/model/trainer.py", line 573, in <module>
    trainer.initialize(restore=save_path, sess=sess)
  File "code/model/trainer.py", line 137, in initialize
    self.model_saver = tf.train.Saver(max_to_keep=2)
  File "/home/ext_user2/.local/lib/python2.7/site-packages/tensorflow/python/training/saver.py", line 1140, in __init__
    self.build()
  File "/home/ext_user2/.local/lib/python2.7/site-packages/tensorflow/python/training/saver.py", line 1172, in build
    filename=self._filename)
  File "/home/ext_user2/.local/lib/python2.7/site-packages/tensorflow/python/training/saver.py", line 688, in build
    restore_sequentially, reshape)
  File "/home/ext_user2/.local/lib/python2.7/site-packages/tensorflow/python/training/saver.py", line 407, in _AddRestoreOps
    tensors = self.restore_op(filename_tensor, saveable, preferred_shard)
  File "/home/ext_user2/.local/lib/python2.7/site-packages/tensorflow/python/training/saver.py", line 247, in restore_op
    [spec.tensor.dtype])[0])
  File "/home/ext_user2/.local/lib/python2.7/site-packages/tensorflow/python/ops/gen_io_ops.py", line 663, in restore_v2
    dtypes=dtypes, name=name)
  File "/home/ext_user2/.local/lib/python2.7/site-packages/tensorflow/python/framework/op_def_library.py", line 767, in apply_op
    op_def=op_def)
  File "/home/ext_user2/.local/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 2630, in create_op
    original_op=self._default_original_op, op_def=op_def)
  File "/home/ext_user2/.local/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 1204, in __init__
    self._traceback = self._graph._extract_stack()  # pylint: disable=protected-access

NotFoundError (see above for traceback): Key entity_lookup_table/entity_lookup_table/Adam_1 not found in checkpoint
         [[Node: save/RestoreV2_9 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_9/tensor_names, save/RestoreV2_9/shape_and_slices)]]
         [[Node: save/RestoreV2_17/_3 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_80_save/RestoreV2_17", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/gpu:0"]()]]
@kunal017 kunal017 reopened this Jan 29, 2019
@shehzaadzd
Copy link
Owner

It looks like the model was training with the train_entity_embeddings parameter set as False. Could you change that in the config and rerun?

@Albert-xj
Copy link

@kunal017 Have you solved this problem? I also encountered
NotFoundError (see above for traceback): Key policy_steps_unroll/multi_rnn_cell/cell_0/lstm_cell/bias not found in checkpoint [[Node: save/RestoreV2_18 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_arg_save/Const_0_0, save/RestoreV2_18/tensor_names, save/RestoreV2_18/shape_and_slices)]]
@shehzaadzd Can you help me?

@shehzaadzd
Copy link
Owner

Can you try training from scratch with the same parameters?

@Albert-xj
Copy link

Yes, retrain and saved model can be successfully loaded

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@shehzaadzd @kunal017 @Albert-xj