GO_bot pretrained model checkpoint is broken

[Timopheym]

Hi, i've download all the data and got an exception once run interact mode

(tensorflow) D:\work\Pilot\deeppavlov>python deep.py interact skills/go_bot/config.json
Using TensorFlow backend.
C:\Users\User\Anaconda3\envs\tensorflow\lib\site-packages\gensim\utils.py:865: UserWarning: detected Windows; aliasing chunkize to chunkize_serial
  warnings.warn("detected Windows; aliasing chunkize to chunkize_serial")
[loading vocabulary from `..\download\vocabs\word.dict`]
[loading vocabulary from `..\download\vocabs\token.dict`]
[loading vocabulary from `..\download\vocabs\tag.dict`]
[loading vocabulary from `..\download\vocabs\char.dict`]
[loading vocabulary from `..\download\vocabs\classes.dict`]
2018-01-30 20:24:57.237947: W d:\nwani\l\tensorflow_1498062690615\work\tensorflow-1.1.0\tensorflow\core\platform\cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE instructions, but these are available on your machine and could speed up CPU computations.
2018-01-30 20:24:57.238765: W d:\nwani\l\tensorflow_1498062690615\work\tensorflow-1.1.0\tensorflow\core\platform\cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE2 instructions, but these are available on your machine and could speed up CPU computations.
2018-01-30 20:24:57.241623: W d:\nwani\l\tensorflow_1498062690615\work\tensorflow-1.1.0\tensorflow\core\platform\cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE3 instructions, but these are available on your machine and could speed up CPU computations.
2018-01-30 20:24:57.243348: W d:\nwani\l\tensorflow_1498062690615\work\tensorflow-1.1.0\tensorflow\core\platform\cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.1 instructions, but these are available on your machine and could speed up CPU computations.
2018-01-30 20:24:57.245338: W d:\nwani\l\tensorflow_1498062690615\work\tensorflow-1.1.0\tensorflow\core\platform\cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use SSE4.2 instructions, but these are available on your machine and could speed up CPU computations.
2018-01-30 20:24:57.246123: W d:\nwani\l\tensorflow_1498062690615\work\tensorflow-1.1.0\tensorflow\core\platform\cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX instructions, but these are available on your machine and could speed up CPU computations.
2018-01-30 20:24:57.247209: W d:\nwani\l\tensorflow_1498062690615\work\tensorflow-1.1.0\tensorflow\core\platform\cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use AVX2 instructions, but these are available on your machine and could speed up CPU computations.
2018-01-30 20:24:57.247976: W d:\nwani\l\tensorflow_1498062690615\work\tensorflow-1.1.0\tensorflow\core\platform\cpu_feature_guard.cc:45] The TensorFlow library wasn't compiled to use FMA instructions, but these are available on your machine and could speed up CPU computations.

:: initializing `GoalOrientedBotNetwork` from saved

:: restoring checkpoint from ..\download\go_bot_rnn\model-0

2018-01-30 20:24:57.578303: W d:\nwani\l\tensorflow_1498062690615\work\tensorflow-1.1.0\tensorflow\core\framework\op_kernel.cc:1152] Not found: Key lstm_cell/weights/Adadelta_1 not found in checkpoint
2018-01-30 20:24:57.583433: W d:\nwani\l\tensorflow_1498062690615\work\tensorflow-1.1.0\tensorflow\core\framework\op_kernel.cc:1152] Not found: Key lstm_cell/weights/Adadelta not found in checkpoint
2018-01-30 20:24:57.585337: W d:\nwani\l\tensorflow_1498062690615\work\tensorflow-1.1.0\tensorflow\core\framework\op_kernel.cc:1152] Not found: Key lstm_cell/weights not found in checkpoint
2018-01-30 20:24:57.587360: W d:\nwani\l\tensorflow_1498062690615\work\tensorflow-1.1.0\tensorflow\core\framework\op_kernel.cc:1152] Not found: Key lstm_cell/biases/Adadelta_1 not found in checkpoint
2018-01-30 20:24:57.589280: W d:\nwani\l\tensorflow_1498062690615\work\tensorflow-1.1.0\tensorflow\core\framework\op_kernel.cc:1152] Not found: Key lstm_cell/biases/Adadelta not found in checkpoint
2018-01-30 20:24:57.592492: W d:\nwani\l\tensorflow_1498062690615\work\tensorflow-1.1.0\tensorflow\core\framework\op_kernel.cc:1152] Not found: Key lstm_cell/biases not found in checkpoint
Exception in <class 'deeppavlov.skills.go_bot.network.GoalOrientedBotNetwork'>
Traceback (most recent call last):
  File "C:\Users\User\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\client\session.py", line 1039, in _do_call
    return fn(*args)
  File "C:\Users\User\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\client\session.py", line 1021, in _run_fn
    status, run_metadata)
  File "C:\Users\User\Anaconda3\envs\tensorflow\lib\contextlib.py", line 88, in __exit__
    next(self.gen)
  File "C:\Users\User\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\framework\errors_impl.py", line 466, in raise_exception_on_not_ok_status
    pywrap_tensorflow.TF_GetCode(status))
tensorflow.python.framework.errors_impl.NotFoundError: Key lstm_cell/weights/Adadelta_1 not found in checkpoint
         [[Node: save/RestoreV2_18 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_recv_save/Const_0, save/RestoreV2_18/tensor_names, save/RestoreV2_18/shape_and_slices)]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "deep.py", line 63, in <module>
    main()
  File "deep.py", line 49, in main
    interact_model(pipeline_config_path)
  File "D:\work\Pilot\deeppavlov\core\commands\infer.py", line 73, in interact_model
    model = build_model_from_config(config)
  File "D:\work\Pilot\deeppavlov\core\commands\infer.py", line 34, in build_model_from_config
    model = from_params(REGISTRY[model_name], model_config, vocabs=vocabs, mode=mode)
  File "D:\work\Pilot\deeppavlov\core\common\params.py", line 49, in from_params
    mode=kwargs['mode'])
  File "D:\work\Pilot\deeppavlov\core\common\params.py", line 52, in from_params
    model = cls(**dict(config_params, **kwargs))
  File "D:\work\Pilot\deeppavlov\core\models\tf_backend.py", line 47, in __call__
    obj.__init__(*args, **kwargs)
  File "D:\work\Pilot\deeppavlov\core\models\tf_backend.py", line 28, in _wrapped
    return func(*args, **kwargs)
  File "D:\work\Pilot\deeppavlov\skills\go_bot\network.py", line 49, in __init__
    self.load()
  File "D:\work\Pilot\deeppavlov\core\models\tf_backend.py", line 28, in _wrapped
    return func(*args, **kwargs)
  File "D:\work\Pilot\deeppavlov\core\models\tf_model.py", line 114, in load
    self._saver().restore(self.sess, ckpt.model_checkpoint_path)
  File "C:\Users\User\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\training\saver.py", line 1457, in restore
    {self.saver_def.filename_tensor_name: save_path})
  File "C:\Users\User\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\client\session.py", line 778, in run
    run_metadata_ptr)
  File "C:\Users\User\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\client\session.py", line 982, in _run
    feed_dict_string, options, run_metadata)
  File "C:\Users\User\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\client\session.py", line 1032, in _do_run
    target_list, options, run_metadata)
  File "C:\Users\User\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\client\session.py", line 1052, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.NotFoundError: Key lstm_cell/weights/Adadelta_1 not found in checkpoint
         [[Node: save/RestoreV2_18 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_recv_save/Const_0, save/RestoreV2_18/tensor_names, save/RestoreV2_18/shape_and_slices)]]

Caused by op 'save/RestoreV2_18', defined at:
  File "deep.py", line 63, in <module>
    main()
  File "deep.py", line 49, in main
    interact_model(pipeline_config_path)
  File "D:\work\Pilot\deeppavlov\core\commands\infer.py", line 73, in interact_model
    model = build_model_from_config(config)
  File "D:\work\Pilot\deeppavlov\core\commands\infer.py", line 34, in build_model_from_config
    model = from_params(REGISTRY[model_name], model_config, vocabs=vocabs, mode=mode)
  File "D:\work\Pilot\deeppavlov\core\common\params.py", line 49, in from_params
    mode=kwargs['mode'])
  File "D:\work\Pilot\deeppavlov\core\common\params.py", line 52, in from_params
    model = cls(**dict(config_params, **kwargs))
  File "D:\work\Pilot\deeppavlov\core\models\tf_backend.py", line 47, in __call__
    obj.__init__(*args, **kwargs)
  File "D:\work\Pilot\deeppavlov\core\models\tf_backend.py", line 28, in _wrapped
    return func(*args, **kwargs)
  File "D:\work\Pilot\deeppavlov\skills\go_bot\network.py", line 49, in __init__
    self.load()
  File "D:\work\Pilot\deeppavlov\core\models\tf_backend.py", line 28, in _wrapped
    return func(*args, **kwargs)
  File "D:\work\Pilot\deeppavlov\core\models\tf_model.py", line 114, in load
    self._saver().restore(self.sess, ckpt.model_checkpoint_path)
  File "C:\Users\User\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\training\saver.py", line 1056, in __init__
    self.build()
  File "C:\Users\User\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\training\saver.py", line 1086, in build
    restore_sequentially=self._restore_sequentially)
  File "C:\Users\User\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\training\saver.py", line 691, in build
    restore_sequentially, reshape)
  File "C:\Users\User\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\training\saver.py", line 407, in _AddRestoreOps
    tensors = self.restore_op(filename_tensor, saveable, preferred_shard)
  File "C:\Users\User\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\training\saver.py", line 247, in restore_op
    [spec.tensor.dtype])[0])
  File "C:\Users\User\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\ops\gen_io_ops.py", line 669, in restore_v2
    dtypes=dtypes, name=name)
  File "C:\Users\User\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\framework\op_def_library.py", line 768, in apply_op
    op_def=op_def)
  File "C:\Users\User\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\framework\ops.py", line 2336, in create_op
    original_op=self._default_original_op, op_def=op_def)
  File "C:\Users\User\Anaconda3\envs\tensorflow\lib\site-packages\tensorflow\python\framework\ops.py", line 1228, in __init__
    self._traceback = _extract_stack()

NotFoundError (see above for traceback): Key lstm_cell/weights/Adadelta_1 not found in checkpoint
         [[Node: save/RestoreV2_18 = RestoreV2[dtypes=[DT_FLOAT], _device="/job:localhost/replica:0/task:0/cpu:0"](_recv_save/Const_0, save/RestoreV2_18/tensor_names, save/RestoreV2_18/shape_and_slices)]]

Should i train my own, is it wrong model or do i have wrong version of TF?

Keras==2.1.2
tensorflow==1.4.0
tensorflow-gpu==1.4.0

Thanks!

[my-master]

Hi, looks like sth is messed up with your tf installation. You have installed tensorflow==1.4.0 on your python virtual env, however, the real tf running version is 1.1.0. Look at these paths:
d:\nwani\l\tensorflow_1498062690615\work\tensorflow-1.1.0\tensorflow\core\platform\cpu_feature_guard.cc:45]

Also it requires names like "lstm_cell/biases' to be present at the checkpoint, this is an old tf naming convention.

I suggest you upgrade your tf version on d:\

[Timopheym]

Thanks you and sorry for disturb with such dump questions!