Testing problem

[DR73]

Thanks for publishing the codebase.I have a question about testing.I set it up exactly as you mentioned in the README documentation.The rest arguments of testing are same as training.For example,'load_model=1' and 'model_load_dir="/output/worksfor/b2b4_3_0.05_100_0.05/model/model.ckpt"'.But I got the following error:

INFO:tensorflow:Restoring parameters from /output/worksfor/b2b4_3_0.05_100_0.05/model/model.ckpt
INFO:tensorflow:Restoring parameters from /output/worksfor/b2b4_3_0.05_100_0.05/model/model.ckpt
07/06/2019 03:42:44 PM: [ Restoring parameters from /output/worksfor/b2b4_3_0.05_100_0.05/model/model.ckpt ]
2019-07-06 15:42:44.906846: W tensorflow/core/framework/op_kernel.cc:1318] OP_REQUIRES failed at save_restore_tensor.cc:170 : Invalid argument: Unsuccessful TensorSliceReader constructor: Failed to get matching files on /output/worksfor/b2b4_3_0.05_100_0.05/model/model.ckpt: Not found: /output/worksfor/b2b4_3_0.05_100_0.05/model; No such file or directory
Traceback (most recent call last):
  File "/home/dr/.local/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1322, in _do_call
    return fn(*args)
  File "/home/dr/.local/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1307, in _run_fn
    options, feed_dict, fetch_list, target_list, run_metadata)
  File "/home/dr/.local/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1409, in _call_tf_sessionrun
    run_metadata)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Unsuccessful TensorSliceReader constructor: Failed to get matching files on /output/worksfor/b2b4_3_0.05_100_0.05/model/model.ckpt: Not found: /output/worksfor/b2b4_3_0.05_100_0.05/model; No such file or directory
	 [[Node: save/RestoreV2 = RestoreV2[dtypes=[DT_FLOAT, DT_INT32, DT_FLOAT, DT_FLOAT, DT_FLOAT, ..., DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT], _device="/job:localhost/replica:0/task:0/device:CPU:0"](_arg_save/Const_0_0, save/RestoreV2/tensor_names, save/RestoreV2/shape_and_slices)]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "code/model/trainer.py", line 574, in <module>
    trainer.initialize(restore=save_path, sess=sess)
  File "code/model/trainer.py", line 144, in initialize
    return  self.model_saver.restore(sess, restore)
  File "/home/dr/.local/lib/python3.5/site-packages/tensorflow/python/training/saver.py", line 1802, in restore
    {self.saver_def.filename_tensor_name: save_path})
  File "/home/dr/.local/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 900, in run
    run_metadata_ptr)
  File "/home/dr/.local/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1135, in _run
    feed_dict_tensor, options, run_metadata)
  File "/home/dr/.local/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1316, in _do_run
    run_metadata)
  File "/home/dr/.local/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1335, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Unsuccessful TensorSliceReader constructor: Failed to get matching files on /output/worksfor/b2b4_3_0.05_100_0.05/model/model.ckpt: Not found: /output/worksfor/b2b4_3_0.05_100_0.05/model; No such file or directory
	 [[Node: save/RestoreV2 = RestoreV2[dtypes=[DT_FLOAT, DT_INT32, DT_FLOAT, DT_FLOAT, DT_FLOAT, ..., DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT], _device="/job:localhost/replica:0/task:0/device:CPU:0"](_arg_save/Const_0_0, save/RestoreV2/tensor_names, save/RestoreV2/shape_and_slices)]]

Caused by op 'save/RestoreV2', defined at:
  File "code/model/trainer.py", line 574, in <module>
    trainer.initialize(restore=save_path, sess=sess)
  File "code/model/trainer.py", line 138, in initialize
    self.model_saver = tf.train.Saver(max_to_keep=2)
  File "/home/dr/.local/lib/python3.5/site-packages/tensorflow/python/training/saver.py", line 1338, in __init__
    self.build()
  File "/home/dr/.local/lib/python3.5/site-packages/tensorflow/python/training/saver.py", line 1347, in build
    self._build(self._filename, build_save=True, build_restore=True)
  File "/home/dr/.local/lib/python3.5/site-packages/tensorflow/python/training/saver.py", line 1384, in _build
    build_save=build_save, build_restore=build_restore)
  File "/home/dr/.local/lib/python3.5/site-packages/tensorflow/python/training/saver.py", line 835, in _build_internal
    restore_sequentially, reshape)
  File "/home/dr/.local/lib/python3.5/site-packages/tensorflow/python/training/saver.py", line 472, in _AddRestoreOps
    restore_sequentially)
  File "/home/dr/.local/lib/python3.5/site-packages/tensorflow/python/training/saver.py", line 886, in bulk_restore
    return io_ops.restore_v2(filename_tensor, names, slices, dtypes)
  File "/home/dr/.local/lib/python3.5/site-packages/tensorflow/python/ops/gen_io_ops.py", line 1463, in restore_v2
    shape_and_slices=shape_and_slices, dtypes=dtypes, name=name)
  File "/home/dr/.local/lib/python3.5/site-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper
    op_def=op_def)
  File "/home/dr/.local/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 3392, in create_op
    op_def=op_def)
  File "/home/dr/.local/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 1718, in __init__
    self._traceback = self._graph._extract_stack()  # pylint: disable=protected-access

InvalidArgumentError (see above for traceback): Unsuccessful TensorSliceReader constructor: Failed to get matching files on /output/worksfor/b2b4_3_0.05_100_0.05/model/model.ckpt: Not found: /output/worksfor/b2b4_3_0.05_100_0.05/model; No such file or directory
	 [[Node: save/RestoreV2 = RestoreV2[dtypes=[DT_FLOAT, DT_INT32, DT_FLOAT, DT_FLOAT, DT_FLOAT, ..., DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT], _device="/job:localhost/replica:0/task:0/device:CPU:0"](_arg_save/Const_0_0, save/RestoreV2/tensor_names, save/RestoreV2/shape_and_slices)]]

[DR73]

I still have a question I would like to ask you. Does the training part also include testing(I found that the test() function in trainer.py is also executed when only training)? Are the results of your papers calculated by training or by testing? Thanks again.

[shehzaadzd]

Can you train the model from scratch? I think there's a mismatch in tf versions.
The trainer class has the testing function. You could call it outside the train method also.
The results reported in the paper are using the test function.

[DR73]

Thank you for your answer.I have already solved the problem of "Testing". It needs to add a "/" symbol, like this: /output/worksfor//b2b4_3_0.05_100_0.05/model/model.ckpt.
What you mean is that when you execute the "sh run.sh configs/xxx.sh" command, you have already included a test, right?
By the way,what is the reason for each test result being different?

[shehzaadzd]

There's no constant seed used which results in different results after every run.

[DR73]

Appreciate your help!

[dmortem]

Hi,
@DR73 How do you solve the problem? What do you mean by "add a '/' symbol"? According to the readme, we need to modify "model_load_dir="saved_models/countries_s2/model.ckpt"", but it doesn't work. How to solve it?

[DR73]

Like this: /output/worksfor//b2b4_3_0.05_100_0.05/model/model.ckpt.Double '/' after relation name ‘worksfor’. But I don't know whether it applies to your situation.