tensorflow.python.framework.errors_impl.InvalidArgumentError: 2 root error(s) found. (0) Invalid argument: Assign requires shapes of both tensors to match

[herogan2017]

when I use python gtp.py for 000990-pallas, it outputs following error.
How to solve it?thanks.

I0521 23:15:21.790073 140320694859520 saver.py:1284] Restoring parameters from /home/gzd/Others/Backup/minigo/model/000990-pallas/v17-19x19_models_000990-pallas
Traceback (most recent call last):
File "/home/gzd/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 1365, in _do_call
return fn(*args)
File "/home/gzd/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 1350, in _run_fn
target_list, run_metadata)
File "/home/gzd/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 1443, in _call_tf_sessionrun
run_metadata)
tensorflow.python.framework.errors_impl.InvalidArgumentError: 2 root error(s) found.
(0) Invalid argument: Assign requires shapes of both tensors to match. lhs shape= [361,256] rhs shape= [128,512]
[[{{node save/Assign_327}}]]
[[save/RestoreV2/_536]]
(1) Invalid argument: Assign requires shapes of both tensors to match. lhs shape= [361,256] rhs shape= [128,512]
[[{{node save/Assign_327}}]]
0 successful operations.
0 derived errors ignored.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/gzd/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/training/saver.py", line 1290, in restore
{self.saver_def.filename_tensor_name: save_path})
File "/home/gzd/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 956, in run
run_metadata_ptr)
File "/home/gzd/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 1180, in _run
feed_dict_tensor, options, run_metadata)
File "/home/gzd/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 1359, in _do_run
run_metadata)
File "/home/gzd/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 1384, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: 2 root error(s) found.
(0) Invalid argument: Assign requires shapes of both tensors to match. lhs shape= [361,256] rhs shape= [128,512]
[[node save/Assign_327 (defined at /home/gzd/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/framework/ops.py:1748) ]]
[[save/RestoreV2/_536]]
(1) Invalid argument: Assign requires shapes of both tensors to match. lhs shape= [361,256] rhs shape= [128,512]
[[node save/Assign_327 (defined at /home/gzd/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/framework/ops.py:1748) ]]
0 successful operations.
0 derived errors ignored.

Original stack trace for 'save/Assign_327':
File "gtp.py", line 93, in
app.run(main)
File "/home/gzd/anaconda3/lib/python3.7/site-packages/absl/app.py", line 299, in run
_run_main(main, args)
File "/home/gzd/anaconda3/lib/python3.7/site-packages/absl/app.py", line 250, in _run_main
sys.exit(main(argv))
File "gtp.py", line 85, in main
minigui_mode=FLAGS.minigui_mode)
File "gtp.py", line 52, in make_gtp_instance
n = DualNetwork(load_file)
File "/home/gzd/Others/Backup/minigo/dual_net.py", line 187, in init
self.initialize_graph()
File "/home/gzd/Others/Backup/minigo/dual_net.py", line 202, in initialize_graph
self.initialize_weights(self.save_file)
File "/home/gzd/Others/Backup/minigo/dual_net.py", line 212, in initialize_weights
tf.train.Saver().restore(self.sess, save_file)
File "/home/gzd/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/training/saver.py", line 828, in init
self.build()
File "/home/gzd/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/training/saver.py", line 840, in build
self._build(self._filename, build_save=True, build_restore=True)
File "/home/gzd/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/training/saver.py", line 878, in _build
build_restore=build_restore)
File "/home/gzd/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/training/saver.py", line 508, in _build_internal
restore_sequentially, reshape)
File "/home/gzd/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/training/saver.py", line 350, in _AddRestoreOps
assign_ops.append(saveable.restore(saveable_tensors, shapes))
File "/home/gzd/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/training/saving/saveable_object_util.py", line 73, in restore
self.op.get_shape().is_fully_defined())
File "/home/gzd/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/ops/state_ops.py", line 227, in assign
validate_shape=validate_shape)
File "/home/gzd/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/ops/gen_state_ops.py", line 66, in assign
use_locking=use_locking, name=name)
File "/home/gzd/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/framework/op_def_library.py", line 794, in _apply_op_helper
op_def=op_def)
File "/home/gzd/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/util/deprecation.py", line 507, in new_func
return func(*args, **kwargs)
File "/home/gzd/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/framework/ops.py", line 3357, in create_op
attrs, op_def, compute_device)
File "/home/gzd/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/framework/ops.py", line 3426, in _create_op_internal
op_def=op_def)
File "/home/gzd/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/framework/ops.py", line 1748, in init
self._traceback = tf_stack.extract_stack()

[amj]

v17 was 40-block, so try running with --trunk_layers=39, or change the default value of the flag in dual_net.py.

(it might be 40 instead of 39, i forget)

[herogan2017]

Hello, thank you for your reply.
I try to change the value in dual_net.py as following:

flags.DEFINE_integer('trunk_layers', go.N,

'The number of resnet layers in the shared trunk.')

flags.DEFINE_integer('trunk_layers', 39,
'The number of resnet layers in the shared trunk.')

But it still doesn't work.

I0522 00:44:33.190119 140452096435968 saver.py:1284] Restoring parameters from /home/gzd/Others/minigo/minigo-models/models/000990-pallas/v17-19x19_models_000990-pallas
2020-05-22 00:44:34.526539: W tensorflow/core/framework/op_kernel.cc:1651] OP_REQUIRES failed at save_restore_v2_ops.cc:184 : Not found: Key batch_normalization_39/beta not found in checkpoint
Traceback (most recent call last):
File "/home/gzd/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 1365, in _do_call
return fn(*args)
File "/home/gzd/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 1350, in _run_fn
target_list, run_metadata)
File "/home/gzd/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 1443, in _call_tf_sessionrun
run_metadata)
tensorflow.python.framework.errors_impl.NotFoundError: 2 root error(s) found.
(0) Not found: Key batch_normalization_39/beta not found in checkpoint
[[{{node save/RestoreV2}}]]
(1) Not found: Key batch_normalization_39/beta not found in checkpoint
[[{{node save/RestoreV2}}]]
[[save/RestoreV2/_49]]
0 successful operations.
0 derived errors ignored.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/gzd/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/training/saver.py", line 1290, in restore
{self.saver_def.filename_tensor_name: save_path})
File "/home/gzd/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 956, in run
run_metadata_ptr)
File "/home/gzd/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 1180, in _run
feed_dict_tensor, options, run_metadata)
File "/home/gzd/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 1359, in _do_run
run_metadata)
File "/home/gzd/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/client/session.py", line 1384, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.NotFoundError: 2 root error(s) found.
(0) Not found: Key batch_normalization_39/beta not found in checkpoint
[[node save/RestoreV2 (defined at /home/gzd/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/framework/ops.py:1748) ]]
(1) Not found: Key batch_normalization_39/beta not found in checkpoint
[[node save/RestoreV2 (defined at /home/gzd/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/framework/ops.py:1748) ]]
[[save/RestoreV2/_49]]
0 successful operations.
0 derived errors ignored.

Original stack trace for 'save/RestoreV2':
File "gtp.py", line 93, in
app.run(main)
File "/home/gzd/anaconda3/lib/python3.7/site-packages/absl/app.py", line 299, in run
_run_main(main, args)
File "/home/gzd/anaconda3/lib/python3.7/site-packages/absl/app.py", line 250, in _run_main
sys.exit(main(argv))
File "gtp.py", line 85, in main
minigui_mode=FLAGS.minigui_mode)
File "gtp.py", line 52, in make_gtp_instance
n = DualNetwork(load_file)
File "/home/gzd/Others/minigo/dual_net.py", line 190, in init
self.initialize_graph()
File "/home/gzd/Others/minigo/dual_net.py", line 205, in initialize_graph
self.initialize_weights(self.save_file)
File "/home/gzd/Others/minigo/dual_net.py", line 215, in initialize_weights
tf.train.Saver().restore(self.sess, save_file)
File "/home/gzd/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/training/saver.py", line 828, in init
self.build()
File "/home/gzd/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/training/saver.py", line 840, in build
self._build(self._filename, build_save=True, build_restore=True)
File "/home/gzd/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/training/saver.py", line 878, in _build
build_restore=build_restore)
File "/home/gzd/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/training/saver.py", line 508, in _build_internal
restore_sequentially, reshape)
File "/home/gzd/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/training/saver.py", line 328, in _AddRestoreOps
restore_sequentially)
File "/home/gzd/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/training/saver.py", line 575, in bulk_restore
return io_ops.restore_v2(filename_tensor, names, slices, dtypes)
File "/home/gzd/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/ops/gen_io_ops.py", line 1696, in restore_v2
name=name)
File "/home/gzd/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/framework/op_def_library.py", line 794, in _apply_op_helper
op_def=op_def)
File "/home/gzd/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/util/deprecation.py", line 507, in new_func
return func(*args, **kwargs)
File "/home/gzd/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/framework/ops.py", line 3357, in create_op
attrs, op_def, compute_device)
File "/home/gzd/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/framework/ops.py", line 3426, in _create_op_internal
op_def=op_def)
File "/home/gzd/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/framework/ops.py", line 1748, in init
self._traceback = tf_stack.extract_stack()

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/gzd/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/training/saver.py", line 1300, in restore
names_to_keys = object_graph_key_mapping(save_path)
File "/home/gzd/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/training/saver.py", line 1618, in object_graph_key_mapping
object_graph_string = reader.get_tensor(trackable.OBJECT_GRAPH_PROTO_KEY)
File "/home/gzd/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/pywrap_tensorflow_internal.py", line 915, in get_tensor
return CheckpointReader_GetTensor(self, compat.as_bytes(tensor_str))
tensorflow.python.framework.errors_impl.NotFoundError: Key _CHECKPOINTABLE_OBJECT_GRAPH not found in checkpoint

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "gtp.py", line 93, in
app.run(main)
File "/home/gzd/anaconda3/lib/python3.7/site-packages/absl/app.py", line 299, in run
_run_main(main, args)
File "/home/gzd/anaconda3/lib/python3.7/site-packages/absl/app.py", line 250, in _run_main
sys.exit(main(argv))
File "gtp.py", line 85, in main
minigui_mode=FLAGS.minigui_mode)
File "gtp.py", line 52, in make_gtp_instance
n = DualNetwork(load_file)
File "/home/gzd/Others/minigo/dual_net.py", line 190, in init
self.initialize_graph()
File "/home/gzd/Others/minigo/dual_net.py", line 205, in initialize_graph
self.initialize_weights(self.save_file)
File "/home/gzd/Others/minigo/dual_net.py", line 215, in initialize_weights
tf.train.Saver().restore(self.sess, save_file)
File "/home/gzd/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/training/saver.py", line 1306, in restore
err, "a Variable name or other graph key that is missing")
tensorflow.python.framework.errors_impl.NotFoundError: Restoring from checkpoint failed. This is most likely due to a Variable name or other graph key that is missing from the checkpoint. Please ensure that you have not altered the graph expected based on the checkpoint. Original error:

2 root error(s) found.
(0) Not found: Key batch_normalization_39/beta not found in checkpoint
[[node save/RestoreV2 (defined at /home/gzd/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/framework/ops.py:1748) ]]
(1) Not found: Key batch_normalization_39/beta not found in checkpoint
[[node save/RestoreV2 (defined at /home/gzd/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/framework/ops.py:1748) ]]
[[save/RestoreV2/_49]]
0 successful operations.
0 derived errors ignored.

Original stack trace for 'save/RestoreV2':
File "gtp.py", line 93, in
app.run(main)
File "/home/gzd/anaconda3/lib/python3.7/site-packages/absl/app.py", line 299, in run
_run_main(main, args)
File "/home/gzd/anaconda3/lib/python3.7/site-packages/absl/app.py", line 250, in _run_main
sys.exit(main(argv))
File "gtp.py", line 85, in main
minigui_mode=FLAGS.minigui_mode)
File "gtp.py", line 52, in make_gtp_instance
n = DualNetwork(load_file)
File "/home/gzd/Others/minigo/dual_net.py", line 190, in init
self.initialize_graph()
File "/home/gzd/Others/minigo/dual_net.py", line 205, in initialize_graph
self.initialize_weights(self.save_file)
File "/home/gzd/Others/minigo/dual_net.py", line 215, in initialize_weights
tf.train.Saver().restore(self.sess, save_file)
File "/home/gzd/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/training/saver.py", line 828, in init
self.build()
File "/home/gzd/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/training/saver.py", line 840, in build
self._build(self._filename, build_save=True, build_restore=True)
File "/home/gzd/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/training/saver.py", line 878, in _build
build_restore=build_restore)
File "/home/gzd/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/training/saver.py", line 508, in _build_internal
restore_sequentially, reshape)
File "/home/gzd/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/training/saver.py", line 328, in _AddRestoreOps
restore_sequentially)
File "/home/gzd/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/training/saver.py", line 575, in bulk_restore
return io_ops.restore_v2(filename_tensor, names, slices, dtypes)
File "/home/gzd/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/ops/gen_io_ops.py", line 1696, in restore_v2
name=name)
File "/home/gzd/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/framework/op_def_library.py", line 794, in _apply_op_helper
op_def=op_def)
File "/home/gzd/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/util/deprecation.py", line 507, in new_func
return func(*args, **kwargs)
File "/home/gzd/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/framework/ops.py", line 3357, in create_op
attrs, op_def, compute_device)
File "/home/gzd/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/framework/ops.py", line 3426, in _create_op_internal
op_def=op_def)
File "/home/gzd/anaconda3/lib/python3.7/site-packages/tensorflow_core/python/framework/ops.py", line 1748, in init
self._traceback = tf_stack.extract_stack()

[huynq55]

@herogan2017 same problem. Did you fix this issue?