Error reported to Coordinator: <class 'tensorflow.python.framework.errors_impl.InvalidArgumentError'>

[TanyaStevens]

Hello,

Thanks for the amazing work! :) I have been trying to run a test data. This is the error I am getting:

INFO:tensorflow:Error reported to Coordinator: <class 'tensorflow.python.framework.errors_impl.InvalidArgumentError'>, 2 root error(s) found.
  (0) Invalid argument: Expect 8 fields but have 13 in record 0
	 [[{{node DecodeCSV}}]]
	 [[IteratorGetNext]]
  (1) Invalid argument: Expect 8 fields but have 13 in record 0
	 [[{{node DecodeCSV}}]]
	 [[IteratorGetNext]]
	 [[IteratorGetNext/_231]]
0 successful operations.
0 derived errors ignored.
I0406 06:04:50.266259 139791091263232 coordinator.py:224] Error reported to Coordinator: <class 'tensorflow.python.framework.errors_impl.InvalidArgumentError'>, 2 root error(s) found.
  (0) Invalid argument: Expect 8 fields but have 13 in record 0
	 [[{{node DecodeCSV}}]]
	 [[IteratorGetNext]]
  (1) Invalid argument: Expect 8 fields but have 13 in record 0
	 [[{{node DecodeCSV}}]]
	 [[IteratorGetNext]]
	 [[IteratorGetNext/_231]]
0 successful operations.
0 derived errors ignored.
Traceback (most recent call last):
  File "test.py", line 399, in <module>
    tf.app.run()
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/platform/app.py", line 40, in run
    _run(main=main, argv=argv, flags_parser=_parse_flags_tolerate_undef)
  File "/usr/local/lib/python2.7/dist-packages/absl/app.py", line 303, in run
    _run_main(main, args)
  File "/usr/local/lib/python2.7/dist-packages/absl/app.py", line 251, in _run_main
    sys.exit(main(argv))
  File "test.py", line 206, in main
    [ins, outs, jitter_outs, step] = sess.run([inputs, outputs, jitter_outputs, global_step])
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 950, in run
    run_metadata_ptr)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 1173, in _run
    feed_dict_tensor, options, run_metadata)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 1350, in _do_run
    run_metadata)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 1370, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: 2 root error(s) found.
  (0) Invalid argument: Expect 8 fields but have 13 in record 0
	 [[{{node DecodeCSV}}]]
	 [[IteratorGetNext]]
  (1) Invalid argument: Expect 8 fields but have 13 in record 0
	 [[{{node DecodeCSV}}]]
	 [[IteratorGetNext]]
	 [[IteratorGetNext/_231]]
0 successful operations.
0 derived errors ignored.

Below is the command I have been using:

python test.py \
--cameras_glob '/home/DNN_Methods/Softwares/matryodshka-main/replica-6dof/6dof/apartment_0_6dof.txt' \
--image_dir '/home/DNN_Methods/Softwares/matryodshka-main/input/' \
--test_type on_video \
--input_type ODS \
--experiment_name matryodshka-with-transform-inverse-reg-checkpoint \
--checkpoint_dir '/home/DNN_Methods/Softwares/matryodshka-main/pretrained-models/' \
--output_root '/home/DNN_Methods/Softwares/matryodshka-main/trial_1/' \
--coord_net

Could you please tell me where it is going wrong?

Thanks very much! Any help would be much appreciated :)

[iszihan]

Hello, this error seems to indicate you are using a wrong txt file as the --cameras_glob input for dataloading. Did you try download the glob files using ./download_glob.sh script? And for testing, you should point to glob/test/regular/apartment_1_6dof.txt file if you are testing on independent frames and glob/test/video/*.txt if testing on video data. Hope this helps!

[TanyaStevens]

Hello @iszihan , thanks very much for your reply :)
I had tried with the "glob/test/video/apartment_0.txt" before. This was the error. I think I am going wrong somewhere while loading the pre-trained models. Would really appreciate any help! Thanks :)

Traceback (most recent call last):
  File "test.py", line 402, in <module>
    tf.app.run()
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/platform/app.py", line 40, in run
    _run(main=main, argv=argv, flags_parser=_parse_flags_tolerate_undef)
  File "/usr/local/lib/python2.7/dist-packages/absl/app.py", line 303, in run
    _run_main(main, args)
  File "/usr/local/lib/python2.7/dist-packages/absl/app.py", line 251, in _run_main
    sys.exit(main(argv))
  File "test.py", line 207, in main
    saver.restore(sess, ckpt_file)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/saver.py", line 1322, in restore
    err, "a mismatch between the current graph and the graph")
tensorflow.python.framework.errors_impl.InvalidArgumentError: **Restoring from checkpoint failed. This is most likely due to a mismatch between the current graph and the graph from the checkpoint. Please ensure that you have not altered the graph expected based on the checkpoint.** Original error:

**Assign requires shapes of both tensors to match. lhs shape= [3,3,128,128] rhs shape= [3,3,129,128]**
	 [[node save/Assign_47 (defined at test.py:193) ]]

Errors may have originated from an input operation.
Input Source operations connected to node save/Assign_47:
 net/conv7_2/weights (defined at /home/DNN_Methods/Softwares/matryodshka-main/matryodshka/nets.py:436)

Original stack trace for u'save/Assign_47':
  File "test.py", line 402, in <module>
    tf.app.run()
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/platform/app.py", line 40, in run
    _run(main=main, argv=argv, flags_parser=_parse_flags_tolerate_undef)
  File "/usr/local/lib/python2.7/dist-packages/absl/app.py", line 303, in run
    _run_main(main, args)
  File "/usr/local/lib/python2.7/dist-packages/absl/app.py", line 251, in _run_main
    sys.exit(main(argv))
  File "test.py", line 193, in main
    saver = tf.train.Saver([var for var in tf.trainable_variables()]+[global_step])

Here is my command line, checkpoints and input directory.
Command Line:

python test.py \
--cameras_glob '/home/DNN_Methods/Softwares/matryodshka-main/glob/test/video/preprocessed_apartment_0.txt' \
--image_dir '/home/DNN_Methods/Softwares/matryodshka-main/input/' \
--test_type on_video \
--input_type ODS \
--experiment_name matryodshka-checkpoint \
--checkpoint_dir '/home/DNN_Methods/Softwares/matryodshka-main/pretrained-models/' \
--output_root '/home/DNN_Methods/Softwares/matryodshka-main/trial_1/' \

Checkpoints: (Inside pretrained directory)

|-matryodshka.pb
|-matryodshka.onnx
|-matryodshka-checkpoint
| |-checkpoint
| |-graph.pbtxt
| |-model.latest-140000.meta
| |-model.latest-140000.index
| |-model.latest-140000.data-00000-of-00001
|-matryodshka-with-transform-inverse-reg.pb
|-matryodshka-with-transform-inverse-reg.onnx
|-matryodshka-with-transform-inverse-reg-checkpoint
| |-checkpoint
| |-graph.pbtxt
| |-model.latest-138000.meta
| |-model.latest-138000.index
| |-model.latest-138000.data-00000-of-00001
|-graph.pbtxt

Input directory:

 |-apartment_0_0003_pos4.jpeg
 |-apartment_0_0013_pos0.jpeg
 |-apartment_0_0017_pos2.jpeg
 |-apartment_0_0017_pos3.jpeg
 |-apartment_0_0032_pos5.jpeg
 |-apartment_0_0034_pos2.jpeg
 |-apartment_0_0037_pos0.jpeg
.....

p.s. I have tried with both the checkpoints. In both the cases, I get the same error. The test images are from your link "test_video_640x320".

[iszihan]

Hey, could you try adding the flag --coord_net and see if the error still occurs? Sorry for not specifying the type of the model for the pre-trained checkpoints.

[TanyaStevens]

Hi @iszihan , thanks for your suggestion. It worked!

One minor addition if I may suggest:
In test.py, if you could add/define "jitter_output_depth_image" in line 165, it can avoid a KeyError in line 237.

Thanks very much for your help :)