Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

training error information #5

Closed
NewEricWang opened this issue Jul 15, 2019 · 6 comments
Closed

training error information #5

NewEricWang opened this issue Jul 15, 2019 · 6 comments

Comments

@NewEricWang
Copy link

NewEricWang commented Jul 15, 2019

like this:
E1: 98% (train err 160.54 ) E0 val err: inf, 3a, GPU:1.
why is E0 err inf?

And then some error information is dumped out:
2019-07-15 11:40:25.994360: W tensorflow/core/framework/op_kernel.cc:1318] OP_REQUIRES failed at tensor_array_ops.cc:497 : Invalid argument: TensorArray map/TensorArray_4_7605: Could not read from TensorArray index 0. Furthermore, the element shape is not fully defined: . It is possible you are working with a resizeable TensorArray and stop_gradients is not allowing the gradients to be written. If you set the full element_shape property on the forward TensorArray, the proper all-zeros tensor will be returned instead of incurring this error.
......

@NewEricWang
Copy link
Author

Is it necessary to run install step 2 and 3? I don't install them.

@anicolson
Copy link
Owner

Validation is not computed for epoch 0, as the network has not completed an epoch, so it is set to infinity until an epoch has been completed, see the initialised param 'val_error' on line 153:

epoch_par = {'epoch_size': len(args.train_s_list), 'epoch_comp': 0,
'start_idx': 0, 'end_idx': args.mbatch_size, 'val_error': float("inf")} # create epoch parameters.

As with your error, can you tell me what line of the script you are getting the error?

@NewEricWang
Copy link
Author

Thank you!
After the first epoch is finished, then I get some error information as follows:

2019-07-16 14:28:08.124081: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: TensorArray map/TensorArray_4_7603: Could not read from TensorArray index 0 because it has not yet been written to.
2019-07-16 14:28:08.124344: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: TensorArray map/TensorArray_4_7603: Could not read from TensorArray index 0 because it has not yet been written to.
[[Node: map/while/TensorArrayReadV3_4 = TensorArrayReadV3[dtype=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:0"](map/while/TensorArrayReadV3_4/Enter, map/while/Identity, map/while/TensorArrayReadV3_4/Enter_1)]]
2019-07-16 14:28:08.124642: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: TensorArray map/TensorArray_4_7603: Could not read from TensorArray index 0 because it has not yet been written to.
[[Node: map/while/TensorArrayReadV3_4 = TensorArrayReadV3[dtype=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:0"](map/while/TensorArrayReadV3_4/Enter, map/while/Identity, map/while/TensorArrayReadV3_4/Enter_1)]]
2019-07-16 14:28:08.124715: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: TensorArray map/TensorArray_4_7603: Could not read from TensorArray index 0 because it has not yet been written to.
[[Node: map/while/TensorArrayReadV3_4 = TensorArrayReadV3[dtype=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:0"](map/while/TensorArrayReadV3_4/Enter, map/while/Identity, map/while/TensorArrayReadV3_4/Enter_1)]]
2019-07-16 14:28:08.124760: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: TensorArray map/TensorArray_4_7603: Could not read from TensorArray index 0 because it has not yet been written to.
[[Node: map/while/TensorArrayReadV3_4 = TensorArrayReadV3[dtype=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:0"](map/while/TensorArrayReadV3_4/Enter, map/while/Identity, map/while/TensorArrayReadV3_4/Enter_1)]]
2019-07-16 14:28:08.124824: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: TensorArray map/TensorArray_4_7603: Could not read from TensorArray index 1 because it has not yet been written to.
2019-07-16 14:28:08.124862: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: TensorArray map/TensorArray_4_7603: Could not read from TensorArray index 0 because it has not yet been written to.
[[Node: map/while/TensorArrayReadV3_4 = TensorArrayReadV3[dtype=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:0"](map/while/TensorArrayReadV3_4/Enter, map/while/Identity, map/while/TensorArrayReadV3_4/Enter_1)]]
2019-07-16 14:28:08.124913: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: TensorArray map/TensorArray_4_7603: Could not read from TensorArray index 0 because it has not yet been written to.
[[Node: map/while/TensorArrayReadV3_4 = TensorArrayReadV3[dtype=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:0"](map/while/TensorArrayReadV3_4/Enter, map/while/Identity, map/while/TensorArrayReadV3_4/Enter_1)]]
2019-07-16 14:28:08.124950: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: TensorArray map/TensorArray_4_7603: Could not read from TensorArray index 0 because it has not yet been written to.
[[Node: map/while/TensorArrayReadV3_4 = TensorArrayReadV3[dtype=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:0"](map/while/TensorArrayReadV3_4/Enter, map/while/Identity, map/while/TensorArrayReadV3_4/Enter_1)]]
2019-07-16 14:28:08.125002: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: TensorArray map/TensorArray_4_7603: Could not read from TensorArray index 2 because it has not yet been written to.
2019-07-16 14:28:08.125037: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: TensorArray map/TensorArray_4_7603: Could not read from TensorArray index 0 because it has not yet been written to.
[[Node: map/while/TensorArrayReadV3_4 = TensorArrayReadV3[dtype=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:0"](map/while/TensorArrayReadV3_4/Enter, map/while/Identity, map/while/TensorArrayReadV3_4/Enter_1)]]
2019-07-16 14:28:08.125086: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: TensorArray map/TensorArray_4_7603: Could not read from TensorArray index 0 because it has not yet been written to.
[[Node: map/while/TensorArrayReadV3_4 = TensorArrayReadV3[dtype=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:0"](map/while/TensorArrayReadV3_4/Enter, map/while/Identity, map/while/TensorArrayReadV3_4/Enter_1)]]
2019-07-16 14:28:08.125121: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: TensorArray map/TensorArray_4_7603: Could not read from TensorArray index 0 because it has not yet been written to.
[[Node: map/while/TensorArrayReadV3_4 = TensorArrayReadV3[dtype=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:0"](map/while/TensorArrayReadV3_4/Enter, map/while/Identity, map/while/TensorArrayReadV3_4/Enter_1)]]
2019-07-16 14:28:08.125173: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: TensorArray map/TensorArray_4_7603: Could not read from TensorArray index 3 because it has not yet been written to.
2019-07-16 14:28:08.125206: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: TensorArray map/TensorArray_4_7603: Could not read from TensorArray index 0 because it has not yet been written to.
[[Node: map/while/TensorArrayReadV3_4 = TensorArrayReadV3[dtype=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:0"](map/while/TensorArrayReadV3_4/Enter, map/while/Identity, map/while/TensorArrayReadV3_4/Enter_1)]]
2019-07-16 14:28:08.125268: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: TensorArray map/TensorArray_4_7603: Could not read from TensorArray index 0 because it has not yet been written to.
[[Node: map/while/TensorArrayReadV3_4 = TensorArrayReadV3[dtype=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:0"](map/while/TensorArrayReadV3_4/Enter, map/while/Identity, map/while/TensorArrayReadV3_4/Enter_1)]]
2019-07-16 14:28:08.125304: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: TensorArray map/TensorArray_4_7603: Could not read from TensorArray index 0 because it has not yet been written to.
[[Node: map/while/TensorArrayReadV3_4 = TensorArrayReadV3[dtype=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:0"](map/while/TensorArrayReadV3_4/Enter, map/while/Identity, map/while/TensorArrayReadV3_4/Enter_1)]]
2019-07-16 14:28:08.125356: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: TensorArray map/TensorArray_4_7603: Could not read from TensorArray index 4 because it has not yet been written to.
2019-07-16 14:28:08.125389: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: TensorArray map/TensorArray_4_7603: Could not read from TensorArray index 0 because it has not yet been written to.
[[Node: map/while/TensorArrayReadV3_4 = TensorArrayReadV3[dtype=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:0"](map/while/TensorArrayReadV3_4/Enter, map/while/Identity, map/while/TensorArrayReadV3_4/Enter_1)]]
2019-07-16 14:28:08.125432: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: TensorArray map/TensorArray_4_7603: Could not read from TensorArray index 0 because it has not yet been written to.
[[Node: map/while/TensorArrayReadV3_4 = TensorArrayReadV3[dtype=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:0"](map/while/TensorArrayReadV3_4/Enter, map/while/Identity, map/while/TensorArrayReadV3_4/Enter_1)]]
2019-07-16 14:28:08.125470: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: TensorArray map/TensorArray_4_7603: Could not read from TensorArray index 0 because it has not yet been written to.
[[Node: map/while/TensorArrayReadV3_4 = TensorArrayReadV3[dtype=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:0"](map/while/TensorArrayReadV3_4/Enter, map/while/Identity, map/while/TensorArrayReadV3_4/Enter_1)]]
2019-07-16 14:28:08.125521: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: TensorArray map/TensorArray_4_7603: Could not read from TensorArray index 5 because it has not yet been written to.
2019-07-16 14:28:08.125578: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: TensorArray map/TensorArray_4_7603: Could not read from TensorArray index 0 because it has not yet been written to.
[[Node: map/while/TensorArrayReadV3_4 = TensorArrayReadV3[dtype=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:0"](map/while/TensorArrayReadV3_4/Enter, map/while/Identity, map/while/TensorArrayReadV3_4/Enter_1)]]
2019-07-16 14:28:08.125650: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: TensorArray map/TensorArray_4_7603: Could not read from TensorArray index 0 because it has not yet been written to.
[[Node: map/while/TensorArrayReadV3_4 = TensorArrayReadV3[dtype=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:0"](map/while/TensorArrayReadV3_4/Enter, map/while/Identity, map/while/TensorArrayReadV3_4/Enter_1)]]
2019-07-16 14:28:08.125722: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: TensorArray map/TensorArray_4_7603: Could not read from TensorArray index 0 because it has not yet been written to.
[[Node: map/while/TensorArrayReadV3_4 = TensorArrayReadV3[dtype=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:0"](map/while/TensorArrayReadV3_4/Enter, map/while/Identity, map/while/TensorArrayReadV3_4/Enter_1)]]
2019-07-16 14:28:08.125806: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: TensorArray map/TensorArray_4_7603: Could not read from TensorArray index 6 because it has not yet been written to.
2019-07-16 14:28:08.125863: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: TensorArray map/TensorArray_4_7603: Could not read from TensorArray index 0 because it has not yet been written to.
[[Node: map/while/TensorArrayReadV3_4 = TensorArrayReadV3[dtype=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:0"](map/while/TensorArrayReadV3_4/Enter, map/while/Identity, map/while/TensorArrayReadV3_4/Enter_1)]]
2019-07-16 14:28:08.125928: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: TensorArray map/TensorArray_4_7603: Could not read from TensorArray index 0 because it has not yet been written to.
[[Node: map/while/TensorArrayReadV3_4 = TensorArrayReadV3[dtype=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:0"](map/while/TensorArrayReadV3_4/Enter, map/while/Identity, map/while/TensorArrayReadV3_4/Enter_1)]]
2019-07-16 14:28:08.125981: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: TensorArray map/TensorArray_4_7603: Could not read from TensorArray index 0 because it has not yet been written to.
[[Node: map/while/TensorArrayReadV3_4 = TensorArrayReadV3[dtype=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:0"](map/while/TensorArrayReadV3_4/Enter, map/while/Identity, map/while/TensorArrayReadV3_4/Enter_1)]]
2019-07-16 14:28:08.126060: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: TensorArray map/TensorArray_4_7603: Could not read from TensorArray index 7 because it has not yet been written to.
2019-07-16 14:28:08.126112: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: TensorArray map/TensorArray_4_7603: Could not read from TensorArray index 0 because it has not yet been written to.
[[Node: map/while/TensorArrayReadV3_4 = TensorArrayReadV3[dtype=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:0"](map/while/TensorArrayReadV3_4/Enter, map/while/Identity, map/while/TensorArrayReadV3_4/Enter_1)]]
2019-07-16 14:28:08.126165: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: TensorArray map/TensorArray_4_7603: Could not read from TensorArray index 0 because it has not yet been written to.
[[Node: map/while/TensorArrayReadV3_4 = TensorArrayReadV3[dtype=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:0"](map/while/TensorArrayReadV3_4/Enter, map/while/Identity, map/while/TensorArrayReadV3_4/Enter_1)]]
2019-07-16 14:28:08.126217: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: TensorArray map/TensorArray_4_7603: Could not read from TensorArray index 0 because it has not yet been written to.
[[Node: map/while/TensorArrayReadV3_4 = TensorArrayReadV3[dtype=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:0"](map/while/TensorArrayReadV3_4/Enter, map/while/Identity, map/while/TensorArrayReadV3_4/Enter_1)]]
2019-07-16 14:28:08.126294: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: TensorArray map/TensorArray_4_7603: Could not read from TensorArray index 8 because it has not yet been written to.
2019-07-16 14:28:08.126346: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: TensorArray map/TensorArray_4_7603: Could not read from TensorArray index 0 because it has not yet been written to.
[[Node: map/while/TensorArrayReadV3_4 = TensorArrayReadV3[dtype=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:0"](map/while/TensorArrayReadV3_4/Enter, map/while/Identity, map/while/TensorArrayReadV3_4/Enter_1)]]
2019-07-16 14:28:08.126399: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: TensorArray map/TensorArray_4_7603: Could not read from TensorArray index 0 because it has not yet been written to.
[[Node: map/while/TensorArrayReadV3_4 = TensorArrayReadV3[dtype=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:0"](map/while/TensorArrayReadV3_4/Enter, map/while/Identity, map/while/TensorArrayReadV3_4/Enter_1)]]
2019-07-16 14:28:08.126452: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: TensorArray map/TensorArray_4_7603: Could not read from TensorArray index 0 because it has not yet been written to.
[[Node: map/while/TensorArrayReadV3_4 = TensorArrayReadV3[dtype=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:0"](map/while/TensorArrayReadV3_4/Enter, map/while/Identity, map/while/TensorArrayReadV3_4/Enter_1)]]
2019-07-16 14:28:08.126499: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: TensorArray map/TensorArray_4_7603: Could not read from TensorArray index 9 because it has not yet been written to.
2019-07-16 14:28:08.126532: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: TensorArray map/TensorArray_4_7603: Could not read from TensorArray index 0 because it has not yet been written to.
[[Node: map/while/TensorArrayReadV3_4 = TensorArrayReadV3[dtype=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:0"](map/while/TensorArrayReadV3_4/Enter, map/while/Identity, map/while/TensorArrayReadV3_4/Enter_1)]]
2019-07-16 14:28:08.126581: W tensorflow/core/framework/op_kernel.cc:1192] Invalid argument: TensorArray map/TensorArray_4_7603: Could not read from TensorArray index 0 because it has not yet been written to.
[[Node: map/while/TensorArrayReadV3_4 = TensorArrayReadV3[dtype=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:0"](map/while/TensorArrayReadV3_4/Enter, map/while/Identity, map/while/TensorArrayReadV3_4/Enter_1)]]
Traceback (most recent call last):
File "/restools/tools/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1323, in _do_call
return fn(*args)
File "/restools/tools/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1302, in _run_fn
status, run_metadata)
File "/restools/tools/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/errors_impl.py", line 473, in exit
c_api.TF_GetCode(self.status.status))
tensorflow.python.framework.errors_impl.InvalidArgumentError: TensorArray map/TensorArray_4_7603: Could not read from TensorArray index 0 because it has not yet been written to.
[[Node: map/while/TensorArrayReadV3_4 = TensorArrayReadV3[dtype=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:0"](map/while/TensorArrayReadV3_4/Enter, map/while/Identity, map/while/TensorArrayReadV3_4/Enter_1)]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "deepxi.py", line 290, in
if args.train: train(sess, net, args)
File "deepxi.py", line 194, in train
net.s_len_ph: args.val_s_len[start_idx:end_idx], net.d_len_ph: args.val_d_len[start_idx:end_idx], net.snr_ph: args.val_snr[start_idx:end_idx]}) # mini-batch.
File "/restools/tools/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 889, in run
run_metadata_ptr)
File "/restools/tools/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1120, in _run
feed_dict_tensor, options, run_metadata)
File "/restools/tools/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1317, in _do_run
options, run_metadata)
File "/restools/tools/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1336, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: TensorArray map/TensorArray_4_7603: Could not read from TensorArray index 0 because it has not yet been written to.
[[Node: map/while/TensorArrayReadV3_4 = TensorArrayReadV3[dtype=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:0"](map/while/TensorArrayReadV3_4/Enter, map/while/Identity, map/while/TensorArrayReadV3_4/Enter_1)]]

Caused by op 'map/while/TensorArrayReadV3_4', defined at:
File "deepxi.py", line 282, in
net = deepxi_net(args)
File "deepxi.py", line 85, in init
args.fs, self.P, args.nconst, args.mu, args.sigma) # feature graph.
File "lib/feat.py", line 43, in xi_mapped
P, nconst), (s, d, s_len, d_len, Q), dtype=(tf.float32, tf.float32, tf.float32)) # padded waveforms.
File "/restools/tools/anaconda3/lib/python3.6/site-packages/tensorflow/python/ops/functional_ops.py", line 389, in map_fn
swap_memory=swap_memory)
File "/restools/tools/anaconda3/lib/python3.6/site-packages/tensorflow/python/ops/control_flow_ops.py", line 2816, in while_loop
result = loop_context.BuildLoop(cond, body, loop_vars, shape_invariants)
File "/restools/tools/anaconda3/lib/python3.6/site-packages/tensorflow/python/ops/control_flow_ops.py", line 2640, in BuildLoop
pred, body, original_loop_vars, loop_vars, shape_invariants)
File "/restools/tools/anaconda3/lib/python3.6/site-packages/tensorflow/python/ops/control_flow_ops.py", line 2590, in _BuildLoop
body_result = body(*packed_vars_for_body)
File "/restools/tools/anaconda3/lib/python3.6/site-packages/tensorflow/python/ops/functional_ops.py", line 378, in compute
packed_values = input_pack([elem_ta.read(i) for elem_ta in elems_ta])
File "/restools/tools/anaconda3/lib/python3.6/site-packages/tensorflow/python/ops/functional_ops.py", line 378, in
packed_values = input_pack([elem_ta.read(i) for elem_ta in elems_ta])
File "/restools/tools/anaconda3/lib/python3.6/site-packages/tensorflow/python/util/tf_should_use.py", line 49, in fn
return method(self, *args, **kwargs)
File "/restools/tools/anaconda3/lib/python3.6/site-packages/tensorflow/python/util/tf_should_use.py", line 49, in fn
return method(self, *args, **kwargs)
File "/restools/tools/anaconda3/lib/python3.6/site-packages/tensorflow/python/ops/tensor_array_ops.py", line 281, in read
name=name)
File "/restools/tools/anaconda3/lib/python3.6/site-packages/tensorflow/python/ops/gen_data_flow_ops.py", line 4445, in _tensor_array_read_v3
dtype=dtype, name=name)
File "/restools/tools/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper
op_def=op_def)
File "/restools/tools/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 2956, in create_op
op_def=op_def)
File "/restools/tools/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1470, in init
self._traceback = self._graph._extract_stack() # pylint: disable=protected-access

InvalidArgumentError (see above for traceback): TensorArray map/TensorArray_4_7603: Could not read from TensorArray index 0 because it has not yet been written to.
[[Node: map/while/TensorArrayReadV3_4 = TensorArrayReadV3[dtype=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:0"](map/while/TensorArrayReadV3_4/Enter, map/while/Identity, map/while/TensorArrayReadV3_4/Enter_1)]]

@anicolson
Copy link
Owner

can you please check your args.val_s, args.val_d, args.val_s_len, args.val_d_len, and args.val_snr numpy arrays? e.g. print(args.val_s.shape) to check if they are the right shape.

@NewEricWang
Copy link
Author

NewEricWang commented Jul 16, 2019

@anicolson , Thank you for your help! I get it. Snr information don't be included in my val data's file name.
Another question:
Must I set snr level for each training data and val data on their file name?

@anicolson
Copy link
Owner

Only for the validation set. It's just so the validation set is always the same. Please read the following and can you tell me if you understand it:


For the validation set only:

Identical filenames for the clean speech and noise must be placed in 'val_clean_speech'
and 'val_noise', with the SNR at the end of the filename. The format of the filename is:
'x_y_zdB', where 'x' is the clean speech name, 'y' is the noise source name and z is the SNR level.
The clean speech and noise will be mixed together at the SNR level specified in the filename during
validation

As an example:
'./val_clean_speech/198_19-198-0003_Machinery17_15dB.wav' contains the clean speech,
and './val_noise/198_19-198-0003_Machinery17_15dB.wav' contains the noise. The number
of samples in the clean speech and noise must be identical.

This must be done, as the clean speech and noise used to create the noisy speech
(noisy speech = clean speech + noise) are required to compute the a priori SNR target.


Thank you

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants