Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Training Error? #8

Closed
rishabh-ojha98 opened this issue Sep 9, 2019 · 12 comments
Closed

Training Error? #8

rishabh-ojha98 opened this issue Sep 9, 2019 · 12 comments

Comments

@rishabh-ojha98
Copy link

While following all the training steps i encountered the following training error again and again. Whereas i am not getting from where this error has been occurred?
Please explain me a bit about error and how to resolve the same?

`E1: 100.0% (train err 109.10), E0 val err: inf, 3a, GPU:0.
Traceback (most recent call last):
File "/home/paperspace/anaconda3/envs/tensorflow_gpuenv/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1356, in _do_call
return fn(*args)
File "/home/paperspace/anaconda3/envs/tensorflow_gpuenv/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1341, in _run_fn
options, feed_dict, fetch_list, target_list, run_metadata)
File "/home/paperspace/anaconda3/envs/tensorflow_gpuenv/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1429, in _call_tf_sessionrun
run_metadata)
tensorflow.python.framework.errors_impl.InvalidArgumentError: 2 root error(s) found.
(0) Invalid argument: Need minval < maxval, got 0 >= -39117
[[{{node map/while/random_uniform}}]]
[[map/while/Slice_1/size/_2979]]
(1) Invalid argument: Need minval < maxval, got 0 >= -39117
[[{{node map/while/random_uniform}}]]
0 successful operations.
0 derived errors ignored.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "deepxi.py", line 290, in
if args.train: train(sess, net, args)
File "deepxi.py", line 195, in train
net.s_len_ph: args.val_s_len[start_idx:end_idx], net.d_len_ph: args.val_d_len[start_idx:end_idx], net.snr_ph: args.val_snr[start_idx:end_idx]}) # mini-batch.
File "/home/paperspace/anaconda3/envs/tensorflow_gpuenv/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 950, in run
run_metadata_ptr)
File "/home/paperspace/anaconda3/envs/tensorflow_gpuenv/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1173, in _run
feed_dict_tensor, options, run_metadata)
File "/home/paperspace/anaconda3/envs/tensorflow_gpuenv/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1350, in _do_run
run_metadata)
File "/home/paperspace/anaconda3/envs/tensorflow_gpuenv/lib/python3.7/site-packages/tensorflow/python/client/session.py", line 1370, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: 2 root error(s) found.
(0) Invalid argument: Need minval < maxval, got 0 >= -39117
[[node map/while/random_uniform (defined at lib/feat.py:434) ]]
[[map/while/Slice_1/size/_2979]]
(1) Invalid argument: Need minval < maxval, got 0 >= -39117
[[node map/while/random_uniform (defined at lib/feat.py:434) ]]
0 successful operations.
0 derived errors ignored.

Original stack trace for 'map/while/random_uniform':
File "deepxi.py", line 284, in
net = deepxi_net(args)
File "deepxi.py", line 86, in init
self.feature = feat.xi_mapped(self.s_ph, self.d_ph, self.s_len_ph, self.d_len_ph, self.snr_ph, args.Nw, args.Ns, args.NFFT, args.fs, self.P, args.nconst, self.mu, self.sigma) # feature graph.
File "lib/feat.py", line 43, in xi_mapped
P, nconst), (s, d, s_len, d_len, Q), dtype=(tf.float32, tf.float32, tf.float32)) # padded waveforms.
File "/home/paperspace/anaconda3/envs/tensorflow_gpuenv/lib/python3.7/site-packages/tensorflow/python/ops/map_fn.py", line 268, in map_fn
maximum_iterations=n)
File "/home/paperspace/anaconda3/envs/tensorflow_gpuenv/lib/python3.7/site-packages/tensorflow/python/ops/control_flow_ops.py", line 3501, in while_loop
return_same_structure)
File "/home/paperspace/anaconda3/envs/tensorflow_gpuenv/lib/python3.7/site-packages/tensorflow/python/ops/control_flow_ops.py", line 3012, in BuildLoop
pred, body, original_loop_vars, loop_vars, shape_invariants)
File "/home/paperspace/anaconda3/envs/tensorflow_gpuenv/lib/python3.7/site-packages/tensorflow/python/ops/control_flow_ops.py", line 2937, in _BuildLoop
body_result = body(*packed_vars_for_body)
File "/home/paperspace/anaconda3/envs/tensorflow_gpuenv/lib/python3.7/site-packages/tensorflow/python/ops/control_flow_ops.py", line 3456, in
body = lambda i, lv: (i + 1, orig_body(*lv))
File "/home/paperspace/anaconda3/envs/tensorflow_gpuenv/lib/python3.7/site-packages/tensorflow/python/ops/map_fn.py", line 257, in compute
packed_fn_values = fn(packed_values)
File "lib/feat.py", line 43, in
P, nconst), (s, d, s_len, d_len, Q), dtype=(tf.float32, tf.float32, tf.float32)) # padded waveforms.
File "lib/feat.py", line 410, in addnoisepad
(y, d) = addnoise(x, d, Q) # compute noisy waveform.
File "lib/feat.py", line 434, in addnoise
i = tf.random_uniform([1], 0, tf.add(1, tf.subtract(d_len, x_len)), tf.int32)
File "/home/paperspace/anaconda3/envs/tensorflow_gpuenv/lib/python3.7/site-packages/tensorflow/python/ops/random_ops.py", line 245, in random_uniform
shape, minval, maxval, seed=seed1, seed2=seed2, name=name)
File "/home/paperspace/anaconda3/envs/tensorflow_gpuenv/lib/python3.7/site-packages/tensorflow/python/ops/gen_random_ops.py", line 919, in random_uniform_int
seed=seed, seed2=seed2, name=name)
File "/home/paperspace/anaconda3/envs/tensorflow_gpuenv/lib/python3.7/site-packages/tensorflow/python/framework/op_def_library.py", line 788, in _apply_op_helper
op_def=op_def)
File "/home/paperspace/anaconda3/envs/tensorflow_gpuenv/lib/python3.7/site-packages/tensorflow/python/util/deprecation.py", line 507, in new_func
return func(*args, **kwargs)
File "/home/paperspace/anaconda3/envs/tensorflow_gpuenv/lib/python3.7/site-packages/tensorflow/python/framework/ops.py", line 3616, in create_op
op_def=op_def)
File "/home/paperspace/anaconda3/envs/tensorflow_gpuenv/lib/python3.7/site-packages/tensorflow/python/framework/ops.py", line 2005, in init
self._traceback = tf_stack.extract_stack()

`
@anicolson
Copy link
Owner

it seems that there is a problem with your validation set, can you check the size of args.val_s, args.val_s_len, args.val_d, args.val_d_len, and args.val_snr?

@rishabh-ojha98
Copy link
Author

rishabh-ojha98 commented Sep 9, 2019 •
edited

Shape of the validation set parameters are as follows:

args.val_s
(1000, 273520)

args.val_s_len
(1000,)

args.val_snr
(1000,)

args.val_d
(1000, 4832000)

args.val_d_len
(1000,)

Note: I have also added few of the noises of my use case in training as well as in validation set, if that is a problem or not?

@anicolson
Copy link
Owner

this should solve your problems: https://github.com/anicolson/DeepXi/blob/master/set/info.txt

@rishabh-ojha98
Copy link
Author

rishabh-ojha98 commented Sep 9, 2019 •
edited

I have followed this info.txt
But given SNR values randomly as [5, 10]

The file name in val_clean_speech are as follows:
1001-134707-0015_noise-free-sound-0210_10dB.wav
1081-125237-0001_noise-free-sound-0531_10dB.wav
1224-121064-0078_noise-free-sound-0768_10dB.wav
1335-163935-0006_noise-free-sound-0592_5dB.wav
1001-134707-0021_uspolo_10dB.wav
1081-125237-0002_noise-free-sound-0365_10dB.wav
1224-121064-0107_noise-free-sound-0157_10dB.wav
1335-163935-0011_noise-free-sound-0768_5dB.wav

The file name in val_noise are as follows:
1001-134707-0015_noise-free-sound-0210_5dB.wav
1081-125237-0001_noise-free-sound-0531_10dB.wav
1224-121064-0078_noise-free-sound-0768_10dB.wav
1335-163935-0006_noise-free-sound-0592_5dB.wav
1001-134707-0021_uspolo_5dB.wav
1081-125237-0002_noise-free-sound-0365_10dB.wav
1224-121064-0107_noise-free-sound-0157_10dB.wav
1335-163935-0011_noise-free-sound-0768_10dB.wav

Is there any issue with randomly generating SNR values? so let me know.

@anicolson
Copy link
Owner

anicolson commented Sep 9, 2019 via email

@rishabh-ojha98
Copy link
Author

rishabh-ojha98 commented Sep 9, 2019 •
edited

Got it!!
Thanks @anicolson
Issue is solved now, will let you know if encountered any problem.

@anicolson
Copy link
Owner

anicolson commented Sep 9, 2019 via email

@rishabh-ojha98
Copy link
Author

does each pair have the same number of samples?
~ Do we need to have equal number of sampled in each pairs??

@anicolson
Copy link
Owner

anicolson commented Sep 10, 2019 via email

@rishabh-ojha98
Copy link
Author

yeah got it, thanks

@rishabh-ojha98
Copy link
Author

Just want to clear my doubt
Is there any significance of having dB value in my inferencing wav file?
CA_CA01_03_voice_babble_5dB.wav

@anicolson
Copy link
Owner

anicolson commented Sep 18, 2019 via email

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@anicolson @rishabh-ojha98