IndexError: index 0 is out of bounds for axis 0 with size 0 #442

mrgloom · 2019-03-17T23:21:49Z

Get this error after runned time ./run_full_voice.sh:

2019-03-18 02:18:11,773 INFO    param_generation: processing    1 of    1: /data/user_data/external_projects/text-to-speech/merlin/egs/slt_arctic/s1/experiments/slt_arctic_full/test_synthesis/wav/test_me.cmp
Traceback (most recent call last):
  File "/data/user_data/external_projects/text-to-speech/merlin/src/run_merlin.py", line 1320, in <module>
    main_function(cfg)
  File "/data/user_data/external_projects/text-to-speech/merlin/src/run_merlin.py", line 971, in main_function
    generator.acoustic_decomposition(gen_file_list, cfg.cmp_dim, cfg.out_dimension_dict, cfg.file_extension_dict, var_file_dict, do_MLPG=cfg.do_MLPG, cfg=cfg)
  File "/data/user_data/external_projects/text-to-speech/merlin/src/frontend/parameter_generation.py", line 164, in acoustic_decomposition
    gen_features = mlpg_algo.generation(current_features, var, out_dimension_dict[feature_name]//3)
  File "/data/user_data/external_projects/text-to-speech/merlin/src/frontend/mlpg_fast.py", line 122, in generation
    var_frames[0, 1] = 100000000000;
IndexError: index 0 is out of bounds for axis 0 with size 0
Lock freed
deleting intermediate synthesis files...
synthesized audio files are in: experiments/slt_arctic_full/test_synthesis/wav
All successfull!! Your demo voice is ready :)

real	85m11.509s
user	525m7.484s
sys	404m31.404s

Also get similar error when run ./merlin_synthesis.sh:

2019-03-18 02:22:23,187 INFO           main    : Input label dimension is 425
2019-03-18 02:22:23,188 INFO           main    : preparing label data (input) using standard HTS style labels
2019-03-18 02:22:23,190 INFO           labels  : loaded /data/user_data/external_projects/text-to-speech/merlin/egs/slt_arctic/s1/experiments/slt_arctic_full/test_synthesis/gen-lab/test_me.lab,   0 labels
WARNING: no silence found!
2019-03-18 02:22:23,395 INFO      acoustic_norm: Loaded min max values from the trained data for feature dimension of 425
2019-03-18 02:22:23,422 INFO           main    : label dimension is 425
2019-03-18 02:22:23,422 INFO           main    : generating from DNN
2019-03-18 02:22:23,504 INFO     dnn_generation: generating    1 of    1: /data/user_data/external_projects/text-to-speech/merlin/egs/slt_arctic/s1/experiments/slt_arctic_full/test_synthesis/wav/test_me.lab
2019-03-18 02:22:24,987 INFO    param_generation: processing    1 of    1: /data/user_data/external_projects/text-to-speech/merlin/egs/slt_arctic/s1/experiments/slt_arctic_full/test_synthesis/wav/test_me.cmp
Traceback (most recent call last):
  File "/data/user_data/external_projects/text-to-speech/merlin/src/run_merlin.py", line 1320, in <module>
    main_function(cfg)
  File "/data/user_data/external_projects/text-to-speech/merlin/src/run_merlin.py", line 971, in main_function
    generator.acoustic_decomposition(gen_file_list, cfg.cmp_dim, cfg.out_dimension_dict, cfg.file_extension_dict, var_file_dict, do_MLPG=cfg.do_MLPG, cfg=cfg)
  File "/data/user_data/external_projects/text-to-speech/merlin/src/frontend/parameter_generation.py", line 164, in acoustic_decomposition
    gen_features = mlpg_algo.generation(current_features, var, out_dimension_dict[feature_name]//3)
  File "/data/user_data/external_projects/text-to-speech/merlin/src/frontend/mlpg_fast.py", line 122, in generation
    var_frames[0, 1] = 100000000000;
IndexError: index 0 is out of bounds for axis 0 with size 0
Lock freed
deleting intermediate synthesis files...
synthesized audio files are in: experiments/slt_arctic_full/test_synthesis/wav

And dir experiments/slt_arctic_full/test_synthesis/wav is empty.

The text was updated successfully, but these errors were encountered:

ZackHodari · 2019-04-06T14:48:35Z

It looks like var_frames is empty figuring out why may help solve the issue.

Following the trace it seems that this comes from features, frame_number = io_funcs.load_binary_file_frame(file_name, dimension)

Try loading that file in a python prompt and see if it is empty or not.

mrgloom · 2019-04-08T00:08:49Z

Looks like the reason that on step 3

merlin/egs/slt_arctic/s1/run_full_voice.sh

Line 15 in 9df27e9

./03_train_duration_model.sh conf/duration_slt_arctic_full.conf

I was getting an error about the same as here:
#447
#414
#418

ERROR (theano.gpuarray): pygpu was configured but could not be imported or is too old (version 0.7 or higher required)

But processing don't stop, at step 4 I see that all CPU cores are 100% load, is it desired behaviour or does it mean that theano failed to run on GPU?

At which step parameter_generation.py is used?

mrgloom · 2019-04-08T09:42:30Z

import theano work without an error.

print(theano.__version__)
1.0.4

Looks like pygpu can be build like:
http://deeplearning.net/software/libgpuarray/installation.html#step-by-step-install

Running step 3 ./03_train_duration_model.sh conf/duration_slt_arctic_full.conf

MERLIN_THEANO_FLAGS:
    cuda.root=/usr/local/8.0
    floatX=float32
    on_unused_input=ignore

Running on GPU id=1 ...
Can not use cuDNN on context None: cannot compile with cuDNN. We got this error:
b'In file included from /tmp/try_flags_hi2fruvy.c:4:0:\n/usr/include/cudnn.h:64:26: fatal error: cuda_runtime.h: No such file or directory\n #include <cuda_runtime.h>\n       $
Mapped name None to device cuda1: GeForce GTX TITAN X (0000:01:00.0)

Theano version: 1.0.4
THEANO_FLAGS: mode=FAST_RUN,device=cuda1,cuda.root=/usr/local/8.0,floatX=float32,on_unused_input=ignore
device: cuda1

As I can see
find /usr/local/cuda/ -name "*cuda_runtime.h"
is under
/usr/local/cuda/targets/x86_64-linux/include/cuda_runtime.h

not sure why /usr/local/8.0/ because I have /usr/local/cuda-9.0 linked to /usr/local/cuda

Path can be changed here:

merlin/src/setup_env.sh

Line 13 in 9df27e9

    
           MERLIN_THEANO_FLAGS="cuda.root=/usr/local/8.0,floatX=float32,on_unused_input=ignore"

But I still get IndexError: index 0 is out of bounds for axis 0 with size 0 error.

ZackHodari · 2019-04-10T12:30:04Z

If pygpu fails, Theano will fall back to using the cpu and everything will work correctly, but may be slower.

parameter_generation.py is called in run_merlin.py when the config option DNNGEN is set to True and either AcousticModel or DurationModel is True.

You shouldn't need to install pygpu from scratch, try using conda or pip.

Did you try changing MERLIN_THEANO_FLAGS?

If you are still getting the IndexError something is probably breaking earlier, you'll need to figure out where based on why var_frames was empty.

mrgloom · 2019-04-15T02:41:16Z

Did you try changing MERLIN_THEANO_FLAGS?
Yes, I set my cuda path location.

The file that is not loaded is used on step 4 and have path is like merlin/egs/slt_arctic/s1/experiments/slt_arctic_full/acoustic_model/gen/feed_forward_6_tanh/arctic_b0539.cmp
Any ideas on which step and where in the code it should be generated?

Looks like it setted here:

merlin/egs/slt_arctic/s1/conf/dnn/acoustic_slt_arctic_full.conf

Line 79 in 9df27e9

model_file_name: feed_forward_6_tanh

Maybe these paths are not relative and need to be changed?

merlin/egs/slt_arctic/s1/conf/dnn/acoustic_slt_arctic_full.conf

Line 5 in 9df27e9

TOPLEVEL: /group/project/cstr1/srikanth/test_v5/merlin/egs/slt_arctic/s1

ZackHodari · 2019-04-15T09:20:01Z

That is the composed (.cmp) predictions made by the model in frontend.parameter_generation.ParameterGeneration within DNNGEN of run_merlin.py.

You can see this will be run from line 123 in your config.

Try loading arctic_b0539.cmp using io_funcs.binary_io.BinaryIOCollection and see if it contains data.

ZackHodari · 2019-04-15T09:23:43Z

Unfortunately if something goes wrong earlier and no frames are produced, generation will "fail" silently and will save an empty array to a file.

Merlin's logging will give you information about when it fails, but it is hard to notice. In fact I just looked back at your original comment and noticed the following:

2019-03-18 02:22:23,190 INFO           labels  : loaded /data/user_data/external_projects/text-to-speech/merlin/egs/slt_arctic/s1/experiments/slt_arctic_full/test_synthesis/gen-lab/test_me.lab,   0 labels

It has 0 labels this could be the root cause of the failure to synthesise anything.

mirfan899 · 2019-06-01T10:16:03Z

I'm getting the same error. I tried many solutions but the problem still exists.

mirfan899 · 2019-06-03T12:18:45Z

I found the solution related to my problem was, festival utts.data format was wrong. Adding proper format solved the issue.

ratz-creator · 2021-09-15T04:02:39Z

can you please specify the format

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

IndexError: index 0 is out of bounds for axis 0 with size 0 #442

IndexError: index 0 is out of bounds for axis 0 with size 0 #442

mrgloom commented Mar 17, 2019 •

edited

Loading

ZackHodari commented Apr 6, 2019

mrgloom commented Apr 8, 2019

mrgloom commented Apr 8, 2019 •

edited

Loading

ZackHodari commented Apr 10, 2019

mrgloom commented Apr 15, 2019 •

edited

Loading

ZackHodari commented Apr 15, 2019 •

edited

Loading

ZackHodari commented Apr 15, 2019

mirfan899 commented Jun 1, 2019

mirfan899 commented Jun 3, 2019

ratz-creator commented Sep 15, 2021

IndexError: index 0 is out of bounds for axis 0 with size 0 #442

IndexError: index 0 is out of bounds for axis 0 with size 0 #442

Comments

mrgloom commented Mar 17, 2019 • edited Loading

ZackHodari commented Apr 6, 2019

mrgloom commented Apr 8, 2019

mrgloom commented Apr 8, 2019 • edited Loading

ZackHodari commented Apr 10, 2019

mrgloom commented Apr 15, 2019 • edited Loading

ZackHodari commented Apr 15, 2019 • edited Loading

ZackHodari commented Apr 15, 2019

mirfan899 commented Jun 1, 2019

mirfan899 commented Jun 3, 2019

ratz-creator commented Sep 15, 2021

mrgloom commented Mar 17, 2019 •

edited

Loading

mrgloom commented Apr 8, 2019 •

edited

Loading

mrgloom commented Apr 15, 2019 •

edited

Loading

ZackHodari commented Apr 15, 2019 •

edited

Loading