AttributeError: 'dict' object has no attribute 'start_shuffle'

[yash-s20]

this is the complete traceback -

Traceback (most recent call last): File ".../espnet/bin/asr_train.py", line 370, in <module> main(sys.argv[1:]) File ".../espnet/bin/asr_train.py", line 357, in main train(args) File ".../espnet/asr/pytorch_backend/asr.py", line 630, in train trainer.run() File ".../espnet/tools/venv/lib/python3.7/site-packages/chainer/training/trainer.py", line 349, in run six.reraise(*exc_info) File ".../espnet/tools/venv/lib/python3.7/site-packages/six.py", line 703, in reraise raise value File ".../espnet/tools/venv/lib/python3.7/site-packages/chainer/training/trainer.py", line 319, in run entry.extension(self) File ".../espnet/espnet/utils/training/iterators.py", line 28, in __call__ iterator.start_shuffle() AttributeError: 'dict' object has no attribute 'start_shuffle'
this is my train config -

# network architecture
# encoder related
etype: blstmp     # encoder architecture type
elayers: 4
eunits: 512
eprojs: 512
subsample: "1_2_2_1_1" # skip every n frame from input to nth layers
dropout-rate: 0.3
# decoder related
dlayers: 1
dunits: 512
dropout-rate-decoder: 0.1
# attention related
atype: location
adim: 512
aconv-chans: 10
aconv-filts: 100

# hybrid CTC/attention
mtlalpha: 0.7

# minibatch related
batch-size: 4
maxlen-in: 800  # if input length  > maxlen_in, batchsize is automatically reduced
maxlen-out: 150 # if output length > maxlen_out, batchsize is automatically reduced

# optimization related
opt: adadelta
epochs: 20
patience: 10
sortagrad: 1
# scheduled sampling option
sampling-probability: 0.0

report-cer: true
report-wer: true

Here's what I see in espnet/asr/pytorch_backend/asr_mix.py -
a list containing just train_iter is passed to ShufflingEnabler, which is a dictionary, and in the call method a start_shuffle attribute is called on the this, which is causing the error. Why is this so?
Also as you can see sortagrad is set to 1, and this error sprouts at the beginning of the second epoch.
Edit: file paths

[sw005320]

Could you tell me a little bit more detail?
What kind of recipes you are using and where did you modify based on it?
Only sortagrad?

[yash-s20]

I've used a mix of an4 and minian4 recipes, and the error does not occur when I use sortagrad: 0 or sortagrad: -1. I've not modified much except for number of units in the model from an4.

[sw005320]

Thanks.
You mean that it's caused by espnet/asr/pytorch_backend/asr_mix.py?
@simpleoier, could you take a look at it?

[simpleoier]

Hi @yash-s20 , I'm a bit confused. From the traceback and the config, I guess you are training the normal single-speaker ASR system. Is it true? If so, you should not use the espnet/asr/pytorch_backend/asr_mix.py.

Based on the traceback, I also suggest you modify this line as:

trainer.extend(ShufflingEnabler([train_iter['main']]),

I'll also examine the error and post more information if I have any updates.

[yash-s20]

Hmm I see I was looking at the wrong file, but that part of code is similar in both, I'll make the changes and re-run.

[ahmedalbahnasawy]

@sw005320 I faced the same issue before 2 weeks but when I changed sortgrad to 0 it works fine. I was using asr_train.py (Transducer model).

[simpleoier]

I think the reason is because of this ShufflingEnabler, in which the train_iter was defined as a dictionary. When the ShufflingEnabler is activated, it will call the train_iter.start_shuffle() from here. In the previous versions, the train_iter wasn't defined as a dictionary. So there was no such problem then.

I am not sure for what reason the train_iter was defined as a dictionary. But the element train_iter['main'] is of type ChainerDataloader which has the method start_shuffle() I think.

So the method I mentioned above is not the final solution. But we'll see what we can do based on the effectiveness of the method.

[JoeyHeisenberg]

I got the same problem when I was training the FastSpeech on 3 GPU, which I set the sortagrad to 5

same happined when training Tacotron2 on single GPU

[yash-s20]

@simpleoier what you mentioned works. Thanks! You could make the changes if you haven't already.
@JoeyHeisenberg try it.

[sw005320]

It was fixed by @1741.
Could you test it?