AttributeError: 'DistributedDataParallel' object has no attribute 'version' at the time of model checkpointing

[rushiagr]

Full stacktrace:

... <snipped> ...
Epoch: [1][97/804]      Time 1.485 (1.640)      Data 0.006 (0.069)      Loss 339.3378 (444.7247)
Epoch: [1][98/804]      Time 1.753 (1.641)      Data 0.012 (0.069)      Loss 372.9253 (443.9921)
Epoch: [1][99/804]      Time 1.500 (1.640)      Data 0.017 (0.068)      Loss 276.2173 (442.2974)
Epoch: [1][100/804]     Time 1.461 (1.638)      Data 0.006 (0.068)      Loss 314.9874 (441.0243)
Saving checkpoint model to /datasets/deepspeech/librispeech/deepspeech_checkpoint_epoch_1_iter_100.pth
Traceback (most recent call last):
  File "train.py", line 284, in <module>
    wer_results=wer_results, cer_results=cer_results, avg_loss=avg_loss),
  File "/workspace/src/deepspeech/deepspeech.pytorch/model.py", line 251, in serialize
    'version': model.version,
  File "/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py", line 585, in __getattr__
    type(self).__name__, name))
AttributeError: 'DistributedDataParallel' object has no attribute 'version'
Traceback (most recent call last):
  File "/opt/conda/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/opt/conda/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/workspace/src/deepspeech/deepspeech.pytorch/multiproc.py", line 46, in <module>
    cmd=p.args)
subprocess.CalledProcessError: Command '['/opt/conda/bin/python', 'train.py', '--rnn-type', 'lstm', '--hidden-size', '1024', '--hidden-layers', '5', '--train-manifest', '/datasets/deepspeech/librispeech/libri_train_manifest.csv', '--val-manifest', '/datasets/deepspeech/librispeech/libri_val_manifest.csv', '--epochs', '60', '--num-workers', '16', '--cuda', '--learning-anneal', '1.01', '--batch-size', '64', '--no-sortaGrad', '--visdom', '--opt-level', 'O1', '--loss-scale', '1', '--id', 'libri', '--checkpoint', '--save-folder', '/datasets/deepspeech/librispeech', '--model-path', '/datasets/deepspeech/librispeech/deepspeech_final.pth', '--checkpoint-per-batch', '100', '--opt-level', 'O1', '--loss-scale', '1.0', '--world-size', '4', '--rank', '0', '--gpu-rank', '0']' returned non-zero exit status 1.

[rajeevbaalwan]

You need to make changes package json in model.py file
change it from :
package = {
'version': model.version,
'hidden_size': model.hidden_size,
'hidden_layers': model.hidden_layers,
'rnn_type': supported_rnns_inv.get(model.rnn_type, model.rnn_type.name.lower()),
'audio_conf': model.audio_conf,
'labels': model.labels,
'state_dict': model.state_dict(),
'bidirectional': model.bidirectional
}

to

package = {
'version': model.module.version,
'hidden_size': model.module.hidden_size,
'hidden_layers': model.module.hidden_layers,
'rnn_type': supported_rnns_inv.get(model.module.rnn_type, model.module.rnn_type.name.lower()),
'audio_conf': model.module.audio_conf,
'labels': model.module.labels,
'state_dict': model.module.state_dict(),
'bidirectional': model.module.bidirectional
}

[farisalasmary]

That didn't help! I've got this error
Traceback (most recent call last): File "train.py", line 291, in <module> wer_results=wer_results, cer_results=cer_results, avg_loss=avg_loss), File "/workspace/deepspeech.pytorch/model.py", line 263, in serialize 'version': model.module.version, File "/opt/conda/lib/python3.7/site-packages/torch/nn/modules/module.py", line 576, in __getattr__ type(self).__name__, name)) AttributeError: 'DeepSpeech' object has no attribute 'module'

[farisalasmary]

Here is my solution:

open model.py and do the following:

import this package at the beginning of the file

from apex.parallel import DistributedDataParallel

then go to the method serialize and replace the following block of code:

package = {
            'version': model.version,
            'hidden_size': model.hidden_size,
            'hidden_layers': model.hidden_layers,
            'rnn_type': supported_rnns_inv.get(model.rnn_type, model.rnn_type.__name__.lower()),
            'audio_conf': model.audio_conf,
            'labels': model.labels,
            'state_dict': model.state_dict(),
            'bidirectional': model.bidirectional,
        }

with this:

if isinstance(model, DistributedDataParallel):
    print('Save Model: DistributedDataParallel...')
    package = {
        'version': model.module.version,
        'hidden_size': model.module.hidden_size,
        'hidden_layers': model.module.hidden_layers,
        'rnn_type': supported_rnns_inv.get(model.module.rnn_type, model.module.rnn_type.__name__.lower()),
        'audio_conf': model.module.audio_conf,
        'labels': model.module.labels,
        'state_dict': model.module.state_dict(),
        'bidirectional': model.module.bidirectional
    }
else:
    package = {
        'version': model.version,
        'hidden_size': model.hidden_size,
        'hidden_layers': model.hidden_layers,
        'rnn_type': supported_rnns_inv.get(model.rnn_type, model.rnn_type.__name__.lower()),
        'audio_conf': model.audio_conf,
        'labels': model.labels,
        'state_dict': model.state_dict(),
        'bidirectional': model.bidirectional,
    }

This would work for training on a single GPU, python train.py, as well as multiple GPUs, python -m multiproc train.py.

[farisalasmary]

Here is a much simpler solution:
Just put this if statement at the beginning of the "serialize" method:

if isinstance(model, DistributedDataParallel):
    model = model.module

So, it will be something like:

    @staticmethod
    def serialize(model, optimizer=None, amp=None, epoch=None, iteration=None, loss_results=None,
                  cer_results=None, wer_results=None, avg_loss=None, meta=None):
        if isinstance(model, DistributedDataParallel):
            model = model.module
        package = {
            'version': model.version,
            'hidden_size': model.hidden_size,
            'hidden_layers': model.hidden_layers,
            'rnn_type': supported_rnns_inv.get(model.rnn_type, model.rnn_type.__name__.lower()),
            'audio_conf': model.audio_conf,
            'labels': model.labels,
            'state_dict': model.state_dict(),
            'bidirectional': model.bidirectional,
        }

DO NOT forget to import this package at the beginning of the file:

from apex.parallel import DistributedDataParallel

[SeanNaren]

thanks @farisalasmary i'm going to have a look at this, will implement one of your proposed solutions in the master branch!

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.