DataParallel is not compatible with pack_padded_sequence

[ZiJianZhao]

If the model has pack_padded_sequence, then with DataParallel module it will output error "ValueError: lengths array has incorrect size"

[ngimel]

Can you please provide minimum reproducer?

[soumith]

@ZiJianZhao still waiting on a response.

[jgc128]

Hi,

I have the same error (there's also this issue #1591). The code below works on one GPU (CUDA_VISIBLE_DEVICES=0 python pack_padded_sequence_data_parallel.py), but fails with "ValueError: lengths array has incorrect size" on two GPUs (CUDA_VISIBLE_DEVICES=0,1 python pack_padded_sequence_data_parallel.py):

import numpy as np
import torch
from torch.autograd import Variable


class RNNDataParallel(torch.nn.Module):
    def __init__(self):
        super(RNNDataParallel, self).__init__()

    def forward(self, inputs, lengths):
        packed = torch.nn.utils.rnn.pack_padded_sequence(inputs, lengths, batch_first=True)

        return packed


model = RNNDataParallel()
model = torch.nn.DataParallel(model)
model = model.cuda()

inputs = Variable(torch.from_numpy(np.array([
    [1, 2, 3],
    [4, 5, 0],
])))
lengths = [3, 2]

packed = model(inputs, lengths)

print(packed)

My PyTorch version is 0.2.0+e02f7bf

[ahmedmagdiosman]

I encountered the same issue as @jgc128 .

EDIT: I think the issue is that DataParallel does not do slice CPU data like the lengths list.

EDIT2: I "fixed" this by transforming the lengths to a Variable(LongTensor.cuda()) before starting the forward pass and reverting it to a List before calling pack_padded_sequence.

[jekbradbury]

pack_padded_sequence now supports the lengths being provided as a Tensor or Variable, I think?

[ahmedmagdiosman]

@jekbradbury Not on my build (conda pytorch 0.2.0). Even so, the issue lies within DataParallel for not slicing lengths. I fixed the issue with a simple hack (check my 2nd edit on the previous comment).