pytorch-seq2seq slower than OpenNMT-py

[kylegao91]

Benchmarked the two implementations using WMT's newstest2013 from German to English. See training logs in the gist. Despite accuracy differences, pytorch-seq2seq is 10 times slower than OpenNMT.py.

[PetrochukM]

@kylegao91 Changed pytorch-seq2seq in a private implementation. We were able to match the speeds of OpenNMT .

Things that made a big difference:

• Removing fixed length batching and introducing variable sized batching had a 3 - 4x speed up. Pooling together similar sized examples reduced padding. We implemented this with torchtext.
• Faster loss function similar to OpenNMT memory efficient loss. Instead of looping row by row evaluating the loss batch times. We transformed the target and output from 2D and 3D to 1D and 2D. Evaluated the loss once for the entire batch.

        # (seq len, batch size, dictionary size) -> (batch size * seq len, dictionary size)
        outputs = outputs.view(-1, outputs.size(2))
        # (seq len, batch size) -> (batch size * seq len)
        targets = targets.view(-1)
        self.criterion(outputs, targets)

• Removed the DecoderRNN loop for updating length. We were able to use tensor operations and not include a python loop.

            eos_batches = symbols.view(-1).data.eq(self.eos_idx).nonzero()
            if eos_batches.dim() > 0:
                # (n, 1) => (n)
                eos_batches = eos_batches.view(-1)
                lengths[eos_batches] = len(sequence_symbols)

[kylegao91]

@Deepblue129 Thanks a lot! I will try the ideas.

[kylegao91]

@Deepblue129 Regarding the 3rd point, your code ignored the condition di < lengths[b_idx] so that the lengths in lengths might be longer than they should be. Look at #32 for modified version.

[cclauss]

If speed is essential, why not step up to Python 3.6 or to pypy? Both are faster that Python 2.7.

[kylegao91]

@cclauss It's like, doing that would speed up an O(N^2) sorting algorithm, but it's still O(N^2) instead of O(Nlog(N))...
We definitely need to support python3 though, would appreciate any contribution to that end.

[PetrochukM]

@kylegao91
OpenNMT/OpenNMT-py#139

[kylegao91]

Now with #32, #55, and #73, this issue is done.