Bytenet - implementation difference from paper

[randomrandom]

I was looking at the bytenet code and noticed that the implementation is different from the Neural Machine Translation in Linear Time paper.

From my understanding of the paper - each atrous conv layer is wrapped in a ResnetBlock, e.g.:

single repeat = [Resnet(conv layer, dilated=1), Resnet(conv layer, dilated=2), Resnet(conv layer, dilated=4)]

while in the current implementation it seems more like a ResnetBlock is composed of many dilated conv layers, e.g.:

single repeat = Resnet[ (conv layer, dilated=1), (conv layer, dilated=4), (conv layer, dilated=8)]

Is the difference from the paper intended?

[lukaszkaiser]

This is a very good question! I thought it's like in our implementation (some of the authors of the original papers had a look), but I'll ask the authors again next time I see them to confirm.

[randomrandom]

The paper is a bit thin on details, so I wanted to make sure that my implementation is correct and checked your code, looks differently from the described here:

Each layer is wrapped in a residual block that contains additional convolutional layers with filters of size 1 × 1 (He et al., 2016).
(from the paper)

From my experiments the current T2T architecture also looks to be working fine, though the gradient flow might not be so optimal if someone stacks lots of dilated convolutions, due to the lack of shortcuts.

Thanks!