A question about input dimensions

[DaiZhenrong]

Hi Ian Shih,
Thank for your amazing job.
In my memories，the input dimesions of Transformer is [batch_size, seq_len,d_model]. So I wonder that why your input dimesions of model is seq_len, batch_size, d_model?
I will appreciate that if I can your reply.

Thanks.

[atosystem]

@DaiZhenrong

For pytorch built-in transformer module, you can input with the dimension order of (batch_size, seq_len, d_model) or seq_len, batch_size, d_model. It depends on batch_first. For the former batch_first=True, and for the latter batch_first=False. I used the default setting in Theme Transformer, which is False.

Here is the doc for torch.nn.Tranformer

[DaiZhenrong]

oh, I got it. Thanks a lot for your reply！

[DaiZhenrong]

And I also wander that why seq_len you chose is 512 instead of 1024 or 256. I would appreciate it if you could reply.

[atosystem]

@DaiZhenrong
Yeah, I start my experiments using 512 (for lower memory). But in the ablation studies, I have run Theme Transformer on sequence length of 1024, which has a better performance in capturing the theme gaps. (For more details, please refer to the paper)

So, in general, the model performs better with a longer sequence length as long as your GPU memory permits.
Also, one may also replace the architecture of vanilla transformer with transformerXL or Longformer, which feature in modeling long sequences.

[DaiZhenrong]

Ok, I see. Thank you for your prompt reply, help and work!