Does the part of the performance gain come from residual attention?

[jerrywn121]

I noticed that this implementation of Transformer here used residual attention, which does not appear in some of the other baselines mentioned in the paper. So I wonder if you have performed additional ablation studies to see the effect of residual attention for forecasting?

[jerrywn121]

Also, it would be great to know how much RevIN benefits the patch Transformer. Thank you!

[yuqinie98]

Hi @jerrywn121 , thanks for your questions! We use residual attention because we want to be as close as possible to the original Transformer model in NLP, and only find the key factors that make it work well when applying it to time series datasets (patch; channel-independence). Also, residual attention was used in some previous papers like TST (https://arxiv.org/abs/2010.02803). We haven't studied the effect of it yet.

For RevIN, we do the additional ablation in table 11 in the paper. It improves the model marginally in general. Hope this helps.

[jerrywn121]

Got it. Thank you for your reply!