What's the need for a different positional encoding?

[shamoons]

transformer/tst/utils.py

Line 32 in 1ac4b34

def generate_regular_PE(length: int, d_model: int, period: Optional[int] = 24) -> torch.Tensor:

Looks like this one is just sin?

[maxjcohen]

Let's answer the question here, once and for all !
The Transformer computes every time step in parallel, it has no notion of time as a sequence. Positional encodings are added in the original paper as a clever way to give the Transformer a notion of what word comes before the others, which is quite important in NLP tasks.

In my case, i'm less interested in knowing which day comes before the next, as I am to tell the Transformer about day/night cycles. This is what the "regular" PE is about, it's just a sin function, acting like a positional encoding, with a period of 24 (cause 24h is a day)

[shamoons]

When using the Transformer model, can we somehow pass a new sequence length to the regular PE instead of 24?

[maxjcohen]

Sure, you can change this value when generating the regular PE, please see the docs here. Feel free to write a modified version of the PE if they still don't match your requirements, examples here.

[shamoons]

My use case is time series prediction for a spectrogram, so 24 isn’t enough. If you’re up for it, I can make a PR that takes a pe_period parameter in the Transformer

[maxjcohen]

In practice, adding PE does not always improve performances. If you still want to use regular PE with a different period, do write a PR, I'll be sure to check it out.

[shamoons]

Will do. But I’m curious why a None PE would work at all? Wouldn’t all the inputs be strictly parallel with no concept of time?

[maxjcohen]

They would, and I couldn't tell you exactly why it still works. Instead, I comfort myself in thinking that the networks simply "pays attention" to related time steps, which happens to be the one you expect. In this example on a month's data, you can see that the Transformer shows day/night cycles, as well as week/weekend, even though there are no PE added.

[shamoons]

#35 - PR up. Please let me know if I have to make any edits

[maxjcohen]

PR was merged, closing.

[outdoteth]

@maxjcohen Have you done a benchmark showing the improved performance of "regular" vs "original" method of positional encoding?

[maxjcohen]

Hi, I haven't done such a benchmark, but I did compare "regular" PE to no PE at all, without much difference, see for instance here and here.

[outdoteth]

Have you got a hypothesis on how it was possible for the network to still learn effectively without the PE? (I know you said you don't know in the comment above^^^ but that was half a year ago now).

[maxjcohen]

I assumed the cyclic nature of the data already strongly encourages the network to learn effectively. For instance, you can see in the example with no PE that the attention map looks very regular (day/night and week/weekend cycles appear clearly).

[shamoons]

I’d love to see how this library works for a variety of datasets. Maybe there’s a paper in it. Any collaborators?

We can evaluate it for various time series data (weather, stocks, audio, etc) to establish “best practices for transformers”

[maxjcohen]

Hi @shamoons, you can see here #48 for a paper comparing the time series Transformer to other recurrent models.