You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The current implementation of relative positional encoding (#139) only supports discrete positions (technically, you can give already it continuous positions, which then get truncated by tensor.long). The most basic version of relative positional encoding for vector spaces would apply a function that discretizes all positions, e.g. by dividing by a specified width and then rounding to the nearest integer. There are various extensions to this that might improve performance:
Discretize positions in a way that has a higher resolution for nearby positions and lower resolution for distant positions (only skimmed this very briefly, but https://arxiv.org/pdf/2107.14222.pdf seems to be doing something similar). The idea is that you might care a lot more about small changes in position for nearby entities, but only care about the rough location for more distant entities. Lots of different possibilities for the discretization function.
When an entity position doesn't fall exactly onto one of the discrete locations, interpolate between nearby positions.
This could be prototyped with the Minefield task (#50), on which relative positional encoding should achieve the same performance as translation.
The text was updated successfully, but these errors were encountered:
The current implementation of relative positional encoding (#139) only supports discrete positions (technically, you can give already it continuous positions, which then get truncated by
tensor.long
). The most basic version of relative positional encoding for vector spaces would apply a function that discretizes all positions, e.g. by dividing by a specified width and then rounding to the nearest integer. There are various extensions to this that might improve performance:This could be prototyped with the Minefield task (#50), on which relative positional encoding should achieve the same performance as translation.
The text was updated successfully, but these errors were encountered: