Implement: Stochastic positional embeddings improve masked image modeling

[IgorSusmelj]

From this paper: https://arxiv.org/pdf/2308.00566

AFAIK we could use that in combination with many other building blocks. Could be interesting to see. E.g. with LeJEPA :)

[gabrielfruet]

Hi @IgorSusmelj , I'll take a look on that paper!

[JiwaniZakir]

The paper (SoPE, Bao et al. 2023) proposes replacing fixed sinusoidal or learned positional embeddings with stochastic ones sampled at training time — effectively adding noise to token positions — which forces the model to rely less on absolute position and learn more semantic patch representations during MIM pretraining. In lightly, the natural integration point would be the positional embedding modules inside lightly/models/modules/ (e.g., wherever VisionTransformer or patch embedding layers inject position encodings), since the stochastic sampling only needs to apply during the forward pass at training time, not at inference. The LeJEPA connection makes sense given its I-JEPA heritage — the context/target block sampling already introduces spatial stochasticity, and SoPE could compound that by also randomizing the positional signal fed to the encoder. Worth noting that SoPE is reported to be largely orthogonal to masking strategy, so it should compose cleanly with existing MIM heads in the codebase without requiring architectural changes beyond the embedding layer.