about position embedding

[leon532]

How does token merging process position embedding? I am wondering whether token merging can adapt relative position embedding like RoPE?
Look forward to ur reply

[dbolya]

ToMe works on any absolute position embeddings, but for relative that depends on the method itself. See facebookresearch/ToMe#29 for additional details, but likely applying ToMe to a specific relative position embedding is a research question of its own!