RotaryEmbedding.mdx

import { Callout } from "nextra-theme-docs" import { Tab, Tabs } from "nextra-theme-docs"

RotaryEmbedding (Attention)

Rotary Position Embedding for RoFormer.

Attention(
    dim=768,
    num_heads=12,
    head_dim= 64,
    plugins=[
        RotaryEmbedding(
            head_dim=64,
        )
    ],
)

Parameters

• head_dim: The dimension size for each attention head.
• seq_len: The length of given sequence.