SubNorm.mdx

import { Callout } from "nextra-theme-docs" import { Tab, Tabs } from "nextra-theme-docs"

SubNorm (Attention)

Apply layer normalization before projection.

PreNorm(
    Attention(
        dim=768,
        plugins=[
            SubNorm(dim=768)
        ],
    ),
    dim=768,
)

This plugin implements Sub-LN for Foundation Transformers. Note that Sub-LN presumes Pre-LN rather than Post-LN