TriGLU v0.2.0 — a controlled study of replacing attention with token-local gating.
This release extends the v0.1.0 RoPE-TriGLU placement suite into a controlled
study of how much attention a small decoder actually needs and where, with
parameter-matched token-local controls. The recommended TriGLU form is now the
no-RoPE variant; the RoPE variant is retained as the ablation origin and a
control.
Headline findings (89M params, 20 layers, 1B FineWeb-Edu tokens at 4K; means
pool three seeds unless marked exploratory):
- Attention is substantially replaceable at quality parity — any parameter-
matched token-local mixer at 25%, and no-RoPE TriGLU at 55%, matches or beats
the all-attention baseline. Placement and amount, not the mixer equation, are
the primary levers; the third multiplicative factor earns its keep only at
aggressive (55%) replacement, where the two-factor control regresses to parity. - The 55% hybrid trains ~19% faster with 55% less KV-cache at 4K, widening with
context (controlled benchmarks). - The triple product is also the strongest FFN tested (all attention retained;
best quality in the study), at negligible controlled-throughput cost — a
separate direction from attention replacement. The two roles do not stack. - The hybrid's quality advantage over a parameter-matched 9-layer collapse is a
genuine structural effect; the collapse is faster, so it is a distinct point on
the speed/quality frontier rather than a refuted alternative.
The exact TriGLU formulation is a GQU-family gate; the contribution is the
controlled study, not the equation. See docs/related-work.md, docs/results.md,
and docs/design.md.
Evidence archive. TriGLU-evidence-v0.2.0.tar.gz (attached) contains the
complete raw records behind every reported number per docs/results-schema.md:
each run's resolved config, metric stream, environment metadata, and data hashes;
raw evaluation/benchmark/rank-analysis JSON; and the generated report tables and
figures for both suites. Checkpoints and token corpora are excluded by design —
the included manifests pin the exact upstream dataset revision and hashes needed
to regenerate them.