TriGLU v0.2.0

TriGLU v0.2.0 — a controlled study of replacing attention with token-local gating.

This release extends the v0.1.0 RoPE-TriGLU placement suite into a controlled
study of how much attention a small decoder actually needs and where, with
parameter-matched token-local controls. The recommended TriGLU form is now the
no-RoPE variant; the RoPE variant is retained as the ablation origin and a
control.

Headline findings (89M params, 20 layers, 1B FineWeb-Edu tokens at 4K; means
pool three seeds unless marked exploratory):

• Attention is substantially replaceable at quality parity — any parameter-
  matched token-local mixer at 25%, and no-RoPE TriGLU at 55%, matches or beats
  the all-attention baseline. Placement and amount, not the mixer equation, are
  the primary levers; the third multiplicative factor earns its keep only at
  aggressive (55%) replacement, where the two-factor control regresses to parity.
• The 55% hybrid trains ~19% faster with 55% less KV-cache at 4K, widening with
  context (controlled benchmarks).
• The triple product is also the strongest FFN tested (all attention retained;
  best quality in the study), at negligible controlled-throughput cost — a
  separate direction from attention replacement. The two roles do not stack.
• The hybrid's quality advantage over a parameter-matched 9-layer collapse is a
  genuine structural effect; the collapse is faster, so it is a distinct point on
  the speed/quality frontier rather than a refuted alternative.

The exact TriGLU formulation is a GQU-family gate; the contribution is the
controlled study, not the equation. See docs/related-work.md, docs/results.md,
and docs/design.md.

Evidence archive. TriGLU-evidence-v0.2.0.tar.gz (attached) contains the
complete raw records behind every reported number per docs/results-schema.md:
each run's resolved config, metric stream, environment metadata, and data hashes;
raw evaluation/benchmark/rank-analysis JSON; and the generated report tables and
figures for both suites. Checkpoints and token corpora are excluded by design —
the included manifests pin the exact upstream dataset revision and hashes needed
to regenerate them.

TriGLU v0.1.0

Initial public release.

TriGLU — the Triple-Product Gated Linear Unit — is a token-local channel mixer
for selectively replacing causal-attention sublayers in decoder-only
Transformers, together with the controlled single-GPU harness (training,
deterministic data preparation, evaluation, benchmarking, mechanistic
diagnostics, and reporting) used to measure the trade-offs.

Headline evidence (89M-parameter, 20-layer suite, 1B FineWeb-Edu tokens,
two confirmatory seeds): two 15-attention/5-TriGLU hybrids match the
all-attention baseline's validation loss within ±0.001 while training ~7%
faster, with 5–9% faster prefill, 8–10% faster cached decode, and exactly 25%
less allocated KV-cache at 4K context. Gains grow at longer contexts
(efficiency extrapolation only above the 4K training length). See
docs/results.md for the complete results and limitations.

Evidence archive. TriGLU-evidence-v0.1.0.tar.gz (attached) contains the
complete raw records behind every reported number, per
docs/results-schema.md: each run's resolved config, metric stream,
environment metadata, and data hashes; raw evaluation/benchmark/rank-analysis
JSON; and the generated report tables and figures. Checkpoints and token
corpora are excluded by design — manifest.json pins the exact upstream
dataset revision and hashes needed to regenerate them.