TriGLU v0.3.0 — the attention-replacement boundary at 40 layers.
This release scales the v0.2.0 winning form (no-RoPE TriGLU) to a 152M-parameter,
40-layer decoder at two billion FineWeb-Edu tokens to map the replacement
*boundary*: how much attention can be removed, where the retained attention must
sit, and what the removal is worth at inference time. 41 training runs across 25
parameter-identical layer plans (152,281,600 parameters each), with three seeds
on every headline contrast and SEM-gated verdicts; evidence spans validation
loss, LM Evaluation Harness capability (LAMBADA, WikiText), needle-in-a-haystack
retrieval, and measured inference cost.
**Headline findings** (means pool three seeds unless marked exploratory):
- **The boundary:** replacing 57.5% of attention layers (p12\_c4 — a 12-layer
attention prefix, then 1:4 attention:TriGLU) beats the all-attention baseline
on loss (3.1536 vs 3.1638, 1.9 SEM), LAMBADA, and retrieval; 65–70% costs a
small, replicated ~0.010–0.015-nat loss while holding benchmark capability;
80% breaks on every instrument at once.
- **The economics are measured, not modeled:** the 57.5% hybrid carries 58% less
KV-cache (34 vs 80 KB/token, counted from the live cache and matching the
analytic value exactly), 2× measured prefill, and +49% cached decode at the 4K
window. Both savings are linear in retained attention layers and therefore
context-independent — at a 128K context that is 4.25 GB of KV cache versus
10 GB.
- **Attention is front-critical:** the identical 8-layer attention budget placed
at the *end* of the network instead of the front is ~0.09 nats worse — the
worst model in the study. A small attention postfix restores output-local
prediction (LAMBADA) but never general quality, and the final layer is the
worst single position for attention. Early attention is the irreplaceable part.
- **Placement within the tail is forgiving:** at fixed budget, even vs growing
vs shrinking gap spacing produces no separation that survives the significance
gate at either prefix, and a tail-phase rotation is a null. Keep a solid early
attention block, spread the rest, and don't tune the schedule.
- **Replication matters at this effect size:** measured seed SD (~0.005 nats)
exceeds most single-seed gaps in the healthy cluster. Two apparent single-seed
effects — a growing-gap penalty and the baseline's own level — reversed or
shrank under replication and are reported as the nulls they are.
Full findings and limitations: docs/results-40l.md; design and verdict policy:
docs/design.md. The v0.2.0 20-layer study is unchanged (docs/results.md).
**New in the harness:** the generated 40-layer config suite
(scripts/gen\_placement\_configs.py, explicit per-layer plans), unattended
skip-complete launchers with shared-GPU waiting, seed-replication tooling with
SEM-gated analysis (scripts/analyze\_seed\_replication.py), a NIAH retrieval
integration (triglu.niah\_eval, against
[Royer-Research-Labs/Niah](https://github.com/Royer-Research-Labs/Niah)), an LM
Evaluation Harness model adapter (triglu model type), the long-context
economics panel, and reproducible README figures. 142 CPU tests. NIAH results in
this release were produced with Royer-Research-Labs/Niah@de5b6be.
**Evidence archive.** TriGLU-evidence-v0.3.0.tar.gz (attached) contains the
complete raw records behind every reported number per docs/results-schema.md:
each of the 41 runs' resolved config, metric stream, environment metadata, and
data hashes; per-run NIAH results and charts; the LM-eval capability slice; the
controlled benchmark records; the long-context economics table; and the generated
report tables and figures. Checkpoints and token corpora are excluded by design —
the included manifests pin the exact upstream dataset revision and hashes needed
to regenerate them.