v0.0.3 trajectory — in-progress release (cheap-path 500-step v0.1.x AV)

In-progress trajectory release. Adapter weights for each saved checkpoint of the cheap-path 500-step v0.1.x AV SFT run, attached as release assets. New checkpoints land every ~2.5 hours of GPU time as training proceeds. Final v0.0.3 release tag with the complete trajectory + H14-style trend evaluation will land once training completes (~2026-05-14).

What's attached

Each adapter_model_step_XXXXXX.safetensors is a LoRA adapter from that checkpoint. Pair with the v0.0.1 AR (Solshine/gemma-4-e2b-nla-L23-ar-v0_0_1) for round-trip eval at that training scale.

For per-checkpoint metadata (adapter_config.json, nla_meta.yaml, README), see trajectory/ in the repo source.

Currently attached

• adapter_model_step_000050.safetensors (387 MB) — loss 2.38 at step 50
• adapter_model_step_000100.safetensors (387 MB) — loss 2.24 at step 100

(More will be added as checkpoints save.)

Mirror

Same checkpoints are also on HuggingFace at Solshine/gemma-4-e2b-nla-L23-av-v0_1_x-trajectory. Use whichever mirror is faster for you.

Hardware-constraint disclosure

Same disclaimers as v0.0.2 apply: this is a small-model variant pushed to the limit of a 4 GB consumer GPU. See main README for the full discussion of AR content-blindness and the empirical ceiling at v0.0.x compute scale.

v0.0.2 — Hardware-constraint-bound release + AR-content-blind investigation

Same trained adapters as v0.0.1, with the AR-content-blind / template-collapse investigation findings now prominent in the model cards.

What changed since v0.0.1

A 5-experiment team investigation against the v0.0.1 NLA pair surfaced findings that revise the original template-collapse diagnosis. The most important: the AR is content-blind.

Mean cos by AR input condition on n=10 rows from rl.parquet (same target activations as the published n=42 eval):

AR input	Mean cos	Above 0.30 floor
Real AV-generated explanation	0.4292	10/10
Random Wikipedia sentences	0.4045	10/10
Random nonsense tokens	0.4135	10/10
Empty string	0.4051	10/10

The "explanation" contributes a mean +0.024 cos over feeding the AR nothing at all. ~95% of the published cos number is the AR's content-independent projection toward OpenWebText activation space; ~5% is explanation-dependent.

Also surfaced: a storage-truncation bug at eval_round_trip.py:195 ("explanation": explanation[:300]) made the original "298-character outputs / 4 templates" finding partly an artifact of display truncation, not model behavior. Fixed.

Full investigation report: ACCURACY_COLLAPSE_LIMITATIONS_ROOT_CAUSES_HYPOTHESIS.md.

Hardware-constraint disclosure (new in v0.0.2)

This release was pushed to the limit of what a single 4 GB consumer GPU (NVIDIA GTX 1650 Ti Max-Q) can practically train. The under-trained AV (~12% effective data exposure at 55 SFT steps × grad-accum 4) and content-blind AR are the predicted small-model failure modes at this compute budget, not defects fixable at the same hardware tier.

v0.0.2 is the best honest artifact achievable on the chosen hardware. A faithful NLA at the methodology's scale requires either (a) a larger consumer GPU with more headroom, or (b) ~$30-50 of cloud A100 time per the cloud_gpu_setup_notes.md plan in the source repo.

What still works

• ✅ Methodology pipeline (Stage 0 → 1 → 2 → 3 → AV SFT → AR SFT → eval) runs end-to-end on a 4 GB GPU in ~6 hours
• ✅ Multi-labeler infrastructure with per-row labeler_model provenance
• ✅ Persona+audit labeling pipeline (Dr Marisol Chen labeler + Dr Riley Otsuka auditor)
• ✅ Honest-accuracy training-trend verdict tool (slope < −0.002 AND R² ≥ 0.10)
• ✅ Restart-safe chunked parquet output (survives daily Gemini quota walls)
• ✅ All four published HuggingFace datasets are independently useful as Stage-0 inputs

What doesn't work yet

• ❌ Per-row activation interpretation — both halves of the pair are under-trained
• ❌ Round-trip cos as a faithfulness metric — at v0.0.2 scale it measures structural projection, not explanation faithfulness
• ❌ Cross-AV pairing — has not been validated

Next steps (in queue)

• Train matched Option B AR on Claude Haiku persona+audit labels (~3-6 GPU-hours, dataset already prepared) to test whether better AR labels alone break content-blindness
• v0.1.0 full retrain on diversified 9-source corpus at 3,000-5,000 SFT steps (requires cloud GPU)
• Second-model Mistral-7B / OLMo-7B variant on rented A100 (~$30-40 spend; runbook ready in source repo)

v0.0.1 — First independent open-source NLA

First open-source Natural Language Autoencoder (NLA) released independently of Anthropic's NLA team. Gemma-4-E2B AV+AR pair trained on a 4 GB consumer GPU.

Honest performance

• Round-trip cosine similarity: 0.438 ± 0.054 on n=42 held-out OpenWebText activations
• 42 of 42 rows clear the 0.30 noise-floor threshold
• Min cos 0.313, max 0.558
• Well below Anthropic's published 7B variants (cos ~0.7+) — framed as methodology validation at small-model scale, not numbers-parity

What ships

Models (HuggingFace):

• Solshine/gemma-4-e2b-nla-L23-av-v0_0_1 — AV (Actor) adapter
• Solshine/gemma-4-e2b-nla-L23-ar-v0_0_1 — AR (Critic) adapter + linear head

Datasets (HuggingFace):

• Solshine/gemma-4-e2b-nla-ar_sft-v0_0_x-haiku-persona-audit — 696-row AR-SFT corpus
• Solshine/gemma-4-e2b-nla-av_sft-v0_1_x-gemini-persona-audit — 4,734-row diversified AV-SFT corpus

In this repo:

• README.md, MODEL_CARD_AV.md, MODEL_CARD_AR.md
• lesswrong_post.md (the announcement)
• figures/ (6 release figures)
• examples/eval_round_trip.py
• LICENSE (CC BY 4.0)

Source repo (full reproducibility)

SolshineCode/deception-nanochat-sae-research — every script, every prompt, every result.

Methodology citation

Fraser-Taliente, K., et al. (2026). Natural Language Autoencoders. https://transformer-circuits.pub/2026/nla/

The Anthropic reference repo is kitft/natural_language_autoencoders.