Skip to content

NanoLens v1.0 — Initial Release

Latest
Latest

Choose a tag to compare

View all tags
@AdityaSinghDevs AdityaSinghDevs released this 28 May 20:22
v1.0
da00632

NanoLens v1.0

First release of NanoLens — a mechanistic interpretability toolkit built
from scratch on a 25M parameter character-level transformer trained on
Dostoevsky.

What's included

  • Pretrained checkpoint: 8 layers, 8 heads, 512 embedding dim
  • Trained for 5000 steps on 4.4M characters (Crime and Punishment,
    Brothers Karamazov, The Idiot)
  • Final val loss: 1.1144 | Train/val gap: 0.053

Checkpoint

  • nanolens_v1.pt — ~111MB, drop into checkpoints/ directory

Research findings in this release

  • 6 documented attention head types across 64 heads
  • Previous token heads, BOS sink heads, word boundary heads,
    abstract semantic routing heads confirmed
  • Early layers (0-2) encode local sequential structure
  • Late layers (5-7) perform abstract semantic routing

Usage

See README for full quickstart. Requires Python 3.10+, PyTorch 2.0+.

python cli.py --checkpoint checkpoints/nanolens_v1.pt
python cli.py --checkpoint checkpoints/nanolens_v1.pt --inspect "Raskolnikov hesitated"

Label

None — this is a stable research release, not a pre-release.

Assets 3
Loading