Repository navigation
v0.3.0
Oumi Weights Lab 0.3 adds tools for looking inside a model: attention heads as weight slices, attention capture, input-side hooks, direction removal, activation gradients, and comparisons across sessions. Version 0.3 is experimental, and snippets run as trusted Python with your filesystem and network access.
Tested on CPU with tiny GPT-2, grouped-query Llama, OPT, and Qwen3.5 models, on the oldest and newest supported versions of PyTorch, Transformers, and PEFT (Qwen3.5 on the newest only, since older Transformers lacks it).
Highlights
Heads and attention
heads()lists every attention head as slices of the model's own weights, solab.zeroandlab.editablate a head reversibly. Grouped-query heads share their keys and values.head_weights()returns one head's query, key, value, output, and gate matrices in one orientation. It supports Llama, Mistral, Qwen2 and 3, Gemma, OLMo, OPT, GPT-2, and Qwen3.5's gated queries.record(..., at="attention")captures attention weights from an eager attention block, with positions selecting queries.
Activations
recordandpatchtakeat="input"to act on what a module is called with, such as one head's slice of the output projection's input.- Positions also take
"all"and slices. Batched scoring refuses only positions that left padding would shift. gradient(..., modules=)returns gradients of the loss with respect to module activations, on frozen models too.
Edits and comparisons
project_outremoves a direction from a weight, bias, or embedding as one undoable edit, in the right orientation for Linear, GPT-2's Conv1D, and embeddings.compare_runs,diff, andcompare_suitestakecross_session=Trueto compare runs from different sessions, for example a model against its export. The protocol must still match, and the report lists how the two setups differ.
New recipes cover steering vectors, the logit lens, direction edits, attention heads and induction-head search, and attribution patching.
Install this version
Claude Code:
claude plugin marketplace add oumi-ai/weights-lab
claude plugin install oumi-weights-lab@oumi-weights-labCLI, pinned to this release, with LoRA adapters, Hugging Face datasets, and multi-GPU loading:
uv tool install 'oumi-weights-lab[adapters,datasets,multi-gpu] @ git+https://github.com/oumi-ai/weights-lab@v0.3.0'See the README for Codex and other agents.
What's Changed
- Compare runs from different sessions when the caller opts in by @jgreer013 in #41
- Add project_out to remove a direction from a weight, bias, or embedding by @jgreer013 in #42
- Record and patch module inputs, all or sliced positions, and attention weights by @jgreer013 in #43
- Add heads() and head_weights() to list attention heads as weight slices by @jgreer013 in #44
- Take gradients with respect to module activations by @jgreer013 in #45
- Set the version to 0.3.0 by @jgreer013 in #50
Full Changelog: v0.2.1...v0.3.0