Skip to content

Releases

v0.5.1

Choose a tag to compare

@jgreer013 jgreer013 released this 08 Nov 01:24
2387139

Oumi Weights Lab 0.5.1 fixes two bugs in 0.5.0 and changes how the skill plans an investigation. Version 0.5 is experimental, and snippets run as trusted Python with your filesystem and network access.

Fixes

  • On Linux, stop no longer fails with "[Errno 3] No such process" when the worker exits while stop checks on it. The worker still stopped in 0.5.0, but stop exited with an error.
  • oumi-weights-lab --help lists every command in its usage line. 0.5.0 left out script.

The skill

  • Agents treat a model as the subject of an experiment. They keep a belief file of established findings, rival hypotheses, assumptions and open questions, and pick each next test by what it would settle and what it costs, instead of following a fixed order of calls.
  • The method map lists each call's cost in forward passes.

Install this version

Claude Code:

claude plugin marketplace add oumi-ai/weights-lab
claude plugin install oumi-weights-lab@oumi-weights-lab

The plugin includes the SAE support. As before, it includes quantized loading except on Intel Macs and macOS before 14.

CLI, pinned to this release, with LoRA adapters, Hugging Face datasets, multi-GPU loading, quantized loading and SAEs:

uv tool install 'oumi-weights-lab[adapters,datasets,multi-gpu,quantized,saes] @ git+https://github.com/oumi-ai/weights-lab@v0.5.1'

See the README for Codex and other agents.

What's Changed

  • Run the model tests once a day instead of on every pull request by @jgreer013 in #98
  • Pick each experiment by what it would settle and what it costs by @jgreer013 in #99
  • List every command in the CLI's usage line by @jgreer013 in #100
  • Treat a worker that exits while stop checks on it as stopped by @jgreer013 in #101
  • Set the version to 0.5.1 by @jgreer013 in #103

Full Changelog: v0.5.0...v0.5.1

v0.5.0

Choose a tag to compare

@jgreer013 jgreer013 released this 07 Nov 06:10
9a114ed

Oumi Weights Lab 0.5 is about understanding a model. Analyses that find and test circuits are now tested lab calls: direct logit attribution, path patching, edge attribution, sparse autoencoders and transcoders, attribution graphs, and contrastive directions. Agents can save the methods they write, and the skill gains a method map and a research protocol. Version 0.5 is experimental, and snippets run as trusted Python with your filesystem and network access.

Tested on CPU with tiny GPT-2 and grouped-query Llama models, on the oldest and newest supported versions of PyTorch, Transformers, PEFT, accelerate and bitsandbytes. The proving ground's tasks planted inside GPT-2 small were validated live, and its network limit was checked live in Docker.

Highlights

Building blocks for new methods

  • run_with runs a forward pass with temporary interventions.
  • jacobian, jvp and vjp take derivatives between any two sites.
  • sweep runs one prompt under many sets of interventions, in batched passes.
  • ablation_scan zeroes each head and MLP, or groups and pairs of them, in batched passes. It is the fast first sweep, and its docs say what a sweep can't show.

Circuit analyses

  • logit_attribution, path_patch, edge_attribution and find_circuit find a circuit and test it.
  • contrast_directions computes directions from contrasting prompts and checks them on held-out rows.

Features

  • load_sae loads sparse autoencoders and transcoders through SAELens (the saes extra), and edits their features.
  • attribution_graph builds feature-level attribution graphs from transcoders.

Methods the agent writes

  • save_method, load_method and run_method keep a method with its source, results and setup.
  • oumi-weights-lab methods promote writes a recipe draft from one for a maintainer.

Scripts

  • open_lab loads a model in a plain Python script.
  • oumi-weights-lab script FILE runs that script in the lab's environment.

The skill

  • A method map says which call answers which question.
  • A research protocol asks for rival hypotheses, faithfulness checks, and a search for backup mechanisms.
  • A quick start for understanding a model opens the skill.

The proving ground

  • Planted mechanisms with held-out answer keys.
  • A Docker runner with a skill arm and a control arm.
  • A scorer, and a paired comparison.
  • The agent's container can reach only the agent API host, so it can't download a model to diff against.
  • The proving-ground page lists what each task family can and can't show, including what still finds the GPT-2 keys from the weights.

Install this version

Claude Code:

claude plugin marketplace add oumi-ai/weights-lab
claude plugin install oumi-weights-lab@oumi-weights-lab

The plugin includes the SAE support. As before, it includes quantized loading except on Intel Macs and macOS before 14.

CLI, pinned to this release, with LoRA adapters, Hugging Face datasets, multi-GPU loading, quantized loading and SAEs:

uv tool install 'oumi-weights-lab[adapters,datasets,multi-gpu,quantized,saes] @ git+https://github.com/oumi-ai/weights-lab@v0.5.0'

See the README for Codex and other agents.

What's Changed

  • Add a proving ground that scores agents on planted mechanisms by @jgreer013 in #85
  • Add run_with, jacobian, jvp, vjp and sweep for writing new methods by @jgreer013 in #86
  • Compute directions from contrasting prompts with a held-out check by @jgreer013 in #87
  • Add direct logit attribution, path patching, and EAP-IG circuit analyses by @jgreer013 in #88
  • Load sparse autoencoders and transcoders and edit their features by @jgreer013 in #89
  • Build feature-level attribution graphs from transcoders by @jgreer013 in #90
  • Save agent-written methods with their source, results and setup by @jgreer013 in #91
  • Add a method map and a research protocol to the skill by @jgreer013 in #92
  • Plant redundant and self-repairing copy heads in the proving ground by @jgreer013 in #93
  • Run the lab from a plain script with open_lab by @jgreer013 in #94
  • Add ablation_scan, a batched sweep that zeroes each head and MLP by @jgreer013 in #95
  • Plant copy mechanisms inside GPT-2 small for the proving ground by @jgreer013 in #96
  • Set the version to 0.5.0 by @jgreer013 in #97

Full Changelog: v0.4.0...v0.5.0

v0.4.0

Choose a tag to compare

@jgreer013 jgreer013 released this 05 Nov 16:53
de54466

Oumi Weights Lab 0.4 loads models quantized to 4 or 8 bits with bitsandbytes, so a 7–30B model fits on one consumer GPU or a Mac, and you can still edit its packed weights and undo exactly. Version 0.4 is experimental, and snippets run as trusted Python with your filesystem and network access.

Tested on CPU with tiny GPT-2 and grouped-query Llama models, on the oldest and newest supported versions of PyTorch, Transformers, PEFT, and bitsandbytes. The live check loaded, edited, saved, resumed, and exported Qwen2.5-7B-Instruct in nf4 and int8 on one NVIDIA L40S, and Qwen2.5-0.5B-Instruct on an Apple-silicon Mac.

Highlights

Loading

  • start --quantize nf4, fp4, or int8 loads a model with bitsandbytes on CUDA, Apple MPS (macOS 14 and torch 2.5 or later), or CPU. Embeddings, norms, biases, and lm_head stay in float.
  • Scoring, generation, record, patch, heads, head_weights, and activation gradients work on the quantized model. parts and measure report the logical shape plus the packed layout.
  • Checkpoints already quantized with bitsandbytes load as they are. LoRA adapters load on a quantized base.
  • Kernel code from the Hugging Face Hub is blocked unless you pass --allow-hub-kernels.

Editing

  • edit, zero, swap, and project_out work on nf4, fp4, and int8 weights. Each edit requantizes only the blocks it touches: 64 weights for 4-bit, one row for int8.
  • undo restores the packed bytes and scales exactly, so logits return bit for bit.
  • Each edit records the change you asked for, the change that landed, and how far the untouched values in its blocks moved. An edit too small to survive rounding warns instead of passing silently.

Saving and export

  • save_edits and --resume keep quantized edits byte for byte, and resume checks the method, block size, and starting bytes before changing anything.
  • export writes the quantized checkpoint, which reloads with plain from_pretrained to the same bytes and logits. Merging a LoRA adapter into a quantized base is refused.

A new recipe covers quantized models, and another walks through finding and testing a circuit with the tools from 0.3.

Install this version

Claude Code:

claude plugin marketplace add oumi-ai/weights-lab
claude plugin install oumi-weights-lab@oumi-weights-lab

The plugin includes quantized loading, except on Intel Macs and macOS before 14, where bitsandbytes has no wheel.

CLI, pinned to this release, with LoRA adapters, Hugging Face datasets, multi-GPU loading, and quantized loading:

uv tool install 'oumi-weights-lab[adapters,datasets,multi-gpu,quantized] @ git+https://github.com/oumi-ai/weights-lab@v0.4.0'

See the README for Codex and other agents.

What's Changed

  • Load models quantized with bitsandbytes, read-only by @jgreer013 in #51
  • Edit nf4 and fp4 weights in place with exact byte-level undo by @jgreer013 in #52
  • Edit int8 weights row by row with exact undo by @jgreer013 in #53
  • Save, resume and export quantized edits by @jgreer013 in #54
  • Add a recipe for finding and testing a circuit by @jgreer013 in #55
  • Set the version to 0.4.0 by @jgreer013 in #56

Full Changelog: v0.3.0...v0.4.0

v0.3.0

Choose a tag to compare

@jgreer013 jgreer013 released this 05 Nov 03:18
8f81097

Oumi Weights Lab 0.3 adds tools for looking inside a model: attention heads as weight slices, attention capture, input-side hooks, direction removal, activation gradients, and comparisons across sessions. Version 0.3 is experimental, and snippets run as trusted Python with your filesystem and network access.

Tested on CPU with tiny GPT-2, grouped-query Llama, OPT, and Qwen3.5 models, on the oldest and newest supported versions of PyTorch, Transformers, and PEFT (Qwen3.5 on the newest only, since older Transformers lacks it).

Highlights

Heads and attention

  • heads() lists every attention head as slices of the model's own weights, so lab.zero and lab.edit ablate a head reversibly. Grouped-query heads share their keys and values. head_weights() returns one head's query, key, value, output, and gate matrices in one orientation. It supports Llama, Mistral, Qwen2 and 3, Gemma, OLMo, OPT, GPT-2, and Qwen3.5's gated queries.
  • record(..., at="attention") captures attention weights from an eager attention block, with positions selecting queries.

Activations

  • record and patch take at="input" to act on what a module is called with, such as one head's slice of the output projection's input.
  • Positions also take "all" and slices. Batched scoring refuses only positions that left padding would shift.
  • gradient(..., modules=) returns gradients of the loss with respect to module activations, on frozen models too.

Edits and comparisons

  • project_out removes a direction from a weight, bias, or embedding as one undoable edit, in the right orientation for Linear, GPT-2's Conv1D, and embeddings.
  • compare_runs, diff, and compare_suites take cross_session=True to compare runs from different sessions, for example a model against its export. The protocol must still match, and the report lists how the two setups differ.

New recipes cover steering vectors, the logit lens, direction edits, attention heads and induction-head search, and attribution patching.

Install this version

Claude Code:

claude plugin marketplace add oumi-ai/weights-lab
claude plugin install oumi-weights-lab@oumi-weights-lab

CLI, pinned to this release, with LoRA adapters, Hugging Face datasets, and multi-GPU loading:

uv tool install 'oumi-weights-lab[adapters,datasets,multi-gpu] @ git+https://github.com/oumi-ai/weights-lab@v0.3.0'

See the README for Codex and other agents.

What's Changed

  • Compare runs from different sessions when the caller opts in by @jgreer013 in #41
  • Add project_out to remove a direction from a weight, bias, or embedding by @jgreer013 in #42
  • Record and patch module inputs, all or sliced positions, and attention weights by @jgreer013 in #43
  • Add heads() and head_weights() to list attention heads as weight slices by @jgreer013 in #44
  • Take gradients with respect to module activations by @jgreer013 in #45
  • Set the version to 0.3.0 by @jgreer013 in #50

Full Changelog: v0.2.1...v0.3.0

v0.2.1

Choose a tag to compare

@jgreer013 jgreer013 released this 05 Nov 02:38
e6c64bd

Oumi Weights Lab 0.2.1 fixes two bugs in 0.2.0. Version 0.2 is experimental, and snippets run as trusted Python with your filesystem and network access.

Fixes

  • Sampled evaluations and generate stop at the end of a chat turn on checkpoints whose generation config leaves out the token the turn ends with, such as Qwen3.5. In 0.2.0, each reply ran two tokens past the turn and kept a trailing newline, and its length counted those tokens. Generation now also stops at the tokenizer's EOS, as vLLM does, unless you set eos_token_id. Runs where this adds a stop token record stop_token_ids in scoring_config, so they don't pair with 0.2.0 runs.
  • stop keeps the final status a worker writes as it exits, so the path of a deadline edit bundle is no longer lost.

Checks

  • tests/live/validate_truncation.py checks the lab's own sampling against vLLM on MATH-500. On H200s, with 40 problems and 4 samples each, both engines truncated 99.4% of the base Qwen3.5-0.8B's samples, and the lab truncated 51.3% of the edited model's against 50.6% for vLLM.

Install this version

Claude Code:

claude plugin marketplace add oumi-ai/weights-lab
claude plugin install oumi-weights-lab@oumi-weights-lab

CLI, pinned to this release, with LoRA adapters, Hugging Face datasets, and multi-GPU loading:

uv tool install 'oumi-weights-lab[adapters,datasets,multi-gpu] @ git+https://github.com/oumi-ai/weights-lab@v0.2.1'

See the README for Codex and other agents.

What's Changed

  • Keep the final status a worker writes as it exits when stop is waiting by @jgreer013 in #40
  • Stop generation at the tokenizer's EOS as well as the model's by @jgreer013 in #47
  • Check the lab's truncation share against vLLM on MATH-500 by @jgreer013 in #48
  • Set the version to 0.2.1 by @jgreer013 in #49

Full Changelog: v0.2.0...v0.2.1

v0.2.0

Choose a tag to compare

@jgreer013 jgreer013 released this 05 Nov 01:04
f62f5c0

Oumi Weights Lab 0.2 rounds out the evaluation tools and loads one model across all the GPUs of a node. Version 0.2 is experimental, and snippets run as trusted Python with your filesystem and network access.

Tested on CPU, Apple MPS, NVIDIA H200, a 72B model split across eight NVIDIA H100s, and SkyPilot with Slurm.

Highlights

Evaluations

  • Sampled evaluations can draw several samples per problem, with pass@k, majority votes through an answer key, and a checker that sees the whole row. Accuracy is then the mean fraction of samples correct, and comparisons use a paired permutation test.
  • score, evaluate, and generate can wrap prompts in the tokenizer's chat template.
  • lab.hf_rows loads evaluation rows from a Hugging Face dataset split pinned to its commit (with the datasets extra).

Multi-GPU loading

  • start --device-map balanced (or sequential) loads one model across every GPU of a node, within --max-memory per GPU. A model that would need CPU or disk offload is refused before its weights download, with the memory needed and available. peek reports how much of the model sits on each GPU.
  • Scoring, generation, records, patches, gradients, exact edits and undo, edit bundles, and exports all work on a split model, and a bundle saved under one layout resumes under another.
  • Exporting a LoRA model merges one layer at a time instead of copying the model, so the GPUs never hold a second copy.
  • Bundle hashing runs in parallel across GPUs, sessions with --device-map get a 300-second shutdown grace, and SkyPilot starts take --disk-size.

Install this version

Claude Code:

claude plugin marketplace add oumi-ai/weights-lab
claude plugin install oumi-weights-lab@oumi-weights-lab

CLI, pinned to this release, with LoRA adapters, Hugging Face datasets, and multi-GPU loading:

uv tool install 'oumi-weights-lab[adapters,datasets,multi-gpu] @ git+https://github.com/oumi-ai/weights-lab@v0.2.0'

See the README for Codex and other agents.

What's Changed

  • Draw several samples per problem, with pass@k and majority vote by @jgreer013 in #15
  • Wrap prompts in the tokenizer's chat template on request by @jgreer013 in #16
  • Add a security policy and a default issue template by @jgreer013 in #17
  • Make the swallowed-stop lifecycle test deterministic by @jgreer013 in #18
  • Load evaluation rows from a Hugging Face dataset split by @jgreer013 in #19
  • Run a model split across devices the same as on one device by @jgreer013 in #34
  • Export a LoRA model one merged layer at a time instead of copying it by @jgreer013 in #35
  • Load one model across several GPUs with start --device-map by @jgreer013 in #36
  • Hash split models faster, give them longer to stop, and validate them on GPUs by @jgreer013 in #37
  • Set the version to 0.2.0 by @jgreer013 in #39

Full Changelog: v0.1.0...v0.2.0

v0.1.0

Choose a tag to compare

@jgreer013 jgreer013 released this 04 Nov 18:03
7eb34ae

First tagged release of Oumi Weights Lab: an agent skill and Python worker that let your coding agent experiment on an open language model. Load it once, edit its weights reversibly, measure the effect, and export the result. Version 0.1 is experimental, and snippets run as trusted Python with your filesystem and network access.

Tested on CPU, Apple MPS, NVIDIA H200, and SkyPilot with Slurm.

Install this version

Claude Code:

claude plugin marketplace add oumi-ai/weights-lab
claude plugin install oumi-weights-lab@oumi-weights-lab

CLI, pinned to this release:

uv tool install git+https://github.com/oumi-ai/weights-lab@v0.1.0

See the README for Codex and other agents.

What's Changed

  • Add Oumi Weights Lab: persistent model worker, reversible edits, and agent skill by @jgreer013 in #1
  • Show reversible weight edits in the README by @jgreer013 in #3
  • Link the README logo to oumi.ai by @jgreer013 in #4
  • Add a pull request template by @jgreer013 in #5
  • Charge one forward pass per sequence by @jgreer013 in #6
  • Score likelihood rows in padded batches by @jgreer013 in #12
  • Generate sampled rows in batches and record why each one stopped by @jgreer013 in #13
  • Request the maintainer's review on every pull request by @jgreer013 in #14

New Contributors

Full Changelog: https://github.com/oumi-ai/weights-lab/commits/v0.1.0