Repository navigation
v0.2.0
Oumi Weights Lab 0.2 rounds out the evaluation tools and loads one model across all the GPUs of a node. Version 0.2 is experimental, and snippets run as trusted Python with your filesystem and network access.
Tested on CPU, Apple MPS, NVIDIA H200, a 72B model split across eight NVIDIA H100s, and SkyPilot with Slurm.
Highlights
Evaluations
- Sampled evaluations can draw several samples per problem, with pass@k, majority votes through an answer key, and a checker that sees the whole row. Accuracy is then the mean fraction of samples correct, and comparisons use a paired permutation test.
score,evaluate, andgeneratecan wrap prompts in the tokenizer's chat template.lab.hf_rowsloads evaluation rows from a Hugging Face dataset split pinned to its commit (with thedatasetsextra).
Multi-GPU loading
start --device-map balanced(orsequential) loads one model across every GPU of a node, within--max-memoryper GPU. A model that would need CPU or disk offload is refused before its weights download, with the memory needed and available.peekreports how much of the model sits on each GPU.- Scoring, generation, records, patches, gradients, exact edits and undo, edit bundles, and exports all work on a split model, and a bundle saved under one layout resumes under another.
- Exporting a LoRA model merges one layer at a time instead of copying the model, so the GPUs never hold a second copy.
- Bundle hashing runs in parallel across GPUs, sessions with
--device-mapget a 300-second shutdown grace, and SkyPilot starts take--disk-size.
Install this version
Claude Code:
claude plugin marketplace add oumi-ai/weights-lab
claude plugin install oumi-weights-lab@oumi-weights-labCLI, pinned to this release, with LoRA adapters, Hugging Face datasets, and multi-GPU loading:
uv tool install 'oumi-weights-lab[adapters,datasets,multi-gpu] @ git+https://github.com/oumi-ai/weights-lab@v0.2.0'See the README for Codex and other agents.
What's Changed
- Draw several samples per problem, with pass@k and majority vote by @jgreer013 in #15
- Wrap prompts in the tokenizer's chat template on request by @jgreer013 in #16
- Add a security policy and a default issue template by @jgreer013 in #17
- Make the swallowed-stop lifecycle test deterministic by @jgreer013 in #18
- Load evaluation rows from a Hugging Face dataset split by @jgreer013 in #19
- Run a model split across devices the same as on one device by @jgreer013 in #34
- Export a LoRA model one merged layer at a time instead of copying it by @jgreer013 in #35
- Load one model across several GPUs with start --device-map by @jgreer013 in #36
- Hash split models faster, give them longer to stop, and validate them on GPUs by @jgreer013 in #37
- Set the version to 0.2.0 by @jgreer013 in #39
Full Changelog: v0.1.0...v0.2.0