Skip to content

v0.2.0

Choose a tag to compare

@jgreer013 jgreer013 released this 05 Nov 01:04
f62f5c0

Oumi Weights Lab 0.2 rounds out the evaluation tools and loads one model across all the GPUs of a node. Version 0.2 is experimental, and snippets run as trusted Python with your filesystem and network access.

Tested on CPU, Apple MPS, NVIDIA H200, a 72B model split across eight NVIDIA H100s, and SkyPilot with Slurm.

Highlights

Evaluations

  • Sampled evaluations can draw several samples per problem, with pass@k, majority votes through an answer key, and a checker that sees the whole row. Accuracy is then the mean fraction of samples correct, and comparisons use a paired permutation test.
  • score, evaluate, and generate can wrap prompts in the tokenizer's chat template.
  • lab.hf_rows loads evaluation rows from a Hugging Face dataset split pinned to its commit (with the datasets extra).

Multi-GPU loading

  • start --device-map balanced (or sequential) loads one model across every GPU of a node, within --max-memory per GPU. A model that would need CPU or disk offload is refused before its weights download, with the memory needed and available. peek reports how much of the model sits on each GPU.
  • Scoring, generation, records, patches, gradients, exact edits and undo, edit bundles, and exports all work on a split model, and a bundle saved under one layout resumes under another.
  • Exporting a LoRA model merges one layer at a time instead of copying the model, so the GPUs never hold a second copy.
  • Bundle hashing runs in parallel across GPUs, sessions with --device-map get a 300-second shutdown grace, and SkyPilot starts take --disk-size.

Install this version

Claude Code:

claude plugin marketplace add oumi-ai/weights-lab
claude plugin install oumi-weights-lab@oumi-weights-lab

CLI, pinned to this release, with LoRA adapters, Hugging Face datasets, and multi-GPU loading:

uv tool install 'oumi-weights-lab[adapters,datasets,multi-gpu] @ git+https://github.com/oumi-ai/weights-lab@v0.2.0'

See the README for Codex and other agents.

What's Changed

  • Draw several samples per problem, with pass@k and majority vote by @jgreer013 in #15
  • Wrap prompts in the tokenizer's chat template on request by @jgreer013 in #16
  • Add a security policy and a default issue template by @jgreer013 in #17
  • Make the swallowed-stop lifecycle test deterministic by @jgreer013 in #18
  • Load evaluation rows from a Hugging Face dataset split by @jgreer013 in #19
  • Run a model split across devices the same as on one device by @jgreer013 in #34
  • Export a LoRA model one merged layer at a time instead of copying it by @jgreer013 in #35
  • Load one model across several GPUs with start --device-map by @jgreer013 in #36
  • Hash split models faster, give them longer to stop, and validate them on GPUs by @jgreer013 in #37
  • Set the version to 0.2.0 by @jgreer013 in #39

Full Changelog: v0.1.0...v0.2.0