Skip to content

v0.8.3

Latest

Choose a tag to compare

@JadenFiotto-Kaufman JadenFiotto-Kaufman released this 18 Nov 21:07
ff4c2a3

0.8.3 is a bug-fix and compatibility release. It works with transformers 5.23, moves the vLLM integration to vLLM 0.33, and fixes several ways a TransformersModel could load the wrong tokenizer or none at all. It also adds an on/off switch for PEFT adapters.

If you use vLLM, read the upgrade notes: the pin moved to 0.33, and results under n > 1 changed shape.

Highlights

transformers 5.23

  • Lazy loading (#844, @Rohit4507). On transformers 5.23, TransformersModel(repo) with the default dispatch=False failed with Impossible to guess which tokenizer to use. It loads again.
  • Tensor parallelism (#846). transformers 5.23 moved the internals nnsight read to learn how a model is sharded, so every tensor-parallel load failed. nnsight now asks the model for its resolved plan.

The tokenizer now comes from where the weights do

(#848, building on #845 by @shruti25838)

  • A cache_dir, token or revision now reaches the tokenizer and other preprocessors on every load path: the lazy build, a config= load, and a pre-loaded module. Before, those paths fetched the tokenizer from the default cache and the main branch. A cache_dir holding the only copy failed offline, and a pinned revision got main's tokenizer with no error.
  • model_kwargs= works. It raised a TypeError with dispatch=True and was ignored by the lazy build.

Gemma: <bos> on plain strings

(#833, @aidxhxr)

Gemma returns noise unless its input starts with <bos>, and the Gemma 4 -it tokenizers leave it off a plain string. model.trace("...") ran without it and returned garbage with no error. nnsight now sets such a tokenizer to add <bos> when the model loads. Chat input and token ids are unchanged, and PaliGemma is left alone.

PEFT adapters

  • Attaching an adapter no longer freezes the base model (#824). On Qwen2.5-0.5B, 290 of 290 parameters required grad before an attach and detach, and 0 after. Every later backward pass was refused.
  • disable_adapters() and enable_adapters() (#824) switch an attached adapter off and on without changing the module tree. Switched off, the forward is the base model's, bit for bit.
  • autocast_adapter_dtype (#824), on the constructor and on load_adapter. False holds the adapter in the model's dtype instead of float32.
  • disable_adapters() unmerges a merged layer first, instead of leaving it merged until the next forward (#827).

vLLM

  • nnsight now runs on vLLM 0.33 and its V2 model runner (#837). 0.33 is the last vLLM release with the V1 runner that nnsight used before.
  • Over serve=, a trace returns what a local engine returns (#834). A name saved in several invokes came back as the last invoke's value alone, and a name saved under n > 1 as the first sequence's alone.
  • A stop no longer hides an error over serve= (#836). A tracer.stop() in one invoke swallowed a real error raised in another.
  • A trace-level lora_request now covers the invokes that name none (#839). It raised a TypeError. Pass lora_request=None on an invoke to run the base model.

Upgrade notes

  • vLLM is pinned to 0.33. The vllm extra requires vllm>=0.33,<0.34 (#837). vLLM 0.33 needs Python 3.11 or newer and transformers<5.21, and publishes only a CUDA 13 wheel.
  • vLLM, n > 1: every saved name is a list, one entry per sampled sequence, tracer.result included (#835). Read result[1].outputs[0].text where you read result.outputs[1].text. Traces with n = 1 are unchanged.
  • vLLM: a trace that raises binds none of its saved names (#836). On a local synchronous engine, code that read partial results after catching a trace's error no longer finds them.
  • vLLM: tracer.stop() on a request with min_tokens ends it at once, not after min_tokens tokens (#837).
  • vLLM refuses three engine setups at startup, each with a message that says what to change (#838): a fallback to the V1 runner, enable_batch_sharded_sampling=True, and taps with a cudagraph_mode that has full graphs.
  • Tensor parallelism on a TransformersModel needs transformers 5.23 or newer, and so Python 3.11 (#846). nnsight itself still supports Python 3.10. Because vLLM 0.33 needs transformers<5.21, the two need separate environments.
  • local_files_only=True raises a TypeError when the model is built (#848). transformers.pipeline does not accept it. Set HF_HUB_OFFLINE=1 instead.
  • Gemma: a string that already starts with <bos>, such as a chat template rendered to text, now gets a second one. Pass the chat itself (#833).

Other fixes

  • del x inside a trace now unbinds the name and frees its value. Before, the next read of x returned the deleted value, and the value stayed alive until the trace ended (#823).
  • anyio 4.15 (#847). In a process that had imported nnsight, anyio's thread pool raised TypeError: Attribute 'save' is missing its type annotation on first use. This broke any FastAPI app that imports nnsight, and on NDIF it rejected every request over 1 MB.
  • PEFT on a shared server (#824). The server applies the requested adapter state on every request, so one request switching an adapter off no longer leaves it off for the next user.

Docs

  • A "LoRA adapters" section for vLLM engines: enable_lora, add_lora, and lora_request on a trace, an invoke or generate (#841).

What's Changed

New Contributors

Full Changelog: v0.8.2...v0.8.3