Repository navigation
0.8.3 is a bug-fix and compatibility release. It works with transformers 5.23, moves the vLLM integration to vLLM 0.33, and fixes several ways a TransformersModel could load the wrong tokenizer or none at all. It also adds an on/off switch for PEFT adapters.
If you use vLLM, read the upgrade notes: the pin moved to 0.33, and results under n > 1 changed shape.
Highlights
transformers 5.23
- Lazy loading (#844, @Rohit4507). On transformers 5.23,
TransformersModel(repo)with the defaultdispatch=Falsefailed withImpossible to guess which tokenizer to use. It loads again. - Tensor parallelism (#846). transformers 5.23 moved the internals nnsight read to learn how a model is sharded, so every tensor-parallel load failed. nnsight now asks the model for its resolved plan.
The tokenizer now comes from where the weights do
(#848, building on #845 by @shruti25838)
- A
cache_dir,tokenorrevisionnow reaches the tokenizer and other preprocessors on every load path: the lazy build, aconfig=load, and a pre-loaded module. Before, those paths fetched the tokenizer from the default cache and the main branch. Acache_dirholding the only copy failed offline, and a pinnedrevisiongot main's tokenizer with no error. model_kwargs=works. It raised aTypeErrorwithdispatch=Trueand was ignored by the lazy build.
Gemma: <bos> on plain strings
Gemma returns noise unless its input starts with <bos>, and the Gemma 4 -it tokenizers leave it off a plain string. model.trace("...") ran without it and returned garbage with no error. nnsight now sets such a tokenizer to add <bos> when the model loads. Chat input and token ids are unchanged, and PaliGemma is left alone.
PEFT adapters
- Attaching an adapter no longer freezes the base model (#824). On Qwen2.5-0.5B, 290 of 290 parameters required grad before an attach and detach, and 0 after. Every later backward pass was refused.
disable_adapters()andenable_adapters()(#824) switch an attached adapter off and on without changing the module tree. Switched off, the forward is the base model's, bit for bit.autocast_adapter_dtype(#824), on the constructor and onload_adapter.Falseholds the adapter in the model's dtype instead of float32.disable_adapters()unmerges a merged layer first, instead of leaving it merged until the next forward (#827).
vLLM
- nnsight now runs on vLLM 0.33 and its V2 model runner (#837). 0.33 is the last vLLM release with the V1 runner that nnsight used before.
- Over
serve=, a trace returns what a local engine returns (#834). A name saved in several invokes came back as the last invoke's value alone, and a name saved undern > 1as the first sequence's alone. - A stop no longer hides an error over
serve=(#836). Atracer.stop()in one invoke swallowed a real error raised in another. - A trace-level
lora_requestnow covers the invokes that name none (#839). It raised aTypeError. Passlora_request=Noneon an invoke to run the base model.
Upgrade notes
- vLLM is pinned to 0.33. The
vllmextra requiresvllm>=0.33,<0.34(#837). vLLM 0.33 needs Python 3.11 or newer andtransformers<5.21, and publishes only a CUDA 13 wheel. - vLLM,
n > 1: every saved name is a list, one entry per sampled sequence,tracer.resultincluded (#835). Readresult[1].outputs[0].textwhere you readresult.outputs[1].text. Traces withn = 1are unchanged. - vLLM: a trace that raises binds none of its saved names (#836). On a local synchronous engine, code that read partial results after catching a trace's error no longer finds them.
- vLLM:
tracer.stop()on a request withmin_tokensends it at once, not aftermin_tokenstokens (#837). - vLLM refuses three engine setups at startup, each with a message that says what to change (#838): a fallback to the V1 runner,
enable_batch_sharded_sampling=True, andtapswith acudagraph_modethat has full graphs. - Tensor parallelism on a
TransformersModelneeds transformers 5.23 or newer, and so Python 3.11 (#846). nnsight itself still supports Python 3.10. Because vLLM 0.33 needstransformers<5.21, the two need separate environments. local_files_only=Trueraises aTypeErrorwhen the model is built (#848).transformers.pipelinedoes not accept it. SetHF_HUB_OFFLINE=1instead.- Gemma: a string that already starts with
<bos>, such as a chat template rendered to text, now gets a second one. Pass the chat itself (#833).
Other fixes
del xinside a trace now unbinds the name and frees its value. Before, the next read ofxreturned the deleted value, and the value stayed alive until the trace ended (#823).- anyio 4.15 (#847). In a process that had imported nnsight, anyio's thread pool raised
TypeError: Attribute 'save' is missing its type annotationon first use. This broke any FastAPI app that imports nnsight, and on NDIF it rejected every request over 1 MB. - PEFT on a shared server (#824). The server applies the requested adapter state on every request, so one request switching an adapter off no longer leaves it off for the next user.
Docs
- A "LoRA adapters" section for vLLM engines:
enable_lora,add_lora, andlora_requeston a trace, an invoke orgenerate(#841).
What's Changed
- tracing: make
delin a trace body unbind the name and free its value by @JadenFiotto-Kaufman in #823 - PEFT: keep requires_grad across attach/detach; add disable_adapters/enable_adapters and autocast_adapter_dtype by @JadenFiotto-Kaufman in #824
- peft: disable_adapters() unmerges a merged layer first by @JadenFiotto-Kaufman in #827
- Serve the DeepWiki README badge from shields.io by @JadenFiotto-Kaufman in #829
- Dev by @JadenFiotto-Kaufman in #830
- vllm: under n > 1 every saved name is a list, tracer.result included by @JadenFiotto-Kaufman in #835
- vllm serve: merge a trace's saves by the rule the local push uses by @JadenFiotto-Kaufman in #834
- vllm: a trace that raises binds nothing; a stop does not hide an error by @JadenFiotto-Kaufman in #836
- vllm: run on vLLM's V2 model runner and target 0.33 by @JadenFiotto-Kaufman in #837
- vllm: refuse three engine setups the V2 runner cannot serve by @JadenFiotto-Kaufman in #838
- vllm: a trace-level lora_request covers the invokes that name none by @JadenFiotto-Kaufman in #839
- docs: LoRA adapters on a vLLM engine by @JadenFiotto-Kaufman in #841
- transformers: resolve dotted impl paths in _loaded_preprocessors by @Rohit4507 in #844
- tp: support transformers 5.23 by reading its resolved plan by @JadenFiotto-Kaufman in #846
- mount: leave
.saveout of anyio's typed-attribute check by @JadenFiotto-Kaufman in #847 - transformers: build the pipeline the same way on every load path by @JadenFiotto-Kaufman in #848
- transformers: prepend for Gemma tokenizers that leave it off by @aidxhxr in #833
- dev → main: transformers 5.23, load-kwargs routing, Gemma , vLLM V2 runner by @JadenFiotto-Kaufman in #849
New Contributors
- @Rohit4507 made their first contribution in #844
- @aidxhxr made their first contribution in #833
- @shruti25838 made their first contribution in #845, which landed through #848
Full Changelog: v0.8.2...v0.8.3