Skip to content

Enhance vllm support and introduce multi-worker inference#707

Merged
HaomingX merged 5 commits into
zjunlp:mainfrom
Carbene:main
Jul 4, 2026
Merged

Enhance vllm support and introduce multi-worker inference#707
HaomingX merged 5 commits into
zjunlp:mainfrom
Carbene:main

Conversation

@Carbene

@Carbene Carbene commented Jul 3, 2026

Copy link
Copy Markdown
Contributor

Features

This PR brings significant upgrades to the vLLM integration and inference performance, along with documentation updates.

  1. Full vLLM Support for Train-Free Methods: Expanded the vLLM support range. It now supports all train-free methods, allowing users to leverage vLLM's high-throughput serving for a wider variety of setups. Now it supports caa, sae, sta and vector prompt. Also, merge is fine too. But to ensure the maximal changes, generation part does not support multi-workers, it uses an in-process method to avoid complex hook method.
  2. Multi-Worker Inference: Introduced a multi-worker architecture for the application stage. This significantly speeds up the inference process by parallelizing workloads.
  3. Documentation Update: Updated readme_2.md to reflect these new capabilities, usage instructions, and configuration changes.

Verification

Successfully running the test in falcon. Observed siginificant result in Qwen3 and Qwen2.5-VL. Test all four methods: CAA, STA, SAE, Vector Prompt.

Carbene added 2 commits July 4, 2026 00:35
…! Furthermore, we introduce multi-workers for application stage to speed up inference. Also update the readme_2.
@HaomingX HaomingX merged commit fed52b9 into zjunlp:main Jul 4, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants