Vision requests fail with `RuntimeError: There is no Stream(gpu, 0) in current thread`

[684xcm4rx8-dev]

[Bug] Vision requests fail with RuntimeError: There is no Stream(gpu, 0) in current thread

vllm-mlx 0.2.9 returns HTTP 500 on any multimodal /v1/chat/completions request to a Qwen3-VL model. The underlying error originates in mlx_vlm/generate.py:323 wired_limit and indicates that vllm-mlx's request handler is dispatching MLX work to a different thread than the one that created the GPU stream — which MLX does not allow (ml-explore/mlx#2133).

The same failure reproduces across at least three configurations (default, --max-num-seqs 1 --prefill-batch-size 1 --completion-batch-size 1, and --continuous-batching --gpu-memory-utilization 0.55), so it is not a flag-tunable workaround.

Environment

vllm-mlx	0.2.9
mlx	0.31.2
mlx-vlm	0.4.4
Python	3.11.15
macOS	26.4.1
Hardware	Apple M1 Max, 32 GB unified memory
Model	mlx-community/Qwen3-VL-8B-Instruct-4bit (loaded from a local directory; behavior identical with HF repo id)

Minimal reproduction

# Pre-download the model (one-time, ~5.4 GB)
vllm-mlx download --mllm mlx-community/Qwen3-VL-8B-Instruct-4bit
# (or use --offline pointed at a local directory; both fail identically)

# Start the server (defaults — no batching, no caching tweaks)
vllm-mlx serve mlx-community/Qwen3-VL-8B-Instruct-4bit \
    --port 8540 \
    --mllm \
    --offline

# In another terminal — confirm /v1/models works
curl -s http://127.0.0.1:8540/v1/models
# {"object":"list","data":[{"id":"mlx-community/Qwen3-VL-8B-Instruct-4bit",...}]}

# Send a vision request
IMG_B64=$(base64 -i /path/to/any/jpeg.jpg | tr -d '\n')
curl -s http://127.0.0.1:8540/v1/chat/completions \
    -H 'Content-Type: application/json' \
    -d "{
      \"model\": \"mlx-community/Qwen3-VL-8B-Instruct-4bit\",
      \"messages\": [{\"role\": \"user\", \"content\": [
        {\"type\": \"text\", \"text\": \"Describe this image in one sentence.\"},
        {\"type\": \"image_url\", \"image_url\": {\"url\": \"data:image/jpeg;base64,$IMG_B64\"}}
      ]}],
      \"max_tokens\": 80
    }"

Expected behavior

A chat completion with a vision-aware description.

Actual behavior

The server returns HTTP 500. Server log shows:

INFO:vllm_mlx.models.mllm:MLLM.chat() called with 1 messages
INFO:vllm_mlx.models.mllm:Applying chat template with 1 messages, 1 images, 0 audios
[transformers] Kwargs passed to `processor.__call__` have to be in `processor_kwargs` dict, not in `**kwargs`
INFO:     127.0.0.1:55398 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error
ERROR:    Exception in ASGI application
Traceback (most recent call last):
  File ".../mlx_vlm/generate.py", line 323, in wired_limit
    yield
  File ".../mlx_vlm/generate.py", line 735, in stream_generate
    for n, (token, logprobs) in enumerate(gen):
  File ".../mlx_vlm/generate.py", line 514, in generate_step
    embedding_output = model.get_input_embeddings(
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".../mlx_vlm/models/qwen3_vl/qwen3_vl.py", line 73, in get_input_embeddings
    hidden_states, deepstack_image_embeds = self.vision_tower(
                                            ^^^^^^^^^^^^^^^^^^
  File ".../mlx_vlm/models/qwen3_vl/vision.py", line 381, in __call__
    pos_embeds = self.fast_pos_embed_interpolate(grid_thw)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".../mlx_vlm/models/qwen3_vl/vision.py", line 332, in fast_pos_embed_interpolate
    idx_list[i].extend(indices[i].tolist())
                       ^^^^^^^^^^^^^^^^^^^
RuntimeError: There is no Stream(gpu, 0) in current thread.

The above exception was the direct cause of the following exception:
... (FastAPI/Starlette/uvicorn frames) ...
  File ".../mlx_vlm/generate.py", line 327, in wired_limit
    mx.synchronize(s)
RuntimeError: There is no Stream(gpu, 0) in current thread.

When --continuous-batching is enabled, the same root error appears earlier in the call path (mllm_batch_generator.py:1310 mx.eval(sampled, logprobs)), with the request returning HTTP 200 but finish_reason: error and content: null:

ERROR:vllm_mlx.mllm_batch_generator:Failed to process batch of 1 prompts: RuntimeError: There is no Stream(gpu, 2) in current thread.
Traceback (most recent call last):
  File ".../vllm_mlx/mllm_batch_generator.py", line 1559, in _next
    new_batch = self._process_prompts(requests)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File ".../vllm_mlx/mllm_batch_generator.py", line 1310, in _process_prompts
    mx.eval(sampled, logprobs)
RuntimeError: There is no Stream(gpu, 2) in current thread.
WARNING:vllm_mlx.mllm_scheduler:Request ad545b9b-... failed during preprocessing
INFO:vllm_mlx.server:Chat completion: 0 tokens in 1.62s (0.0 tok/s)
INFO:     127.0.0.1:62161 - "POST /v1/chat/completions HTTP/1.1" 200 OK

The stream id varies (Stream(gpu, 0), Stream(gpu, 2)) — consistent with each request being dispatched to a different ThreadPoolExecutor worker.

Root-cause hypothesis

MLX's Stream and wired_limit are not thread-safe: a stream allocated on one thread cannot be mx.synchronize()-d from another. The relevant upstream issue is ml-explore/mlx#2133.

vllm-mlx's request handler appears to dispatch MLX work via asyncio.to_thread (default ThreadPoolExecutor), so each request lands on whichever worker happens to be free. The model is loaded on one thread (during startup); subsequent inference is attempted on different threads, which trips MLX's per-thread stream invariant when the wired_limit context manager exits.

The well-known workaround in the mlx-vlm community is to pin all MLX inference to a single, dedicated thread via something like ThreadPoolExecutor(max_workers=1) and route every MLX call through that one worker — see e.g. LM Studio's MLX backend which does this internally, and hand-rolled servers that follow the same pattern. Fixing this likely requires vllm-mlx to introduce a single MLX-pinned worker rather than delegating to a generic thread pool.

Configurations tested (all fail identically)

Config	Result
serve --mllm --offline (defaults)	RuntimeError: There is no Stream(gpu, 0) in current thread
serve --mllm --offline --max-num-seqs 1 --prefill-batch-size 1 --completion-batch-size 1	same (stream id 0)
serve --mllm --offline --continuous-batching --gpu-memory-utilization 0.55	same (stream id 2) — request returns 200 with finish_reason: error

Impact

vllm-mlx is otherwise an excellent fit for the use case (OpenAI-compatible server for Apple Silicon), but the inability to run any multimodal request blocks adoption. Drew's homelab vision pipeline currently uses a hand-rolled HTTP wrapper around mlx-vlm with the single-thread-pinning workaround; the hope was to drop ~500 LOC of custom code in favor of vllm-mlx, but the cutover is blocked on this bug.

Happy to help

I can provide additional logs, run alternative configurations, or test patches if helpful. Thank you for the project — looking forward to using it once this is resolved.

[janhilgard]

Fixed in current main (since commit 0d8c6d1, 2026-04-23).

The root cause was exactly as described — MLX streams are thread-local, and asyncio.to_thread dispatched inference to arbitrary thread pool workers. The fix (vllm_mlx/mlx_streams.py) creates a dedicated stream on the worker thread and rebinds mlx_lm.generate.generation_stream / mlx_vlm.generate.generation_stream at worker-entry boundaries.

This is called in all three engine paths:

• engine_core.py (MLLM continuous batching)
• mllm_scheduler.py (MLLM scheduler)
• engine/simple.py (SimpleEngine)

Upgrading to any version after 0.2.9 (i.e., current main) resolves the issue. No configuration flags needed.