VRAM control anomaly

[Kenwwww]

Description

I deployed the VLLM backend service using qwen-asr-serve. Even though I set --gpu-memory-utilization to 0.2, in the actual transcription service process, the GPU memory usage far exceeded this value.

Also, are there any suggestions regarding the recommended VRAM and related configuration parameters?

Reproduction

CUDA_VISIBLE_DEVICES=6 qwen-asr-serve ./project_qwen_asr/Qwen3-ASR-0.6B --served-model-name qwen3_asr --port 6022 --gpu-memory-utilization 0.2 --max-model-len 10000

Logs

WARNING 01-30 06:01:32 [registry.py:801] Model architecture Qwen3ASRForConditionalGeneration is already registered, and will be overwritten by the new model class <class 'qwen_asr.core.vllm_backend.qwen3_asr.Qwen3ASRForConditionalGeneration'>.
(APIServer pid=3784343) INFO 01-30 06:01:33 [api_server.py:1272] vLLM API server version 0.14.0
(APIServer pid=3784343) INFO 01-30 06:01:33 [utils.py:263] non-default args: {'model_tag': './project_qwen_asr/Qwen3-ASR-0.6B', 'port': 6022, 'model': './project_qwen_asr/Qwen3-ASR-0.6B', 'max_model_len': 10000, 'served_model_name': ['qwen3_asr'], 'gpu_memory_utilization': 0.2}
(APIServer pid=3784343) INFO 01-30 06:01:33 [model.py:530] Resolved architecture: Qwen3ASRForConditionalGeneration
(APIServer pid=3784343) ERROR 01-30 06:01:33 [repo_utils.py:65] Error retrieving safetensors: Repo id must be in the form 'repo_name' or 'namespace/repo_name': './project_qwen_asr/Qwen3-ASR-0.6B'. Use `repo_type` argument if needed., retrying 1 of 2
(APIServer pid=3784343) ERROR 01-30 06:01:35 [repo_utils.py:63] Error retrieving safetensors: Repo id must be in the form 'repo_name' or 'namespace/repo_name': './project_qwen_asr/Qwen3-ASR-0.6B'. Use `repo_type` argument if needed.
(APIServer pid=3784343) INFO 01-30 06:01:35 [model.py:1866] Downcasting torch.float32 to torch.bfloat16.
(APIServer pid=3784343) INFO 01-30 06:01:35 [model.py:1545] Using max model len 10000
(APIServer pid=3784343) INFO 01-30 06:01:35 [scheduler.py:229] Chunked prefill is enabled with max_num_batched_tokens=2048.
(APIServer pid=3784343) INFO 01-30 06:01:35 [vllm.py:630] Asynchronous scheduling is enabled.
(APIServer pid=3784343) INFO 01-30 06:01:35 [vllm.py:637] Disabling NCCL for DP synchronization when using async scheduling.
(APIServer pid=3784343) The tokenizer you are loading from './project_qwen_asr/Qwen3-ASR-0.6B' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
(APIServer pid=3784343) The following generation flags are not valid and may be ignored: ['temperature']. Set `TRANSFORMERS_VERBOSITY=info` for more details.
WARNING 01-30 06:01:49 [registry.py:801] Model architecture Qwen3ASRForConditionalGeneration is already registered, and will be overwritten by the new model class <class 'qwen_asr.core.vllm_backend.qwen3_asr.Qwen3ASRForConditionalGeneration'>.
(EngineCore_DP0 pid=3784648) INFO 01-30 06:01:50 [core.py:97] Initializing a V1 LLM engine (v0.14.0) with config: model='./project_qwen_asr/Qwen3-ASR-0.6B', speculative_config=None, tokenizer='./project_qwen_asr/Qwen3-ASR-0.6B', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=10000, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=0, served_model_name=qwen3_asr, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'level': None, 'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['none'], 'splitting_ops': ['vllm::unified_attention', 'vllm::unified_attention_with_output', 'vllm::unified_mla_attention', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::gdn_attention_core', 'vllm::kda_attention', 'vllm::sparse_attn_indexer'], 'compile_mm_encoder': False, 'compile_sizes': [], 'compile_ranges_split_points': [2048], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256, 272, 288, 304, 320, 336, 352, 368, 384, 400, 416, 432, 448, 464, 480, 496, 512], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': False, 'fuse_act_quant': False, 'fuse_attn_quant': False, 'eliminate_noops': True, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': False}, 'max_cudagraph_capture_size': 512, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': True}, 'local_cache_dir': None}
(EngineCore_DP0 pid=3784648) The tokenizer you are loading from './project_qwen_asr/Qwen3-ASR-0.6B' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
(EngineCore_DP0 pid=3784648) INFO 01-30 06:01:51 [parallel_state.py:1214] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://115.239.230.131:56799 backend=nccl
(EngineCore_DP0 pid=3784648) INFO 01-30 06:01:56 [parallel_state.py:1425] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A
(EngineCore_DP0 pid=3784648) The tokenizer you are loading from './project_qwen_asr/Qwen3-ASR-0.6B' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
(EngineCore_DP0 pid=3784648) The tokenizer you are loading from './project_qwen_asr/Qwen3-ASR-0.6B' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
(EngineCore_DP0 pid=3784648) INFO 01-30 06:02:02 [gpu_model_runner.py:3808] Starting to load model ./project_qwen_asr/Qwen3-ASR-0.6B...
(EngineCore_DP0 pid=3784648) INFO 01-30 06:02:03 [mm_encoder_attention.py:86] Using AttentionBackendEnum.FLASH_ATTN for MMEncoderAttention.
(EngineCore_DP0 pid=3784648) INFO 01-30 06:02:03 [vllm.py:630] Asynchronous scheduling is enabled.
(EngineCore_DP0 pid=3784648) INFO 01-30 06:02:03 [cuda.py:351] Using FLASH_ATTN attention backend out of potential backends: ('FLASH_ATTN', 'FLASHINFER', 'TRITON_ATTN', 'FLEX_ATTENTION')
Loading safetensors checkpoint shards:   0% Completed | 0/1 [00:00<?, ?it/s]
Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:00<00:00,  1.79it/s]
Loading safetensors checkpoint shards: 100% Completed | 1/1 [00:00<00:00,  1.79it/s]
(EngineCore_DP0 pid=3784648) 
(EngineCore_DP0 pid=3784648) INFO 01-30 06:02:04 [default_loader.py:291] Loading weights took 0.63 seconds
(EngineCore_DP0 pid=3784648) INFO 01-30 06:02:05 [gpu_model_runner.py:3905] Model loading took 1.53 GiB memory and 0.990190 seconds
(EngineCore_DP0 pid=3784648) INFO 01-30 06:02:05 [gpu_model_runner.py:4715] Encoder cache will be initialized with a budget of 2048 tokens, and profiled with 5 audio items of the maximum feature size.
(EngineCore_DP0 pid=3784648) INFO 01-30 06:02:13 [backends.py:644] Using cache directory: /home/.cache/vllm/torch_compile_cache/02e6010edd/rank_0_0/backbone for vLLM's torch.compile
(EngineCore_DP0 pid=3784648) INFO 01-30 06:02:13 [backends.py:704] Dynamo bytecode transform time: 7.19 s
(EngineCore_DP0 pid=3784648) INFO 01-30 06:02:17 [backends.py:226] Directly load the compiled graph(s) for compile range (1, 2048) from the cache, took 1.434 s
(EngineCore_DP0 pid=3784648) INFO 01-30 06:02:17 [monitor.py:34] torch.compile takes 8.62 s in total
(EngineCore_DP0 pid=3784648) INFO 01-30 06:02:18 [gpu_worker.py:358] Available KV cache memory: 1.74 GiB
(EngineCore_DP0 pid=3784648) INFO 01-30 06:02:18 [kv_cache_utils.py:1305] GPU KV cache size: 16,272 tokens
(EngineCore_DP0 pid=3784648) INFO 01-30 06:02:18 [kv_cache_utils.py:1310] Maximum concurrency for 10,000 tokens per request: 1.63x
Capturing CUDA graphs (mixed prefill-decode, PIECEWISE): 100%|█████████████████████████████████████████████████████████████████████████████████████| 51/51 [00:01<00:00, 43.52it/s]
Capturing CUDA graphs (decode, FULL): 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████| 35/35 [00:00<00:00, 48.03it/s]
(EngineCore_DP0 pid=3784648) INFO 01-30 06:02:21 [gpu_model_runner.py:4856] Graph capturing finished in 2 secs, took 0.50 GiB
(EngineCore_DP0 pid=3784648) INFO 01-30 06:02:21 [core.py:273] init engine (profile, create kv cache, warmup model) took 15.60 seconds
(APIServer pid=3784343) INFO 01-30 06:02:21 [api_server.py:1014] Supported tasks: ['generate', 'transcription']
(APIServer pid=3784343) WARNING 01-30 06:02:21 [model.py:1358] Default sampling parameters have been overridden by the model's Hugging Face generation config recommended from the model creator. If this is not intended, please relaunch vLLM instance with `--generation-config vllm`.
(APIServer pid=3784343) INFO 01-30 06:02:21 [serving_responses.py:224] Using default chat sampling params from model: {'temperature': 1e-06}
(APIServer pid=3784343) INFO 01-30 06:02:21 [serving_chat.py:146] Using default chat sampling params from model: {'temperature': 1e-06}
(APIServer pid=3784343) INFO 01-30 06:02:21 [serving_chat.py:182] Warming up chat template processing...
(APIServer pid=3784343) The tokenizer you are loading from './project_qwen_asr/Qwen3-ASR-0.6B' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
(APIServer pid=3784343) INFO 01-30 06:02:22 [chat_utils.py:599] Detected the chat template content format to be 'openai'. You can set `--chat-template-content-format` to override this.
(APIServer pid=3784343) INFO 01-30 06:02:22 [serving_chat.py:218] Chat template warmup completed in 1074.1ms
(APIServer pid=3784343) INFO 01-30 06:02:22 [serving_completion.py:78] Using default completion sampling params from model: {'temperature': 1e-06}
(APIServer pid=3784343) The tokenizer you are loading from './project_qwen_asr/Qwen3-ASR-0.6B' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
(APIServer pid=3784343) INFO 01-30 06:02:23 [speech_to_text.py:111] Overwriting default completion sampling param with: {'temperature': 1e-06}
(APIServer pid=3784343) INFO 01-30 06:02:23 [speech_to_text.py:138] Warming up audio preprocessing libraries...
(APIServer pid=3784343) INFO 01-30 06:02:24 [speech_to_text.py:174] Audio preprocessing warmup completed in 1.11s
(APIServer pid=3784343) INFO 01-30 06:02:24 [speech_to_text.py:201] Warming up multimodal input processor...
(APIServer pid=3784343) The tokenizer you are loading from './project_qwen_asr/Qwen3-ASR-0.6B' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
(APIServer pid=3784343) The tokenizer you are loading from './project_qwen_asr/Qwen3-ASR-0.6B' with an incorrect regex pattern: https://huggingface.co/mistralai/Mistral-Small-3.1-24B-Instruct-2503/discussions/84#69121093e8b480e709447d5e. This will lead to incorrect tokenization. You should set the `fix_mistral_regex=True` flag when loading this tokenizer to fix this issue.
(APIServer pid=3784343) INFO 01-30 06:02:30 [speech_to_text.py:234] Input processor warmup completed in 5.27s
(APIServer pid=3784343) INFO 01-30 06:02:30 [speech_to_text.py:111] Overwriting default completion sampling param with: {'temperature': 1e-06}
(APIServer pid=3784343) INFO 01-30 06:02:30 [speech_to_text.py:138] Warming up audio preprocessing libraries...
(APIServer pid=3784343) INFO 01-30 06:02:30 [speech_to_text.py:174] Audio preprocessing warmup completed in 0.01s
(APIServer pid=3784343) INFO 01-30 06:02:30 [speech_to_text.py:201] Warming up multimodal input processor...
(APIServer pid=3784343) INFO 01-30 06:02:30 [speech_to_text.py:234] Input processor warmup completed in 0.00s
(APIServer pid=3784343) INFO 01-30 06:02:30 [serving_chat.py:146] Using default chat sampling params from model: {'temperature': 1e-06}
(APIServer pid=3784343) INFO 01-30 06:02:30 [api_server.py:1346] Starting vLLM API server 0 on http://0.0.0.0:6022
(APIServer pid=3784343) INFO 01-30 06:02:30 [launcher.py:38] Available routes are:
(APIServer pid=3784343) INFO 01-30 06:02:30 [launcher.py:46] Route: /openapi.json, Methods: HEAD, GET
(APIServer pid=3784343) INFO 01-30 06:02:30 [launcher.py:46] Route: /docs, Methods: HEAD, GET
(APIServer pid=3784343) INFO 01-30 06:02:30 [launcher.py:46] Route: /docs/oauth2-redirect, Methods: HEAD, GET
(APIServer pid=3784343) INFO 01-30 06:02:30 [launcher.py:46] Route: /redoc, Methods: HEAD, GET
(APIServer pid=3784343) INFO 01-30 06:02:30 [launcher.py:46] Route: /scale_elastic_ep, Methods: POST
(APIServer pid=3784343) INFO 01-30 06:02:30 [launcher.py:46] Route: /is_scaling_elastic_ep, Methods: POST
(APIServer pid=3784343) INFO 01-30 06:02:30 [launcher.py:46] Route: /tokenize, Methods: POST
(APIServer pid=3784343) INFO 01-30 06:02:30 [launcher.py:46] Route: /detokenize, Methods: POST
(APIServer pid=3784343) INFO 01-30 06:02:30 [launcher.py:46] Route: /inference/v1/generate, Methods: POST
(APIServer pid=3784343) INFO 01-30 06:02:30 [launcher.py:46] Route: /pause, Methods: POST
(APIServer pid=3784343) INFO 01-30 06:02:30 [launcher.py:46] Route: /resume, Methods: POST
(APIServer pid=3784343) INFO 01-30 06:02:30 [launcher.py:46] Route: /is_paused, Methods: GET
(APIServer pid=3784343) INFO 01-30 06:02:30 [launcher.py:46] Route: /metrics, Methods: GET
(APIServer pid=3784343) INFO 01-30 06:02:30 [launcher.py:46] Route: /health, Methods: GET
(APIServer pid=3784343) INFO 01-30 06:02:30 [launcher.py:46] Route: /load, Methods: GET
(APIServer pid=3784343) INFO 01-30 06:02:30 [launcher.py:46] Route: /v1/models, Methods: GET
(APIServer pid=3784343) INFO 01-30 06:02:30 [launcher.py:46] Route: /version, Methods: GET
(APIServer pid=3784343) INFO 01-30 06:02:30 [launcher.py:46] Route: /v1/responses, Methods: POST
(APIServer pid=3784343) INFO 01-30 06:02:30 [launcher.py:46] Route: /v1/responses/{response_id}, Methods: GET
(APIServer pid=3784343) INFO 01-30 06:02:30 [launcher.py:46] Route: /v1/responses/{response_id}/cancel, Methods: POST
(APIServer pid=3784343) INFO 01-30 06:02:30 [launcher.py:46] Route: /v1/messages, Methods: POST
(APIServer pid=3784343) INFO 01-30 06:02:30 [launcher.py:46] Route: /v1/chat/completions, Methods: POST
(APIServer pid=3784343) INFO 01-30 06:02:30 [launcher.py:46] Route: /v1/completions, Methods: POST
(APIServer pid=3784343) INFO 01-30 06:02:30 [launcher.py:46] Route: /v1/audio/transcriptions, Methods: POST
(APIServer pid=3784343) INFO 01-30 06:02:30 [launcher.py:46] Route: /v1/audio/translations, Methods: POST
(APIServer pid=3784343) INFO 01-30 06:02:30 [launcher.py:46] Route: /ping, Methods: GET
(APIServer pid=3784343) INFO 01-30 06:02:30 [launcher.py:46] Route: /ping, Methods: POST
(APIServer pid=3784343) INFO 01-30 06:02:30 [launcher.py:46] Route: /invocations, Methods: POST
(APIServer pid=3784343) INFO 01-30 06:02:30 [launcher.py:46] Route: /classify, Methods: POST
(APIServer pid=3784343) INFO 01-30 06:02:30 [launcher.py:46] Route: /v1/embeddings, Methods: POST
(APIServer pid=3784343) INFO 01-30 06:02:30 [launcher.py:46] Route: /score, Methods: POST
(APIServer pid=3784343) INFO 01-30 06:02:30 [launcher.py:46] Route: /v1/score, Methods: POST
(APIServer pid=3784343) INFO 01-30 06:02:30 [launcher.py:46] Route: /rerank, Methods: POST
(APIServer pid=3784343) INFO 01-30 06:02:30 [launcher.py:46] Route: /v1/rerank, Methods: POST
(APIServer pid=3784343) INFO 01-30 06:02:30 [launcher.py:46] Route: /v2/rerank, Methods: POST
(APIServer pid=3784343) INFO 01-30 06:02:30 [launcher.py:46] Route: /pooling, Methods: POST
(APIServer pid=3784343) INFO:     Started server process [3784343]
(APIServer pid=3784343) INFO:     Waiting for application startup.
(APIServer pid=3784343) INFO:     Application startup complete.

Environment Information

python 3.12 torch 2.9.1 flash_attn 2.8.3

Known Issue

• The issue hasn't been already addressed in Documentation, Issues, and Discussions.

[kyr0]

You might wanna try my inference server implementation. I haven't seen any memory allocation issues with it. It's consistently < 2GB VRAM usage under load / PR'ed it here. https://github.com/kyr0/fast-qwen-asr-inference-vllm

I'd be interested to see how well it performs in your HW setup.

Edit: https://github.com/kyr0/fast-qwen-asr-inference-vllm/blob/main/server.py#L150 <- please be aware that I forgot to pass an env variable here.

[github-actions]

This issue has been automatically marked as inactive due to lack of recent activity. Should you believe it remains unresolved and warrants attention, kindly leave a comment on this thread.