Bug in Qwen2.5-Math-7B-Instruct's config file

[tristancinquin]

Describe the bug

There seems to be a bug in Qwen2.5-Math-7B-Instruct's configuration file. Launching Qwen2.5-Math-7B-Instruct fails with the following error:

ValueError: User-specified max_model_len (32768) is greater than the derived max_model_len
(max_position_embeddings=4096 or model_max_length=None in model's config.json). 
This may lead to incorrect model outputs or CUDA errors. 
To allow overriding this maximum, set the env var VLLM_ALLOW_LONG_MAX_MODEL_LEN=1

To Reproduce

vec-inf launch Qwen2.5-Math-7B-Instruct then vec-inf status {$JobID} where JobID is the Slurm Job ID.

Screenshots

Error:

Expected:

but with Qwen2.5-Math-7B-Instruct rather than Meta-Llama-3.1-8B-Instruct

Version

0.4.0.post1

[amrit110]

Hi @tristancinquin, i can reproduce this bug. We will fix the config with the next package release, but to get the inference server working, you can manually override. So you can do vec-inf launch Qwen2.5-Math-7B-Instruct --max-model-len 4096

[tristancinquin]

Thanks, I also noticed that Qwen2.5-Math-1.5B-Instruct had the same issue.