[Usage]: Qwen2 GGUF model can't run successfully #7689
Labels
usage
How to use vllm
Comments
A new problem |
|
When I cloned and installed the latest transformers(4.45.0.dev0) library locally, I re-ran the inference of Qwen2 GGUF's vllm and encountered this new issue. |
|
Did other models like Qwen2-7B-GGUF/Qwen2-14B-GGUF also encounter this error? Or just 72B has this new issue? (72B is too large for me to reproduce this. And 7B should have same root issue with 72B, so we could reproduce this if this is related to the root issue.) BTW, I can run the 7B inference with transformers(4.45.0.dev0) without any issue and the new issue above is very strange and shouldn't be encountered in most of cases: qwen2-7b-instruct-q2_k.gguf: 100%|██████████████████████████████████████████████████████████████████████████████| 3.02G/3.02G [00:22<00:00, 134MB/s]
INFO 08-20 13:19:46 config.py:1552] Downcasting torch.float32 to torch.float16.
WARNING 08-20 13:19:46 config.py:312] gguf quantization is not fully optimized yet. The speed can be slower than non-quantized models.
INFO 08-20 13:19:46 llm_engine.py:182] Initializing an LLM engine (v0.5.4) with config: model='/root/.cache/huggingface/hub/models--Qwen--Qwen2-7B-Instruct-GGUF/snapshots/7c1879f2983b48bb6a5609f7546299b833d25d13/qwen2-7b-instruct-q2_k.gguf', speculative_config=None, tokenizer='/root/.cache/huggingface/hub/models--Qwen--Qwen2-7B-Instruct-GGUF/snapshots/7c1879f2983b48bb6a5609f7546299b833d25d13/qwen2-7b-instruct-q2_k.gguf', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, rope_scaling=None, rope_theta=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.float16, max_seq_len=32768, download_dir=None, load_format=LoadFormat.GGUF, tensor_parallel_size=1, pipeline_parallel_size=1, disable_custom_all_reduce=False, quantization=gguf, enforce_eager=True, kv_cache_dtype=auto, quantization_param_path=None, device_config=cuda, decoding_config=DecodingConfig(guided_decoding_backend='outlines'), observability_config=ObservabilityConfig(otlp_traces_endpoint=None, collect_model_forward_time=False, collect_model_execute_time=False), seed=0, served_model_name=/root/.cache/huggingface/hub/models--Qwen--Qwen2-7B-Instruct-GGUF/snapshots/7c1879f2983b48bb6a5609f7546299b833d25d13/qwen2-7b-instruct-q2_k.gguf, use_v2_block_manager=False, enable_prefix_caching=False)
/opt/conda/envs/vllm/lib/python3.10/site-packages/transformers/tokenization_utils_base.py:1601: FutureWarning: `clean_up_tokenization_spaces` was not set. It will be set to `True` by default. This behavior will be deprecated in transformers v4.45, and will be then set to `False` by default. For more details check this issue: https://github.com/huggingface/transformers/issues/31884
warnings.warn(
INFO 08-20 13:20:51 selector.py:217] Cannot use FlashAttention-2 backend for Volta and Turing GPUs.
INFO 08-20 13:20:51 selector.py:116] Using XFormers backend.
/opt/conda/envs/vllm/lib/python3.10/site-packages/xformers/ops/fmha/flash.py:211: FutureWarning: `torch.library.impl_abstract` was renamed to `torch.library.register_fake`. Please use that instead; we will remove `torch.library.impl_abstract` in a future version of PyTorch.
@torch.library.impl_abstract("xformers_flash::flash_fwd")
/opt/conda/envs/vllm/lib/python3.10/site-packages/xformers/ops/fmha/flash.py:344: FutureWarning: `torch.library.impl_abstract` was renamed to `torch.library.register_fake`. Please use that instead; we will remove `torch.library.impl_abstract` in a future version of PyTorch.
@torch.library.impl_abstract("xformers_flash::flash_bwd")
INFO 08-20 13:20:53 model_runner.py:889] Starting to load model /root/.cache/huggingface/hub/models--Qwen--Qwen2-7B-Instruct-GGUF/snapshots/7c1879f2983b48bb6a5609f7546299b833d25d13/qwen2-7b-instruct-q2_k.gguf...
INFO 08-20 13:21:11 selector.py:217] Cannot use FlashAttention-2 backend for Volta and Turing GPUs.
INFO 08-20 13:21:11 selector.py:116] Using XFormers backend.
INFO 08-20 13:21:30 model_runner.py:900] Loading model weights took 2.9129 GB
INFO 08-20 13:27:13 gpu_executor.py:113] # GPU blocks: 2847, # CPU blocks: 4681
Processed prompts: 100%|████████████████████████████████████████| 8/8 [00:15<00:00, 1.97s/it, est. speed input: 24.03 toks/s, output: 60.25 toks/s]
Prompt: '<|system|>\nYou are a friendly assistant chatbot.</s>\n<|user|>\nvLLM is a high-throughput and memory-efficient inference and serving engine for LLMs.\n</s>\n<|assistant|>\n', Generated text: "Yes, that's correct! VLLM stands for Vectorized Large Language Model and it's designed specifically for inference and serving tasks involving Large Language Models (LLMs). It aims to provide high throughput and memory efficiency by leveraging vectorized operations and optimized memory management techniques.\n\nInference engines like VLLM are crucial for applications that require processing large amounts of text data quickly and efficiently, such as in natural language processing tasks like text generation, question answering, or sentiment analysis. By optimizing these tasks, VLLM can help improve the performance of AI systems deployed in various industries including tech companies, research institutions, and more.\n\nThe vectorized"
Prompt: '<|system|>\nYou are a friendly assistant chatbot.</s>\n<|user|>\nBriefly describe the major milestones in the development of artificial intelligence from 1950 to 2020.\n</s>\n<|assistant|>\n', Generated text: "Artificial Intelligence (AI) development has seen significant milestones since its inception in the mid-twentieth century. Here are some major milestones:\n\n### Early Milestones (1950s)\n\n- **Origins**: AI was conceptualized around the mid-1950s with the creation of the first AI program by Alan Turing himself.\n\n### 1960s Milestones\n\n- **Early AI Programs**: Programs like the Logic Theorist were developed which could prove mathematical theorems using symbolic logic.\n\n### Late 1960s Milestones\n\n- **Early AI Failures**: AI's first major setback" |
My original command for running the program was as follows: I found that the error occurred when I used --cpu_offload_gb 80. When I removed --cpu_offload_gb 80, it ran normally: Cool! Thank you~ |
I've found that the official qwen2-72b-instruct-q2_k.gguf model in ModelScope still throws an error with the assertion |
|
I found that the error was caused by my transformers code not being updated to the latest version, which made the official modelscope model unable to be used. After updating, it worked just fine. 😂 |
|
run sucess,but template how to create? BadRequestError: Error code: 400 - {'object': 'error', 'message': 'As of transformers v4.44, default chat template is no longer allowed, so you must provide a chat template if the tokenizer does not define one.', 'type': 'BadRequestError', 'param': None, 'code': 400} |
1 task
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Your current environment
How would you like to use vllm
When I ran inference of a qwen2-72b-instruct-q2_k.gguf. I got an error，I don't know how to deal with:
The text was updated successfully, but these errors were encountered: