[Bug]: [benchmark_app] : Can not create StaticMemory object. The memory desc is undefined

[HPUedCSLearner]

OpenVINO Version

openvino24.1

Operating System

Ubuntu 20.04 (LTS)

Device used for inference

CPU

Framework

None

Model used

LLM : qwen

Issue description

benchmark_app -m ../Qwen1.5-1.8B-Chat-OpenVINO-int4/openvino_model.xml -d CPU -shape "input_ids[1,786],attention_mask[1,786],position_ids[1,786],beam_idx[1]"

Step-by-step reproduction

benchmark_app -m ../Qwen1.5-1.8B-Chat-OpenVINO-int4/openvino_model.xml -d CPU -shape "input_ids[1,786],attention_mask[1,786],position_ids[1,786],beam_idx[1]"

Relevant log output

(openvino24.1) yongshuai_wang@ubuntu-DeepLearning-4801031:/juicefs-algorithm/workspace/vision/yongshuai_wang/models/hf-models$ benchmark_app -m ../Qwen1.5-1.8B-Chat-OpenVINO-int4/openvino_model.xml -d CPU  -shape "input_ids[1,786],attention_mask[1,786],position_ids[1,786],beam_idx[1]"  

[Step 1/11] Parsing and validating input arguments
[ INFO ] Parsing input parameters
[Step 2/11] Loading OpenVINO Runtime
[ INFO ] OpenVINO:
[ INFO ] Build ................................. 2024.1.0-33-20d7ce923d1
[ INFO ] 
[ INFO ] Device info:
[ INFO ] CPU
[ INFO ] Build ................................. 2024.1.0-33-20d7ce923d1
[ INFO ] 
[ INFO ] 
[Step 3/11] Setting device configuration
[ WARNING ] Performance hint was not explicitly specified in command line. Device(CPU) performance hint will be set to PerformanceMode.THROUGHPUT.
[Step 4/11] Reading model files
[ INFO ] Loading model files
[ INFO ] Read model took 131.17 ms
[ INFO ] Original model I/O parameters:
[ INFO ] Model inputs:
[ INFO ]     input_ids (node: input_ids) : i64 / [...] / [?,?]
[ INFO ]     attention_mask (node: attention_mask) : i64 / [...] / [?,?]
[ INFO ]     position_ids (node: position_ids) : i64 / [...] / [?,?]
[ INFO ]     beam_idx (node: beam_idx) : i32 / [...] / [?]
[ INFO ] Model outputs:
[ INFO ]     logits (node: aten::to/Convert) : f32 / [...] / [?,?,151936]
[Step 5/11] Resizing model to match image sizes and given batch
[ INFO ] Model batch size: 1
[ INFO ] Reshaping model: 'input_ids': [1,786], 'attention_mask': [1,786], 'position_ids': [1,786], 'beam_idx': [1]
[ INFO ] Reshape model took 318.09 ms
[Step 6/11] Configuring input of the model
[ INFO ] Model inputs:
[ INFO ]     input_ids (node: input_ids) : i64 / [...] / [1,786]
[ INFO ]     attention_mask (node: attention_mask) : i64 / [...] / [1,786]
[ INFO ]     position_ids (node: position_ids) : i64 / [...] / [1,786]
[ INFO ]     beam_idx (node: beam_idx) : i32 / [...] / [1]
[ INFO ] Model outputs:
[ INFO ]     logits (node: aten::to/Convert) : f32 / [...] / [1,786,151936]
[Step 7/11] Loading the model to the device
[ ERROR ] Exception from src/inference/src/cpp/core.cpp:106:
Exception from src/inference/src/dev/plugin.cpp:54:
Exception from src/plugins/intel_cpu/src/cpu_memory.cpp:475:
Can not create StaticMemory object. The memory desc is undefined


Traceback (most recent call last):
  File "/data/yongshuai_wang/miniconda3/envs/openvino24.1/lib/python3.10/site-packages/openvino/tools/benchmark/main.py", line 408, in main
    compiled_model = benchmark.core.compile_model(model, benchmark.device, device_config)
  File "/data/yongshuai_wang/miniconda3/envs/openvino24.1/lib/python3.10/site-packages/openvino/runtime/ie_api.py", line 521, in compile_model
    super().compile_model(model, device_name, {} if config is None else config),
RuntimeError: Exception from src/inference/src/cpp/core.cpp:106:
Exception from src/inference/src/dev/plugin.cpp:54:
Exception from src/plugins/intel_cpu/src/cpu_memory.cpp:475:
Can not create StaticMemory object. The memory desc is undefined

Issue submission checklist

• I'm reporting an issue. It's not a question.
• I checked the problem with the documentation, FAQ, open issues, Stack Overflow, etc., and have not found a solution.
• There is reproducer code and related data files such as images, videos, models, etc.

[HPUedCSLearner]

This problem has troubled me for a long time. I have tried many versions of OpenVINO, including building from source code, installing with pip, and installing with apt, and all of them report the same error.
I would like to know how to use OpenVINO's benchmark_app tool to perform performance tests on current mainstream large models such as qwen, chatglm, etc.
Is it that benchmark_app testing of large models is not yet supported?
Please let me know, thank you very much!

[dmitry-gorokhov]

Hi @HPUedCSLearner!
We recommend to use dedicated llm_bench for these purposes. It is the most convinient way to benchmark LLM workloads with OpenVINO runtime.
Benchmark_app is mostly dedicated to profile "traditional" models and might lack some LLM related functionality.

[HPUedCSLearner]

Thank you for your suggestion, I will try your suggestion next.