[qnn] How to lower the generation rate to reduce the bandwith/cpu usage?

[ecccccsgo]

Hi Teams,

if i depoly the model successfully, when decoding, the NPU will take amost all bandwith, like 30g/s, Which is unacceptable if other apps running on the device.

It seems we can use htp_backend_ext_config.json, to set the perf_profile to power_saver (try it but not work with qnn-2.39) in the QNN notebook:

{ 
  "graphs": [{
    "num_cores": 1,
    "O":3.0,
    "vtcm_mb":16
    }],
    "devices": [{
            "device_id": 0,
            "core_type": 0,
            "core_id":[0],
            "dsp_arch": "v68",
            "soc_id":39,
            "soc_model":39,    
            "cores":[{
        "perf_profile": "burst"
      }]            
    }]
}

so we hope to know how to limit the decode rate in executorch with qnn?

looking forward to your help :)

cc @cccclai @cbilgin @abhinaykukkadapu @winskuo-quic @shewu-quic @haowhsu-quic @DannyYuyang-quic

[abhinaykukkadapu]

@ecccccsgo can you try this when lowering the model.

htp_options.performance_mode = QnnExecuTorchHtpPerformanceMode.kHtpBalanced # (or kHtpPowerSaver)
htp_options.use_dlbc = True #AOT only flag                                   
htp_options.use_slc_allocator = True # AOT only flag

If you are using qnn_executor_runner, you can also use the runtime override for the performance mode: qnn_executor_runner --model_path model.pte --htp_performance_mode 4

htp_backend_ext_config seems to be for native QNN path and not for Executorch QNN. CC: @shewu-quic @haowhsu-quic

[ecccccsgo]

Hello @abhinaykukkadapu , Sorry for late. I did find it in qnn_executor_runner:

-htp_performance_mode (HTP Performance mode between 0-8. This is a runtime
      option and will override the performance mode set during AOT. Refer to
      QnnExecuTorchHtpPerformanceMode under qc_compiler_spec.fbs for more
      info.) type: int32 default: 0

but it not present in ./qnn_llama_runner --help and ./qnn_multimodal_runner -help.

Maybe I can try to modify the cpp code and add this option, taking qnn_executor_runner as an example?

[abhinaykukkadapu]

Hello @abhinaykukkadapu , Sorry for late. I did find it in qnn_executor_runner:

-htp_performance_mode (HTP Performance mode between 0-8. This is a runtime
option and will override the performance mode set during AOT. Refer to
QnnExecuTorchHtpPerformanceMode under qc_compiler_spec.fbs for more
info.) type: int32 default: 0
but it not present in ./qnn_llama_runner --help and ./qnn_multimodal_runner -help.

Maybe I can try to modify the cpp code and add this option, taking qnn_executor_runner as an example?

Thanks @ecccccsgo, will look forward for your contribution.

[abhinaykukkadapu]

CC: @shewu-quic @haowhsu-quic @winskuo-quic i think we should unify the cli harness to make it uniform for all the runners to have same options available, align llama runner and multimodal runner with qnn runner?

[ecccccsgo]

@abhinaykukkadapu Unfortunately, it not works via add same code in qnn_executor_runner (rebuild the cpp):

DEFINE_int32(
    htp_performance_mode,
    0,
    "HTP Performance mode between 0-8. "
    "This is a runtime option and will override the performance mode set during AOT. "
    "Refer to QnnExecuTorchHtpPerformanceMode under qc_compiler_spec.fbs for more info.");

...(some other code)...

and not works when compiling the model vim htp_options.performance_mode = QnnExecuTorchHtpPerformanceMode.kHtpLowPowerSaver as it was said.

did you have some good idea?

CC: @shewu-quic @haowhsu-quic @winskuo-quic

[winskuo-quic]

@abhinaykukkadapu Unfortunately, it not works via add same code in qnn_executor_runner (rebuild the cpp):

Hi @ecccccsgo,
In recent mainline, we have exposed setting performance mode, so you don't need to perform performance mode changing in runtime.
Reference:

executorch/examples/qualcomm/utils.py

Line 1066 in d93ef2c

parser.add_argument(

If you still would like to perform runtime update, you will need to add set_option logic in runner besides from adding the gflag.
Reference:

executorch/examples/qualcomm/executor_runner/qnn_executor_runner.cpp

Line 230 in d93ef2c

backend_options.set_option(

If there’s a strong use case for this, we can also add it to our roadmap and enable it officially for qnn_llama_runner.

[ecccccsgo]

Thanks @winskuo-quic , i actually add the code like this, and not works:

  executorch::runtime::BackendOptions<1> backend_options;
  if (!gflags::GetCommandLineFlagInfoOrDie("htp_performance_mode").is_default) {
    ET_LOG(
        Info,
        "Setting runtime performance mode: %d",
        FLAGS_htp_performance_mode);
    ET_CHECK_MSG(
        backend_options.set_option(
            QNN_RUNTIME_HTP_PERFORMANCE_MODE, FLAGS_htp_performance_mode) ==
            executorch::runtime::Error::Ok,
        "Failed to set backend options: %s",
        QNN_RUNTIME_HTP_PERFORMANCE_MODE);
    ET_CHECK_MSG(
        executorch::runtime::set_option(QNN_BACKEND, backend_options.view()) ==
            executorch::runtime::Error::Ok,
        "Failed to set runtime options.");
  }

As your suggestions, I will update my code and try it again.

[ecccccsgo]

Hello @winskuo-quic , maybe it not works via args override the AOT?

I test ./qnn_llama_runner --decoder_model_version qwen3 --tokenizer_path ./tokenizer.json --model_path hybrid_llama_qnn.pte --seq_len 1024 --prompt "write a paper about the Spring in Chinese." -htp_performance_mode 4

 "write a paper about the AI." -htp_performance_mode 4                                 <
I tokenizers:regex.cpp:27] Registering override fallback regex
I 00:00:00.000460 executorch:qnn_llama_runner.cpp:292] Setting runtime performance mode: 4
I 00:00:00.000578 executorch:runner.cpp:149] creating module: model_path=hybrid_llama_qnn.pte
I 00:00:00.000581 executorch:runner.cpp:150] creating runner: tokenizer_path=./tokenizer.json
I 00:00:00.000582 executorch:runner.cpp:151] eval mode=1
I tokenizers:hf_tokenizer.cpp:142] Setting up normalizer...
I tokenizers:normalizer.cpp:89] Using NFC normalizer. Please notice that our implementation may not handle all edge cases.
I tokenizers:hf_tokenizer.cpp:146] Normalizer set up
I tokenizers:hf_tokenizer.cpp:160] Setting up pretokenizer...
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
E0000 00:00:1776137985.457681   26234 re2.cc:237] Error parsing '((?i:'s|'t|'re|'ve|'m|'ll|'d)|[^\r\n\p{L}\p{N}]?\p{L}+|\p{N}| ?[^\s\p{L}\p{N}]+[\r\n]*|\s*[\r\n]+|\s...': invalid perl operator: (?!
I tokenizers:re2_regex.cpp:27] Re2 failed to compile regex: ((?i:'s|'t|'re|'ve|'m|'ll|'d)|[^\r\n\p{L}\p{N}]?\p{L}+|\p{N}| ?[^\s\p{L}\p{N}]+[\r\n]*|\s*[\r\n]+|\s+(?!\S)|\s+), error: invalid perl operator: (?!
This may be ok if a fallback regex is used.
I tokenizers:regex_lookahead.cpp:27] Creating PCRE2 regex
I tokenizers:hf_tokenizer.cpp:164] Pretokenizer set up
I tokenizers:hf_tokenizer.cpp:180] Loading BPE merges...
I tokenizers:hf_tokenizer.cpp:240] Loaded 151387 BPE merge rules
I tokenizers:hf_tokenizer.cpp:252] Built merge ranks map with 151387 entries
I 00:00:01.267699 executorch:llm_runner_helper.cpp:54] Loaded json tokenizer
I tokenizers:hf_tokenizer.cpp:415] normalized input: '' -> ''
[INFO] [Qnn ExecuTorch]: Deserializing processed data using QnnContextCustomProtocol
[INFO] [Qnn ExecuTorch]: Creating new backend bundle.
[INFO] [Qnn ExecuTorch]: create QNN Logger with log_level 1
[INFO] [Qnn ExecuTorch]: Initialize Qnn backend parameters for Qnn executorch backend type 2
[INFO] [Qnn ExecuTorch]: Caching: Caching is in RESTORE MODE.
[INFO] [Qnn ExecuTorch]: QnnContextCustomProtocol expected magic number: 0x5678abcd but get: 0x2000000
[INFO] [Qnn ExecuTorch]: Running level=1 optimization.
[INFO] [Qnn ExecuTorch]: Running level=1 optimization.
[INFO] [Qnn ExecuTorch]: Deserializing processed data using QnnContextCustomProtocol
[INFO] [Qnn ExecuTorch]: Use cached delegate handle for current method: kv_forward
I 00:00:02.051937 executorch:runner.cpp:230] Reading metadata from model
I 00:00:02.051987 executorch:runner.cpp:351] creating io_memory
I tokenizers:hf_tokenizer.cpp:415] normalized input: '' -> ''
I tokenizers:hf_tokenizer.cpp:415] normalized input: 'user
write a paper about the Spring in Chinese.' -> 'user
write a paper about the Spring in Chinese.'
I tokenizers:hf_tokenizer.cpp:415] normalized input: '
' -> '
'
I tokenizers:hf_tokenizer.cpp:415] normalized input: 'assistant' -> 'assistant'
I 00:00:02.086509 executorch:prompt_processor.cpp:267] Prompt Processor: total 17 prompt tokens (AR-128 * 1 iters)
I 00:00:02.146458 executorch:runner.cpp:462] RSS after prompt prefill: 1050.937500 MiB (0 if unsupported)

I 00:00:28.636681 executorch:token_generator.cpp:345] 
Reached to the end of generation
I 00:00:28.636689 executorch:runner.cpp:477] RSS after finishing text generation: 1050.937500 MiB (0 if unsupported)
PyTorchObserver {"prefill_token_per_sec":274.194,"decode_token_per_sec":37.5613,"prompt_tokens":17,"generated_tokens":995,"model_load_start_ms":1776137984860,"model_load_end_ms":1776137986944,"inference_start_ms":1776137986944,"inference_end_ms":1776138013496,"prompt_eval_end_ms":1776137987006,"first_token_ms":1776137987006,"aggregate_sampling_time_ms":338,"SCALING_FACTOR_UNITS_PER_SECOND":1000}
I 00:00:28.636738 executorch:stats.h:161] 	Prompt Tokens: 17    Generated Tokens: 995
I 00:00:28.636741 executorch:stats.h:167] 	Model Load Time:		2.084000 (seconds)
I 00:00:28.636750 executorch:stats.h:177] 	Total inference time:		26.552000 (seconds)	 Rate: 	37.473637 (tokens/second)
I 00:00:28.636765 executorch:stats.h:185] 		Prompt evaluation:	0.062000 (seconds)	 Rate: 	274.193548 (tokens/second)
I 00:00:28.636777 executorch:stats.h:196] 		Generated 995 tokens:	26.490000 (seconds)	 Rate: 	37.561344 (tokens/second)
I 00:00:28.636782 executorch:stats.h:204] 	Time to first generated token:	0.062000 (seconds)
I 00:00:28.636784 executorch:stats.h:211] 	Sampling time over 1012 tokens:	0.338000 (seconds)
[INFO] [Qnn ExecuTorch]: Destroy Qnn context
[INFO] [Qnn ExecuTorch]: Destroy Qnn device
[INFO] [Qnn ExecuTorch]: Destroy Qnn backend
/qnn_llama_runner --decoder_model_version qwen3 --tokenizer_path ./tokenizer.json --model_path hybrid_*

and
./qnn_llama_runner --decoder_model_version qwen3 --tokenizer_path ./tokenizer.json --model_path hybrid_llama_qnn.pte --seq_len 1024 --prompt "write a paper about the Spring in Chinese." -htp_performance_mode 0

 "write a paper about the Spring in Chinese." -htp_performance_mode 0                                 <
I tokenizers:regex.cpp:27] Registering override fallback regex
I 00:00:00.000509 executorch:qnn_llama_runner.cpp:292] Setting runtime performance mode: 0
I 00:00:00.000647 executorch:runner.cpp:149] creating module: model_path=hybrid_llama_qnn.pte
I 00:00:00.000665 executorch:runner.cpp:150] creating runner: tokenizer_path=./tokenizer.json
I 00:00:00.000668 executorch:runner.cpp:151] eval mode=1
I tokenizers:hf_tokenizer.cpp:142] Setting up normalizer...
I tokenizers:normalizer.cpp:89] Using NFC normalizer. Please notice that our implementation may not handle all edge cases.
I tokenizers:hf_tokenizer.cpp:146] Normalizer set up
I tokenizers:hf_tokenizer.cpp:160] Setting up pretokenizer...
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
E0000 00:00:1776137880.539710   23368 re2.cc:237] Error parsing '((?i:'s|'t|'re|'ve|'m|'ll|'d)|[^\r\n\p{L}\p{N}]?\p{L}+|\p{N}| ?[^\s\p{L}\p{N}]+[\r\n]*|\s*[\r\n]+|\s...': invalid perl operator: (?!
I tokenizers:re2_regex.cpp:27] Re2 failed to compile regex: ((?i:'s|'t|'re|'ve|'m|'ll|'d)|[^\r\n\p{L}\p{N}]?\p{L}+|\p{N}| ?[^\s\p{L}\p{N}]+[\r\n]*|\s*[\r\n]+|\s+(?!\S)|\s+), error: invalid perl operator: (?!
This may be ok if a fallback regex is used.
I tokenizers:regex_lookahead.cpp:27] Creating PCRE2 regex
I tokenizers:hf_tokenizer.cpp:164] Pretokenizer set up
I tokenizers:hf_tokenizer.cpp:180] Loading BPE merges...
I tokenizers:hf_tokenizer.cpp:240] Loaded 151387 BPE merge rules
I tokenizers:hf_tokenizer.cpp:252] Built merge ranks map with 151387 entries
I 00:00:01.212074 executorch:llm_runner_helper.cpp:54] Loaded json tokenizer
I tokenizers:hf_tokenizer.cpp:415] normalized input: '' -> ''
[INFO] [Qnn ExecuTorch]: Deserializing processed data using QnnContextCustomProtocol
[INFO] [Qnn ExecuTorch]: Creating new backend bundle.
[INFO] [Qnn ExecuTorch]: create QNN Logger with log_level 1
[INFO] [Qnn ExecuTorch]: Initialize Qnn backend parameters for Qnn executorch backend type 2
[INFO] [Qnn ExecuTorch]: Caching: Caching is in RESTORE MODE.
[INFO] [Qnn ExecuTorch]: QnnContextCustomProtocol expected magic number: 0x5678abcd but get: 0x2000000
[INFO] [Qnn ExecuTorch]: Running level=1 optimization.
[INFO] [Qnn ExecuTorch]: Running level=1 optimization.
[INFO] [Qnn ExecuTorch]: Deserializing processed data using QnnContextCustomProtocol
[INFO] [Qnn ExecuTorch]: Use cached delegate handle for current method: kv_forward
I 00:00:01.953622 executorch:runner.cpp:230] Reading metadata from model
I 00:00:01.953728 executorch:runner.cpp:351] creating io_memory
I tokenizers:hf_tokenizer.cpp:415] normalized input: '' -> ''
I tokenizers:hf_tokenizer.cpp:415] normalized input: 'user
write a paper about the Spring in Chinese.' -> 'user
write a paper about the Spring in Chinese.'
I tokenizers:hf_tokenizer.cpp:415] normalized input: '
' -> '
'
I tokenizers:hf_tokenizer.cpp:415] normalized input: 'assistant' -> 'assistant'
I 00:00:01.988834 executorch:prompt_processor.cpp:267] Prompt Processor: total 17 prompt tokens (AR-128 * 1 iters)
I 00:00:02.055443 executorch:runner.cpp:462] RSS after prompt prefill: 1050.972656 MiB (0 if unsupported)

I 00:00:28.613930 executorch:token_generator.cpp:345] 
Reached to the end of generation
I 00:00:28.614006 executorch:runner.cpp:477] RSS after finishing text generation: 1050.972656 MiB (0 if unsupported)
PyTorchObserver {"prefill_token_per_sec":246.377,"decode_token_per_sec":37.4652,"prompt_tokens":17,"generated_tokens":995,"model_load_start_ms":1776137879957,"model_load_end_ms":1776137881943,"inference_start_ms":1776137881943,"inference_end_ms":1776137908570,"prompt_eval_end_ms":1776137882012,"first_token_ms":1776137882012,"aggregate_sampling_time_ms":333,"SCALING_FACTOR_UNITS_PER_SECOND":1000}
I 00:00:28.614036 executorch:stats.h:161] 	Prompt Tokens: 17    Generated Tokens: 995
I 00:00:28.614039 executorch:stats.h:167] 	Model Load Time:		1.986000 (seconds)
I 00:00:28.614042 executorch:stats.h:177] 	Total inference time:		26.627000 (seconds)	 Rate: 	37.368085 (tokens/second)
I 00:00:28.614047 executorch:stats.h:185] 		Prompt evaluation:	0.069000 (seconds)	 Rate: 	246.376812 (tokens/second)
I 00:00:28.614050 executorch:stats.h:196] 		Generated 995 tokens:	26.558000 (seconds)	 Rate: 	37.465171 (tokens/second)
I 00:00:28.614053 executorch:stats.h:204] 	Time to first generated token:	0.069000 (seconds)
I 00:00:28.614056 executorch:stats.h:211] 	Sampling time over 1012 tokens:	0.333000 (seconds)
[INFO] [Qnn ExecuTorch]: Destroy Qnn context
[INFO] [Qnn ExecuTorch]: Destroy Qnn device
[INFO] [Qnn ExecuTorch]: Destroy Qnn backend

the decode rate almost the same.

[winskuo-quic]

Hello @winskuo-quic , maybe it not works via args override the AOT?

I test ./qnn_llama_runner --decoder_model_version qwen3 --tokenizer_path ./tokenizer.json --model_path hybrid_llama_qnn.pte --seq_len 1024 --prompt "write a paper about the Spring in Chinese." -htp_performance_mode 4

@ecccccsgo Could you do a quick verification by adding a log message under:

executorch/backends/qualcomm/runtime/backends/htp/HtpDevice.cpp

Line 161 in 7e099b4

case QnnExecuTorchHtpPerformanceMode::kHtpPowerSaver:

.
And do backends/qualcomm/scripts/build.sh --no_clean.

If it does goes in this case, that means PowerSaver is properly set. If not, could you create a PR and I can try to reproduce and see how to resolve the issue.

[ecccccsgo]

@winskuo-quic Yes it goes to into this case:

for other cases, the decode speed not change:

[winskuo-quic]

In this case, it should be setting to QNN SDK.
If you want to double check, you can also set --log_level 5 with another runtime option:

executorch/examples/qualcomm/executor_runner/qnn_executor_runner.cpp

Line 219 in 37b12c8

backend_options.set_option(QNN_RUNTIME_LOG_LEVEL, FLAGS_log_level) ==

Please also ensure executorch option num from 1 to 2., like: runtime::BackendOptions<2> backend_options;
With --htp_performance_mode 4 --log_level 5, you should be able to see verbose messages from QNN SDK. You can see configs that aligns with values given in

executorch/backends/qualcomm/runtime/backends/htp/HtpDevice.cpp

Line 161 in 7e099b4

case QnnExecuTorchHtpPerformanceMode::kHtpPowerSaver:

.

For example, for power saver, you should see coreVoltageCornerMin 64, like following:

If all above aligns, could you profile again to see if decoding is taking all NPU bandwidth?
Thanks

[ecccccsgo]

Yes, I found coreVoltageCornerMin 64, and the generation rate almost the same, but with different bandwith:
--htp_performance_mode=0

++++++++++++++++++++++++++++++++++++++++++++++++++SUMMARY++++++++++++++++++++++++++++++++++++++++++++++++++

                             Metric  |                 Min  |                 Avg  |                 Max  |
            CDSP 128B_Read_BW(MB/s)  |               86.00  |              193.55  |              223.00  |
           CDSP 128B_Write_BW(MB/s)  |               42.00  |               52.45  |               80.00  |
            CDSP 256B_Read_BW(MB/s)  |                0.00  |            15826.45  |            20483.00  |
           CDSP 256B_Write_BW(MB/s)  |                0.00  |                3.45  |               38.00  |
             CDSP 64B_Read_BW(MB/s)  |                0.00  |                0.00  |                0.00  |
            CDSP 64B_Write_BW(MB/s)  |                0.00  |                0.00  |                0.00  |
                CDSP Frequency(KHz)  |                1.00  |          1013488.38  |          1236165.00  |
                   CDSP HMX_Load(%)  |                0.00  |               11.00  |               13.98  |
                   CDSP HVX_Load(%)  |                0.00  |               17.49  |               22.44  |
                       CDSP Load(%)  |               27.53  |               44.10  |               48.72  |
                CDSP Total_BW(MB/s)  |              128.00  |            16075.91  |            20756.00  |
           CDSP Total_Read_BW(MB/s)  |               86.00  |            16020.00  |            20706.00  |
          CDSP Total_Write_BW(MB/s)  |               42.00  |               55.91  |              118.00  |

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

[INFO] [Qnn ExecuTorch]: QnnDsp <V> sleepLatency 1000

[INFO] [Qnn ExecuTorch]: QnnDsp <V> setSleepDisable 0

[INFO] [Qnn ExecuTorch]: QnnDsp <V> sleepDisable 0

[INFO] [Qnn ExecuTorch]: QnnDsp <V> setBusParams 1

[INFO] [Qnn ExecuTorch]: QnnDsp <V> busVoltageCornerMin 64

[INFO] [Qnn ExecuTorch]: QnnDsp <V> busVoltageCornerTarget 64

--htp_performance_mode=4

++++++++++++++++++++++++++++++++++++++++++++++++++SUMMARY++++++++++++++++++++++++++++++++++++++++++++++++++

                             Metric  |                 Min  |                 Avg  |                 Max  |
            CDSP 128B_Read_BW(MB/s)  |               34.00  |              183.64  |              219.00  |
           CDSP 128B_Write_BW(MB/s)  |               16.00  |               48.27  |               88.00  |
            CDSP 256B_Read_BW(MB/s)  |                0.00  |            15979.00  |            20406.00  |
           CDSP 256B_Write_BW(MB/s)  |                0.00  |                3.45  |               38.00  |
             CDSP 64B_Read_BW(MB/s)  |                0.00  |                0.00  |                0.00  |
            CDSP 64B_Write_BW(MB/s)  |                0.00  |                0.00  |                0.00  |
                CDSP Frequency(KHz)  |           902400.00  |          1099306.75  |          1191397.00  |
                   CDSP HMX_Load(%)  |                0.00  |               11.11  |               14.00  |
                   CDSP HVX_Load(%)  |                0.00  |               17.68  |               22.40  |
                       CDSP Load(%)  |               10.74  |               42.05  |               48.76  |
                CDSP Total_BW(MB/s)  |               50.00  |            16214.36  |            20675.00  |
           CDSP Total_Read_BW(MB/s)  |               34.00  |            16162.64  |            20625.00  |
          CDSP Total_Write_BW(MB/s)  |               16.00  |               51.73  |              126.00  |

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

[INFO] [Qnn ExecuTorch]: QnnDsp <V> sleepLatency 2000

[INFO] [Qnn ExecuTorch]: QnnDsp <V> setSleepDisable 0

[INFO] [Qnn ExecuTorch]: QnnDsp <V> sleepDisable 0

[INFO] [Qnn ExecuTorch]: QnnDsp <V> setBusParams 1

[INFO] [Qnn ExecuTorch]: QnnDsp <V> busVoltageCornerMin 48

[INFO] [Qnn ExecuTorch]: QnnDsp <V> busVoltageCornerTarget 48

[INFO] [Qnn ExecuTorch]: QnnDsp <V> busVoltageCornerMax 48

the verbose log seems not same, but the TPS are the same.
Maybe my devices are almost no other app running, and it will take almost all badwith in low power mode?