QNN runtime fails to load all context binaries when model contains >50 context binaries

[goka-wu]

Description

I am currently deploying the Llama-3.2-1B-Instruct model on QCS8550. I customized the QNN partitioner to offload Linear operations to the CPU, while delegating all non-Linear operations to the QNN Backend (HtpV73). This partitioning resulted in 66 QNN backend subgraphs, consequently generating 66 context binaries serialized within the resulting *.pte file.

However, upon execution on the QCS8550, the process failed. The error log indicates an issue during context loading.

I have attached the detailed logs for reference. Please review them.
err_20251009.log

The source code is from main branch with the following commit info：

commit cf6e895c53bd1052f1266821a76bd7c5a85ace52 (HEAD -> dev)
Author: Šimon Strýček <simon.strycek@nxp.com>
Date:   Tue Sep 16 11:33:01 2025 +0200

    NXP backend: Relocation of remove_io_quant_ops_pass.py (#14202)

    ### Summary
    Relocate `remove_io_quant_ops_pass.py` to `nxp/edge_passes`.

    ### Test plan
    Should be covered by already existing unit tests.

    Co-authored-by: Roman Janik <roman.janik@nxp.com>

Thank you for your time!

Analyze

(1) It appears that the QnnExecuTorchBackend first loads and manages all 66 Context Binaries upfront. It then executes the specific graph stored within a Context Binary as needed, and finally destroys all context binaries and other resources together.

(2) I believe the 66 context binaries are inherently valid because I successfully executed all of them sequentially (Load one Context Binary → Execute the model graph → Destroy the Context Binary ). The main modification for test is as belows:

Result<DelegateHandle*> QnnExecuTorchBackend::init(
    BackendInitContext& context,
    FreeableBuffer* processed,
    ArrayRef<CompileSpec> compile_specs) const {
    // convert CompileSpec to qnn ExecuTorch option
    for (auto& compile_spec : compile_specs) {
        if (std::strcmp(compile_spec.key, QNN_COMPILE_SPEC) == 0)
        qnn_executorch_options_ =
            GetQnnExecuTorchOptions(compile_spec.value.buffer);
        else
        QNN_EXECUTORCH_LOG_WARN("unknown argument: %s", compile_spec.key);
    }

  ET_LOG(Info, "WQnnExecuTorchBackend::init() is a dummpy function.");
  return processed;
}

Error QnnExecuTorchBackend::execute(
    BackendExecutionContext& context,
    DelegateHandle* handle,
    Span<EValue*> args) const {
  FreeableBuffer* processed = (FreeableBuffer*)(handle);
  QnnExecuTorchContextBinary qnn_context_blob;
  auto [status, signature, ctx_size, ctx_bin] =
      QnnContextCustomProtocol().DeserializeContextCustomBuffer(
          const_cast<void*>(processed->data()));
  if (status == Error::Ok) {
    QNN_EXECUTORCH_LOG_INFO(
        "Deserializing processed data using QnnContextCustomProtocol");
    // After this stage, qnn_context_blob.nbytes & qnn_context_blob.buffer will
    // only store qnn_context_binary.
    qnn_context_blob.nbytes = ctx_size;
    qnn_context_blob.buffer = ctx_bin;
    std::string file_name = "contexts/context_"  + std::to_string(backend_cnt_) + ".txt";
    write_file(file_name.c_str(), ctx_bin, ctx_size);
  } else {
    // This buffer will be verified again in QnnBackendCache.
    QNN_EXECUTORCH_LOG_INFO("Deserializing processed data using Dlc");
    qnn_context_blob.buffer = const_cast<void*>(processed->data());
    qnn_context_blob.nbytes = processed->size();
  }


  // Create QnnManager
  MemoryAllocator* runtime_allocator = context.get_temp_allocator();
  QnnManager* qnn_manager = runtime_allocator->allocateInstance<QnnManager>();
  if (qnn_manager == nullptr) {
    return Error::MemoryAllocationFailed;
  }
  // NOTE: Since we use placement new and since this type is not trivially
  // destructible, we must call the destructor manually in destroy().
  new (qnn_manager) QnnManager(qnn_executorch_options_, qnn_context_blob);
  // TODO: this is a temporal solution for multi-graph support, will be
  //       removed once framework starts to accept runtime configuration
  // ---
  // check if current context binary has already been initialized
  // return cached one for reducing memory footprint
  ET_CHECK_OR_RETURN_ERROR(
      qnn_manager->Init() == Error::Ok,
      Internal,
      "Fail to initialize Qnn Manager");
    
   ......

  ET_CHECK_OR_RETURN_ERROR(
      qnn_manager->Execute(
          method_name,
          input_tensor_structs,
          output_tensor_structs,
          context.event_tracer()) == Error::Ok,
      Internal,
      "Fail to execute graph");
  ET_CHECK_OR_RETURN_ERROR(
      qnn_manager->ProfileExecuteData(method_name, context.event_tracer()) ==
          Error::Ok,
      Internal,
      "Fail to profile graph");
  
 qnn_manager->Destroy();
  return Error::Ok;
}

cc @cccclai @winskuo-quic @shewu-quic @haowhsu-quic @DannyYuyang-quic @cbilgin

[goka-wu]

@cccclai Please take a look when you have a chance. Thanks a lot!

[cccclai]

cc: @haowhsu-quic mind taking a look

[haowhsu-quic]

Yes, will take some time to investigate.

[goka-wu]

Hello team, @haowhsu-quic
Could you please provide some suggestions on possible directions to investigate? Thanks a lot.

[haowhsu-quic]

Hi, I'm seeing part of log message saying not able to find available PD for certain context binary, I think that out of resource issue is not very easy to resolve. Could you try to reduce the partition number a bit to see if it still fails?
If I understand correctly, you're trying to do dynamic quantization with this approach. My suggestion would be just partitioning out those sensitive ones only instead of all of the linear layers: e.g. for llama3.2_1b_instruct, the down_proj, lm_head layers need more precision to preserve the accuracy.

[goka-wu]

It succeeds when the number of context binaries reaches 18. I attaches the success log below.

test_success.log

Actually, our team is planning to offload linear ops to our in-memory chips to get better performance. However, we unfortunately encountered an error. If you have any suggestions, I’d really appreciate your feedback.