Is the code multithreaded for arm64?

[RuABraun]

I'm using QNNPACK for inference for my models, I was testing it on a v7a phone where it worked well. Now I got a newer phone (S9) and while it sometimes is a lot faster (I think there is some CPU throttling happening at times which can make it very slow), I noticed in the profiler that the CPU is at 80%, compared to ~35% on the older phone. I cannot explain how that could happen other than the code is doing multithreading?

[hlu1]

It depends on your models and the number of big cores in your phone. We do have specially optimized kernels in QNNPACK for high end ARM CPUs.

[RuABraun]

Is it possible to turn it off? Despite the higher CPU usage (implying better performance) it's actually slower most of the time than when running on a significantly older phone (this is probably not the code's fault, but to try and diagnose what is going wrong it would helpful for me to test that option).

[hlu1]

That kernel runs only on high end CPUs. The default kernel is used on older phones.

[RuABraun]

? Like I said on the newer phone it is slower than on the older phone.

it's actually slower most of the time than when running on a significantly older phone

[RuABraun]

Nevermind I tried a model without QNNPACK and still have the same issues.

[RuABraun]

I'm curious about this again. I'd rather have the other cores working on other tasks. Is there a flag to disable using multiple cores?

[AshkanAliabadi]

As far as I can tell if the threadpool argument to qnnp_run_operator is NULL, the dispatch calls reduce to a sequential loop.