Default value for the number of threads

[avdosev]

The current default value is cpu_count/2:

llama-cpp-python/llama_cpp/llama.py

Line 102 in b2a24bd

self.n_threads = n_threads or max(multiprocessing.cpu_count() // 2, 1)

This value does not seem to be optimal for multicore systems. For example, a CPU with 8 cores will have 4 cores idle. Or to put it simply, we will get twice the slowdown (if there are no more nuances in model execution).

Related issues: #71

In this discussion I would like to know the motivation for such a default value, as it seems that it is not obvious to most users.

[gjmulder]

• Most physical systems are hyperthreaded
• Hyperthreading doesn't seem to improve performance due to the memory I/O bound nature of llama.cpp
• Might be invalid for VMs

[avdosev]

Hyperthreading doesn't seem to improve performance due to the memory I/O bound nature of llama.cpp

It's a bit counterintuitive for me. Hypertreading was created to fully utilize the CPU during memory bound programs.

If we are talking about limited CPU utilization on the VM, in my opinion, the library should not solve such case and should offer the most productive option for the base scenario.

In general, it is necessary to test the performance on different systems and choose the most performance default value.

[Priestru]

This was tested in original llama. C++ implementation already utilities full available compute of physical cores, increasing amount of threads beyond would only to lower performance due to reason mentioned above.

[avdosev]

Okay. If the systems with hypertrading receive reductions, then for them this logic can be left, but why for systems with all physical cores logic is not different?

[gjmulder]

Closing as it is impossible to support every config of hardware.

[kevintsq]

As of today, many CPU cores have removed hyperthreading. Using psutil.cpu_count(logical=False) instead of simply assuming all cores are hyperthreaded should be a better way to do this.

[serhii-nakon]

And on servers sometimes hyper-threading has 2-3 virtual cores