server: comment --threads option behavior

[phymbert]

As we are using batching, I am wondering what is the purpose of --threads N parameter in the server.

Should we remove it ?

[ggerganov]

It's used for bs = 1 decoding:

llama.cpp/llama.cpp

Line 9082 in b2075fd

int n_threads = n_tokens == 1 ? cparams.n_threads : cparams.n_threads_batch;

[phymbert]

OK, later I will add a this precision in the readme, also specifying this is not used at all when using --ngl.

[Jeximo]

As we are using batching

What about those who don't use batching?

[phymbert]

As we are using batching

What about those who don't use batching?

The server is using batching by design