-ngl to load ·last n layers· to gpu

[Bobchenyx]

Hi there,

I would like to know if we could use -ngl to load last N layers to GPU instead of first N.

If possible can someone please point me to a place where I should modify the source code?

llama-bench for example.

[Bobchenyx]

take MoE models as an example. I would like to load first M Dense layers to CPU, then rest N MoE layers to GPU

[foldl]

My proposal: update -ngl to something like this:

https://github.com/foldl/chatllm.cpp/blob/master/docs/gpu.md#usage

[github-actions]

This issue was closed because it has been inactive for 14 days since being marked as stale.