Repository navigation
Replies: 1 comment
|
Short version: with the default What each flag does (from
Practical way to get "weights mostly on one card, other card does something else": use On your general questions: the code gives you the mechanisms above but no built-in cost model, and I have not benchmarked your two-GPU setup, so I can't tell you which split is fastest. With the default layer split the GPUs run one after the other (pipelined) for a single sequence, so a slow card does not run in parallel with the fast one, it adds its share of time. That is why giving the slow card a smaller share than its VRAM would suggest is often worth testing. Measure with As for new flags or treating RAM and mmap as devices, that is a design question for the maintainers and not something the current code does, so it would be a feature request. (I looked this up in the code with help from an AI assistant; the links are to the lines I checked.) |
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
I have questions regarding llama.cpp and the various data-structures involved. Allow me to set the stage for the questions.
Hardware:
One Turing card with 6GB of memory, and an Ampere card with 24GB. The 6GB Turing card is slow (compute/memory) for a GPU, still faster than CPU/RAM. Which in turn is faster than NVME, of course.
One:
llama.cpp offers some knobs to put mmproj and MTP heads on specific devices.
Two:
These flags do not appear to take effect unless one also list the devices in question after
--deviceThree:
There does not appear to be a way to explicitly state that the model weights only should live on a subset of devices.
--deviceis just a list of devices llama.cpp can make use of.--tensor-splitand--split-modeappear to apply equally to 'everything'?This is based on my own tests today. (If you think this is the result of me having fumbled, feel free to let me know the correct way to fire up llama.cpp.)
Four:
I do not have any 'instinct' for where the various datastructures should live in a heterogenous system (different Nvidia GPUs) or a hybrid system (utilizing GPU/CPU/NVME), for the optimal performance.
Would I be better off with all the model weights on the 3090 and kv-cache split on the RTX 3090 and the RTX 3000? Or would it be better to have the kv-cache on the 3090 and the model weights distributed? Similar questions for MTP and mmproj weights. Or is the current strategy actually the best one? With multiple GPUs, there is potentially also an element of parallelism. One does not want one piece of hardware to go idle because another piece is slow.
In short, I assume there is some hierarchy of these datastructures, in terms of size, how often they are accessed and how much data is read/written.
--tensor-splitdoes not seem to care about any of that?This hierarchy should in my mind impact on what device(s) these structures are located. Am I wrong?
Finally, the questions at the front of my mind:
fit off)RAMandMMAPbe made 'devices', to simplify the configuration syntax?All reactions