Replies: 5 comments
|
Hi, is it possible to use with the AtomicQuant GGUF? Also, isnt lazy-load doing this? I need to try it further. |
Not at this moment, I would need to adapt the branch. I guess I over engineering by having the manual split and separate arguments.
Yes, just found out about Seems like tensor-read-lazy was merged yesterday, which is why I missed it (my branch had a previous base than unsloth's PR) . |
|
I'm following this work mostly as an interested local-inference user, not as someone familiar with llama.cpp internals. I should also mention that I used an AI assistant to help me research the related issues/PRs and to understand some of the implementation details and terminology below. The underlying question came from thinking about how local inference behaves on machines with limited VRAM, but some of the technical context and references were gathered with AI assistance. So please take this as a question / possible direction rather than a proposed design. After reading this discussion and the recent related work, I wondered whether there might eventually be value in going one step beyond lazy reads + readahead and looking at access-frequency-aware residency for PLE/ngram tables. What led me to the thought:
These are obviously different kinds of tensors and execution paths, but together they made me wonder whether PLE tables might benefit from a similar working-set view: with the existing lazy-read/readahead path handling misses. I'm not assuming that this would actually be beneficial. The OS page cache may already capture most useful locality, explicit caching could duplicate that work, PCIe transfers or synchronization could cost more than they save, and the PLE access distribution might simply be too flat for a useful small hot set to exist. So perhaps the first useful experiment would not be an implementation at all, but simply instrumentation:
If, for example, a relatively small fraction of PLE pages accounts for a large fraction of repeated accesses, then a small explicit VRAM cache could potentially complement readahead: readahead hides predictable cold misses, while the cache avoids repeatedly fetching genuinely hot entries at all. Conversely, if the distribution is close to uniform or reuse is too low, the measurements would kill the idea cheaply and confirm that lazy mmap + page cache + readahead is already the right architecture. Longer term, if this did show useful locality, I wonder whether the interesting abstraction would be less "PLE caching" specifically and more a small tiered residency mechanism for sparse/indexed model data, with different policies for things such as PLE rows and MoE experts. But I realise that is a much larger architectural question and probably premature here. Mostly posting this because the recent PLE lazy-loading work and the MoE expert-cache experiments seem to be approaching a similar memory-hierarchy problem from two directions, and I was curious whether anyone has already measured the PLE access distribution this way. Apologies if I'm overlooking an obvious constraint in the model loader/backend architecture. |
In my branch, I tried to push GPT to analyze the access patterns and it concluded that it could benefit from some kind of "smart" cache and tried to implement it. However, I did not see any difference from the naive fseek/fread whenever a row was required. I suspect filesystem paging is already smart enough to keep the most recently accessed pages in memory (I did not pursue this very much TBH as the performance of the simple implementation was good enough) |
Uh oh!
There was an error while loading. Please reload this page.
Since Qwen 3.8 Flash Next came out, I kept reading about the 51B n-gram part of the model which may be offloaded to RAM and even to SSD, and decided to do my own investigation on the topic. While I found that SSD offloading will have an impact (how much depends on hardware), it is still a practical option to significantly reduce memory requirements for running local single-user inference.
I wanted to share some benchmarks I ran from my branch, which also has other fixes and Metal optimizations. The idea might be worth exploring as a general mechanism in llama.cpp to offload the PLE as new LLM architectures implement this.
In the branch, the table is stored in a separate GGUF in order for offload to work in the simplest manner possible: Just fopen on the split GGUF, fseeko+fread to fetch the queried rows and dequantize with CPU. No caching is done other than filesystem page cache, but it might become more efficient with mmap. Could also work with the table merged with the main GGUF if it is stored continuously (though it could be tricky in the case of split GGUFs).
The flag to load the separate table is
--model-ngram, with--ngram-load-mode residentfor keeping the table in memory and--ngram-load-mode readto read the rows on demand. This is the benchmark forresidentmode, which keeps the table in-memory:And this is the same benchmark with
readmode, which loads rows on demand from SSD. Note the strangeness that sometimes a longer prefill can get a faster pp, possibly due to luck in Macos filesystem cache hits:In case someone wants to try it out, I've published split ngram GGUF here: https://huggingface.co/tarruda/Qwen3.8-Flash-Next-GGUF
All reactions