airllm fork and some questions

[Sophist-UK]

I don't blame you for forking airllm which feels abandoned. Well done for taking the plunge and committing the time.

A few questions...

1. New models implement new inference techniques and e.g. llamacp gets updated to improve performance and handle these new model internals in a cleverer way.

   Qwen definitely has some of these internal techniques (dual attention if I remember). Does RabbitLLM include all similar updates and fully support and optimise these?

   Similarly MoE models limit what gets executed using a technique other than layers. Does RabbitLLM fully support and optimise MoE models?

2. Does it only support 70B models in 4GB vRAM or is it more flexible about model and vRAM size? (I have a 6GB a3000 and I desperately want to do local intense with a decent model with reasonable performance.)

3. Does it manage normal RAM to cache or pre-load shards ready to be loaded into vRAM? Having now read the code, I know that it starts to pre-fetch the next layer from disk, but e.g

   • If you have sufficient vRAM to hold 2 layers, can it have the next layer loaded and ready to run?
   • If you have sufficient memory to hold the entire model, can it simply hold the entire model in main memory to swap into the GPU as needed? I believe hardware for this use case is reasonably common and I don't believe it has ever been benchmarked. My laptop is capable of having 128GB of memory, so in theory I can hold the 127B parameter Qwen3-Coder in main memory and swap in layer by layer to my 6GB GPU.
   • If you have sufficient memory to hold (e.g.) 16 of 20 layers, could it hold 1-4, 6-9, 11-14, and 16-19 in memory and read only 5, 10, 15 and 20 from disk because e.g. it could start reading 5 from disk as soon as 20 has been loaded, and it won't need to have it in memory until 4 has finished executing - so it will have 5x as long to do the prefetch from disk to main memory before it needs to load it.

4. What llamacp or LM studio or ollama options does it support? K/V cache quantisation feels the most important because of the vRAM usage implications.

5. For unified memory systems (e.g. with Apple silicon or AMD Ryzen AI CPUs), can it use exactly the same techniques to keep only the executing layer in unified memory?

6. What capabilities are there to analyse and predict and report on the likely performance of a particular model on particular hardware?

   I don't want to spend time testing models when most of them will be unsuitable. And for those that are suitable, I would like to know whether I can get 90% of the quality with 5x the throughout with LLM A compared to LLM B e.g. because model A can have preloading of layers whilst model B can't. Ideally this would include performance comparisons between paged models and models that fit entirely into vRAM and don't need to be pushed and including paged models with some layers being CPU inference.

7. Do I need to prepare models locally and how long does that take? Can we have prepared models uploaded to HF to be used without local effort?

8. Do I need to write code to use RabbitLLM or can I load a model (like I can with llamacp or ollama or LM Studio) and call it with an Anthropic or OpenAI API? This is the biggest use case, so for significant take up of RabbitLLM, this would need to be supported.

9. And is there a docker image that I can use instead of Ollama? Dockerfile provided. But without an HTTP server, much less useful.

Thanks again for taking this on.

S

[Sophist-UK]

P.S. I appreciate why you worked AirLLM, and that this is the existing code base for this layer paging approach, but as a long term strategy would it be better to use llamacp as a base and add functionality to that?

The reason I am suggesting this is that llamacpp already had the ability to split layers between vRAM & RAM and the logic to decide how to prioritise them, so it should hopefully be reasonably easy to turn that from a static loading action into something more dynamic like RabbitLLM.

[Sophist-UK]

Please note that I have thunked further and have updated the original post with some new ideas.

[Sophist-UK]

I added the following points to #3 after Manuel commented that the current performance is 4mins/token:

As you clearly realise 4 minutes for 1 token is unusable (and slower than CPU inference).

My guess is that swapping layers in and out of memory takes substantially longer than the time for a single layer to execute.

But in essence my suggestions are as follows:

1. We should try to improve on the existing llama.cpp approach of loading most layers into vRAM and run the remaining few on CPU, by tweaking this approach to keep a most layers permanently in vRAM and swap a few in and out so that everything runs on GPU. This is a substantially different approach to swapping every layer in 1-by-1.

2. Running any layers on CPU or not allowing enough vRAM for context means using shared memory for the context and that in itself slows things down, so any strategy which allows all layers to execute in memory with the context also in vRAM without any wait states will likely be fastest. I suspect that if you need to put context in shared memory, then if you can run all layers on GPU without having to wait for them to load, then that will be next fastest. But that any delays for layers to load will be slower than executing those layers on CPU. But to know this...

3. We need to know the relative times for: a) executing a layer on GPU with context entirely in vRAM, b) executing a layer on GPU with context in shared memory, c) executing a layer on CPU with context entirely in normal RAM, d) executing a layer in CPU with context in shared memory, e) loading a layer into GPU from main memory, f) moving context between vRAM and shared memory or shared memory and normal memory.

4. These can then inform us as to what strategies are worth pursuing - to focus only on use cases where performance is better than can be achieved using normal mixed GPU / CPU execution.

5. Based on me doing some simple manual tests on my own GPU, loading a 5GB model takes several seconds (even if it was previously read and likely in the Windows disk cache. So presumably loading a 5GB layer will take a similar time. But I suspect that a single layer execution time is measured in 0.1, 0.01 or 0.001 seconds. This suggests to me that there may be ways to make layer swapping work, but they are much more likely to be ways to make a model that doesn't quite fit into vRAM run fully on the GPU by swapping out (say) 10% of the layers (which might make a substantial improvement in performance) cf. running a huge model where only one or two layers fit into vRAM.

6. So for example, suppose you have a model with 20 layers, but only 18 fit in GPU. This means that context will be in shared memory, 18 layers will be in vRAM and execute on GPU but slowly because of the context being in shared memory, and 2 layers will execute slowly on CPU. Performance will be much much lower than a slightly smaller model that can run entirely on the GPU with context entirely in vRAM. There would be little point simply in running 17 in vRAM to allow context to be in vRAM also, because the other layers will need context in shared memory. But if we stored 16 layers permanently in vRAM, plus context fully in vRAM, and could hold the 4 other layers in RAM, and were clever about choosing which 16 layers to store in vRAM e.g. 1-4, 6-9, 11-14, 16-19, then once layer 5 (which we swapped in) finishes executing, we can start to swap in layer 10, and we have whatever the execution time of 4 layers is to swap in only 1/18 of the vRAM.

7. Indeed, because we will be getting full GPU inference speeds, we can even afford to have short waits for layer 10 to be loaded and still get substantial performance improvements over a hybrid model.

8. Clearly the larger the context, the less vRAM you have to hold layers - and this is IMO another key factor. The Anthropic system prompt starts the context with c. 10,000 tokens which requires substantial vRAM for context before you add the user's context - and this might by itself crush any use case. (Is your 4 mins per token based on context being fully in vRAM? And what size context did this measurement have?) But there are use cases (e.g. agentic development with Pi) where the system context is c. 300 tokens which might have an entirely different outcome. The difficulty we have is that (AFAIK at least) we cannot use CLIs or UIs to drive models because we currently can only call from python code. Getting real-life use cases may be more difficult than if we had an API and could e.g. get existing API use cases to try out our approach in real-life cases.

9. Given a model and a context size, plus some rules of thumb derived from the one off benchmarks in point 3., it should be possible for RabbitLLM to estimate what performance gains it can achieve. Indeed, it should be possible to ask RabbitLLM to run a specific model in hybrid mode and in GPU swapping mode and measure whether there are performance gains or not.

10. When you are selecting which layers to keep permanently in memory, the choice might make a difference. Some layers are likely to run more slowly that others, and it might make sense to choose to hold these permanently in vRAM to allow the other layers more time to be loaded. Some layers may be smaller than others and thus load quickly. We probably need have some rules of thumb and / or some measurement capabilities to drive an optimisation algorithm.

Narrower Use Cases

The net result of the above is a guess that we probably need to narrow down the use cases where this can be beneficial.

• If we assume that we are only going to get performance gains if everything runs on GPU and with context also in GPU, this would allow us to focus on a less ambitious but more achievable significant incremental performance for models that are slightly too big for vRAM and which can currently run mixed GPU/CPU inference but slowly, and not try to achieve even reasonable performance for huge models that cannot otherwise run at all in available vRAM + RAM.

• Whilst it probably makes sense to start with a very small context in order to focus on other factors and whilst context size isn't a factor (i.e. I suspect that 4min/token is so bad that the context size doesn't matter), we need to bear this in mind from the start and plan to build a test harness that uses some realistic contexts so the measurements are more realistic.

Development strategy

I agree with your comment on not developing an API until performance gains have been proven.

But at that point you should probably ask whether it makes sense:

1. to add e.g. all the other functionality in llama.cpp to create a fully functional LLM runner, and maybe even need to create UI functionality like Ollama and LM Studio; or

2. to treat RabbitLLM as a prototype and have the additional functionality redeveloped as an enhancement to llama.cpp.

[ManuelSLemos]

Since I wanted to develop these features using other techniques, I created this project, which outperforms Ollama and vLLM. So I'm leaving this research repository here https://github.com/ferrumox/fox

[Sophist-UK]

Judging only by the ReadME, this looks like a VERY impressive piece of coding, synergizing the best ideas from multiple sources.

A couple of comments:

1, It is not clear whether Ferrumox uses the ideas from AirLLM or the ideas I put forwards above that RabbitLLM is trying out, but as the originator of those ideas I would be very happy for you to use them.

2, Any chance of supporting the Anthropic API as well?

[ManuelSLemos]

Here's my response:

1. Not all of Rabbit's features are ready yet; working with Rust has been quite a challenge. But many of the lessons learned—such as using CPU + GPU, KV-Cache, and other techniques—are already in place. These will be implemented over the course of future releases.

2. Yes, in version 1.1 I’ll start opening up the APIs to all kinds of tools—Claude, OpenClaws, and all the popular ones. Right now, OpenAI and Ollama are available for the first global test.

[Sophist-UK]

That sounds great!!!!