multiple llama-server processes to achieve the parallel sequence processing?

[magikRUKKOLA]

Is it possible to run several processes of llama-server at the 512GB machine with two processes that uses separate GPUs for the head and the KV-cache etc. of MoE LLM such as Kimi-K2.5 IQ3_K (~460GB total) to achieve the parallel processing -- so that instead of --parallel option and multiple problems related to it I could simply run several instances of llama-server with the same LLM so all the processes would share the same CPU offloaded parts of the LLM in question?

[EDIT]: If possible, I wonder what the total decode would be in such a case.

[ikawrakow]

This would require a new buffer type that makes the tensors loaded in RAM available to other processes via shared memory.

[magikRUKKOLA]

But ... what about the mmap and MAP_SHARED ?

[ikawrakow]

But ... what about the mmap and MAP_SHARED ?

Have you tried? The mmap system call uses MAP_SHARED, so theoretically it could work. One cannot use anything flags that would turn off mmap (-rtr, -mqkv, -muge come to mind).

[magikRUKKOLA]

Have you tried?

Unfortunately no, I haven't tried anything.

Details

I had been thinking regarding the usage of RPC-CUDA devices to say, speed up the prefill and reduce hence the need to increase the -b and -ub . So theoretically it should be possible to offload the prefill via the network (via the RPC-connected instance), dump the kv-cache from the there, load it up into the main instance (with load batch sizes) and proceed with the decode. This way it should be possible to optimally offload the layers of MoE MLA LLMs when dealing with small-sized GPUs and the split mode layer ...

[saood06]

-- so that instead of --parallel option and multiple problems related to it

What problems?

[magikRUKKOLA]

@saood06

What problems?

Constant VRAM consumption. With --parallel 2 the KV-cache allocation is such that one have to sacrifice the ctx length to fit two parallel sessions.
But in case if one have some extra GPU in the system it would be much cleaner just to on-demand allocate another llama-server which would use the shared RAM segments but will use some spare GPUs (possibly including the RPC i.e. RDMA). This way the ctx length that can be supported is always better with such config.
That is what I meant.

[saood06]

That is what I meant.

Thanks for the clarification.