Per-Model Bring Up / Tear Down Hooks

[dinerburger]

Llama.cpp's llama-server allows users to save, and then restore, slot data at run-time. We can use this feature to persist the KV cache state on teardown, and then seamlessly reload it on bringup, allowing KV cache state to be persisted while another model (think RAG stack) does some work, and then recover for processing my the "master" LLM on the way back in.

I'm proposing allowing lifecycle hooks per-model, similar to that of the application lifecycle hooks. Right now the only lifecycle hook exposed at the application level is on_startup, but I'm proposing models have a broader spread. Probably:

[on_will_start (before application invocation), on_startup (after health check is passed), on_will_stop (before the termination is sent), on_stopped (after application is verified as being stopped)]. The content is up for debate; I could see something like:

models:
  fancymodel:
    on_will_stop:
      http:
        POST: http://localhost:${PORT}/slots/0?action=save
        POST: http://localhost:${PORT}/slots/1?action=save
        POST: http://localhost:${PORT}/slots/2?action=save

or arbitrary commands:

models:
  fancymodel:
    on_will_stop:
      cmd: /tmp/saveSlots.sh ${PORT}

But I'm open for debate on all this. Thanks for making this killer tool!!

[chrispaulm]

Any news on this topic? I think it would be awesome to have the possibility to save KV cache between swaps. 👍

[dinerburger]

I slopped up a little wrapper utility to do it, which obviated the need to make changes to llama-swap. https://github.com/dinerburger/llama-save-wrapper

[chrispaulm]

Thanks for the hint! I was just slopping my own wrapper server ;-) But I think it would be better, if it would be just possible to solve the whole topic in llama-swap. :-)

[mostlygeek]

A wrapper written in Go would be my preferred way to do this. Something like cmd/vllm-wrapper with the arg changes proposed in #979.

[chrispaulm]

I am happy to test the wrapper if it is ready in llama-swap 👍