[Feature]: Expose option to load new model weights from disk

[edbeeching]

🚀 The feature, motivation and pitch

In an async RL setting, we often want to perform fast generation with a vllm endpoint on a separate node and occasionally sync model weights from disk. It would be good if this option was available on the vllm endpoint.

Alternatives

SGLang already exposes this option: https://docs.sglang.ai/backend/native_api.html#Update-Weights-From-Disk

Additional context

No response

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[mgoin]

Hi @edbeeching can you see if this feature achieves what you need? #12084

We have been actively working on adding new features to better support RL workflows

[danya0123]

@mgoin it would be nice to allow unload model(to save gpu memory) /reload model too

#6566 can only unload lora
#3281 require to reprogram the entire http interface

[lewtun]

@mgoin thanks for the pointer to #12084 !

What Ed is referring to is whether this collective can be exposed in the OpenAI-compatible server as a dedicated endpoint. For context, we'd like to spin up a vllm server on N nodes and run training on M nodes. At each training step, we'd like to synchronise the weights so that the vllm server is generating from the current policy.

We did look at #12084, but it seems to require us to adopt ray which adds a rather high amount of complexity to trl