You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Cross-instance prefix cache transfer for load-balanced oMLX replicas
#5217
We run multiple independent oMLX servers serving the same model behind an external router. Keeping each conversation on the node holding its prefix cache saves a lot of prefill work. When that node is busy or the conversation needs to move, the destination has to prefill the long history again.
We would like to transfer a compatible cached prefix from one replica to another, then let the destination continue the conversation using its local cache. The router could compare transfer cost with waiting or re-prefilling before deciding to move a session.
Is there any follow-up work or a planned API for cache transfer between independent replicas? Would you welcome a PR that revisits #3615 against current main?
A useful first scope would be authenticated export/import of a completed prefix, including hybrid-model state, with strict model/tokenizer/runtime/cache-format compatibility checks and safe handling of RAM-only blocks and concurrent requests. Cross-node transfer could use TCP first, with RDMA as a later transport option. Benchmarks would compare transfer plus restore time against a cold prefill and verify that the imported prefix is actually reused.
I'm interested in contributing a PR once the scope and integration approach are agreed. Should this build on the artifact work in #3615, or is there a newer cache interface we should use?
reacted with thumbs up emoji reacted with thumbs down emoji reacted with laugh emoji reacted with hooray emoji reacted with confused emoji reacted with heart emoji reacted with rocket emoji reacted with eyes emoji
Uh oh!
There was an error while loading. Please reload this page.
We run multiple independent oMLX servers serving the same model behind an external router. Keeping each conversation on the node holding its prefix cache saves a lot of prefill work. When that node is busy or the conversation needs to move, the destination has to prefill the long history again.
We would like to transfer a compatible cached prefix from one replica to another, then let the destination continue the conversation using its local cache. The router could compare transfer cost with waiting or re-prefilling before deciding to move a session.
I found these related efforts:
Is there any follow-up work or a planned API for cache transfer between independent replicas? Would you welcome a PR that revisits #3615 against current main?
A useful first scope would be authenticated export/import of a completed prefix, including hybrid-model state, with strict model/tokenizer/runtime/cache-format compatibility checks and safe handling of RAM-only blocks and concurrent requests. Cross-node transfer could use TCP first, with RDMA as a later transport option. Benchmarks would compare transfer plus restore time against a cold prefill and verify that the imported prefix is actually reused.
I'm interested in contributing a PR once the scope and integration approach are agreed. Should this build on the artifact work in #3615, or is there a newer cache interface we should use?
All reactions