Skip to content

b317

Latest

Choose a tag to compare

@github-actions github-actions released this 04 Sep 21:51
a52ce59

feat(serve): pool hardware across daemons as an aggregation (#405)

Name peer daemons in [aggregation] peers (or LLMMAN_PEERS) and a
request to any node is served by whichever has the model loaded, or the
most room to load it. Forwarded requests carry x-llmman-hop so they are
never forwarded again; ps, tags, models and stop cover every node.

Fixes #404.