IICP now supports MeshLLM as a distributed inference backend #979
RobLe3
started this conversation in
Show and tell
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
Hi MeshLLM maintainers and community,
We implemented MeshLLM support on the IICP side and wanted to validate that we
are using the intended public API boundary.
IICP is an open capability-discovery and routing protocol. An IICP provider can
use MeshLLM as its local inference runtime. The boundary is deliberately small:
What the IICP integration does:
/v1/modelsinventory;http://127.0.0.1:9337/v1gateway;model: "mesh"ensemble alias from ordinaryautomatic capability advertisement.
What it deliberately does not do:
The initial IICP compatibility profile has been exercised against a real
MeshLLM gateway, and a MeshLLM-backed provider is currently visible in the IICP
directory. For a user-facing overview of the connection and how to try it,
see the IICP community post.
Could you let us know whether
/readyz,/v1/models, and OpenAI-compatiblechat completions are the intended stable external boundary for integrations
such as this? We welcome corrections to any assumption we have made.
All reactions