Repository navigation
Running Qwen locally on Apple Silicon as an agent backend with Rapid-MLX #1929
raullenchai
started this conversation in
Show and tell
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
Disclosure: I maintain Rapid-MLX, an Apache-2.0 local inference server for Apple Silicon.
For Qwen users who want to connect a local model to an agent, the useful path is:
Qwen MLX checkpoint → Rapid-MLX on a Mac → local OpenAI/Anthropic-compatible API → agent client.
A small-model starting point is:
Query
http://localhost:8000/v1/modelsfor the served model ID, then use the OpenAI-compatible base URLhttp://localhost:8000/v1in a client. Core inference is local after downloading weights. Mac memory and context requirements depend on the checkpoint and request; this is not a claim that every model fits every Mac.Installation/API setup and agent setup guides explain the client side. On the model side, our Mac mini qualification report documents Qwen3.5-4B and Qwen3.8-27B text generation on M2 Pro/32GB, including the environment, checkpoint identities, methods and limitations. Those are historical, workload-specific measurements; they are not new tests for this post and do not establish all-model tool-call reliability or multimodal performance.
Your docs already have Run Locally/MLX LM guidance. If a community serving example is useful, the contribution could focus on one pinned Qwen checkpoint, a local HTTP request, and a reproducible agent/tool-call check, with package versions and explicit text/vision boundaries. Happy to follow the preferred docs location after maintainer feedback.
I am using Show and tell because this is a community project/example, not a model bug. Rapid-MLX is not an official Qwen runtime. Anonymous metadata telemetry is enabled by default and can be disabled with
RAPID_MLX_TELEMETRY=0orrapid-mlx telemetry off. AI-assisted research and drafting.All reactions