Repository navigation
Replies: 1 comment
|
Hi, I'd like to build this myself and check the approach with you first, since it will be a larger change. A bit about me: I'm new to LLM internals and not a Python developer, so I'm working with an AI coding assistant. I'll make sure I understand and can explain every change, and I'll validate on real models as the contributing guide asks. What oMLX already has
What's missing for my case What I would contribute, in two steps
Questions
Validation: I'm on a Mac with 64 GB. I've already produced reference outputs for the smaller 9B JEV model with PyTorch, so I can compare oMLX against them exactly. For the 27B I'd report how much quantization changes the results on a sample of the published test set. |
Uh oh!
There was an error while loading. Please reload this page.
Hi, and thanks for oMLX! I use a lot, mostly because MTP makes it really fast.
What I'd like to do
I'd like one local model to be my personal assistant for both chatting and quick decisions: AutoTrust JEV-27B.
JEV-27B is the regular Qwen3.8-27B with a small add-on (a LoRA adapter of about 110M parameters):
Why in oMLX
What would help
Nice to have: for decisions, a way to limit the answer to a few tokens and read their probabilities, or JEV support in the existing /v1/systemone endpoint.
Background
Thanks!
All reactions