Running mlx-community/gpt-oss-20b-MXFP4-Q8 with 16GB

[mingodad]

Today I found that I can successfully ran https://huggingface.co/ggml-org/gpt-oss-20b-GGUF on a mac mini m4 with 16GB locally with llama.cpp (see here ggml-org/llama.cpp#15396) ( macmon reports: RAM 13.05 / 16 GB -- SWAP 0.04 / 1 GB):

llama-cli -hf ggml-org/gpt-oss-20b-GGUF --n-cpu-moe 12 -fa on -c 32768 --jinja --no-mmap

And I'm getting around 26 tokens/s (without need any workaround like https://github.com/ml-explore/mlx-lm/tree/main#large-models ) .

Then I've tried to test with https://huggingface.co/mlx-community/gpt-oss-20b-MXFP4-Q8 but failed ( macmon reports: RAM 12.9 / 16 GB -- SWAP 0.98 / 2 GB):

mlx_lm.chat --model mlx-community/gpt-oss-20b-MXFP4-Q8
/Users/mingo/Library/Python/3.9/lib/python/site-packages/urllib3/__init__.py:35: NotOpenSSLWarning: urllib3 v2 only supports OpenSSL 1.1.1+, currently the 'ssl' module is compiled with 'LibreSSL 2.8.3'. See: https://github.com/urllib3/urllib3/issues/3020
  warnings.warn(
Fetching 9 files: 100%|█████████████████████████████████████████████████████████████| 9/9 [00:00<00:00, 100129.27it/s]
[INFO] Starting chat session with mlx-community/gpt-oss-20b-MXFP4-Q8.
The command list:
- 'q' to exit
- 'r' to reset the chat
- 'h' to display these commands
>> when have you been built
[WARNING] Generating with a model that requires 11516 MB which is close to the maximum recommended size of 10922 MB. This can be slow. See the documentation for possible work-arounds: https://github.com/ml-explore/mlx-lm/tree/main#large-models
libc++abi: terminating due to uncaught exception of type std::runtime_error: [METAL] Command buffer execution failed: Insufficient Memory (00000008:kIOGPUCommandBufferCallbackErrorOutOfMemory)
zsh: abort      mlx_lm.chat --model mlx-community/gpt-oss-20b-MXFP4-Q8
mingo@MacMiniM4Dad bin % /Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/multiprocessing/resource_tracker.py:216: UserWarning: resource_tracker: There appear to be 1 leaked semaphore objects to clean up at shutdown
  warnings.warn('resource_tracker: There appear to be %d '

So the question is if llama.cpp managed to get it running (using the CPU performance cores and all GPU) could this project do it too and beat the speed of llama.cpp ?

[awni]

I think it should run if you set sudo sysctl iogpu.wired_limit_mb=14000 (e.g. set it to a larger limit, like 14GB or so).

[mingodad]

Thank you for reply !
As I said llama.cpp run it without need any special OS settings and it seems that the mlx program+model is using more memory than llama.cpp .

[awni]

I don't think it's using more memory. We just require the user to be explicit about when they want to use a lot of memory on the device.

[mingodad]

Thank you again for replay !
Look at the numbers on the first message:
llama.cpp : RAM 13.05 / 16 GB -- SWAP 0.04 / 1 GB
mlx_lm.chat: RAM 12.9 / 16 GB -- SWAP 0.98 / 2 GB

Somehow it got a peak to force a bigger swap.

[awni]

I see.. yes it might be too large for your machine.

[mingodad]

It's really at the limit, I'm using it as a headless server only for local LLM use but somehow pepole from llama.cpp figured out how to squeeze it and get reasonable speed .

[awni]

I would try setting the wired limit if you can and see if it works. But it will be quite close to the limit, even in headless mode the device needs some RAM for the OS. And you will be limited in terms of how much context you can use with the model.

[zcbenz]

Sorry but I'm closing non-essential issues so it would be possible for us to actually work on important items.