Cookbook Model List

[pewdiepie-archdaemon]

Cookbook is a curated list of models.

If you have a request for models missing, please leave a comment below:

• Hf Link
• What hardware you ran on (optional)
• What engine (llama.cpp, vllm, etc)
• Reason to add

Thank you

[DeanLemans]

i think one of the fine tuned 'uncencored" ai should be added:
one of the dphn models would be great: https://huggingface.co/dphn
i personally use:

• https://huggingface.co/dphn/dolphin-2.9-llama3-8b
• https://huggingface.co/dphn/Dolphin-X1-Trinity-Nano

[CommanderTurtle]

I have just updated a solid 5090 setup for one of these, actually.

Would love to see further integration or (maybe if you know) LiteLLM filtering for said models. Anything honestly at this point to fix-up and filter runners that contain "unhinged" outputs. I was using DavidAUs' for a time, until eventually switching over to AEON's nvfp4 instructs for very big context that also worked with toolcalls (although, they were not as interesting for creative writing, like DavidAUs' reasoning window, which was indeed quite good)

If you have a 5090, 4090, or 5080, blackwell, a solid choice for these types of models is my repo where I seem to have gotten modelopt turbo running well on consumer hardware. That is, unquanted "but nvfp4" vision-capable Google Gemma4 with both chat templates preserved from David's "instruct" (i.e. toolcalling) and "non instruct" which just has an insanely detailed reasoning block of the model re-reasoning with itself. Around 170tok/s on Gemma A4B MoE model at four parallel 65k context chats capable with this kind of setup.

https://huggingface.co/sHEL1562/gemma-4-A4B-it-MoE-HERETIC-nvfp4-blackwell

[nutter77-fossmc]

I think lfm2.5 models should be added

its better then most qwen models around its weights

also running on m1 air base model 8gb ram 7gpu 8cpu 256 gb the very budget friendly MacBook Air m1 sold from Walmart

[pewdiepie-archdaemon]

Can you link so I know which one you mean, updated the original post

[nutter77-fossmc]

Yeah sure

I’ve got a few more I can link to they are people / orgs /authors tho
https://huggingface.co/Crownelius
https://huggingface.co/DavidAU
https://huggingface.co/Goekdeniz-Guelmez
https://huggingface.co/LiquidAI

Bit to sort through

also if it helps I think mobile apps might be cool for this

[RenanOSX]

• https://huggingface.co/HauhauCS/Qwen3.6-35B-A3B-Uncensored-HauhauCS-Aggressive (Q4_K_M)
• RTX 4060 8Gb + 32 DDR4 RAM (It bleeds a lot but I can run with a stable 22 tokens/s at 8k context window) [Fit: TOO TIGHT]
• llama.cpp
• Qwen performance but allows users to study cybersecurity and RE without refusals from default models (Good for coding and dissecting samples)

[Whitefirelion]

Would also be great if we got the uncensored qwen3.5-9b model, works great on my older harder
https://huggingface.co/HauhauCS/Qwen3.5-9B-Uncensored-HauhauCS-Aggressive (Q4_K_M)

[SPOOKEXE]

Qwopus models are really nice, qwen trained on opus traces;
https://huggingface.co/Jackrong/Qwopus3.6-27B-v2-MTP-GGUF
https://huggingface.co/Jackrong/Qwopus3.5-9B-Coder
https://huggingface.co/Jackrong/collections

[sacb0y]

Good uncensored Gemma 4 model that won't stop working cause it encounters spicy words >_>
https://huggingface.co/zerofata/G4-MeroMero-26B-A4B-gguf

[josephspurrier]

Gemma 4 12B

• HF Link: https://huggingface.co/google/gemma-4-12b
• Hardware: Runs on consumer laptops with 16GB VRAM or unified memory
• Engine: Ollama, LM Studio, vLLM
• Reason to add: Google's first mid-sized model with native audio inputs alongside text and vision. Encoder-free architecture (no separate vision/audio encoders) reduces latency. Performance nears their larger 27B MoE model at less than half the memory footprint. Includes Multi-Token Prediction drafters for faster inference. Apache 2.0 license.

Blog post: https://blog.google/innovation-and-ai/technology/developers-tools/introducing-gemma-4-12B/

[AbishekKoushik]

Gemma 4 12B

1. HF LINK: google/gemma-4-12B-it-qat-q4_0-gguf
2. Hardware: Runs on comsumer gpu with 12GB VRAM

[DericHuynh]

Qwen3.5 Mixture of Quants + Multi-Token-Prediction

1. HF LINK: w-ahmad/Qwen3.5-9B-GGUF-MoQ-MTP
2. Hardware: Runs on consumer GPU with 8GB VRAM

Specifically moq-3.6, runs on 8GB Vram, fast generation, modern model with quantization that is 97% accuracy against baseline. A generally good model for constrained hardware.

[T-Bozz28]

HI,
Anyone have an idea why i can't see models?
"Couldn't scan device" error

[deucebucket]

https://huggingface.co/deucebucket/Qwen3.6-35B-A3B-Cerebellum-GGUF
Cerebellum v3 Qwen3.6-35B-A3B-Cerebellum-v3.gguf, 11 GB

https://huggingface.co/deucebucket/Gemma-4-26B-A4B-it-Heretic-Cerebellum-GGUF
gemma-4-26B-A4B-it-heretic-cerebellum-v1.1-templatefix.gguf, 11.7 G

https://huggingface.co/deucebucket/Qwen3.6-27B-Cerebellum-GGUF
Ablation-informed mixed-precision quantization of, 12 GB file size, 7.034 perplexity, 181 per-tensor quant overrides.

[rendika695]

put uncensored to model filter

[Stoltverd]

Hf Link: https://huggingface.co/zai-org/GLM-5.2
What engine (llama.cpp, vllm, etc): vllm
Reason to add: Solid 1M Context

[IanADN]

• Hf Link: https://huggingface.co/unsloth/gemma-4-E4B-it-GGUF
• What hardware you ran on (optional): M1 2021 32GB Macbook Pro
• What engine (llama.cpp, vllm, etc): llama.cpp
• Reason to add: Its not available yet but its previous versions are (e.g. gemma-3n-e2b), no reason to not add to cookbook if generally safe.

[CommanderTurtle]

Worth looking at both the Agents A1 project as well as the Qwen Agentworld development. Both came out as recent pushes from official qwen team as well as a science lab for what's best for agents (over the last 2 weeks)

Submitted pewdiepie-archdaemon/odysseus/discussions/5493 on this, as well as it being a fully curated vllm command guide for WSL 5090 owners. I opt for debian myself, as it's fairly straightforward. And, minimalist windows if you're really privacy-focused these days puts in some decent work on the privacy front, especially if you don't mind group policy and correct opts in telemetry. So a mini cookbook is on that discussion page for any card with over 24gb vram. Might have to adjust the quant slightly (int3 or something) but int4 is what I use. Would need two 5090s for fp8.

Agents A1 has documentation about the science lab here 9:10 in the video

[gprocunier]

@pewdiepie-archdaemon models I have configured with Odysseus and tested as working with tool use:

https://huggingface.co/zai-org/GLM-5.2 BF16 non quantized, and quantized FP8 and Int4/Int8 variants.
https://huggingface.co/nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16
https://huggingface.co/thinkingmachines/Inkling
https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro

• Hardware: HPE ProLiant DL380 Gen12, 256 logical CPUs, ~2.0 TiB RAM, and 16× NVIDIA H200 141GB GPUs (~2.25 TB HBM), arranged as four 4-GPU NVLink islands over CerIO/PCIe. Models use TP4/PP4/EP4 and a 9 TiB CephFS cache.
• Platform: OpenShift 4.21.15, Red Hat OpenShift AI 3.4.2 (vllm).
• GLM-5.2 BF16 256K: custom vLLM 0.25.0rc3-derived runtime
• Nemotron 3 Ultra 550B BF16 256K: custom vLLM 0.25.0rc4.dev0 runtime, pinned to rc3 commit 3d1c21a6

Should have Mistral Large 3 675B FP8 tires kicked in the next week or so.

I have been exploring agentic ops/sre assist using local models similar to what Anthropic and OpenAI do with Claude.ai and chatgpt.